A pallet recognition method based on 3D camera

Abstract

The present invention discloses a pallet recognition method based on a 3D camera, comprising the following steps: firstly, the RGB image and the depth image of the pallet to be recognized are collected, and the images are transmitted to the front-end Yolov5 neural network model for pallet recognition detection, and then the region of interest ROI of the pallet is determined according to the recognition result of the front-end neural network model, and the right line of the left jack of the pallet and the left line of the right jack are respectively detected by sliding to both sides of the ROI image according to the center of the ROI, and then the intersection of the side lines of the jacks on both sides and the upper side lines is calculated, and the center point of the two intersections is taken as the midpoint of the pallet. Finally, the coordinates of the center point of the pallet in the camera coordinate system are calculated, and the heading of the pallet in the camera coordinate system is obtained by combining the depth image. The present invention combines the RGB image and the depth image, and utilizes the advantages of the two image information to complete the pallet posture detection, reduce the influence of environmental factors on the pallet detection, improve the pallet detection rate, and ensure a high detection speed at the same time, which is suitable for complex industrial environments.

Description

Technical Field

[0001] The invention belongs to the field of unmanned forklift target recognition, and in particular relates to a pallet recognition method based on a 3D camera. Background Art

[0002] With the large-scale application of unmanned forklifts in smart warehousing and logistics, the increasingly complex working scenes have put forward higher requirements for the functions of unmanned forklifts. Among them, the ability to adaptively identify pallets in complex environments has become one of the key functions of unmanned forklifts, which also has a significant impact on the automation level and operating efficiency of the entire warehousing or logistics system.

[0003] Existing pallet recognition solutions are mainly divided into two types: laser and camera according to the type of sensor used. Among them, the laser solution mainly collects pallet point clouds and uses point cloud matching to achieve pallet recognition. Its advantages are that it is not affected by ambient light and has good stability, but it has problems such as large amount of calculation, slow recognition speed and high cost. The camera solution mainly uses a 3D camera (RGB-D camera) to collect the depth image of the pallet, generates a point cloud template based on the size information of the standard pallet, and then uses the template to match the point cloud collected by the camera to complete the pallet recognition. Its advantages are low cost and high recognition accuracy, but it also has problems such as the need to set multiple types of templates, large amount of calculation, and difficulty in realizing rapid forking by unmanned forklifts. Summary of the invention

[0004] The present invention mainly solves the problem that the existing technology has a large amount of calculation and cannot quickly identify pallets. A pallet identification method based on a 3D camera is provided, which uses an edge detection algorithm to extract pallet features and key points in an RGB image, and then uses depth information to complete the calculation of the pallet position and heading. The embodiments of the present invention can be applied to indoor and outdoor unmanned forklifts.

[0005] The technical solution proposed by the present invention includes the following contents:

[0006] A pallet recognition method based on a 3D camera, the recognition method comprising the following steps:

[0007] Step 1: Acquire an original RGB image and a depth image of the pallet to be inspected from a 3D camera, and preprocess the original image to obtain a first image;

[0008] Step 2: passing the first image to the front-end Yolov5 neural network model for pallet detection to obtain the pallet detection result;

[0009] Step 3: Determine the pallet target ROI according to the pallet detection result, obtain the upper side, left and right side edges of the pallet by using an image processing method, and determine the left and right vertex coordinates of the pallet;

[0010] Step 4: Calculate the coordinates of the center point of the pallet target ROI based on the coordinates of the left and right vertices of the pallet;

[0011] Step 5: According to the center point of the tray target ROI, slide to the left and right sides to detect the left and right jack edges;

[0012] Step 6: Use the intersection of the left and right jack edges and the upper side edge to determine the center point of the tray;

[0013] Step 7: Calculate the coordinates of the center point of the tray, the intersection of the left and right jack edges and the upper edge line in the camera coordinate system according to the camera intrinsic parameters and the depth value corresponding to the center point of the tray.

[0014] Step 8: Calculate the pallet heading based on the coordinates of the intersection of the left and right socket edges and the upper edge.

[0015] Furthermore, in step 1, a 3D camera is used to obtain an RGB image and a depth image of the tray as two images for data alignment.

[0016] Furthermore, the preprocessing in step 1 is: removing outliers through Gaussian filtering to remove interference and noise.

[0017] Furthermore, the front-end Yolov5 neural network model in step 2 is pre-trained with a large amount of pallet image data, and its output detection results include the pallet detection evaluation value, and the coordinates of the upper left corner and lower right corner of the pallet target in the RGB image.

[0018] Furthermore, in step 3, a pallet target ROI is determined in the RGB image according to the pallet detection result, and subsequent image processing is performed in the pallet target ROI area.

[0019] Furthermore, in step 3, image processing is performed on the target ROI area of ​​the tray to obtain the upper side and left and right side lines of the tray, and the left and right vertices of the tray are obtained by intersecting the upper side line with the left and right side lines.

[0020] Furthermore, in step 4, the center point of the upper edge of the pallet is calculated according to the coordinates of the left and right vertices of the pallet, and then the center point of the pallet target ROI is determined by moving along the normal line of the upper edge toward the pallet target ROI.

[0021] Furthermore, in step 5, starting from the center point of the target ROI of the tray, slide left and right along the upper side line of the tray to detect the right side line of the left plug hole and the left side line of the right plug hole.

[0022] Furthermore, in step 6, the intersection points of the right line of the left plug hole, the left line of the right plug hole and the upper side line of the tray are calculated respectively, and then the center point of the tray is obtained according to the two intersection points.

[0023] Furthermore, in step 7, the camera imaging principle and the camera internal parameters are used to obtain the coordinates of the intersection of the center point of the tray, the right line of the left plug hole, the left line of the right plug hole and the upper side line of the tray in the camera coordinate system using the following formula:

[0024] ;

[0025] In the formula, is the point coordinate in the camera coordinate system, is the point coordinate in the pixel coordinate system, is the depth information of the point, and Represent the pixel coordinates of the principal point, and Respectively represent the focal length in the x and y directions in the camera coordinate system.

[0026] Further, in step 8, the pallet heading is calculated based on the coordinates of the intersection of the right line of the left jack, the left line of the right jack and the upper side line of the pallet. :

[0027] ;

[0028] Where ( , ) is the coordinate of the intersection of the right line of the left socket and the upper side line of the tray, ( , ) is the coordinate of the intersection of the left line of the right socket and the upper side line of the tray.

[0029] Compared with the prior art, the technical solution of the present invention has the following advantages:

[0030] Compared with the method of using deep point cloud for template matching, the Yolov5 neural network model is used to identify the pallet target, which effectively solves the problem of quickly identifying the pallet in a complex environment and shortens the pallet positioning time. Secondly, in the process of extracting pallet features, the detection range is limited to the area where the pallet target is output by the neural network model, which greatly reduces the amount of data calculation, improves the real-time performance of pallet identification, and provides favorable conditions for unmanned forklifts to quickly pick up pallets. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1A flow chart of a pallet recognition method based on a 3D camera according to the present invention;

[0033] Figure 2 The pallet target ROI is determined according to the neural network recognition result of the present invention;

[0034] Figure 3 This is a schematic diagram of the upper side edge detection of a pallet according to the present invention;

[0035] Figure 4 This is a schematic diagram of the left and right side line detection of the pallet of the present invention;

[0036] Figure 5 This is a schematic diagram of marking the left and right vertices of the tray of the present invention;

[0037] Figure 6 This is a schematic diagram of the tray ROI center mark of the present invention;

[0038] Figure 7 It is a schematic diagram of the detection of the edge lines of the left and right insertion holes of the tray according to the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described in detail below by taking a depth camera sensor as an example. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The invention discloses a pallet recognition method based on a 3D camera. The method comprises the following steps: firstly, the RGB image and the depth image of the pallet to be recognized are collected by using a 3D camera, and noise reduction processing is performed. Then, the RGB image is transmitted to the front-end Yolov5 neural network model for pallet recognition detection, and then the recognition result is transmitted to the pallet feature back-end detection module, which determines the pallet region of interest ROI in the RGB image according to the recognition result of the front-end neural network model, and then uses Gaussian filtering, canny edge detection operator and Hough transform and other methods to process the image, extract the upper side and left and right side lines of the pallet, determine the left and right vertices of the pallet, and then preliminarily determine the midpoint of the upper side line of the pallet according to the left and right vertices of the pallet, and determine the center of the pallet ROI downward along the normal direction of the upper side line of the pallet according to the midpoint position, and then slide to the left and right sides of the ROI image respectively according to the ROI center to detect the right side line of the left jack of the pallet and the left side line of the right jack, and then calculate the intersection of the side lines of the jacks on both sides and the upper side line, and take the center point of the two intersections as the midpoint of the pallet. Finally, the camera internal parameters are used to calculate the coordinates of the center point of the pallet in the camera coordinate system, and the heading of the pallet in the camera coordinate system is obtained by combining the depth image.

[0041] Embodiment 1:

[0042] Please refer to Figure 1 , Figure 1 A flowchart of a pallet recognition method based on a 3D camera provided in this application includes:

[0043] S1: The 3D camera used in this embodiment is a FM851-E2 depth camera, which outputs RGB image and depth image data. First, the camera is used to collect the RGB image and depth image of the pallet to be identified, and then the image is preprocessed, that is, outliers are removed.

[0044] S2: The RGB image is passed to the pre-trained Yolov5 neural network model for pallet inspection to obtain the pallet detection result (the coordinates of the upper left vertex and the lower right vertex of the pallet target rectangular area). The neural network model is pre-trained by a data set of actual images of multiple pallets of the same type after data amplification.

[0045] S3: According to the neural network recognition results, determine the tray target ROI (Region of Interest) in the RGB image, please refer to Figure 2 , Figure 2 The black rectangular area is the ROI of the tray to be identified in the frame image. The area is segmented by Otsu threshold, and then image Gaussian filtering is performed, and canny operator edge extraction is performed. Then, the Hough transform straight line method is used to remove the straight lines with less pixel content to filter out the lower side interference lines of the tray, and then the least squares method is used to accurately fit the upper side line of the tray. Please refer to Figure 3 .

[0046] S3: Perform edge detection from both left and right directions in the left area of ​​ROI to extract the left edge of the pallet outline. Similarly, perform edge detection from both left and right directions in the right area of ​​ROI to extract the right edge of the pallet outline. Please refer to Figure 4 ;

[0047] S3: Calculate the intersection points of the extracted upper side line of the pallet with the left side line and the right side line as the left and right vertices of the pallet, and verify by calculating the ratio of the pixel distance between the two intersection points and the overall width. Please refer to Figure 5 .

[0048] S4: Calculate the center point of the upper edge of the pallet based on the left and right vertices of the pallet, and then move along the normal line of the upper edge to the target ROI of the pallet to determine the center point of the ROI. Please refer to Figure 6 .

[0049] S5: Move from the center point of ROI to the left along the upper line to extract the right line of the left jack. Move from the center point of ROI to the right along the upper line to extract the left line of the right jack. Please refer to Figure 7 .

[0050] S6: Calculate the intersection points of the right line of the left side plug hole, the left line of the right side plug hole and the upper line of the tray respectively, and record them as the center point of the tray.

[0051] S7: Using the camera imaging principle and camera internal parameters, the coordinates of the center point of the pallet, the right line of the left socket, the intersection of the left line of the right socket and the upper side line of the pallet in the camera coordinate system are obtained using the following formula:

[0052] ;

[0053] In the formula, is the point coordinate in the camera coordinate system, is the point coordinate in the pixel coordinate system, is the depth information of the point, and They represent the pixel coordinates of the principal point of the image. The principal point is the vertical point from the camera center to the photo, that is, the intersection of the main optical axis and the photo, also called the principal point of the image. and Respectively represent the focal length of the camera in the x and y directions. In this embodiment, , , , .

[0054] S8: The pallet heading is calculated based on the coordinates of the intersection of the right line of the left socket, the left line of the right socket and the upper side line of the pallet. .

[0055] ;

[0056] Where ( , ) is the coordinate of the intersection of the right line of the left socket and the upper side line of the tray, ( , ) is the coordinate of the intersection of the left line of the right socket and the upper side line of the tray.

[0057] It should be noted that the parts of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.

[0058] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pallet recognition method based on a 3D camera, characterized in that: The identification method comprises the following steps: Step 1: Acquire an original RGB image and a depth image of the pallet to be inspected from a 3D camera, and preprocess the original RGB image to obtain a first image; Step 2: Pass the first image to the front-end Yolov5 neural network model for pallet detection to obtain the pallet detection result; Step 3: Determine the pallet target ROI based on the pallet detection results, obtain the upper side, left and right side edges of the pallet using image processing methods, and determine the coordinates of the left and right vertices of the pallet; Step 4: Calculate the coordinates of the center point of the pallet target ROI based on the coordinates of the left and right vertices of the pallet; Step 5: Based on the center point of the tray target ROI, slide to the left and right sides to detect the left and right jack edges; Step 6: Use the intersection of the left and right jack edges and the upper side edge to determine the center point of the tray; Step 7: Based on the camera intrinsic parameters and the depth value corresponding to the center of the tray, calculate the coordinates of the center of the tray, the intersection of the left and right jack edges, and the upper edge in the camera coordinate system; Step 8: Calculate the pallet heading based on the coordinates of the intersection of the left and right socket edges and the upper side edge.

2. A pallet recognition method based on a 3D camera according to claim 1, characterized in that: In step 1, a 3D camera is used to obtain an RGB image and a depth image of the tray as two images for data alignment.

3. The method for pallet recognition based on a 3D camera according to claim 1, characterized in that: The front-end Yolov5 neural network model in step 2 is pre-trained using pallet image data, and its output detection results include the pallet detection evaluation value and the coordinates of the upper left corner and lower right corner of the pallet target area in the RGB image.

4. The method for pallet recognition based on a 3D camera according to claim 1, characterized in that: In step 3, a pallet target ROI is determined in the RGB image according to the pallet detection result, and subsequent image processing is performed in the pallet target ROI area.

5. The method for pallet recognition based on a 3D camera according to claim 1, characterized in that: In step 3, image processing is performed on the target ROI area of ​​the tray to obtain the upper side and left and right side lines of the tray, and the left and right vertices of the tray are obtained by intersecting the upper side line with the left and right side lines.

6. The method for pallet recognition based on a 3D camera according to claim 1, characterized in that: In step 4, the center point of the upper edge of the pallet is calculated according to the coordinates of the left and right vertices of the pallet, and then the center point of the pallet target ROI is determined by moving along the normal line of the upper edge toward the pallet target ROI.

7. The method for pallet recognition based on a 3D camera according to claim 1, characterized in that: In step 5, starting from the center point of the target ROI of the tray, the tray slides leftward and rightward along the upper side line of the tray to detect the right line of the left jack and the left line of the right jack.

8. The method for pallet recognition based on a 3D camera according to claim 1, characterized in that: In step 6, the intersection points of the right line of the left plug hole, the left line of the right plug hole and the upper side line of the tray are calculated respectively, and then the center point of the tray is obtained according to the two intersection points.

9. The method for pallet recognition based on a 3D camera according to claim 1, characterized in that: In step 7, the camera imaging principle and camera intrinsic parameters are used to obtain the coordinates of the center point of the tray, the right line of the left jack, the left line of the right jack, and the intersection of the upper side line of the tray in the camera coordinate system using the following formula: ； Where, is the point coordinate in the camera coordinate system, is the point coordinate in the pixel coordinate system, is the depth information of the point, and Represent the pixel coordinates of the principal point, and Represents the focal length in the x and y directions in the camera coordinate system respectively.

10. The method for pallet recognition based on a 3D camera according to claim 1, characterized in that: In step 8, the pallet heading is calculated based on the coordinates of the intersection of the right line of the left jack, the left line of the right jack and the upper side line of the pallet. : ； Where ( , ) is the coordinate of the intersection of the right line of the left socket and the upper side line of the tray, ( , ) is the coordinate of the intersection of the left line of the right socket and the upper side line of the tray.

Images