Pallet pose estimation method and pallet pose estimation system

Abstract

The invention provides a pallet pose estimation method and system, and the method comprises the steps: obtaining a color image which comprises a pallet region and at least part of pixels of which are aligned, and point cloud data, and cutting the color image to obtain a target image which only comprises the pallet region; obtaining a point cloud of an area corresponding to the target image from the point cloud data according to an alignment relationship between the color image and the point cloud data, and obtaining a to-be-fitted point cloud; performing plane fitting on the to-be-fitted point cloud to obtain a plane fitting model; angle information and center coordinates of the pallet are calculated based on the plane fitting model and a preset reference coordinate system, and the angle information and the center coordinates of the pallet are utilized to construct a pose matrix of the pallet to obtain pose information of the pallet. The pallet pose estimation method provided by the invention has higher robustness and higher accuracy.

Description

Technical Field

[0001] The present application relates to the field of image deep learning, and more specifically, to a pallet pose estimation method and a pallet pose estimation system. Background Art

[0002] Forklifts need to identify and predict the pallet's pose during pallet-handling operations to assist forklift operators in more efficient operations. Related solutions combine pallet image processing and point cloud information to predict the pallet's pose. However, these solutions require auxiliary markers and / or use non-universal templates, resulting in poor generalization. Summary of the Invention

[0003] The present application provides a pallet pose estimation method and a pallet pose estimation system, which can have stronger robustness and higher accuracy when estimating the pallet pose.

[0004] In a first aspect, a method for estimating a pallet pose is provided, comprising: obtaining a color image and point cloud data including a pallet area and having at least partial pixel alignment, cropping the color image to obtain a target image including only the pallet area; obtaining a point cloud of an area corresponding to the target image from the point cloud data based on an alignment relationship between the color image and the point cloud data to obtain a point cloud to be fitted; performing plane fitting on the point cloud to be fitted to obtain a plane fitting model; calculating the angle information of the pallet and the center coordinates of the pallet based on the plane fitting model and a preset reference coordinate system, and constructing a pose matrix of the pallet using the angle information of the pallet and the center coordinates of the pallet to obtain the pose information of the pallet.

[0005] Based on the above technical solution, the plane fitting model used to obtain the pallet pose matrix can be directly obtained by plane fitting of the point cloud to be fitted corresponding to the target image, so that the pose information of the pallet can be calculated without the need for external markers and / or specific templates on the pallet, thereby improving the robustness and accuracy of the pallet pose estimation method, further improving the generalization ability of the pallet pose estimation method, and reducing the deployment cost of the pallet pose estimation method.

[0006] In combination with the first aspect, in certain implementations of the first aspect, a point cloud of an area corresponding to a target image is obtained from the point cloud data based on an alignment relationship between the color image and the point cloud data to obtain a point cloud to be fitted, including: performing a pallet quality judgment on the color image or the target image to determine whether to obtain the point cloud to be fitted based on the judgment result; when the pallet quality in the color image or the target image is that the pallet is complete, a point cloud of an area corresponding to the target image is obtained from the point cloud data based on the alignment relationship between the color image and the point cloud data to obtain the point cloud to be fitted.

[0007] Based on the above technical solution, a more appropriate process execution action can be selected in combination with the actual state of the pallet, thereby providing the operator with effective pallet posture information while saving process action overhead.

[0008] In combination with the first aspect, in certain implementations of the first aspect, plane fitting is performed on the point cloud to be fitted to obtain a plane fitting model, including: preprocessing the point cloud to be fitted to obtain a first point cloud; plane fitting is performed on the first point cloud to obtain a plane fitting model; wherein the preprocessing includes at least one or a combination of the following: pass-through filtering, downsampling, outlier removal, cluster filtering, filtering according to the normal direction of the point cloud to be fitted, and distance filtering of the clustered point cloud in the normal direction of the point cloud to be fitted.

[0009] Based on the above technical solution, before plane fitting is performed on the point cloud to be fitted, the points in the point cloud to be fitted are preprocessed in advance, thereby improving the confidence of the point cloud to be fitted, and further improving the accuracy of the plane fitting model obtained by subsequent plane fitting of the point cloud to be fitted.

[0010] In combination with the first aspect, in certain implementations of the first aspect, a plane fitting is performed on the first point cloud to obtain a plane fitting model, including: randomly selecting several points in the first point cloud to fit the first plane and obtain the corresponding first plane equation, and calculating the first distance of each remaining point in the first point cloud to the first plane one by one through the coordinates of each point in the first point cloud and the first plane equation corresponding to the first plane; if the first distance of the point in the first point cloud is less than or equal to a preset distance threshold, it means that the point is an inner point; conversely, if the first distance is greater than the preset distance threshold, it means that the point is not an inner point, and after traversing each point in the first point cloud, the inner points are collected to obtain a first inner point set; based on the first inner point set, a second plane is fitted and the corresponding second plane equation is obtained, and the second distance of each remaining point in the first inner point set to the second plane is calculated through the coordinates of each point in the first inner point set and the second plane equation corresponding to the second plane, and the second distance is compared with the preset distance threshold to update the first inner point set to obtain a second inner point set; plane fitting and updating are performed on the second inner point set until a preset number of updates is reached to obtain a third inner point set, and the plane fitting model is constructed using the third inner point set.

[0011] Based on the above technical solution, plane fitting is performed on the points in the fitting point cloud, so that the points in the final plane fitting model can better fit the actual pallet posture, thereby improving the accuracy of pallet posture estimation.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the normal vectors of the plane fitting model and the reference plane of the preset reference coordinate system are obtained, and the angle between the plane fitting model and the reference plane of the preset reference coordinate system is calculated through the normal vector to obtain the first angle information of the stack; the plane fitting model is projected onto a specific plane in the reference plane to form a two-dimensional image, and the minimum circumscribed rectangle of the two-dimensional image is calculated to obtain the second angle information and the center coordinates of the stack through the minimum circumscribed rectangle; and the posture matrix of the stack plane is constructed according to the first angle information, the second angle information and the center coordinates of the stack to obtain the posture information of the stack.

[0013] In combination with the first aspect, in certain implementations of the first aspect, the first angle information of the stack includes a yaw angle between the plane fitting model and the YOZ reference plane of the preset reference coordinate system, and a pitch angle between the plane fitting model and the XOY reference plane of the preset reference coordinate system; the second angle information of the stack includes a minimum circumscribed rectangle corresponding to the plane fitting model and a roll angle of the horizontal coordinate axis of the preset reference coordinate system.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the method for obtaining the yaw angle or the pitch angle includes: defining the plane coefficient of the YOZ reference plane of the preset reference coordinate system where the plane fitting model is located as (1, 0, 0, 0), or defining the plane coefficient of the XOY reference plane of the preset reference coordinate system where the plane fitting model is located as (0, 0, 1, 0), where the plane coefficient is used to represent the coefficient of the equation of a plane; obtaining the first normal vector of the plane fitting model and the second normal vector of the YOZ reference plane or the third normal vector of the XOY reference plane, calculating the first dot product and the first modulus of the first normal vector and the second normal vector, or calculating the second dot product and the second modulus of the first normal vector and the third normal vector; calculating the first cosine value of the yaw angle according to the first dot product and the first modulus; or calculating the second cosine value of the pitch angle according to the second dot product and the second modulus; and using the first cosine value or the second cosine value to solve the inverse cosine function to obtain the yaw angle or the pitch angle.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the roll angle is obtained by projecting the plane fitting model onto a YOZ specific plane in a preset reference coordinate system reference plane to form a two-dimensional image; calculating the minimum circumscribed rectangle of the two-dimensional image, and calculating the counterclockwise rotation angle of the relatively wider side of the minimum circumscribed rectangle relative to the horizontal line of the YOZ specific plane to obtain the roll angle.

[0016] In conjunction with the first aspect, in certain implementations of the first aspect, a method for obtaining the center coordinates of the stack includes: obtaining a plane equation of a plane fitting model and coordinates corresponding to the center point of a minimum circumscribed rectangle, calculating an x-coordinate value corresponding to the center coordinates of the stack using the coordinates corresponding to the center point of the minimum circumscribed rectangle and the plane equation of the plane fitting model; and obtaining the center coordinates of the stack by combining the y-coordinate value and the z-coordinate value corresponding to the center point of the minimum circumscribed rectangle.

[0017] Based on the above technical solution, the pallet pose matrix can be obtained directly by fitting a plane model corresponding to the target image without the need for external markers and / or specific templates on the pallet, thereby improving the generalization ability of the pallet pose estimation method and reducing the deployment cost of the pallet pose estimation method.

[0018] In combination with the first aspect, in certain implementations of the first aspect, the color image is cropped to obtain a target image that only includes the pallet area, including: performing semantic segmentation on the color image to identify the area where the pallet is located to obtain a bounding box and a mask image that only includes the pallet area; and using the bounding box and the mask image to crop the color image to obtain a target image that only includes the pallet area.

[0019] Based on the above technical solution, the target image including the pallet area can be obtained by cropping the input image.

[0020] In the second aspect, a pallet posture estimation system is provided, which is applied to a forklift. The system includes: a two-dimensional visual sensor for collecting color images including the pallet area; a three-dimensional visual sensor for collecting point cloud data including the pallet area; and a main control chip for processing the color image and point cloud data according to the method of the first aspect to obtain the pallet posture information.

[0021] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores program code, and when the computer program code is run on a computer, the computer executes the method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the architecture of a pallet pose estimation system provided in an embodiment of the present application.

[0023] Figure 2 A schematic flow chart of a pallet pose estimation method 100 provided in an embodiment of the present application.

[0024] Figure 3 A schematic diagram of the architecture of a preset segmentation network model 300 provided in an embodiment of the present application.

[0025] Figure 4 A flow chart of a pallet quality determination method provided in an embodiment of the present application.

[0026] Figure 5 A schematic diagram of the architecture of a preset classification network provided in an embodiment of the present application.

[0027] Figure 6 A schematic flow chart of a method for estimating the position and posture of a pallet based on a plane fitting model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solution in this application will be described below with reference to the accompanying drawings.

[0029] In the description of the embodiments of this application, unless otherwise specified, " / " represents or. For example, A / B can represent A or B. "And / or" in this application is simply a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In this application, "at least one" means one or more, and "more than one" means two or more.

[0030] In the embodiments of this application, prefixes such as "first" and "second" are used only to distinguish different description objects and have no limiting effect on the position, order, priority, quantity, or content of the described objects. The use of prefixes such as ordinal numbers in the embodiments of this application to distinguish description objects does not constitute a limitation on the described objects. For a statement of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary limitation.

[0031] To simplify the drawings, the drawings in the embodiments of this application schematically illustrate only the portions relevant to the corresponding embodiments and do not represent the actual structure of the products. Furthermore, to simplify the drawings and facilitate understanding, some drawings schematically depict only a portion of the structures or components; in practice, more or fewer identical or similar structures or components may exist.

[0032] Some related technologies require forklifts to identify and predict the pallet's pose during pallet-related operations, thereby assisting forklift operators in making more efficient operations. Related solutions combine pallet image processing information with point cloud information to predict the pallet's pose. However, these solutions require auxiliary markers and / or use non-universal templates, resulting in poor generalization.

[0033] Based on this, this application proposes a pallet pose estimation system, which is applied to forklifts, such as Figure 1As shown, the system includes a main control chip 10, a two-dimensional (2D) vision sensor 20 and a three-dimensional (3D) vision sensor 30, wherein: the main control chip 10 is coupled with the 2D vision sensor 20 and the 3D vision sensor 30, the 2D vision sensor 20 is used to collect color images including the pallet area, the 3D vision sensor 30 is used to collect point cloud data including the pallet area, and the main control chip 10 is used to read the color images and point cloud data of the 2D vision sensor 20 and the 3D vision sensor 30 to execute the pallet pose estimation method provided by one or more embodiments of the present application to obtain the pallet pose information.

[0034] Optionally, the 2D vision sensor 20 may be a color camera, and the captured color image may be an RGB image or a YUV image including a complete pallet area, or an RGB image or a YUV image including a partial pallet area.

[0035] Optionally, the 3D vision sensor 30 may be a depth camera and / or a lidar, the depth camera may be one or more combinations of a structured light camera, a time-of-flight camera, or a binocular camera, the depth camera and the color camera may be separately or integrated into one, which is not limited in this application.

[0036] In one possible scenario, when the 3D vision sensor 30 is a depth camera, the 3D vision sensor 30 first obtains a depth image, that is, the depth camera is used to collect a depth image including the pallet area. Furthermore, the main control chip needs to convert the depth image into point cloud data for subsequent processing.

[0037] In another possible scenario, when the 3D vision sensor 30 is a lidar, the 3D vision sensor 30 directly obtains point cloud data. Specifically, the lidar is used to collect point cloud data encompassing the pallet area. Furthermore, when the lidar outputs point cloud data, if the sparseness of a single frame of data does not meet the requirements, an accumulation method can be used.

[0038] Figure 2 A schematic diagram of a pallet pose estimation method 100 provided in an embodiment of the present application.

[0039] like Figure 2 As shown, a pallet pose estimation method 100 provided in one embodiment of the present application includes:

[0040] S110: Acquire a color image and point cloud data that include a pallet area and are at least partially pixel-aligned, and crop the color image to obtain a target image that only includes the pallet area.

[0041] Specifically, the alignment of at least some pixels of the color image and the point cloud data can be understood as the fact that the 2D vision sensor used to capture the color image and the 3D vision sensor used to capture the point cloud data have at least partially overlapping fields of view. In other words, a pixel in the color image can be found in the point cloud data based on the conversion relationship between the 2D vision sensor and the 3D vision sensor. It should be noted that the conversion relationship between the 2D vision sensor and the 3D vision sensor can be obtained by calibrating the extrinsic parameters of the two sensors using a known calibration algorithm in the prior art, and this application does not impose any limitations thereon.

[0042] Optionally, cropping the color image to obtain a target image that includes only the pallet region includes: performing semantic segmentation on the color image to identify the pallet region to obtain a bounding box and a mask image that only includes the pallet region; and cropping the color image using the bounding box and the mask image to obtain a target image that only includes the pallet region. It should be understood that the bounding box is a minimum bounding rectangle that includes the outer contour of the pallet region, and the mask image is a binary image, i.e., the image corresponding to the region of interest where the pallet is located is set to 1, and other regions of no interest are set to 0, to extract the pallet region from the color image.

[0043] Optionally, semantic segmentation is performed on the color image to identify the area where the pallet area is located to obtain a bounding box and a mask image that only includes the pallet area. This can be implemented using an image semantic segmentation method 200 provided in an embodiment of the present application. The method 200 may include the following steps:

[0044] S210: Performing deep feature extraction on the color image to obtain feature images, where the feature images include a feature image A and a feature image B. Feature image A and feature image B have different scales.

[0045] In one possible scenario, deep features of different scales may be extracted from the color image, thereby obtaining feature images A and feature images B of different scales.

[0046] Another possible scenario is to first perform fixed-scale deep feature extraction on the color image to obtain feature image A, and then perform fixed-scale deep feature extraction on feature image A to obtain feature image B with a different scale from feature image A.

[0047] It is understood that feature image B has different resolution and number of channels than feature image A. Feature image B has lower resolution and higher number of channels than feature image A. S220: Sampling feature image B to obtain feature image C. Feature image C has the same scale as feature image A.

[0048] As an example, feature image B passes through the sampling layer and is sampled to obtain feature image C with the same scale as feature image A.

[0049] S230: Splicing the feature image A and the feature image C to obtain a spliced ​​image A.

[0050] As an example, feature image A and feature image C with the same scale are passed through a splicing layer, and feature image A and feature image C are spliced ​​to obtain a spliced ​​image A.

[0051] S240: Performing depth feature extraction on the spliced ​​image A to obtain a feature image D.

[0052] S250: Decoupling the feature image D to obtain a bounding box and the mask image, where the bounding box and the mask image include the pallet area.

[0053] Specifically, the feature image D is decoupled to obtain multiple candidate bounding boxes and corresponding confidences and candidate mask maps, and the candidate bounding boxes and corresponding confidences and candidate mask maps are post-processed to obtain bounding boxes and mask maps. For example, based on a preset confidence threshold and the confidence of the candidate bounding boxes, the candidate bounding boxes are screened to obtain multiple candidate bounding boxes that include the pallet area, thereby reducing the number of candidate bounding boxes; the best bounding box (i.e., bounding box) is determined from the multiple candidate bounding boxes that include the pallet area by using intersection-over-union (IoU), and the mask map corresponding to the best bounding box is the best mask map (i.e., mask map), so as to eliminate redundant bounding boxes on the same target and retain only the box with the highest confidence and low overlap with other boxes. Furthermore, the confidence of the candidate bounding boxes is compared with the preset confidence threshold, and the candidate bounding boxes outside the preset confidence threshold are eliminated; the remaining candidate bounding boxes are sorted according to the confidence of the candidate bounding boxes and the candidate bounding box with the highest confidence is selected as the reference bounding box; the intersection of union (IoU) between the reference bounding box and other boxes is calculated, and the calculated IoU is compared with the preset IoU threshold; the candidate bounding boxes with IoU outside the preset IoU threshold are regarded as overlapping boxes and removed, and the candidate bounding boxes with the highest confidence and low overlap with other boxes are retained as the best bounding boxes for the corresponding prediction branch.

[0054] Optionally, before S250, method 200 also includes: S241, downsampling the feature image D to obtain a feature image E, and the feature image E has the same scale as the feature image B; S242, splicing the feature image B and the feature image E to obtain a spliced ​​image B; S243, performing deep feature extraction on the spliced ​​image B to obtain a feature image F.

[0055] Optionally, when the method 200 further includes S241 , S242 and S243 , S250 may further include: decoupling the feature image D and the feature image F respectively to obtain a bounding box and a mask image.

[0056] Specifically, the decoupling and post-processing provided in the above embodiment are performed on the feature image D to obtain a first bounding box and a first mask image. The decoupling and post-processing provided in the above embodiment are performed on the feature image F to obtain a second bounding box and a second mask image. The first bounding box and the second bounding box are post-processed to obtain a bounding box. The first mask image and the second mask image are post-processed to obtain a mask image. For example, the best bounding box is selected from the first bounding box and the second bounding box to obtain a bounding box. The mask image (first mask image and / or second mask image) corresponding to the best bounding box is the best mask image (i.e., the mask image).

[0057] In some embodiments, method 100 further includes: inputting the color image into a preset segmentation network model, where the preset segmentation network model performs semantic segmentation on the color image to obtain a bounding box and a mask image.

[0058] Specifically, the input image is input into the preset segmentation network model for semantic segmentation to obtain a bounding box and mask map including the stack area; in other words, the cropping information of the color image can be obtained by performing semantic segmentation on the input image through the preset segmentation network model.

[0059] Optionally, a color image is input into a preset segmentation network model, and feature extraction is performed on the color image to obtain multiple feature images of different scales; the multiple feature images of different scales are upsampled and / or downsampled to obtain multiple feature images of the same scale; the multiple feature images of the same scale are spliced ​​to obtain a spliced ​​image; the spliced ​​image is predicted to obtain a candidate bounding box and a candidate mask map; the candidate bounding box and the candidate mask map are post-processed to obtain a bounding box and a mask map.

[0060] Optionally, the preset segmentation network model includes at least one of the following: a downsampling layer, a feature extraction layer, an upsampling layer, a splicing layer, and a prediction layer; specifically, the feature extraction layer in the preset segmentation network model is used to extract features from the color image to obtain multiple feature images of different scales; the upsampling layer and / or downsampling layer in the preset segmentation network model is used to upsample and / or downsample multiple feature images of different scales to obtain multiple feature images of the same scale; the splicing layer in the preset segmentation network model is used to splice multiple feature images of the same scale to obtain a spliced ​​image; the prediction layer in the preset segmentation network model is used to decouple the spliced ​​image and / or feature image to obtain a bounding box and a mask image.

[0061] Based on this technical solution, a preset segmentation network model is used to extract features from color images to varying degrees. By upsampling / downsampling and splicing, the features of images of different scales are integrated, effectively combining global and local information from the color image and improving segmentation accuracy. Furthermore, the preset segmentation network model further performs deep feature extraction on the spliced ​​images, which helps reduce image processing data and retains advanced and rich semantic features, ensuring both model accuracy and operational efficiency.

[0062] Figure 3 A schematic diagram of a preset segmentation network model 300 provided in an embodiment of the present application.

[0063] Optionally, combined Figure 3 As shown, the preset segmentation network model may include: a first downsampling layer, a first feature extraction layer, a second downsampling layer, a second feature extraction layer, a third downsampling layer, and a third feature extraction layer.

[0064] Specifically, the first downsampling layer is used to downsample the input color image to obtain a first sampling image, and the resolution and size of the first sampling image are smaller than the original resolution and size of the color image; the first feature extraction layer is used to perform feature extraction on the first sampling image to obtain a first feature image; the second downsampling layer is used to downsample the first feature image to obtain a second sampling image, and the resolution and size of the second sampling image are smaller than the resolution and size of the first feature image; the second feature extraction layer is used to perform feature extraction on the second sampling image to obtain a second feature image; the third downsampling layer is used to downsample the second feature image to obtain a third sampling image, and the resolution and size of the third sampling image are smaller than the resolution and size of the second feature image; the third feature extraction layer is used to perform feature extraction on the third sampling image to obtain a third feature image.

[0065] It should be understood that the input color image passes through the above-mentioned downsampling layer and feature extraction layer in sequence, and the different feature images obtained in the feature extraction layer of each layer have different resolutions and numbers of channels, where the resolution shows a downward trend and the number of channels shows an upward trend, and then the first feature image, the second feature image and the third feature extraction image of different scales are obtained in the first feature extraction layer, the second feature extraction layer and the third feature extraction layer respectively.

[0066] Optionally, the preset segmentation network model may further include: a first upsampling layer, a first splicing layer, a fourth feature extraction layer, a second upsampling layer, a second splicing layer, and a fifth feature extraction layer.

[0067] Specifically, the first upsampling layer is used to upsample the third feature image in the third feature extraction layer to obtain a fourth feature image of the same scale as the second feature image; the first splicing layer is used to splice the second feature image in the second feature extraction layer with the fourth feature image in the first upsampling layer to obtain a first spliced ​​image; the fourth feature extraction layer is used to perform feature extraction on the first spliced ​​image in the first splicing layer to obtain a fifth feature image; the second upsampling layer is used to upsample the fifth feature image in the fourth feature extraction layer to obtain a sixth feature image of the same scale as the first feature image; the second splicing layer is used to splice the first feature image in the first feature extraction layer with the sixth feature image in the second upsampling layer to obtain a second spliced ​​image; the fifth feature extraction layer is used to perform feature extraction on the second spliced ​​image in the second splicing layer to obtain a seventh feature image. It should be noted that the technical solution for upsampling and splicing feature images has been explained above and will not be repeated here.

[0068] Optionally, the preset segmentation network model may further include: a fourth downsampling layer, a third splicing layer, a sixth feature extraction layer, a fifth downsampling layer, a fourth splicing layer, and a seventh feature extraction layer.

[0069] Specifically, the fourth downsampling layer is used to downsample the seventh feature image in the fifth feature extraction layer to obtain an eighth feature image of the same scale as the fifth feature image; the third splicing layer is used to splice the fifth feature image in the fourth feature extraction layer with the eighth feature image in the fourth downsampling layer to obtain a third spliced ​​image; the sixth feature extraction layer is used to perform feature extraction on the third spliced ​​image in the third splicing layer to obtain a ninth feature image; the fifth downsampling layer is used to downsample the ninth feature image in the sixth feature extraction layer to obtain a tenth feature image of the same scale as the third feature image; the fourth splicing layer is used to splice the third feature image in the third feature extraction layer with the tenth feature image in the fifth downsampling layer to obtain a fourth spliced ​​image; the seventh feature extraction layer is used to perform feature extraction on the fourth spliced ​​image in the fourth splicing layer to obtain an eleventh feature image. It should be noted that the technical solutions for feature extraction, downsampling and splicing of feature images have been explained above and will not be repeated here.

[0070] Optionally, the preset segmentation network model may further include: a first prediction layer, a second prediction layer, and a third prediction layer.

[0071] Specifically, the first prediction layer is used to decouple the seventh feature image in the fifth feature extraction layer to obtain a first candidate bounding box and a first candidate mask map. The second prediction layer is used to decouple the ninth feature image in the sixth feature extraction layer to obtain a second candidate bounding box and a second candidate mask map. The third prediction layer is used to decouple the eleventh feature image in the seventh feature extraction layer to obtain a third candidate bounding box and a third candidate mask map. It should be noted that the technical solution for decoupling feature images to obtain candidate bounding boxes and candidate mask maps has been explained above and will not be repeated here.

[0072] In some embodiments, before inputting the color image into the preset segmentation network model, method 100 further includes: normalizing the color image, i.e., scaling the color image to a standard size so that it meets the input standard of the preset segmentation model to obtain a preprocessed color image; and inputting the preprocessed color image into the preset segmentation network model, which helps the color image to be more easily processed by the preset segmentation network model.

[0073] S120: acquiring a point cloud of an area corresponding to the target image from the point cloud data according to an alignment relationship between the color image and the point cloud data, and obtaining a point cloud to be fitted (also referred to as a ROI point cloud).

[0074] In one possible implementation, when a color camera is used as a 2D vision sensor to capture color images and a depth camera is used as a 3D vision sensor to acquire point cloud data, the depth camera first captures a depth image. Since the target image is cropped from the color image, the area corresponding to the target image can be cropped from the depth image based on the conversion relationship between the color camera and the depth camera to obtain a depth region that only includes the pallet. The depth region is then converted into a point cloud to obtain an ROI point cloud. When a lidar is used as a 3D vision sensor, the lidar can directly output point cloud data. The point cloud data is converted to the color camera coordinates based on the conversion relationship between the color camera and the lidar, and projected onto the color camera imaging plane based on the color camera intrinsic parameters to obtain a depth image. The depth image and the target image are located on the same plane in the same coordinate system. The depth image converted from the point cloud is segmented based on the target image to obtain a depth region that only includes the pallet. The depth region is then converted to the lidar coordinate system based on the conversion relationship to obtain an ROI point cloud.

[0075] Optionally, step S120 further includes: performing a pallet quality assessment on the color image or target image to determine whether to obtain a point cloud to be fitted based on the assessment result; wherein, when the pallet quality in the color image or target image indicates that the pallet is complete, obtaining a point cloud corresponding to the target image from the point cloud data based on the alignment relationship between the color image and the point cloud data to obtain the point cloud to be fitted. It should be understood that the quality assessment of the color image or target image can be directly obtained from a storage medium of a local device and / or a cloud device, and this application is not limited thereto.

[0076] Based on the above technical solution, a more appropriate process execution action can be selected in combination with the actual state of the pallet, thereby providing the operator with effective pallet posture information while saving process action overhead.

[0077] like Figure 4 As shown, in one possible implementation, the judgment result is obtained by performing a pallet quality judgment on the directly acquired color image; in another possible implementation, the pallet quality judgment can also be performed on the target image obtained by cropping the color image using the bounding box and / or mask image based on step S110 to obtain the judgment result.

[0078] Optionally, the judgment result includes any of the following: the pallet is intact, the pallet is obscured, or the pallet is damaged and incomplete. When the judgment result is that the pallet is intact, it indicates that the pallet is in an ideal state and the current pallet can be used; when the judgment result is that the pallet is obscured, it means that there is an obstacle blocking the front of the pallet; when the judgment result is that the pallet is damaged and incomplete, it means that the pallet is damaged. For judgment results of obscured or incomplete pallets, some interactive reminders can be provided, such as reminding the operator that there is an obstacle blocking the front of the pallet or the pallet is damaged, allowing the operator to evaluate whether the pallet can continue to be used or whether it should be replaced.

[0079] Based on the above technical solution, a more appropriate process execution action can be selected in combination with the actual state of the pallet, thereby providing the operator with effective pallet posture information while saving process action overhead.

[0080] Optionally, the color image and / or target image is input into a preset classification network for quality judgment, such as Figure 5 As shown, Figure 5A schematic diagram of a preset classification network provided in an embodiment of the present application. The preset classification network includes an initial convolutional layer, a bottleneck layer, a pooling layer, and a fully connected layer connected in series, wherein the initial convolutional layer includes a convolution module, a batch normalization module, and an activation module; the convolution module is used to perform feature extraction on the color image and / or target image to extract high-level features in the color image and / or target image, thereby reducing the dimension of the color image and / or target image; the batch normalization module is used to normalize the image after feature extraction to accelerate training and stabilize the network; the activation module is used to introduce nonlinearity to the normalized image, thereby obtaining image A, to increase the expressive power of the model. The bottleneck layer includes a depthwise separable convolution module, which is used to further perform deep feature extraction on image A to reduce the amount of computation while maintaining performance to obtain image B; the pooling layer is used to compress image B into a single vector to obtain image C, thereby reducing the number of parameters while retaining global information; the fully connected layer is used to perform classification prediction on image C to obtain the classification result (i.e., the quality judgment result), where the classification result is divided into 0, 1, and 2; where 0 indicates that the pallet is intact and in an ideal state, 1 indicates that the pallet is blocked, and 2 indicates that the pallet is damaged and incomplete.

[0081] S130: Perform plane fitting on the point cloud to be fitted to obtain a plane fitting model.

[0082] Optionally, the point cloud to be fitted is preprocessed to obtain a first point cloud, and the first point cloud is used to perform plane fitting to obtain a plane fitting model.

[0083] Specifically, the point cloud to be fitted can be preprocessed to obtain the first point cloud by at least one of the following methods: through filtering, downsampling, outlier removal, cluster filtering, filtering based on the normal direction of the point cloud to be fitted, and distance filtering based on clustered point clouds along the normal direction of the point cloud to be fitted. Six examples of the above-mentioned preprocessing methods for the point cloud to be fitted are described below.

[0084] Example 1, straight-through filtering.

[0085] A cropping bounding box is defined, and the portion of the point cloud that lies within the bounding box is extracted from the point cloud to be fitted based on the specified six coordinate boundary values ​​(corresponding to the minimum and maximum values ​​of the x, y, and z axes, respectively), thereby obtaining a first point cloud. It is understood that the six coordinate boundary values ​​can be configured based on the actual environment of the pallet and are not limited in this application.

[0086] Example 2, downsampling.

[0087] The point cloud to be fitted is divided into multiple voxels according to a preset scale, and the center point of each voxel is selected as the representative point after downsampling of the point cloud to be fitted, thereby achieving uniform downsampling of the point cloud to be fitted to obtain the first point cloud. Specifically, the preset scale can be set according to the density of the point cloud to be fitted, and then the point cloud to be fitted is uniformly downsampled: if the point cloud to be fitted has uniform density, the average point spacing of the point cloud to be fitted is calculated, and the preset scale is set based on the average point spacing and a preset fixed coefficient. The point cloud to be fitted is evenly divided according to the preset scale to obtain multiple voxels, and the center point of each voxel is selected as the representative point after downsampling of the point cloud to be fitted to obtain the first point cloud; if the point cloud to be fitted has non-uniform density, multi-resolution voxels are preferably used, such as using small voxels for local dense density areas in the point cloud to be fitted and using large voxels for local sparse density areas in the point cloud to be fitted, and then the center point of each small voxel and the center point of each large voxel are selected as the representative point after downsampling of the point cloud to be fitted, thereby obtaining the first point cloud.

[0088] In one possible implementation, small voxels are used for local dense density areas in the point cloud to be fitted, and large voxels are used for local sparse density areas in the point cloud to be fitted, including: for local high-density areas, a first preset scale is set using the average point spacing and a first fixed coefficient, and the local dense density areas in the point cloud to be fitted are divided into small voxels based on the first preset scale; for local sparse density areas in the point cloud to be fitted, a second preset scale is set using the average point spacing and a second fixed coefficient, and the local sparse density areas in the point cloud to be fitted are divided into large voxels based on the second preset scale, wherein the first fixed coefficient is smaller than the second fixed coefficient.

[0089] The method used in this embodiment can well control the distribution density of each point in the point cloud, making the downsampled point cloud more uniform in space, while reducing the number of points in the point cloud while maintaining the geometric shape and distribution characteristics of the point cloud.

[0090] Example 3, outlier removal.

[0091] Traverse each point in the point cloud to be fitted, take each point as the sphere center and construct a sphere centered on each point based on a preset specified radius, count the number of neighboring points in each sphere, compare the number of neighboring points with the preset number threshold to eliminate isolated points, and obtain the first point cloud.

[0092] It is understood that by default, valid point cloud data should be composed of dense points, while isolated points may be noise or outliers. For example, if the number of neighboring points is less than a preset threshold, it means that there are not enough neighboring points in the sphere constructed with the point as the center. The point can be considered an isolated point and removed from the point cloud. Conversely, if the number of neighboring points is greater than or equal to the preset threshold, it means that there are enough neighboring points in the sphere constructed with the point as the center. It is considered a valid point. The above steps are performed on each point in the point cloud to obtain the first point cloud.

[0093] Example 4, cluster filtering.

[0094] Based on the density of samples in the point cloud to be fitted, the point cloud to be fitted is divided into multiple dense clusters, and noise points are identified to remove noise points from the point cloud to be fitted to obtain the first point cloud. The specific steps are as follows:

[0095] Step 1: For each point in the fitted point cloud, calculate the number of points in its ε neighborhood. If the number of points is greater than or equal to the preset point threshold (MinPts), it is a core point; where the ε neighborhood refers to the set of all other points in a circular (two-dimensional) or spherical (three-dimensional) area with a radius of ε centered on a certain point.

[0096] Step 2: Starting from any core point, find all reachable points in its ε neighborhood to form a cluster.

[0097] Step 3, repeat step 2 for the points that have not been visited until all points have been processed.

[0098] Step 4: Points that do not belong to any cluster are called noise points. Noise points are removed from the point cloud to be fitted to obtain the first point cloud.

[0099] In Example 5, the point cloud to be fitted is filtered according to the normal direction of the point cloud to be fitted to obtain a first point cloud.

[0100] Step 1: Calculate the normal direction of the point cloud to be fitted. The calculation method is:

[0101] ①For each point in the point cloud to be fitted, select neighboring points and calculate their covariance matrix;

[0102] ② Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors. The eigenvector corresponding to the smallest eigenvalue is the normal vector.

[0103] ③ Adjust the consistency of the normals facing the X-axis, that is, make the directions of the normals consistent and all facing the positive or negative direction of the X-axis;

[0104] Step 2: Eliminate points in the point cloud to be fitted whose normal x vector is smaller than the y and z vectors to obtain the first point cloud.

[0105] Example 6: A first point cloud is obtained by filtering the clustered point clouds according to the distance in the normal direction of the point cloud to be fitted.

[0106] Based on the normal vectors of each point obtained in Example 5, all normal vectors are regarded as independent point clouds. Clustering operations are performed on each independent point cloud, and only the largest point cloud in the cluster is retained. This point cloud represents the main normal direction; the point cloud in the main normal direction is screened to obtain the first point cloud. Specifically, first, a DBSCAN clustering operation is performed on each point in the point cloud located in the main normal direction to obtain multiple clustered point clouds, and on this basis, the normal vectors of the multiple clustered point clouds are obtained. Based on the multiple normal vectors, the mean of the normal vector and the projection distance of the centroid of each clustered point cloud on the mean normal vector are calculated. According to the pre-set distance threshold and the size of the projection distance, the clustered point clouds that meet the conditions are further screened to obtain the first point cloud.

[0107] It should be noted that Examples 1 to 6 above are for illustrative purposes only and are not intended to be limiting in this application. Furthermore, preprocessing the fitted point cloud to obtain the first point cloud may be performed using any one of the above examples or a combination of multiple examples, or by performing filtering in sequence, which is not intended to be limiting in this application.

[0108] Based on the above technical solution, before plane fitting is performed on the point cloud to be fitted, the points in the point cloud to be fitted are preprocessed in advance, thereby improving the confidence of the point cloud to be fitted, and further improving the accuracy and efficiency of the plane fitting model obtained by subsequent plane fitting of the point cloud to be fitted.

[0109] In one possible implementation, a plane fitting model is obtained by using the first point cloud to perform plane fitting, including: randomly selecting a number of points in the first point cloud to fit the first plane and obtaining the corresponding first plane equation (as an example, randomly selecting 3 points), calculating the distance from each remaining point in the first point cloud to the first plane one by one by using the coordinates of each point in the first point cloud and the first plane equation corresponding to the first plane, if the distance is less than or equal to a preset distance threshold, it means that the point is an interior point (that is, it is regarded as a point on the first plane), conversely, if the distance is greater than the preset distance threshold, it means that the point is not an interior point, and after traversing each interior point, the interior points are collected to obtain a first interior point set; based on the first The inlier point set is fitted to a second plane and the corresponding second plane equation is obtained. The distances from the remaining points in the first inlier point set to the second plane are calculated using the coordinates of each point in the first inlier point set and the second plane equation corresponding to the second plane. A similar method to the above-mentioned method for obtaining the first inlier point set is used to compare the first point cloud with the preset distance threshold to update the first inlier point set to obtain the second inlier point set. The second inlier point set is plane fitted and updated, i.e., the steps of updating the first inlier point set are repeated until a preset number of updates (also referred to as an iteration number threshold) is reached to obtain a third inlier point set, and an optimal plane fitting model is constructed based on the third inlier point set.

[0110] It should be noted that the preset number of updates (also called the iteration threshold) determines the accuracy and running time of the embodiment of the present application. The more updates (also called the iteration threshold), the higher the accuracy of the fitted plane model, but the longer the running time. The preset number of updates (also called the iteration threshold) can be set according to actual task requirements, and this application does not limit it here.

[0111] Based on the above technical solution, plane fitting is performed on the points in the fitting point cloud, so that the points in the final plane fitting model can better fit the actual pallet posture, thereby improving the accuracy of pallet posture estimation.

[0112] S140: Calculating the angle information and the center coordinates of the pallet based on the plane fitting model and the preset reference coordinate system, and constructing a pose matrix of the pallet using the angle information and the center coordinates of the pallet to obtain the pose information of the pallet.

[0113] Figure 6 The following is a flow chart of a method for estimating the position and posture of a pallet based on a plane fitting model provided in an embodiment of the present application. Figure 6As shown, the method includes: obtaining normal vectors of a plane fitting model and a reference plane of a preset reference coordinate system, calculating the angle between the plane fitting model and the reference plane of the preset reference coordinate system using the normal vectors of the two, and obtaining first angle information of the pallet; projecting the plane fitting model onto a specific plane in the reference plane to form a two-dimensional image, and calculating the minimum bounding rectangle of the two-dimensional image to obtain second angle information and the center coordinates of the pallet through the minimum bounding rectangle; and constructing a pose matrix of the pallet plane based on the first angle information, the second angle information, and the center coordinates of the pallet to obtain pose information of the pallet. It should be understood that the coordinate axis of the plane fitting model is the same as the coordinate axis of the preset reference coordinate system.

[0114] Specifically, combined Figure 6 As shown, the first angle information of the pallet includes the yaw angle between the plane fitting model and the YOZ reference plane of the preset reference coordinate system, and the pitch angle between the plane fitting model and the XOY reference plane of the preset reference coordinate system. The second angle information of the pallet includes the minimum bounding rectangle corresponding to the plane fitting model and the roll angle of the horizontal axis of the reference coordinate system. Examples 7 through 9 below describe how to calculate these angles (yaw, pitch, and roll).

[0115] Example 7, calculation method of yaw angle.

[0116] Step 1: Define the plane coefficient of the YOZ reference plane of the preset reference coordinate system where the plane fitting model is located as (1, 0, 0, 0). The plane coefficient is used to represent the equation coefficient of a plane.

[0117] Step 2: Calculate the angle between the plane fitting model plane and the YOZ reference plane:

[0118] ① Obtain the first normal vector of the plane fitting model and the second normal vector of the YOZ plane;

[0119] ②Calculate the first dot product of the first normal vector and the second normal vector;

[0120] ③Calculate the first modulus of the first normal vector and the second normal vector;

[0121] ④ Calculate the first cosine of the yaw angle based on the first dot product and the first modulus;

[0122] ⑤ Solve the inverse cosine function for the first cosine value to obtain the yaw angle (in radians as an example).

[0123] Example 8, how to calculate the pitch angle.

[0124] Step 1: Define the plane coefficient of the preset reference coordinate system XOY reference plane where the plane fitting model is located as (0, 0, 1, 0).

[0125] Step 2: Calculate the angle between the plane fitting model plane and the XOY plane:

[0126] ① Obtain the first normal vector of the plane fitting model and the third normal vector of the XOY plane;

[0127] ②Calculate the second dot product of the first normal vector and the third normal vector;

[0128] ③Calculate the second modulus of the first normal vector and the third normal vector;

[0129] ④ Calculate the second cosine of the pitch angle based on the second dot product and the second modulus;

[0130] ⑤ Solve the inverse cosine function for the second cosine value to obtain the pitch angle (in radians as an example).

[0131] Example 9, how to calculate the roll angle.

[0132] Step 1: Project the plane fitting model onto the YOZ specific plane in the reference plane of the preset reference coordinate system to form a two-dimensional image.

[0133] Step 2: Calculate the minimum bounding rectangle of the two-dimensional image.

[0134] Step 3: Calculate the counterclockwise rotation angle of the wider side of the minimum circumscribed rectangle relative to the horizontal line of the YOZ specific plane to obtain the roll angle.

[0135] Step 4: Convert the roll angle to radians.

[0136] Combined with Examples 7 to 9 above, the pallet angle information rpy (roll, pitch, yaw) is obtained. It should be understood that the pallet angle information can be expressed in radians or degrees. To facilitate subsequent calculations, this application prefers to use radians, which is not limited here.

[0137] Specifically, the center coordinates of the stack are obtained through the minimum enclosing rectangle, including: obtaining the plane equation ax+by+cz+d=0 of the plane fitting model and the coordinates center(0, y, z) corresponding to the center point of the minimum enclosing rectangle, calculating the x-coordinate value corresponding to the center coordinates of the stack through the coordinates corresponding to the center point of the minimum enclosing rectangle and the plane equation of the plane fitting model, and combining the y-coordinate value and the z-coordinate value corresponding to the center point of the minimum enclosing rectangle to obtain the center coordinates of the stack.

[0138] As an example, obtain the plane equation ax+by+cz+d=0 corresponding to the plane fitting model and the y and z coordinates of the center point of the minimum bounding rectangle. Substitute the y and z coordinates of the center coordinates of the minimum bounding rectangle into the plane equation ax+by+cz+d=0 to calculate the x coordinate. The center coordinates of the minimum bounding rectangle are the y and z coordinates of the center coordinates of the pallet. Thus, the pallet center coordinates Loc(x, y, z) are obtained.

[0139] Specifically, the pallet pose matrix is ​​constructed according to the pallet angle information rpy (roll, pitch, yaw) and the pallet center coordinates Loc (x, y, z) to obtain the pallet pose information.

[0140] As an example, first construct the following three rotation matrices based on the stack angle information rpy (roll, pitch, yaw):

[0141] Rotation matrix for rotation around the x-axis

[0142] Rotation matrix for rotation around the y-axis

[0143] Rotation matrix for rotation around the z-axis

[0144] The above rotation matrix R x (roll), R y (pitch) and R z (yaw) are multiplied in sequence to obtain the following final rotation matrix R:

[0145]

[0146] The final rotation matrix R and the pallet center coordinates Loc (x, y, z) representing the translation matrix are combined to form the pallet pose matrix H to obtain the pallet pose information. The pallet pose matrix H is in the following form:

[0147]

[0148] Based on the above technical solution, the plane fitting model of the pallet can be obtained by plane fitting of the point cloud data to be fitted corresponding to the color image. The pallet pose estimation can be directly performed based on the plane fitting model, and the pallet pose information can be calculated without the need for external marks and / or specific templates on the pallet. While improving the robustness and accuracy of the pallet pose estimation method, the generalization ability of the pallet pose estimation method is further improved, and the deployment cost of the pallet pose estimation method is reduced.

[0149] The present application also provides a computer-readable storage medium storing a program code, which, when executed on a computer, enables the computer to execute the following Figures 2 to 6 The pallet pose estimation method described in the embodiment.

[0150] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0151] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0155] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0156] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A pallet pose estimation method, characterized in that: The method comprises: Acquire a color image and point cloud data that includes the pallet area and is at least partially pixel-aligned, and crop the color image to obtain a target image that only includes the pallet area; According to the alignment relationship between the color image and the point cloud data, the point cloud of the area corresponding to the target image is obtained from the point cloud data to obtain the point cloud to be fitted; Performing plane fitting on the point cloud to be fitted to obtain a plane fitting model; The angle information of the pallet and the center coordinates of the pallet are calculated based on the plane fitting model and the preset reference coordinate system, and the pose matrix of the pallet is constructed using the angle information of the pallet and the center coordinates of the pallet to obtain the pose information of the pallet.

2. The method according to claim 1, characterized in that The step of acquiring a point cloud corresponding to an area of ​​the target image from the point cloud data according to the alignment relationship between the color image and the point cloud data to obtain a point cloud to be fitted further comprises: Performing a pallet quality judgment on the color image or the target image to determine whether to obtain a point cloud to be fitted according to the judgment result; When the pallet quality in the color image or the target image is complete, a point cloud of an area corresponding to the target image is obtained from the point cloud data according to an alignment relationship between the color image and the point cloud data to obtain a point cloud to be fitted.

3. The method according to claim 1 or 2, characterized in that The performing plane fitting on the point cloud to be fitted to obtain a plane fitting model includes: Preprocessing the point cloud to be fitted to obtain a first point cloud; Performing plane fitting on the first point cloud to obtain the plane fitting model; The preprocessing includes at least one or more of the following combinations: through filtering, downsampling, outlier removal, cluster filtering, filtering according to the normal direction of the point cloud to be fitted, and distance filtering of clustered point clouds in the normal direction of the point cloud to be fitted.

4. The method according to claim 3, characterized in that The performing plane fitting on the first point cloud to obtain a plane fitting model includes: Randomly selecting a number of points in the first point cloud to fit a first plane and obtaining the corresponding first plane equation, and calculating the first distance from each remaining point in the first point cloud to the first plane one by one using the coordinates of each point in the first point cloud and the first plane equation corresponding to the first plane; If the first distance of a point in the first point cloud is less than or equal to a preset distance threshold, it is indicated that the point is an inlier; otherwise, if the first distance is greater than the preset distance threshold, it is indicated that the point is not an inlier, and after traversing each point in the first point cloud, the inliers are collected to obtain a first inlier set; Fitting a second plane based on the first inlier point set and obtaining a corresponding second plane equation, calculating a second distance from each remaining point in the first inlier point set to the second plane using the coordinates of each point in the first inlier point set and the second plane equation corresponding to the second plane, and comparing the second distance with the preset distance threshold to update the first inlier point set to obtain a second inlier point set; Performing plane fitting and updating on the second inlier point set until a preset number of updates is reached to obtain a third inlier point set, and constructing a plane fitting model using the third inlier point set.

5. The method according to claim 3, characterized in that The method of obtaining a pallet pose matrix based on the plane fitting model and the reference coordinate system includes: Obtaining a normal vector of the plane fitting model and a reference plane of a preset reference coordinate system, and calculating an angle between the plane fitting model and the reference plane of the preset reference coordinate system using the normal vector to obtain first angle information of the pallet; Projecting the plane fitting model onto a specific plane in the reference plane to form a two-dimensional image, and calculating a minimum bounding rectangle of the two-dimensional image to obtain second angle information of the pallet and the center coordinates of the pallet through the minimum bounding rectangle; A posture matrix of the pallet plane is constructed according to the first angle information, the second angle information and the center coordinates of the pallet to obtain the posture information of the pallet.

6. The method according to claim 5, characterized in that The first angle information of the pallet includes the yaw angle of the plane fitting model and the YOZ reference plane of the preset reference coordinate system, and the pitch angle of the plane fitting model and the XOY reference plane of the preset reference coordinate system; the second angle information of the pallet includes the minimum circumscribed rectangle corresponding to the plane fitting model and the roll angle of the horizontal coordinate axis of the preset reference coordinate system.

7. The method according to claim 6, characterized in that Methods for obtaining the yaw angle or the pitch angle include: Define the plane coefficient of the YOZ reference plane of the preset reference coordinate system where the plane fitting model is located as (1, 0, 0, 0), or define the plane coefficient of the XOY reference plane of the preset reference coordinate system where the plane fitting model is located as (0, 0, 1, 0), where the plane coefficient is used to represent the coefficient of the equation of a plane; Obtain a first normal vector of the plane fitting model and a second normal vector of the YOZ reference plane or a third normal vector of the XOY reference plane, and calculate a first dot product and a first modulus of the first normal vector and the second normal vector or calculate a second dot product and a second modulus of the first normal vector and the third normal vector; Calculating a first cosine value of the yaw angle based on the first dot product and the first modulus; or calculating a second cosine value of the pitch angle based on the second dot product and the second modulus; An inverse cosine function is solved using the first cosine value or the second cosine value to obtain the yaw angle or the pitch angle.

8. The method according to claim 6, characterized in that The roll angle is obtained by: Projecting the plane fitting model onto a YOZ specific plane in the reference plane of the preset reference coordinate system to form a two-dimensional image; The roll angle is obtained by calculating the minimum circumscribed rectangle of the two-dimensional image and calculating the counterclockwise rotation angle of a relatively wide side of the minimum circumscribed rectangle relative to the horizontal line of the YOZ specific plane.

9. The method according to claim 8, characterized in that The method for obtaining the center coordinates of the stack includes: Obtaining a plane equation of the plane fitting model and coordinates corresponding to the center point of the minimum circumscribed rectangle, and calculating an x-coordinate value corresponding to the center coordinate of the stack using the coordinates corresponding to the center point of the minimum circumscribed rectangle and the plane equation of the plane fitting model; The center coordinates of the pallet are obtained by combining the y-coordinate value and the z-coordinate value corresponding to the center point of the minimum circumscribed rectangle.

10. The method according to claim 1 or 2, characterized in that The step of cropping the color image to obtain a target image including only the pallet area includes: Performing semantic segmentation on the color image to identify the area where the pallet is located to obtain a bounding box and a mask image that only includes the pallet area; The color image is cropped using the bounding box and the mask image to obtain a target image that only includes the pallet area.

11. A pallet pose estimation system, applied to a forklift, characterized in that: The system comprises: A two-dimensional vision sensor for capturing color images of the pallet area; 3D vision sensor for collecting point cloud data including the pallet area; A main control chip is used to process the color image and the point cloud data according to the method according to any one of claims 1 to 10 to obtain the position information of the pallet.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program code, and when the computer program code is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 10.

Images