A pallet pose estimation method based on attention mechanism

Abstract

The present invention discloses a pallet pose estimation method based on an attention mechanism, specifically a pallet detection and direction estimation method for a mobile forklift, which can be used to guide the forklift toward a target pallet. The method primarily includes: capturing pallet images with an RGB-D depth camera; detecting the pallet region using an instance segmentation method incorporating an attention mechanism to obtain mask information for the pallet image; extracting the pallet's point cloud data based on the mask information and the correspondence between the two-dimensional image and the three-dimensional point cloud; and calculating the normal vector of the pallet's front face using a principal component analysis method based on the pallet's point cloud data to obtain the pallet's pose relative to the forklift. The present invention uses an instance segmentation method incorporating an attention mechanism to detect the pallet region in the image and segment the pallet's point cloud data, thereby improving the accuracy of pallet segmentation and the efficiency of forklift guidance.

Description

Technical Field

[0001] The present invention relates to the field of pallet detection, and in particular to a pallet pose estimation method based on an attention mechanism, which is used to guide a movable forklift to approach a target pallet. Background Art

[0002] Robotic forklifts are already widely used in the logistics and manufacturing sectors of the future. In unmanned factories, driverless automated guided vehicles (AGVs) are primarily used as material handling equipment for in-factory logistics applications. During the material transfer process, the AGV must perform a series of operations, such as pallet detection, pallet pose estimation, pallet picking, and loading. The AGV must accurately detect the pallet's position and pose in order to quickly, accurately, and safely approach the pallet. Generally, pallet docking for an AGV is defined as the problem of positioning the AGV to a target position with the desired direction and distance using visual servo navigation. However, docking operations require high precision, and inaccurate operations can easily lead to collisions with the target, resulting in significant economic losses. Furthermore, the docking task must also be subject to the additional transient constraints imposed by the docking mechanism structure. Therefore, accurately estimating the pallet's pose is crucial for the smooth docking of the AGV.

[0003] Experts and scholars at home and abroad have conducted in-depth research on the pose estimation of pallets. Shao Yiping et al. (A Novel Pallet Detection Method for Automated Guided Vehicles Based on Point Cloud Data, 2022, 22(20): 8019-136) proposed a pallet detection method for automated guided vehicles based on point cloud data. This method combines color with the geometric features of the pallet point cloud, and constructs a new adaptive fast color point feature histogram (ACFPFH) feature descriptor by adaptively selecting the optimal neighborhood for pallet detection. Tsiogas, Efthimios et al. (Pallet detection and docking strategy for autonomous pallet truck AGV operation, 2021, 3444-3451) proposed using a dedicated perception topology based on monocular vision and laser measurement for pallet detection. First, a deep neural network is used to quickly separate the pallet area of ​​interest, and then the point cloud data is used to perform model-based geometric pattern matching to extract the pallet pose. Michela Zaccaria et al. (Real-time Pallet Localization with 3D Camera Technology for Forklifts in Logistic Environments, 2018, pp. 297-302) proposed using an RGB camera to capture pallet images and a convolutional neural network-based method to detect the front and socket positions of the pallet to achieve vehicle guidance. Benjamin Molter et al. (B. Molter and J. Fottner, Real-time Pallet Localization with 3DCamera Technology for Forklifts in Logistic Environments, 2018, pp. 297-302) proposed a method for detecting and locating standardized Euro pallets. Without prior information about the pallet, the method searches for planes in the collected pallet point cloud data to fit the three wooden blocks on the front of the Euro pallet, thereby achieving pallet positioning and detection. Walter et al. (Closed-loop pallet manipulation in unstructured environments, 2010, 5119-5126) used lidar measurements, perception algorithms, and motion controllers to allow a forklift to pick up an unknown pallet and place it at a specified ground location or on a truck bed.Wu Wenhan et al. (A Pallet Detection Method for Warehousing Robots Based on Contour Matching, 2019, 2:197-202) proposed a detection method based on point cloud plane contour matching. This method uses a region growing algorithm with normal constraints to segment the plane and project it along its main normal direction to generate a grid image. After extracting the contour of the grid image, the contour features that fuse Hu invariant moments and scale ratio features are used to match the target and template to achieve pallet detection. Wang Pingkai et al. (Research on Pallet Detection Method Based on Improved YOLOv3, 2021, 7:29-32) proposed a pallet detection method based on YOLOv3. The K-Means++ clustering method is used to re-cluster anchor boxes that are more suitable for pallet detection. The pallet is located by analyzing the density distribution of the horizontal and vertical axes of the pallet image in the image coordinate system. Wu Denglu et al. (Research on Pallet Detection Technology Based on 3D Point Clouds and Image Edges, 2019, 3:40-42) proposed a point cloud matching technique that combines image edge and depth information to solve the problem of pallet detection and positioning, targeting the specific motion constraints and actual usage scenarios of forklifts. Liu Jiangyu (Research on Warehouse Pallet Detection Algorithms Based on Deep Learning, 2017, 2:78-84) proposed a deep learning-based convolutional neural network method to enable warehouse robots to detect pallets. By comparing four convolutional neural network models for object detection, they established the Faster RCNN model.

[0004] The above methods have achieved certain results in pallet detection. However, the following challenges remain for pallet docking with autonomous mobile forklifts: 1) In complex industrial environments, due to factors such as lighting variations, the pallet in the image may be blurred or obscured, making target docking difficult; 2) Most vision-based pallet docking systems focus only on detecting the pallet itself, ignoring the fact that the accurate position of the pallet is key to guiding the forklift to perform the docking task. This leads to performance issues with pallet docking, such as slow guidance speed and inaccurate target detection. Therefore, designing a real-time and accurate pallet pose estimation method is of great significance for improving the operating efficiency of forklift robots and promoting the intelligent development of autonomous mobile robots. Summary of the Invention

[0005] The purpose of the present invention is to disclose a pallet pose estimation method for a forklift, which is used to guide a mobile forklift to dock with a pallet.

[0006] A pallet pose estimation method based on an attention mechanism includes the following steps:

[0007] Step 1: Data acquisition: Install the RGB-D camera on the front of the forklift to obtain the image I, the point cloud data set S, and the corresponding set IS of the image I and the point cloud set S; where image I = {(r uv ,guv ,b uv )|u=1,2,...,H;v=1,2,...,W}, W represents the width of image I, H represents the height of image I, CH represents the number of channels of image I, (r uv ,g uv ,b uv ) represents the R, G, and B values ​​of the pixel in the uth row and vth column in the image I; the point cloud set S = {(x i ,y i ,z i )|i=1,2,...,U}, U represents the number of points in the point cloud set S; the corresponding set IS={(u i ,v i )|i=1,2,...,U},(u i ,v i ) indicates that the i-th point in the point cloud set S corresponds to the pixel in the u-th row and v-th column in the image I;

[0008] Step 2: Pallet segmentation: Use a convolutional neural network to train the pallet sample image to obtain a trained pallet detection network model MP. Use the pallet detection network model MP to extract the pallet area and achieve pallet image segmentation. Specifically:

[0009] Step 2.1: Construct a convolutional neural network, modifying the Mask R-CNN model, including feature extraction of the backbone network, multi-scale feature integration with attention mechanism, region proposal network, region feature alignment, branch prediction network, and loss function design;

[0010] Step 2.1.1: Backbone network feature extraction: Use the standard convolutional neural network ResNet as the backbone network for feature extraction to obtain the pallet feature C1;

[0011] Step 2.1.2: Multi-scale feature integration with attention mechanism, add attention mechanism module A to FPN to obtain multi-level feature prediction; specifically, in the bottom-up network of FPN, obtain multi-level feature vectors C2, C3, C4, C5; add attention mechanism module A in the FPN horizontal connection; in the top-down network of FPN, upsample the high-level feature C5 and the feature vectors M4 and M3 output by the attention mechanism module, and finally obtain feature vectors M5, M4, M3, M2 containing attention information; in the convolution fusion of FPN, perform conv convolution operation on each feature vector in the feature set {M2, M3, M4, M5} to obtain the final prediction, and obtain the feature vector set {P2, P3, P4, P5}, which corresponds to the feature set {C2, C3, C4, C5} respectively; Among them, the attention mechanism model A specifically includes:

[0012] Step 2.1.2.1: Attention mechanism module input: feature vector C i With the eigenvector M i+1 When input to the attention module;

[0013] Step 2.1.2.2.: For feature C i With feature M i+1 Splice and obtain the feature vector A Conv , as shown in formula (1):

[0014] A Conv =concat(C i ,up (2) (M i +1)) (1)

[0015] Among them, concat(·) represents the concatenation function, up (2) represents the up sampling function, A Conv The size is w*h*(c1+c2); where c1 represents C i The number of channels of the eigenvector, c2 represents the eigenvector M i+1 The number of channels;

[0016] Step 2.1.2.3: For feature A Conv Perform conv_1 convolution operation to obtain feature layer A Conv1 , as shown in formula (2):

[0017]

[0018] Among them, ω (c2,1×1) Represents the conv_1 convolution operation function, the kernel of the convolution operation is 1×1, the number of channels c is c2, A Conv1 The size of is w*h*c2;

[0019] Step 2.1.2.4: For feature A Conv1 Perform ReLU function, conv_2 convolution function and Sigmoid function operations respectively to obtain feature layer A sig , as shown in formula (3):

[0020] A Sig =σ(ω (c1,1×1) (ReLU(A Conv1 ))) (3)

[0021] Where ReLU(·) represents the ReLU function, ω (c,1×1) Represents the conv_2 convolution operation function, the kernel of the convolution operation is 1×1, the number of channels c is c1, σ(·) represents the Sigmoid activation function, then the feature vector Asig The size of is w*h*c1;

[0022] Step 2.1.2.5: Place A sig and M i+1 Multiply the two feature matrices to obtain the output feature vector M of the attention module i , as shown in formula (4):

[0023]

[0024] Among them, M i The feature size is w×h and the number of channels is c1;

[0025] Step 2.1.2.6: Output feature vector M of the attention module i Perform the convolution operation of conv to obtain the output feature P of the AFPN network with attention mechanism i As shown in formula (5):

[0026] P i =ω (c,3×3) (M i ) (5)

[0027] Among them, ω (c,3×3) Represents the conv convolution operation function, the kernel of the convolution operation is 3×3, the number of channels c is c1, P i The size of the feature layer is w*h*c1;

[0028] Step 2.1.3: Region Proposal Network, output P according to the feature i Generate proposed regions and perform feature extraction;

[0029] Step 2.1.4: The regional feature alignment network samples different points of the feature map and applies bilinear interpolation to obtain the region of interest with a size of 7*7;

[0030] Step 2.1.5: Branch prediction network, obtain boundary area information, category information and mask set B, B = {b uv |u=1,2,…,H,v=1,2,…,W}, for any element b in set B, uv Indicates whether the pixel at row u and column v in image I is the pallet area, b uv =1 indicates the pallet area, otherwise it indicates the non-pallet area;

[0031] Step 2.1.6: The loss function is designed as shown in formula (6):

[0032] Loss = L cls +L bbox +L mask (6)

[0033] Among them, L cls Represents the classification loss value of the target, L bbox Represents the regression loss value of the boundary area, L mask Indicates the loss value of the mask part;

[0034] Step 2.2: Training the convolutional network. The network is pre-trained using the MS COCO dataset. Based on the pre-trained model, a dataset of pallet images captured by the RGB-D camera is used. Transfer learning is performed on this dataset to obtain the pallet detection model M. The training tool is PyTorch, and the optimizer is SGD.

[0035] Step 2.3: Pallet segmentation. For any image I input to the convolutional neural network, obtain the output mask image B. Based on the correspondence set IS between the image I and the point cloud data set S, obtain the pallet point cloud data set SS. The specific steps include:

[0036] Step 2.3.1: Traverse element b in mask set B uv , for satisfying b uv =1 are denoted as u* and v*;

[0037] Step 2.3.2: Traverse the image and point cloud correspondence set IS, and take all elements i in the set IS that satisfy formula (7) to form the pallet point cloud identification set and record it as the set F = {f j |j=1,2,...,V};

[0038]

[0039] Where V represents the number of elements in the set F, that is, the number of pixels on the pallet;

[0040] Step 2.3.3: Traverse the set F and the point cloud set S, and take all the triples (x i ,y i ,z i ) constitutes the pallet point cloud set and is recorded as SS = {(x k ,y k ,z k )|k=1,2,...,K};

[0041]

[0042] Where K represents the number of points in the pallet point cloud set SS;

[0043] Step 3: Pallet pose estimation: Principal component analysis is used to estimate the normal vector of the point cloud set SS, and the specific pose of the pallet is calculated according to formula (9):

[0044]

[0045] Among them, x k ∈SX,y k ∈SY, z k ∈SD, (x k ,y k ,z k )∈SS,SX={x k |i=1,2,...,K}、SY={y k |i=1,2,...,K}、SD={z k |i=1,2,…,K}, SX, SY, and SD are the sets of x, y, and z coordinate values ​​extracted from the point cloud set SS, K represents the number of points in the point cloud set SS, V(·) represents the normal vector function based on principal component analysis, (d, l, s) represent the relative position of the center point of the front face of the pallet with respect to the forklift, and θ represents the orientation of the front face of the pallet with respect to the forklift.

[0046] The advantages of the present invention are: first, a pallet detection method for forklifts is implemented, which can be used to guide forklifts to dock pallets; second, the pallet detection method based on instance segmentation can accurately estimate the pallet's posture due to the accurate extraction of the pallet's area and point cloud data; third, in the pallet segmentation method based on instance segmentation, the addition of an attention mechanism module makes the instance segmentation edge more accurate, and more effectively improves the accuracy of the pallet's posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is the network structure diagram of the improved MaskR-CNN;

[0048] Figure 2 This is the network structure diagram of the attention mechanism module. DETAILED DESCRIPTION

[0049] The present invention will be further described below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In this embodiment, the network structure diagram of the improved Mask R-CNN is shown in FIG. Figure 1 As shown, the network structure of the attention mechanism module Figure 2 As shown, this method includes the following steps:

[0050] Step 1: Data acquisition: Install the RGB-D camera on the front of the forklift to obtain the image I, the point cloud data set S, and the corresponding set IS of the image I and the point cloud set S; where image I = {(r uv ,g uv ,buv )|u=1,2,...,W;v=1,2,...,H}, W represents the width of image I, H represents the height of image I, CH represents the number of channels of image I, (r uv ,g uv ,b uv ) represents the R, G, and B values ​​of the pixel in the vth row and uth column in the image I; the point cloud set S = {(x i ,y i ,z i )|i=1,2,…U}, U represents the number of points in the point cloud set S; the corresponding set IS={(u i ,v i )|i=1,2,…U},(u i ,v i ) represents the pixel at the vth row and uth column in the image I corresponding to the i-th point in the point cloud set S; in this embodiment, W=1280, H=720, CH=3;

[0051] Step 2: Pallet segmentation: Use convolutional neural network to train pallet sample images to obtain the trained pallet detection network model M P , using the pallet detection network model M P Extract the pallet area and segment the pallet image, such as Figure 1 As shown, specifically:

[0052] Step 2.1: Construct a convolutional neural network, modifying the Mask R-CNN model, including feature extraction of the backbone network, multi-scale feature integration with attention mechanism, region proposal network, region feature alignment, branch prediction network, and loss function design;

[0053] Step 2.1.1: Backbone network feature extraction: a standard convolutional neural network (ResNet) is used as the backbone network to extract features and obtain pallet features C. In this embodiment, ResNet101 is used as the backbone network.

[0054] Step 2.1.2: Multi-scale feature integration with attention mechanism, add attention mechanism module A in FPN to obtain multi-level feature prediction; specifically, in the bottom-up network of FPN, obtain multi-level feature vectors C2, C3, C4, C5; add attention mechanism module A in the FPN horizontal connection; in the top-down network of FPN, perform upsampling operation on feature vector C5 and feature vectors M4 and M3 output by the attention mechanism module, and finally obtain feature vectors M5, M4, M3, M2 containing attention information; in the convolution fusion of FPN, perform conv convolution operation on each feature vector in the feature set {M2, M3, M4, M5} to obtain the final prediction, and obtain feature vector set {P2, P3, P4, P5}, which corresponds to the feature set {C2, C3, C4, C5} respectively; in this embodiment, the interval of upsampling operation up is 2; where, as Figure 2 As shown, the attention mechanism model A specifically includes:

[0055] Step 2.1.2.1: Attention mechanism module input: feature vector C i With the eigenvector M i+1 When input to the attention module;

[0056] Step 2.1.2.2.: For the eigenvector C i With the eigenvector M i+1 Splice and obtain the feature vector A Conv , as shown in formula (1):

[0057] A Conv =concat(C i ,up( 2) (M i +1)) (1)

[0058] Among them, concat(·) represents the concatenation function, up (2) Represents the upsampling function up with a pixel interval of 2, A Conv The size is w*h*(c1+c2); where c1 represents C i The number of channels of the eigenvector, c2 represents the eigenvector M i+1 The number of channels; in this embodiment, c1 and c2 are both 256;

[0059] Step 2.1.2.3: For feature A Conv Perform conv_1 convolution operation to obtain feature layer A Conv1 , as shown in formula (2):

[0060]

[0061] Among them, ω(c2,1×1) Represents the conv_1 convolution operation function, the kernel of the convolution operation is 1×1, and the number of channels c is the feature vector M i+1 The number of channels is c2, stride is 2, A Conv1 The size of is w*h*c2; in this embodiment, c2 is 256;

[0062] Step 2.1.2.4: For feature A Conv1 Perform ReLU function, conv_2 convolution function and Sigmoid function operations respectively to obtain feature layer A sig , as shown in formula (3):

[0063] A Sig =σ(ω (c1,1×1) (ReLU(A Conv1 ))) (3)

[0064] Where ReLU(·) represents the ReLU function, ω (c,1×1) Represents the conv_2 convolution operation function, the kernel of the convolution operation is 1×1, and the number of channels c is the feature vector C i The number of channels is c1, the stride is 2, and σ(·) represents the Sigmoid activation function, then the feature vector A sig The size of is w*h*c1; in this embodiment, c1 is 256;

[0065] Step 2.1.2.5: Place A sig and M i+1 Multiply the two feature matrices to obtain the output feature vector M of the attention module i , as shown in formula (4):

[0066]

[0067] Among them, M i The feature size is w×h and the number of channels is c1;

[0068] Step 2.1.2.6: Output feature vector M of the attention module i Perform the convolution operation of conv to obtain the predicted feature P i , as shown in formula (5):

[0069] P i =ω (c,3×3) (M i ) (5)

[0070] Among them, ω (c,3×3) Represents the conv convolution operation function, the kernel of the convolution operation is 3×3, and the number of channels c is c1; P iThe size of the feature layer is w*h*c1; in this embodiment, the stride of the conv convolution operation function is 2, the padding is 1, and the number of channels is 256;

[0071] Step 2.1.3: Region Proposal Network, output P according to the feature i Generate proposed regions and perform feature extraction;

[0072] Step 2.1.4: The regional feature alignment network samples different points of the feature map and applies bilinear interpolation to obtain the region of interest with a size of 7*7;

[0073] Step 2.1.5: Branch prediction network, obtain boundary area information, category information and mask set B, B = {b uv |u=1,2,…,H,v=1,2,…,W}, for any element b in set B, uv Indicates whether the pixel at row u and column v in image I is the pallet area, b uv =1 indicates the pallet area, otherwise it indicates the non-pallet area;

[0074] Step 2.1.6: The loss function is designed as shown in formula (6):

[0075] Loss = L cls +L bbox +L mask (6)

[0076] Among them, L cls Represents the classification loss value of the target, L bbox Represents the regression loss value of the boundary area, L mask Indicates the loss value of the mask part;

[0077] Step 2.2: Training the convolutional network. The network is pre-trained using the MSCOCO dataset. Based on the pre-trained model, a dataset of pallet images collected by the RGB-D camera is used. Transfer learning is performed on this dataset to obtain a pallet detection model M. The training tool is pytorch and the optimizer is SGD. In this embodiment, the initial learning rate is 0.001, the momentum and weight decay are 0.9 and 0.0001 respectively, and the number of iterations is 100,000.

[0078] Step 2.3: Pallet segmentation. For any image I input to the convolutional neural network, obtain the output mask image B. Based on the correspondence set IS between the image I and the point cloud data set S, obtain the pallet point cloud data set SS. The specific steps include:

[0079] Step 2.3.1: Traverse element b in mask set Buv , for satisfying b uv =1 are denoted as u* and v*;

[0080] Step 2.3.2: Traverse the image and point cloud correspondence set IS, and take all elements i in the set IS that satisfy formula (7) to form the pallet point cloud identification set and record it as the set F = {f j |j=1,2,...,V};

[0081]

[0082] Where V represents the number of elements in the set F, that is, the number of pixels on the pallet;

[0083] Step 2.3.3: Traverse the set F and the point cloud set S, and take all the triples (x i ,y i ,z i ) constitutes the pallet point cloud set and is recorded as SS = {(x k ,y k ,z k )|k=1,2,...,K};

[0084]

[0085] Where K represents the number of points in the pallet point cloud set SS;

[0086] Step 3: Pallet pose estimation: Principal component analysis is used to estimate the normal vector of the point cloud set SS, and the specific pose of the pallet is calculated according to formula (9):

[0087]

[0088] Among them, x k ∈SX,y k ∈SY, z k ∈SD, (x k ,y k ,z k )∈SS,SX={x k |i=1,2,...,K}、SY={y k |i=1,2,...,K}、SD={z k |i=1,2,…,K}, SX, SY, and SD are the sets of x, y, and z coordinate values ​​extracted from the point cloud set SS, K represents the number of points in the point cloud set SS, V(·) represents the normal vector function based on principal component analysis, (d, l, s) represent the relative position of the center point of the front face of the pallet with respect to the forklift, and θ represents the orientation of the front face of the pallet with respect to the forklift.

[0089] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A pallet pose estimation method based on attention mechanism, characterized in that: The steps include: Step 1: Data acquisition: Install the RGB-D camera on the front of the forklift to obtain the image I, the point cloud set S, and the corresponding set IS of the image I and the point cloud set S; where image I = {(r uv ,g uv ,b uv )|u=1,2,...,H;v=1,2,...,W}, W represents the width of image I, H represents the height of image I, CH represents the number of channels of image I, (r uv ,g uv ,b uv ) represents the R, G, and B values ​​of the pixel in the uth row and vth column in the image I; the point cloud set S = {(x i ,y i ,z i )|i=1,2,...,U}, U represents the number of points in the point cloud set S; the corresponding set IS={(u i ,v i )|i=1,2,...,U},(u i ,v i ) indicates that the i-th point in the point cloud set S corresponds to the pixel in the u-th row and v-th column in the image I; Step 2: Pallet segmentation: Use a convolutional neural network to train the pallet sample image to obtain a trained pallet detection network model MP. Use the pallet detection network model MP to extract the pallet area and achieve pallet image segmentation. Specifically: Step 2.1: Construct a convolutional neural network, modifying the Mask R-CNN model, including feature extraction of the backbone network, multi-scale feature integration with attention mechanism, region proposal network, region feature alignment, branch prediction network, and loss function design; Step 2.1.1: Backbone network feature extraction: Use the standard convolutional neural network ResNet as the backbone network for feature extraction to obtain the pallet feature C1; Step 2.1.2: Multi-scale feature integration with attention mechanism: Add attention mechanism module A to FPN to obtain multi-level feature prediction; In the bottom-up network of FPN, multi-level feature vectors C2, C3, C4, and C5 are obtained; an attention mechanism module A is added to the FPN horizontal connection; in the top-down network of FPN, feature C5 and the output features M4 and M3 of the attention mechanism module are upsampled, and finally intermediate feature vectors M5, M4, M3, and M2 containing attention information are obtained; in the convolutional fusion of FPN, each feature vector in the feature set {M2, M3, M4, M5} is subjected to a conv convolution operation to obtain the final prediction, and a feature vector set {P2, P3, P4, P5} is obtained, which corresponds to the feature set {C2, C3, C4, C5} respectively; wherein, the attention mechanism model A specifically includes: Step 2.1.2.1: Attention mechanism module input: feature vector C i With the eigenvector M i+1 When input to the attention module; Step 2.1.2.2: For the eigenvector C i With the eigenvector M i+1 Perform splicing to obtain the feature vector A Conv , as shown in formula (1): A Conv =concat(C i ,up (2) (M i +1)) (1) Among them, concat(·) represents the concatenation function, up (2) represents the up sampling function, A Conv The size is w*h*(c1+c2); where c1 represents C i The number of channels of the eigenvector, c2 represents the eigenvector M i+1 The number of channels; Step 2.1.2.3: For the eigenvector A Conv Perform conv_1 convolution operation to obtain feature layer A Conv1 , as shown in formula (2): Among them, ω (c2,1×1) Represents the conv_1 convolution operation function, the kernel of the convolution operation is 1×1, the number of channels c is c2, A Conv1 The size of is w*h*c2; Step 2.1.2.4: For feature A Conv1 Perform ReLU function, conv_2 convolution function and Sigmoid function operations respectively to obtain feature layer A sig , as shown in formula (3): A Sig =σ(ω (c1,1×1) (ReLU(A Conv1 ))) (3) Where ReLU(·) represents the ReLU function, ω (c,1×1) Represents the conv_2 convolution operation function, the kernel of the convolution operation is 1×1, the number of channels c is c1, σ(·) represents the Sigmoid activation function, then the feature vector A sig The size of is w*h*c1; Step 2.1.2.5: Place A sig and M i+1 Multiply the two feature matrices to obtain the output feature vector M of the attention module i , as shown in formula (4): Among them, M i The feature size is w×h and the number of channels is c1; Step 2.1.2.6: Output feature vector M of the attention module i Perform the convolution operation of conv to obtain the output feature P of the AFPN network with attention mechanism i , as shown in formula (5): P i =ω (c,3×3) (M i ) (5) Among them, ω (c,3×3) Represents the conv convolution operation function, the kernel of the convolution operation is 3×3, the number of channels c is c1, P i The size of the feature layer is w*h*c1; Step 2.1.3: Region Proposal Network, output P according to the feature i Generate proposed regions and perform feature extraction; Step 2.1.4: The regional feature alignment network samples different points of the feature map and applies bilinear interpolation to obtain the region of interest with a size of 7*7; Step 2.1.5: Branch prediction network, obtain boundary area information, category information and mask set B, B = {b uv |u=1,2,…,H,v=1,2,…,W}, for any element b in set B, uv Indicates whether the pixel at row u and column v in image I is the pallet area, b uv =1 indicates the pallet area, otherwise it indicates the non-pallet area; Step 2.1.6: The loss function is designed as shown in formula (6): Loss=L cls +L bbox +L mask (6) Among them, L cls Represents the classification loss value of the target, L bbox Represents the regression loss value of the boundary area, L mask Represents the loss value of the mask; Step 2.2: Training the convolutional network. The network is pre-trained using the MSCOCO dataset. Based on the pre-trained model, a dataset of pallet images collected by the RGB-D camera is used. Transfer learning is performed on this dataset to obtain the pallet detection model M. The training tool is PyTorch, and the optimizer is SGD. Step 2.3: Pallet segmentation. For any image I input to the convolutional neural network, obtain the output mask image B. Based on the correspondence set IS between the image I and the point cloud data set S, obtain the pallet point cloud data set SS. The specific steps include: Step 2.3.1: Traverse element b in mask set B uv , for satisfying b uv =1 are denoted as u* and v*; Step 2.3.2: Traverse the image and point cloud correspondence set IS, and take all elements i in the set IS that satisfy formula (7) to form the pallet point cloud identification set and record it as the set F = {f j |j=1,2,...,V}; Where V represents the number of elements in the set F, that is, the number of pixels on the pallet; Step 2.3.3: Traverse the set F and the point cloud set S, and take all the triples (x i ,y i ,z i ) constitutes the pallet point cloud set and is recorded as SS = {(x k ,y k ,z k )|k=1,2,...,K}; Where K represents the number of points in the pallet point cloud set SS; Step 3: Pallet pose estimation: Principal component analysis is used to estimate the normal vector of the point cloud set SS, and the specific pose of the pallet is calculated according to formula (9): Among them, x k ∈SX,y k ∈SY, z k ∈SD, (x k ,y k ,z k )∈SS,SX={x k |i=1,2,...,K}、SY={y k |i=1,2,...,K}、SD={z k |i=1,2,…,K}, SX, SY, and SD are the sets of x, y, and z coordinate values ​​extracted from the point cloud set SS, K represents the number of points in the point cloud set SS, V(·) represents the normal vector function based on principal component analysis, (d, l, s) represent the relative position of the center point of the front face of the pallet with respect to the forklift, and θ represents the orientation of the front face of the pallet with respect to the forklift.

Images