Industrial tray detection method

Abstract

The invention discloses an industrial tray detection method. The method specifically comprises the following steps: S1, acquiring a tray data set; S2, constructing an AM-Mask R-CNN network model; and S3, training and detecting. Wherein a feature extraction layer of the AM-Mask R-CNN network model is a ResNet50-CBAM-FPN layer, and a CBAM attention mechanism is introduced into an effective feature layer extracted by a Resnet network and the last three output feature layers, so that channel features of a tray main body are enhanced, channel features of goods and backgrounds are inhibited, the capacity of extracting tray features by the network is improved, and the characterization capacity of a feature map is enhanced. Therefore, the network can better capture key features in the image, so that the performance of the network is improved, and meanwhile, the robustness of the model is improved. According to the invention, the unmanned industrial vehicle can efficiently and accurately detect different types of trays in different scenes.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to an industrial pallet detection method based on a neural network model. Background Art

[0002] Cargo handling is one of the core tasks of warehousing logistics. Due to the differences in size, shape, quality, etc., it is not easy to transport goods in batches. In actual industrial applications, pallets are usually used to transport goods. Pallets are devices that serve as load-bearing surfaces and include auxiliary structural parts on the load-bearing surfaces when items are organized into cargo units during transportation, handling, and storage. They are bridges connecting production, transportation, warehousing, and distribution, and have universal specifications and shapes. As a carrier for cargo loading, unloading, storage, and transportation, storage pallets can facilitate the smooth consignment of goods and the integration of supply chains.

[0003] In warehousing and logistics, traditional logistics operations mainly rely on manually operated forklifts to move pallets, with a low degree of automation. AGV is an autonomous driving vehicle that can effectively reduce labor costs. The premise for realizing the automated handling of goods through AGV is to accurately identify and locate the handling target, that is, pallet detection. Pallet detection belongs to the category of target detection. It is the application and expansion of target detection technology in the field of intelligent logistics, and has become a research hotspot in the warehousing field. However, with the increasing complexity of the warehousing environment, unstructured scenes, occlusions, lighting changes and other problems make pallet detection challenging. Therefore, improving the robustness of the algorithm, overcoming the detection difficulties of multi-scene and multi-type pallets in complex environments, and meeting the high-precision work requirements of unmanned industrial vehicles in complex environments are of great significance to the realization of unmanned and intelligent warehousing.

[0004] In recent years, convolutional neural networks are one of the most popular methods used in the field of image processing. Deeply integrating convolutional neural networks with industrial image data and combining them with industrial robots to complete corresponding tasks to achieve less-manned and automated logistics is a direction worth studying in the future. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention proposes an industrial pallet detection method, which overcomes the shortcomings of low precision and poor robustness in the existing pallet detection technology, and realizes efficient and accurate detection of pallets of different scenarios and types by unmanned industrial vehicles.

[0006] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0007] An industrial pallet detection method, characterized in that it comprises the following steps:

[0008] S1. Obtain a pallet dataset, where the pallet dataset includes a training set, a validation set, and a test set;

[0009] S2. Construct an AM-Mask R-CNN network model, wherein the AM-Mask R-CNN network model specifically includes:

[0010] ResNet50-CBAM-FPN feature extraction layer uses ResNet50 as the base network and combines the CBAM attention mechanism and FPN structure to extract feature maps;

[0011] The region production layer uses RPN to slide a small network on the feature map to generate a set of candidate RoIs;

[0012] RoIAlign layer, spatially calibrating the candidate RoIs and extracting the calibrated RoIs;

[0013] The task execution layer classifies and regresses each of the calibrated RoIs and generates a segmentation mask for each RoI to achieve pixel-level object segmentation;

[0014] S3, training and testing, inputting the training set and the validation set into the AM-Mask R-CNN network model for training and verification, and inputting the test set into the pre-trained AM-Mask R-CNN network model for testing, and outputting the target detection result.

[0015] Among them, ResNet50-CBAM-FPN introduces the CBAM attention mechanism in the effective feature layer extracted by the ResNet network and the last three output feature layers. Specifically, by enhancing the channel features of the pallet body and suppressing the channel features of the cargo and background, the network's ability to extract pallet features can be significantly improved, thereby enhancing the representation ability of the feature map. The feature extraction process of ResNet50-CBAM-FPN mainly includes the following steps: After the pallet dataset is input into the ResNet50-CBAM-FPN feature extraction network, it is first processed by the convolution layer and the residual block. On this basis, the CBAM module first applies the channel attention module, which obtains the descriptor of each channel through global average pooling and global maximum pooling, and then uses these descriptors to calculate the weight of each channel to determine which channels are more important and which channels can be ignored or reduced in importance. After the channel attention module, CBAM further applies the spatial attention module, using 2D convolution operations to calculate the weight of each spatial position, with the aim of determining which spatial positions in the input feature map are more important. Finally, the features output by CBAM are input into the FPN (Feature Pyramid Networks) module, where upsampling and downsampling operations are performed respectively, and finally the extracted feature maps are output.

[0016] The region generation layer (RPN) first slides a small sliding window (usually 3×3 size) on the feature map O. The center position of the window corresponds to an anchor point on the original image. For each anchor point, RPN predicts multiple bounding boxes of different sizes and aspect ratios, and predicts the coordinates of each bounding box as well as the object score. Finally, RPN uses non-maximum suppression (NMS) to remove overlapping proposals and outputs a set of final object proposals.

[0017] The region of interest alignment (ROIAlign) layer is used to accurately extract features of target proposals from feature maps. The input is the feature map obtained by ResNet-CBAM-FPN described in S34 and the target proposal generated by RPN described in S35. First, for each target proposal, RoIAlign maps the target proposal to the feature map. Next, RoIAlign defines a fixed number of sampling points on the mapped proposal and extracts features of these points from them through bilinear interpolation. For each sampling point, RoIAlign performs bilinear interpolation using the four feature points around it to obtain the feature value of the point. For each target proposal, RoIAlign finally outputs a feature map of fixed size.

[0018] The task execution layer classifies and regresses each of the calibrated RoIs, generates a segmentation mask for each RoI to achieve pixel-level object segmentation, and outputs the target result. The target result includes the image category, image location, and corresponding segmentation mask.

[0019] Since ResNet-FPN cannot fully integrate and utilize shallow detail information and deep semantic information, the feature extraction layer in this technical solution is the ResNet50-CBAM-FPN layer. By introducing the CBAM attention mechanism in the effective feature layer extracted by the Resnet network and the last three output feature layers, the channel attention module weights the different channels of the feature map by learning the importance of each channel. This allows the network to automatically identify which channels are critical for identifying the pallet body and assign higher weights to these channels. The feature channels of the pallet body (for example, the part of the pallet with a specific texture, shape or color) are given higher weights, while the channels related to the background and cargo are given lower weights, thereby enhancing the pallet features in the channel dimension. The channel attention in CBAM is adaptive, which means that it can dynamically adjust the weights according to different input images. This adaptive learning enables the network to be optimized for different scenarios and pallet types. In different environments and conditions, the network can continue to focus on the pallet body instead of being disturbed by changes in the background or cargo.

[0020] Therefore, using the ResNet50-CBAM-FPN layer as the feature extraction layer can enhance the channel characteristics of the pallet body, suppress the channel characteristics of the goods and background, improve the network's ability to extract pallet features, and enhance the representation ability of feature maps. This helps the network better capture key features in the image, thereby improving its performance and the robustness of the model. This enables unmanned industrial vehicles to efficiently and accurately detect different types of pallets in different scenarios.

[0021] As a preferred embodiment of the present invention, the AM-Mask R-CNN network model further includes a color space conversion layer, which converts the pallet data set from an RGB color space to an HSV color space and inputs the color space into the ResNet50-CBAM-FPN feature extraction layer.

[0022] The RGB color space is an unevenly distributed color space, and the color difference between two colors cannot be directly expressed as the distance between two points in the corresponding color space. The HSV color space is a color model based on perception. Compared with the RGB space, it has stronger recognition ability and is more in line with human visual characteristics.

[0023] As a preferred embodiment of the present invention, the S3, training and testing, specifically comprises the following steps:

[0024] S31, constructing a loss function, wherein the loss function includes a coordinate loss function, a confidence loss function and a classification loss function, and calculating corresponding coordinate loss values, confidence loss values ​​and classification loss values;

[0025] S32, calculating the gradient of the loss function to the model parameters by a back propagation algorithm;

[0026] S33. According to the calculated gradient, the model parameters are updated using an optimization algorithm to reduce the loss value; the process of forward propagation, loss calculation, back propagation and parameter update is repeated until a stopping condition is reached.

[0027] As a preferred embodiment of the present invention, the loss function is a CDFL hybrid function, and the CDFL hybrid function is composed of CELoss, Dice Loss and Focal Loss.

[0028] In order to better capture the characteristics of the data and the requirements of the model, this paper uses a hybrid loss function to narrow the gap between the predicted results and the true values ​​and achieve high confidence in the boundaries of the segmented image.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention proposes a pallet detection method based on Mask R-CNN: 1. HSV color space is used as the input of the network, 2. ResNet-CBAM-FPN is used as the feature extraction layer, and 3. CDFL hybrid function is constructed as the loss function. Compared with the traditional Mask R-CNN, AM-Mask R-CNN can be effectively applied to pallet detection tasks in complex storage environments, has a better ability to overcome problems such as unstructured scenes, occlusion and illumination changes, and shows good robustness. Its beneficial effects are mainly reflected in the following aspects:

[0031] First, the AM-Mask R-CNN model has better segmentation and recognition capabilities, and can more accurately identify pallets at a distance. At the same time, it improves the positioning accuracy of pallets, fully meeting the high-precision and real-time working requirements of unmanned industrial vehicles, and provides reliable data for the efficient and accurate estimation of pallet postures by subsequent unmanned industrial vehicles. Secondly, the model can ensure the segmentation accuracy in complex working environments, especially in the edge segmentation accuracy of pallets at a distance, which is significantly better than the traditional Mask R-CNN. Finally, the segmentation effect of this method is better than other methods, and the model has good generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 It is a flowchart for obtaining image dataset;

[0034] Figure 2 This is the network structure diagram of AM-Mask R-CNN;

[0035] Figure 3 This is the Resnet50-CBAM-FPN structure diagram;

[0036] Figure 4 It is a schematic diagram of the test results. DETAILED DESCRIPTION

[0037] The present invention is further described below by specific examples and drawings. The examples of the present invention are intended to enable those skilled in the art to better understand the present invention, and do not limit the present invention in any way.

[0038] This embodiment provides an industrial pallet detection method, the flow chart is as follows Figure 1 As shown, the following specific steps are included:

[0039] S1. Get the pallet dataset

[0040] Build an image acquisition environment

[0041] An automatically controlled reach forklift was selected as the research object, and the industrial vision sensor PercipioFM851-E23D vision sensor was used to collect color images. The ranging subject is an active binocular with a working range of 0.7-6.0m. The vision sensor is installed on the top of the forklift fork frame, and the sensor moves up and down with the fork. In addition, in order to be able to capture the pallets on the ground and the shelves, the camera of the vision sensor is slightly tilted downward by 10°. According to the angle and horizontal deviation adjustment requirements of the forklift AGV, the forklift AGV stops at 500mm from the front face of the pallet and collects scene data. When collecting scene data, the pallet needs to be placed on a flat ground to ensure that the fork is perpendicular to the front face of the pallet and the center of the sensor is in a straight line with the center of the pallet.

[0042] Video data recording and image frame extraction

[0043] After the image acquisition environment is set up, video data in the real working environment is recorded, and color data of trays with different colors, different placement angles, different distances, and different placement environments are captured. In the experiment, the trays used are blue Sichuan-shaped plastic trays, black Sichuan-shaped plastic trays, and wooden Tian-shaped trays, with dimensions of 1200mm×1000mm×150mm. Video data of a day's normal operation of an unmanned forklift in a warehousing environment is collected. Using the commercial video processing software Premiere, each operation video is frame-extracted at intervals of 10s and converted into frame data, with an image resolution of 1920×1080PPI. The frame data is preprocessed, including but not limited to image, image scaling, image normalization, data augmentation, image format conversion, image denoising, etc., to ensure the quality of the dataset.

[0044] Image annotation

[0045] The images generated in step S22 are identified by manually using the labelme software, and all tray shapes, positions, and categories that appear during the operation of the unmanned forklift in the images are annotated to obtain a tray dataset. In addition, the obtained tray dataset format is the COCO dataset format. The COCO dataset includes all training image folders, all validation image folders, and corresponding annotation folders. The corresponding annotation folders include training sets, validation sets, and test set json annotation files for corresponding object detection and segmentation tasks.

[0046] The tray dataset is split into a training set, a validation set, and a test set in the ratio of 8:1:1 for subsequent training. In an example of this embodiment, the annotation includes 3 types of trays: blue Sichuan-shaped plastic trays, black Sichuan-shaped plastic trays, and wooden Tian-shaped trays.

[0047] S2. Build an AM-Mask R-CNN industrial tray detection model

[0048] The structural schematic diagram of the AM-Mask R-CNN network is as Figure 2 shown. For a tray image with an input shape of P×Q, first adjust its size to M×N, then convert the RGB space to the HSV color space, map the image to a feature map through ResNet50-CBAM-FPN, use RPN to generate a set of candidate RoIs, extract accurate RoI features from the feature map through the RoIAlign layer, perform object classification and bounding box regression on each RoI to determine the category and position of the object, and use the segmentation mask branch to generate a segmentation mask for each RoI. In addition, combine CE Loss, Dice Loss, and Focal Loss to form a CDFL hybrid function as the loss function to guide the training process.

[0049] S2.1. RGB to HSV

[0050] The color image acquired by the Percipio FM851-32 vision sensor contains the coordinates and color information of the target to be measured. The RGB color space is an unevenly distributed color space, and the color difference between two colors cannot be directly expressed as the distance between two points in the corresponding color space. The HSV color space is a color model based on perception. Compared with the RGB space, it has stronger recognition ability and is more in line with human visual characteristics. This paper uses the HSV color space for feature extraction, where H represents the hue of the point cloud, that is, the position of the spectral color, S represents the saturation of the point cloud, and V represents the brightness of the point cloud color. The three color components of the HSV color space are extracted to represent the color features of each key point PFi in the point cloud. The conversion relationship from RGB space to HSV space is shown in equations (1)-(3), where the value range of R, G, and B is [0,255], the value range of H is [0,360], the value range of S is [0,1], and the value range of V is [0,255].

[0051] V=max(R,G,B) (1)

[0052]

[0053] If H<0, then H=H+360

[0054] S2.2, map the image into a feature map through ResNet50-CBAM-FPN;

[0055] esNet is a deep convolutional neural network architecture that solves the gradient vanishing and gradient exploding problems in deep network training by introducing residual blocks. Its core idea is to introduce skip connections in each residual block, allowing the gradient to be directly back-propagated to earlier layers.

[0056] CBAM (Convolutional Block Attention Module) is an attention mechanism for computer vision tasks such as image classification and object detection

[18] . The CBAM module can generate channel attention maps and spatial attention maps based on the input feature map to improve the model's attention to key features. Given a feature map F, the channel attention module first generates a channel attention map using Mc, a shared network that focuses on the content of the channel to refine the feature F′. Then, a spatial attention matrix Ms is generated in the spatial dimension to focus on the location of valid features. Finally, F″ is the output of the CBAM module, which has an efficient feature descriptor. The expressions are shown in (4)-(5):

[0057]

[0058] FPN uses a top-down lateral connection architecture to build a feature pyramid within the network from a single-scale input, and extracts RoI features from different levels of the feature pyramid according to the scale. By fusing high-level features with low-level features from top to bottom, the semantics of the output low-level features is improved.

[0059] ResNet50-CBAM-FPN introduces the CBAM attention mechanism in the effective feature layer extracted by the ResNet network and the last three output feature layers. Specifically, by enhancing the channel features of the pallet body and suppressing the channel features of the cargo and background, the network's ability to extract pallet features can be significantly improved, thereby enhancing the representation ability of the feature map. Compared with VGGNet, using ResNet-FPN as the backbone network of MaskR-CNN for feature extraction has certain improvements in accuracy and speed. However, ResNet-FPN has problems such as being unable to fully integrate and utilize shallow detail information and deep semantic information. Therefore, the CBAM attention mechanism is introduced in the effective feature layer extracted by the Resnet network and the last three output feature layers. By enhancing the channel features of the pallet body and suppressing the channel features of the cargo and background, the network's ability to extract pallet features is improved, and the representation ability of the feature map is enhanced. This helps the network better capture the key features in the image, thereby improving its performance and improving the robustness of the model.

[0060] like Figure 3 As shown in the figure, the feature extraction process of Resnet-CBAM-FPN mainly includes the following steps:

[0061] (a) The input image is denoted as I, and its pixel position is (i, j), where i represents the row number and j represents the column number.

[0062] (b) First, the image I passes through the convolutional layer C(I) = w c *I+b c , where w c is the convolution weight, b c is the bias, and * indicates the convolution operation.

[0063] (c) The convolution output C(I) is then input into the residual block R, R(C) = C + F(C), where F(C) is the convolution operation inside the residual block. The convolution output C(I) is then input into the residual block R, R(C) = C + F(C), where F(C) is the convolution operation inside the residual block.

[0064] (d) On this basis, the CBAM module first applies a channel attention module, which obtains the descriptors of each channel through global average pooling and global maximum pooling, and then uses these descriptors to calculate the weight C(SA) of each channel, which aims to determine which channels are more important and which channels can be ignored or have their importance reduced. Where C(SA) is the channel attention weight, Represents channel multiplication.

[0065] (e) After the channel attention module, CBAM further applies a spatial attention module, which uses a 2D convolution operation to calculate the weight of each spatial position, aiming to determine which spatial positions in the input feature map are more important, SA(R) = CA⊙S(R), where S(R) is the spatial attention weight and ⊙ represents element-wise multiplication.

[0066] (f) Finally, the features output by CBAM are input into the FPN module, where upsampling and downsampling operations are performed respectively, and the extracted feature maps are finally output.

[0067] FPN(CA)=U(SA)+D(SA), where U(SA) is the upsampling operation and D(SA) is the downsampling operation. The final output feature O=FPN(CA). The expression of the entire process is shown in formula (6):

[0068] O=FPN(SA(CA(R(C(I)))))(6)

[0069] Among them, the backbone network maps the input image I to a feature map O, and each position (i, j, k) of O represents a feature value with k feature channels: O(i, j, k).

[0070] S2.3. Region Candidates and Screening

[0071] The Proposal Network (RPN) is a fully convolutional network for generating bounding box proposals for objects in images. It shares convolutional features with the backbone network and operates on these features to generate a set of rectangular object proposals, each with an object score.

[0072] The input image is represented as I, and the feature map obtained by ResNet-CBAM-FPN is represented as O. In RPN, a small sliding window (usually 3x3 size) is first slid on the feature map O, and the center position of this window corresponds to an anchor point on the original image. For each anchor point, RPN predicts bounding boxes of multiple sizes and aspect ratios, and predicts an object score for each bounding box. P(O) = w p *O+b p , where w pis the weight of RPN, b p is the bias, * indicates the convolution operation, and P(O) is the output of the RPN, including the coordinates of the bounding box (x, y, w, h) and the object score s. Where x, y are the coordinates of the center of the bounding box, and w, h are the width and height of the bounding box. The object score s is a value between 0 and 1, indicating whether the bounding box contains an object. Finally, the RPN uses non-maximum suppression (NMS) to remove overlapping proposals and output a set of final target proposals.

[0073] The expression of the entire RPN process is shown in formula (7):

[0074] P=NMS(P(O)) (7)

[0075] Among them, O(i,j,k) is the feature value of each position of the feature map obtained by ResNet-CBAM-FPN, i and j are spatial positions, and k is the number of channels.

[0076] S2.4. RoI feature extraction

[0077] RoIAlign (Region of Interest Align) is a key component in Faster R-CNN, which is used to accurately extract the features of the object proposal P from the feature map O without any spatial misalignment problem.

[0078] Input the feature map O obtained by ResNet-CBAM-FPN and the target proposal P generated by RPN. First, for each target proposal p in P, RoIAlign first maps p to the feature map O, represented as Map(p). Then, RoIAlign defines a fixed number of sampling points on the mapped proposal Map(p) and extracts the features of these points from O using bilinear interpolation. For each sampling point, RoIAlign uses the four feature points around it for bilinear interpolation to obtain the feature value B(O,Map(p)) of the point. For each target proposal p, RoIAlign outputs a fixed-size feature map, such as 7×7×k, where k is the number of channels. This output is represented as A(p,O).

[0079] The expression of the entire RoIAlign layer process is as follows:

[0080] A(p,O)=B(O,Map(p)) (8)

[0081] A={A(p i ,O)|p i ∈P} (9)

[0082] Where: A(p,O) is the feature extracted from O using RoIAlign for proposal p, Map(p) is the operation of mapping proposal p to feature map O. B is the bilinear interpolation operation, and A is the set of extracted features of all target proposals.

[0083] S2.5. Pallet Recognition and Instance Segmentation

[0084] Object detection and instance segmentation are advanced tasks in deep learning that require not only identifying objects in an image, but also determining their precise locations (via bounding boxes) and generating a segmentation mask for each object.

[0085] Input: input image I, feature map O obtained by ResNet-CBAM-FPN, object proposal P generated by RPN, fixed-size feature A extracted from O by RoIAlign.

[0086] (1) Object Classification: For each proposal feature a in A, a classifier is used to predict the object category in the proposal. This can be represented as Classify(a), which outputs a class score vector S. At the same time, a regressor is used to fine-tune the proposed coordinates to more accurately match the location of the object. This can be represented as Regress(a), which outputs a corrected bounding box B.

[0087] (2) Mask segmentation: For each proposed feature a in A, a pixel-level segmentation mask of the object is generated for each RoI through the segmentation mask branch. This allows each object to be accurately segmented. This can be represented as Segment(a), and a binary mask M is output, as shown in (10)-(12)

[0088] S(a,O)=Classify(A(p i ,O)) (10)

[0089] B(a,O)=Regress(A(p i ,O)) (11)

[0090] M(a,O)=Segment(A(p i ,O)) (12)

[0091] Where: S(a,O) is the class score vector of the object, B(a,O) is the corrected bounding box, and M(a,O) is the binary mask of the object.

[0092] The expression of the entire target detection and instance segmentation process is as follows:

[0093] (S,B,M)={(S(a i ,O),B(a i,O),M(a i ,O))|a i ∈A} (13)

[0094] where (S, B, M) is the set of class scores, corrected bounding boxes, and binary masks of all object proposals.

[0095] S3. Training and testing

[0096] Input the training set and validation set into the AM-Mask R-CNN network model for training and validation, and input the test set into the pre-trained AM-Mask R-CNN network model for detection, and output the target detection results. Specifically: Construct a loss function, which includes a coordinate loss function, a confidence loss function, and a classification loss function, and calculate the corresponding coordinate loss value, confidence loss value, and classification loss value. Calculate the gradient of the loss function to the model parameters through the back-propagation algorithm. According to the calculated gradient, use the optimization algorithm to update the model parameters to reduce the loss value; repeat the process of forward propagation, loss calculation, back-propagation, and parameter update until the stop condition is reached.

[0097] In order to better capture the characteristics of the data and the requirements of the model, this paper uses the CDFL hybrid loss function to narrow the gap between the predicted results and the true values ​​and achieve high confidence in the boundaries of the segmented image. The CDFL hybrid loss is shown in formula (14).

[0098]

[0099] In this way, the model can take into account classification accuracy, regional similarity, learning of difficult samples, and label smoothness during training. Among them, α, β, γ, δ are the weight coefficients of each loss function, which can be determined by cross-validation. Through this comprehensive loss function, the model can fully consider various factors during training, thereby improving its performance in practical applications. The expressions of CE Loss, Dice Loss, and Focal Loss are as follows (15)-(17):

[0100]

[0101] In this formula, n represents the total number of pallet samples, y i represents the true category of the input sample, p i represents the possibility that the sample is 1, and γ represents the modulation coefficient. Correspondingly, TP (True Positive), FP (False Positive), and FN (False Negative) represent the total number of true positives, false positives, and false negatives, respectively.

[0102] During the training process of deep learning models, especially in classification tasks, the model may be overconfident about certain categories, resulting in a decrease in generalization ability on new and unseen data. To solve this problem, label smoothing technology

[23] is introduced into the loss function. Its core idea is to convert hard labels (such as one-hot encoding) into flexible labels so that the model will not be overconfident about a specific category during training. The mathematical expression of label smoothing is as follows:

[0103]

[0104] Where y is the original one-hot encoded label, p is the model's predicted probability, ∈ is a smoothing factor, usually a positive number less than 1, K is the number of categories, and LS(y,p) is the prediction loss.

[0105] In this way, label smoothing technology can prevent the model from overfitting to a specific category during training, thereby improving the generalization ability of the model and reducing the impact of labeling errors on model training.

[0106] In order to prevent the model from performing too perfectly on the training data but performing poorly on unseen test data, the L2 regularization technique is introduced into the loss function to prevent the model from overfitting. The L2 regularization is shown in formula (19).

[0107]

[0108] Among them, λ is a regularization parameter that controls the strength of regularization. When the value of λ increases, the regularization effect will be stronger and the model weight will be smaller; when the value of λ decreases, the regularization effect will weaken. In this way, L2 regularization helps to balance the complexity of the model and its fit on the training data, thereby improving the performance of the model on unseen data.

[0109] The test set is input into the pre-trained AM-Mask R-CNN network model for detection, and the target detection results are output, such as Figure 4As shown. The present invention proposes a pallet detection method based on Mask R-CNN: AM-Mask R-CNN. This method aims to solve the problem that the features of pallets of different types of colors are similar and difficult to accurately distinguish under complex lighting conditions. For this purpose, HSV color space is used as network input. In view of the poor segmentation performance of pallets farther away in the scene, ResNet-CBAM-FPN is used as a feature extraction network, and a CDFL hybrid function is constructed as a loss function. Compared with the traditional Mask R-CNN, this method has higher recognition accuracy and better segmentation performance for small target pallets on the shelf while ensuring the segmentation accuracy of conventional scene images, overcoming the shortcomings of low accuracy and poor robustness of traditional pallet detection methods. The AM-Mask R-CNN model has better segmentation and recognition capabilities, can more accurately identify pallets farther away, and at the same time improves the positioning accuracy of pallets, better meeting the high-precision and real-time working requirements of unmanned industrial vehicles, and providing reliable data for subsequent unmanned industrial vehicles to efficiently and accurately estimate the position and posture of pallets.

[0110] It should be noted that the above embodiments are only detailed descriptions of the preferred embodiments and principles of the present invention. For ordinary technicians in this field, there will be changes in the specific implementation methods based on the ideas provided by the present invention, and these changes are also considered to be within the scope of protection of the present invention.

Claims

1. An industrial pallet detection method, characterized in that: The following steps are involved: S1. Obtain a pallet dataset, where the pallet dataset includes a training set, a validation set, and a test set; S2. Construct an AM-Mask R-CNN network model, wherein the AM-Mask R-CNN network model specifically includes: ResNet50-CBAM-FPN feature extraction layer uses ResNet50 as the base network and combines the CBAM attention mechanism and FPN structure to extract feature maps; The region proposal layer uses RPN to slide a small network on the feature map to generate a set of candidate RoIs; RoIAlign layer, spatially calibrating the candidate RoIs and extracting the calibrated RoIs; The task execution layer classifies and regresses each of the calibrated RoIs and generates a segmentation mask for each RoI to achieve pixel-level object segmentation; S3, training and testing, inputting the training set and the validation set into the AM-Mask R-CNN network model for training and verification, and inputting the test set into the pre-trained AM-Mask R-CNN network model for testing, and outputting the target detection result.

2. The industrial pallet detection method according to claim 1, characterized in that: The AM-Mask R-CNN network model also includes a color space conversion layer, which converts the pallet dataset from RGB color space to HSV color space and inputs it into the ResNet50-CBAM-FPN feature extraction layer.

3. The industrial pallet detection method according to claim 1, characterized in that: The S3, training and testing, specifically includes the following steps: S31, constructing a loss function, wherein the loss function includes a coordinate loss function, a confidence loss function and a classification loss function, and calculating corresponding coordinate loss values, confidence loss values ​​and classification loss values; S32, calculating the gradient of the loss function to the model parameters by a back propagation algorithm; S33. According to the calculated gradient, the model parameters are updated using an optimization algorithm to reduce the loss value; the process of forward propagation, loss calculation, back propagation and parameter update is repeated until a stopping condition is reached.

4. The industrial pallet detection method according to claim 3, characterized in that: The loss function is a CDFL hybrid function, and the CDFL hybrid function is composed of CE Loss, Dice Loss and Focal Loss.

5. An industrial pallet detection method according to any one of claims 1 to 4, characterized in that: The step S1, obtaining the pallet data set, specifically comprises the following steps: S11, acquiring video data, and performing frame extraction conversion on the video data to obtain image frame data; S12, preprocessing the image frame data to obtain a preprocessed image; S13, annotating the preprocessed image with pallet information to obtain the pallet dataset.

6. The industrial pallet detection method according to claim 5, characterized in that: The specific steps of obtaining the video data in S11 are: installing the visual sensor on the top of the forklift fork frame, recording the video data in the real working environment, and the video data includes the color data of the pallet with different colors, different placement angles, different distances, and different placement environments.

7. The industrial pallet detection method according to claim 5, characterized in that: The industrial pallet detection method further comprises a format conversion step after S1, obtaining the pallet dataset: converting the pallet dataset into a COCO dataset.

Images