Film-sleeved hard carton unstacking method based on deep learning

Abstract

The invention provides a film-covered hard strip box unstacking method based on deep learning, and relates to the technical field of hard strip box unstacking, and the method comprises the following steps: obtaining image data of a film-covered hard strip box; segmenting a pixel region where the hard bar box is located from the image data through a detection model; converting the pixel area into three-dimensional coordinate data corresponding to the hard strip box; according to the three-dimensional data of the hard strip box, positioning a central point coordinate value of the hard strip box; and unstacking the hard strip boxes according to the central point coordinate values of the hard strip boxes. According to the method, the YOLOv8 model is adopted to accurately recognize the film-sleeved hard strip box, the pixel area where the hard strip box is located is segmented from the image information, the pixel area is converted into the three-dimensional coordinate data corresponding to the hard strip box, and the coordinate value of the center point is positioned, so that the film-sleeved hard strip box is better in unstacking effect, and the unstacking success rate is effectively increased.

Description

Technical Field

[0001] The present invention relates to the technical field of depalletizing strip boxes, and in particular, to a method for depalletizing film-coated rigid strip boxes based on deep learning. Background Art

[0002] In domestic packaging workshops, cigarette cartons are mostly unpacked manually, which has the disadvantages of occupying a large space, high labor intensity, and cumbersome procedures. Many cigarette cartons robot unpacking systems have also been designed on the market, using advanced mechanical, electrical and other technologies to achieve automated sorting of cigarette cartons, thereby improving production efficiency and product quality, and reducing labor and logistics costs. However, due to the film on the cartons, conventional image recognition methods such as template matching often fail to accurately identify the cartons, which ultimately leads to robot unpacking failure.

[0003] With the rapid development of artificial intelligence technology, deep learning technology represented by convolutional neural networks has been widely used in the field of image recognition. Deep learning is now used for tobacco leaf grade classification and feature visualization, as well as for cigar cigarette appearance quality detection, and has achieved the expected results. At present, the YOLO series of algorithms are still widely used in image recognition, which has a good speed-accuracy balance and realizes end-to-end training and detection. Among them, the single-stage detector YOLO v1 innovatively proposed to divide the image into multiple grids (Grid Cell), and then predict the bounding box and category probability for each grid. Subsequently, the original YOLOv1 team successively proposed YOLOv2 and YOLO v3, which improved the detection accuracy and speed. YOLO v3 replaced the backbone network with DarkNet53, used the idea of ​​FPN to detect targets, and achieved an accuracy of 79.3% on the PASCAL VOC12 dataset. Subsequently, YOLOv4, which achieved 65.7% mAP on the COCO dataset, and YOLO v5, which integrated a large number of Tricks based on YOLO v3 and whose latest version achieved 68.9% mAP on the COCO dataset and surpassed YOLOv4 in accuracy and speed, were proposed one after another.

[0004] However, there is still no recognition technology for film-coated rigid-strip cartons. Even with conventional deep learning to recognize the information of rigid-strip cartons for depalletizing, it is still impossible to effectively identify and depalletize film-coated rigid-strip cartons, and the depalletizing failure rate of film-coated rigid-strip cartons is still very high.

[0005] Therefore, in order to solve the above problems, it is necessary for us to design a reasonable method for depalletizing rigid boxes with film sleeves based on deep learning. Summary of the invention

[0006] The purpose of the present invention is to provide a method for destacking film-coated rigid strip boxes based on deep learning, which adopts the YOLOv8 model to accurately identify the film-coated rigid strip boxes, segment the pixel area where the rigid strip boxes are located from the image information, convert the pixel area into three-dimensional coordinate data corresponding to the rigid strip boxes, and locate the coordinate value of the center point, so that the destacking effect of the film-coated rigid strip boxes is better and the destacking success rate is effectively improved.

[0007] The target rigid box is thus depalletized.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A method for depalletizing a film-wrapped rigid box based on deep learning, comprising the following steps:

[0010] Acquire image data of the sleeve rigid strip box;

[0011] Segment the pixel area where the hard box is located from the image data through the detection model;

[0012] Convert the pixel area into three-dimensional coordinate data corresponding to the hard box;

[0013] Locate the center point coordinate value of the rigid box according to the three-dimensional coordinate data of the rigid box;

[0014] The hard box is depalletized according to the coordinate value of the center point of the hard box.

[0015] As one of the preferred embodiments of the present invention, before segmenting the pixel area where the hard box is located from the image information, a detection model is constructed;

[0016] The construction of the detection model includes BackBone part, PANet part and Output part;

[0017] The BackBone part includes C2F module and SPPF module; the PANet part includes several Conv sampling modules and C2F modules; the Output part includes a decoupling head structure module, which is used to separate the classification module and the detection module;

[0018] Among them, the C2F module contains a convolutional layer and a fully connected layer, and an attention mechanism is added to the C2F module; the SPPF module is used to convert feature maps of any size into feature vectors of fixed size.

[0019] As one of the preferred embodiments of the present invention, the pixel area where the hard box is located is segmented from the image information, specifically:

[0020] The image information is input into the detection model. The BackBone part of the detection model divides the image into S*S grids. In each grid, the C2F module is used to detect the hard box as the target. The detected target hard box is sampled by the Conv sampling module of the PANet part. The C2F module obtains the center point coordinates and boundary point coordinates of the target hard box. Finally, the target is decoupled through the Output, and the area where the target hard box is located is output classified and the confidence test is performed on the area where the target hard box is located, and the pixel area where the hard box is located in the image is obtained.

[0021] As one of the preferred embodiments of the present invention, after the detection model is constructed, a plurality of sets of film-wrapped rigid box images are provided to be input into the detection model for detection model training.

[0022] As one of the preferred embodiments of the present invention, when the detection model is trained, the detection model further includes a loss calculation module and a label assignment module connected to the Output;

[0023] The loss calculation module includes the classification loss calculation module and the regression loss calculation module

[0024] When the detection model is trained, the detection model is trained based on the training samples and the preset benchmark value as the target, and the model output is adjusted and calculated according to the loss parameter calculated by the classification loss calculation module. When the predicted value of the model output calculated by the regression loss calculation module is the same as the preset benchmark value, the model converges and the training is completed, and the label assignment module is used to assign labels to the output hard box pixel area;

[0025] Among them, the classification loss calculation module uses VFL Loss for calculation;

[0026] The regression loss calculation module uses DFL loss + CIOU Loss for calculation;

[0027] In the label assignment module, the Task-AlignedAssigner positive and negative sample assignment strategy is adopted.

[0028] As one of the preferred embodiments of the present invention, the pixel area is converted into three-dimensional coordinate data corresponding to the rigid box, specifically:

[0029] The camera is used to obtain preset images of the depalletizing station at multiple angles; a laser scanner is used to scan the depalletizing station to obtain the laser point information of each point of the depalletizing station, and three-dimensional coordinate point cloud data is constructed; and the preset image and the point cloud data are paired and bound;

[0030] Construct a mapping relationship between the position of each point in the preset image and the three-dimensional coordinate point cloud data;

[0031] According to the mapping relationship between the position of each point of the preset image and the three-dimensional coordinate cloud data, the pixel area of ​​the sleeve film hard box image is converted into the three-dimensional coordinate data corresponding to the hard box.

[0032] As one of the preferred embodiments of the present invention, the hard box is destacking according to the coordinate value of the center point of the hard box, specifically:

[0033] Send the center point coordinate value of the rigid box to the joint robot;

[0034] The articulated robot is driven to reach the position of the hard box;

[0035] The fixture at the front end of the articulated robot completes the depalletizing of the hard box;

[0036] The clamp is a composite clamp, which includes three functional structures: a clamping assembly for grabbing hard boxes, a suction cup assembly for grabbing partition paper, and a connecting rod clamp hook assembly for grabbing and stacking pallets.

[0037] The beneficial effect of the destacking method of the film-coated rigid strip boxes based on deep learning of the present invention is that: the film-coated rigid strip boxes are accurately identified by using the YOLOv8 model, the pixel area where the rigid strip boxes are located is segmented from the image information, the pixel area is converted into three-dimensional coordinate data corresponding to the rigid strip boxes, and the coordinate value of the center point is located, so that the destacking effect of the film-coated rigid strip boxes is better, and the destacking success rate is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic flow chart of a method for depalletizing a sleeve rigid box based on deep learning according to the present invention;

[0039] Figure 2 A schematic diagram of a YOLOv8 model framework in a method for depalletizing a film-encased rigid box based on deep learning according to the present invention;

[0040] Figure 3 This is a schematic structural diagram of a depalletizing device in a method for depalletizing a sleeve rigid box based on deep learning according to the present invention. DETAILED DESCRIPTION

[0041] The following are specific embodiments of the present invention, which further describe the technical solutions of the present invention, but the present invention is not limited to these embodiments.

[0042] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangements of modules and structures set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0043] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0044] Technologies, methods, and systems known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and systems should be considered part of the authorization specification.

[0045] Embodiment 1: Figure 1 What is shown is only one of the embodiments of the present invention, a method for depalletizing film-coated rigid strip boxes based on deep learning, based on the current fully automatic multi-degree-of-freedom robotic arm robot hardware equipment, and introducing the current YOLO series algorithm as software, to design a high-precision depalletizing method for film-coated rigid strip boxes, wherein the YOLO series algorithm adopts the newly developed YOLOv8 model, which can be used for image recognition of film-coated rigid strip box materials, and visual positioning of film-coated rigid strip box materials, and then the positioning result is sent to the multi-degree-of-freedom robotic arm robot, and the film-coated rigid strip boxes are effectively depalletized by multi-degree-of-freedom and multi-clamps. In short, the depalletizing of film-coated rigid strip boxes is completed by high-precision visual positioning based on deep learning and composite clamps in cooperation with articulated robots.

[0046] Specifically, the method of the present invention comprises the following steps:

[0047] S100: Acquire image data of the film-covered rigid box;

[0048] S200: Segmenting the pixel area where the hard box is located from the image data by using a detection model;

[0049] S300: Convert the pixel area into three-dimensional coordinate data corresponding to the hard box;

[0050] S400: Locating the coordinate value of the center point of the rigid box according to the three-dimensional data of the rigid box;

[0051] S500: Destacking the rigid box is completed according to the coordinate value of the center point of the rigid box.

[0052] The existing hard box depalletizing technology directly obtains the image of the hard box through a camera, and then directly converts the image into the three-dimensional coordinate data corresponding to the hard box. Finally, the robotic arm completes the depalletizing of the hard box according to the three-dimensional coordinate data. This method can only obtain the hard box data through ordinary graphic contours. Once the outside of the hard box is covered with an irregular film, there may be errors in grasping the hard box, resulting in depalletizing failure.

[0053] The present invention is aimed at destacking of film-covered rigid boxes. The image of the rigid boxes is still obtained by a camera, but the image data is input into a detection model to segment the pixel area where the rigid boxes are located, and the pixel area is converted into three-dimensional coordinate data corresponding to the rigid boxes. In this way, high-precision recognition of the film-covered rigid boxes can be performed. Finally, the three-dimensional coordinate data of the rigid boxes are still converted into center point coordinate values ​​for positioning. Finally, the composite fixture cooperates with the articulated robot to complete the destacking of the rigid boxes according to the center point coordinate value.

[0054] When executing step S200, the detection model is a YOLOv8 model;

[0055] Before segmenting the pixel area where the hard box is located from the image information, build a YOLOv8 detection model;

[0056] The construction of the detection model, the YOLOv8 model specifically includes the BackBone part, the PANet part and the Output part;

[0057] The BackBone part includes the C2F module (cross stage partial feature fusion module, from convolution to full connection) and the SPPF module (Spatial Pyramid Pooling-Fast); the PANet part includes several Conv sampling modules and C2F modules; the Output part includes a decoupling head structure module, which is used to separate the classification module and the detection module;

[0058] Among them, the C2F module contains a convolutional layer and a fully connected layer, and an attention mechanism is added to the C2F module; the SPPF module is used to convert feature maps of any size into feature vectors of fixed size.

[0059] Then, when executing step 200 to segment the pixel area where the hard box is located from the image information, specifically:

[0060] The image information is input into the detection model. The BackBone part of the detection model divides the image into S*S grids. The C2F module is used in each grid to detect the hard box as the target. Each grid is responsible for detecting whether there is a target in the grid and obtaining the location and category of the target.

[0061] The detected target rigid box is sampled by the Conv sampling module of the PANet part, and the C2F module obtains the coordinates of the center point and the boundary point of the target rigid box, that is, the area where the target rigid box is located;

[0062] Finally, the target is decoupled through Output, and the area where the target hard box is located is output classified and confidence detected to obtain the pixel area where the hard box is located in the image.

[0063] Specifically, the YOLOv8 model divides the image into S*S grids. If the center point of a specific target (object) in the image happens to belong to a certain grid after the division, the target will be predicted by this specific grid; each grid after the division will predict the bounding box of the specific target and its confidence score. Each predicted bounding box will include five parameters: x, y, w, h and confidence. Among them, (x, y) is the coordinate of the center point of the bounding box, which is the relative position of the center point of the bounding box and the grid. (w, h) is the width and height of the bounding box, which is the width and height of the bounding box relative to the entire image. Confidence is the confidence, which reflects the possibility of the specified target predicted by the network in the bounding box.

[0064] The edge, fold, and text of the cardboard box are all specific targets; each specific target has a center point coordinate (x, y) and multiple bounding box coordinates (w, h), which express the location, size, and category of the specific target.

[0065] If there is no specified target in the grid, the target index Pr(Object) = 0, and the predicted specified target confidence is also equal to 0. If there is a target whose center falls in the grid, the target index Pr(Object) = 1, and the location (x, y) and confidence of its center point are obtained according to the detection network, and the coordinates of each specific target location on the hard box are calculated according to the confidence comparison of specific targets in multiple grids and the loss value of the matching model.

[0066] In short, after the YOLOv8 model calculates the confidence of the coordinates of the center point of each specific target in the image and the parameters of the bounding box, we can get the area where each specific target is located, find the pixel point corresponding to this area in the image, and form a pixel area, which is the pixel area where the hard box is located.

[0067] The biggest differences between the present invention and the existing depalletizing technology are the following two points: 1. The positioning data source adopts high-precision visual positioning of deep learning of the YOLOv8 model; 2. The depalletizing behavior is completed by a composite fixture in conjunction with an articulated robot.

[0068] In summary, the present invention adopts the YOLOv8 model to achieve higher recognition accuracy for the film-covered rigid strip boxes, segments the pixel area where the rigid strip boxes are located from the image information, converts the pixel area into three-dimensional coordinate data corresponding to the rigid strip boxes, and locates the center point coordinate value, so that the composite clamp cooperates with the articulated robot to accurately destacker the film-covered rigid strip boxes, thereby achieving better destacking effect for the film-covered rigid strip boxes and effectively improving the destacking success rate.

[0069] Embodiment 2: Figure 1 , 2 The figure is only one of the embodiments of the present invention. On the basis of the first embodiment, in a method for depalletizing a sleeve rigid box based on deep learning of the present invention, after constructing the detection model, several groups of sleeve rigid box images are provided to be input into the detection model for detection model training; several groups of sleeve rigid box images are provided, for example, 400 sleeve rigid box material images, the training set and the verification set adopt a ratio of 7:3, all image data are extracted, screened, and annotated in the YOLOv8 model to obtain training data, and a trained YOLOv8 detection model is obtained.

[0070] YOLO is a real-time target detection algorithm that divides an image into S*S grids, each of which is responsible for predicting whether there is a target in the grid and the location and category of the target. This design enables YOLO to complete target detection in one forward propagation, greatly improving the detection speed. In addition, YOLO also uses technologies such as multi-scale feature fusion and global loss function to further improve the detection accuracy; the YOLOv8 model is used in this application;

[0071] The construction of the YOLOv8 model is as follows:

[0072] Based on the YOLOv5 model, the YOLOv5 model includes the BackBone part, the PANet (path aggregation network) part, and the Head Output part;

[0073] In BackBone, the CSP (Cross Stage Partial) module concept of the YOLOv5 model is still used, but the C3 module (Concentrated-Comprehensive Convolution Module) in YOLOv5 is replaced by the C2f module (cross stage partial feature fusion module), making the entire model more lightweight. The SPP module in YOLOv5 is replaced by the SPPF module, and models of different scales are fine-tuned to improve the performance of the entire model.

[0074] Specifically, in the BackBone part of the YOLOv8 model, the input data is detected by at least 4 groups of Conv convolution modules and C2f modules, and the input data of different scales are preliminarily processed and reach the PANet part, such as Figure 2 shown.

[0075] In PANet, the Conv1*1 sampling layer module in YOLOv5 is deleted, and the Conv module sampling is performed directly, and the C3 module in YOLOv5 is replaced by the C2f module; thus, in the PANet part of the YOLOv8 model, multiple sampling data are directly sampled and transmitted to the Head Output part through at least three groups of Concat modules and C2f modules, such as Figure 2 shown.

[0076] In Head Output, the YOLOv8 model uses a decoupled head structure to separate the classification module and the detection module. That is, each group of data sampled and transmitted by the PANet part is decoupled into two parts, and the loss is calculated after passing through a Conv module and a Concat module respectively. For example, Figure 2 shown.

[0077] In addition, when training the detection model, the detection model also includes a loss calculation module and a label assignment module connected to the Output;

[0078] The loss calculation module includes a classification loss calculation module and a regression loss calculation module;

[0079] When the detection model is trained, the detection model is trained based on the training samples and with the preset benchmark value as the target. The model output is adjusted and calculated according to the loss parameter calculated by the classification loss calculation module. When the predicted value of the model output calculated by the regression loss calculation module is the same as the preset benchmark value, the model converges and the training is completed. The label assignment module is used to assign labels to the output hard strip box pixel area so that when the detection model is trained, the loss parameter of the sleeve hard strip box image recognition is calculated;

[0080] Among them: the classification loss calculation module uses VFL Loss for calculation, as shown in formula (1.1);

[0081]

[0082] Among them, p represents the probability of the text existing as predicted by the model, and q represents the probability of the text existing in the actual label.

[0083] When q>0, the loss function is given by the cross entropy loss function, which measures the consistency between the predicted probability of text existence and the actual label. When q=0, the loss function is given by another term, which is a custom function with parameters α and γ to handle the case where text does not exist. This loss function is designed to consider both the presence and absence of text. By adjusting the α and γ parameters, the loss weights of these two cases can be balanced. This loss function can help the model better extract text information from images.

[0084] The regression loss calculation module uses DFL loss + CIOU Loss for calculation.

[0085] Among them, the DFL loss target recognition loss is shown in formula (1.2);

[0086] DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(yy i )log(S i+1 )) (1.2)

[0087] Among them, S i and S i+1 Respectively represent the model's predicted probability for the i-th character and the i+1-th character in the text sequence; y i and i+1 is the label value of the corresponding character in the actual label.

[0088] The loss function is based on the cross entropy loss function, which is used to measure the consistency between the character probability predicted by the model and the actual label character. The loss function is calculated by weighting the character probability predicted by the model according to the position of the actual label character. If the model's prediction at position i matches the label, the loss function is affected by y i+1 -y penalty; if the model's prediction at position i+1 matches the label, the loss function is penalized by yy i By minimizing this loss function, the model can better extract text information from images and improve the accuracy of text recognition.

[0089] Among them, the CIOULoss target recognition loss is shown in formula (1.3);

[0090]

[0091] Where IoU represents the intersection over union ratio between the predicted text area and the true text area, Dc represents the diagonal length of the text area, and D 2 Indicates the length of the short side of the text area.

[0092] CIoU represents a comprehensive loss function, which consists of the intersection over union (IoU), the length of the diagonal, and an adjustment parameter αv; α is a dynamic adjustment parameter determined by the ratio of the intersection over union (IoU) and another parameter v. Parameter v is a penalty term based on the aspect ratio of the text area, which is used to penalize the difference between the aspect ratio and the reference aspect ratio. h is the height of the text area, and w is the width of the text area. The arctan function in the formula is used to calculate the aspect ratio of the text area, and the difference between the predicted text area shape and the reference shape is penalized by the squared difference. By minimizing this loss function, the model can be better extracted from the image and the accuracy of text detection can be improved.

[0093] In the label assignment module, the YOLOv8 model abandons the forgotten IoU assignment or unilateral proportional assignment method and adopts the Task-AlignedAssigner positive and negative sample assignment strategy, as shown in formula (1.4);

[0094] t=S A *u B (1.4)

[0095] This formula describes a feature fusion method for text detection and recognition tasks. Here, s and u represent feature vectors from two different sources, and a and B are the corresponding weight matrices. The fused feature vector t is obtained by weighting the feature vectors s and u with the weight matrices a and B respectively, and then multiplying them. This feature fusion method can help the model comprehensively utilize feature information from different sources and improve the accuracy and robustness of text detection and recognition.

[0096] In summary, the YOLOv8 model can greatly improve the pixel detection speed in the image of the film-encased rigid box, improve the accuracy and robustness of the recognition of the film-encased rigid box, and solve the problem of robot depalletizing failure caused by inaccurate recognition.

[0097] Embodiment 3: Figure 1 , 2 , 3 are only one of the embodiments of the present invention. On the basis of any of the above embodiments, the present invention still relies on a destacking device in a method for destacking a film rigid box based on deep learning. The destacking device structure is as follows Figure 3 shown.

[0098] When depalletizing, open the main power cabinet 01, and send the pallet with hard strip boxes to the hard strip box waiting station 09 through the AGV trolley. After the hard strip box depalletizing station 08 completes a pallet depalletizing process and is emptied, the materials on the hard strip box pallet on the hard strip box waiting station are sent to the depalletizing station through the chain conveyor, and after the positioning is completed by the image recognition system 02, a signal is given to the robot 03 to drive the multi-functional clamp 04 to clamp the two rows of hard strip boxes and transfer them to the downstream machine; when all the hard strip boxes on the hard strip box depalletizing station are taken away, the machine The robot drives the multifunctional clamp to grab the partition from the hard box depalletizing station to the partition stacking station 11. When the partitions stacked on the partition stacking station reach the preset number of layers, the chain machine will transport the partition pallet to the partition pallet recovery station 07, and the AGV will be responsible for transferring the pallet stacked with partitions to the designated recycling area 10. After the partition stacking station is emptied, the chain transplanter on the empty pallet waiting station 06 will transfer the pallet to the partition stacking station, and the empty pallet waiting station will be replenished by the pallet emptied from the hard box depalletizing station. If there is no need for an empty pallet at the empty pallet waiting station and the hard box and partitions on the pallet at the hard box depalletizing station are emptied, the pallet will be transferred to the pallet code extension machine 05 through multiple sets of chain machines and chain transplanters. When the pallet code extension machine is full, the chain machine will send the pallet stack to the pallet recovery station, waiting for the AGV to transfer the pallet stack to the designated recycling area.

[0099] In this way, based on the block coding method and bagging packaging method of the rigid box pallet, the system completes the depalletizing of the film-covered rigid boxes through high-precision visual positioning based on deep learning and a composite fixture in conjunction with an articulated robot, which can realize the forward and reverse judgment of the rigid box and the precise positioning of the relative coordinate values ​​of each rigid box relative to the mortar plate.

[0100] When executing step S300, converting the pixel area into the three-dimensional coordinate data corresponding to the rigid box, specifically:

[0101] S310: obtaining preset images of the depalletizing station at multiple angles through a camera; scanning the depalletizing station with a laser scanner to obtain laser point information of each point of the depalletizing station, and constructing three-dimensional coordinate point cloud data;

[0102] S320: Constructing a mapping relationship between the position of each point of the preset image and the three-dimensional coordinate point cloud data;

[0103] S330: According to the mapping relationship between the position of each point of the preset image and the three-dimensional coordinate point cloud data, the pixel area of ​​the sleeve film hard strip box image is converted into the three-dimensional coordinate data corresponding to the hard strip box.

[0104] That is, laser technology is used to continuously scan the coordinate point data of the surface of the objects on the depalletizing station to form three-dimensional coordinate point cloud data. At the same time, multi-angle images of the surface of the objects on the depalletizing station are obtained through the camera, and the images are bound to the point cloud data one by one to generate a mapping relationship between the two. Then, only the pixel area of ​​the image of the sleeve rigid box needs to be obtained, and the pixel of each point can be brought into the mapping relationship to obtain the three-dimensional coordinate data of the sleeve rigid box.

[0105] As one of the embodiments of the present invention, when performing step S500 and completing the destacking of the rigid box according to the coordinate value of the center point of the rigid box, specifically:

[0106] S510: Send the center point coordinate value of the rigid box to the joint robot;

[0107] S520: The joint robot is driven to reach the position of the hard box;

[0108] S530: The fixture at the front end of the articulated robot completes the depalletizing of the hard box;

[0109] The clamp is a composite clamp, which includes three functional structures: a clamping assembly for grabbing hard boxes, a suction cup assembly for grabbing partition paper, and a connecting rod clamp hook assembly for grabbing and stacking pallets.

[0110] Embodiment 4: Figure 1 , 2 3 is only one of the embodiments of the present invention. Based on any of the above embodiments, in a method for depalletizing a film-wrapped rigid box based on deep learning, during actual training, a hardware platform of 48GB memory, Geforce RTX 2080Ti GPU, Intel(R) Core(TM) i7-9700KCPU and Windows operating system are used, the deep learning environment is CUDA11.1, CUDNN8.2.0, torch-1.9.0, and the compilation environment is Pycharm / python language.

[0111] In addition, the training parameters include input image size of 640*640, batchsize of 8, no pre-training weights, 300 training rounds, AdamW optimizer and its parameters, learning rate of 0.01, momentum of 0.937, and three parameter groups are defined, each of which includes parameter index, parameter type and weight decay settings; the first parameter group includes weight, and its weight decay is 0.0; the second parameter group also includes weight, but its weight decay is 0.0005; the last parameter group includes bias, and its weight decay is 0.0.

[0112] Furthermore, after executing step S100 and obtaining the image data of the sleeve rigid box, the image data needs to be preprocessed using various image enhancement techniques; the preprocessing includes image hue, saturation, brightness enhancement, image translation, flipping, cropping, scaling, mosaic enhancement, etc., wherein the hue enhancement coefficient is 0.015, the saturation enhancement coefficient is 0.7, the brightness enhancement coefficient is 0.4, the translation coefficient is 0.1, and the scaling coefficient is 0.5.

[0113] In the object detection of the film-coated hard-strip box, the evaluation indicators commonly used are: accuracy (Precision), recall rate (Recall), average precision (Average Precision, AP), mean average precision (mean Average Precision, mAP), etc., and the formula is shown in formula (1.5);

[0114]

[0115] Among them, TP represents the number of correctly detected targets; FN represents the number of incorrectly detected targets; FP represents the number of undetected targets; and m represents the number of categories.

[0116] Recall represents the ratio of predicted boxes to real boxes, and Precision represents the ratio of successful predictions in the prediction results. A PR curve is established with accuracy as the ordinate and recall as the abscissa. The area enclosed by the curve and the abscissa is defined as AP. The average AP value of each class is taken to get mAP. The larger the mAP, the better the overall detection effect of the model and the higher the detection accuracy.

[0117] Experience shows that the training effect is good when using the labeled data set for training. The accuracy and recall are close to 1. After about 200 rounds of training, the mAP50 is close to 1, and after about 290 rounds of training, the mAP50-95 reaches 0.8, which can achieve the expected recognition effect. After testing with the trained model, the pixel area of ​​each part of the film box can be effectively obtained, and then according to the mapping relationship between the image and the point cloud data, the pixel area is converted into the three-dimensional coordinates corresponding to each box, and the coordinate value of the center point of each box is located.

[0118] In summary, the use of YOLOv8 for rigid box material identification improves the accuracy and robustness of sleeve rigid box identification, and solves the problem of robot depalletizing failure caused by inaccurate identification. Compared with traditional manual depalletizing, the rigid box robot depalletizing system based on deep learning can reduce labor costs, meet 60 pieces / minute high-speed rigid box sorting, and eliminate the situation of being unable to depalletize due to recognition failure. The speed of automated depalletizing is much higher than that of manual operation, and the number of manual operations is reduced. The site is clean and orderly, which is conducive to future capacity expansion. The test results show that the rigid box depalletizing system can realize fully automatic depalletizing, replace manual labor with mechanization, improve production efficiency, and ensure stable product quality.

[0119] The present invention discloses a method for destacking film-coated rigid strip boxes based on deep learning. The method adopts the YOLOv8 model to accurately identify the film-coated rigid strip boxes, segments the pixel area where the rigid strip boxes are located from image information, converts the pixel area into three-dimensional coordinate data corresponding to the rigid strip boxes, and locates the coordinate value of the center point. In this way, the destacking effect of the film-coated rigid strip boxes is better, and the destacking success rate is effectively improved.

[0120] The present invention is not limited to the above specific implementations, and the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made to the above implementations based on the technical essence of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for depalletizing a film-wrapped rigid box based on deep learning, characterized in that: The following steps are involved: Acquire image data of the sleeve rigid strip box; Segment the pixel area where the hard box is located from the image data through the detection model; Convert the pixel area into three-dimensional coordinate data corresponding to the hard box; Locate the center point coordinate value of the rigid box according to the three-dimensional coordinate data of the rigid box; The hard box is depalletized according to the coordinate value of the center point of the hard box.

2. The method for depalletizing a film-wrapped rigid box based on deep learning according to claim 1, characterized in that: Before segmenting the pixel area where the hard box is located from the image information, a detection model is constructed; The construction of the detection model includes BackBone part, PANet part and Output part; The BackBone part includes C2F module and SPPF module; the PANet part includes several Conv sampling modules and C2F modules; the Output part includes a decoupling head structure module, which is used to separate the classification module and the detection module; Among them, the C2F module contains a convolutional layer and a fully connected layer, and an attention mechanism is added to the C2F module; the SPPF module is used to convert feature maps of any size into feature vectors of fixed size.

3. The method for depalletizing a film-wrapped rigid box based on deep learning according to claim 2, characterized in that: The pixel area where the hard box is located is segmented from the image information, specifically: The image information is input into the detection model. The BackBone part of the detection model divides the image into S*S grids. In each grid, the C2F module is used to detect the hard box as the target. The detected target hard box is sampled by the Conv sampling module of the PANet part. The C2F module obtains the center point coordinates and boundary point coordinates of the target hard box. Finally, the target is decoupled through the Output, and the area where the target hard box is located is output classified and the confidence test is performed on the area where the target hard box is located, and the pixel area where the hard box is located in the image is obtained.

4. The method for depalletizing a film-wrapped rigid box based on deep learning according to claim 2, characterized in that: After the detection model is constructed, several groups of film-wrapped rigid box images are provided as input to the detection model for detection model training.

5. The method for depalletizing a film-wrapped rigid box based on deep learning according to claim 4, characterized in that: When training the detection model, the detection model also includes a loss calculation module and a label assignment module connected to the Output; The loss calculation module includes a classification loss calculation module and a regression loss calculation module; When the detection model is trained, the detection model is trained based on the training samples and the preset benchmark value as the target, and the model output is adjusted and calculated according to the loss value calculated by the classification loss calculation module and the regression loss calculation module. When the predicted value output by the model is the same as the preset benchmark value, the model converges and the training is completed, and the label is assigned to the output hard box pixel area through the label assignment module; The loss calculation module includes a classification loss calculation module and a regression loss calculation module; Among them, the classification loss calculation module uses VFL Loss for calculation; The regression loss calculation module uses DFLloss+CIOU Loss for calculation.

6. The method for depalletizing a film-wrapped rigid box based on deep learning according to claim 1, characterized in that: Convert the pixel area into the three-dimensional coordinate data corresponding to the hard box, specifically: Acquire preset images of the depalletizing station at multiple angles through the camera; Use a laser scanner to scan the depalletizing station, obtain the laser point information of each point of the depalletizing station, and construct three-dimensional coordinate point cloud data; Construct a mapping relationship between the position of each point in the preset image and the three-dimensional coordinate point cloud data; According to the mapping relationship between the position of each point of the preset image and the three-dimensional coordinate point cloud data, the pixel area of ​​the sleeve film hard strip box image is converted into the three-dimensional coordinate data corresponding to the hard strip box.

7. The method for depalletizing a film-wrapped rigid box based on deep learning according to claim 1, characterized in that: The hard box is depalletized according to the coordinate value of the center point of the hard box. Specifically: Send the center point coordinate value of the rigid box to the joint robot; The articulated robot is driven to reach the position of the hard box; The fixture at the front end of the articulated robot completes the depalletizing of the hard box; The clamp is a composite clamp, which includes three functional structures: a clamping assembly for grabbing hard boxes, a suction cup assembly for grabbing partition paper, and a connecting rod clamp hook assembly for grabbing and stacking pallets.

Images