Bag sorting machine and bag sorting method thereof

Through the automatic finishing technology combined with bag handling machine and camera detection, the cumbersome and instability of packaging bag sorting are solved, efficient and accurate packaging bag sorting is achieved, and work efficiency and safety are improved.

CN117657715BActive Publication Date: 2024-11-29杭州灵智科技数字化装备有限公司
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410051153.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-15
Publication Date
2024-11-29
Estimated Expiration
2044-01-15

AI Technical Summary

Technical Problem

The existing packaging bags are cumbersome and can easily lead to unstable center of gravity, causing the packaging bags to tilt or collapse, reducing work efficiency.

Method used

A bag processing machine was designed, combining robots and cameras to detect the status of packaging bags in real time, and automatically organize the packaging bags through robots to ensure the positive and negative alignment of packaging bags and the orientation of packaging bags is detected using improved YOLOv5 network and image processing algorithm. and quantity.

Benefits of technology

It improves the accuracy and work efficiency of packaging bags, avoids the tilt or collapse of packaging bags when stacking too high, reduces manual labor, and improves the convenience of operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117657715B_ABST
    Figure CN117657715B_ABST
Patent Text Reader

Abstract

The present invention discloses a bag sorting machine and a bag sorting method thereof, which belong to the field of packaging machinery. The bag sorting machine comprises: a frame; the bag sorting machine also comprises: a placement platform, which is arranged at the front end of the frame and is used to place unsorted woven bags; a bag sorting platform, which is arranged at the middle end of the frame and is used to sort the woven bags; a first manipulator, which is arranged on the frame and is located between the placement platform and the bag sorting platform, and is used to move the woven bags on the placement platform to the bag sorting platform in sequence; a conveyor belt, which is arranged on the bag sorting platform and is used to transport the woven bags; the bag sorting platform, which is arranged at the rear end of the frame; a second manipulator, which is arranged on the frame and is located above the bag sorting platform and the bag sorting platform, and is used to transport the woven bags on the bag sorting platform to the bag sorting platform and then sort the woven bags on the bag sorting platform. The beneficial effect of the present application is to provide a bag sorting machine and a bag sorting method thereof that improve work efficiency by automatically sorting packaging bags.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of packaging machinery, in particular to a bag sorting machine and a bag sorting method thereof. Background Art

[0002] At present, the packaging bags needed by the factory are all purchased and used in the factory, and the purchased packaging bags are delivered to the factory in bundles and piles. The front and back of the packaging bags are different. In order to facilitate bagging and use, the operators need to repeatedly sort out the front and back of the packaging bags and make them neat during use, which is very troublesome and reduces work efficiency.

[0003] If the bottom edges of the packaging bags are not aligned during manual sorting, the packaging bags will become unstable after being stacked too much, causing the entire bag to tilt or even collapse. Summary of the invention

[0004] The purpose of the present invention is to provide a bag sorting machine and a bag sorting method thereof. The present invention can sort the bags by detecting the status of the bags in real time through a manipulator and a camera, thereby improving the work efficiency and the accuracy of sorting and stacking the bags, and effectively preventing the bags from tilting or collapsing due to being stacked too high.

[0005] The content of this application is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this application is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] In order to solve the technical problems mentioned in the above background technology section, some embodiments of the present application provide a bag sorting machine, including:

[0007] frame;

[0008] The bag unpacking machine also includes:

[0009] The placing platform is located at the front end of the frame and is used to place unsorted woven bags;

[0010] The bag separation platform is located at the middle end of the frame and is used to separate woven bags;

[0011] The first manipulator is arranged on the frame and located between the placing platform and the bag distributing platform, and is used to move the woven bags on the placing platform to the bag distributing platform in sequence;

[0012] Conveyor belt, installed on the bag distributing platform, used to transport woven bags;

[0013] The bag sorting platform is located at the rear end of the frame;

[0014] The second manipulator is arranged on the frame, located above the bag-separating platform and the bag-sorting platform, and is used to transport the woven bags on the bag-separating platform to the bag-sorting platform and then sort the woven bags on the bag-sorting platform;

[0015] Among them, there are multiple cameras on the bag sorting platform for detecting the status of woven bags.

[0016] A bag sorting method for a bag sorting machine comprises the following steps:

[0017] S1. Collect image information of woven bags placed on the bag sorting platform through a camera;

[0018] S2, preprocessing the collected images of the woven bag;

[0019] S3, performing real-time image detection of woven bags based on the pre-processed images;

[0020] S4. According to the graphic detection results, the second robot and the bag sorting platform move to sort the woven bags.

[0021] Furthermore, in S2, the image preprocessing is to first filter the original image, then grayscale the filtered image, and then perform image enhancement.

[0022] Furthermore, the image filtering uses bilateral filtering, and its formula is

[0023]

[0024] In the formula, g(x,y) is the filtered image, ω s (i,j) is the spatial domain weight, ω r (i,j) is the grayscale domain weight, ω P is the normalization parameter, I(i,j) is the noise image, and Ω is the neighborhood range at pixel (x,y);

[0025] in,

[0026]

[0027]

[0028]

[0029] Furthermore, the image grayscale is to use the weighted average method to perform weighted average on the three components according to the importance of each component using weights. The calculation formula is:

[0030] Gray=ω1R+ω2G+ω3B;

[0031] Among them, R, G, B are the three primary colors, and ω1, ω2, ω3 are weight coefficients.

[0032] Furthermore, the image enhancement process first uses the Sobel operator for edge detection, which is to use two convolution kernels to perform horizontal and vertical edge detection for any point P0 in the image:

[0033] G x (P0)=(P3+2P5+P8)-(P1+2P4+P6);

[0034] G y (P0)=(P6+2P7+P8)-(P1+2P2+P3);

[0035] In the formula, G x (P0) and G y (P0) is the horizontal edge response intensity and the vertical edge response intensity at point P0, and its value indicates the possibility of it being the edge of the image;

[0036] Then perform feature extraction, starting from the upper right corner of the image, traverse all pixels of the image from left to right and from top to bottom, and take the following operations on the pixels: obtain the gradient value of the edge detection image, if the gradient of a point is greater than the threshold T and the gradient value of the pixel point at a horizontal distance L from it is also greater than the threshold T, then this is the edge point of the inner support ring, set all the pixel values ​​between these two points to 255, and set the pixel values ​​of the pixels that do not meet the conditions to 0; at this point, the image enhancement is completed.

[0037] Furthermore, the image detection includes detecting the orientation of the woven bags, detecting whether the woven bags are neatly placed, and detecting the number of woven bags;

[0038] The detection of the orientation of the woven bags and whether the woven bags are neatly placed is performed using the improved YOLOv5 network. The improvements of the YOLOv5 network include:

[0039] Using the wiou loss function: Assume that the anchor box is B = [x, y, w, h] and the target box is B gt =[x gt ,y gt ,w gt ,h gt ], the calculation formula of wiou is as follows:

[0040]

[0041]

[0042]

[0043]

[0044] L WloUv1=R WloU L loU ;

[0045]

[0046] Among them, W i , H i is the size of the overlap between the two frames, W g , H g is the size of the minimum bounding box, α is the weight coefficient, v is the similarity calculation of the aspect ratio index of the two boxes; x, y, w, h are the horizontal and vertical coordinates of the center point of the anchor box, the width and height of the anchor box, respectively, gt ,y gt ,w gt ,h gt are the horizontal and vertical coordinates of the center point of the prediction box, the width and height of the anchor box, respectively. g , H g is the size of the minimum bounding box, and the superscript * indicates that W g , H g Separate it from the computation graph, turn it from a variable into a constant, and prevent R WloU Produces gradients that hinder convergence;

[0047] Use channel attention: First, use global average pooling to convert channel global spatial information into channel descriptors:

[0048]

[0049] Among them, X c Indicates the cth channel X c The value at position (i, j), H p As a global pooling function, the size of the feature map changes from C×H×W to C×1×1. In order to obtain different weights for different channels, the features are then passed through two convolutional layers and sigmoid and ReLu activation functions:

[0050] CA c =σ(Conv(δ(Conv(g c ))));

[0051] Where σ is the sigmoid function and δ is the ReLU function;

[0052] Finally, the input F is converted into c With channel CA c Multiply the corresponding elements by the weights

[0053] Furthermore, improving the training of the YOLOv5 network includes the following steps:

[0054] a. Constructing a data set: Collect a number of forward and reverse woven bag images, annotate them, and divide the annotated data set into a data set with an upward direction and a side direction. The upward data set is divided into a training set and a validation set, with a ratio of one to one;

[0055] b. Send the data set to the YOLOv5 network for pre-training to obtain pre-training weights;

[0056] c. Use the obtained pre-trained weights to send the data set to the improved YOLOv5 network for training again;

[0057] d. Write a script to count the accuracy of the validation set under different thresholds. Set the accuracy below the threshold as a backward woven bag, otherwise it is a forward woven bag;

[0058] e. Test the threshold at the highest accuracy.

[0059] Furthermore, the number of woven bags is detected by using a graph segmentation method, which first performs threshold segmentation on the enhanced image, then performs canny edge detection, and finally uses Hough line detection to output a vector set of detected lines, where the vectors in the set represent the number of woven bags.

[0060] Furthermore, threshold segmentation is performed using the Otsu algorithm, and the solution is as follows:

[0061]

[0062] in:

[0063]

[0064]

[0065] Where: represents the inter-class variance; p1 is the probability of a pixel being classified into C1; p2 is the probability of being classified into C2, m1 is the average gray value of pixels assigned to C1; m2 is the average gray value of pixels assigned to C2; k is the gray level; pi is the probability of a pixel having a gray level of ; i is the gray level i;

[0066] Let {0,1,2,…,L-1} represent the L different gray levels in a digital image of size M×N pixels, n i represents the number of pixels with gray level i, and the probability that the gray level of the pixel is and Suppose we choose a threshold T(k)=k, 0<k<L-1, and use it to threshold the input image into two categories C1 and C2, where C1 consists of all pixels in the image whose grayscale values ​​are in the range [0, k], and C2 consists of all pixels whose grayscale values ​​are in the range [k+1, L-1]; then the probability of a pixel being classified into C1 is The probability of classification to C2 is Then the average gray value of the pixels assigned to C1 is The one assigned to C2 is The average gray value of pixels with gray levels from 0 to k is: The average gray value of the entire image is: The between-class variance is The global variance is:

[0067] Canny edge detection uses Gaussian filtering to process the image after threshold segmentation to amplify the image contrast and enhance the edge contour information. The processing formula is as follows:

[0068]

[0069] Where: f s (x, y) is the result of convolution of Gaussian function and image; G(x, y) is a one-dimensional zero-mean Gaussian kernel function; f(x, y) is the gray value of the original pixel; σ is the standard deviation;

[0070]

[0071] Then calculate the image grayscale gradient: use the Sobel operator horizontal and vertical operators to convolve with the image to calculate dx and dy, as follows:

[0072] dx=f(x,y)·Sobel x (x, y);

[0073] dy=f(x,y)·Sobel y (x, y);

[0074] Then the image gradient amplitude and angle are obtained as follows:

[0075]

[0076] Where M[x, y] is the gradient value, θ M is the gradient direction, dx and dy represent the gradient of the image in the horizontal and vertical directions respectively;

[0077] Finally, the extracted edge features are screened using a double threshold. The final edge pixels are determined by setting two different thresholds, high and low. If the edge gradient of the pixel neighborhood obtained in the third step of the algorithm is greater than the set high threshold, it is determined to be an edge, and if it is less than the set low threshold, it is determined to be a non-edge. Pixels between the thresholds are judged based on whether they are connected to edge pixels. If they are connected, they are determined to be edge feature pixels.

[0078] The Hough transform is a set of straight line vectors detected based on the output of edge feature pixel points. Each straight line in the vector set is represented by a vector (x1, y1, x2, y2) with four elements, where (x1, y1) represents the starting point of the line segment and (x2, y2) represents the end point of the line segment.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] 1. Through the setting of the robot, the packaging bags can be automatically taken, which reduces manual labor and improves work efficiency.

[0081] 2. The camera detects the status of the packaging bag in real time to adjust the front and back and neatness of the packaging bag, so that the operator can use the packaging bag more conveniently when using it. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The illustrative embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application.

[0083] In addition, throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the components and elements are not necessarily drawn to scale.

[0084] In the attached picture:

[0085] Figure 1 is an overall schematic diagram according to Embodiment 1 of the present application;

[0086] Figure 2 It is a schematic diagram of the bag sorting process of the second embodiment;

[0087] Figure 3 It is a flowchart of real-time camera detection;

[0088] Figure 4 It is a flowchart of image preprocessing;

[0089] Figure 5 is a schematic diagram of edge detection;

[0090] Figure 6It is a flowchart of feature extraction;

[0091] Figure 7 It is a flowchart of building a data set;

[0092] Figure 8 is a schematic diagram of the forward detection result;

[0093] Fig. 9 It is a schematic diagram of the test results of multiple woven bags;

[0094] Fig.10 is a schematic diagram of the backward detection result;

[0095] Fig.11 This is a schematic diagram of the detection of woven bags that are not neatly placed;

[0096] Fig.12 It is a schematic diagram of the image segmentation process;

[0097] Fig.13 It is a schematic diagram of adjusting the conveyor belt speed according to the target number of woven bags.

[0098] Reference numerals:

[0099] Embodiment 1

[0100] 100. Bag sorting machine;

[0101] 101, rack;

[0102] 102. Placement platform;

[0103] 103. Bag distribution platform;

[0104] 104. The first manipulator;

[0105] 105. Conveyor belt;

[0106] 106. Bag sorting platform;

[0107] 107. Second manipulator. DETAILED DESCRIPTION

[0108] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0109] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0110] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0111] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0112] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0113] Embodiment 1

[0114] Reference Figure 1 A bag sorting machine 100 includes: a frame 101, a placing platform 102, a bag dividing platform 103, a first manipulator 104, a conveyor belt 105, a bag sorting platform 106, a second manipulator 107, and a third manipulator 108; the placing platform is arranged on the frame 101, located at the front end of the frame 101, and unsorted woven bags are placed on the placing platform 102; the bag dividing platform 103 is arranged on the frame 101, located at the middle end of the frame 101; the first manipulator 104 is arranged on the frame 101, located between the placing platform 102 and the bag dividing platform 103, and is used to place the unsorted woven bags on the placing platform 102 onto the bag dividing platform 103 one by one; the conveyor belt 105 is arranged On the bag sorting platform 103, it is used to transport the woven bags on the bag sorting platform 103 to the bag sorting platform 106; the bag sorting platform 106 is arranged on the frame 101, located at the rear end of the frame 101, and cameras for monitoring the real-time status of the woven bags are installed above and on the side of the bag sorting platform 106, respectively, and the cameras are CCD cameras; the second manipulator 107 is arranged on the frame 101, located above the bag sorting platform 106 and the bag sorting platform 103, and is used to put the woven bags on the conveyor belt 105 onto the bag sorting platform 106, and then tidy up the messy woven bags on the bag sorting platform 106; the first manipulator 104 and the second manipulator 107 both use suction cups to grab the woven bags.

[0115] When using the bag sorting machine 100, the unsorted woven bags are stacked on the placement platform 102, and then the bag sorting machine 100 is started. The first manipulator 104 places the woven bags on the placement platform 102 one by one on the conveyor belt 105 of the bag sorting platform 106. The conveyor belt 105 transports the woven bags. The second manipulator 107 places the woven bags on the conveyor belt 105 on the bag sorting platform 106. Then the second manipulator 107 sorts the messy woven bags according to the real-time status of the woven bags monitored by the camera.

[0116] Through the automatic operation of the manipulator and the conveyor belt 105, the messy woven bags are neatly arranged, automated, and manual labor is reduced, thereby improving efficiency.

[0117] Embodiment 2

[0118] Reference Figure 2-13 , a bag sorting method of a bag sorting machine, comprising the following steps:

[0119] S1. Collect image information of woven bags placed on the bag sorting platform through a camera;

[0120] S2, preprocessing the collected images of the woven bag;

[0121] S3, performing real-time image detection of woven bags based on the pre-processed images;

[0122] S4. According to the graphic detection results, the second robot and the bag sorting platform move to sort the woven bags.

[0123] Reference Figure 2 , which is the working process of the bag sorting machine.

[0124] Reference Figure 3 , which is the process of visual inspection.

[0125] Reference Figure 4 ,In S2, the image preprocessing is to first filter the original image, then grayscale the filtered image, and then perform image enhancement ,processing.

[0126] Image filtering uses bilateral filtering, the formula is:

[0127]

[0128] In the formula, g(x,y) is the filtered image, ω s (i,j) is the spatial domain weight, ω r (i,j) is the grayscale domain weight, ω P is the normalization parameter, I(i,j) is the noise image, Ω is the neighborhood range of the pixel (x,y), where

[0129]

[0130]

[0131]

[0132] Image grayscale uses the weighted average method to average the three components according to the importance of each component using weights. The calculation formula is:

[0133] Gray=ω1R+ω2G+ω3B;

[0134] In the formula, R, G, B represent the three primary colors, and ω1, ω2, ω3 represent weight coefficients.

[0135] Reference Figure 5 , 6 ,The image enhancement process is to first use the Sobel operator for edge detection, and use a 3*3 convolution kernel to convolve the grayscale image for edge detection. The convolution kernel template is as follows Figure 5 (1), assuming that the image to be edge detected is Figure 5 As shown in (1), for any point P0 in this image, two convolution kernels are used to perform edge detection in the horizontal and vertical directions respectively:

[0136] G x (P0)=(P3+2P5+P8)-(P1+2P4+P6)

[0137] G y (P0)=(P6+2P7+P8)-(P1+2P2+P3);

[0138] G x (P0) and G y (P0) is the horizontal edge response intensity and the vertical edge response intensity at point P0, and its value indicates the possibility of being an image edge.

[0139] Then feature extraction starts from the upper right corner of the image, traverses all pixels of the image from left to right and from top to bottom, and takes the following operations on the pixels:

[0140] The first step is to obtain the gradient value g(x, y);

[0141] The second step is to use the following formula to determine whether the pixel point may be an edge point of the inner support ring, g(x, y)>T&&g(x, y)>T;

[0142] Step 3: If the formula is true, this pixel point may be the edge point of the inner support ring, and all pixel values ​​between these two points are set to 255; if the formula is not true, the pixel value of the pixel point is set to 0;

[0143] Where g(x,y) is the gradient value of the edge detection image, T is the gradient threshold, and L is the lane line width.

[0144] When the gradient g(x,y) of a certain point is greater than the threshold T and the gradient value of the pixel g(x+L,y) at a horizontal distance L is also greater than the threshold T, then this is a possible inner support ring edge point, and the pixel values ​​between these two points are all set to 255, and the pixel values ​​of the pixels that do not meet the conditions are set to 0. At this point, the image enhancement is completed.

[0145] The constructed data set includes upward and side direction data sets. The upward data set can detect the orientation of the woven bags and whether the woven bags are neatly placed, and the side direction data set can detect the number of woven bags.

[0146] In the above dataset, images are divided into two categories: forward and backward. First, a number of forward and backward woven bag images are collected and labeled using labelimg software. Then, the labeled dataset is divided into a training set and a validation set in a one-to-one ratio. Then, the dataset is sent to the YOLOv5 network for pre-training and the pre-training weights are obtained. After that, the weights are used to send the dataset to the improved YOLOv5 network for training again. Finally, a script is written to count the accuracy of the validation set under different thresholds. The ones below the threshold are set as backward woven bags, otherwise they are forward woven bags, and the threshold under the highest map (accuracy) is tested.

[0147] In step S3, real-time image detection of woven bags is performed based on the preprocessed image, including the following steps:

[0148] a. Detect the direction of the woven bag;

[0149] b. Check whether the woven bags are neatly arranged;

[0150] c. Check the quantity of woven bags.

[0151] refer to Figure 8-11 , the figure shows some detection results, in which forward and backward represent categories, and the numbers behind represent credibility. If there are n target boxes in the image, and the accuracy of each target box is ξ_1, ξ_2, ..., ξ_n, then the threshold at the highest accuracy is ξ_1·ξ_2·...·ξ_n, for example Fig.10 The threshold in is 0.97×0.89×0.87=0.75, Fig.10 The test results are used as verification: Fig.10The Backward credibility is 0.97, which is converted into a Forward credibility of 0.03, which is lower than the set threshold of 0.75 and is recorded as Backward. The detection result is accurate.

[0152] In the step of detecting the orientation of woven bags in real-time image detection, the real-time collected image is input into the detect prediction script, and judged according to the threshold divided by the training script. If it is below the threshold, it is initially detected as an inverted woven bag, otherwise it is a forward woven bag.

[0153] Method to improve YOLOv5 network: First, use wiou loss function: Assume that the anchor box is B = [x, y, w, h] and the target box is B gt =[x gt ,y gt ,w gt ,h gt ], the calculation formula of wiou is as follows:

[0154]

[0155]

[0156]

[0157]

[0158] L WloUv1 =R WloU L loU ;

[0159]

[0160] Among them, W i , H i is the size of the overlap between the two frames, W g , H g is the size of the minimum bounding box, α is the weight coefficient, and v is the similarity calculation of the aspect ratio index of the two boxes. x, y, w, h are the horizontal and vertical coordinates of the center point of the anchor box, the width and height of the anchor box, respectively. gt ,y gt ,w gt ,h gt are the horizontal and vertical coordinates of the center point of the prediction box, the width and height of the anchor box, respectively. g , H g is the size of the minimum bounding box, and the superscript * indicates that W g , H g Separating it from the computation graph and turning it from a variable into a constant will prevent R WloU Produces gradients that hinder convergence.

[0161] The second step is to use channel attention (CA): channel attention focuses on the fact that different channel features have completely different weighted information for DCP. First, global average pooling is used to convert channel global spatial information into channel descriptors:

[0162]

[0163] Among them, X c Indicates the cth channel X c The value at position (i, j), H p As a global pooling function, the size of the feature map changes from C×H×W to C×1×1. In order to obtain different weights for different channels, the features are then passed through two convolutional layers and sigmoid and ReLu activation functions:

[0164] CA c =σ(Conv(δ(Conv(g c ))));

[0165] Among them, σ is the sigmoid function and δ is the ReLU function.

[0166] Finally, the input F is converted into c With channel CA c Multiply the corresponding elements by the weights

[0167]

[0168] In the step of detecting whether the woven bags are neatly arranged, after entering the detect prediction script, the number of target boxes is counted in real time. If the number of target boxes is greater than or equal to 2, the woven bags are not neatly arranged. Then the original image of the prediction result is collected and added to the training set to enrich the diversity of the training set.

[0169] Reference Fig.11 When the woven bags are not placed neatly, the angle between the two bags is detected to be α. At this time, the mechanical claw rotates the woven bags so that the two bags overlap.

[0170] Reference Fig.12 ,For the side direction woven bag dataset, the image segmentation method is adopted, including the following steps:

[0171] a. Enhanced image;

[0172] b. Threshold segmentation;

[0173] c. Canny edge detection;

[0174] d. Hough line detection.

[0175] In step b, threshold segmentation, the Otsu algorithm is used for threshold segmentation. The main principle of the Otsu algorithm is to divide the image into foreground and background based on the grayscale distribution characteristics of the whole image, and to make segmentation judgments by calculating the inter-class variance between the two. If the variance value is larger, the grayscale difference between the two is larger, and then the contrast is further improved for segmentation, so the Otsu algorithm is also called the maximum inter-class variance method. In the selection of grayscale threshold, the algorithm usually uses a cyclic iterative optimization mechanism to find the maximum inter-class variance. The algorithm solves the inter-class variance as follows:

[0176] (k is omitted here;) where:

[0177]

[0178]

[0179] Where: represents the inter-class variance; p1 is the probability of a pixel being classified into C1; p2 is the probability of being classified into C2, m1 is the average gray value of pixels assigned to C1; m2 is the average gray value of pixels assigned to C2; k is the gray level; pi is the probability of a pixel having a gray level of ; i is the gray level i;

[0180] Let {0,1,2,...,L-1} represent the L different gray levels in a digital image of size M×N pixels, n i represents the number of pixels with gray level i, and the probability that the gray level of the pixel is and Suppose we choose a threshold value T(k)=k, 0<k<L-1, and use it to threshold the input image into two categories C1 and C2, where C1 consists of all pixels in the image whose grayscale values ​​are in the range [0, k], and C2 consists of all pixels whose grayscale values ​​are in the range [k+1, L-1]. Then the probability of a pixel being classified into C1 is The probability of classification to C2 is Then the average gray value of the pixels assigned to C1 is The one assigned to C2 is The average gray value of pixels with gray levels from 0 to k is: The average gray value of the entire image is: The between-class variance is (k is omitted), the global variance is:

[0181] Then use Gaussian filtering to process the image to amplify the contrast of the image and enhance the edge contour information. The processing formula is as follows:

[0182]

[0183] Where: f s (x, y) is the result of convolution of Gaussian function and image; G(x, y) is a one-dimensional zero-mean Gaussian kernel function; f(x, y) is the gray value of the original pixel; σ is the standard deviation.

[0184] Calculate the grayscale gradient of the image and use the Sobel operator to convolve the horizontal and vertical operators with the image to calculate dx and dy, as shown below:

[0185] dx=f(x,y)·Sobel x (x, y);

[0186] dy=f(x,y)·Sobel y (x, y);

[0187] Then the image gradient amplitude and angle are obtained as follows:

[0188]

[0189] Where M[x, y] is the gradient value, θ M is the gradient direction, dx and dy represent the gradient of the image in the horizontal and vertical directions respectively.

[0190] Finally, a double threshold is used to screen the extracted edge features. The final edge pixels are determined by setting two different thresholds, high and low. If the edge gradient of the pixel neighborhood obtained in the third step of the algorithm is greater than the set high threshold, it is judged as an edge, and if it is less than the set low threshold, it is judged as a non-edge. Pixels between the thresholds will be judged based on whether they are connected to edge pixels. If they are connected, they are judged as edge feature pixels.

[0191] Hough transform, in the rectangular coordinate system, a straight line can be expressed by the following formula in the rectangular coordinate system: y = kx + b; where k and b are parameters, representing the slope and intercept of the straight line. The parameters of all straight lines passing through a certain point (x0, y0) satisfy y0 = -kx0 + y0. If the equation is rewritten as:

[0192] b=-kx0+y0, then the equation corresponds to a straight line in the parameter space kb.

[0193] Convert it into polar coordinate parameter space: r = xcosθ + ysinθ; r is the distance from the origin to the straight line in the polar coordinate system, and θ is the angle of the straight line.

[0194] Probabilistic Hough line detection is used. The principle is to adopt a probabilistic selection mechanism to randomly select individual points for calculation, which reduces the amount of calculation and calculation time, and outputs a vector set of detected lines. Each line is represented by a vector with four elements (x1, y1, x2, y2), where (x1, y1) represents the starting point of the line segment and (x2, y2) represents the end point of the line segment. If 3 to 4 sets of vectors can be detected, it means that the storage bag mechanism is full.

[0195] Conveyor belt conveying speed control: Considering the robot's grasping time, position adjustment time, and image detection time, when there are fewer target woven bags on the conveyor belt, there is generally no missed or wrong grasping of the target object. The conveyor belt speed can be appropriately increased to improve the overall grasping efficiency of the system. When the target woven bags are densely distributed on the conveyor belt, the real-time image detection and grasping system response time is short, and there will be cases of missing woven bags. At this time, the conveyor belt speed should be slowed down to give the robot enough reaction time to ensure the reliability of the system. In the process of different speed changes, it is also necessary to reduce sudden shocks and reduce damage to hardware. In view of this situation, when controlling the conveyor belt speed, select a suitable speed regulation strategy. Use stepless speed regulation to achieve conveyor belt speed control.

[0196] Stepless speed control method:

[0197]

[0198] Where: v i is the conveyor belt speed, v max is the maximum speed that the conveyor belt can reach, x min , x max The upper and lower limits for the robot to grasp the target object.

[0199] The stepless speed regulation has no sense of start-stop frustration, the transmission efficiency is much higher, and the overall impact is reduced, which also has a better protective effect on the system.

[0200] Taking into account that the number of woven bags in the bag storage device will continue to decrease as the woven bags are grabbed, the manipulator moves downward and the platform rises upward.

[0201] In summary, the present invention automatically arranges woven bags through the detection algorithm of the bag sorting machine and the camera, reduces labor and improves work efficiency.

[0202] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A bag sorting machine, comprising: Rack(101); Features: The bag unpacking machine also includes: A placement platform (102), arranged at the front end of the frame (101), for placing unsorted woven bags; A bag sorting platform (103) is provided at the middle end of the frame (101) and is used for sorting woven bags; A first manipulator (104) is disposed on the frame (101), between the placement platform (102) and the bag separation platform (103), and is used to sequentially move the woven bags on the placement platform (102) to the bag separation platform (103); A conveyor belt (105), arranged on the bag distributing platform (103), for transporting woven bags; A bag sorting platform (106) is arranged at the rear end of the frame (101); A second robot is arranged on the frame (101), located above the bag-separating platform (103) and the bag-sorting platform (106), and is used for transporting the woven bags on the bag-separating platform (103) to the bag-sorting platform (106) and then arranging the woven bags on the bag-sorting platform (106); Wherein, the bag sorting platform (106) is provided with a plurality of cameras for detecting the status of the woven bags; The bag sorting method of the bag sorting machine, The following steps are involved: S1. Collect image information of woven bags placed on the bag sorting platform through a camera; S2, preprocessing the collected images of the woven bag; S3, performing real-time image detection of woven bags based on the pre-processed images; S4. According to the graphic detection results, the second manipulator and the bag sorting platform move to sort the woven bags; The image detection includes detecting the orientation of the woven bags, detecting whether the woven bags are neatly placed, and detecting the number of woven bags; The detection of the orientation of the woven bags and the detection of whether the woven bags are neatly placed are performed by using an improved YOLOv5 network. The improvements of the YOLOv5 network include: Using the wiou loss function: Assume that the anchor box is B = [x, y, w, h] and the target box is B gt =[x gt ,y gt ,w gt ,h gt ], the calculation formula of wiou is as follows: L WloUv1 =R WloU L loU ; Among them, W i , H i is the size of the overlap between the two frames, W g , H g is the size of the minimum bounding box, α is the weight coefficient, v is the similarity calculation of the aspect ratio index of the two boxes; x, y, w, h are the horizontal and vertical coordinates of the center point of the anchor box, the width and height of the anchor box, respectively, gt ,y gt ,w gt ,h gt are the horizontal and vertical coordinates of the center point of the prediction box, the width and height of the anchor box, respectively. g , H g is the size of the minimum bounding box, and the superscript * indicates that W g , H g Separate it from the computation graph, turn it from a variable into a constant, and prevent R WloU Produces gradients that hinder convergence; Use channel attention: First, use global average pooling to convert channel global spatial information into channel descriptors: Among them, X c Indicates the cth channel X c The value at position (i, j), H p As a global pooling function, the size of the feature map changes from C×H×W to C×1×1. In order to obtain different weights for different channels, the features are then passed through two convolutional layers and sigmoid and ReLu activation functions: THAT c =σ(Conv(δ(Conv(g c )))): Where σ is the sigmoid function and δ is the ReLU function; Finally, the input F is converted into c With channel CA c Multiply the corresponding elements by the weights The training of the improved YOLOv5 network includes the following steps: a. Constructing a data set: Collect a number of forward and reverse woven bag images, annotate them, and divide the annotated data set into a data set with an upward direction and a side direction. The upward data set is divided into a training set and a validation set, with a ratio of one to one; b. Send the data set to the YOLOv5 network for pre-training to obtain pre-training weights; c. Use the obtained pre-trained weights to send the data set to the improved YOLOv5 network for training again; d. Write a script to count the accuracy of the validation set under different thresholds. Set the accuracy below the threshold as a backward woven bag, otherwise it is a forward woven bag; e. Test the threshold at the highest accuracy.

2. The bag sorting method of the bag sorting machine according to claim 1, characterized in that: In S2, the image preprocessing is to first perform image filtering on the original image, then grayscale the filtered image, and then perform image enhancement.

3. The bag sorting method of the bag sorting machine according to claim 2, characterized in that: The image filtering uses bilateral filtering, and its formula is: In the formula, g(x,y) is the filtered image, ω s (i,j) is the spatial domain weight, ω r (i,j) is the grayscale domain weight, ω P is the normalization parameter, I(i,j) is the noise image, and Ω is the neighborhood range at pixel (x, y); in, 4. The bag sorting method of the bag sorting machine according to claim 3, characterized in that: The image grayscale is to use the weighted average method to perform weighted average on the three components according to the importance of each component using the weight, and the calculation formula is: Gray=ω1R+ω2G+ω3B; Among them, R, G, B are the three primary colors, and ω1, ω2, ω3 are weight coefficients.

5. The bag sorting method of the bag sorting machine according to claim 3, characterized in that: The image enhancement process is to first use the Sobel operator to perform edge detection, and the method is to use two convolution kernels to perform horizontal and vertical edge detection respectively for any point P0 in the image: G x (P0)=(P3+2P5+P8)-(P1+2P4+P6); G y (P0)=(P6+2P7+P8)-(P1+2P2+P3); In the formula, G x (P0) and G y (P0) is the horizontal edge response intensity and the vertical edge response intensity at point P0, and its value indicates the possibility of it being the edge of the image; Then perform feature extraction, starting from the upper right corner of the image, traverse all pixels of the image from left to right and from top to bottom, and take the following operations on the pixels: obtain the gradient value of the edge detection image, if the gradient of a point is greater than the threshold T and the gradient value of the pixel point at a horizontal distance L from it is also greater than the threshold T, then this is the edge point of the inner support ring, set all the pixel values ​​between these two points to 255, and set the pixel values ​​of the pixels that do not meet the conditions to 0; at this point, the image enhancement is completed.

6. The bag sorting method of the bag sorting machine according to claim 5, characterized in that: The method of detecting the number of woven bags adopts a graphic segmentation method, firstly threshold segmentation is performed on the enhanced image, then Canny edge detection is performed, and finally Hough line detection is used to output a vector set of detected lines, and the vectors in the set represent the number of woven bags.

7. The bag sorting method of the bag sorting machine according to claim 6, characterized in that: The threshold segmentation is performed using the Otsu algorithm, and the solution is as follows: in: Where: represents the inter-class variance; p1 is the probability of a pixel being classified into C1; p2 is the probability of being classified into C2, m1 is the average gray value of pixels assigned to C1; m2 is the average gray value of pixels assigned to C2; k is the gray level; pi is the probability of a pixel having a gray level of ; i is the gray level i; Let {0, 1, 2, ..., L-1} represent L different gray levels in a digital image of size M×N pixels, n i represents the number of pixels with gray level i, and the probability that the gray level of the pixel is and Suppose we choose a threshold T(k)=k, 0<k<L-1, and use it to threshold the input image into two categories C1 and C2, where C1 consists of all pixels in the image whose grayscale values ​​are in the range [0, k], and C2 consists of all pixels whose grayscale values ​​are in the range [k+1, L-1]; then the probability of a pixel being classified into C1 is The probability of classification to C2 is The average gray value of the pixels assigned to C1 is The one assigned to C2 is The average gray value of pixels with gray levels from 0 to k is: The average gray value of the entire image is: The between-class variance is The global variance is: The canny edge detection is to first use Gaussian filtering to process the image after threshold segmentation to amplify the image contrast and enhance the edge contour information. The processing formula is as follows: Where: f s (,y) is the result of convolution of Gaussian function and image; G(x,y) is a one-dimensional zero-mean Gaussian kernel function; f(x,y) is the gray value of the original pixel; σ is the standard deviation; Then calculate the image grayscale gradient: use the Sobel operator horizontal and vertical operators to convolve with the image to calculate dx and dy, as follows: dx=f(x,y)·Sobel x (x,y); dy=f(x,y)·Sobel y (x, y)? Then the image gradient amplitude and angle are obtained as follows: Where M[x, y] is the gradient value, θ M is the gradient direction, dx and dy represent the gradient of the image in the horizontal and vertical directions respectively; Finally, the extracted edge features are screened using a double threshold. The final edge pixels are determined by setting two different thresholds, high and low. If the edge gradient of the pixel neighborhood obtained in the third step of the algorithm is greater than the set high threshold, it is determined to be an edge, and if it is less than the set low threshold, it is determined to be a non-edge. Pixels between the thresholds are judged based on whether they are connected to edge pixels. If they are connected, they are determined to be edge feature pixels. The Hough line detection is to output a set of detected straight line vectors based on edge feature pixel points, and each straight line in the vector set is represented by a vector (x1, y1, x2, y2) with four elements, where (x1, y1) represents the starting point of the line segment and (x2, y2) represents the end point of the line segment.

Citation Information

Patent Citations

  • Bag inserting and arranging all-in-one machine

    CN116119112A

  • Visual real-time detection method for accurate quantity and abnormal condition of densely and vertically arranged bagged food on assembly line

    CN116758017A

  • Tubular part direction adjuster based on visual detection

    CN205855295U