Conveyor belt cigarette carton stacking collapse detection method

Abstract

The invention discloses a conveyor belt cigarette carton stacking collapse detection method, and relates to the technical field of image recognition, and the method comprises the following steps: S1, collecting a cigarette carton end image sequence on a conveyor belt, carrying out the positioning detection of each cigarette carton end in the cigarette carton end image sequence, and obtaining the coordinates of four vertexes of each cigarette carton end; s2, based on the coordinates of the four vertexes of each cigarette end, tracking the cigarette end head pixel points of the cigarette end head image sequence, and obtaining the coordinates of the cigarette end head pixel points in subsequent frames; s3, according to the tracking result of the cigarette end head pixel points, the motion direction of the same cigarette end head pixel point between adjacent frames is calculated; and S4, carrying out statistical analysis on the motion direction of the pixel points of the cigarette end head by utilizing the motion direction histogram, and identifying the collapsed cigarette in the cigarette end head image sequence. According to the method, through detection and identification of collapse of the cigarette pile, the position of the collapsed cigarette carton is adjusted in time before packaging, unpacking and secondary packaging are avoided, and the sorting efficiency is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and more specifically to a method for detecting the collapse of cigarette stacks on a conveyor belt. Background Art

[0002] On the cigarette production line, different customers have different needs and need to sort the cigarettes in a customized way before packaging them. On the cigarette sorting line, the cigarettes are usually piled up and then placed on the conveyor belt for transmission. And the cigarettes cannot be placed randomly, but must be placed according to the pre-designed cigarette drop sequence diagram, such as Figure 3 As shown. After packing, the staff will refer to the falling order diagram to check whether the first and last cigarettes are placed correctly. However, since standard cigarettes and special-shaped cigarettes are often mixed together, there are cigarettes of different sizes in the cigarette pile, which makes it difficult to ensure neat stacking, causing it to collapse easily during transportation. Once the cigarette pile collapses, the position of the first and last cigarettes changes, and the staff must conduct more complex audits and corrections on the cigarette pile, which seriously affects the overall sorting efficiency.

[0003] In the prior art, the patent with publication number CN117274887A discloses a method for detecting the end of a cigarette and a method for identifying the specifications and quantity of cigarettes, which relates to the field of computer vision technology and solves the technical problem that the irregular packaging of cigarettes in the photographed photos will lead to missed detection. The method includes the following steps: S1, obtaining an Euler angle prediction model and an end detection model through training; S2, inputting the image of the cigarette to be detected into the Euler prediction model to obtain the Euler angle of the end surface of the cigarette in the cigarette image; wherein the cigarette image includes multiple boxes of cigarettes; S3, correcting the cigarette image according to the Euler angle to obtain a corrected cigarette image; S4, inputting the corrected cigarette image into the end detection model to obtain the vertex coordinates of all cigarette ends in the cigarette image. The present invention can effectively avoid the problem of missed detection by correcting the end surface of the cigarette. However, it cannot detect the collapse of cigarette piles.

[0004] In summary, the current research on smoke pile collapse detection is still quite lacking, and there is no technical means to remedy smoke pile collapse when it occurs. Summary of the invention

[0005] In order to overcome the defects in the above-mentioned prior art, the present invention discloses a method for detecting the collapse of cigarette cartons on a conveyor belt. The present invention detects and identifies the collapse of cigarette cartons by sequentially performing cigarette carton end detection, cigarette carton end tracking, cigarette carton end motion judgment, and cigarette carton stack collapse detection, and timely adjusts the position of collapsed cigarette cartons before packaging, avoiding unpacking and secondary packaging, and effectively improving sorting efficiency.

[0006] In order to achieve the above objectives, the technical solution adopted by the present invention is:

[0007] A method for detecting the collapse of cigarette stacks on a conveyor belt comprises the following steps:

[0008] 1. Cigarette end detection

[0009] S1, collecting a sequence of cigarette end images on a conveyor belt, and performing positioning detection on each cigarette end in the sequence of cigarette end images to obtain the coordinates of four vertices of each cigarette end;

[0010] Preferably, in step S1, a cigarette rod end detection model is established, a cigarette rod end image sequence is input into the cigarette rod end detection model, and the cigarette rod end detection model outputs the coordinates of the four vertices of each cigarette rod end in each cigarette rod end image.

[0011] Preferably, in step S1, the cigarette end detection model detects the cigarette end based on a deep learning target detection algorithm, and the target detection algorithm includes YOLOv8 and DINO target detection algorithms.

[0012] Preferably, the model training of the cigarette end detection model in step S1 includes:

[0013] Collect 100,000 images of cigarette carton ends on the sorting line, obtain the coordinates of the four vertices of the cigarette carton ends in the images through manual annotation, and obtain the end labels;

[0014] The cigarette end detection model is trained based on cigarette end images and end label data.

[0015] Preferably, in step S1, the four vertices of the cigarette rod end are A, B, C, and D, and the cigarette rod end is represented by a quadrilateral ABCD, (x i ,y i ), i∈{A, B, C, D}, represents the coordinates of the four vertices, and the left, right, top, and bottom sides of the rectangular frame circumscribing the cigarette end are expressed as:

[0016]

[0017] Among them, min{} means taking the minimum value, max{} means taking the maximum value, x l 、x r ,y t ,y b They are the left, right, top and bottom sides of the rectangular frame circumscribing the end of the cigarette, respectively. A 、x B 、x C 、x D are the horizontal coordinates of the four vertices A, B, C, and D at the end of the cigarette, and y A ,y B ,yC ,y D They are the vertical coordinates of the four vertices A, B, C, and D at the end of the cigarette strip.

[0018] 2. Cigarette end tracking

[0019] S2, based on the coordinates of the four vertices of each cigarette end, tracking the cigarette end pixel points in the cigarette end image sequence, and obtaining the coordinates of the cigarette end pixel points in subsequent frames;

[0020] Preferably, in step S2, the tracked pixel points are obtained by uniform sampling, and the coordinates of the uniform sampling points are:

[0021]

[0022] in:

[0023]

[0024] Among them, x ij is the horizontal coordinate of the uniform sampling point, y ij is the ordinate of the uniform sampling point, d is the grid spacing, x l 、x r ,y t ,y b They are the left, right, top and bottom sides of the rectangular frame circumscribing the end of the cigarette strip, i is the horizontal serial number of the grid point, and j is the vertical serial number of the grid point.

[0025] Preferably, in step S2, only the tracked pixel points of the cigarette end are retained that are inside the quadrilateral ABCD; and the pointPolygonTest function of the OpenCV library is used to determine whether the coordinates of the tracked pixel points of the cigarette end are inside the quadrilateral ABCD.

[0026] Preferably, in step S2, starting from the pixel point of the cigarette end selected in the first frame, the pixel points are tracked according to the similarity of the same pixel points between adjacent frames. The tracking algorithm includes the Co-Tracker tracking algorithm. The Co-Tracker tracking algorithm combines the ideas of optical flow and tracking, and is based on the Transformer network. The correlation of different time points is modeled through the attention layer, and is applied to the cigarette end image sequence in a sliding window manner.

[0027] 3. Determine the direction of movement of the cigarette end

[0028] S3, calculating the movement direction of the same pixel point at the end of the cigarette between adjacent frames according to the tracking result of the pixel point at the end of the cigarette;

[0029] Preferably, step S3 includes: assuming that the coordinates of the pixel point k at the end of the cigarette strip in the t-1th frame are The coordinates of pixel k at the end of the cigarette strip in frame t are: The running direction of pixel k at the end of the cigarette for:

[0030]

[0031] Angle corresponding to the running direction for:

[0032]

[0033] Among them, atan2 is the inverse tangent function, PI is the pi, and % is the modulo operation.

[0034] Preferably, step S3 further includes: Quantized into 36 directions:

[0035]

[0036] is the moving direction of the pixel, and the moving directions of all tracked pixels represent the moving directions of the end where the pixel is located.

[0037] 4. Detection of cigarette stack collapse

[0038] S4. Statistically analyze the movement directions of the pixels at the ends of cigarette strips using a movement direction histogram to identify collapsed cigarette strips in a sequence of cigarette strip end images.

[0039] Preferably, step S4 comprises:

[0040] For the moving direction of the pixel in the t-th frame of the cigarette end image N is the number of pixels tracked, and the moving direction histogram H is statistically i ,i∈{0,1,…,35}, where there are 36 elements in the motion direction histogram, and the value of each element records the number of pixels in the corresponding direction;

[0041] When the group of elements H in the statistical motion direction histogram represents downward motion j ,j∈{19,20,…,35} satisfies max j∈{19,20,…,35} H j >T, it is determined that the cigarette end image of the tth frame has collapsed, where T is the pixel number threshold.

[0042] Beneficial effects of the present invention:

[0043] 1. The present invention detects and identifies the collapse of cigarette piles by sequentially performing cigarette carton end detection, cigarette carton end tracking, cigarette carton end movement judgment, and cigarette carton stack collapse detection, and timely adjusts the position of collapsed cigarette cartons before packaging to avoid unpacking and secondary packaging, thereby effectively improving sorting efficiency.

[0044] 2. The present invention is based on the continuous tracking of the end pixel points in the end image sequence, calculates the movement direction and angle of each pixel point between consecutive frames, uses the movement direction histogram to quantify the distribution of the movement direction, and identifies the collapse event of cigarette stacking by judging the numerical value of downward movement in the movement direction histogram.

[0045] 3. The present invention adopts the Transformer-based Co-Tracker tracking algorithm, combined with optical flow technology and time correlation modeling, which enhances the accuracy and robustness of tracking. It can handle complex dynamic scenes and is suitable for the challenges of changes in sorting line speed and different types of cigarettes. Traditional technologies may be limited to static image analysis or simple dynamic tracking, and do not use complex models of deep learning to track long sequences of pixels.

[0046] 4. The present invention utilizes statistical analysis of the movement direction of cigarette rod ends to identify collapse events, and uses a movement direction histogram to quantify the distribution of movement directions, especially to monitor downward movement, thereby achieving early detection of signs before collapse. The prior art lacks effective methods for detecting and preventing collapse of cigarette rod stacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The present invention is a process flow for processing a cigarette end image sequence;

[0048] Figure 2 The present invention is a visualization of the cigarette stack collapse detection process, wherein:

[0049] a) A sequence of collected cigarette end images;

[0050] b) The cigarette end detection result of the first frame image, where the detected end is marked with a quadrilateral;

[0051] c) The tracking result of the cigarette tip in the image of frame t-1, where the blue line is the movement trajectory of the tip pixel;

[0052] d) The tracking result of the cigarette tip in the t-th frame image. The blue line is the movement trajectory of the tip pixel.

[0053] e) The histogram of the movement direction of the collapsed cigarette end in the t-th frame image shows that the number of pixels moving in the downward direction is significantly greater than that in other directions;

[0054] f) Detection result of smoke stack collapse in the t-th frame image. The collapsed end is marked with a rectangular frame.

[0055] Figure 3 This is a diagram of the cigarette falling sequence of a certain tobacco company. DETAILED DESCRIPTION

[0056] The concept, specific structure and technical effects of the present invention will be clearly and completely described below in conjunction with the embodiments and drawings to fully understand the purpose, characteristics and effects of the present invention.

[0057] A method for detecting the collapse of cigarette stacks on a conveyor belt, such as Figure 1 and Figure 2 As shown, the following steps are included:

[0058] S1, collecting a sequence of cigarette end images on a conveyor belt, and performing positioning detection on each cigarette end in the sequence of cigarette end images to obtain the coordinates of four vertices of each cigarette end;

[0059] S2, based on the coordinates of the four vertices of each cigarette end, tracking the cigarette end pixel points in the cigarette end image sequence, and obtaining the coordinates of the cigarette end pixel points in subsequent frames;

[0060] S3, calculating the movement direction of the same pixel point at the end of the cigarette between adjacent frames according to the tracking result of the pixel point at the end of the cigarette;

[0061] S4. Statistically analyze the movement directions of the pixels at the ends of cigarette strips using a movement direction histogram to identify collapsed cigarette strips in a sequence of cigarette strip end images.

[0062] The present invention is a method for detecting the collapse of cigarette carton stacks based on machine vision and deep learning. The present invention mainly includes four major steps: a cigarette carton end detection step, a cigarette carton end tracking step, a cigarette carton end motion judgment step, and a cigarette carton stack collapse detection step. The cigarette carton end detection step locates each end and obtains the coordinates of the four vertices of the end. The cigarette carton end tracking step inputs the entire image sequence, tracks the pixels of the first frame, and obtains the coordinates of the pixels of the first frame in subsequent frames. The cigarette carton end motion direction judgment step calculates the motion direction of the pixels between adjacent frames based on the tracking results of the pixels. The cigarette carton stack collapse detection step performs statistical analysis on the motion direction of the pixels to identify collapsed cigarette carton. The processing flow of the invention is as follows: Figure 1 and Figure 2 The cigarette end image is processed as follows:

[0063] 1. The cigarette end detection step processes the first frame of the end image and outputs the coordinates of the four vertices of each end. The target detection algorithm based on deep learning is used to detect the cigarette ends. First, 100,000 images of the ends on the sorting line are taken, and the coordinates of the four vertices of the ends are obtained by manual annotation to obtain the end labels; then the cigarette end detection model is trained based on the end images and label data. The cigarette end detection can use but is not limited to target detection algorithms such as YOLOv8 and DINO.

[0064] 2. The cigarette end tracking step processes the end image sequence and outputs the running trajectory of the end pixel points. First, the pixels to be tracked are selected from the first frame, and the end tracking is achieved by tracking these pixels. The tracked pixels are obtained by uniform sampling. The cigarette end detection step outputs the four vertices of the end, which are A, B, C, and D. The end can be represented as a quadrilateral ABCD. (x i ,y i ), i∈{A,B,C,D}, represents the coordinates of these four points. The left, right, top and bottom sides of the external rectangular frame of the terminal are expressed as:

[0065]

[0066] Among them, min{} means taking the minimum value, and max{} means taking the maximum value. The grid spacing is set to d. The coordinates of the uniform sampling points are:

[0067]

[0068] in:

[0069]

[0070] In order to ensure that background pixels are not tracked, only the points inside the quadrilateral ABCD are retained. To determine whether the coordinates of a point are inside the quadrilateral ABCD, the pointPolygonTest function of the OpenCV library can be used. According to the actual experimental results, the tracking effect is good when d is 15 pixels. Starting from the pixel points selected in the first frame, the pixel points are tracked according to the similarity of the same pixel points between adjacent frames. The tracking algorithm can be selected but is not limited to any point tracking algorithm such as Co-Tracker. Co-Tracker combines the ideas from optical flow and tracking. Based on the Transformer network, it models the correlation of different time points through a dedicated attention layer and is applied to long videos in a sliding window manner. It is a flexible and powerful tracking algorithm.

[0071] 3. Step of judging the movement direction of the cigarette end Based on the tracking results of the end pixel points, the movement direction of the same pixel point between adjacent frames is calculated. Assume that the coordinates of the pixel point k in the t-1th frame are The coordinates of pixel k in the tth frame are Then the running direction of the pixel is:

[0072]

[0073] The corresponding angles are:

[0074]

[0075] Among them, atan2 is the inverse tangent function, PI is the pi, and % is the modulo operation. In order to facilitate subsequent processing, Quantized into 36 directions (360 degrees is divided into 36 directions, so one direction every 10 degrees, the degree / 10 and rounded down is the direction number of the angle):

[0076]

[0077] The movement direction of all tracked pixels represents the movement direction of the end where the pixel is located.

[0078] 4. The step of detecting the collapse of cigarette stacks is to statistically analyze the movement direction of the end pixel points. N is the number of pixels tracked, and the moving direction histogram H is statistically i ,i∈{0,1,…,35}. The size of the motion direction histogram is 36, where the value of each element records the number of pixels in the corresponding direction. When the cigarette stack collapses, the movement direction of the pixel at the collapsed end is downward. At this time, a group of elements H in the motion direction histogram representing downward movement j ,j∈{19,20,…,35}, will have a relatively large value. When the following conditions are met, it is determined that the tth frame has collapsed:

[0079] max j∈{19,20,…,35} H j >T

[0080] Where T is the pixel number threshold. According to the actual test results, T is set to 50 to obtain the best detection accuracy.

[0081] When photographing cigarette stacks, the present invention has no special requirements on the conveyor belt speed, but needs to keep the illumination in the conveyor belt area relatively uniform.

[0082] The present invention tested 1000 videos collected on a cigarette sorting line, each of which contained a complete moving process of a pile of cigarettes on a conveyor belt. The recognition rate of the present invention for the end collapse detection of the test data was 100%.

[0083] The above is a specific description of the implementation mode of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention, and these equivalents or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for detecting the collapse of cigarette stacks on a conveyor belt, characterized in that: The following steps are involved: S1, collecting an image sequence of cigarette ends on a conveyor belt, and performing positioning detection on each cigarette end in the image sequence of cigarette ends to obtain the coordinates of four vertices of each cigarette end; S2, based on the coordinates of the four vertices of each cigarette end, tracking the cigarette end pixel points in the cigarette end image sequence, and obtaining the coordinates of the cigarette end pixel points in subsequent frames; S3, calculating the movement direction of the same pixel point at the end of the cigarette between adjacent frames according to the tracking result of the pixel point at the end of the cigarette; S4. Statistically analyze the movement directions of the pixels at the ends of cigarette strips using a movement direction histogram to identify collapsed cigarette strips in a sequence of cigarette strip end images.

2. A method for detecting the collapse of cigarette stacks on a conveyor belt as claimed in claim 1, characterized in that: In step S1, a cigarette end detection model is established, and a cigarette end image sequence is input into the cigarette end detection model, and the cigarette end detection model outputs the coordinates of the four vertices of each cigarette end in each cigarette end image; In step S1, the cigarette end detection model detects the cigarette end based on a deep learning target detection algorithm, and the target detection algorithm includes YOLOv8 and DINO target detection algorithms.

3. A method for detecting the collapse of cigarette stacks on a conveyor belt as claimed in claim 2, characterized in that: The model training of the cigarette end detection model in step S1 includes: Collect 100,000 images of cigarette carton ends on the sorting line, obtain the coordinates of the four vertices of the cigarette carton ends in the images through manual annotation, and obtain the end labels; The cigarette end detection model is trained based on cigarette end images and end label data.

4. A method for detecting the collapse of cigarette stacks on a conveyor belt as claimed in claim 1, characterized in that: In step S1, the four vertices of the cigarette end are A, B, C, and D, and the cigarette end is represented by a quadrilateral ABCD. i ,y i ), i∈{A, B, C, D}, represents the coordinates of the four vertices, and the left, right, top, and bottom sides of the rectangular frame circumscribing the cigarette end are expressed as: Among them, min{} means taking the minimum value, max{} means taking the maximum value, x l 、x r ,y t ,y b They are the left, right, top and bottom sides of the rectangular frame circumscribing the end of the cigarette, respectively. A 、x B 、x C 、x D are the horizontal coordinates of the four vertices A, B, C, and D at the end of the cigarette, and y A ,y B ,y C ,y D They are the vertical coordinates of the four vertices A, B, C, and D at the end of the cigarette strip.

5. A method for detecting the collapse of cigarette stacks on a conveyor belt as claimed in claim 4, characterized in that: In step S2, the tracked pixel points are obtained by uniform sampling, and the coordinates of the uniform sampling points are: in: Among them, x ij is the horizontal coordinate of the uniform sampling point, y ij is the ordinate of the uniform sampling point, d is the grid spacing, x l 、x r ,y t ,y b They are the left, right, top and bottom sides of the rectangular frame circumscribing the end of the cigarette strip, i is the horizontal serial number of the grid point, and j is the vertical serial number of the grid point.

6. A method for detecting the collapse of cigarette stacks on a conveyor belt as claimed in claim 4, characterized in that: In step S2, only the tracked pixel points of the cigarette end are retained that are inside the quadrilateral ABCD; through the pointPolygonTest function of the OpenCV library, it is determined whether the coordinates of the tracked pixel points of the cigarette end are inside the quadrilateral ABCD.

7. A method for detecting the collapse of cigarette stacks on a conveyor belt as claimed in claim 1, characterized in that: In step S2, starting from the pixel point of the cigarette end selected in the first frame, the pixel points are tracked according to the similarity of the same pixel points between adjacent frames. The tracking algorithm includes the Co-Tracker tracking algorithm. The Co-Tracker tracking algorithm combines the ideas of optical flow and tracking, and is based on the Transformer network. The correlation of different time points is modeled through the attention layer, and is applied to the cigarette end image sequence in a sliding window manner.

8. A method for detecting the collapse of cigarette stacks on a conveyor belt as claimed in claim 1, characterized in that: Step S3 includes: assuming that the coordinates of pixel point k at the end of the cigarette strip in frame t-1 are The coordinates of pixel k at the end of the cigarette strip in frame t are: The running direction of pixel k at the end of the cigarette for: Angle corresponding to the running direction for: Among them, atan2 is the inverse tangent function, PI is the pi, and % is the modulo operation.

9. A method for detecting the collapse of cigarette stacks on a conveyor belt as claimed in claim 8, characterized in that: Step S3 also includes: Quantized into 36 directions: is the moving direction of the pixel, and the moving directions of all tracked pixels represent the moving directions of the end where the pixel is located.

10. A method for detecting the collapse of cigarette stacks on a conveyor belt as claimed in claim 1, characterized in that: The S4 step includes: For the moving direction of the pixel in the t-th frame of the cigarette end image N is the number of pixels tracked, and the moving direction histogram H is statistically i ,i∈{0,1,…,35}, where there are 36 elements in the motion direction histogram, and the value of each element records the number of pixels in the corresponding direction; When the group of elements H in the statistical motion direction histogram represents downward motion j ,j∈{19,20,…,35} satisfies max j∈{19,20,…,35} H j >T, it is determined that the cigarette end image of the tth frame has collapsed, where T is the pixel number threshold.

Images