A foreign body detection method for cigarette package slice sections based on Transformer

Abstract

The present invention discloses a method for detecting foreign objects in a cigarette pack slice section based on Transformer, and constructs a network model for detecting foreign objects in a cigarette pack slice section based on Transformer. The network model includes four modules: a local information enhancement module, a feature extraction module, a candidate region generation module, and a target detection module. The local information enhancement module enhances the global modeling capability of the network, the feature extraction module proposes a new Swin Transformer D-Block structure, and the candidate region generation module generates a feature map of a set size, which is used by the target detection module to identify and detect the type and location of foreign objects. The present invention combines the advantages of local information enhancement, Swin Transformer D-Block and other methods to improve the detection capability and efficiency of the detection network for disguised target foreign objects and small target foreign objects, and improve the detection accuracy of the algorithm for disguised target foreign objects and small target foreign objects.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and intelligent manufacturing, and in particular relates to a method for detecting foreign matter in a cigarette package section based on a Transformer. Background Art

[0002] Common foreign objects on the cut surface of cigarette packs include mold spots and impurities, which affect the quality and taste of cigarettes. In order to ensure the quality of finished cigarettes, it is necessary to strictly check the foreign objects on the cut surface of cigarette packs during the production process to monitor the quality of the cut surface of cigarette packs. In the prior art, there are two methods for detecting foreign objects on the cut surface of cigarette packs during the production process: manual inspection and machine vision inspection. The manual inspection method uses the naked eye to check the foreign objects on the cut surface of cigarette packs one by one. This method has the disadvantages of being time-consuming, labor-intensive, easy to fatigue, and poor repeatability. Although the machine vision inspection method solves the shortcomings of poor repeatability and easy fatigue of the manual visual inspection method, it has the disadvantages of long development cycle and high cost.

[0003] In order to solve the shortcomings of poor repeatability and fatigue of manual detection methods, a Chinese invention patent with publication number "CN107330882A" discloses an online detection method for foreign matter after tobacco shreds based on machine vision. The specific scheme is as follows: obtain a color image of the tobacco to be detected; filter and contrast enhance the color image; extract the color space statistical features of the tobacco to be detected from the processed color image through a feature extraction algorithm; use multiple preset color space classifiers for joint detection based on the color space statistical features; perform statistical judgment on the detection results of each color space classifier, and delete the connected domains whose area is less than the preset area threshold; use an OR operation to fuse the above judgment results to obtain a pixel-level detection result; use multiple preset foreign matter identifiers in parallel to identify the above pixel-level detection results, and exclude image areas that do not meet the foreign matter characteristics; fuse the above image data to obtain the final detection result. However, the problem with this prior art is that, in the detection of foreign objects in cigarette pack slice sections that contain a large number of small foreign objects and disguised foreign objects, if the threshold detection method is used, there will be a problem of insufficient detection accuracy, which will lead to obvious missed detections and false detections.

[0004] With the rapid development of Transformer and its outstanding performance in the field of target detection, it has become the first choice for solving the traditional machine vision algorithm problems. The Chinese invention patent with the publication number "CN114241431A" proposes a small target detection method based on an improved Transformer, and its specific scheme is as follows: the preprocessed collected image is loaded into the trained SR-DETR target detection model, and the target detection model processes the image and outputs the final target detection result image; the SR-DETR target detection model includes: feature extraction module, Transformer model, classification prediction module, super-resolution module, image fusion module. However, when the target detection model is applied to the detection of foreign objects in the section of cigarette packages with a large number of small foreign objects and disguised foreign objects, the global modeling ability of the model is low, resulting in insufficient detection ability for small foreign objects and disguised foreign objects; at the same time, the high algorithm complexity of the model leads to low detection efficiency. Summary of the invention

[0005] The present invention provides a method for detecting foreign objects in cigarette packet slice sections based on Transformer, aiming to solve the problems of missed detection and false detection caused by insufficient detection accuracy of target detection models and low detection efficiency caused by high algorithm complexity in the detection of a large number of small target foreign objects and disguised target foreign objects in cigarette packet slice sections.

[0006] In order to solve the above technical problems, the present invention provides a method for detecting foreign matter in a cigarette packet section based on Transformer, comprising the following steps:

[0007] S1: Construct a network model for foreign body detection on cigarette package slice sections based on Transformer. The network model includes four modules: local information enhancement module, feature extraction module, candidate region generation module and target detection module. The training process of the network model includes the following steps:

[0008] S1-1: Establish a cigarette package cross-section foreign body detection dataset as input image;

[0009] S1-2: The local information enhancement module performs image information enhancement on the input image and outputs a feature map D;

[0010] S1-3: The feature extraction module extracts features from the feature map D and outputs feature maps F, G, H, and I;

[0011] S1-4: The candidate region generation module fuses the feature maps F, G, H, and I to generate a feature map K of a set size;

[0012] S1-5: The target detection module identifies and detects the type and position of the foreign object in the feature map K;

[0013] S1-6: Iteratively execute steps S1-2 to S1-5 using the input image until the set number of training times is reached, and save the model of the last training;

[0014] S2: Obtaining a cross-sectional image of a cigarette package during the production process as an image to be detected;

[0015] S3: Load the image to be detected in step S2 into the model saved in step S1-6 to obtain the target detection result in the production process.

[0016] Preferably, the steps of establishing a cigarette packet slice cross-section foreign body detection data set in step S1-1 are as follows: first, filter out cigarette packet slice cross-section images containing foreign bodies, count the types and quantities of foreign bodies, so that the number of each type of foreign bodies is roughly the same; then, mark the foreign bodies in the image; and finally, convert the marked data into a data set in a required format;

[0017] Preferably, the local information enhancement module consists of three branches, branch a consists of a 3×3 convolution operation; branch b consists of a 5×5 convolution operation and a 3×3 deconvolution operation; branch c consists of a 7×7 convolution operation and a 5×5 deconvolution operation.

[0018] Preferably, the feature extraction module is divided into five parts: Patch Partition, Stage 1, Stage 2, Stage 3, and Stage 4. Stage 1 consists of Linear Embedding and Swin Transformer D-Block; Stage 2, Stage 3, and Stage 4 all consist of Patch Merging and Swin Transformer D-Block.

[0019] Preferably, the Swin Transformer D-Block simplifies the dense connection structure and combines it with the Swin Transformer Block, and adds the feature map input to the Swin Transformer D-Block with the feature maps calculated by MLP1 and SW-MSA, MLP2, so that the feature map input to the Swin Transformer D-Block can be reused, and the feature map calculated by W-MSA and MLP1 is not added to the subsequent steps.

[0020] Preferably, the steps of generating candidate regions by the candidate region generation module are as follows: first, FPN fuses the shallow detail information and high-level semantic information extracted by the feature extraction module, and outputs a feature map J containing more information; then, RPN further processes the generated feature map J to generate a series of candidate regions with different lengths and widths; finally, Rol Align adjusts the candidate regions of different lengths and widths to become a feature map Feature Map with a set length and width, which is output as a feature map K.

[0021] Preferably, the task of the target detection module is to identify and detect the feature map generated by the candidate region generation module, and mainly complete two tasks: classifying the targets in the feature map K and obtaining the precise position of the target through bounding box regression F by Box head.

[0022] Compared with the prior art, the present invention has the following technical effects:

[0023] 1. The present invention proposes a new local information enhancement model. After local information enhancement, the size of the image does not change, but the information can be made richer, thereby improving the global modeling ability of the network and the detection ability of the network for disguised target foreign objects and small target foreign objects, which can effectively improve the problems of missed detection and false detection caused by insufficient detection accuracy of the target detection model.

[0024] 2. A new Swin Transformer D-Block structure is proposed in the feature extraction module of the present invention. Based on the Swin Transformer Block structure in the prior art, the Swin Transformer D-Block structure introduces dense connections, enhances feature reuse, avoids overfitting on the cigarette package cross-section foreign body data set, and improves the detection efficiency of the target detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a network model framework diagram of a method for detecting foreign matter in a cigarette package section based on Transformer according to the present invention;

[0026] Figure 2 It is a schematic diagram of a local information enhancement module of a method for detecting foreign matter in a cigarette package cross section based on Transformer according to the present invention;

[0027] Figure 3 It is a schematic diagram of the SwinTransformer D-Block structure of the Transformer-based cigarette package cross-section foreign body detection method described in the present invention;

[0028] Figure 4 This is a network model training flow chart of a method for detecting foreign matter in a cigarette package section based on Transformer described in the present invention.

[0029] In the figure: 1. Local information enhancement module; 2. Feature extraction module; 3. Candidate region generation module; 4. Target detection module. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in combination with specific embodiments of the present application and with reference to the accompanying drawings.

[0031] In order to efficiently complete the task of detecting foreign objects in a cigarette pack slice section, the present invention proposes a method for detecting foreign objects in a cigarette pack slice section based on a Transformer, by detecting foreign objects in a cigarette pack slice section, and then determining whether there are foreign objects in the measured cigarette pack slice section and the type and position of the foreign objects; in order to solve the problem of missed detection and high false detection rate of small target foreign objects and disguised target foreign objects, the present invention provides a local information enhancement module 1; a SwinTransformer D-Block (ShiftedWindowsTransformer D-Block, a simplified densely connected Transformer block based on a shifted window) structure using simplified dense connections; and a training flowchart for identifying and locating foreign objects in a cigarette pack slice section based on an image.

[0032] First, a network model for foreign body detection on cigarette package slices based on Transformer is constructed. Figure 1 As shown in FIG. 1 , the network model includes four modules: a local information enhancement module 1, a feature extraction module 2, a candidate region generation module 3, and a target detection module 4. Among them, the local information enhancement module 1 is used to enhance the local information of the input foreign body image; the feature extraction module 2 is used to extract features from the enhanced image and generate a feature map containing rich semantic information; the candidate region generation module 3 processes the feature map generated by the feature extraction module 2 and generates a series of fixed-size feature maps that may contain foreign bodies for the target detection module 4; the target detection module 4 realizes the recognition and positioning of foreign bodies based on the generated feature map.

[0033] The local information enhancement module 1 uses convolution kernels of different sizes to gather local information of different sizes for the image and enhance the global modeling ability of the network. Figure 2As shown, the local information enhancement module 1 of the present invention is composed of three branches: branch a is composed of 3×3 convolution operations; branch b is composed of 5×5 convolution operations and 3×3 deconvolution operations; branch c is composed of 7×7 convolution operations and 5×5 deconvolution operations. The feature maps processed by each branch are superimposed along the channel direction to enhance the local information of the image. The feature extraction module 2 extracts semantic information from the feature map generated by the local information enhancement module 1. The feature extraction module 2 can be divided into five parts: Patch Partition (block partitioning, used to divide the input feature map into a set of non-overlapping patches and convert the smallest unit of the image from pixels to patches), Stage 1, Stage 2, Stage 3, and Stage 4. Stage 1 consists of Linear Embedding (using projection to map features to any dimension) and Swin Transformer D-Block; Stage 2, Stage 3, and Stage 4 all consist of Patch Merging (used to connect a set number of adjacent patches in Stage to reduce the image resolution) and Swin Transformer D-Block. The task of the candidate region generation module 3 is to further process the multi-scale feature map generated by the feature extraction module 2 and generate a feature map suitable for the target detection module 4.

[0034] The task of the target detection module 4 is to identify and detect the category and position of foreign objects in the feature map generated by the candidate region generation module 3.

[0035] The construction of a network model for foreign body detection on cigarette package slice sections based on Transformer includes the following steps S1-1 to S1-6:

[0036] S1-1: Establish a cigarette package cross-section foreign body detection dataset Src as the input image;

[0037] The specific steps to establish the data set Src are as follows: first, filter out the cross-section images of cigarette packages containing foreign objects, count the types and quantities of foreign objects, and make the number of each type of foreign objects roughly the same; then mark the foreign objects in the image, and the foreign objects must be accurately classified and of appropriate size; finally, convert the marked data into a data set in the required format.

[0038] S1-2: The local information enhancement module 1 enhances the local information of the input image and outputs a feature map D;

[0039] After passing through the local information enhancement module 1, the image size does not change, but the information can be enriched. Figure 2As shown in the figure, the processing steps of the local information enhancement module 1 are as follows: First, when the input image is transmitted to the local information enhancement module 1, the input data is transmitted to branches a, b, and c respectively, and branches a, b, and c process the image at the same time. Then, branch a uses 3×3 convolution to process the image and generates feature map A; branch b uses 5×5 convolution to process the image, and uses 3×3 deconvolution to adjust the feature map to the same size as feature map A to generate feature map B; branch c uses 7×7 convolution to process the image, and uses 5×5 deconvolution to adjust the feature map to the same size as feature map A to generate feature map C. Finally, feature maps A, B, and C are superimposed along the channel direction to generate the final feature map D of the local information enhancement module 1.

[0040] Compared with the existing technology, after local information enhancement, the size of the image does not change, but the information can be made richer, thereby improving the global modeling ability of the network, improving the network's detection ability for disguised target foreign objects and small target foreign objects, and effectively improving the problem of missed detection and false detection caused by insufficient detection accuracy of the target detection model.

[0041] S1-3: Use feature extraction module 2 to extract features and output feature maps F, G, H, I;

[0042] After passing through the feature extraction module 2, the data in the cigarette package cross-section foreign body detection dataset becomes a feature map with rich semantic information. Figure 1 As shown in FIG. 1 , the specific processing steps of the feature extraction module 2 are as follows: first, PatchPartition is used to block the feature map D generated by the local information enhancement module, the feature map is divided into a series of non-overlapping Patches, and flattened along the channel direction to obtain a feature map E; then, the feature map E is input into Stage 1, and in Stage 1, the channel data of the feature map E is linearly changed through the Linear Embedding layer. The feature map processed by Linear Embedding is input into Swin Transformer D-Block for feature extraction and feature map F is generated. Then, feature map F is input into Stage2 for feature extraction. In Stage2, the length and width of feature map F are halved and the depth is doubled through Patch Merging. The feature map processed by Patch Merging is input into Swin Transformer D-Block for feature extraction and feature map G is generated. Finally, feature map G is input into Stage3 for feature extraction to obtain feature map H. Feature map H is input into Stage4 for feature extraction to obtain feature map I. The processing flow of feature maps in Stage3 and Stage4 is exactly the same as that in Stage2.

[0043] Furthermore, the structure of Swin Transformer D-Block is as follows Figure 3 As shown, the structure of Swin Transformer D-Block is as follows Figure 3 As shown in the figure. Swin Transformer D-Block simplifies the dense connection structure and combines it with Swin Transformer Block (Shifted Window Transformer Block, Transformer block based on shifted window). On the basis of Swin Transformer Block, it adds the feature map of the input Swin Transformer D-Block with the feature map calculated by LN (Layer Normalization), MLP1 (Multi-Layer Perception), SW-MSA (Shifted Window based Self-Attention, self-attention mechanism based on shifted window), and MLP2, so that the input feature information can be reused. The feature map generated by W-MSA (Window based Self-Attention, self-attention mechanism based on window) and MLP1 is not added to the subsequent steps to avoid excessive increase in calculation amount and interference from redundant information. Compared with the existing technology, Swin TransformerD-Block introduces dense connections and enhances feature reuse on the basis of the original Swin Transformer Block, avoiding overfitting on the foreign body dataset of cigarette package cross-sections, which can effectively improve detection efficiency.

[0044] S1-4: Candidate region generation module 3 fuses feature maps F, G, H, and I to generate a fixed-size feature map K;

[0045] First, FPN (Feature Pyramid Networks) fuses the shallow detail information extracted by the feature extraction module with the high-level semantic information, and outputs a feature map J containing more information; then, RPN (Region Proposal Networks) further processes the generated feature map J to generate a series of candidate regions with different lengths and widths; finally, Rol Align adjusts the candidate regions of different lengths and widths to become a feature map Feature Map with a fixed length and width, which is output as the feature map K.

[0046] S1-5: The target detection module 4 identifies and detects the type and position of the foreign object in the feature map K;

[0047] The target detection module is used to detect the feature map. The class head is used to classify the feature map and determine the category of the foreign object; the box head is used to regress the position of the true bounding box. Through the detection module, prediction results with appropriate positions and correct categories can be obtained. The target detection module mainly completes the following two tasks: first, classify the objects in the feature map K, the class head flattens the feature map K and obtains the classification results of the feature map K through a series of fully connected layers to determine the category of the foreign object in the cross section of the cigarette package; second, the box head obtains the precise position of the target through bounding box regression F. The bounding box regression F uses translation and scaling operations to:

[0048] F(P x ,P y ,P w ,P h )≈(GT x ,GT y ,GT w ,GT h )

[0049] Among them (P x ,P y ,P w ,P h ) are the center point coordinates, width and height of feature map K; (GT x ,GT y ,GT w ,GT h ) are the center point coordinates, width and height of the real annotation box.

[0050] S1-6: Iteratively execute steps S1-2 to S1-5 using the input image until the set number of training times is reached, and save the model of the last training.

[0051] Then, during the production process, an imaging device is used to obtain a slice cross-sectional image of the cigarette package in real time as an image to be detected;

[0052] Finally, the image to be detected is loaded into the model saved by the network model based on Transformer cigarette package cross-section foreign body detection to obtain the target detection result image. During the production process, corresponding operations are taken according to the target detection results.

[0053] In order to verify the effectiveness of the proposed method for detecting foreign matter in the slice section of a cigarette pack based on Transformer, the Mask RCNN (Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., ... & Guo, B. (2021). Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 10012-10022).) with Swin Transformer as the backbone is compared with the proposed method. The dataset uses the foreign matter detection dataset of the slice section of a cigarette pack established in step 1-1 above, and the evaluation index uses AP (AP) when IoU is 0.5. 50 ), AP (AP) when IoU is 0.75 75 ), AP (AP) when IoU is 0.5::0.95 bbox ). The results are shown in Table 1:

[0054]

[0055] Table 1

[0056] It can be seen from Table 1 that the method proposed in the present invention has higher detection accuracy than the existing Mask RCNN with Swin Transformer as the backbone.

[0057] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for detecting foreign matter in a cigarette packet section based on Transformer, characterized in that: The following steps are involved: S1: A network model for detecting foreign bodies in cigarette package slice sections based on Transformer is constructed. The network model includes four modules: a local information enhancement module (1), a feature extraction module (2), a candidate region generation module (3) and a target detection module (4). The training process of the network model includes the following steps: S1-1: Establish a cigarette package cross-section foreign body detection dataset as input image; S1-2: The local information enhancement module (1) performs image information enhancement on the input image and outputs a feature map D; S1-3: The feature extraction module (2) extracts features from the feature map D and outputs feature maps F, G, H, and I; S1-4: the candidate region generation module (3) fuses the feature maps F, G, H, and I to generate a feature map K of a set size; S1-5: the target detection module (4) identifies and detects the type and position of the foreign object in the feature map K; S1-6: Iteratively execute steps S1-2 to S1-5 using the input image until the set number of training times is reached, and save the model of the last training; S2: Obtaining a cross-sectional image of a cigarette package during the production process as an image to be detected; S3: Load the image to be detected in step S2 into the model saved in step S1-6 to obtain the target detection result in the production process; The feature extraction module (2) is divided into five parts: Patch Partition, Stage 1, Stage 2, Stage 3, and Stage 4; Stage 1 is composed of Linear Embedding and Swin Transformer D-Block; Stage 2, Stage 3, and Stage 4 are all composed of Patch Merging and Swin Transformer D-Block; The Swin Transformer D-Block simplifies the dense connection structure and combines it with the Swin TransformerBlock, and adds the feature map input to the Swin Transformer D-Block with the feature maps calculated by MLP1 and SW-MSA and MLP2, so that the feature map input to the Swin Transformer D-Block can be reused, and the feature map calculated by W-MSA and MLP1 is not added to the subsequent steps.

2. According to claim 1, a method for detecting foreign matter in a cigarette packet section based on a Transformer, characterized in that: The steps of establishing a cigarette pack cross-section foreign body detection data set in step S1-1 are as follows: first, filter out cigarette pack cross-section images containing foreign bodies, count the types and quantities of the foreign bodies, and make the number of each type of foreign bodies the same; then mark the foreign bodies in the image; finally, convert the marked data into data in the required format, and use the converted data to establish a cigarette pack cross-section foreign body detection data set.

3. The method for detecting foreign matter in a cigarette packet section based on Transformer according to claim 1, characterized in that: The local information enhancement module (1) is composed of three branches, wherein branch a is composed of a 3×3 convolution operation; branch b is composed of a 5×5 convolution operation and a 3×3 deconvolution operation; Branch c consists of a 7×7 convolution operation and a 5×5 deconvolution operation.

4. The method for detecting foreign matter in a cigarette packet section based on Transformer according to claim 1, characterized in that: The specific steps of generating candidate regions by the candidate region generation module (3) are as follows: first, FPN fuses the shallow detail information and high-level semantic information extracted by the feature extraction module (2) and outputs a feature map J containing more information; then, RPN further processes the generated feature map J to generate a series of candidate regions with different lengths and widths; finally, Rol Align adjusts the candidate regions with different lengths and widths to convert them into a feature map Feature Map with a set length and width, which is output as a feature map K.

Images