A model-fusion online cigarette package surface mildew detection method and system

Abstract

The present invention proposes a model-fused online cigarette package surface mildew detection method, comprising the following steps: obtaining online cigarette package surface images; performing image preprocessing on the online cigarette package surface images, and identifying online cigarette package surface images that are mildewed using detection rules; intercepting online cigarette package surface images that are mildewed to obtain mildewed area images, and obtaining labeling frame data cleaning based on histogram analysis based on the mildewed area images; establishing a target detection model, using the mildewed area images to train the target detection model, and the target detection model outputs a target frame screenshot image; establishing an image classification model, using the target frame screenshot image to train the image classification model; the present invention fuses the target detection model and the image classification model to perform real-time mildew detection, and when surface mildew is detected, an early warning alarm signal is issued. The present invention adds an image classification model to further reduce false positives.

Description

Technical Field

[0001] The present invention relates to the technical field of tobacco leaf mildew detection, and in particular to a model-fused online cigarette package surface mildew detection method and system. Background Art

[0002] The online cigarette pack unpacking system has a total of 3 sets of CCD visual systems, which detect the cigarette box strapping, the upper surface and the lower surface of the cigarette pack. It is impossible to monitor the surface mildew, which poses a quality risk of mildew. There are two problems with the current manual visual inspection method of the production line: First, it takes multiple workstations to complete the mildew detection on the six sides of the cigarette pack, which consumes a lot of manpower; second, long-term manual work can easily cause visual fatigue, and there is a risk of missing small areas of mildew or debris. Therefore, it is necessary to install visual inspection at appropriate places on the production line. Referring to a method, device and system for detecting mildew of cigarette packs based on an image with Chinese patent publication number CN109903288A, the method includes obtaining a photo of the surface of the cigarette pack; extracting color features from the photo of the surface of the cigarette pack; judging whether the color features of the cigarette pack meet the mildew condition; if so, judging the cigarette pack as a mildewed cigarette pack; otherwise, judging the cigarette pack as a non-mildewed cigarette pack. The model of the target detection algorithm for single-class mildew detection cannot adapt to the distinction of false positives of sequined tobacco leaves similar to mildew. Summary of the invention

[0003] The present invention solves the problem that the existing mildew detection using image recognition is prone to false alarms, proposes an online cigarette package surface mildew detection method and system with model fusion, and adds an image classification model to further reduce false alarms.

[0004] In order to achieve the above objectives, the following technical solutions are proposed:

[0005] A model-fused online cigarette package surface mildew detection method, characterized by comprising the following steps:

[0006] S100, acquiring an online cigarette package surface image;

[0007] S200, performing image preprocessing on the online cigarette package surface images, and identifying online cigarette package surface images that are moldy using detection rules;

[0008] S300, intercepting the surface image of the online cigarette package that has become moldy to obtain an image of the moldy area, and performing labeling frame data cleaning based on histogram analysis according to the image of the moldy area;

[0009] S400, establishing a target detection model, using the moldy area image to train the target detection model, and having the target detection model output a target frame screenshot image;

[0010] S500, establishing an image classification model, and training the image classification model using the target frame screenshot image;

[0011] S600 integrates the target detection model and image classification model to perform real-time mold detection and issue a warning alarm signal when surface mold is detected.

[0012] Because the target detection model searches for targets in the entire image, but the background of the moldy target is relatively complex, there will be false alarms of shiny tobacco leaves or tobacco stems. However, target detection cannot simply filter out false alarms through the detection threshold. In order to reduce false alarms while ensuring a high detection rate, we reduce the threshold of the target detection model and add an image classification model. All targets detected by the target detection model are divided into false alarm targets and real moldy targets. False alarm targets include shiny tobacco leaves and tobacco stems. False alarm targets and real moldy targets are input into the image classification model for training, that is, the fine screening model for mold detection is trained based on the target detection result image, further reducing the model false alarm rate. During online real-time detection, the two models are fused, that is, the results of the target detection model are output to the image classification model for real-time mold detection. When the system detects surface mold, it issues a warning alarm signal, thereby achieving the purpose of real-time detection of online tobacco package surface mold.

[0013] Preferably, the step S100 specifically includes the following steps:

[0014] The images of each surface after the online cigarette package unpacking process are collected, and the surface images of the online cigarette packages are obtained by using special optical effects for lighting. The special optical effects light selects the light in the band with large difference between moldy and background tobacco leaves.

[0015] Preferably, the step S200 specifically includes the following steps:

[0016] Image preprocessing includes image enhancement, image denoising and image segmentation processing on online cigarette pack images.

[0017] The image enhancement processing is one or more of grayscale transformation and histogram equalization; the image denoising processing is one or more of various filtering algorithms based on spatial domain and frequency domain; the image segmentation method is one of a threshold-based segmentation method, a region growing method, a clustering segmentation method and an edge-based segmentation method; the image segmentation method is one of a threshold-based segmentation method, a region growing method, a clustering segmentation method and an edge-based segmentation method. The threshold-based segmentation method is one of a fixed threshold method and an adaptive threshold method. The edge-based segmentation method is one of a Robert operator, a Sobel operator, a Prewitt operator, a Kirsch operator, a Canny operator, a Log operator, and a Laplacian operator.

[0018] Preferably, the image enhancement process is grayscale transformation, grayscale transformation is performed on the online cigarette package image, that is, the image is converted from a color image to a gray image; the grayscale image is subjected to mean smoothing filtering, and the current pixel point (x, y) is used as the grayscale g(x, y) of the processed image at this point, that is, g(x, y) = ∑f(x, y) / m, m is the total number of pixels in the template including the current pixel; the image after mean smoothing filtering is segmented based on a fixed threshold, the original image is set to f(x, y), and the segmentation threshold T is set in f(x, y), the image is segmented into two parts, and the segmented image is: b0 = 0 (black), b1 = 1 (white), that is, the image is binarized. Select the area in the binary image where the connected domain meets the set area, aspect ratio and circularity.

[0019] Preferably, the detection rules are established based on existing online cigarette package moldy image samples, and can identify shape features of complex texture objects, including area, perimeter, center of mass, dispersion, rectangularity, aspect ratio, direction, rectangularity of the region, circularity, sphericity, eccentricity, area-to-perimeter ratio, fineness, and one or more of Hu invariant moment features.

[0020] Preferably, the minimum bounding rectangle of the selected area is calculated, and the original image is intercepted using the minimum bounding rectangle.

[0021] Preferably, the labeling frame data cleaning based on histogram analysis is the aspect ratio and area histogram distribution of the moldy area image, and the distributed discrete points are deleted or the aspect ratio or area corresponding to the discrete points is adjusted according to the concentrated distribution situation.

[0022] Preferably, the target detection model is a YOLO model or an SSD model or a Fast-RCNN model or a Faster-RCNN model or a Mask-RCNN model.

[0023] Preferably, the image classification model is an AlexNet model, an ImageNet model, a Vgg16 model, or a Resnet model.

[0024] Preferably, the S600 specifically includes the following steps: a fusion target detection model and an image classification model are sequentially connected, and the prediction result is:

[0025]

[0026] Among them, P class is the final prediction result of the fusion model. A prediction of 1 indicates that mold is detected. pre_conf is the prediction confidence of the target detection model. thred_conf is the confidence threshold of the target detection model for detecting mold. seg Predict results for image classification.

[0027] Preferably, the P seg The image classification prediction results are:

[0028]

[0029] Among them, pre_calss is the image classification prediction label. The prediction label of 0 indicates the moldy category, and the prediction label of other categories indicates normal.

[0030] A model-fused online cigarette package surface mildew detection system, applicable to the above-mentioned model-fused online cigarette package surface mildew detection method, comprising:

[0031] An acquisition module, used for acquiring online cigarette package surface images;

[0032] The preprocessing module is used to preprocess the image into the data format input into the target detection model;

[0033] Object detection training module, used for online cigarette package surface mold detection based on object detection;

[0034] Image classification training module, used for online cigarette package surface mold detection based on image classification;

[0035] A real-time detection module is used to fuse the data of the target detection training module and the image classification training module to perform real-time mold detection;

[0036] The alarm module is used to send out a warning alarm signal when the system detects mildew on the surface.

[0037] The beneficial effect of the present invention is that the present invention adds an image classification model on the basis of the target detection model, and fuses the two models for recognition, thereby further reducing the occurrence of false alarms. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0039] Figure 2 It is a schematic diagram of the overall structure of the present invention;

[0040] Figure 3 The effect picture of normal tobacco leaves under special lighting;

[0041] Figure 4 Effect picture of specially polished moldy tobacco leaves;

[0042] Figure 5 Image mean filtering effect;

[0043] Figure 6 Image selection feature area effect;

[0044] Figure 7 The distribution of the length-to-width ratio of the annotation box before optimization;

[0045] Figure 8 Distribution of the area value of the marked box before cleaning;

[0046] Fig. 9 Distribution of the aspect ratio of the annotation box after cleaning;

[0047] Fig.10 Distribution of the area value of the annotation box after cleaning;

[0048] Fig.11 YOLO model recognition effect. DETAILED DESCRIPTION

[0049] Example:

[0050] This embodiment proposes a model fusion online cigarette package surface mildew detection method, such as Figure 1 As shown, the following steps are included:

[0051] S100, obtaining an online cigarette package surface image;

[0052] Industrial cameras are used to collect images of various surfaces of the online cigarette pack unpacking process. Special optical effects are used for lighting, and the wavelength band with large differences between moldy and background tobacco leaves is selected for lighting. Specially lit tobacco leaves and moldy images are shown in Figure 1. Figure 3-4 As shown;

[0053] S200, online cigarette pack surface mildew detection based on detection rules, i.e., establishing specific detection rules based on existing online cigarette pack mildew image samples, including: image preprocessing and online cigarette pack mildew shape feature determination, and designing an online cigarette pack surface mildew detection model based on detection rules;

[0054] The preprocessing of the moldy online cigarette package image is as follows: grayscale conversion is performed on the online cigarette package image, that is, the image is converted from a color image to a gray image; the grayscale image is mean smoothed and filtered, and the current pixel point (x, y) is used as the grayscale g(x, y) of the processed image at this point, that is, g(x, y) = ∑f(x, y) / m, where m is the total number of pixels in the template including the current pixel, and m is 9. The effect of mean smoothing filtering is as follows: Figure 5-6 As shown;

[0055] The image after mean smoothing filtering is segmented based on a fixed threshold. Let the original image be f(x, y). Set the segmentation threshold T in f(x, y). T is 128. The image is divided into two parts. The segmented image is: b0 = 0 (black), b1 = 1 (white), which is what we usually call image binarization.

[0056] Selecting a connected domain in the binary image that satisfies an area in the range of 20 to 500, an aspect ratio in the range of 0.25 to 4, and a circularity in the range of 0.2 to 0.5;

[0057] S300, marking and cleaning of detection rule result data, including: intercepting the image of the moldy area detected by the rule and cleaning of the annotation box data based on histogram analysis;

[0058] Calculate the minimum bounding rectangle of the selected area and use the minimum bounding rectangle to intercept the original image; perform histogram analysis on the intercepted original image data, delete the distributed discrete points or adjust the aspect ratio or area corresponding to the discrete points according to the concentrated distribution, and clean the data with relatively discrete aspect ratio and area histogram distribution of the intercepted image. The distribution of area and aspect ratio histogram before and after data cleaning is as follows: Figure 7-10 As shown;

[0059] S400, online cigarette package surface mold detection based on target detection, that is, training a preliminary screening model for mold detection based on online cigarette package images; establishing a target detection model, training the target detection model using moldy area images, and having the target detection model output a target frame screenshot image; the target detection model is a YoloV5 model, and its loss function is divided into position loss and confidence loss:

[0060] Loss = L iou +L object

[0061]

[0062]

[0063] Among them, S represents the grid size, and the value of S is 13; B represents box;

[0064] If the box at position i,j has a target, its value is 1, otherwise it is 0;

[0065] If the box at position i,j has no object, its value is 1, otherwise it is 0.

[0066] Online cigarette pack surface mold detection based on the YOLO model, that is, the preliminary screening model for mold detection is trained based on online cigarette pack images. The effect of the YOLO model is as follows: Fig.11 As shown;

[0067] S500, online cigarette package surface mildew detection based on image classification, that is, training a fine screening model for mildew detection based on online cigarette package images; establishing an image classification model, and training the image classification model using target frame screenshot images; the image classification model uses a Resnet network.

[0068] S600, the two models are integrated to perform real-time detection of mildew; when the system detects surface mildew, it sends out an early warning alarm signal, thereby achieving the purpose of real-time detection of mildew on the surface of online cigarette packages.

[0069] Through the technical solution of the present invention, it is possible to accurately know which link has a problem and whether there is mildew. If mildew is identified, the system will send out an early warning alarm signal to alarm.

[0070] This embodiment also proposes a model fusion online cigarette package surface mildew detection system, such as Figure 2 As shown, it includes: an acquisition module 100, a rule detection module 200, a target detection training module 300, an image classification training module 400, a real-time detection module 500 and an alarm module 600;

[0071] The acquisition module 100 is used to acquire online cigarette pack surface images;

[0072] The preprocessing module 200 is used for image preprocessing and online cigarette package mold shape feature determination, and designs an online cigarette package surface mold detection model based on detection rules;

[0073] The target detection training module 300 is used for online cigarette package surface mildew detection based on target detection, that is, a preliminary screening model for mildew detection is trained based on online cigarette package images;

[0074] The image classification training module 400 is used for online cigarette package surface mildew detection based on image classification, that is, a fine screening model for mildew detection is trained based on online cigarette package images;

[0075] The real-time detection module 500 is used to merge the two models to perform real-time mildew detection;

[0076] The alarm module 600 is used to send out a warning alarm signal when the system detects that the surface is moldy, thereby achieving the purpose of real-time detection of mold on the surface of online cigarette packages.

[0077] Through the technical solution of the device of the present invention, it is possible to accurately know which link has a problem and whether it exceeds the early warning alarm threshold. If it exceeds, the system will send out an early warning alarm signal to alarm.

Claims

1. A model-fused online cigarette package surface mildew detection method, characterized in that: The following steps are involved: S100, using light in a wavelength band with a large difference between moldy and background tobacco leaves to illuminate, and obtaining an online cigarette package surface image; S200, performing image preprocessing on the online cigarette package surface images, and identifying online cigarette package surface images that are moldy using detection rules; S300, intercepting the surface image of the online cigarette package that has become moldy to obtain an image of the moldy area, and obtaining a labeling frame data cleaning based on histogram analysis according to the image of the moldy area; the labeling frame data cleaning based on histogram analysis is the aspect ratio and area histogram distribution of the moldy area image; S400, establishing a target detection model, using the moldy area image to train the target detection model, and having the target detection model output a target frame screenshot image; S500, establishing an image classification model, and training the image classification model using the target frame screenshot image; All targets detected by the target detection model are divided into false positive targets and real moldy targets, and the false positive targets and real moldy targets are input into the image classification model for training; S600, integrates the target detection model and the image classification model to perform real-time mildew detection. When surface mildew is detected, a warning alarm signal is issued to determine whether the prediction confidence of the target detection model is greater than the confidence threshold. If so, the prediction result is that mildew is detected. If not, the result of the target detection model is output to the image classification model to obtain the image classification prediction result.

2. The model-fusion online cigarette package surface mildew detection method according to claim 1, characterized in that: The step S100 specifically includes the following steps: Collect images of each surface of the online cigarette package after the unpacking process, use special optical effects to illuminate, and obtain the online cigarette package surface image.

3. The model-fusion online cigarette package surface mildew detection method according to claim 1, characterized in that: The step S200 specifically includes the following steps: Image preprocessing includes image enhancement, image denoising and image segmentation processing on online cigarette pack images.

4. The model-fusion online cigarette package surface mildew detection method according to claim 1, characterized in that: The labeling frame data cleaning based on histogram analysis is the aspect ratio and area histogram distribution of the moldy area image, and the distributed discrete points are deleted or the aspect ratio or area corresponding to the discrete points is adjusted according to the concentrated distribution situation.

5. The model-fusion online cigarette package surface mildew detection method according to claim 1, characterized in that: The target detection model is a YOLO model, an SSD model, a Fast-RCNN model, a Faster-RCNN model, or a Mask-RCNN model.

6. The model-fusion online cigarette package surface mildew detection method according to claim 1, characterized in that: The image classification model is an AlexNet model, an ImageNet model, a Vgg16 model, or a Resnet model.

7. The model-fusion online cigarette package surface mildew detection method according to claim 1, characterized in that: The S600 specifically includes the following steps: the fusion target detection model and the image classification model are sequentially connected, and the prediction result is: Among them, P class is the final prediction result of the fusion model. A prediction of 1 indicates that mold is detected. pre_conf is the prediction confidence of the target detection model. thred_conf is the confidence threshold of the target detection model for detecting mold. seg Predict the results for image classification.

8. The model-fusion online cigarette package surface mildew detection method according to claim 7, characterized in that: The P seg The image classification prediction results are: Among them, pre_calss is the image classification prediction label. The prediction label of 0 indicates the moldy category, and the prediction label of other categories indicates normal.

9. A model-fused online cigarette package surface mildew detection system, applicable to the model-fused online cigarette package surface mildew detection method according to claim 1, characterized in that: include: An acquisition module, used for acquiring online cigarette package surface images; The preprocessing module is used to preprocess the image into the data format input into the target detection model; Object detection training module, used for online cigarette package surface mold detection based on object detection; Image classification training module, used for online cigarette package surface mold detection based on image classification; A real-time detection module is used to fuse the data of the target detection training module and the image classification training module to perform real-time mold detection; The alarm module is used to send out a warning alarm signal when the system detects mildew on the surface.

Images