Lollipop defect detection method based on improved convolutional neural network algorithm

Abstract

The invention discloses a lollipop defect detection method based on an improved convolutional neural network algorithm, and the method comprises the steps: carrying out the preprocessing of a collected image; important features are extracted by applying phantom convolution, decoders of different levels are designed to output and add correction values to relieve gradient disappearance of the depth model, and redundant features are removed from an extracted feature map by applying multi-scale channel pruning to ensure that the network is more efficient. The local feature learning ability is enhanced through additional information correction and feature fusion by applying an attention mechanism, multi-scale fusion is enhanced by adopting a simplified feature aggregation strategy, and real-time detection of lollipop defects is realized by applying a lightweight detection head. According to the method, the defect identification accuracy and the positioning target recall ratio are improved, the calculated amount parameter quantity is reduced, and the cost of detecting the surface and internal defects of the lollipop is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of defect detection and deep learning technology, and specifically relates to a lollipop defect detection method based on an improved convolutional neural network algorithm. Background Art

[0002] During the production process of lollipops, due to improper control of syrup temperature and humidity, pigment residue, and excessive cooling speed, color defects such as surface pigment residue, missing sugar pieces on the surface, and internal bubble defects may occur. These defects affect its appearance and taste. At present, the detection of lollipop defects is mostly done by manual visual inspection, but the labor intensity is high, the time consumption is long, and the subjectivity is high. It is easy to miss detections and the efficiency is low. In the field of target detection, traditional neural networks can learn to recognize and classify various categories of images through a large amount of data and appropriate training, and show excellent performance in target detection tasks. However, there are also certain limitations, including: (1) Limited global perception: The model can sometimes only detect in a fixed window or area of the entire image, and cannot adaptively focus on the key areas where the target may appear. (2) Influence of interference information: The model is easily affected by interference information in the image that is not related to the target, resulting in a high false detection rate. (3) Poor scale invariance: The detection performance of the model may decrease at different scales, especially for small or large targets. Based on deep learning technology and combined with the characteristics of lollipop defects, the present invention develops a special detection model and algorithm for lollipop defects. Summary of the Invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a method for real-time detection of surface and internal defects of lollipops, to solve the problems of poor detection effect and poor real-time performance such as inaccurate measurement of color defects on the surface of lollipops, diversity and complexity of shape defects, and difficulty in detecting small internal bubble defects, to meet the needs of industrial automation and lay a technical foundation for establishing a real-time defect detection system for lollipops.

[0004] Technical solution: To achieve the above objectives, the present invention provides a lollipop defect detection method based on an improved convolutional neural network algorithm, comprising the following steps:

[0005] S1: Synchronously collect the color image and depth image of the lollipop, and perform preprocessing to obtain the color image C1 and depth image D1 of the lollipop detection image;

[0006] S2: Extract multi-scale features through phantom convolution and input them into the TEDM module for multimodal feature fusion;

[0007] S3: Use the EMSPC module to enhance multi-scale feature interaction and output optimized feature maps;

[0008] S4: Align multi-scale features through the Slimneck module and input them into the lightweight detection head to generate prediction boxes;

[0009] S5: Train the improved YOLOv11 model and deploy it to achieve real-time defect detection.

[0010] S6: Use the improved network model from S2 to S4 to train the lollipop dataset preprocessed by S1 to obtain a weight file, and use the weight file to locate and identify defective lollipops.

[0011] Furthermore, in S2, phantom convolution is applied to the image C1 processed by S1 to calculate a feature map p1 with the same number of channels but different feature sizes; the feature is input into the calculation module TEDM of the YOLOv11 model to obtain a new output feature p2; the calculation module TEDM includes 4 convolutional layers, M encoders and N decoders, N+2 fully connected layers, 2 LeakyReLU activation function layers, and 4 channel stacking calculations, and the number of channels of the output feature is twice the number of channels of the input feature; the calculated encoder modules include MHSA, FFN, LN, and RC modules, the M encoders have the same structure, and the number of input and output channels of each encoder is consistent; the calculated decoder modules include CAM, FFN, LN, and RC modules, the N encoders have the same structure, and the number of input and output channels of each encoder is consistent; the calculation module TE DM performs deep feature extraction on the encoder module in the C1 input structure, generating a color image feature output, PC1. After processing by M encoders, the color image feature output is PC2. Simultaneously, phantom convolution is applied to D1 to calculate a deep feature map, PD1, with the same number of channels but different feature sizes. PD1 is then used as input and fused with PC2 in the first decoder to generate the fused feature PF1. Within the N encoder structures, PF1 and PD1 are fused in the second decoder to generate PF2. PF2 and PD1 are fused in the third decoder to generate PF3, and this process continues until the Nth decoder outputs PFN. Finally, the five decoder outputs, PFN-4, PFN-3, PFN-2, PFN-1, and PFN, are mapped through the MLP layer, respectively, to generate the final features Q2, Q3, Q4, Q5, and Q6, denoted as output p2. The number of M and N can be adjusted, as different defects require different inference capabilities and a balance between speed and accuracy is needed. This approach can also be applied to different candies or different defects in the same candy.

[0012] Furthermore, in S3, the feature p2 output by the TEDM structure is used as the input of the calculation module EMSPC; the calculation module is divided into four equal parts according to the channel dimension, and convolution operations of different scales are performed respectively. The first part performs 1*1 convolution, the second part performs 3*3 convolution, the third part performs 5*5 convolution, and the fourth part performs 7*7 convolution. The features of the four scales are spliced along the channel dimension to obtain the multi-scale enhanced feature PMS; the 1*1 convolution is applied to the PMS for channel fusion to generate the final efficient feature enhancement output Q2 MS 、Q3 MS 、Q4 MS 、Q5 MS 、Q6 MS , denoted as p3.

[0013] Furthermore, in S4, the feature output p3 of Backbone is input into the Slimneck calculation module. The five features output by the calculation module are P2, P3, P4, P5, and P6, and the output is recorded as p4; the calculation module Slimneck includes 4 upsampling nn.Upsample modules, 8 channel splicing Concat modules, 8 VoVGSCSP lightweight modules, and 4 GSConv convolution modules. The calculation module Slimneck performs feature alignment through upsampling or convolution downsampling with a stride of 2, and uses Concat channel splicing to complete multi-scale feature interaction fusion; feature extraction is performed through the VoVGSCSP lightweight module to obtain enhanced scale features.

[0014] Furthermore, in S4, P2 and P6 are input into the TEfficientHead and XEfficientHead calculation modules to obtain two prediction box matrices of different scales, each of which contains information on the target position, whether there is a defect, confidence, defect category and category probability; the TEfficientHead and XEfficientHead contain classification and regression branches, two upsampling layers and a downsampling layer, a LeakyReLU activation function layer, and a Layer Normalization normalization layer, and the outputs of TEfficientHead and XEfficientHead are calculated as two tensors representing lollipop defects of different scales, where each tensor includes the position, size, defect category and confidence information of the prediction box.

[0015] Furthermore, the S1 synchronously collects the color image and the corresponding depth information image of the lollipop. The original lollipop image includes a color image and a depth image containing height information. The color image has 3 channels, namely R, G, and B. The depth image is a single-channel grayscale image. The depth image corresponds one-to-one to the color image.

[0016] Furthermore, the image preprocessing in S1 includes noise reduction processing, contrast enhancement, edge enhancement processing, and data expansion processing of the color image and the depth map.

[0017] Furthermore, the phantom convolution, TEDM, and EMSPC formulas are as follows:

[0018] Phantom Convolution:

[0019] Encoder part in TEDM:

[0020] Decoder part in TEDM:

[0021]

[0022] EMSPC:

[0023] Where P1 RGB is the color image feature output, P1 Depth is the depth map feature output, P2 RGB is the color image feature output after processing by M Encoder modules; P N+2 is the output of N decoders; P 1x1 ,P 3x3 ,P 5x5 ,P 7x7 is the feature after convolution operation of different scales, P MS It is a multi-scale enhancement feature.

[0024] Furthermore, the specific steps of model training and application in S6 include:

[0025] S6.1: Network training, the specific steps include:

[0026] S6.11: Configure the training environment, including initializing the software environment to ensure support for the deep learning framework.

[0027] S6.12: Import the preprocessed image data and annotation information into the training framework and configure the dataset path and category information;

[0028] S6.13: Build and improve the object detection network structure;

[0029] S6.14: Set training parameters, including dataset path, image size, number of training rounds, batch size, etc., start the training process, and generate the weight file;

[0030] S6.2: Defect detection: Use the trained weight file to locate and identify the lollipop to determine whether it is a defective lollipop.

[0031] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0032] (1) The present invention improves the YOLOv11 model for lollipop defect detection by adding multiple Encoder and Decoder modules to the backbone network to reduce the receptive field of the model, which is beneficial for the recognition of small targets. The number M of Encoders and the number N of Decoders can be adjusted, which can also be applied to different candies or different defects of the same candy. Because different defects require different reasoning capabilities, a balance between speed and accuracy is achieved.

[0033] (2) The present invention simultaneously fuses RGB images and depth images, so that defects that cannot be observed in ordinary images (such as fine cracks, etc.) are detected, and the fused features contain as much effective information as possible, which is convenient for subsequent network positioning and classification, and improves the accuracy and robustness of target detection; Multi-scale fusion: A correction value is designed in each decoder output, that is, an additional path is added to connect the image and numerical features. In essence, it is equivalent to providing each other's forward paths for the image and numerical features. Its function is similar to the residual connection, which alleviates the problem of failure in deep network training. Each correction constitutes a feature fusion process of different depths. The existence of two relatively independent paths ensures that each fusion contains only currently known valid data, thereby eliminating the interference of irrelevant information. The data fusion is dispersed in each layer, and the overall network presents a smooth process of gradual fusion, reducing the problem of uneven weights of fusion positions. Ultimately, the network based on the Autoformer encoder structure only needs fewer layers to achieve the prediction accuracy of the existing deep network.

[0034] (3) Based on the idea of convolution and GhostConv phantom convolution, the EMSCP module is designed to remove redundant feature information and improve the computational complexity of the convolution kernel channel to achieve multi-scale feature extraction, which makes the number of parameters and computational complexity lower than that of a single convolution and meets the needs of real-time detection and sorting.

[0035] (4) Adding large and small target lightweight detection heads can not only identify different types of defects, the small target detection head identifies bubbles inside the candy and tiny cracks on the candy block, and the large target detection head detects shape defects in the entire candy block, but also reduce the amount of calculation and parameters under the premise of increasing convolution, thereby achieving lightweight and high efficiency improvements. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1Schematic diagram of the algorithm flow of a lollipop defect detection method based on an improved convolutional neural network algorithm of the present invention;

[0037] Figure 2 It is a schematic diagram of the structure of the TEDM calculation module of the present invention;

[0038] Figure 3 It is a schematic diagram of the structure of the EMSCP calculation module of the present invention;

[0039] Figure 4 Schematic diagram of the structure of the Slimneck, TEfficientHead, and XEfficientHead computing modules of the present invention;

[0040] Figure 5 Schematic diagram of the improved YOLOv11 model structure of the present invention;

[0041] Figure 6 It is the noise reduction result diagram of the present invention;

[0042] Figure 7 It is the enhancement result diagram of the present invention;

[0043] Figure 8 is the accuracy graph of the present invention;

[0044] Figure 9 is the recall graph of the present invention;

[0045] Figure 10 1 is a diagram of detection results of different models in an embodiment of the present invention;

[0046] Figure 11 It is a visualization heat map of different model features in the embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of the present invention. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0048] like Figure 1-5 The lollipop defect detection method based on an improved convolutional neural network algorithm includes the following steps:

[0049] S1: synchronously collect the color image and the corresponding depth information image of the lollipop, pre-process the collected lollipop image, and obtain the color image and depth image of the lollipop detection image;

[0050] S2: Apply phantom convolution to the image processed by S1 to calculate the feature map p1 with the same number of channels but different feature sizes; input the features into the calculation module TEDM of the YOLOv11 model to obtain the new output feature p2;

[0051] S3: Use the output feature p2 as the input of the calculation module EMSPC to obtain the efficient feature enhanced output p3;

[0052] S4: Output feature p3 is used as the input of the calculation module Slimneck. The five output features are P2, P3, P4, P5, and P6, and the output is recorded as p4. P2 and P6 are input into the TEfficientHead and XEfficientHead calculation modules to obtain two prediction box matrices of different scales. Each prediction box matrix contains information such as the target location, whether there is a defect, confidence, defect category, and category probability.

[0053] S5: The improved network model of S2 to S4 is trained on the lollipop dataset preprocessed by S1 to obtain a weight file, and the weight file is used to locate and identify defective lollipops.

[0054] S1 simultaneously captures a color image of the lollipop and its corresponding depth information. The original lollipop image consists of a color image and a depth image containing height information. The image width is 2376 pixels, the height is 584 pixels, and the color image has three channels: R, G, and B. The depth image is a single-channel grayscale image, with a one-to-one correspondence between the depth image and the color image. Image preprocessing in S1 includes noise reduction, contrast enhancement, edge enhancement, and data augmentation for both the color and depth images.

[0055] The calculation module TEDM in S2 includes 4 convolutional layers, M encoders and N decoders, N+2 fully connected layers, 2 LeakyReLU activation function layers, and 4 channel stacking calculations. The number of channels of the output feature is twice the number of channels of the input feature. The number of M and N can be adjusted because different defects require different inference capabilities and a balance needs to be achieved between speed and accuracy. It can also be applied to different candies or different defects of the same candy. The calculated encoder modules include MHSA, FFN, LN, and RC modules. The M encoders have the same structure, and the number of input and output channels of each encoder is the same. The calculated decoder modules include CAM, FFN, LN, and RC modules. The N encoders have the same structure, and the number of input and output channels of each encoder is the same. The calculation formulas for phantom convolution, TEDM, and EMSPC are as follows:

[0056]

[0057] Where P1 RGB is the color image feature output, P2 RGB is the color image feature output after processing by M Encoder modules; P N+2 is the output of N decoders; P 1x1 ,P 3x3 ,P 5x5 ,P 7x7 is the feature after convolution operation of different scales, P MS It is a multi-scale enhancement feature.

[0058] In S4, TEfficientHead and XEfficientHead contain classification and regression branches, two upsampling layers and a downsampling layer, a LeakyReLU activation function layer, and a Layer Normalization layer. The outputs of TEfficientHead and XEfficientHead are two tensors representing lollipop defects of different scales. Each tensor includes the location, size, defect category, and confidence information of the predicted box.

[0059] The specific steps for model training and application in S6 include:

[0060] S6.1: Network training, the specific steps include:

[0061] S6.11: Configure the training environment, including initializing the software environment to ensure support for the deep learning framework.

[0062] S6.12: Import the preprocessed image data and annotation information into the training framework and configure the dataset path and category information;

[0063] S6.13: Build and improve the object detection network structure;

[0064] S6.14: Set training parameters, including dataset path, image size, number of training rounds, batch size, etc., start the training process, and generate the weight file;

[0065] S6.2: Defect detection: Use the trained weight file to locate and identify the lollipop to determine whether it is a defective lollipop.

[0066] The above lollipop defect detection method is used to collect lollipop image data as a data set to verify the detection performance of the lollipop defect detection method based on the improved convolutional neural network algorithm proposed in the present invention and compare it with the existing method. The results are as follows: Figure 10-11 As shown in Table 1.

[0067] Table 1 Statistics of lollipop defect detection performance of different inspection methods

[0068] Detection method Accuracy (%) Recall rate (%) Mean average precision (%) YOLOv5 89.6 88.4 90.1 YOLOv8 91.2 92.3 93.8 Method of the present invention 94.8 95.2 95.2

[0069] YOLOv5 References: Zhang Y, Guo Z, Wu J, et al.Real-time vehicle detection based on improved yolo v5[J].Sustainability, 2022,14(19):12274.

[0070] YOLOv8 reference: Lou H, Duan X, Guo J, et al. DC-YOLOv8: Small-size object detection algorithm based on camera sensor[J]. Electronics, 2023, 12(10): 2323.

[0071] By analyzing Table 1, it can be seen that the lollipop defect detection method based on the improved convolutional neural network algorithm proposed in the present invention is relatively improved by 1.4% to 5.1% in terms of mean average precision index compared with other methods.

Claims

1. A lollipop defect detection method based on an improved convolutional neural network algorithm, characterized in that: The following steps are involved: S1: Synchronously collect the color image and depth image of the lollipop, and perform preprocessing to obtain the color image C1 and depth image D1 of the lollipop detection image; S2: Extract multi-scale features through phantom convolution and input them into the TEDM module for multimodal feature fusion; S3: Use the EMSPC module to enhance multi-scale feature interaction and output optimized feature maps; S4: Align multi-scale features through the Slimneck module and input them into the lightweight detection head to generate prediction boxes; S5: Train the improved YOLOv11 model and deploy it to achieve real-time defect detection. S6: The improved network model of S2 to S4 is trained on the lollipop dataset preprocessed by S1 to obtain a weight file, and the weight file is used to locate and identify defective lollipops.

2. The lollipop defect detection method based on an improved convolutional neural network algorithm according to claim 1, characterized in that: In S2, phantom convolution is applied to the image C1 processed by S1 to calculate a feature map p1 with the same number of channels but different feature sizes; the feature is input into the calculation module TEDM of the YOLOv11 model to obtain a new output feature p2; the calculation module TEDM includes 4 convolutional layers, M encoders and N decoders, N+2 fully connected layers, 2 LeakyReLU activation function layers, and 4 channel stacking calculations. The number of channels of the output feature is twice the number of channels of the input feature; the calculated encoder module includes MHSA, FFN, LN, and RC modules. The M encoders have the same structure, and each encoder has The number of input and output channels is consistent; the calculated decoder module includes CAM, FFN, LN, and RC modules, and the N encoders have the same structure, and the number of input and output channels of each encoder is consistent; the calculation module TEDM performs deep feature extraction on the encoder module in the C1 input structure to obtain a color image feature output PC1. After processing by M encoders, the feature output of the color image is PC2; at the same time, phantom convolution is applied to D1 to calculate a deep feature map PD1 with the same number of channels but different feature sizes; then PD1 is used as input to perform feature fusion with PC2 in the first decoder to obtain a fused feature PF1; In the N encoder structures, PF1 and PD1 are fused through the second decoder to obtain PF2, and PF2 and PD1 are fused through the third decoder to obtain PF3, and so on, until the Nth decoder outputs PFN; the last five decoder outputs PFN-4, PFN-3, PFN-2, PFN-1, and PFN are mapped through the MLP layer respectively to obtain the final features Q2, Q3, Q4, Q5, and Q6; recorded as output p2.

3. The lollipop defect detection method based on an improved convolutional neural network algorithm according to claim 1, characterized in that: In S3, the feature p2 output by the TEDM structure is used as the input of the calculation module EMSPC; the calculation module is divided into four equal parts according to the channel dimension, and convolution operations of different scales are performed respectively. The first part performs 1*1 convolution, the second part performs 3*3 convolution, the third part performs 5*5 convolution, and the fourth part performs 7*7 convolution. The features of the four scales are spliced along the channel dimension to obtain the multi-scale enhanced feature PMS; the 1*1 convolution is applied to the PMS for channel fusion to generate the final efficient feature enhancement output Q2 MS 、Q3 MS 、Q4 MS 、Q5 MS 、Q6 MS , denoted as p3.

4. The lollipop defect detection method based on an improved convolutional neural network algorithm according to claim 3, characterized in that: In S4, the feature output p3 of Backbone is input into the Slimneck calculation module. The five features output by the calculation module are P2, P3, P4, P5, and P6, and the output is recorded as p4; the calculation module Slimneck includes 4 upsampling nn.Upsample modules, 8 channel splicing Concat modules, 8 VoVGSCSP lightweight modules, and 4 GSConv convolution modules. The calculation module Slimneck performs feature alignment through upsampling or convolution downsampling with a stride of 2, and uses Concat channel splicing to complete multi-scale feature interaction fusion; feature extraction is performed through the VoVGSCSP lightweight module to obtain enhanced scale features.

5. The lollipop defect detection method based on an improved convolutional neural network algorithm according to claim 1, characterized in that: In S4, P2 and P6 are input into the TEfficientHead and XEfficientHead calculation modules to obtain two prediction box matrices of different scales, each of which contains information on the target position, whether there is a defect, confidence, defect category and category probability; the TEfficientHead and XEfficientHead contain classification and regression branches, two upsampling layers and a downsampling layer, a LeakyReLU activation function layer, and a Layer Normalization normalization layer, and the outputs of TEfficientHead and XEfficientHead are calculated as two tensors representing lollipop defects of different scales, where each tensor includes the position, size, defect category and confidence information of the prediction box.

6. The lollipop defect detection method based on an improved convolutional neural network algorithm according to claim 1, characterized in that: The S1 synchronously collects the color image and the corresponding depth information image of the lollipop. The original lollipop image includes a color image and a depth image containing height information. The color image has 3 channels, namely R, G, and B. The depth image is a single-channel grayscale image. The depth image corresponds one-to-one to the color image.

7. The lollipop defect detection method based on an improved convolutional neural network algorithm according to claim 1, characterized in that: The image preprocessing in S1 includes noise reduction processing, contrast enhancement, edge enhancement processing, and data expansion processing of the color image and the depth map.

8. The lollipop defect detection method based on an improved convolutional neural network algorithm according to claim 2 or 3, characterized in that: The phantom convolution, TEDM, and EMSPC formulas are as follows: Phantom Convolution: Encoder part in TEDM: Decoder part in TEDM: EMSPC: Where P1 RGB is the color image feature output, P1 Depth is the depth map feature output, P2 RGB is the color image feature output after processing by M Encoder modules; P N+2 is the output of N decoders; P 1x1 ,P 3x3 ,P 5x5 ,P 7x7 is the feature after convolution operation of different scales, P MS It is a multi-scale enhancement feature.

9. The lollipop defect detection method based on an improved convolutional neural network algorithm according to claim 1, characterized in that: The specific steps of model training and application in S6 include: S6.1: Network training, the specific steps include: S6.11: Configure the training environment, including initializing the software environment to ensure support for the deep learning framework. S6.12: Import the preprocessed image data and annotation information into the training framework and configure the dataset path and category information; S6.13: Build and improve the object detection network structure; S6.14: Set training parameters, including dataset path, image size, number of training rounds, batch size, etc., start the training process, and generate the weight file; S6.2: Defect detection: Use the trained weight file to locate and identify the lollipop to determine whether it is a defective lollipop.

Images