Feed outer packaging defect classification detection method based on Mask RCNN

Abstract

The present invention discloses a method for grading feed packaging defects based on Mask RCNN, comprising the following steps: collecting images of feed packaging; annotating different types of packaging defects and corresponding defect levels in each image; inputting the annotated images into a Mask RCNN network, training and saving the network; and after collecting the feed packaging images, loading the Mask RCNN network, processing and evaluating the images. If a defect is detected, the detected defect category and corresponding defect level are output. The present invention utilizes an improved Mask RCNN network to perform real-time image processing, evaluation, and analysis on feed packaging images collected, providing online defect type and defect grading detection results. This facilitates the adoption of different subsequent production processes based on different defect levels, thereby optimizing the production line structure, improving production efficiency, increasing the yield rate of finished feed products, and enhancing economic benefits.

Description

Technical Field

[0001] The present invention belongs to the field of computers, and in particular relates to a feed outer packaging defect grading detection method based on Mask RCNN. Background Art

[0002] The feed production industry has largely achieved automated production lines, resulting in increasingly faster feed filling and higher output. However, during this rapid production process, finished feed packaging is prone to defects such as discoloration, tearing, damage, and misprints. These defects vary widely in severity, and the corresponding subsequent production processes are significantly different depending on the defect level. Therefore, a detection method is needed that can not only identify the type of feed packaging defect but also classify the defect to facilitate appropriate treatment of each defect level. Summary of the Invention

[0003] Purpose of the invention: In response to the problems existing in the prior art, the present invention discloses a method for grading feed packaging defects based on Mask RCNN.

[0004] Technical solution: To achieve the above-mentioned purpose, the present invention adopts the following technical solution:

[0005] A method for detecting defects in feed packaging based on Mask RCNN, comprising the following steps:

[0006] S1: Collect images of feed packaging and build a feed packaging image dataset;

[0007] S2: Use the image annotation tool Labelme to annotate the images in the dataset, marking different types of outer packaging defects and corresponding defect levels in each image in the dataset;

[0008] S3: Input the labeled dataset into the Mask RCNN network and train the Mask RCNN network using the cross-validation method. During training, set batch_size to 1 and epoch to 120. After the Mask RCNN network is trained, save the Mask RCNN network model parameter file.

[0009] S4: After the industrial camera captures the image of the feed packaging, it transmits the image to the industrial computer. The industrial computer uses the saved Mask RCNN network model parameter file to load the Mask RCNN network, inputs the image according to the sampling period, and processes and evaluates the image. If a defect is detected, the detected defect category and corresponding defect level are output, and the sound and light alarm device is controlled to issue a warning message for repackaging.

[0010] Preferably, the Mask RCNN network includes a backbone network composed of a RepVGG network and an FPN network, an RPN network and an image prediction network. After the image in the labeled dataset is input into the Mask RCNN network, the RepVGG network extracts the features of the input image. After the RepVGG network completes feature extraction of the input image, the extracted features are input into the FPN network for feature pyramid feature extraction. After the FPN network outputs the final feature map of the input image, the final feature map is input into the RPN network to generate candidate boxes. After the RPN network outputs the ROI area corresponding to each final candidate box, the image prediction network calculates the loss of each candidate box output by the RPN network through a multi-task loss function.

[0011] Preferably, the image prediction network calculates the loss of each candidate box output by the RPN network through a multi-task loss function including:

[0012] Total loss function L all :

[0013] L all =L bbox +L class +L mask +L grade

[0014] L bbox Bounding box regression loss function for defect identification:

[0015]

[0016]

[0017] Among them, t u represents the coordinate position of the candidate box predicted by the defect category u, and v represents the actual coordinate position of the marked bounding box; Represents the coordinate position of the i-th candidate box predicted by defect category u, i is the candidate box sequence number, where and Indicates the center coordinates of the candidate box, and Indicates the width and height of the candidate box; v i Indicates the true coordinate position of the marked i-th bounding box;

[0018] L class Classification loss function for different defect categories of candidate boxes:

[0019] L class (p,u)=-logp u

[0020] Where p = (p0, p1, ...) represents the probability that the candidate box output by the network belongs to each category and background, u is the real defect category annotation information, and p u The probability that the candidate box belongs to the real defect category u;

[0021] L mask Generate a loss function for the defect mask within the candidate box:

[0022]

[0023] Among them, M all is the total number of masks, y i Represents the i-th ROI region generated by the RPN network, represents the corresponding annotation true value;

[0024] L grade The loss function for measuring defect level is:

[0025]

[0026] where N grade is the total number of ROIs in the image; x il is a binary indicator variable. When the i-th candidate box belongs to defect level l, x il =1, otherwise it is 0. il It represents the probability that the i-th candidate box output by the deep neural network belongs to defect level l, where L is the total number of defect levels.

[0027] Preferably, calculate L grade The corresponding neural network structure includes two fully connected layers. The feature map containing the final candidate box output by the RPN network is sequentially input into the fully connected layer with the activation function of tanh and the fully connected layer with the activation function of softmax, and then the confidence of each candidate box belonging to each defect level is output.

[0028] Beneficial effects: Compared with the prior art, the present invention has the following significant beneficial effects:

[0029] Based on feed packaging images captured by industrial cameras, the present invention uses an improved Mask RCNN network to perform real-time image processing, evaluation, and analysis, and provides online defect type and defect classification detection results. This facilitates the adoption of different subsequent production processes based on different defect levels, thereby optimizing the production line structure, improving production efficiency, and increasing the yield rate of finished feed products, thereby enhancing economic benefits.

[0030] The feature extraction network layer in the improved Mask RCNN network adopts the RepVGG network, which can further improve the network's feature extraction accuracy for the input image and enhance the defect grading detection effect. A loss function for measuring the defect grade is added to the calculation of the multi-task loss function. While detecting the defect type, the detected defect grade can also be given at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the process of the feed outer packaging defect classification detection method based on Mask RCNN according to the present invention;

[0032] Figure 2 Schematic diagram of the structure of the improved Mask RCNN network described in the present invention;

[0033] Figure 3 This is a schematic diagram of the structure of the backbone network described in the present invention;

[0034] Figure 4 The calculation L grade Schematic diagram of the corresponding neural network structure. DETAILED DESCRIPTION

[0035] The present invention will be further described below with reference to the accompanying drawings.

[0036] The present invention discloses a method for detecting defects in feed packaging based on Mask RCNN. Figure 1 As shown, the following steps are included:

[0037] Step S1: Collect feed outer packaging images and build a feed outer packaging image dataset.

[0038] Step S2: Use the image annotation tool Labelme to annotate the images in the dataset, and annotate different types of outer packaging defects and corresponding defect levels in each image in the dataset.

[0039] Step S3: Input the labeled data set into the neural network and train the neural network using the cross-validation method. During training, set batch_size to 1 and epoch to 120. After the neural network training is completed, save the neural network model parameter file.

[0040] When using the cross-validation method to train a neural network, the labeled data set is divided into 5 equal parts. One of them is used as the test set for each training, and the remaining 4 parts are used as training sets. The test sets are selected in turn from the data set divided into 5 equal parts to complete the neural network training.

[0041] The neural network in the present invention adopts an improved Mask RCNN network, and the improved Mask RCNN network framework structure is as follows: Figure 2 As shown in the figure, it includes a backbone network composed of a RepVGG network and an FPN (Feature Pyramid Networks) network, an RPN (Region Proposal Network) network, and an image prediction network. The backbone network is used to extract some low-level common features, including color, shape, and texture. The RPN network is used to further process these common features. The image prediction network outputs results in different forms, such as defect type probability, defect level probability, etc.

[0042] First, after the images in the labeled dataset are input into the improved Mask RCNN network, the feature extraction network layer in the backbone network extracts the features of the input image. In order to improve the efficiency and accuracy of feature extraction and increase the overall training speed of the network, the feature extraction network layer in the classic Mask RCNN network is the ResNet network. In order to further improve the network's feature extraction accuracy for the input image and improve the defect classification detection effect, the feature extraction network layer in the backbone network of the present invention is the RepVGG network, and the structure is shown in Table 1 below:

[0043] Table 1

[0044]

[0045] According to Table 1, the entire RepVGG network contains 5 Stages. Stage 1 only contains one residual block, and the output feature size is 1024×1024. The number of channels in the residual block is taken as min(64, 64a), where min(64, 64a) represents the smaller value between 64 and 64a; Stage 2 contains 2 residual blocks, and the output feature size is 512×512, and the number of channels in the residual block is 64a; Stage 3 contains 4 residual blocks, and the output feature size is 256×256, and the number of channels in the residual block is 128a; Stage 4 contains 14 residual blocks, and the output feature size is 128×128, and the number of channels in the residual block is 256a; Stage 5 contains 1 residual block, and the output feature size is 64×64, and the number of channels in the residual block is 512b. The entire RepVGG network forms different versions of variants through parameter a and parameter b. Parameter a and parameter b are used to control the number of channels in the residual block. Parameter a is used to control the number of channels in the residual blocks in stages 1 to 4, and parameter b is used to control the number of channels in the residual block in stage 5. Generally, a < b. The main reason is that the residual block in Stage 5 needs to have more channels to obtain some more robust feature representations, which is beneficial for the subsequent image prediction network to output more accurate prediction results. In this invention, parameter a = 1.5 and parameter b = 2.75. The size of the input image to the RepVGG network is 2048×2048.

[0046] After the RepVGG network completes the feature extraction of the input image, the extracted features are input into the FPN network for feature pyramid feature extraction, so as to map RoIs (Regions of Interest) of different scales to the corresponding feature layers. By performing upsampling on the feature maps of different levels respectively and fusing them with the bottom-layer features, the final features with high resolution and strong semantics are obtained.

[0047] The backbone network structure after combining the RepVGG network and the FPN network is as Figure 3 shown. The RepVGG network includes stages 1 to 5, and the FPN network includes convolutional blocks Conv2d_1 to Conv2d_8 and the max pooling layer MaxPool. Specifically:

[0048] Stage 2 in the RepVGG network outputs features to the convolution block Conv2d_1 in the FPN network, and the output feature size of the convolution block Conv2d_1 is 512×512; Stage 3 in the RepVGG network outputs features to the convolution block Conv2d_2 in the FPN network, and the output feature size of the convolution block Conv2d_2 is 256×256; Stage 4 in the RepVGG network outputs features to the convolution block Conv2d_3 in the FPN network, and the output feature size of the convolution block Conv2d_3 is 128×128; Stage 5 in the RepVGG network outputs features to the convolution block Conv2d_4 in the FPN network, and the output feature size of the convolution block Conv2d_4 is 64×64. Convolution block Conv2d_4 outputs features to convolution block Conv2d_8, and the output feature size of convolution block Conv2d_8 is 64×64. Convolution block Conv2d_8 outputs features to the maximum pooling layer MaxPool, and the output feature size of the maximum pooling layer MaxPool is 32×32. The features output by convolution block Conv2d_4 are upsampled and then superimposed with the features output by convolution block Conv2d_3. The superimposed features are input to convolution block Conv2d_7, and the output feature size of convolution block Conv2d_7 is The features output by convolution block Conv2d_3 are upsampled and then superimposed with those output by convolution block Conv2d_2. The superimposed features are then fed into convolution block Conv2d_6, and the output features of convolution block Conv2d_6 are of size 256×256. The features output by convolution block Conv2d_2 are upsampled and superimposed with those output by convolution block Conv2d_1. The superimposed features are then fed into convolution block Conv2d_5, and the output features of convolution block Conv2d_5 are of size 512×512. Convolution blocks Conv2d_5–Conv2d_8 and the max pooling layer MaxPool output features to the subsequent RPN network.

[0049] After the FPN network outputs the final feature map of the input image, it is fed into the RPN network for further generation of candidate boxes. The RPN network divides the feature map into m×n regions and, through the anchor mechanism, generates k candidate boxes at each pixel where an object may be located. Each candidate box is then judged and the position of the final candidate box is determined through NMS (non-maximum suppression).

[0050] After the RPN network outputs the ROI area corresponding to each final candidate box, the image prediction network calculates the loss of each candidate box output by the RPN network through the following multi-task loss function:

[0051] L all =L bbox +L class+L mask +L grade

[0052] Among them L all Represents the total loss function.

[0053] L bbox For defect recognition bounding box regression loss function, calculate L bbox The corresponding neural network structure is consistent with the corresponding part in the classic MaskRCNN.

[0054]

[0055]

[0056] Among them, t u represents the coordinate position of the candidate box predicted by the defect category u, and v represents the actual coordinate position of the marked bounding box; Represents the coordinate position of the i-th candidate box predicted by defect category u, i is the candidate box sequence number, where and Indicates the center coordinates of the candidate box, and Indicates the width and height of the candidate box; v i Represents the true coordinate position of the marked i-th bounding box.

[0057] L class Calculate L for the classification loss function of different defect categories of the candidate box class The corresponding neural network structure is consistent with the corresponding part in the classic Mask RCNN.

[0058] L class (p,u)=-logp u

[0059] Where p = (p0, p1, ...) represents the probability that the candidate box output by the network belongs to each category and background, u is the real defect category annotation information, and p u The probability that the candidate box belongs to the true defect category u.

[0060] L mask Generate loss function for the defect mask in the candidate frame and calculate L mask The corresponding neural network structure is consistent with the corresponding part in the classic Mask RCNN. mask It is defined as the average binary cross entropy loss function, that is:

[0061]

[0062] Among them, M all is the total number of masks, y iRepresents the i-th ROI region generated by the RPN network, Indicates the corresponding true value.

[0063] L grade is the loss function to measure the defect level.

[0064]

[0065] where N grade is the total number of ROIs in the image, which is consistent with the total number of masks, that is, M all =N grade ;x il is a binary indicator variable. When the i-th candidate box belongs to defect level l, x il =1, otherwise it is 0. il It represents the probability that the i-th candidate box output by the deep neural network belongs to defect level l, where L is the total number of defect levels.

[0066] Calculate L grade The corresponding neural network structure is as follows Figure 4 As shown in the figure, it includes two fully connected layers. The feature map containing the final candidate box output by the RPN network is sequentially input into the fully connected layer with the activation function of tanh and the fully connected layer with the activation function of softmax, and then the confidence of each candidate box belonging to each defect level is output.

[0067] During the training of the improved Mask RCNN network, according to the final calculated loss L all , perform error back propagation calculation, continuously optimize the Mask RCNN network parameters, and save the final optimized Mask RCNN network parameter file.

[0068] Step S4: During actual application, after the industrial camera captures the image of the feed packaging, it transmits the image to the industrial computer. The industrial computer uses the saved Mask RCNN network parameter file to load the Mask RCNN network, inputs the image according to the sampling period, processes and evaluates the image, and outputs the detected defect category and corresponding defect level if a defect is detected. The sound and light alarm device is controlled to issue a warning message for repackaging.

[0069] The detection method proposed in the present invention can not only simultaneously detect multiple types of feed outer packaging defects, but also simultaneously give the level of each detected defect (level I, level II, level III, level IV, etc.), making it convenient to select different subsequent production processes according to the defect level.

[0070] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting defects in feed packaging based on Mask RCNN, characterized in that: The steps include: S1: Collect images of feed packaging and build a feed packaging image dataset; S2: Use the image annotation tool Labelme to annotate the images in the dataset, marking different types of outer packaging defects and corresponding defect levels in each image in the dataset; S3: Input the labeled dataset into the Mask RCNN network and train the Mask RCNN network using the cross-validation method. During training, set batch_size to 1 and epoch to 120. After the Mask RCNN network is trained, save the Mask RCNN network model parameter file. S4: After the industrial camera captures the image of the feed packaging, it transmits the image to the industrial computer. The industrial computer uses the saved Mask RCNN network model parameter file to load the Mask RCNN network, inputs the image according to the sampling period, processes and evaluates the image, and outputs the detected defect category and corresponding defect level if a defect is detected. The sound and light alarm device is controlled to issue a warning message for repackaging. The Mask RCNN network includes a backbone network composed of a RepVGG network and an FPN network, an RPN network, and an image prediction network. After the image in the labeled dataset is input into the Mask RCNN network, the RepVGG network extracts the features of the input image. After the RepVGG network completes feature extraction of the input image, the extracted features are input into the FPN network for feature pyramid feature extraction. After the FPN network outputs the final feature map of the input image, the final feature map is input into the RPN network to generate candidate boxes. After the RPN network outputs the ROI area corresponding to each final candidate box, the image prediction network calculates the loss of each candidate box output by the RPN network through a multi-task loss function. Among them, the total loss function : ， Bounding box regression loss function for defect recognition, is the classification loss function for different defect categories of the candidate box, Generate a loss function for the defect mask within the candidate frame; The loss function for measuring defect level is: ， in is the total number of ROIs in the image; is a binary indicator variable. candidate boxes belong to the defect level hour, , otherwise it is 0; Represents the output of the deep neural network candidate boxes belong to the defect level The probability of L is the total number of defect levels; calculate The corresponding neural network structure includes two fully connected layers. The feature map containing the final candidate box output by the RPN network is sequentially input into the fully connected layer with the activation function of tanh and the fully connected layer with the activation function of softmax, and then the confidence of each candidate box belonging to each defect level is output.

2. The method for detecting feed outer packaging defects based on Mask RCNN according to claim 1, wherein: Bounding box regression loss function for defect identification: ， in, Indicates defect category The predicted coordinate position of the candidate box, Indicates the real coordinate position of the marked bounding box; Indicates defect category The predicted The coordinate position of the candidate box, is the candidate frame number, where and Indicates the center coordinates of the candidate box, and Indicates the width and height of the candidate box; Indicates the marked The true coordinate position of the candidate box; Classification loss function for different defect categories of candidate boxes: ， in Indicates the probability that the candidate box output by the network belongs to each category and background, Label information for real defect categories, The candidate box belongs to the real defect category probability; Generate a loss function for the defect mask within the candidate box: ， in is the total number of masks, Represents the first ROI area, Indicates the corresponding true value.

Images