A scrap steel grading method based on region segmentation and Yolov5

Abstract

The invention discloses a scrap steel grading method based on regional segmentation and Yolov5. The Mask R-CNN segmentation model is adopted to segment the carriage from the background. The detection head of Yolov5 is improved to extract the feature of each segmented region, that is, the material type category and the confidence of the material type in the region. During training, the network is trained by back-propagating the classification loss and sigmoid confidence product of the segmented region. During reasoning, after the original image is input, the category and confidence of each region are directly output. Through a preset threshold, the invalid region is filtered out, that is, the grayscale region is filled. Finally, the scrap steel type of the whole carriage and its proportion are calculated. The advantages of the classification network and the detection network are absorbed, and good effects are achieved in speed and accuracy. The invention not only does not miss any region, but also focuses the model training on the classification model, and the network model is easier to converge.

Description

Technical Field

[0001] The invention belongs to the technical field, and in particular relates to a scrap steel grading method based on regional segmentation and Yolov5. Background Art

[0002] In order to reduce costs, increase efficiency and improve market competitiveness during the steel production process, steel companies usually recycle scrap steel for reuse. Generally, steel companies are equipped with a quality inspection team to conduct quality inspections on scrap steel. The quality inspectors use the steel company's standards and tools to determine the grade and penalty of scrap steel. Although the traditional quality inspection process clearly stipulates the size (length, width, thickness) of each scrap steel category, it is impossible to measure each piece of scrap steel in each car piece by piece during the actual quality inspection process. The scrap steel grade determined by the quality inspector is only based on years of experience. Different quality inspectors will have different results for the same car of scrap steel, especially due to the influence of human subjective factors, and no grading standard has been formed. This will result in steel companies being unable to effectively measure the scrap steel recovery output ratio, affecting the overall benefits of steel companies.

[0003] Thanks to the rapid development of computer hardware in recent years, deep learning technology has made great progress. It has not only achieved good results in natural language processing and speech, but also achieved better results than humans in the field of image recognition. By analyzing the artificial scrap steel judgment standards and the characteristics of various types of scrap steel, it is theoretically feasible to use deep learning to identify the material type, but there are the following problems in actual operation: the scrap steel carried by the carriage may contain multiple types of materials, and the whole vehicle needs to be classified into one category, such as heavy scrap steel, medium scrap steel and small scrap steel; the target detection method is not effective because the distribution of each type of scrap steel at different times is very different; when using classification recognition, it is impossible to calculate the included material types and their proportions. In summary, combining the advantages of feature extraction of target detection and the characteristics of large proportion of scrap steel types carried by carriages, a scrap steel grading method based on regional segmentation combined with Yolov5 is proposed. Summary of the invention

[0004] The purpose of the present invention is to provide a scrap steel grade judgment model based on regional segmentation and combined with Yolov5. The present invention can not only calculate the scrap steel material type contained in the carriage, but also calculate the proportion of different material types in the whole vehicle. The present invention uses the Mask R-CNN instance segmentation model to segment the scrap steel carriage in the image and remove the pixels around the carriage; secondly, the Yolov5 target detection model is improved, by deleting redundant detection branches, retaining the minimum Feature Map output by the PAN network structure, and then performing 1*1 convolution and Average Pooling on these Feature Maps, and outputting the Feature Map of the 6*6 segmented area, where the number of channels of the Feature Map is the sum of the confidence and the material type category; during model training, the classification loss is calculated by the extracted segmented area features, and the final loss is calculated in combination with the normalized confidence and gradient back propagation; during reasoning, the category and confidence of each area are calculated according to the pre-set segmented area; finally, the scrap steel material type and its proportion contained in the whole vehicle are calculated according to the category and confidence of all segmented areas.

[0005] To achieve the above object, the present invention provides the following technical solutions: a scrap steel grading method based on regional segmentation combined with Yolov5, S1: using the Mask R-CNN instance segmentation model to segment the carriage from the original image and remove the complex background in the original image;

[0006] S2: Transform the region segmentation coordinate frame and map the annotated original image material type label to the segmentation region;

[0007] S3: Improve the Yolov5 network, change the output branch of Yolov5 from three to one, output the Feature Map with the largest receptive field, and then pass through 1*1 convolution and Average Pooling layer to finally output the feature map with H and W both being 6. The number of channels of the feature map is the sum of the number of categories and the number of confidences.

[0008] S4: In the model training phase, the improved Yolov5 output, which includes the features of regional confidence and category, is used for classification loss calculation and confidence loss calculation, and then weighted gradient back propagation; in the inference phase, the category and confidence value of the segmented region are directly output;

[0009] S5: Finally, according to the confidence and area category of different segmented areas, the material types and proportions contained in the whole vehicle scrap steel are calculated.

[0010] Preferably, the following steps are further included before using the Mask R-CNN instance segmentation model to extract the carriage from the original image:

[0011] S11: Collect large-scale scrap steel data on vehicles and produce high-quality training sets, validation sets, and test sets;

[0012] S12: Use gradient descent to train the optimization model on the training set;

[0013] S13: Test the trained model on the test set and select the model with the best effect on the test set; if the test requirements are not met, repeat step S12 until a model that meets the requirements on the test set is trained.

[0014] Preferably, the region segmentation data coordinate frame and label conversion includes the following steps:

[0015] S21: resize the original image;

[0016] S22: mapping the coordinates of the segmented regions for the scrap steel types marked in the original image;

[0017] S23: Calculate the confidence of the category to which each segmented region belongs according to the intersection of the segmented region and the labeled region.

[0018] Preferably, the Yolov5 network is improved, and the improvement process is as follows:

[0019] S31: Remove the Yolov5 detection head and discard the box regression;

[0020] S32: Add a 1*1 convolutional layer and an Average Pooling layer at the end of the PAN network to output a 6*6 feature map with a receptive field of 160*160.

[0021] Preferably, after Yolov5 outputs the regional Feature Map, the post-processing also includes a training phase and an inference phase, and the process is as follows:

[0022] S41: When the model is in the training stage, the category loss and regional confidence sigmoid of the Feature Map are calculated;

[0023] S42: The loss of gradient back propagation is sigmoid, and the confidence is multiplied by the category loss;

[0024] S43: When the model is in the inference stage, the confidence and category corresponding to the Feature Map are directly output.

[0025] Preferably, the material types and proportions of the whole truckload of scrap steel are calculated as follows:

[0026] S51: candidate threshold selection;

[0027] S52: Use the threshold to delete useless areas, calculate the categories of the remaining areas, find out the proportion of the material types contained in each area, and find out the final vehicle grade.

[0028] Compared with the prior art, the invention has the following beneficial effects: the invention proposes a scrap steel grading model based on regional segmentation and Yolov5, after using Mask-RCNN to segment the carriage and the background, the improved version of Yolov5 network extracts a Feature Map with a large receptive field, and then the material type and proportion of the scrap steel contained in the entire carriage are calculated based on the confidence and category of the Feature Map; the invention absorbs the respective advantages of the classification network and the detection network, and has good effects in terms of speed and accuracy;

[0029] The improved Yolov5 regional feature extraction model proposed in the present invention can extract the Feature Map of a fixed field of view. The model using this method combines the advantages of target detection and shields the disadvantage that the detection network can only perform regression and classification based on boxes.

[0030] Compared with directly classifying the original image, it is impossible to obtain the material types and their proportions contained in the carriage; the regional segmentation model proposed in the present invention can not only obtain the material types but also their proportions; the present invention not only does not miss any area, but also focuses the model training on the classification model, and the network model is easier to converge. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the process of the present invention;

[0032] Figure 2 This is a schematic diagram of the improvement of Yolov5 of the present invention;

[0033] Figure 3 It is a characteristic schematic diagram of the segmentation area 1*1*C of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] See also Figures 1 to 3 ,The present invention provides a technical solution: S1: transmitting the video stream collected by the camera to the NVR for video storage and producing the required data set;

[0036] S2 trains the Mask R-CNN instance segmentation model to segment the car from the original image and remove the complex background in the original image;

[0037] S3: Segment and map the scrap steel type labels and coordinate frames of the original image;

[0038] S4: Through the improved Yolov5 output branch, a feature map with a larger receptive field is output, such as Figure 3 As shown;

[0039] S5: After step S4, the model is divided into training phase and inference phase, such as Figure 2 As shown, if it is the training stage, the regional features of the feature map will be classified and loss calculated and multiplied by the confidence value after the corresponding sigmoid;

[0040] S6: After the model is optimally trained, it enters the inference stage. In this stage, the different material types and their proportions contained in the whole vehicle scrap steel are calculated based on the confidence and area categories of different segmented areas.

[0041] After the video stream collected by the camera is transmitted to the NVR for video storage, the production of the required data set also includes the following process:

[0042] S11: Convert the original video to images, and select valid images with certain differences;

[0043] S12: Label the compartment segmentation images and create the segmentation dataset required by Mask-RCNN;

[0044] S13: marking the material type of the segmented carriage data set to produce the data set required by the present invention.

[0045] Training the Mask R-CNN instance segmentation model includes the following steps:

[0046] S21: Parameter selection before training, using gradient descent to train the Mask-RCNN model, using the SGD optimizer; the model input image size is img = (512,512,3);

[0047] S22: Model training. By observing the training loss and validation set loss, several sets of better models are determined. Then, the stability of the model is verified again on the test set to determine the final model. If the effect does not meet the expectations, the training parameters need to be changed and training continues until the expectations are met. If the pre-fetching is still not achieved, the training set is increased in a targeted manner, and the parameters are adjusted and training is iterated to finally meet the segmentation requirements.

[0048] The steps for splitting the coordinate frame and label mapping are as follows:

[0049] S31: resize the original image to 960*960*3;

[0050] S32: Segment the resized image into 6*6 regions;

[0051] S33: Mapping the original image frame coordinates to the segmented area;

[0052] S34: Determine the category to which each segmented region belongs based on the intersection of each segmented region and the labeled region.

[0053] The calculation process can be divided into two types:

[0054] First, when the segmented area contains only one type of material (including grayscale filled areas), the category of the segmented area is the normal mapping of the original image annotation box;

[0055] Second, when the segmented area contains multiple material types, it is necessary to calculate the proportion of each material type in the segmented area, and use the material type with the largest proportion as the material type of this segmented area, but the confidence value needs to be multiplied by the proportion as the confidence value of this segmented area.

[0056] Through the improved Yolov5 output branch, a feature map with a larger receptive field is output. The process includes the following:

[0057] S41: Remove Yolov5 detection head;

[0058] S42: Add a 1*1 convolutional layer and an Average Pooling layer at the end of the PAN network to output a 6*6 feature map with a receptive field of 160*160, such as Figure 2 shown.

[0059] After Yolov5 outputs the regional Feature Map, the post-processing is as follows:

[0060] S51: When the model is in the training stage, the Feature Map outputted in the step, i.e., the category and confidence, is used to calculate the category loss and the regional confidence sigmoid respectively; L p is the classification loss, the formula is as follows:

[0061]

[0062] p i is the model prediction value, y i is the true label, K is the number of categories;

[0063] The confidence is sigmoid, the formula is as follows:

[0064]

[0065] where y i is the confidence value of the segmented area;

[0066] S52: The total training loss is the confidence after sigmoid multiplied by the class classification cross entropy loss, which is calculated as follows:

[0067]

[0068] Where α is the proportion calculated by S34, and is 1 if there is only one type of material.

[0069] According to the confidence of the segmented area and the area category, the types of scrap steel contained in the vehicle and their proportions are calculated. The process is as follows:

[0070] S61: Solve the candidate threshold, the threshold is 1.1 times the confidence value of the original image (960*960) with the left vertex (0, 800) and the lower right point (160, 960) mapped to the Feature Map area;

[0071] S62: After removing useless areas through the threshold, the categories of the remaining areas are calculated, the proportion of the material types contained in each area is calculated, and the final vehicle classification is calculated. The calculation formula is as follows:

[0072]

[0073] Where M is the number of segmented regions without grayscale filling. The category of the kth region, P i For a certain material type.

[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A scrap steel grading method based on regional segmentation and Yolov5, Features: S1: Use the Mask R-CNN instance segmentation model to segment the car from the original image and remove the complex background in the original image; S2: Transform the region segmentation coordinate frame and map the annotated original image material type label to the segmentation region; S3: Improve the Yolov5 network, change the output branch of Yolov5 from three to one, output the Feature Map with the largest receptive field, and then pass through 1*1 convolution and Average Pooling layer to finally output the feature map with H and W both being 6. The number of channels of the feature map is the sum of the number of categories and the number of confidences. The improvement process is as follows: S31: Remove the Yolov5 detection head and discard the box regression; S32: Add a 1*1 convolutional layer and an Average Pooling layer at the end of the PAN network to output a 6*6 feature map with a receptive field of 160*160; S4: In the model training phase, the improved Yolov5 output, which includes the features of regional confidence and category, is used for classification loss calculation and confidence loss calculation, and then weighted for gradient inversion; in the inference phase, the category and confidence value of the segmented region are directly output; S5: Finally, according to the confidence and area category of different segmented areas, the material types and their proportions contained in the whole vehicle scrap steel are calculated.

2. A scrap steel grading method based on regional segmentation and Yolov5 according to claim 1, Features: The Mask R-CNN instance segmentation model is used to extract the car from the original image, and the following steps are included: S11: Collect large-scale scrap steel data on vehicles and produce high-quality training sets, validation sets, and test sets; S12: Use gradient descent to train the optimization model on the training set; S13: Test the trained model on the test set and select the model with the best effect on the test set; if the test requirements are not met, repeat step S12 until a model that meets the requirements on the test set is trained.

3. The scrap steel grading method based on regional segmentation and Yolov5 according to claim 1, Features: The region segmentation coordinate frame conversion and mapping of the annotated original image material type label to the segmentation region include the following steps: S21: resize the original image; S22: mapping the coordinates of the segmented regions for the scrap steel types marked in the original image; S23: Calculate the confidence of the category to which each segmented region belongs according to the intersection of the segmented region and the labeled region.

4. The scrap steel grading method based on regional segmentation and Yolov5 according to claim 1, Features: After Yolov5 outputs the regional Feature Map, it also includes the training phase and the inference phase. The process is as follows: S41: When the model is in the training stage, the category loss and regional confidence sigmoid of the Feature Map are calculated; S42: The loss of gradient back propagation is the confidence after sigmoid multiplied by the category loss; S43: When the model is in the inference stage, the confidence and category corresponding to the Feature Map are directly output.

5. The scrap steel grading method based on regional segmentation and Yolov5 according to claim 1, Features: The calculation process of the material types and proportions of scrap steel in a vehicle is as follows: S51: candidate threshold selection; S52: Use the threshold to delete useless areas, calculate the categories of the remaining areas, find out the proportion of the material types contained in each area, and find out the final vehicle grade.

Images