A method for identifying floating garbage on the water surface based on an improved convolutional neural network

Abstract

The present invention relates to the field of image recognition technology, and in particular to a method for identifying floating garbage on a water surface based on an improved convolutional neural network, comprising S1, expanding a dataset of floating garbage images on a water surface; S2, using ResNet50 as a feature extraction network layer, adopting a pyramid anchor frame to generate an anchor frame, screening out multiple candidate target frames suspected of being garbage through Soft‑NMS, and classifying and regressing the targets based on the classification discriminant map mapped by the candidate frames using ROI Pooling; S3, training the improved Faster RCNN network model; and S4, testing a test set. The present invention proposes a pyramid anchor frame generation method based on the original Faster RCNN to improve the detection accuracy when the illumination reflection is uneven; in view of the fact that the target size is too small, a classification discriminant map is proposed to improve the detection accuracy of small targets, and Soft‑NMS is adopted to improve the detection accuracy of occluded targets.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method for identifying floating garbage on a water surface based on an improved convolutional neural network. Background Art

[0002] Currently, the removal of floating garbage on the water surface mainly relies on manual salvage, but this method has problems such as high cost and slow cleaning. Research has shown that combining surface robots with real-time video monitoring technology can effectively improve its automation level and reduce costs. When it comes to detecting floating garbage on the water surface, traditional detection methods mainly include background modeling and optical flow methods. However, due to problems such as uneven light reflection on the water surface, complex water surface scenes, and large differences in target size that interfere with detection, the above algorithms are difficult to adapt to various environmental changes.

[0003] The development of computer hardware and computing power has fueled the growth of convolutional neural networks (CNNs), and CNN-based object detection algorithms have gradually emerged. Compared with traditional detection algorithms, CNN-based object detection algorithms have better feature representation and generalization capabilities. CNN-based object detection algorithms are categorized into two types: one-stage algorithms, such as YOLOv2, YOLOv3, and SSD. These algorithms are referred to as one-stage algorithms, while two-stage algorithms, such as Faster RCNN, FastRCNN, and RCNN, are referred to as two-stage algorithms. Faster RCNN is a classic object detection algorithm. Its backbone network uses VGG16 and proposes a region proposal network (RPN). This method generates a series of candidate boxes based on feature maps and finally obtains predictions through ROI pooling. This method achieves high detection accuracy for general objects, but it still performs poorly for objects with occlusion, uneven illumination, or small size in the image. Summary of the Invention

[0004] The technical problem to be solved by the present invention is as follows: the present invention improves Faster RCNN and proposes a pyramid anchor frame generation method to address the problem of uneven light reflection; introduces the Soft-NMS algorithm to address the problem of mutual occlusion of targets; and proposes a classification discriminant graph to address the problem of target size being too small. This method can effectively detect floating garbage on the water surface and achieve high detection accuracy.

[0005] The technical solution adopted by the present invention is: a method for identifying floating garbage on the water surface based on an improved convolutional neural network comprises the following steps:

[0006] S1: Expand the floating garbage image dataset and divide it into three parts: training set, validation set and test set;

[0007] The training set accounts for 80%, the validation set accounts for 10%, and the test set accounts for 10% of the dataset;

[0008] S2: ResNet50 is used as the feature extraction network layer. After the floating garbage image is extracted through the feature layer, a feature map with better semantic information is generated. The RPN network then generates anchor frames using a pyramid-like anchor frame generation method. The Soft-NMS algorithm then filters out a series of target candidate frames that may be floating garbage on the water surface. The candidate frames extracted by the RPN network are then mapped to the corresponding classification discriminant map. ROI Pooling performs classification and regression based on the classification discriminant map mapped by the candidate frames.

[0009] Furthermore, the pyramid anchor box generation method includes the following steps:

[0010] S21, using the annotation box to generate a binary label map for each image of floating garbage on the water surface, wherein the position of the annotation box containing the target is coded as 1, and the rest is coded as 0, that is, the foreground part is marked as 1, and the background part is marked as 0;

[0011] S22, the coordinate position information of the annotation frame (x g ,y g ,w g ,h g ) is mapped to different semantic information feature map scales to obtain the corresponding coordinate position information (x' g ,y' g ,w' g ,h' g ), where w g is the width of the annotation box, h g is the height of the annotation box, x g is the horizontal coordinate of the center of the annotation box, y g is the vertical coordinate of the center of the annotation box, w' g is the width of the annotation box after mapping to the semantic information feature map, h' g is the height of the annotation box after mapping to the semantic information feature map, x' g y' is the horizontal coordinate of the center of the annotation box after it is mapped to the semantic information feature map. g The vertical coordinate of the center of the annotation box after it is mapped to the semantic information feature map, and then the annotation box is divided into three categories:

[0012] 1. Internal area CA = (x' g ,y' g ,σ1w' g ,σ1h' g ) defines the central area of the annotation box. This part is the most central area of the annotation box, and the anchor box generated with this part as the center belongs to the positive sample;

[0013] 2. Buffer area IA = (x' g ,y' g ,σ2w' g ,σ2h' g ) is a large area. If the anchor box is generated with this part as the center, its IoU is relatively low, so this part is ignored and used as a buffer area;

[0014] 3. External area OA: This area is the area outside the feature map except CA and IA. The anchor box generated with this area as the center is treated as a negative sample.

[0015] Among them, σ1 and σ2 are the coefficients of the inner area and buffer area respectively;

[0016] S23, comparing the anchor frame and the annotation frame, and predicting the shape of the floating garbage on the water surface based on the IoU between the anchor frame and the annotation frame compared with it;

[0017] In addition, considering that the sizes of annotation boxes are different and vary greatly, three levels of anchor boxes are used, as shown in the following table:

[0018] Table 1 Anchor box levels

[0019]

[0020] The aspect ratios of the three anchor boxes are 1:1, 1:2, and 2:1 respectively. The aspect ratios of the three levels are unified, and the formula is as follows:

[0021]

[0022] Among them, w g is the width of the annotation box, h g is the height of the annotation box, l is the level of anchor box generation;

[0023] Soft-NMS then filters out a series of target candidate frames that may be floating garbage on the water surface. Then, the candidate frames extracted by the RPN network are mapped to the corresponding classification discriminant graph.

[0024] Further classification discriminant diagram includes the following steps:

[0025] The fifth upsampling layer of the S241 and ResNet50 network feature layers and the fourth layer feature map are fused and passed to ROIPooling. The proposal box generated after RPN processing is scaled up and mapped to the fused feature map to form a classification discriminant map.

[0026] S242, ROI Pooling is based on the classification discriminant map, after maximum pooling, and then classification and regression operations;

[0027] S3: Load the training set and validation set of floating garbage image data into the improved Faster RCNN network for model training, and then save the trained result model. The network training result model is the joint training result of the RPN network and the Fast RCNN network;

[0028] S4: Use the trained result model to test the test set to obtain the test results and complete the detection of floating garbage on the water surface;

[0029] To test the test set images, the images of the test set of floating garbage on the water surface are imported into the training result model. The output image can mark the location of the floating garbage on the water surface and display the category and accuracy of the floating garbage on the water surface, thereby completing the detection of floating garbage targets on the water surface.

[0030] The beneficial effects of the present invention are:

[0031] 1. A pyramid-based anchor frame generation method is proposed based on the original Faster RCNN to improve detection accuracy when the illumination reflection is uneven;

[0032] 2. As the target size is too small, a classification discriminant map is proposed to improve the accuracy of small target detection, and Soft-NMS is used to replace NMS. After adjusting the parameters through the iterative threshold shrinkage algorithm, the accuracy of occluded target detection is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the method for identifying floating garbage on the water surface based on the improved Faster RCNN of the present invention;

[0034] Figure 2 This is a ResNet50 structure diagram of the surface floating garbage recognition method based on the improved Faster RCNN of the present invention;

[0035] Figure 3 This is a classification discriminant graph of the method for identifying floating garbage on the water surface based on the improved Faster RCNN of the present invention;

[0036] Figure 4 This is a schematic diagram of the pyramid anchor frame generation method of the present invention based on the improved Faster RCNN floating garbage recognition method;

[0037] Figure 5 This is a partial result display of the method for identifying floating garbage on the water surface based on the improved Faster RCNN of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore only shows the components related to the present invention.

[0039] like Figure 1 As shown, a method for identifying floating garbage on the water surface based on an improved convolutional neural network includes the following steps:

[0040] S1. The collected images of floating garbage on the water surface are folded and rotated for image expansion, and divided into three parts: training, validation, and testing, of which 80% is the training set, 10% is the validation set, and 10% is the test set.

[0041] S2. Use ResNet50 as the feature extraction network layer of Faster RCNN. The ResNet50 network is as follows Figure 2 As shown in the figure, the floating garbage image is resized to a uniform 800×600 size. The floating garbage image is then extracted through the feature layer to generate a feature map with better semantic information. The RPN network then generates anchor frames using a pyramid-like anchor frame generation method. The anchor frame is generated with the annotation frame as the center, which can effectively avoid interference from the target's surrounding environment, such as uneven light reflection.

[0042] The specific steps of the pyramid anchor frame generation method are as follows (such as Figure 3 shown):

[0043] S21. Generate a binary label map for each image using the annotation box, where the position of the annotation box containing the target is encoded as 1, and the rest are encoded as 0, that is, the foreground part is marked as 1 and the background part is marked as 0.

[0044] S22, the coordinate position information of the annotation frame (x g ,y g ,w g ,h g ) is mapped to different feature map scales to obtain the corresponding coordinate position information (x' g ,y' g ,w' g ,h' g ), where x g is the horizontal coordinate of the center of the annotation box, y g is the vertical coordinate of the center of the annotation box, σ1 and σ2 are coefficients, and then the annotation box is divided into three categories:

[0045] 1. Internal area CA = (x' g ,y' g ,σ1w' g ,σ1h' g) defines the central area of the annotation box. This part is the most central area of the annotation box, and the anchor box generated with this part as the center belongs to the positive sample;

[0046] 2. Buffer area IA = (x' g ,y' g ,σ2w' g ,σ2h' g ) is a large area. If the anchor box is generated with this part as the center, its IoU is relatively low, so this part is ignored and used as a buffer area;

[0047] 3. External area OA: This area is the area outside the feature map except CA and IA. The anchor box generated with this area as the center is treated as a negative sample.

[0048] S23, compare the anchor box and the annotation box, and make a prediction based on the IoU between the anchor box and the annotation box compared with it;

[0049] Where w g is the width of the annotation box, h g is the height of the annotation box, x g is the horizontal coordinate of the center of the annotation box, y g is the vertical coordinate of the center of the annotation box, σ1 and σ2 are the coefficients of the inner area and the buffer area, respectively. In this embodiment, when σ1 = 0.18 and σ2 = 0.54, the detection accuracy is the highest;

[0050] In addition, considering that the sizes of annotation boxes are different and vary greatly, the present invention adopts three levels of anchor boxes, as shown in Table 1 below:

[0051] Table 1 Anchor box levels

[0052]

[0053] The aspect ratios of the three anchor boxes are 1:1, 1:2, and 2:1 respectively; the aspect ratios of the three levels are unified, and the formula is expressed as follows:

[0054]

[0055] Among them, w g is the width of the annotation box, h g is the height of the annotation box, l is the level of anchor box generation;

[0056] Then, Soft-NMS is used to screen out a series of target candidate frames that may be floating garbage on the water surface. Since NMS does not process prediction frames with high overlap well, Soft-NMS can effectively improve NMS's processing problem of high overlap. Soft-NMS replaces NMS. The NMS algorithm is shown in (2), where M is the prediction frame with the highest score, is the prediction frame that overlaps with the current prediction frame, and is the set processing threshold. The main idea is that once the IoU exceeds a certain threshold, the corresponding value is set to zero.

[0057]

[0058] The Soft-NMS algorithm is shown in formula (3). Its main idea is that once the IoU exceeds a certain threshold, it is attenuated using the corresponding formula instead of directly setting the corresponding value to zero:

[0059]

[0060] Where s i is the target score, M is the prediction box with the highest score, and b i is the prediction box that overlaps with the current prediction box, N t The processing threshold is set.

[0061] This paper optimizes and adjusts the parameters of Soft-NMS. After adjusting the parameters of the iterative threshold shrinkage algorithm, the N t The threshold has the best effect in the parameter range of 0.45-0.65. If it exceeds this range, the average detection accuracy will be significantly reduced.

[0062] Then, the candidate boxes extracted based on the RPN network are mapped to the corresponding classification discriminant graph, which can greatly improve the accuracy of small target detection;

[0063] The specific implementation of the classification discriminant graph is as follows:

[0064] S241, such as Figure 4 As shown in the figure, the fifth upsampling layer of the ResNet50 network feature layer and the fourth layer feature map are fused and then passed to ROI Pooling. The proposal box generated after RPN processing is scaled up and mapped to the fused feature map to form a classification discriminant map.

[0065] S242, ROI Pooling is based on the classification discriminant map, after maximum pooling, and then classification and regression operations;

[0066] S3. Load the training set and validation set of floating garbage image data into the improved Faster RCNN network for model training. Then save the trained result model and use the saved result model to test the images of the test set to obtain the test results, thus completing the detection of floating garbage on the water surface.

[0067] S4, testing the test set images, is to import the floating garbage test set images into the training result model, the output image can mark the location of the floating garbage part and show the category and accuracy of the floating garbage on the water surface, such as Figure 5 Shown are the detection results of different types of floating garbage on the water surface;

[0068] In order to evaluate the effectiveness of the present invention in detecting floating garbage on the water surface, the experiment uses precision and recall to measure the effectiveness of the method. The calculation formulas are shown in formulas (4) and (5).

[0069]

[0070]

[0071] In the formula, TP (True Positive) represents the positive sample predicted by the model as positive; FP (False Positive) represents the positive sample predicted by the model as negative; FN (False Negative) represents the positive sample predicted by the model as negative.

[0072] The proposed method and the Faster RCNN (ResNet50), YOLOX, and SSD methods were used to test this dataset respectively, and the average detection accuracy and recall rate were recorded and summarized as shown in Table 2:

[0073] Table 2 Results of recall and precision under different models

[0074]

[0075] As can be seen from Table 2, the proposed method improves by 4 percentage points compared with Faster RCNN (ResNet50), improves by 7.3% compared with the SSD algorithm, and improves by 3.6% compared with the latest YOLO series algorithm - YOLOX, with an overall recognition rate of 86.4%.

[0076] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A method for identifying floating garbage on the water surface based on an improved convolutional neural network, characterized in that: The following steps are involved: S1. Expand the dataset of floating garbage images on the water surface and divide the dataset into training set, validation set and test set; S2. After the image of floating garbage on the water surface is extracted through the ResNet50 feature layer, a semantic information feature map is generated. The RPN network uses a pyramid anchor frame generation method to generate anchor frames. The Soft-NMS algorithm then screens out multiple target candidate frames suspected of floating garbage on the water surface. The candidate frames extracted by the RPN network are then mapped to the corresponding classification discriminant map. ROIPooling classifies and regresses the target based on the classification discriminant map mapped by the candidate frames, resulting in an improved FasterRCNN network model. The pyramid anchor box generation methods include: S21, using the annotation box to generate a binary label map for each image of floating garbage on the water surface, wherein the position of the annotation box containing the target is coded as 1, and the rest are coded as 0; S22, the coordinate position information of the annotation frame (x g ,y g ,w g ,h g ) is mapped to different semantic information feature map scales to obtain the corresponding coordinate position information (x' g ,y' g ,w' g ,h' g ), where x g and x' g are the horizontal coordinates of the center of the annotation box and the annotation box after being mapped to the semantic information feature map, y g and y' g are the annotation box and the vertical coordinate of the center after the annotation box is mapped to the semantic information feature map, w g and w' g are the width of the annotation box and the width of the annotation box after mapping to the semantic information feature map, h g and h' g are the annotation box and the height of the annotation box after being mapped to the semantic information feature map; The annotation box is divided into an inner area CA = (x' g ,y' g ,σ1w' g ,σ1h' g ), buffer area IA=(x' g ,y' g ,σ2w' g ,σ2h' g ) and the outer area OA; where σ1 and σ2 are the coefficients of the inner area and the buffer area respectively; The anchor boxes are divided into three levels: level 1, 32×32, 64×64, 128×128; level 2, 64×64, 128×128, 256×256; and level 3, 128×128, 256×256, 512×512. The aspect ratios of the three levels of anchor boxes are 1:1, 1:2, and 2:1, respectively. The formula is expressed as follows: Among them, l is the anchor box generation level; S23, comparing the anchor frame and the annotation frame, and predicting the shape of the floating garbage on the water surface based on the IoU between the anchor frame and the annotation frame; S3. Send the training set and validation set of floating garbage image data to the improved Faster RCNN network for training, and save the trained result model; S4. Use the trained result model to test the test set images to obtain the test results and complete the detection of floating garbage on the water surface.

2. The method for identifying floating garbage on water surface based on improved convolutional neural network according to claim 1 is characterized in that: The classification discriminant graph implementation includes: The fifth upsampling layer of the S241 and ResNet50 network feature layers and the fourth layer feature map are fused and passed to ROI Pooling. The proposal box generated after RPN processing is scaled up and mapped to the fused feature map to form a classification discriminant map. S242, ROIPooling is based on the classification discriminant map, undergoes maximum pooling, and then performs classification and regression operations.

Images