A high-resolution representation-based water garbage detection method

Abstract

The application relates to a water garbage detection method based on high-resolution characterization. Image data containing water garbage is input into a detection network which is constructed and trained in advance, and a water garbage detection result is output. The detection network comprises a high-resolution feature extraction module, a semantic feature extraction module and a feature information fusion module. The high-resolution feature extraction module processes the image data into a high-resolution feature map and extracts spatial information. The semantic feature extraction module extracts semantic information of the high-resolution feature map. The feature information fusion module fuses the spatial information output by the high-resolution feature extraction module and the semantic information output by the semantic feature extraction module, and outputs fusion information. The multi-scale detection module detects objects of multiple scales according to the fusion information, and obtains the water garbage detection result. Compared with the prior art, the application can obtain a high-resolution feature map with rich spatial and semantic information, and can compensate for the shortcomings of the prior art water garbage detection method and improve the detection performance by combining a multi-branch structure and multi-scale detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and object detection, and in particular to a water garbage detection method based on high-resolution representation. BACKGROUND

[0002] The prior art processes water garbage by manual inspection and manual salvage, but the efficiency is low, the cost is high and the danger is high.

[0003] With the rise of artificial intelligence, water garbage can be identified by a deep learning-based object detection algorithm to realize the automation and intelligentization of water garbage inspection, reduce the consumption of human resources and economic expenses. Compared with the land environment, the quantity and quality of existing data sets in the water field are far from enough, and the water environment has the characteristics of multi-complexity, low safety and instability, so the detection of water garbage is more challenging.

[0004] Existing object detection algorithms can be roughly divided into two categories: two-stage methods based on region recommendation and single-stage methods based on regression ideas. The representative algorithm of the two-stage detection method is the R-CNN series. This method has high detection accuracy, but the calculation amount is large and cannot achieve real-time detection. Single-stage detection methods include YOLO series and SSD algorithm, which directly complete the positioning and classification tasks at one time to realize end-to-end detection. The advantage is that it has real-time detection capability, but the accuracy is slightly lower than that of the two-stage detection algorithm.

[0005] Although water unmanned equipment is constantly improving and its performance is gradually increasing, there is still a lot of room for development in garbage detection in water scenes. It is an urgent need to promote the green development of river channels to improve the detection of water garbage. SUMMARY

[0006] The purpose of the present application is to overcome the defects of the prior art and provide a water garbage detection method based on high-resolution representation. This method can obtain a high-resolution feature map with rich spatial and semantic information, has strong representation ability, and can compensate for the shortcomings of existing water garbage detection methods by combining multi-branch structure and multi-scale detection, thereby improving detection performance.

[0007] The purpose of the present application can be achieved by the following technical solutions:

[0008] The present application provides a water garbage detection method based on high-resolution representation. The image data containing water garbage obtained is input into a detection network constructed and trained in advance, and the water garbage detection result is output.

[0009] The detection network comprises:

[0010] High-resolution feature extraction module: used to process image data into high-resolution feature maps and extract spatial information;

[0011] Semantic feature extraction module: used to extract semantic information from high-resolution feature maps;

[0012] Feature information fusion module: used to fuse the spatial information output by the high-resolution feature extraction module and the semantic information output by the semantic feature extraction module, and output fused information;

[0013] Multi-scale detection module: Used to detect objects at multiple scales based on fused information to obtain the detection results of marine debris.

[0014] Preferably, the high-resolution feature extraction module's processing of image data specifically includes the following steps:

[0015] S201: Downsample the image data to obtain the initial feature map P0;

[0016] S202: Divide the initial feature map P0 equally on the channel to obtain the first feature map P1 and the second feature map P2 respectively. Perform convolution on the first feature map P1 and the second feature map P2 respectively to extract the local context.

[0017] S203: The channels of the convolution result of the first feature map P1 and the second feature map P2 are added together. The channel-added result is normalized by BatchNorm and ReLU function to obtain the third feature map P3.

[0018] S204: The result of the initial feature map P0 after being processed by the CBMA attention mechanism is skipped and connected with the third feature map P3 to generate the fused feature map P4.

[0019] S205: Downsample and extract features from the fused feature map P4 to generate the final high-resolution feature map X1.

[0020] Preferably, in step S203, the expression for generating the third feature map P3 from the first feature map and the second feature map is:

[0021] F 3×3 (·)=γ(B(C 3×3 (·)))

[0022] F 1×1 (·)=γ(B(C 1×1 (·)))

[0023] P3=γ(B(Cat[C 3×3 (P1), C 1×1 (P2)]))

[0024] In the formula, C 3×3 (·) represents a convolution operation with a 3×3 kernel, C 1×1 (·) represents a convolution operation with a 1×1 kernel, B(·) represents batch normalization, and γ(·) represents the ReLU function.

[0025] Preferably, in step S204, the expression for generating the fused feature map P4 is:

[0026] P4 = F 1×1 (M(P0)+P3)

[0027] In the formula, M(·) represents the CBMA attention mechanism.

[0028] Preferably, in step S205, feature extraction is performed on the fused feature map P4 using a 1*1 convolution kernel.

[0029] Preferably, the specific process by which the semantic feature extraction module extracts semantic information from the high-resolution feature map is as follows:

[0030] Multiple convolution operations are performed on the high-resolution feature map to generate multiple feature maps with different resolutions. The feature map with the lowest resolution is convolved with 1*1 to generate the first fused feature map. Multiple deconvolution operations are performed on the first fused feature map. After each deconvolution operation, the fused feature map after the deconvolution operation is fused and concatenated with the feature map with the same resolution as the current fused feature map. The fused feature map output by the last deconvolution operation is the final fused feature map output by the semantic feature extraction module.

[0031] Preferably, the process by which the feature information fusion module fuses spatial information and semantic information is as follows:

[0032] The CBMA mechanism is used to weight the reliable regions of the high-resolution feature map and then perform pixel-level fusion with the final fused feature map to obtain the final fused feature map.

[0033] Preferably, convolution operations are performed on the multiple fusion feature maps generated during the semantic information extraction process in the semantic feature extraction module and the final fusion feature map. The corresponding fusion feature maps are then detected by the detection heads corresponding to each fusion feature map to obtain the final water surface detection result.

[0034] Preferably, the corresponding detection head is obtained by performing a 1*1 convolution operation on each fused feature map.

[0035] Preferably, during the training phase of the detection network, the image data input to the detection network is augmented using a data augmentation strategy, and the augmented image data is used as the training set for the input detection network.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] This invention provides a high-resolution representation-based method for detecting marine debris. By acquiring and fusing spatial and semantic information, it obtains a high-resolution feature map with rich spatial and semantic information, which has a strong representation capability. Combined with multi-branch structure and multi-scale detection, it can make up for the shortcomings of existing marine debris detection methods, thereby improving detection performance. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating a method for detecting marine debris based on high-resolution characterization, as provided in this embodiment.

[0039] Figure 2 for Figure 1 A schematic diagram of the network structure of the high-resolution feature extraction module in the embodiment shown.

[0040] Figure 3 for Figure 1 The network structure diagram of the semantic feature extraction module, feature information fusion module and multi-scale detection module in the embodiment shown is illustrated. Detailed Implementation

[0041] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0042] refer to Figure 1 As shown, this embodiment provides a method for detecting marine debris based on high-resolution characterization, including the following steps:

[0043] S1: Obtain image data containing marine debris, preprocess the images, and divide them into training sets;

[0044] As an optional implementation, preprocessing includes data augmentation. In this embodiment, data augmentation is performed using a data augmentation strategy, i.e., copying targets from the image, pasting them into the corresponding positions, and then using CutMix for data augmentation. This increases the number of targets in the sample and enhances the model's localization ability, thereby improving the image data of marine debris. In this embodiment, the training set uses the FloW_IMG dataset.

[0045] S2: Construct a detection network, input the training set into the constructed detection network, and train the detection network so that the trained detection network can detect water debris in the image data.

[0046] The detection network includes:

[0047] High-resolution feature extraction module: used to process image data into high-resolution feature maps with rich spatial information, forming detailed spatial information;

[0048] As an optional implementation method, refer to Figure 2 As shown, the high-resolution feature extraction module's processing of image data specifically includes the following steps:

[0049] S201: Downsample the image data to obtain the initial feature map P0 (where P0∈R). C×H×W );

[0050] S202: Divide the initial feature map P0 equally on the channel to obtain the first feature map P1 and the second feature map P2 (where P1 and P2 ∈ RC / 2×H×W). Convolve the first feature map P1 and the second feature map P2 respectively, and use the key information obtained by convolution to achieve effective extraction of local context.

[0051] S203: Perform a concatenation operation on the result of the convolution of the first feature map P1 and the second feature map P2 to add the channels together. Normalize the result of the channel addition using BatchNorm and the ReLU function to obtain the third feature map P3 (where P3∈R2C×H / 2×W / 2). The expression for generating the third feature map P3 from the first feature map and the second feature map is as follows:

[0052] F 3×3 (·)=γ(B(C 3×3 (·)))

[0053] F 1×1 (·)=γ(B(C 1×1 (·)))

[0054] P3=γ(B(Cat[C 3×3 (P1), C 1×1 (P2)]))

[0055] In the formula, C 3×3 (·) represents a convolution operation with a 3×3 kernel, C 1×1 (·) represents a convolution operation with a 1×1 kernel, B(·) represents batch normalization, and γ(·) represents the ReLU function.

[0056] S204: The result of processing the initial feature map P0 through the CBMA attention mechanism is skipped and connected to the third feature map P3 to fuse all information, obtaining detailed spatial information beneficial for small object detection, i.e., the fused feature map P4. The expression for generating the fused feature map P4 is:

[0057] P4 = F 1×1 (M(P0)+P3)

[0058] In the formula, M(·) represents the CBMA attention mechanism.

[0059] S205: The fused feature map P4 is downsampled and further feature extracted. This feature extraction uses a 1*1 convolution kernel to reduce computation, ultimately obtaining a high-resolution feature map X1 rich in spatial information. The expression for generating the high-resolution feature map X1 is:

[0060] X1 = F 1×1 (P4)

[0061] Semantic feature extraction module: used to extract semantic information from high-resolution feature map X1, with an additional pyramid structure;

[0062] The specific process by which the semantic feature extraction module extracts the semantic information of the high-resolution feature map X1 is as follows:

[0063] Multiple convolution operations are performed on the high-resolution feature map X1 to generate multiple feature maps with different resolutions. The feature map with the lowest resolution is convolved with 1*1 to generate the first fused feature map. Multiple deconvolution operations are performed on the first fused feature map. After each deconvolution operation, the fused feature map after the deconvolution operation is fused and connected with the feature map with the same resolution as the current fused feature map. The fused feature map output by the last deconvolution operation and fusion connection is the final fused feature map output by the semantic feature extraction module.

[0064] In this embodiment, reference Figure 3 As shown, after multiple convolution operations on the high-resolution feature map X1, a second feature map X2, a third feature map X3, and a fourth feature map X4 with gradually decreasing resolution and size are generated sequentially. The fourth feature map X4 is then convolved with a 1*1 to generate a first fused feature map F0. After deconvolution, the first fused feature map F0 is fused and connected with the third feature map X3 to generate a second fused feature map F1. After deconvolution, the second fused feature map F1 is fused and connected with the second feature map X2 to generate a third fused feature map F2. After deconvolution, the third fused feature map F2 is fused and connected with the high-resolution feature map X1 to generate the final fused feature map X5.

[0065] Feature information fusion module: used to fuse the spatial information output by the high-resolution feature extraction module and the semantic information output by the semantic feature extraction module, and output fused information;

[0066] As an optional implementation, the credible region of the high-resolution feature map X1 is weighted using the CBMA mechanism and then fused with the final fused feature map X5 at the pixel level to obtain the final fused feature map F3.

[0067] Multi-scale detection module: Used to detect objects at multiple scales based on fused information to obtain the detection results of marine debris.

[0068] As an optional implementation, after performing a 1*1 convolution operation on the multiple fusion feature maps generated during the semantic information extraction process of the semantic feature extraction module and the final fusion feature map F3, the fusion feature map is detected by the detection head corresponding to each fusion feature map to obtain the final water surface detection result.

[0069] In this embodiment, the sizes of the first fusion feature map F0, the second fusion feature map F1, the third fusion feature map F2, and the final fusion feature map F3 are 20*20, 40*40, 80*80, and 160*160, respectively. They are detected by four detection heads, Head3, Head2, Head1, and Head0, respectively, to obtain the final waterborne detection result.

[0070] S3: Input the image data of the marine debris to be detected into the trained detection network to obtain the marine debris segmentation result of the image data.

[0071] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for detecting marine debris based on high-resolution characterization, characterized in that, The acquired image data containing marine debris is input into a pre-built and trained detection network, which outputs the marine debris detection results. The detection network includes: High-resolution feature extraction module: used to process image data into high-resolution feature maps and extract spatial information; Semantic feature extraction module: used to extract semantic information from high-resolution feature maps; Feature information fusion module: used to fuse the spatial information output by the high-resolution feature extraction module and the semantic information output by the semantic feature extraction module, and output fused information; Multi-scale detection module: used to detect objects at multiple scales based on fused information to obtain the detection results of marine debris; The high-resolution feature extraction module processes image data through the following steps: S201: Downsample the image data to obtain the initial feature map P0; S202: Divide the initial feature map P0 equally on the channel to obtain the first feature map P1 and the second feature map P2 respectively. Perform convolution on the first feature map P1 and the second feature map P2 respectively to extract the local context. S203: The channels of the convolution result of the first feature map P1 and the second feature map P2 are added together. The channel-added result is normalized by BatchNorm and ReLU function to obtain the third feature map P3. S204: The result of the initial feature map P0 after being processed by the CBMA attention mechanism is skipped and connected with the third feature map P3 to generate the fused feature map P4. S205: Downsample and extract features from the fused feature map P4 to generate the final high-resolution feature map X1; The specific process by which the semantic feature extraction module extracts semantic information from the high-resolution feature map is as follows: Multiple convolution operations are performed on the high-resolution feature map to generate multiple feature maps with different resolutions. The feature map with the lowest resolution is convolved with 1*1 to generate the first fused feature map. Multiple deconvolution operations are performed on the first fused feature map. After each deconvolution operation, the fused feature map after the deconvolution operation is fused and concatenated with the feature map with the same resolution as the current fused feature map. The fused feature map output by the last deconvolution operation is the final fused feature map output by the semantic feature extraction module.

2. The method for detecting marine debris based on high-resolution characterization according to claim 1, characterized in that, In step S203, the expression for generating the third feature map P3 from the first feature map and the second feature map is: In the formula, This represents a convolution operation with a 3×3 kernel. B represents a convolution operation with a 1×1 kernel. Represents batch normalization. This represents the ReLU function.

3. The method for detecting marine debris based on high-resolution characterization according to claim 1, characterized in that, In step S204, the expression for generating the fused feature map P4 is: In the formula, M(•) represents the CBMA attention mechanism.

4. The method for detecting marine debris based on high-resolution characterization according to claim 1, characterized in that, In step S205, feature extraction is performed on the fused feature map P4 using a 1*1 convolution kernel.

5. The method for detecting marine debris based on high-resolution characterization according to claim 1, characterized in that, The process by which the feature information fusion module fuses spatial and semantic information is as follows: The CBMA mechanism is used to weight the reliable regions of the high-resolution feature map and then perform pixel-level fusion with the final fused feature map to obtain the final fused feature map.

6. The method for detecting marine debris based on high-resolution characterization according to claim 5, characterized in that, Convolutional operations are performed on multiple fusion feature maps generated during the semantic information extraction process in the semantic feature extraction module, as well as the final fusion feature map. The corresponding fusion feature maps are then detected by the detection heads corresponding to each fusion feature map to obtain the final water surface detection results.

7. The method for detecting marine debris based on high-resolution characterization according to claim 6, characterized in that, The corresponding detection head is obtained by performing a 1*1 convolution operation on each fused feature map.

8. The method for detecting marine debris based on high-resolution characterization according to claim 1, characterized in that, During the training phase of the detection network, the image data input to the detection network is augmented using data augmentation strategies, and the augmented image data is used as the training set for the input detection network.

Images