A water surface garbage recognition method based on deep learning
By improving the U-Net network using Laplacian convolution and depthwise separable convolution combined with channel attention mechanism for water surface debris recognition, the problems of noise interference and large model size are solved, thereby improving recognition accuracy and efficiency.
Patent Information
- Application Number
- CN202211285444.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-20
- Publication Date
- 2026-01-06
- Estimated Expiration
- 2042-10-20
AI Technical Summary
Existing deep learning methods for identifying debris on the water surface suffer from problems such as noise interference affecting the identification results and excessively large network model size, which consumes a lot of storage space.
We employ Laplacian convolution to extract image features, and combine depthwise separable convolution and channel attention mechanisms to improve the U-Net network structure, suppress noise, and reduce the number of model parameters.
It improves the accuracy and efficiency of surface debris identification, reduces the consumption of computing resources, and achieves efficient automatic identification of surface debris.
Smart Images
Figure CN115601627B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a deep learning-based method for identifying debris on the water surface. Background Technology
[0002] Surface debris identification is related to water environment protection and pollution control. With increasing environmental awareness, water pollution is a growing concern worldwide. Floating debris, formed by various types of litter, is one of the main pollutants in water bodies. Various plastic products and aluminum cans cause significant damage to water quality and the ecosystem. Cleaning up surface debris is an important task for environmental protection. The invention of surface debris identification methods can facilitate automatic monitoring of water pollution or aid in the automatic retrieval of surface debris by unmanned vessels.
[0003] Current methods for identifying debris on the water surface mainly include those based on digital image processing and those based on deep learning. Traditional methods include threshold segmentation, Gabor filters, histograms, and random structure forests. In recent years, with the development of artificial intelligence, deep learning methods have begun to be applied to the task of identifying debris on the water surface. Deep learning-based methods for identifying debris on the water surface mainly utilize deep neural networks for tasks such as image classification, object detection, and semantic segmentation.
[0004] In the task of identifying debris on the water surface, although deep learning methods have improved the accuracy of identification compared with traditional methods, they still have many shortcomings. On the one hand, water surface scenes have a lot of noise interference such as reflections and shadows, which can easily form false targets and affect the identification results. On the other hand, the network structure models of existing deep learning methods are large in scale, occupy a lot of storage space, and consume a lot of computing resources, which makes them difficult to apply in practice. Summary of the Invention
[0005] To address the shortcomings of existing algorithms, the technical solution adopted in this invention is: a deep learning-based method for identifying surface debris in water, comprising the following steps:
[0006] Step 1: Cropping the images in the water surface debris recognition dataset and dividing them into training, validation, and test sets;
[0007] Step 2: Use Laplacian convolution to obtain a preliminary denoised feature map F containing edge information. L , feature map F L After being added to the input image I through residual connections, features are further extracted through multiple convolutions, and the suppression weight ω is obtained through the Sigmoid activation function. Each channel of the input image I is multiplied element-wise with the suppression weight (1-ω) to obtain a weighted feature map. The weighted feature map is then output to the subsequent encoder-decoder.
[0008] Furthermore, a preliminary denoised feature map F containing edge information is obtained using Laplacian convolution. L The formula is:
[0009] F L =Conv L (I), F L ∈R H×W×3 (1)
[0010] Among them, Conv L (·) represents Laplace convolution.
[0011] Furthermore, the feature map F L After being added to the input image I through residual connections, features are further extracted through multiple convolutions, and the suppression weight ω is obtained through the Sigmoid activation function.
[0012] The calculation process for the suppression weight ω is as follows:
[0013] ω=σ(Conv(I+F L )), ω∈R H×W×1 (3)
[0014] Conv(·) contains one 3×3 convolution, two 7×7 convolutions, and one 1×1 convolution. Each convolution contains Batch Normalization and ReLU activation functions.
[0015] Furthermore, the process of obtaining a weighted feature map by element-wise multiplying each channel of the input image I with the subtracted suppression weight (1-ω) is as follows:
[0016] Conv(f)=Conv 1×1 (Conv 7×7 (Conv 7×7 (Conv 3×3 (f)))) (4)
[0017] Among them, Conv 3×3 (·), Conv 7×7 (·), Conv 1×1 (·) represent 3×3 convolution, 7×7 convolution, and 1×1 convolution, respectively;
[0018] The Laplacian feature map is processed by a 3×3 convolution with an expanded channel count of 16, followed by a 7×7 convolution to extract information from the feature map, while the number of channels remains constant at 16. Finally, a 1×1 convolution is used to integrate the channel information to generate the suppression weight ω.
[0019] The output of the weighted feature map is:
[0020]
[0021] in, For Hadamard products, I i Let I be the i-th channel, Concat(·) is the feature map concatenation, and O is the module output.
[0022] Step 3: Improve the U-Net network by using two depthwise separable convolutions to form a DSC module in the deep layers and Res modules in the shallow layers.
[0023] Furthermore, specifically including:
[0024] First, the image is processed in the editor through a Res module, then through three DSC modules. Then, in the decoder, upsampling is performed through a symmetrical process, that is, through three DSC modules and one Res module in sequence, and finally through a 1×1 convolution to reduce the number of channels.
[0025] Furthermore, the calculation formula for the DSC module is as follows:
[0026] DSC(f) = Conv 1×1 (DW(Conv 1×1 (DW(f)))) (9)
[0027] Where DW(·) is a depthwise convolution, and Conv 1×1 (·) represents pointwise convolution, and f is the feature map.
[0028] Step 4: Use depthwise separable convolution with attention mechanism. The channel attention mechanism first performs global average pooling on the feature map space to compress spatial dimensions such as length and width, while retaining the channel dimension, to obtain a one-dimensional feature vector. Then, one-dimensional convolution is used to generate channel weights. Finally, the channel weights are multiplied with the original feature map. An ECA module is added after the Res module and DSC module of the encoder. The decoder uses a depthwise separable convolution module.
[0029] Furthermore, the calculation method for the channel attention mechanism is as follows:
[0030] ω c =σ(Conv 1d (Ave(f))), f∈R H×w×C ω c ∈R 1×1×C (11)
[0031]
[0032] Where f is the input feature map, f′ is the output feature map, Ave(·) is the global average pooling, and Conv 1d(·) represents a one-dimensional convolution, σ(·) is the Sigmoid activation function, and ω c This represents the channel weight.
[0033] Furthermore, the decoder uses a depthwise separable convolutional module, which includes a deep three-layer DSC module and a shallow one-layer Res module. The feature map passes through the deep module and the shallow module in sequence. The feature map of each layer is concatenated with the feature map of the corresponding encoder through skip connections. Transposed convolution upsampling increases the resolution by 2 times. Four upsampling operations gradually restore the resolution to the original size. Finally, the feature map reduces the number of channels through a 1×1 convolution.
[0034] Step 5: Train the network model under the deep learning framework to obtain the network parameter values, input the image of water surface debris into the network, and output the image segmentation result of water surface debris.
[0035] The beneficial effects of this invention are:
[0036] 1. Learnable Laplacian convolution is used to extract image features. Convolution is used to process the edge information of the Laplacian feature map to suppress noise and enhance the recognition accuracy of floating objects on the water surface.
[0037] 2. In the encoder and decoder parts, depthwise separable convolutions are used to replace traditional two-layer convolution operations, reducing the number of parameters in the network model;
[0038] 3. The channel attention mechanism is used in the encoding part, which enhances the ability to extract channel information and improves the performance of the method. Attached Figure Description
[0039] Figure 1 This is a flowchart of the deep learning-based water surface debris identification method of the present invention;
[0040] Figure 2 This is a schematic diagram of the reflection suppression module of the present invention;
[0041] Figure 3 This is a schematic diagram of the residual module of the present invention;
[0042] Figure 4 This is a schematic diagram of the depth-separable convolution module of the present invention;
[0043] Figure 5 This is a schematic diagram of the channel attention mechanism module of the present invention. Detailed Implementation
[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.
[0045] like Figure 1 As shown, a deep learning-based method for identifying debris on the water surface includes the following steps:
[0046] Step 1: Cropped the water surface debris recognition dataset into 416×416 images and divided it into three parts: training, validation, and testing.
[0047] Step 2: To address the issue of significant reflected noise interference in water surface scenarios, a reflection suppression module is designed to suppress the noise.
[0048] A reflection suppression module is designed at the beginning of the network, such as Figure 2 To suppress reflection noise, traditional denoising methods require manually setting hyperparameters such as thresholds, and are prone to damaging noise-free images. Using convolutional neural networks, the network can distinguish the features of the target and noise, and perform adaptive noise filtering on the image to achieve lossless denoising.
[0049] The Laplacian operator is a second-order partial derivative operator that can remove low-frequency noise and extract edge information. In water scenes, low-frequency noise refers to blurred water reflections, and the Laplacian operator can easily distinguish the target from the noise. This invention utilizes learnable Laplacian convolution to extract image features, and uses convolution to further process the edge information of the Laplacian feature map to distinguish the target and suppress noise. The feature map obtained by Laplacian convolution has three channels, containing edge information and the denoising result.
[0050] F L =Conv L (I), F L ∈R H×W×3 (1)
[0051] Among them, Conv L (·) denotes a Laplace convolution, which is a 3×3 convolution, denoted by k. L initialization:
[0052]
[0053] During the training process, k L As trainable parameters, they are optimized during gradient backpropagation to better complement network feature extraction.
[0054] The feature map F obtained by Laplacian convolution L After being added to the input image I via residual connections, features are further extracted through four convolutions. These convolutions combine information from the original image with edge information from the Laplacian feature map to identify debris on the water surface. The suppression weight ω is obtained through the Sigmoid activation function σ.
[0055] ω=σ(Conv(I+F L)), ω∈R H×W×1 (3)
[0056] Conv(·) consists of one 3×3 convolution, two 7×7 convolutions, and one 1×1 convolution, each containing Batch Normalization and ReLU activation functions. The Laplacian feature map is refined through the 3×3 convolution, which expands the number of channels to 16. Subsequently, a 7×7 convolution with a larger receptive field is used to extract information from the feature map, while maintaining the same number of channels. Finally, a 1×1 convolution integrates the channel information to generate the suppression weight ω. The process is as follows:
[0057] Conv(f)=Conv 1×1 (Conv 7×7 (Conv 7×7 (Conv 3×3 (f)))) (4)
[0058] Among them, Conv 3×3 (·), Conv 7×7 (·), Conv 1×1 (·) represent 3×3 convolution, 7×7 convolution, and 1×1 convolution, respectively;
[0059] Each channel of the input image I is element-wise multiplied with the subtracted suppression weights (1-ω) to obtain a weighted feature map. This weighted feature map is then output to the subsequent encoder-decoder for further identification of surface debris. Since the Laplacian feature map contains edge information, which aids in fine segmentation, it is used to... Figure 1 Start output:
[0060]
[0061] in, For Hadamard products, I i Let I be the i-th channel, Concat(·) is the feature map concatenation, and O is the module output.
[0062] Step 3: Construct the U-Net network. The encoding part extracts features of debris on the water surface, and the decoding part restores the image and outputs the final feature map.
[0063] The main structure of the network is similar to U-Net, consisting of downsampling and upsampling parts. The downsampling process reduces the feature map resolution, increases the number of channels, and extracts deep semantic information. The upsampling process reduces the number of feature map channels, restores the resolution, and generates the result. The lightweight network of this invention is built using depthwise separable convolution, which includes two parts: depthwise convolution and pointwise convolution. In this invention, each kernel of the depthwise convolution operates on only one input feature map, while the pointwise convolution uses a 1×1 kernel to fuse channel information.
[0064] The network uses depthwise separable convolutions in its deeper layers. Unlike ordinary convolutions, depthwise separable convolutions utilize depthwise convolutions and pointwise convolutions to process spatial and channel information respectively, significantly reducing the number of training parameters. Let c be the number of channels in the input feature map. in The number of channels in the output feature map is c. out If the kernel size is k×k, then the number of parameters in a regular convolution is:
[0065] k×k×c in ×c out (6)
[0066] The number of parameters for depthwise separable convolution is:
[0067] k×k×c in +c in ×c out (7)
[0068] Compared to regular convolution, depthwise separable convolution reduces the number of parameters by a factor of:
[0069]
[0070] Each layer of the original U-Net network contains two 3×3 convolutions. This invention uses two depthwise separable convolutions to form a depthwise separable convolution module (DSC module, such as...). Figure 3 ), replacing the original module in U-Net:
[0071] DSC(f) = Conv 1×1 (DW(Conv 1×1 (DW(f)))) (9)
[0072] Where DW(·) is a depthwise convolution, and Conv 1×1 (·) represents pointwise convolution, and f is the feature map.
[0073] The shallow layers of the network contain shallow semantic information such as edges and textures. The first and last layers are the image input and the predicted image output, respectively. These layers have a small number of channels and parameters. Using depthwise separable convolutions can only reduce the number of parameters to a limited extent, and depthwise separable convolutions cannot simultaneously fuse spatial and channel information, which can easily affect network performance. Therefore, the shallow layers use residual modules (Res modules, such as...). Figure 4 (This is achieved by) enhancing feature representation through residual structure:
[0074] ReS(f)=f+Conv 3×3 (Conv 3×3 (f)) (10)
[0075] Among them, Conv 3×3 (·) represents a 3×3 convolution, and all convolutions used in the module include Batch Normalization and ReLU activation functions.
[0076] The input water surface image first passes through a Res module in the encoder, then through three DSC modules. During this process, the resolution continuously decreases while the number of channels continuously increases. Then it reaches the bottleneck layer. Subsequently, the decoder passes through three DSC modules and one Res module in a symmetrical process, restoring the resolution to the original size. Finally, it reduces the number of channels through 1×1 convolution to form the recognition result.
[0077] Step 4: To address the issue of excessively large network model size, depthwise separable convolution combined with an attention mechanism is used in the network to reduce the model size. The channel attention mechanism first performs global average pooling on the feature map space, compressing spatial dimensions such as length and width while retaining the channel dimension, resulting in a one-dimensional feature vector. Then, one-dimensional convolution is used to generate channel weights, enabling the channel attention module to have higher computational efficiency without sacrificing accuracy. Finally, the channel weights are multiplied by the original feature map to efficiently implement channel attention.
[0078] like Figure 5 Channel attention is a type of attention mechanism that aims to make the model focus on information from some channels rather than all channels, thereby eliminating channels that are not very useful. The calculation method of channel attention is as follows:
[0079] ω c =σ(Conv 1d (Ave(f))), f∈R H×W×C ω c ∈R 1×1×C (11)
[0080]
[0081] Where f is the input feature map, f′ is the output feature map, Ave(·) is the global average pooling, and Conv 1d (·) represents a one-dimensional convolution, σ(·) is the Sigmoid activation function, and ω c This represents the channel weight.
[0082] The U-Net network of this invention has an excessive number of channels, and some channels contain less effective information. Using channel attention would result in greater gain. This invention adds an ECA module after the Res and DSC modules of the encoder to compensate for the network's shortcomings. Feature maps pass through these modules sequentially.
[0083] f1 = ECA(Res(f)) (13)
[0084] f2=ECA(DSC(d(f1))) (14)
[0085] f3=ECA(DSC(d(f2))) (15)
[0086] f4=ECA(DSC(d(f3))) (16)
[0087] f5 = DSC(d(f4)) (17)
[0088] Where d(·) represents average pooling downsampling, ECA(·) represents channel attention mechanism, and f i (i = 0, 1, ..., 5) represents the feature map of the intermediate process of the network.
[0089] The network does not use channel attention in the decoder; instead, the decoder continues to use depthwise separable convolutions to reduce the number of parameters. The specific structure includes a deep three-layer DSC module and a shallow one-layer Res module. Feature maps are passed sequentially through the deep and shallow modules to gradually form segmentation results. The feature maps of each layer are concatenated with the corresponding encoder feature maps via skip connections. Transposed convolution upsampling increases the resolution by a factor of two, and four upsampling operations gradually restore the resolution to its original size. Finally, the feature maps are reduced in number of channels using a 1×1 convolution to obtain the final result.
[0090] f4′=DSC(up(f5)+f4) (18)
[0091] f3′=DSC(up(f4′)+f3) (19)
[0092] f2′=DSC(up(f3′)+f2) (20)
[0093] f1′=Conv 1×1 (Res(up(f′2)+f1)) (21)
[0094] Where up(·) is the transposed convolution upsampling, f i and f i ′(i=1,2,...,4) is the feature map of the intermediate process of the network, Conv 1×1 (·) represents a 1×1 convolution.
[0095] Step 5: Load the training and validation sets of the water surface debris image data into the improved network for training and validation, obtain the network parameter values, and save the model with the best performance; use the saved model to test the test set images and output the water surface debris image segmentation results.
[0096] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.
Claims
1. A water surface garbage recognition method based on deep learning, characterized in that, Comprise the following steps: Step one, the water surface garbage identification data set picture is cut; Step two, using Laplace convolution to get the preliminary denoising feature map containing edge information , the feature map is obtained by subtracting the inhibition weight from the input image , and then further extracting features through convolution, and then obtaining the inhibition weight through the Sigmoid activation function Omega ; each channel of the input image is respectively multiplied by the inhibition weight (1- Omega ) after subtraction to obtain a weighted feature map; the weighted feature map is output to the subsequent encoder-decoder; Step three, using two deep separable convolution in the deep layer of U-Net network to constitute DSC module, using Res module in shallow layer; Step three specifically includes: The picture is first extracted features in the encoder through a layer of Res module and three layers of DSC module, then completes the up sampling through the symmetrical process in the decoder, and finally reduces the channel number through 1x1 convolution; The decoder uses deep separable convolution module, including three layers of deep DSC module and one layer of shallow Res module, the feature map passes through the deep module and the shallow module in turn, the feature map of each layer is spliced with the feature map of the corresponding encoder through the jump connection, the transposed convolution up sampling makes the resolution increase by 2 times, the four times of up sampling operation makes the resolution gradually restore to the original size, and the feature map finally reduces the channel number through 1x1 convolution; Step four, using deep separable convolution combined with channel attention mechanism, the channel attention mechanism first performs global average pooling on the feature map space, compresses the length and width space dimensions, retains the channel dimension, obtains a one-dimensional feature vector, uses one-dimensional convolution to generate channel weight, and finally multiplies the channel weight with the original feature map; The Res module and the DSC module of the encoder are added with the ECA module; The decoder uses deep separable convolution module; Step five, training the network model under the deep learning framework, obtaining the network parameter value, inputting the water surface garbage picture into the network, and outputting the water surface garbage picture segmentation result. 2.The water surface garbage recognition method based on deep learning according to claim 1, characterized in that, using laplacian convolution to obtain a preliminary denoised feature map containing edge information The formula is: (1) wherein, denotes a Laplacian convolution, H is the picture height, W is the picture width. 3.The deep learning-based water surface garbage recognition method according to claim 1, characterized in that, The feature map With the input graph After adding by the residual connection, further extract features by convolution, get the inhibition weight through the Sigmoid activation function Omega The process is: (3) wherein, Conv contains 1 3x3 convolution, 2 7x7 convolutions, 1 1x1 convolution, each containing Batch Normalization and ReLU activation function. 4.The deep learning-based water surface garbage recognition method according to claim 1, characterized in that, input map Each channel of the input map is multiplied element-wise with the difference between the inhibition weight (1- Omega ) to obtain a weighted feature map. (4) wherein , , are 3x3 convolution, 7x7 convolution and 1x1 convolution, respectively; The Laplacian feature map is expanded to 16 channels by 3x3 convolution, information is extracted from the feature map by 7x7 convolution without changing the channel number, and finally 1x1 convolution is used to integrate the channel information to generate the inhibition weight Omega ; The output of the feature map with weight is: (5) wherein, is a Hadamard product, is I the first i channel, Concat is a feature map concatenation, O is a module output. 5.The deep learning-based water surface garbage recognition method according to claim 1, wherein, The calculation formula of DSC module is: (9) wherein, DW is a depthwise convolution, is a pointwise convolution, f is a feature map. 6.The deep learning-based water surface garbage recognition method according to claim 1, wherein, The calculation method of channel attention mechanism is: (11) (12) wherein, f is an input feature map, f' is an output feature map, Ave is a global average pooling, is a one-dimensional convolution, Sigma is a Sigmoid activation function, is a channel weight.