Stained sea surface image detection method based on spatial alignment semantic segmentation network, electronic equipment and readable storage medium

Abstract

The invention relates to the field of computer vision and image processing, in particular to a stained sea surface image detection method based on a spatial alignment semantic segmentation network, electronic equipment and a readable storage medium. The detection method comprises the following steps: 1, inputting a to-be-detected stained sea surface image and a reference image; 2, multi-level feature maps of the stained sea surface image and the reference image are extracted through an encoder; 3, performing space alignment on the feature map of the reference image and the feature map of the stained sea surface image by using a space alignment module based on deformable convolution to obtain a feature map after space alignment; 4, through a space attention module, enabling the semantic segmentation network to focus attention on a stained region of the image, and obtaining a space alignment attention feature map; and 5, fusing the space alignment attention feature maps at different stages through a decoder, and outputting a segmentation result of the stained region. The defect that training can only be carried out on artificially synthesized data is overcome, and the stained area in the image can be accurately detected.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to a method for detecting polluted sea surface images based on a spatially aligned semantic segmentation network, an electronic device, and a readable storage medium. Background Art

[0002] Detecting fouled sea surface images plays a crucial role in the visual perception systems of unmanned surface vehicles (USVs). In complex marine environments, USVs rely on their visual perception systems to acquire crucial information, such as surface obstacles and the marine ecosystem. However, during long missions, their camera lenses inevitably become eroded by various sources of contamination, causing stains to accumulate on the lens surface. These stains can cause partial loss of information in captured images, severely compromising image integrity and accuracy. Therefore, real-time image restoration techniques are needed to automatically fill in the fouled areas to restore the image's true visual content and lay the foundation for subsequent visual missions. However, the morphology of residual stains on the lens is often highly random and unknown, making it impossible to detect fouled areas based on accurate a priori information. Detection errors caused by fouled areas pose significant potential risks to the safety of USVs, potentially leading to collisions and other safety incidents, threatening the safety of both the USV and the surrounding marine environment.

[0003] Existing image restoration methods mainly include traditional methods and deep learning-based methods. Traditional methods are mainly based on the assumptions of image structural similarity and texture feature consistency, and fill in damaged areas by diffusing the boundaries of damaged areas or extracting texture patches from known areas of the image. However, in image scenes with complex textures and large areas of defects, such methods have difficulty in accurately restoring meaningful texture structures and cannot ensure that the restored image is consistent with the real scene. Compared with traditional methods, deep learning-based image restoration methods can link the underlying features of the image with high-level semantic information to infer the content of the missing area of the image, so its restoration results are often closer to the judgment of the human eye. However, existing deep learning methods still infer and restore the content of the missing area based on the known content of the damaged image, resulting in the problem that the restoration results are inconsistent with the real scene. In addition, since it is difficult for environmental perception cameras to simultaneously obtain paired defaced images and true images in the real world, existing deep learning methods almost all train and test models on artificially synthesized damaged images. However, there are huge differences between these images and real defaced images in terms of the morphology of the defective areas and the form of missing information. As a result, the models trained on artificially synthesized data cannot be applied to the restoration of real defaced images. Summary of the Invention

[0004] In view of the problems and shortcomings in the prior art, the object of the present invention is to provide a method for detecting polluted sea surface images based on a spatially aligned semantic segmentation network, an electronic device, and a readable storage medium.

[0005] Based on the above purpose, the present invention adopts the following technical solutions:

[0006] A first aspect of the present invention provides a method for detecting polluted sea surface images based on a spatially aligned semantic segmentation network, comprising the following steps:

[0007] Step 1: Input the polluted sea surface image to be detected and the reference image;

[0008] Step 2: Extract multi-level feature maps of the polluted sea surface image and the reference image through the encoder;

[0009] Step 3: Using a spatial alignment module based on deformable convolution, the feature map of the corrupted sea surface image and the feature map of the reference image are first spliced along the channel direction. Convolution is performed to reduce the number of channels and perform nonlinearity to obtain an intermediate feature map. A 3×3 deformable convolution layer is used to learn a parameter tensor from the intermediate feature map. The sampling position of the convolution kernel is adaptively adjusted using the offset vector and modulation weight to ensure that the feature map of the reference image is spatially aligned with the feature map of the corrupted sea surface image, resulting in a spatially aligned feature map.

[0010] Step 4: Through the spatial attention module, the semantic segmentation network focuses on the contaminated area of the image to obtain a spatially aligned attention feature map; the spatially aligned feature map obtained in step 3 is spliced with the feature map of the contaminated sea surface image along the channel direction, and sequentially passes through a 1×1 convolution layer, a batch normalization layer, and a ReLU activation layer to obtain an intermediate feature map, which is then sent to another 1×1 convolution layer, a batch normalization layer, and a Sigmoid activation layer to generate a spatial attention coefficient matrix. The coefficient matrix is element-wise multiplied with the spatially aligned feature map to obtain a spatial attention weighted aligned feature map, which is then spliced with the feature map of the contaminated sea surface image along the channel direction, and sequentially passes through a 1×1 convolution layer, a batch normalization layer, and a ReLU activation layer to obtain a spatially aligned attention feature map;

[0011] Step 5: The decoder fuses the spatially aligned attention feature maps from different stages and outputs the segmentation result of the stained area.

[0012] Furthermore, in step 2, the encoder adopts the encoder structure of the lightweight network FSSNet, which is mainly composed of an initialization module, a decomposition convolution module and a dilation convolution module; the encoder gradually downsamples the input corrupted sea surface image and the reference image through a series of convolution layers and pooling layers, and extracts multi-level feature maps of the corrupted sea surface image and the reference image.

[0013] Furthermore, the encoder is divided into three stages: an L1 initialization stage, an L2 continuous decomposition convolution stage, and an L3 continuous expansion convolution stage; the initialization stage includes an initialization module, which preliminarily extracts shallow features of the image through convolution and pooling operations; the continuous decomposition convolution stage is composed of a downsampling module and four consecutive decomposition convolution modules in series, the downsampling module is mainly used to reduce the resolution of the feature map so that the network can focus on the most important features or elements, and the decomposition convolution module reduces the number of network parameters through decomposition convolution operations; the continuous expansion convolution stage includes a downsampling module and six consecutive expansion convolution modules, the expansion convolution module expands the receptive field through convolution operations with different expansion rates and extracts higher-level semantic information.

[0014] Furthermore, the decomposition convolution module replaces the 3×3 convolution in the nonlinear mapping branch of the residual module with two one-dimensional convolutions in different directions, thereby reducing the number of network parameters and computational complexity; the dilated convolution module replaces it with a 3×3 dilated convolution, increasing the receptive field of the convolution kernel while keeping the convolution kernel parameters unchanged, so that the output of each convolution contains a wider range of information to enhance the robustness of the model; six different dilation rates of 2, 5, 9, 2, 5, and 9 are used in six consecutive dilated convolution modules to avoid the grid effect caused by dilated convolution.

[0015] Furthermore, in step 3, a 3×3 deformable convolution layer is used to extract the intermediate feature map We learn a parameter tensor T of size 27×W×H n , by offsetting the vector Δp k and modulation weight Δm k Adaptively adjust the sampling position of the convolution kernel to ensure the reference image feature map Feature map of polluted sea surface image Align spatially to generate aligned feature maps

[0016]

[0017] Among them, w k is the learnable weight, p0 is each position on the feature map, p k is the sampling position and p k ∈R, R is the sampling grid, Δp k is the offset vector, Δm k is the modulation weight;

[0018] The feature map is realized by bilinear interpolation:

[0019]

[0020] Where p represents the feature map Any of the above sampling positions may be a fraction, and p=p0+p k +Δp k ;q represents the feature map Any integer spatial position on the y-axis; G(·,·) represents the bilinear interpolation kernel, which is two-dimensional and can be decomposed into two one-dimensional kernels:

[0021] G(q,p)=g(q x ,p x )·g(q y ,p y )

[0022] Where g(a,b)=max(0,1-|ab|).

[0023] Furthermore, in step 4,

[0024] After spatial alignment, the feature map is rewritten as

[0025] The feature map of the polluted sea surface image is rewritten as

[0026] Where C and M are the number of channels and the number of feature vectors of the feature map, respectively, and M = H × W, where H and W represent the height and width of the feature map, respectively;

[0027] Spatial Attention Weighted Aligned Feature Maps The eigenvector at any position on It can be defined as:

[0028]

[0029] Among them, the weight α n,m is defined as follows:

[0030]

[0031] Where, and Feature maps and The spatial position index on is the eigenvector of m, ∪ represents the channel splicing operation, and and are the linear transformation parameters and bias parameters of the two 1×1 convolutional layers, δ and σ represent the ReLU and Sigmoid activation functions, respectively.

[0032] Furthermore, in step 5, the decoder includes two upsampling fusion modules, which upsample and fuse the spatially aligned attention feature maps at different stages to gradually restore high-resolution feature maps and output the final semantic segmentation results. The decoder operates as follows:

[0033] ①Align the space of stage n+1 with the attention feature map It is fed into a 1×1 convolution layer, a batch normalization layer, and a ReLU activation layer in sequence to make the number of channels of the feature map the same as the feature map of the nth stage. The number of channels remains consistent;

[0034] ②Adjust its resolution to the same level as the shallow feature map through a bilinear interpolation upsampling layer Consistent, that is

[0035] ③The upsampled feature map With shallow feature map Add element by element and pass through a 3×3 convolution layer, a batch normalization layer and a ReLU activation layer in sequence to finally achieve the fusion of feature maps at different stages;

[0036] ④ In the final stage of the decoder, a bilinear interpolation upsampling layer is used to adjust the spatial resolution of the input feature map to the size of the original image, and a 1×1 convolutional layer is used to output the semantic segmentation result of the damaged area.

[0037] Furthermore, two loss functions are used to supervise the learning of the network, namely feature alignment loss L align And the cross entropy loss L ce :

[0038] L=λ1L align +λ2L ce

[0039] Where λ1 and λ2 are L align , L ce The weight parameter of the loss function; L ce The cross entropy loss function is used in the output layer of the network to supervise the training of the network. align Used to strengthen the network's spatial alignment feature map The learning formula is:

[0040]

[0041] Where L1(·,·) is the L1 loss function.

[0042] A second aspect of the present invention provides an electronic device comprising a memory and a processor, wherein a computer program is stored on the memory, and when the processor executes the computer program, any step of the method for detecting contaminated sea surface images based on a spatially aligned semantic segmentation network as described in the first aspect is implemented.

[0043] The third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer processor, it implements any step in the method for detecting dirty sea surface images based on a spatially aligned semantic segmentation network as described in the first aspect.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention proposes a lightweight semantic segmentation network to detect damaged areas in images, and then fills the damaged areas with a uniform color based on the detection results, so that the defect form of the image is consistent with the training data used by the image restoration network, overcoming the disadvantage of the existing technology that it can only be trained on artificially synthesized data.

[0046] This method can accurately detect contaminated areas in images. It improves detection accuracy by incorporating reference image information from adjacent viewpoints. Furthermore, it utilizes a spatial attention module to focus the network on contaminated areas, enhancing detection robustness. The method also offers real-time performance, meeting the real-time processing requirements of unmanned surface vehicle vision systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is the overall framework diagram of the method of the present invention;

[0048] Figure 2 This is a network structure diagram of the initialization module and downsampling module of the present invention;

[0049] Figure 3 This is a network structure diagram of the decomposition convolution module and the expansion convolution module of the present invention;

[0050] Figure 4 This is a structural diagram of the spatial alignment attention module of the present invention;

[0051] Figure 5 This is a position distribution diagram of different 3×3 convolution kernel sampling points of the present invention;

[0052] Figure 6 This is a qualitative comparison example of different models in the fouled sea surface image detection experiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] Example 1

[0055] This embodiment provides a method for detecting polluted sea surface images based on a spatially aligned semantic segmentation network. Figure 1 As shown, the following steps are included:

[0056] Step 1: Input the polluted sea surface image to be detected and the reference image;

[0057] The input image includes the main view of the polluted sea surface image I Src and the reference image I of the adjacent view Ref The two images are collected simultaneously by a multi-camera vision system to ensure time synchronization.

[0058] Step 2: Extract multi-level feature maps of the polluted sea surface image and the reference image through the encoder;

[0059] The encoder uses the lightweight FSSNet encoder architecture, primarily consisting of an initialization module, a factorized convolution (FConv) module, and a dilated convolution (DConv) module. This reduces the number of network parameters and expands the receptive field of convolution operations, ensuring faster inference without sacrificing segmentation accuracy. The encoder progressively downsamples the input image through a series of convolutional and pooling layers, extracting multi-level feature maps.

[0060] Specifically, the encoder is divided into three stages: initialization stage (L1), continuous decomposition convolution stage (L2) and continuous expansion convolution stage (L3). The initialization stage (L1) contains an initialization module that preliminarily extracts shallow features of the image through convolution and pooling operations. Its structure is as follows Figure 2 As shown in (a); the continuous decomposition convolution stage (L2) is composed of a downsampling module and four consecutive decomposition convolution modules in series. The downsampling module is mainly used to reduce the resolution of the feature map so that the network can focus on the most important features or elements. Its structure is as follows Figure 2 As shown in (b), the decomposition convolution module reduces the number of network parameters by decomposing the convolution operation. Its structure is as follows Figure 3 (a); The subsequent dilated convolution stage (L3) includes a downsampling module and six consecutive dilated convolution modules. The dilated convolution module expands the receptive field through convolution operations with different dilation rates and extracts higher-level semantic information. Its structure is as follows Figure 3 (b) shown.

[0061] Among them, the decomposed convolution module and the dilated convolution module are both improved versions of the standard residual module. The decomposed convolution module replaces the 3×3 convolution in the nonlinear mapping branch of the residual module with two one-dimensional convolutions in different directions, thereby reducing the number of parameters and computational complexity of the network; the dilated convolution module replaces it with a 3×3 dilated convolution, aiming to increase the receptive field of the convolution kernel while keeping the convolution kernel parameters unchanged, so that the output of each convolution contains a wider range of information to enhance the robustness of the model. In addition, six different dilation rates of 2, 5, 9, 2, 5, and 9 are used in consecutive dilated convolution modules to avoid the grid effect caused by dilated convolution. Finally, the input image (I Src ) and the reference image of the adjacent view (I Ref ) to extract features and obtain a multi-level feature map similar to a pyramid shape ( and ).

[0062] Step 3: The structure of the spatial alignment module is as follows Figure 4 As shown in (b), the feature map of the reference image is spatially aligned with the feature map of the polluted sea surface image using a spatial alignment module based on deformable convolution. The specific steps are as follows:

[0063] ①Splicing along the channel direction;

[0064] Assume that the sea surface image is corrupted (I Src ) and the reference image (I Ref ) The feature maps at the nth stage of the encoder are and To make the reference image feature map and the feature map of polluted sea surface image Keeping them aligned in space, and Splice along the channel direction.

[0065] ② Reduce the number of channels to C through a 1×1 convolution layer, and then use the PReLU activation function for nonlinearization to obtain the intermediate feature map

[0066] ③Use the deformable convolution layer to spatially transform the feature map of the reference image to generate the aligned feature map

[0067] In traditional two-dimensional convolution, image feature extraction is mainly achieved through the following two steps: (i) using a regularized grid R to sample the input feature map F, and (ii) multiplying these sampling results by a learnable weight w k, and then merge these results. Figure 5 As shown in (a), the sampling grid R = {(-1,-1), (-1,0), ..., (0,1), (1,1)} defines a standard 3 × 3 convolution kernel, which has 9 sampling positions, and each sampling position is denoted by p k ∈R, for each position p0 on its output feature map F', there is:

[0068]

[0069] Since traditional standard convolution only samples and extracts image features at fixed positions in the feature map, it cannot adapt to scaled or rotated objects. To avoid this problem, the characteristic of deformable convolution that changes the theoretical receptive field can be used to construct the spatial geometric transformation relationship between the defaced sea surface image and the adjacent view images by introducing it into different stages of the network, thereby eliminating parallax and spatially aligning feature maps from different viewpoints. For example, Figure 5 (b) shown.

[0070] Specifically, if Figure 4 As shown in (b), a 3×3 deformable convolution layer is used to extract the intermediate feature map We learn a parameter tensor T of size 27×W×H n Since the 3×3 deformable convolution kernel has 9 sampling positions, and each sampling position needs to learn an offset vector Δp k =[Δx,Δy] T With a modulation weight Δm k Therefore, when the spatial resolution of the input and output feature maps of the deformable convolution remains unchanged, it is necessary to pre-learn a parameter tensor Tn with 27-dimensional channels. 。 Among them, the first 18 dimensions are the offsets of the 9 sampling positions of the convolution kernel in the horizontal and vertical directions {Δp k} k＝1:9 , and the last 9 dimensions are the modulation weights {Δm k} k＝1:9 Finally, using the parameter tensor T of the deformable convolution, the offset vector Δp k and modulation weight Δm k Adaptively adjust the sampling position of the convolution kernel to ensure the reference view feature map Feature map of the corrupted image Align spatially to generate aligned feature maps

[0071]

[0072] Since the offset vector Δp kWith the existence of , the deformable convolution has the ability to adaptively adjust the convolution kernel sampling position. However, it is necessary to take into account the offset vector Δp learned through the convolution operation. k It may be a fraction, so formula (2) also needs to be implemented through bilinear interpolation:

[0073]

[0074] Where p represents the feature map Any of the above sampling positions may be a fraction, and p=p0+p k +Δp k ;q represents the feature map Any integer spatial position on the y-axis; G(·,·) represents the bilinear interpolation kernel, which can be decomposed into two one-dimensional kernels because it is two-dimensional:

[0075] G(q,p)=g(q x ,p x )·g(q y ,p y ) (4)

[0076] In the formula, g(a,b)=max(0,1-|ab|). Since only a small portion of q is non-zero in formula (3), the corresponding result can be quickly solved.

[0077] Step 4: Through the spatial attention module, the semantic segmentation network focuses on the stained area of the image; the structure of the spatial attention module is as follows Figure 4 (c) As shown. Theoretically, at the nth stage of the encoder, the feature map after spatial alignment Feature map of the defaced image There will only be obvious differences in the defective areas. Therefore, based on this characteristic, the present invention designs a spatial attention module to make the semantic segmentation network pay more attention to the damaged areas of the image. The specific steps are as follows:

[0078] ① Feature map after spatial alignment and the feature map of the defaced image Splicing along the channel direction;

[0079] Let C and M be the number of channels and feature vectors of the feature map, and M = H × W. H and W are the height and width of the feature map, respectively. The spatially aligned feature map can be rewritten as The feature map of the corrupted image is rewritten as The feature map and Splice by channel.

[0080] ② Obtain the intermediate feature map through the 1×1 convolution layer, batch normalization layer and ReLU activation layer in sequence

[0081] ③ Send the intermediate feature map to another 1×1 convolution layer, batch normalization layer and Sigmoid activation layer to generate the spatial attention coefficient matrix α n ;

[0082] ④The coefficient matrix α n With the aligned feature map Multiply element by element to obtain the attention-weighted aligned feature map

[0083] Feature Map The eigenvector at any position on It can be defined as:

[0084]

[0085] Among them, the weight α n,m is defined as follows:

[0086]

[0087] Where, and Feature maps and The spatial position index on is the eigenvector of m; ∪(·,·) represents the channel splicing operation; and and are the linear transformation parameters and bias parameters of the two 1×1 convolutional layers; δ and σ represent the ReLU and Sigmoid activation functions, respectively.

[0088] ⑤The spatial attention weighted feature map Feature map of the defaced image Splice along the channel direction and pass through a 1×1 convolution layer, a batch normalization layer, and a ReLU activation layer in sequence to obtain a spatially aligned attention feature map

[0089] Step 5: The decoder fuses the spatially aligned attention feature maps from different stages and outputs the segmentation result of the defaced area.

[0090] The decoder consists of two upsampling fusion modules, the structure is as follows Figure 1 As shown in , it is mainly used to upsample and fuse the spatially aligned attention feature maps at different stages to gradually restore the high-resolution feature maps and output the final semantic segmentation results. The specific steps of the decoder are as follows:

[0091] ①The feature map of stage n+1 It is fed into a 1×1 convolution layer, a batch normalization layer, and a ReLU activation layer in sequence to make the number of channels of the feature map the same as the feature map of the nth stage. The number of channels remains consistent;

[0092] ②Adjust its resolution to the same level as the shallow feature map through a bilinear interpolation upsampling layer Consistent, that is

[0093] ③The upsampled feature map With shallow feature map Add element by element and pass through a 3×3 convolution layer, a batch normalization layer and a ReLU activation layer in sequence to finally achieve the fusion of feature maps at different stages;

[0094] ④ In the final stage of the decoder, a bilinear interpolation upsampling layer is used to adjust the spatial resolution of the input feature map to the size of the original image, and a 1×1 convolutional layer is used to output the semantic segmentation result of the damaged area.

[0095] In addition, the present invention uses two loss functions to supervise the learning of the network, namely feature alignment loss L align And the cross entropy loss L ce , as shown below:

[0096] L=λ1L align +λ2L ce (7)

[0097] Where λ1 and λ2 are L align , L ce The weight parameters of the loss function, λ1 and λ2 are set to 0.5 and 1.0 respectively; L ce The cross entropy loss function is mainly used in the output layer of the network to supervise the training of the network. align It is mainly used to strengthen the network's spatial alignment feature map The learning formula is:

[0098]

[0099] Where L1(·,·) is the L1 loss function.

[0100] In a specific embodiment, based on the Ubuntu 16.04 operating system, using the Python programming language and the PyTorch deep learning framework, the experimental hardware environment includes two NVIDIA GTX1080Ti GPUs and an Intel Core i7-6700K CPU. The specific steps are as follows:

[0101] (1) Experimental data preparation;

[0102] A dataset of defaced sea surface images covering coastal and port scenes was collected using a custom-built unmanned surface vehicle. The dataset consists of 1,600 pairs of images from adjacent viewpoints at a resolution of 512×384, with the left view representing the defaced image and the right view representing the reference image. The dataset was randomly divided into a training set (1,000 pairs of images), a validation set (200 pairs of images), and a test set (400 pairs of images). Each defaced image was manually annotated with the defaced area.

[0103] (2) Model building and training;

[0104] ① The encoder in the lightweight network FSSNet is selected as the backbone network and pre-trained on the public urban road scene dataset Cityscapes;

[0105] ② The input image resolution is uniformly scaled to 512×384, and the training data is augmented using random color jittering;

[0106] ③Use feature alignment loss L align and cross entropy loss L ce The loss function composed of (L = λ1L align +λ2L ce ,λ1=0.5,λ2=1.0) supervised network learning;

[0107] ④ The batch size is set to 8, and the stochastic gradient descent (SGD) method with a momentum factor of 0.9 is used as the parameter optimizer. The convolutional layer is initialized with the Kaiming method, the initial learning rate is set to 0.001, and the Poly learning strategy with an exponent of 0.9 is used to update the learning rate. A total of 30 epochs are trained.

[0108] (3) Selection of performance evaluation indicators;

[0109] This example uses commonly used indicators in the field of semantic segmentation to evaluate the model detection performance, including precision, recall, and F1-score. Precision is calculated by calculating the ratio of the number of pixels correctly classified as positive samples to the total number of pixels classified as positive samples; recall refers to the proportion of actual positive sample pixels predicted by the model; and F1-score is the harmonic mean of precision and recall. The calculation formula is as follows:

[0110]

[0111] Among them, (i) True Positive (TP), that is, the pixel predicted as a positive sample is actually a positive sample; (ii) False Positive (FP), that is, the pixel predicted as a positive sample is actually a negative sample; (iii) True Negative (TN), that is, the pixel predicted as a negative sample is actually a negative sample; (iv) False Negative (FN), that is, the pixel predicted as a positive sample is actually a negative sample. It shows the four situations of the prediction results for a pixel in the semantic segmentation task. In addition, in order to evaluate the execution efficiency and time complexity of the model, this embodiment uses the average operation time commonly used in the field of machine learning as the performance evaluation indicator of the model.

[0112] (4) Selection of control model;

[0113] A variety of semantic segmentation networks with excellent performance, such as ENet, FSSNet, PSPNet, and DeepLabv3+, were selected as control models. Considering that the task of detecting contaminated areas in sea surface images has high real-time requirements for the algorithm, the ResNet-50 network with moderate complexity was selected as the backbone network for PSPNet and DeepLabv3+. Among them, except for this embodiment, the other semantic segmentation networks only need to input the sea surface image to be detected, while the network of this embodiment also needs to input additional adjacent perspective reference images. In order to ensure the fairness of the comparative experiment, all networks were pre-trained on the public dataset Cityscapes and trained using the learning strategy described in step (2).

[0114] (5) Comparison and analysis of experimental results;

[0115] The quantitative evaluation results of different models in the fouled sea surface image detection experiment are shown in Table 1.

[0116] Table 1 Quantitative evaluation results of different models in the fouled sea surface image detection experiment

[0117] Model backbone network Accuracy (%) Recall rate (%) F1 value (%) ENet - 95.63 92.52 94.05 FSSNet - 94.88 96.15 95.51 PSPNet ResNet-50 95.21 95.98 95.59 DeepLabv3+ ResNet-50 96.62 95.47 96.04 Example 1 FSSNet encoder 98.12 99.06 98.59

[0118] ① Quantitative evaluation: Experiments were conducted on a self-made dataset, and the results are shown in Table 1. It was found that the spatial alignment semantic segmentation network of this embodiment achieved 98.12%, 99.06% and 98.59% in precision, recall and F1 value, respectively, which is better than other control models. Compared with the second-place DeepLabv3+ model, the precision rate increased by 1.5%, the recall rate increased by 3.59%, and the comprehensive index F1 value directly increased by 2.55%. The encoder used in this embodiment is the same as that of the FSSNet network, but the image information of adjacent perspectives is additionally introduced into the network, which makes it perform much better than FSSNet in the defaced area detection experiment, indirectly verifying the effectiveness of the spatial alignment module and spatial attention module proposed in this invention.

[0119] ② Qualitative evaluation: By qualitatively comparing the detection results of each semantic segmentation model on the contaminated area of the sea surface image, such as Figure 6 As shown, it is found that the semantic segmentation network of this embodiment can more accurately detect the damaged areas in various complex background images than other existing models, and has strong robustness and practicality.

[0120] ③ Computation time evaluation: Table 2 shows the average computation time of different semantic segmentation models in the defaced area detection experiment.

[0121] Table 2 Average operation time of different semantic segmentation models in the pollution sea surface image detection experiment

[0122] Semantic Segmentation Model ENet FSSNet PSPNet DeepLabv3+ Example 1 Computation time / ms 10 7 249 491 8

[0123] As can be seen from Table 2, for input images with a resolution of 512×384, the DeepLabv3+ network has the longest average computation time, taking approximately 491ms per frame; while the spatial alignment semantic segmentation network of this embodiment only takes an average of 8ms. Although this embodiment requires processing two input images simultaneously for each computation (i.e., the sea surface image to be detected and the reference image from the adjacent viewpoint), due to the use of a lightweight spatial alignment module and spatial attention module, the computation speed is only slightly slower than that of the FSSNet network, meeting the real-time requirements of the unmanned surface vehicle vision system.

[0124] Example 2

[0125] This embodiment provides an electronic device, including a memory and a processor, wherein a computer program is stored on the memory, and when the processor executes the computer program, any step of the method for detecting contaminated sea surface images based on a spatially aligned semantic segmentation network as described in Example 1 is implemented.

[0126] The hardware of the electronic device in this embodiment also includes a GPU, a display buffer memory, a RAMD / A converter, and a heat sink that cooperate with the processor; the GPU is responsible for processing the graphic display of the electronic device, providing image rendering and acceleration functions, and using its parallel computing advantages to accelerate the processing of large-scale data-intensive tasks.

[0127] Furthermore, the method for detecting contaminated sea surface images based on a spatially aligned semantic segmentation network described in Example 1 can be implemented as a computer software program. For example, this embodiment includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the method. In such an embodiment, the computer program can be downloaded and installed from a network and / or installed from removable media. When executed by a processor, the computer program performs the aforementioned functions defined in the method of this application.

[0128] Example 3

[0129] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, any step of the method for detecting contaminated sea surface images based on a spatially aligned semantic segmentation network as described in Example 1 is implemented.

[0130] The computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0131] The computer program code for performing the operations of the present application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as python, C++, and also conventional procedural programming languages or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0132] The computer-readable storage medium of this embodiment can be accelerated by hardware such as a GPU, and the parallel computing advantages of the GPU can be used to accelerate the processing of any step in the method for detecting contaminated sea surface images based on a spatially aligned semantic segmentation network as described in Example 1.

[0133] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the scope of protection of the present invention. Those skilled in the art can modify or replace the technical solutions of the present invention according to the concept of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A method for detecting polluted sea surface images based on a spatially aligned semantic segmentation network, characterized in that: The following steps are involved: Step 1: Input the polluted sea surface image to be detected and the reference image; Step 2: Extract multi-level feature maps of the polluted sea surface image and the reference image through the encoder; Step 3: Using a spatial alignment module based on deformable convolution, the feature map of the corrupted sea surface image and the feature map of the reference image are first spliced along the channel direction. Convolution is performed to reduce the number of channels and perform nonlinearity to obtain an intermediate feature map. A 3×3 deformable convolution layer is used to learn a parameter tensor from the intermediate feature map. The sampling position of the convolution kernel is adaptively adjusted using the offset vector and modulation weight to ensure that the feature map of the reference image is spatially aligned with the feature map of the corrupted sea surface image, resulting in a spatially aligned feature map. Step 4: Through the spatial attention module, the semantic segmentation network focuses on the stained area of the image to obtain a spatially aligned attention feature map; The spatially aligned feature map obtained in step 3 is concatenated with the feature map of the polluted sea surface image along the channel direction, and sequentially passes through a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation layer to obtain an intermediate feature map. The intermediate feature map is then fed into another 1×1 convolutional layer, a batch normalization layer, and a Sigmoid activation layer to generate a spatial attention coefficient matrix. The coefficient matrix is element-wise multiplied with the spatially aligned feature map to obtain a spatial attention weighted aligned feature map. The spatial attention weighted aligned feature map is then concatenated with the feature map of the polluted sea surface image along the channel direction, and sequentially passes through a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation layer to obtain a spatially aligned attention feature map. Step 5: The decoder fuses the spatially aligned attention feature maps from different stages and outputs the segmentation result of the stained area.

2. The method for detecting contaminated sea surface images based on a spatially aligned semantic segmentation network according to claim 1, characterized in that: In step 2, the encoder adopts the encoder structure of the lightweight network FSSNet, which is mainly composed of an initialization module, a decomposition convolution module and a dilation convolution module; the encoder gradually downsamples the input defaced sea surface image and the reference image through a series of convolution layers and pooling layers, and extracts multi-level feature maps of the defaced sea surface image and the reference image.

3. The method for detecting contaminated sea surface images based on a spatially aligned semantic segmentation network according to claim 2, characterized in that: The encoder is divided into three stages: an L1 initialization stage, an L2 continuous decomposition convolution stage, and an L3 continuous dilation convolution stage; the initialization stage includes an initialization module that preliminarily extracts shallow features of the image through convolution and pooling operations; the continuous decomposition convolution stage is composed of a downsampling module and four consecutive decomposition convolution modules in series. The downsampling module is mainly used to reduce the resolution of the feature map so that the network can focus on the most important features or elements, and the decomposition convolution module reduces the number of network parameters through decomposition convolution operations; the continuous dilation convolution stage includes a downsampling module and six consecutive dilation convolution modules. The dilation convolution module expands the receptive field through convolution operations with different dilation rates to extract higher-level semantic information.

4. The method for detecting contaminated sea surface images based on a spatially aligned semantic segmentation network according to claim 3, characterized in that: The decomposition convolution module replaces the 3×3 convolution in the nonlinear mapping branch of the residual module with two one-dimensional convolutions in different directions, thereby reducing the number of network parameters and computational complexity; the dilated convolution module replaces the 3×3 convolution in the nonlinear mapping branch of the residual module with a 3×3 dilated convolution, increasing the receptive field of the convolution kernel while keeping the convolution kernel parameters unchanged, so that the output of each convolution contains a wider range of information to enhance the robustness of the model; six different dilation rates of 2, 5, 9, 2, 5, and 9 are used in six consecutive dilated convolution modules to avoid the grid effect caused by the dilated convolution.

5. The method for detecting contaminated sea surface images based on a spatially aligned semantic segmentation network according to claim 1, characterized in that: In step 3, a 3×3 deformable convolution layer is used to extract the feature map from the intermediate feature map. We learn a parameter tensor T of size 27×W×H n , by offsetting the vector Δp k and modulation weight Δm k Adaptively adjust the sampling position of the convolution kernel to ensure the reference image feature map Feature map of polluted sea surface image Align spatially to generate aligned feature maps Among them, w k is the learnable weight, p0 is each position on the feature map, p k is the sampling position and p k ∈R, R is the sampling grid, Δp k is the offset vector, Δm k is the modulation weight; The feature map is realized by bilinear interpolation: Where p represents the feature map Any of the above sampling positions may be a fraction, and p=p0+p k +Δp k ;q represents the feature map Any integer spatial position on the y-axis; G(·,·) represents the bilinear interpolation kernel, which is two-dimensional and can be decomposed into two one-dimensional kernels: G(q,p)=g(q x ,p x )·g(q y ,p y ) Where g(a,b)=max(0,1-|ab|).

6. The method for detecting contaminated sea surface images based on a spatially aligned semantic segmentation network according to claim 1, characterized in that: In the step 4, After spatial alignment, the feature map is rewritten as The feature map of the polluted sea surface image is rewritten as Where C and M are the number of channels and the number of feature vectors of the feature map, respectively, and M = H × W, where H and W represent the height and width of the feature map, respectively; Spatial Attention Weighted Aligned Feature Maps The eigenvector at any position on It can be defined as: Among them, the weight α n,m is defined as follows: Where, and Feature maps and The spatial position index on is the eigenvector of m, ∪ represents the channel splicing operation, and and are the linear transformation parameters and bias parameters of the two 1×1 convolutional layers, δ and σ represent the ReLU and Sigmoid activation functions, respectively.

7. The method for detecting polluted sea surface images based on a spatially aligned semantic segmentation network according to claim 1, characterized in that: In step 5, the decoder includes two upsampling fusion modules, which upsample and fuse the spatially aligned attention feature maps at different stages to gradually restore high-resolution feature maps and output the final semantic segmentation results. The decoder operates as follows: ①Align the space of stage n+1 with the attention feature map It is fed into a 1×1 convolution layer, a batch normalization layer, and a ReLU activation layer in sequence to make the number of channels of the feature map the same as the feature map of the nth stage. The number of channels remains consistent; ②Adjust its resolution to the same level as the shallow feature map through a bilinear interpolation upsampling layer Consistent, that is ③The upsampled feature map With shallow feature map Add element by element and pass through a 3×3 convolution layer, a batch normalization layer and a ReLU activation layer in sequence to finally achieve the fusion of feature maps at different stages; ④ In the final stage of the decoder, a bilinear interpolation upsampling layer is used to adjust the spatial resolution of the input feature map to the size of the original image, and a 1×1 convolutional layer is used to output the semantic segmentation result of the damaged area.

8. The method for detecting contaminated sea surface images based on a spatially aligned semantic segmentation network according to claim 1, characterized in that: Two loss functions are used to supervise the learning of the network, namely feature alignment loss L align And the cross entropy loss L ce : L=λ1L align +λ2L ce Where λ1 and λ2 are L align , L ce The weight parameter of the loss function; L ce The cross entropy loss function is used in the output layer of the network to supervise the training of the network. align Used to strengthen the network's spatial alignment feature map The learning formula is: Where L1(·,·) is the L1 loss function.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, it implements any step in the method for detecting polluted sea surface images based on a spatially aligned semantic segmentation network as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a computer processor, implements any step in the method for detecting contaminated sea surface images based on a spatially aligned semantic segmentation network as described in any one of claims 1 to 8.

Images