Offshore oil spill detection method, device, medium and equipment

Abstract

The invention discloses an offshore oil spill detection method, device, medium and equipment, and relates to the technical field of ocean monitoring. Comprising the step of constructing an improved U-Net network by adding a multi-head cross attention mechanism module at the input end of a decoder deconvolution module of an original U-Net network and replacing an encoder in the original U-Net network with a ResNet50 network. The output of the ResNet50 network with the same scale and the input of the decoder are spliced and then are input into the multi-head cross attention mechanism module, and the multi-head cross attention mechanism module can capture the long-distance dependency relationship in the spliced feature map. Therefore, a higher weight is only given to the supplementary features obviously associated with other oil spill features instead of weighting all the features supplemented through the splicing module, useful information in the supplementary features is transmitted, and the obtained isolated oil spill area information is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of marine monitoring technology, and in particular to a method, device, medium and equipment for detecting marine oil spills. Background Art

[0002] Marine oil spills usually refer to marine accidents that cause marine pollution, such as ship collisions, reef strikes, and fuel oil leaks from cargo ships due to wave damage. These leaks usually occur in areas far from land, causing oil to spread, disperse, and float on the surface of the ocean, causing widespread pollution and damage to marine life and ecosystems. Therefore, effective monitoring and estimation of oil spills are key steps in dealing with leaks.

[0003] In the prior art, the U-Net network is a method for image segmentation, which can realize offshore oil spill detection by segmenting oil spill images. Its network structure consists of two parts: an encoder and a decoder, with features transferred through jump connections in the middle. The encoder extracts high-level features of the image through a series of convolution operations; the jump connection transfers the encoder's feature map to the decoder part to retain and utilize detail information, so that the decoder can obtain fine-grained information from the encoder during upsampling; the decoder gradually restores the spatial resolution of the image through upsampling, and uses the feature map in the jump connection to assist in reconstruction.

[0004] However, after multiple convolution operations of the encoder, the features of the larger oil spill areas in the oil spill image will be well extracted. However, the information of some isolated oil spill areas in the oil spill image may be lost, resulting in inaccurate oil spill detection results. Summary of the invention

[0005] Based on this, in order to solve the technical problem that the existing technology has low accuracy in detecting offshore oil spills, the present invention provides a method, device, medium and equipment for detecting offshore oil spills.

[0006] The present invention provides a method for detecting oil spills at sea, comprising:

[0007] An improved encoder is constructed by replacing the encoder in the original U-Net network with a ResNet50 network, wherein the ResNet50 network includes a splicing module for splicing the input of the convolution operation and the output of the convolution operation; an improved decoder is constructed by adding a multi-head cross attention mechanism module at the input end of the deconvolution module of the original U-Net network decoder; and an improved U-Net network including an improved encoder, an improved decoder and a jump connection of the original U-Net network is constructed;

[0008] Collect offshore oil spill images to build a data set, use the data set to train the improved U-Net network, and obtain an offshore oil spill detection model;

[0009] The offshore oil spill image to be detected is input into the offshore oil spill detection model, and the offshore oil spill image to be detected is convolved by improving the encoder to reduce the size of the offshore oil spill image to be detected. During the convolution operation, the input of the convolution operation is spliced ​​with the output of the convolution operation by a splicing module to compensate for the loss of oil spill features caused by the size reduction of the offshore oil spill image to be detected during the convolution operation, so as to obtain oil spill feature maps of different scales; the first decoder deconvolution module in the improved decoder is deconvolved to perform a deconvolution operation on the oil spill feature map of the smallest scale to restore the size of the oil spill feature map; before the remaining decoder deconvolution modules perform deconvolution operations, the oil spill feature map of the corresponding scale is transmitted to the input end of the corresponding decoder deconvolution module through a jump connection. The oil spill recovery feature map output from the previous decoder deconvolution module is spliced ​​to obtain a spliced ​​feature map of the corresponding scale; the long-distance dependency in the spliced ​​feature map is captured through a multi-head cross attention mechanism module to obtain the association between the oil spill features supplemented by the splicing module in the splicing feature map and other oil spill features in the splicing feature map, and the oil spill features supplemented by the residual connection module that are associated with other oil spill features in the splicing feature map are weighted to obtain an oil spill weighted feature map of the corresponding scale; the residual decoder deconvolution module performs a deconvolution operation on the oil spill weighted feature map of the corresponding scale to obtain oil spill recovery feature maps of different scales; the pixels in the oil spill recovery feature map of the largest size are classified into oil spill pixels and non-oil spill pixels, and the oil spill area in the marine oil spill image to be detected is output.

[0010] Furthermore, the ResNet50 network includes 5 residual modules connected in sequence, and the standard convolution layer in each residual module is replaced by a convolution block and an identity block structure connected in sequence; the convolution block is used to adjust the channel and size of the oil spill feature map; the identity block is used to increase the depth of the network without changing the channel and size of the oil spill feature map to prevent the gradient from disappearing during the oil spill feature transmission process.

[0011] Furthermore, the oil spill feature map of the corresponding scale is transmitted to the input end of the corresponding decoder deconvolution module through the jump connection to be spliced ​​with the oil spill recovery feature map output from the previous decoder deconvolution module to obtain the spliced ​​feature map of the corresponding scale, specifically including:

[0012] The fourth oil spill feature map output by the fourth residual module in the ResNet50 network and the first oil spill recovery feature map output by the first decoder deconvolution module are spliced ​​to obtain a first spliced ​​feature map;

[0013] The third oil spill feature map output by the third residual module in the ResNet50 network and the second oil spill recovery feature map output by the second decoder deconvolution module are spliced ​​to obtain a second spliced ​​feature map;

[0014] The second oil spill feature map output by the second residual module in the ResNet50 network and the third oil spill recovery feature map output by the third decoder deconvolution module are spliced ​​to obtain a third spliced ​​feature map;

[0015] The first oil spill feature map output by the first residual module in the ResNet50 network and the fourth oil spill recovery feature map output by the fourth decoder deconvolution module are spliced ​​to obtain a fourth spliced ​​feature map.

[0016] Furthermore, the multi-head cross attention mechanism module is added to the input end of the original U-Net network decoder deconvolution module, specifically including:

[0017] In the original U-Net network, the first multi-head cross-attention mechanism module, the second multi-head cross-attention mechanism module, the third multi-head cross-attention mechanism module and the fourth multi-head cross-attention mechanism module are added to the input ends of the second decoder deconvolution module, the third decoder deconvolution module, the fourth decoder deconvolution module and the fifth decoder deconvolution module respectively:

[0018] The i-th multi-head cross attention mechanism module is used to capture the long-distance dependency in the i-th spliced ​​feature map, and output the i-th oil spill weighted feature map corresponding to the i-th spliced ​​feature map, i=1,2,3,4; specifically includes:

[0019] The i-th multi-head cross attention mechanism module uses the oil spill feature map from the splicing module in the i-th splicing feature map as the query Q and key K in the attention mechanism, and uses the oil spill recovery feature map from the decoder deconvolution module in the i-th splicing feature map as the value V in the attention mechanism;

[0020] The attention weight matrix is ​​calculated based on the query Q and the key K, and the attention weight matrix is ​​multiplied by the value V and then upsampled to obtain a first feature matrix; the oil spill feature map from the residual module is position-encoded and dot-multiplied with the first feature matrix to obtain a first oil spill weighted feature map; the oil spill recovery feature map from the decoder deconvolution module is position-encoded and upsampled to obtain a second oil spill weighted feature map;

[0021] The first oil spill weighted feature map and the second oil spill weighted feature map are spliced ​​to obtain the i-th oil spill weighted feature map output by the i-th multi-head cross attention mechanism module.

[0022] The present invention provides a marine oil spill detection device, comprising:

[0023] A model building module, for building an improved encoder by replacing the encoder in the original U-Net network with a ResNet50 network, wherein the ResNet50 network includes a splicing module for splicing the input of the convolution operation and the output of the convolution operation; building an improved decoder by adding a multi-head cross attention mechanism module at the input end of the deconvolution module of the original U-Net network decoder; and building an improved U-Net network including an improved encoder, an improved decoder and a jump connection of the original U-Net network;

[0024] The model training module is used to collect offshore oil spill images to build a data set, and use the data set to train the improved U-Net network to obtain an offshore oil spill detection model;

[0025] The oil spill detection module is used to input the marine oil spill image to be detected into the marine oil spill detection model, and perform convolution operation on the marine oil spill image to be detected by improving the encoder to reduce the size of the marine oil spill image to be detected. During the convolution operation, the input of the convolution operation is spliced ​​with the output of the convolution operation by the splicing module to compensate for the loss of oil spill features caused by the size reduction of the marine oil spill image to be detected during the convolution operation, so as to obtain oil spill feature maps of different scales; the first decoder deconvolution module in the improved decoder is used to perform deconvolution operation on the oil spill feature map of the smallest scale to restore the size of the oil spill feature map; before the remaining decoder deconvolution modules perform deconvolution operation, the oil spill feature map of the corresponding scale is transmitted to the corresponding decoder deconvolution module through jump connection. The input end is used to splice the oil spill recovery feature map output by the previous decoder deconvolution module to obtain a spliced ​​feature map of the corresponding scale; and the long-distance dependency in the spliced ​​feature map is captured through the multi-head cross attention mechanism module to obtain the association between the oil spill features supplemented by the splicing module in the splicing feature map and other oil spill features in the splicing feature map, and the oil spill features supplemented by the residual connection module that are associated with other oil spill features in the splicing feature map are weighted to obtain an oil spill weighted feature map of the corresponding scale; the residual decoder deconvolution module performs a deconvolution operation on the oil spill weighted feature map of the corresponding scale to obtain oil spill recovery feature maps of different scales; the pixels in the oil spill recovery feature map of the largest size are classified into oil spill pixels and non-oil spill pixels, and the oil spill area in the marine oil spill image to be detected is output.

[0026] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned offshore oil spill detection method is implemented.

[0027] The present invention provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned offshore oil spill detection method when executing the program.

[0028] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0029] In the offshore oil spill detection method provided by the present invention, the encoder is replaced with a ResNet50 network and a multi-head cross-attention mechanism module is introduced to improve the U-Net network; the ResNet50 module splices the input of the convolution operation with the output of the convolution operation to supplement the loss of oil spill features caused by the size reduction of the offshore oil spill image to be detected during the convolution operation, thereby avoiding the loss of isolated oil spill area features during the convolution process; the output of the ResNet50 network of the same scale is spliced ​​with the output of the decoder deconvolution, and then input into the multi-head cross-attention mechanism module. Since the multi-head cross-attention mechanism module can capture the long-distance dependency in the spliced ​​feature map, the association between the oil spill features supplemented by the splicing module in the spliced ​​feature map and other oil spill features in the spliced ​​feature map can be obtained, so that only those supplementary features that are significantly associated with other oil spill features are given higher weights, instead of weighting all supplementary features, thereby transmitting useful information in the supplementary features, so that the obtained isolated oil spill area information is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0031] Figure 1 A schematic flow chart of a method for detecting oil spills at sea provided by the present invention;

[0032] Figure 2 A schematic diagram of the U-Net network structure provided by the present invention;

[0033] Figure 3 Schematic diagram of the Resnet50 convolutional block provided by the present invention;

[0034] Figure 4 Schematic diagram of the Resnet50 identity block provided by the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in combination with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Ships, aircraft, and satellites are widely used to identify and monitor oil spills. Traditional methods include aerial photography and field surveys, but they are costly. Some ships are equipped with special radars to detect oil at sea, but their visibility and coverage are limited. Therefore, satellite observations have become one of the main systems for monitoring and controlling climate change. Remote sensing observations, especially those from SAR (Synthetic Aperture Radar) sensors, are favored due to their data collection capabilities. SAR sensors record backscattered signals of different frequencies and polarizations to form a two-dimensional image. SAR sensors are widely used in marine oil spill detection because they are not affected by sunlight and cloudy weather, have a wide coverage range, and are less expensive than aircraft equipped with SAR sensors.

[0037] In this context, this embodiment provides a method for detecting oil spills at sea. The method improves the original U-Net network by replacing the encoder part of the U-Net network with the residual block of Resnet50 and introducing a multi-head cross-attention mechanism in the residual connection of the decoder part to efficiently extract image information. The SAR image is segmented into two categories: oil spills and background using the improved U-Net network.

[0038] Figure 1 The flow chart of the offshore oil spill detection method provided by this embodiment is shown. Figure 1 The offshore oil spill detection method is described in detail, and specifically comprises the following steps:

[0039] S1: An improved encoder is constructed by replacing the encoder in the original U-Net network with a ResNet50 network, which includes a splicing module for splicing the input of the convolution operation and the output of the convolution operation; an improved decoder is constructed by adding a multi-head cross attention mechanism module at the input of the deconvolution module of the original U-Net network decoder; and an improved U-Net network including an improved encoder, an improved decoder and a skip connection of the original U-Net network is constructed.

[0040] The U-Net network structure is as follows Figure 2 As shown, Figure 2The left part, i.e., the encoder part, is replaced by a ResNet50 network, which includes five residual modules connected in sequence; the decoder includes five decoder deconvolution modules connected in sequence; the residual module outputs of the same scale are transmitted to the input end of the corresponding decoder deconvolution module through jump connections, so as to splice the oil spill feature map and the oil spill recovery feature map of the same scale, and input the spliced ​​feature map into the multi-head cross attention mechanism module for weighted operation, and input the weighted spliced ​​feature map into the next decoder deconvolution module for further decoding.

[0041] In the ResNet50 network, the standard convolutional layer in each residual module is replaced by a sequentially connected convolutional block and identity block structure. The convolutional block is used to adjust the feature map channel and feature map size to extract features of different scales; the identity block is used to increase the depth of the network without changing the size of the feature map or the number of channels to prevent the gradient from disappearing during feature transfer.

[0042] The network structure of the convolutional block is as follows Figure 3 As shown, it includes an input layer, two parallel convolution branches connected to the output end of the input layer at the same time, a splicing layer connected to the output ends of the two parallel convolution branches at the same time, a ReLU activation function layer connected to the output end of the splicing layer, and an output layer; one of the convolution branches includes a 1×1 convolution layer, a batch normalization layer, a ReLU activation function layer, a 3×3 convolution layer, a batch normalization layer, a ReLU activation function layer, a 1×1 convolution layer, and a batch normalization layer connected in sequence; the other convolution branch includes a 1×1 convolution layer and a batch normalization layer connected in sequence.

[0043] The network structure of the identity block is as follows Figure 4 As shown, it includes an input layer, a 1×1 convolution layer, a batch normalization layer, a ReLU activation function layer, a 1×1 convolution layer, a batch normalization layer, a ReLU activation function layer, a 1×1 convolution layer, a batch normalization layer, a fusion layer, a ReLU activation function layer and an output layer connected in sequence; wherein the output end of the input layer is connected to the input end of the fusion layer through a residual connection.

[0044] The working steps of improving the encoder are:

[0045] The input SAR image has a size of 256x256x1, where 1 represents the number of channels. The image is fed into the convolutional block of the first residual module, where a branch performs the following operations:

[0046] First, a convolution operation is performed using a 1x1 convolution kernel and 16 convolution layers, with a step size of 1, and the size of the output feature map is 256x256x16; the output feature map is normalized and processed with the ReLU activation function. Then a 3x3 convolution operation is performed, the number of output channels is set to 16, and padding is "same" to keep the size unchanged; the size of the feature map generated in this step is still 256x256x16; the feature map is batch normalized and processed with the ReLU activation function again. Then a convolution operation is performed again using a 1x1 convolution kernel and 16 convolution layers, and normalized. In another branch in the convolution block, the input image is convolved and normalized once with a 1x1 convolution operation through a residual connection, and then spliced ​​with the output of the previous branch. The spliced ​​result is processed with the ReLU activation function to obtain the output of the convolution block, and the output size is 256x256x16.

[0047] The features of size 256x256x16 are input into the identity block module and the following operations are performed:

[0048] First, a convolution operation is performed using a 1x1 convolution kernel and 16 convolution layers. The size of the output feature map is 256x256x16. The output feature map is normalized and processed with the ReLU activation function. Then a second 3x3 convolution operation is performed, the number of output channels is set to 16, and padding is "same" to keep the size unchanged. The size of the feature map generated in this step is still 256x256x16. The feature map is batch normalized and processed with the ReLU activation function again. A convolution operation is performed again using a 1x1 convolution kernel and 16 convolution layers. After normalization, it is spliced ​​with the input features through a residual connection. The spliced ​​result is processed with the ReLU activation function to obtain the output of the identity block, and the output size is 256x256x16. The convolution block is placed in the first position to adjust the channel and feature map size, and the identity block is used to deepen the network. After a 2x2 maximum pooling operation, a feature map of size 128x128x16 pixels is obtained, which becomes the input of the second residual module.

[0049] Similarly, in the second residual module, the steps are the same as the above. First, after 1x1 convolution, normalization, and ReLU activation function processing in the convolution block, a 3x3 convolution operation, normalization, and ReLU activation function processing are performed, and then 1x1 convolution and normalization operations are performed. The difference from the first residual module step is that the number of convolution layers in the second residual module is 32, 32, and 32 respectively. Finally, after a 1x1 convolution operation is performed on the input features of the second residual module, a residual connection is performed with the output part, and a feature map with a size of 128x128x32 is output. Then, after 1x1 convolution, normalization, and ReLU activation function processing in the identity block, a 3x3 convolution operation, normalization, and ReLU activation function processing are performed, and then 1x1 convolution and normalization operations are performed. The number of convolution layers in the identity block is 32, 32, and 32 respectively. Finally, the input features are connected to the output part through residual connection, and the output feature map with the size of 128x128x32 is obtained. After the maximum pooling operation of 2x2, the feature map with the size of 64x64x32 is obtained. The input of the third residual module is obtained.

[0050] The processing method of the third residual module is the same as above. The number of convolutional layers in the convolutional block is replaced by 64, and the number of convolutional layers in the identity block is replaced by 64. After the maximum pooling operation, a 32x32x64 feature map is obtained, which is the input of the fourth residual module.

[0051] The processing method of the fourth residual module is the same as above. The number of convolutional layers in the convolutional block is replaced by 128, and the number of convolutional layers in the identity block is replaced by 128. After the maximum pooling operation, a 16x16x128 feature map is obtained, which is the input of the fifth residual module.

[0052] The fifth residual module is processed in the same way as above, except that the number of convolutional layers in the convolutional block is replaced by 256, and the number of convolutional layers in the identity block is replaced by 256. Then, the nearest neighbor interpolation is used to perform a two-fold upsampling operation to obtain a feature map of size 32x32x128 pixels.

[0053] The upsampled feature map will be spliced ​​with the corresponding feature map of the encoder. After splicing, the number of channels of the feature map will increase. During the splicing process, a multi-head cross-attention mechanism module is added. The multi-head cross-attention mechanism module is used after the jump connection. It mainly combines the feature map with richer high-level semantics with the high-resolution map from the jump connection. The core idea of ​​the multi-head cross-attention mechanism module is to find a way to filter out irrelevant or noisy areas in the jump connection and highlight the relevant areas. The inputs of the multi-head cross-attention mechanism module are the result S from the jump connection (the output of each residual module) and the result Y after processing by the previous layer of decoder (the output of each decoder deconvolution module). Specifically:

[0054] The input of the first decoder deconvolution module is the fifth oil spill feature map output by the fifth residual module in the ResNet50 network, and the output of the first decoder deconvolution module is the first oil spill recovery feature map.

[0055] The fourth oil spill feature map output by the fourth residual module in the ResNet50 network and the first oil spill recovery feature map are spliced ​​to obtain a first spliced ​​feature map; the first spliced ​​feature map is weighted by the first multi-head cross attention mechanism module to obtain a first oil spill weighted feature map; the input of the second decoder deconvolution module is the first oil spill weighted feature map, and the output of the second decoder deconvolution module is the second oil spill recovery feature map.

[0056] The third oil spill feature map output by the third residual module in the ResNet50 network and the second oil spill recovery feature map are spliced ​​to obtain a second spliced ​​feature map; the second spliced ​​feature map is weighted by the second multi-head cross attention mechanism module to obtain a second oil spill weighted feature map; the input of the third decoder deconvolution module is the second oil spill weighted feature map, and the output of the third decoder deconvolution module is the third oil spill recovery feature map.

[0057] The second oil spill feature map output by the second residual module in the ResNet50 network and the third oil spill recovery feature map are spliced ​​to obtain a third spliced ​​feature map; the third spliced ​​feature map is weighted by the third multi-head cross attention mechanism module to obtain a third oil spill weighted feature map; the input of the fourth decoder deconvolution module is the third oil spill weighted feature map, and the output of the fourth decoder deconvolution module is the fourth oil spill recovery feature map.

[0058] The first oil spill feature map output by the first residual module in the ResNet50 network and the fourth oil spill recovery feature map are spliced ​​to obtain a fourth spliced ​​feature map; the fourth spliced ​​feature map is weighted by the fourth multi-head cross attention mechanism module to obtain a fourth oil spill weighted feature map; the input of the fifth decoder deconvolution module is the fourth oil spill weighted feature map, and the output of the fifth decoder deconvolution module is the fifth oil spill recovery feature map.

[0059] The inputs of the multi-head cross attention mechanism module are the result S of the jump connection and the result Y after processing in the previous layer. The result after Y is embedded is used as K and Q in the attention mechanism, and the result output by S is used as V. That is, Q and K are taken from the current decoding features, and V is taken from the corresponding encoding features. The combination of features consists of two ways: one is to use Q and K to obtain the weight matrix of the current feature and multiply it by V to obtain the information of the important part in the segmentation, then upsample, and then take the position-encoded S for dot multiplication. The second is to upsample the position-encoded Y and then perform a convolution operation. Finally, the final features of one and two are spliced ​​together to obtain the new decoding features. Specifically:

[0060] The i-th multi-head cross attention mechanism module uses the features from the residual module in the i-th concatenated feature map as the query Q and key K in the attention mechanism, and uses the features from the decoder deconvolution module in the i-th concatenated feature map as the value V in the attention mechanism; where i = 1, 2, 3, 4. The attention weight matrix is ​​calculated based on the query Q and key K, and the attention weight matrix is ​​multiplied by the value V and then upsampled to obtain the first feature matrix; the features from the residual module are positionally encoded and dot-multiplied with the first feature matrix to obtain the first global feature map; the features from the decoder deconvolution module are positionally encoded and upsampled to obtain the second global feature map; the first global feature map and the second global feature map are concatenated to obtain the output of each multi-head cross attention mechanism module.

[0061] The first decoder deconvolution module in the decoder uses two ordinary convolutional layers, each of which includes 128 convolution kernels, each of which is of size (3,3), and uses the ReLU activation function to implement the decoding operation. The remaining decoder deconvolution modules are the same as above, all undergoing two ordinary convolutional layers, with the size of the convolution kernels being (3,3), but the number of convolution kernels is changed to 64, 32, and 16 respectively. The fifth decoder deconvolution module uses a 1x1 convolution operation to obtain a feature map of size 256x256x1 pixels, which is used as the final output of the improved U-Net network.

[0062] The above improved Unet network has the following improvements:

[0063] 1. The convolutional blocks and identity block structures in ResNet50 replace the ordinary convolutional layers. By introducing residual connections, the network can be deepened and the gradient vanishing problem can be solved. The input of the convolution operation is concatenated with the output of the convolution operation to supplement the oil spill features lost during the convolution operation, thus avoiding the loss of isolated oil spill area features during the convolution process. In the feature extraction of offshore oil spill areas, complex image features can be better captured and oil spill area features can be more accurately identified.

[0064] 2. A multi-head cross-attention mechanism is introduced into the jump connection of the U-Net network to interact the features of the encoder and decoder, extract and fuse key features, and reduce the impact of noise. The multi-head cross-attention mechanism allows the model to focus on the most relevant features in the encoder during the decoding stage, so that the model can more accurately identify and locate the oil spill area during the feature extraction process of the offshore oil spill area. Since the multi-head cross-attention mechanism module can capture the long-distance dependencies in the spliced ​​feature map, the output of the ResNet50 network of the same scale is spliced ​​with the input of the decoder and then input into the multi-head cross-attention mechanism module. Only those supplementary features that are significantly associated with other oil spill features can be given higher weights, instead of weighting all features supplemented by residual connections. The useful information in the supplementary features is transmitted, making the isolated oil spill area information obtained more accurate.

[0065] S2: Collect offshore oil spill images to build a dataset, use the dataset to train the improved U-Net network, and obtain an offshore oil spill detection model.

[0066] Firstly, a large number of oil spill data sets from different regions of the world are selected to train and evaluate the performance of the proposed framework. Each type is divided into a training set and a test set in a ratio of 7:3. In addition, the SAR images in the data set are geometrically corrected, and 700 SAR images are obtained for constructing the data set. Then Python is used to process the SAR images in blocks: open the input digital elevation model file, obtain the spectral bands of red R, green G, and blue B; obtain the width (number of columns) and height (number of rows) of the image from the attributes of the red band, divide the width and height of the image by the pixel size, and the pixel size is 256, respectively, to obtain the integer number of rows and columns to divide the image into fixed-size blocks. Then calculate the remainder of the image width and height divided by the pixel size to process incomplete pixel blocks. Then use a double loop to process the SAR image in blocks. The outer loop is responsible for traversing the rows of the image, and then the inner loop traverses the columns of each row, so that the image can be accessed and processed block by block without loading the entire image into memory at one time, which improves the efficiency of data processing. In the loop, first read a part of the image data from the red, green and blue channels to form an RGB image block. Then convert the RGB image block to a grayscale image by calling the function. Finally, check whether white pixels need to be added according to the conditions. If the conditions are met (the height and width still have remainders after segmentation and the current image block is in the last row or column of the image), adjust the size of the grayscale image block and add white pixels to the edge of the image block. Finally, multiply the pixel value of the processed grayscale image block by 512 and convert it to a 16-bit unsigned integer type. Save the processed image block data as a file in ".TIFF" format. Use the function to splice the processed image blocks (or partitions) into a whole image.

[0067] Next, the improved U-Net network is trained using the dataset. The remote sensing images in the training set are converted into 256x256x1 pixel images through preprocessing and input into CNN. During the model training process, the performance of the model is evaluated using precision, recall, and F1-score.

[0068] S3: Input the marine oil spill image to be detected into the marine oil spill detection model, perform convolution operation on the marine oil spill image to be detected by improving the encoder to reduce the size of the marine oil spill image to be detected, and during the convolution operation, splice the input of the convolution operation with the output of the convolution operation through the splicing module to compensate for the loss of oil spill features caused by the size reduction of the marine oil spill image to be detected during the convolution operation, so as to obtain oil spill feature maps of different scales; perform deconvolution operation on the oil spill feature map of the smallest scale by improving the first decoder deconvolution module in the decoder to restore the size of the oil spill feature map; before the remaining decoder deconvolution modules perform deconvolution operation, transmit the oil spill feature map of the corresponding scale to the input of the corresponding decoder deconvolution module through jump connection. At the end, the oil spill recovery feature map output from the previous decoder deconvolution module is spliced ​​to obtain a spliced ​​feature map of the corresponding scale; and the long-distance dependency in the spliced ​​feature map is captured through the multi-head cross attention mechanism module to obtain the association between the oil spill features supplemented by the splicing module in the splicing feature map and other oil spill features in the splicing feature map, and the oil spill features supplemented by the residual connection module that are associated with other oil spill features in the splicing feature map are weighted to obtain an oil spill weighted feature map of the corresponding scale; the residual decoder deconvolution module performs a deconvolution operation on the oil spill weighted feature map of the corresponding scale to obtain oil spill recovery feature maps of different scales; the pixels in the oil spill recovery feature map of the largest size are classified into oil spill pixels and non-oil spill pixels, and the oil spill area in the marine oil spill image to be detected is output.

[0069] The above is a method for detecting oil spills at sea provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding device for detecting oil spills at sea, including:

[0070] A model building module is provided for building an improved encoder by replacing the encoder in the original U-Net network with a ResNet50 network, wherein the ResNet50 network includes a splicing module for splicing the input of the convolution operation and the output of the convolution operation; building an improved decoder by adding a multi-head cross attention mechanism module at the input end of the deconvolution module of the original U-Net network decoder; and building an improved U-Net network including an improved encoder, an improved decoder and a jump connection of the original U-Net network.

[0071] The model training module is used to collect offshore oil spill images to build a data set, and use the data set to train the improved U-Net network to obtain an offshore oil spill detection model.

[0072] The oil spill detection module is used to input the marine oil spill image to be detected into the marine oil spill detection model, and perform convolution operation on the marine oil spill image to be detected by improving the encoder to reduce the size of the marine oil spill image to be detected. During the convolution operation, the input of the convolution operation is spliced ​​with the output of the convolution operation by the splicing module to compensate for the loss of oil spill features caused by the size reduction of the marine oil spill image to be detected during the convolution operation, so as to obtain oil spill feature maps of different scales; the first decoder deconvolution module in the improved decoder is used to perform deconvolution operation on the oil spill feature map of the smallest scale to restore the size of the oil spill feature map; before the remaining decoder deconvolution modules perform deconvolution operation, the oil spill feature map of the corresponding scale is transmitted to the corresponding decoder deconvolution module through jump connection. The input end is used to splice the oil spill recovery feature map output by the previous decoder deconvolution module to obtain a spliced ​​feature map of the corresponding scale; and the long-distance dependency in the spliced ​​feature map is captured through the multi-head cross attention mechanism module to obtain the association between the oil spill features supplemented by the splicing module in the splicing feature map and other oil spill features in the splicing feature map, and the oil spill features supplemented by the residual connection module that are associated with other oil spill features in the splicing feature map are weighted to obtain an oil spill weighted feature map of the corresponding scale; the residual decoder deconvolution module performs a deconvolution operation on the oil spill weighted feature map of the corresponding scale to obtain oil spill recovery feature maps of different scales; the pixels in the oil spill recovery feature map of the largest size are classified into oil spill pixels and non-oil spill pixels, and the oil spill area in the marine oil spill image to be detected is output.

[0073] The specific definition of the offshore oil spill detection device can be found in the definition of the offshore oil spill detection method above, which will not be repeated here. Each module in the above-mentioned offshore oil spill detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0074] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provided methods for detecting marine oil spills.

[0075] The present invention also provides a computer device structure. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Provided methods for detecting marine oil spills.

[0076] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0077] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A method for detecting oil spills at sea, characterized in that: include: An improved encoder is constructed by replacing the encoder in the original U-Net network with a ResNet50 network, wherein the ResNet50 network includes a splicing module for splicing the input of the convolution operation and the output of the convolution operation; an improved decoder is constructed by adding a multi-head cross attention mechanism module at the input end of the deconvolution module of the original U-Net network decoder; and an improved U-Net network including an improved encoder, an improved decoder and a jump connection of the original U-Net network is constructed; Collect offshore oil spill images to build a data set, use the data set to train the improved U-Net network, and obtain an offshore oil spill detection model; The offshore oil spill image to be detected is input into the offshore oil spill detection model, and the offshore oil spill image to be detected is convolved by improving the encoder to reduce the size of the offshore oil spill image to be detected. During the convolution operation, the input of the convolution operation is spliced ​​with the output of the convolution operation by a splicing module to compensate for the loss of oil spill features caused by the size reduction of the offshore oil spill image to be detected during the convolution operation, so as to obtain oil spill feature maps of different scales; the first decoder deconvolution module in the improved decoder is used to perform a deconvolution operation on the minimum-scale oil spill feature map to restore the size of the oil spill feature map; Before the residual decoder deconvolution module performs deconvolution operation, the oil spill feature map of the corresponding scale is transmitted to the input end of the corresponding decoder deconvolution module through jump connection to be spliced ​​with the oil spill recovery feature map output from the previous decoder deconvolution module to obtain the spliced ​​feature map of the corresponding scale; and the long-distance dependency in the spliced ​​feature map is captured through the multi-head cross attention mechanism module to obtain the association between the oil spill features supplemented by the splicing module in the splicing feature map and other oil spill features in the splicing feature map, and the oil spill features supplemented by the residual connection module that are associated with other oil spill features in the splicing feature map are weighted to obtain the oil spill weighted feature map of the corresponding scale; the residual decoder deconvolution module performs deconvolution operation on the oil spill weighted feature map of the corresponding scale to obtain oil spill recovery feature maps of different scales; the pixels in the oil spill recovery feature map of the largest size are classified into oil spill pixels and non-oil spill pixels, and the oil spill area in the marine oil spill image to be detected is output.

2. The method for detecting oil spill at sea as claimed in claim 1, characterized in that: The ResNet50 network includes 5 residual modules connected in sequence, and the standard convolution layer in each residual module is replaced by a convolution block and an identity block structure connected in sequence; the convolution block is used to adjust the channel and size of the oil spill feature map; the identity block is used to increase the depth of the network without changing the channel and size of the oil spill feature map to prevent the gradient from disappearing during the oil spill feature transmission process.

3. The method for detecting oil spill at sea as claimed in claim 2, characterized in that: The step of transmitting the oil spill feature map of the corresponding scale to the input end of the corresponding decoder deconvolution module through the jump connection to be spliced ​​with the oil spill recovery feature map output from the previous decoder deconvolution module to obtain the spliced ​​feature map of the corresponding scale specifically includes: The fourth oil spill feature map output by the fourth residual module in the ResNet50 network and the first oil spill recovery feature map output by the first decoder deconvolution module are spliced ​​to obtain a first spliced ​​feature map; The third oil spill feature map output by the third residual module in the ResNet50 network and the second oil spill recovery feature map output by the second decoder deconvolution module are spliced ​​to obtain a second spliced ​​feature map; The second oil spill feature map output by the second residual module in the ResNet50 network and the third oil spill recovery feature map output by the third decoder deconvolution module are spliced ​​to obtain a third spliced ​​feature map; The first oil spill feature map output by the first residual module in the ResNet50 network and the fourth oil spill recovery feature map output by the fourth decoder deconvolution module are spliced ​​to obtain a fourth spliced ​​feature map.

4. The method for detecting oil spill at sea as claimed in claim 3, characterized in that: The method adds a multi-head cross attention mechanism module to the input of the original U-Net network decoder deconvolution module, specifically including: In the original U-Net network, the first multi-head cross-attention mechanism module, the second multi-head cross-attention mechanism module, the third multi-head cross-attention mechanism module and the fourth multi-head cross-attention mechanism module are added to the input ends of the second decoder deconvolution module, the third decoder deconvolution module, the fourth decoder deconvolution module and the fifth decoder deconvolution module respectively: The i-th multi-head cross attention mechanism module is used to capture the long-distance dependency in the i-th spliced ​​feature map, and output the i-th oil spill weighted feature map corresponding to the i-th spliced ​​feature map, i=1,2,3,4; specifically includes: The i-th multi-head cross attention mechanism module uses the oil spill feature map from the splicing module in the i-th splicing feature map as the query Q and key K in the attention mechanism, and uses the oil spill recovery feature map from the decoder deconvolution module in the i-th splicing feature map as the value V in the attention mechanism; The attention weight matrix is ​​calculated based on the query Q and the key K, and the attention weight matrix is ​​multiplied by the value V and then upsampled to obtain a first feature matrix; the oil spill feature map from the residual module is position-encoded and dot-multiplied with the first feature matrix to obtain a first oil spill weighted feature map; the oil spill recovery feature map from the decoder deconvolution module is position-encoded and upsampled to obtain a second oil spill weighted feature map; The first oil spill weighted feature map and the second oil spill weighted feature map are spliced ​​to obtain the i-th oil spill weighted feature map output by the i-th multi-head cross attention mechanism module.

5. A marine oil spill detection device, characterized in that: include: A model building module, for building an improved encoder by replacing the encoder in the original U-Net network with a ResNet50 network, wherein the ResNet50 network includes a splicing module for splicing the input of the convolution operation and the output of the convolution operation; building an improved decoder by adding a multi-head cross attention mechanism module at the input end of the deconvolution module of the original U-Net network decoder; and building an improved U-Net network including an improved encoder, an improved decoder and a jump connection of the original U-Net network; The model training module is used to collect offshore oil spill images to build a data set, and use the data set to train the improved U-Net network to obtain an offshore oil spill detection model; The oil spill detection module is used to input the marine oil spill image to be detected into the marine oil spill detection model, and perform convolution operation on the marine oil spill image to be detected by improving the encoder to reduce the size of the marine oil spill image to be detected. During the convolution operation, the input of the convolution operation is spliced ​​with the output of the convolution operation by the splicing module to compensate for the loss of oil spill features caused by the size reduction of the marine oil spill image to be detected during the convolution operation, so as to obtain oil spill feature maps of different scales; the first decoder deconvolution module in the improved decoder is used to perform deconvolution operation on the oil spill feature map of the smallest scale to restore the size of the oil spill feature map; Before the residual decoder deconvolution module performs deconvolution operation, the oil spill feature map of the corresponding scale is transmitted to the input end of the corresponding decoder deconvolution module through jump connection to be spliced ​​with the oil spill recovery feature map output from the previous decoder deconvolution module to obtain the spliced ​​feature map of the corresponding scale; and the long-distance dependency in the spliced ​​feature map is captured through the multi-head cross attention mechanism module to obtain the association between the oil spill features supplemented by the splicing module in the splicing feature map and other oil spill features in the splicing feature map, and the oil spill features supplemented by the residual connection module that are associated with other oil spill features in the splicing feature map are weighted to obtain the oil spill weighted feature map of the corresponding scale; the residual decoder deconvolution module performs deconvolution operation on the oil spill weighted feature map of the corresponding scale to obtain oil spill recovery feature maps of different scales; the pixels in the oil spill recovery feature map of the largest size are classified into oil spill pixels and non-oil spill pixels, and the oil spill area in the marine oil spill image to be detected is output.

6. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

7. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of claims 1 to 4 is implemented.

Images