A multi-task generative adversarial network-based SAR image oil spill classification and semantic segmentation method

Abstract

A multi-task generative adversarial network-based SAR image oil spill classification and semantic segmentation method belongs to the interdisciplinary field of marine remote sensing and artificial intelligence. It includes the classification and identification of oil spills and oil spill similarities and the semantic segmentation of oil spill images. First, the discriminator and classifier of the generative adversarial network are combined to achieve the classification and discrimination of oil spills and oil spill similarities; secondly, the oil spill image after discrimination enters the next generative adversarial network; finally, the generator is used to generate a segmentation map of the oil spill image, thereby achieving semantic segmentation. The present invention can solve the problems of difficulty in distinguishing oil spills and oil spill similarities and low oil spill detection accuracy. The entire model framework can achieve both the classification of oil spills and oil spill similarities and the semantic segmentation of oil spill images. All tasks can be completed within a single framework. The method proposed in the present invention achieves high accuracy in both oil spill classification tasks and oil spill area detection, and can be used to monitor marine oil spills in SAR images.

Description

Technical Field

[0001] The present invention belongs to the field of intersectional technology of marine remote sensing and artificial intelligence, and relates to a SAR image oil spill classification and semantic segmentation method based on a multi-task generative adversarial network. Background Art

[0002] The ocean is a valuable resource for humanity, providing us with abundant mineral resources, aquatic biological resources, and more. With the rapid development of science and technology, the ocean is suffering from varying degrees of pollution, with oil pollution being a particularly prominent problem. Oil spills in the ocean not only have a serious impact on the marine environment but also harm marine life. Using satellite remote sensing to monitor marine oil spills can promptly detect spills and address them before a disaster occurs, effectively reducing marine pollution. Synthetic Aperture Radar (SAR) remote sensing images have the ability to observe the ground around the clock and in all weather conditions, and their coverage is extensive. Therefore, SAR images have become an important tool for oil spill monitoring in remote sensing oil spill monitoring methods.

[0003] Marine oil spills appear as black areas in SAR images. However, distinguishing between spilled oil and oil-like objects remains a challenge when using SAR imagery for oil spill monitoring. This is because SAR images often contain a large number of oil-like objects, such as low-wind areas, internal waves, and eddies, which can appear as black areas in SAR images and lead to misidentification. Therefore, a major challenge in extracting the extent of an oil spill from SAR imagery is accurately distinguishing between spilled oil and oil-like objects.

[0004] When it comes to oil spill area extraction, traditional machine learning methods cannot guarantee effective oil spill detection. In recent years, deep learning has achieved promising results in oil spill detection, but most methods rely on large amounts of training data to ensure accurate detection results. However, large amounts of oil spill data are difficult to obtain in reality. Therefore, a current limitation of deep learning is how to solve the problem of training deep learning networks with small sample sizes.

[0005] To address the above two issues, research that can both classify and identify oil spills and extract information about oil spill areas has yet to be conducted. Therefore, a multi-task generative adversarial network (GAN) was designed that can achieve both oil spill classification and information extraction within a single framework. The discriminator of the GAN is used to accurately distinguish between oil spills and oil spill-like objects, while the generator can not only generate semantic segmentation maps of the oil spill area, but also expand the number of samples and increase adversarial samples to improve the generalization ability of the discriminant. This achieves the effect of requiring only a small number of training samples as a training set during the training process. Therefore, the multi-task GANs network can be used as a method for oil spill classification and identification, as well as to generate oil spill detection results and perform oil spill segmentation tasks, effectively improving the accuracy of oil spill detection. Summary of the Invention

[0006] This invention primarily addresses the difficulty in identifying spilled oil and oil-like objects in marine oil spills, as well as detecting spilled areas. A multi-task generative adversarial network-based method for SAR imagery oil spill classification and semantic segmentation is proposed. The model is constructed using two generative adversarial networks: the first for classifying and identifying spilled oil and oil-like objects, and the second for semantic segmentation of oil spill images. This framework achieves both oil spill classification and oil spill area segmentation.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A multi-task generative adversarial network (GAN)-based method for oil spill classification and semantic segmentation in SAR images involves two steps: classification and recognition of oil spills and oil spill mimics, and semantic segmentation of oil spill images. The classification and recognition approach combines the discriminator and classifier of a GAN to achieve classification and recognition of oil spills and oil spill mimics. The discriminated oil spill image then enters the next GAN, where the generator generates a segmentation map of the oil spill image, thereby achieving semantic segmentation. Using two GANs, a multi-task GANs network model is ultimately constructed. The steps involved are:

[0009] The first step is remote sensing image preprocessing. SAR data processing operations include thermal noise removal, radiometric calibration, speckle filtering, and terrain correction. Both oil spill and oil spill mimic data require data preprocessing.

[0010] 1.1) Thermal noise removal: The acquired SAR images contain thermal noise, which affects the accuracy of the radar backscatter signal and needs to be removed.

[0011] 1.2) Radiometric calibration: Due to the penetrability of clouds, SAR data requires radiometric calibration. Radiometric calibration eliminates sensor errors and determines the exact radiometric value at the sensor entrance.

[0012] 1.3) Speckle filtering. Speckle is a common phenomenon in SAR images. Speckle filtering can reduce speckle noise. There are many filters that can remove speckle. The Refined Lee filter used here is the most commonly used speckle filter and provides excellent processing results.

[0013] 1.4) Terrain correction: The final step is terrain correction. Terrain correction is to ensure that the real geographic coordinates correspond to the remote sensing image. The range Doppler method is used here for terrain correction.

[0014] After the original images were processed using the above methods, the oil spill and oil spill similarities were selected through expert interpretation, and then the oil spill and oil spill similarity areas were cropped out using the SNAP tool to obtain the oil spill image and oil spill similarity image. The label value of the oil spill similarity image was marked as 0, and the label value of the oil spill image was marked as 1. After marking, they were used as the input of the first generative adversarial network.

[0015] In the second step, the pre-processed images of oil spills and oil spill mimics are input into the network. The input images are passed through the first generative adversarial network to complete the classification task of oil spills and oil spill mimics, realizing the distinction between oil spills and oil spill mimics (the classification operation in this step is completed by the discriminator of the first generative adversarial network, that is, the discriminator in step 2.5)). The details are as follows:

[0016] 2.1) Build the first generative adversarial network model, consisting of the generator G c and the discriminator D c The network of the generator is mainly composed of a convolution layer, a LeakyReLU activation function, an upsampling layer and a normalization layer. The upsampling layer, convolution layer, normalization layer and LeakyReLU activation function are regarded as a whole module, and the four layers in the whole module are connected end to end. The structure of the generator is first a normalization layer, then two or more whole modules, and finally a convolution layer and a Tanh activation function. The whole module is composed of an upsampling layer, a convolution layer, a normalization layer and a LeakyReLU activation function. The input of the generator is random noise z, and then a linear transformation is performed. The linear transformation formula is shown in (1).

[0017] y=zA T +c (1)

[0018] Where z is the random noise input, A is the weight, c is the bias, and y is the linear transformation output.

[0019] 2.2) After linear transformation, the data enters the normalization layer to solve the problem of data distribution changes during training, prevent gradient disappearance or explosion, and speed up training.

[0020] 2.3) After the normalization layer, the data must pass through the generator's upsampling layer, convolution layer, normalization layer, and LeakyReLU activation function in sequence. The upsampling layer increases the image's length and width by a factor of two before passing through the convolution layer. The convolution layer formula is shown in (2).

[0021]

[0022] Among them, a is the output of the convolution layer; W g is the weight of the g-th convolution kernel; x i is the i-th input; b g is the g-th bias; I is the total number of inputs; G is the total number of convolution kernels.

[0023] In order to prevent gradient disappearance and overfitting during feature extraction network training, a normalization layer is added after the convolution layer. The output of the convolution layer is used as the input of the normalization layer as shown in formula (3).

[0024]

[0025] Where ε is the minimum value automatically generated by the system, and n is the output of the convolutional layer.

[0026] In order to enable the feature extraction network to converge quickly, the LeakReLU activation function is added after the normalization layer, as shown in the formula (4).

[0027]

[0028] The value of l is usually around 0.01. L(n) is the output of the LeakReLU function.

[0029] 2.4) After passing through the above-mentioned upsampling layer, convolution layer, normalization layer, and LeakyReLU activation function, the same process is repeated (the data is again processed by the upsampling layer, convolution layer, normalization layer, and LeakyReLU activation function). The final output is then passed through another convolution layer and the Tanh activation function. The formula for the Tanh activation function is shown in (5).

[0030]

[0031] Where L is the output of the final LeakyReLU function, and Tanh(L) is the output of the Tanh activation function. The generator will eventually generate pseudo samples that are close to oil spills and oil spill-like objects, which are then input into the discriminator.

[0032] 2.5) Discriminator

[0033] The discriminator consists of four convolution blocks, each of which consists of a convolution layer, a LeakyReLU activation function, and a Dropout layer connected in sequence. Adding a Dropout layer can prevent the model from overfitting and improve the generalization ability of the model. The sigmoid and softmax activation functions are connected at the end of the discriminator. An output layer is added after the last convolution block of the discriminator, and the output of the last convolution block serves as the input of the sigmoid and softmax activation functions in the output layer. The formula for the sigmoid function is shown in (6). The formula for the softmax function is shown in (7). The sigmoid function is used to distinguish true from false. The softmax function is used for classification, converting it into a multi-classification problem.

[0034]

[0035]

[0036] The discriminator's input consists of real samples and fake samples. The real samples are the oil spill images and oil spill mimics from the first step, while the fake samples are the oil spills and oil spill mimics generated by the generator. When these two samples are fed together, the discriminator is not only able to distinguish between real and fake samples but also to classify the real samples. The discriminator in the first generative adversarial network learns the characteristics of oil spills and oil spill mimics, automatically distinguishing them from unknown images. For unlabeled images, the final softmax function in the second step outputs class probabilities. The class with the highest probability is determined as the class. For example, if the probability of class 0 is high, the output is 0, indicating that the image is an oil spill mimic. If the output is 1, it indicates that the image is an oil spill image. The output values ​​can be used to distinguish between oil spills and oil spill mimics.

[0037] In the third step, after the images are classified into oil spills and oil spill-like objects in the second step, some of the classified oil spill images may contain misclassification errors. We need to remove oil spill-like images. Only oil spill images are used as input for the fourth step, so the third step is a judgment operation. Since we previously defined the label value of oil spill-like images as 0 and the label value of oil spill images as 1, we retain images with a label value of 1 in the second step and the remaining images are not used as input for the fourth step.

[0038] In the fourth step, the oil spill images filtered in the third step are run through a second GAN to perform semantic segmentation. This semantic segmentation task extracts the oil spill area. The second GAN also consists of a generator and a discriminator. The following is the details of the second GAN:

[0039] 4.1) Build a segmentation network model, consisting of the generator G s and the discriminator D s The generator network consists of a U-net network with four downsampling and four upsampling steps, forming a U-shaped structure. The U-net network is followed by a convolutional layer, followed by three convolutional blocks, each of which has a convolutional layer, a normalization layer, and a LeakyReLU activation function. Finally, there is a convolutional layer and a Tanh activation function. The discriminator is constructed with five convolutional blocks, each of which contains a convolutional layer, a normalization layer, and a LeakyReLU activation function. The last convolutional block contains only a convolutional layer.

[0040] The generator takes as input the original oil spill image, which is the filtered oil spill image from the third step, and outputs the generated oil spill segmentation map. The discriminator takes as input the original oil spill image, the oil spill segmentation map generated by the generator, and the oil spill segmentation label map. The oil spill segmentation label map is the oil spill area determined by expert interpretation, and then the area is drawn using the labelme tool to serve as the oil spill segmentation label map. The training process of the second generative adversarial network is as follows: First, the oil spill segmentation map initially generated by the generator does not approximate the oil spill segmentation label map. Then, the discriminator uses the original oil spill image, the oil spill segmentation map, and the oil spill segmentation label map as input to perform a discrimination. The original oil spill image and the oil spill segmentation label map allow the discriminator to learn which areas in the original oil spill image are oil spill areas. The discriminator then distinguishes between the oil spill segmentation map generated by the generator and the oil spill segmentation label map, distinguishing whether the image is the oil spill segmentation label map or the oil spill segmentation map generated by the generator, learning the differences in data distribution between them, and outputting a discrimination score. The discriminator's results are used to reverse-optimize the generator, guiding it to produce more accurate oil spill segmentation maps. Ultimately, the generator can produce realistic oil spill segmentation maps, successfully deceiving the discriminator. The discriminator is unable to distinguish between the generator-generated oil spill segmentation map and the oil spill segmentation label map. After training, the trained generator can be used to generate oil spill segmentation maps, achieving accurate extraction of the oil spill area.

[0041] The objective function of the generator of the second generative adversarial network (the generative adversarial network here consists of a generator and a discriminator. During the training process, the goal of the generator is to generate as many real images as possible to deceive the discriminator. The goal of the discriminator is to distinguish the images generated by the generator from the real images as much as possible. In this way, the generator and the discriminator constitute a dynamic game process) is shown in the following formula (8):

[0042]

[0043] Among them, λ1 is the weight balance parameter, S is the input true segmentation label map, is the segmentation map generated by the generator, and I is the real oil spill image. is the L1 norm, which penalizes the pixel-by-pixel distance between the segmentation label map and the generated segmentation map. Minimizing the training generator to generate more realistic oil spill segmentation maps can ultimately successfully deceive the discriminator.

[0044] The objective function of the second generative adversarial network discriminator is shown in Equation (9):

[0045]

[0046] Among them, λ2 is the weight balance parameter, S is the input true segmentation label map, is the segmentation map generated by the generator, and I is the real oil spill image. This term represents the adversarial loss and increases the discriminative ability of the discriminator. Represents the penalty gradient loss, which produces a stable gradient that neither disappears nor explodes. Indicates that in S and Minimize the training of the discriminator to distinguish the generated oil spill segmentation map from the oil spill segmentation label map.

[0047] 4.2) The overall objective function of the second generative adversarial network is shown in (10):

[0048]

[0049] During the segmentation phase, a second GAN is used (in this invention, the first GAN only performs the classification task, while the second GAN performs the segmentation task). A U-net structure is added to the network's generator to improve the accuracy of the generated oil spill segmentation map. The generator and discriminator are continuously optimized and trained. The network is trained using the loss formula (10) until the second GAN reaches equilibrium. After training, the generator is retained and the discriminator is discarded. The generator is then used to generate the oil spill segmentation map.

[0050] The beneficial effects of the present invention are:

[0051] The SAR image oil spill classification and semantic segmentation method proposed in the present invention solves the problems of difficulty in distinguishing between oil spills and oil spill similarities and low oil spill detection accuracy. A generative adversarial network is used to simultaneously achieve oil spill classification and segmentation tasks, with the discriminator of the generative adversarial network used as a classifier and the generator used to generate semantic segmentation results. In this way, the entire model framework can not only achieve the classification of oil spills and oil spill similarities, but also achieve semantic segmentation of oil spill images. Ultimately, a complete oil spill detection framework is formed. All tasks can be completed within a single framework. The method proposed in the present invention achieves high accuracy in both oil spill classification tasks and oil spill area detection, and can be used to monitor marine oil spills in SAR images. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The overall framework of a multi-task generative adversarial network-based SAR image oil spill classification and semantic segmentation method;

[0053] Figure 2 The original images after preprocessing, (a) is the original image, (b) is the cropped oil spill image, and (d) is the cropped oil spill similarity image.

[0054] Figure 3 The discriminant results after preprocessing are shown in Figure 1. (a) and (c) are distinguished as oil spill images, and (b) is distinguished as an oil spill-like image.

[0055] Figure 4 Details of the results after segmentation of the Sentinel-1 satellite oil spill image: (a) is the original image of the oil spill area, (b) is the segmented label image of the oil spill area, and (c) is the result of the patented method, with an accuracy of 98.88% and a precision of 93.59%.

[0056] Figure 5 Details of the extraction results after segmentation of the ERS-1 satellite oil spill image: (a) the original image of the oil spill area, (b) the segmented label image of the oil spill area, and (c) the result of the patented method, with an accuracy of 96.62% and a precision of 91.93%.

[0057] Figure 6 Details of the extraction results after segmentation of the GF-3 satellite oil spill image: (a) the original image of the oil spill area, (b) the segmented label image of the oil spill area, and (c) the result of the patented method, with an accuracy of 98.77% and a precision of 91.80%. DETAILED DESCRIPTION

[0058] To make the method problems solved by the present invention, the method solutions adopted, and the method effects achieved more clearly, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the drawings only show portions relevant to the present invention, not all of the contents.

[0059] like Figure 1 As shown, an embodiment of the present invention provides a method for oil spill classification and semantic segmentation of SAR images using a multi-task generative adversarial network, including:

[0060] The program was compiled in Python 3.8, PyTorch 1.7.1, and Cuda 10.1 under Windows 10, and run on an RTX 2080 GPU. The input data size was 256×256 SAR images.

[0061] The first step is remote sensing image preprocessing. SAR data processing operations include thermal noise removal, radiometric calibration, speckle filtering, and terrain correction. Both oil spill data and oil spill mimic data must undergo data preprocessing.

[0062] 1.1) Thermal noise removal: Thermal noise removal can improve the thermal noise problem. Use SNAP software to perform Remove Thermal Noise with default parameters.

[0063] 1.2) Radiometric calibration. Due to the penetrability of clouds, SAR data requires radiometric calibration. Radiometric calibration eliminates sensor errors and determines the exact radiometric value at the sensor entrance. This is performed using SNAP software with default parameters.

[0064] 1.3) Speckle Filtering: Speckle is a common phenomenon in SAR images. Various filters are available for removing speckle. The Refined Lee filter used here is the most commonly used speckle filter. It is an adaptive filter with excellent processing performance. The Refined Lee filter has a filter window size setting. The default setting in SNAP used in this patent is 7x7.

[0065] 1.4) Terrain correction: The final step is terrain correction. Terrain correction is to ensure that the real geographic coordinates correspond to the remote sensing image. The range Doppler method is used here for terrain correction.

[0066] After the original image is processed by the above methods, the oil spill and oil spill similar objects are selected through expert interpretation, and then the oil spill and oil spill similar object areas are cropped out through the SNAP tool. The obtained oil spill image and oil spill similar object images are marked with the label value of the oil spill similar image as 0 and the label value of the oil spill image as 1, which are used as the input of the first generative adversarial network. Figure 2 The resulting image will be used as input for the second step.

[0067] In the second step, the pre-processed oil spill and oil spill-like images are input into the network. The input images are passed through the first generative adversarial network to distinguish between oil spill and oil spill-like objects. Finally, the discriminator of the first generative adversarial network distinguishes the categories. The detailed network structure is shown in the figure. Figure 1 As shown in the first stage, this part of the network was trained for 300 iterations.

[0068] 2.1) Build the first generative adversarial network model, consisting of the generator G c and the discriminator D c The above sampling layer, convolution layer, normalization layer and LeakyReLU activation function are regarded as an overall module, and the four layers in the overall module are connected end to end. The generator is constructed by first having a normalization layer, then an overall module consisting of two or more sampling layers, convolution layers, normalization layers and LeakyReLU activation functions, and finally a convolution layer and Tanh activation function. The input of the generator is random noise z, which is then linearly transformed. The linear transformation formula is shown in (11).

[0069] y=zA T +c (11)

[0070] Where z is the random noise input, A has weights of [32768, 200], c has biases of

[32768] , and y is the output of the linear transformation.

[0071] 2.2) After the linear transformation, the data is fed into the normalization layer for batch normalization. This addresses the issue of shifting data distribution during training, preventing gradient vanishing or exploding, and speeding up training.

[0072] 2.3) After the normalization layer, the data must pass through the upsampling layer, convolution layer, normalization layer, and LeakyReLU activation function in sequence. The upsampling layer increases the length and width of the image to twice its original size before the convolution layer is performed. The convolution kernel size is 3x3, the stride is 1, and the convolution layer formula is shown in (12).

[0073]

[0074] Among them, a is the output of the convolution layer; W gis the weight of the g-th convolution kernel; x i is the i-th input; b g is the g-th bias; I is the total number of inputs; G is the total number of convolution kernels.

[0075] In order to prevent gradient disappearance and overfitting during feature extraction network training, a normalization layer is added after the convolution layer. The output of the convolution layer is used as the input of the normalization layer as shown in formula (13).

[0076]

[0077] Among them, a is the output of the convolution layer and the input of the normalization layer; G is the total number of convolution kernels; ε is the minimum value automatically generated by the system; and n is the output of the convolution layer.

[0078] In order to enable the feature extraction network to converge quickly, the LeakyReLU activation function is added after the normalization layer, as shown in the formula (14).

[0079]

[0080] Here, the l value is set to 0.2 in this patent; L(n) is the output of the LeakReLU layer.

[0081] 2.4) After passing through the above-mentioned upsampling layer, convolution layer, normalization layer, and LeakyReLU activation function, the same process is repeated. The final output is passed through another convolution layer and Tanh activation function. The formula of Tanh activation function is shown in (15).

[0082]

[0083] Where L is the output of the LeakyReLU activation function, and Tanh(L) is the output of the Tanh activation function. After the generator's final Tanh activation function output, the output is a pseudo sample, which will serve as the input of the discriminator.

[0084] 2.5) Discriminator. The discriminator consists of four convolution blocks. Each convolution block has a convolution layer, a LeakyReLU activation function and a Dropout layer connected in sequence. Because the discriminator has to perform classification tasks, adding a Dropout layer can prevent the model from overfitting and improve the generalization ability of the model. The sigmoid and softmax activation functions are connected at the end of the discriminator. An output layer is added after the last convolution block of the discriminator, and the output of the last convolution block is used as the input of the sigmoid and softmax activation functions in the output layer. The formula of the sigmoid function is shown in (16). The formula of the softmax function is shown in (17). The sigmoid function is used to distinguish between true and false. The softmax function is used for classification. In order to distinguish between oil spills and oil spill similarities, it is converted into a multi-classification problem. In this patent, the number of output channels of the softmax function is set to 3, and the final result is divided into three categories. Each channel represents the probability of oil spills, the probability of oil spill similarities and the probability of false samples. The classification results are shown in Figure 3 shown.

[0085]

[0086]

[0087] The input of the discriminator is composed of true samples and false samples. The oil spill images and oil spill similarities in the first step are true samples, and the oil spill and oil spill similarities generated by the generator are false samples. They are input into the discriminator together. The discriminator can not only distinguish true from false, but also classify true samples. Because the discriminator in the first generative adversarial network will learn the characteristics of oil spill and oil spill similarity images, it will learn to automatically distinguish oil spill and oil spill similarities from unknown images. For images with unlabeled label values, the last softmax function in the second step outputs the category probability value. The category with the higher probability value is determined. For example, if the probability value of the false 0 class is high, 0 will be output in the result, which means that the image is an oil spill similarity. If the output is 1, it means that the image is an oil spill image. The distinction between oil spill and oil spill similarities can be achieved based on the output value. Figure 3 The results of the classification of oil spills and oil spill-like objects are shown.

[0088] In the third step, after the images are classified into oil spills and oil spill-like objects in the second step, some of the classified oil spill images may contain misclassification errors. We need to remove oil spill-like images. Only oil spill images are used as input for the fourth step, so the third step is a judgment operation. Since we previously defined the label value of oil spill-like images as 0 and the label value of oil spill images as 1, we retain images with a label value of 1 in the second step and the remaining images are not used as input for the fourth step.

[0089] In the fourth step, the oil spill images filtered in the third step are passed through the second generative adversarial network for semantic segmentation. The semantic segmentation task is used to obtain the extraction results of the oil spill area. The second generative adversarial network is also built with a generator and a discriminator. The detailed network structure is as follows Figure 1 The following are the details of the second generation adversarial network:

[0090] 4.1) Build a segmentation network model, consisting of the generator G s and the discriminator D s The generator network consists of a U-net network with four downsampling and four upsampling steps, forming a U-shaped structure. The downsampling layer includes a pooling layer and a double-layer convolution layer, which in turn consists of two convolution layers and a ReLU activation function. The upsampling layer includes a deconvolution layer and a double-layer convolution layer, which in turn consists of two convolution layers and a ReLU activation function. The U-net network is followed by a convolution layer, followed by three convolution blocks, each of which includes a convolution layer, a normalization layer, and a LeakyReLU activation function. Finally, there is a convolution layer and a Tanh activation function. The discriminator is constructed with five convolution blocks, each of which includes a convolution layer, a normalization layer, and a LeakyReLU activation function. The last convolution block contains only a convolution layer.

[0091] The generator input is a real oil spill image, which is the oil spill image filtered in the third step, and outputs an oil spill segmentation map. The discriminator inputs include the real oil spill image, the oil spill semantic segmentation map generated by the generator, and the oil spill image segmentation label map. The oil spill segmentation label map is the oil spill area determined by expert interpretation, and then the area is drawn using the Labelme tool to serve as the oil spill segmentation label map. The second generative adversarial network training process is as follows: First, the oil spill segmentation map initially generated by the generator does not approximate the oil spill segmentation label map. Then, the discriminator uses the oil spill image, the oil spill segmentation map, and the oil spill segmentation label map as input to perform a discrimination. The oil spill image and the oil spill segmentation label map allow the discriminator to learn which areas in the oil spill image are oil spill areas. The discriminator distinguishes between the oil spill segmentation map generated by the generator and the oil spill segmentation label map, distinguishing whether the image is the oil spill label map or the oil spill segmentation map generated by the generator, learning the data distribution differences between them, and outputs a discrimination score. The discriminator's results are then used to reverse-optimize the generator, guiding it to produce more accurate oil spill segmentation maps. Ultimately, the generator can produce realistic oil spill segmentation maps, successfully deceiving the discriminator. The discriminator is unable to distinguish between the generator-generated oil spill segmentation map and the oil spill segmentation label map. After training, the trained generator can be used to generate oil spill segmentation maps, achieving accurate extraction of the oil spill area.

[0092] 4.2) The objective function of the generator of the second generative adversarial network is shown in the following formula (18):

[0093]

[0094] Among them, λ1 is the weight balance parameter, S is the input true segmentation label map, is the segmentation map generated by the generator, and I is the real oil spill image. is the L1 norm, which penalizes the pixel-by-pixel distance between the segmentation label map and the generated segmentation map. Minimizing the training generator to generate more realistic oil spill segmentation maps can ultimately successfully deceive the discriminator.

[0095] 4.3) The objective function of the discriminator of the second generative adversarial network is shown in the following formula (19):

[0096]

[0097] Among them, λ2 is the weight balance parameter, S is the input true segmentation label map, is the segmentation map generated by the generator, and I is the real oil spill image. This term represents the adversarial loss and increases the discriminative ability of the discriminator. Represents the penalty gradient loss, which produces a stable gradient that neither disappears nor explodes. Indicates that in S and Minimize the training of the discriminator to distinguish the generated oil spill segmentation map from the oil spill segmentation label map.

[0098] 4.4) The overall objective function of the second generative adversarial network is shown in (20):

[0099]

[0100] The U-net structure is added to the generator of the second GAN to improve the accuracy of the generated segmentation map. The generator and discriminator are continuously optimized and trained until the second GAN reaches a balanced state. After training, the generator is retained to generate the semantic segmentation map. The final oil spill area segmentation result is as follows: Figure 4 、 Figure 5 and Figure 6 Shown are the segmentation results of oil spill images on Sentinel-1, ERS-1 and GF-3 satellites respectively.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the method scheme of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary method personnel in this field should understand that modifying the method schemes described in the above embodiments, or equivalently replacing some or all of the method features therein, does not cause the essence of the corresponding method scheme to deviate from the scope of the method schemes of the embodiments of the present invention.

Claims

1. A multi-task generative adversarial network-based SAR image oil spill classification and semantic segmentation method, characterized by: The method includes two parts: classification and recognition of oil spills and oil spill-like objects, and semantic segmentation of oil spill images. First, the discriminator and classifier of the generative adversarial network are combined to achieve classification and discrimination of oil spills and oil spill-like objects. Second, the discriminated oil spill image is fed into the next generative adversarial network, and the generator is used to generate a segmentation map of the oil spill image to achieve semantic segmentation. Through the two generative adversarial networks, a multi-task GANs network model is finally constructed. The method includes the following steps: In the first step, after preprocessing the original remote sensing image, the oil spill and oil spill resemblances are selected through expert interpretation. The oil spill and oil spill resemblance areas are then cropped to obtain the oil spill image and oil spill resemblance image. The oil spill resemblance image is labeled with a value of 0, and the oil spill image is labeled with a value of 1. These labeled values ​​serve as the input to the first generative adversarial network. In the second step, the pre-processed images of oil spills and oil spill-like objects are input into the first generative adversarial network. The input images are then used to complete the classification task of oil spills and oil spill-like objects through the first generative adversarial network, thereby achieving the distinction between oil spills and oil spill-like objects. In the third step, after the image is divided into oil spill and oil spill-like objects in the second step, it is necessary to remove the oil spill-like objects from the classified oil spill image, and only use the oil spill image as the input of the fourth step. The third step only performs the judgment operation; In the fourth step, the oil spill images filtered out in the third step are subjected to the semantic segmentation task by the second generative adversarial network. The semantic segmentation task is used to obtain the extraction results of the oil spill area. The second generative adversarial network is also built with a generator and a discriminator, as follows: 4.1) Build a segmentation network model, consisting of the generator G s and the discriminator D s composition; The input of the generator is the original oil spill image, that is, the oil spill image after filtering in the third step, and the output is the generated oil spill segmentation map; the input of the discriminator is the original oil spill image, the oil spill segmentation map generated by the generator, and the oil spill segmentation label map; The training process of the second generative adversarial network is as follows: first, the oil spill segmentation map initially generated by the generator cannot be close to the oil spill segmentation label map. The discriminator discriminates by inputting the original oil spill image, the oil spill segmentation map and the oil spill segmentation label map. The discriminator distinguishes the oil spill segmentation map generated by the generator and the oil spill segmentation label map, distinguishes whether the image is an oil spill segmentation label map or an oil spill segmentation map generated by the generator, learns the data distribution differences between them, and outputs a discrimination score; the discriminator's judgment result is used to reversely optimize the generator, guiding the generator to generate a more accurate oil spill segmentation map; finally, the generator can generate a realistic oil spill segmentation map. After the training is completed, the trained generator is used to generate an oil spill segmentation map to achieve accurate extraction of the oil spill area; The objective function of the generator of the second generative adversarial network is shown in the following formula (8): Among them, λ1 is the weight balance parameter, S is the input true segmentation label map, is the segmentation map generated by the generator, and I is the real oil spill image; is the L1 norm, which penalizes the pixel-by-pixel distance between the segmentation label map and the generated segmentation map; minimizing it trains the generator to generate more realistic oil spill segmentation maps, and ultimately successfully deceives the discriminator; The objective function of the second generative adversarial network discriminator is shown in Equation (9): Among them, λ2 is the weight balance parameter, S is the input true segmentation label map, is the segmentation map generated by the generator, and I is the real oil spill image; This term represents the adversarial loss, which increases the discriminative ability of the discriminator; Represents the penalty gradient loss, which produces a stable gradient that neither disappears nor explodes; Indicates that in S and A random variable uniformly sampled between ; minimize the training discriminator to distinguish the generated oil spill segmentation map and the oil spill segmentation label map; 4.2) The overall objective function of the second generative adversarial network is shown in (10): The generator and discriminator are continuously optimized and trained; the network is trained using the loss formula (10) until the second generative adversarial network reaches a balanced state; after the training is completed, the generator is retained and the discriminator is discarded; the generator is used to generate the oil spill segmentation map.

2. The SAR image oil spill classification and semantic segmentation method based on a multi-task generative adversarial network according to claim 1 is characterized in that: The pre-processing process in the first step includes thermal noise removal, radiation calibration, speckle filtering, and terrain correction.

3. The SAR image oil spill classification and semantic segmentation method based on a multi-task generative adversarial network according to claim 1 is characterized in that: The specific steps of the second step of distinguishing oil spills from oil spill-like objects are as follows: 2.1) Build the first generative adversarial network model, consisting of the generator G c and the discriminator D c The generator is constructed by firstly normalizing the layer, then two or more overall modules, and finally a convolution layer and a Tanh activation function. The overall module consists of an upsampling layer, a convolution layer, a normalization layer, and a LeakyReLU activation function. The input of the generator is random noise z, which is then linearly transformed. The linear transformation formula is shown in (1). y=zA T +c (1) Where z is the random noise input, A is the weight, c is the bias, and y is the linear transformation output; 2.2) The data after linear transformation enters the normalization layer; 2.3) After the normalization layer, the data passes through the generator’s upsampling layer, convolution layer, normalization layer, and LeakyReLU activation function in sequence; the upsampling layer increases the length and width of the image to twice its original size before the convolution layer is performed. The convolution layer formula is shown in (2); Among them, a is the output of the convolution layer; W g is the weight of the g-th convolution kernel; x i is the i-th input; b g is the g-th bias; Q is the total number of inputs; G is the total number of convolution kernels; In order to prevent gradient disappearance and overfitting during feature extraction network training, a normalization layer is added after the convolution layer, as shown in formula (3); Among them, ε is the minimum value automatically generated by the system; n is the output of the normalization layer; In order to make the feature extraction network converge quickly, the LeakyReLU activation function is added after the normalization layer, as shown in formula (4); Among them, the value of l is 0.01; L(n) is the output of the LeakyReLU function; 2.4) After passing through the above-mentioned upsampling layer, convolution layer, normalization layer and LeakyReLU activation function, the same process is repeated; the final output result passes through another convolution layer and Tanh activation function; the formula of Tanh activation function is shown in (5); Among them, L is the output of the final LeakyReLU function, and Tanh(L) is the output of the Tanh activation function; the generator will eventually generate pseudo samples close to oil spills and oil spill-like objects, which are then input into the discriminator; 2.5) Discriminator The discriminator consists of four convolution blocks, each of which consists of a convolution layer, a LeakyReLU activation function, and a Dropout layer connected in sequence. The sigmoid and softmax activation functions are connected at the end of the discriminator. An output layer is added after the last convolution block of the discriminator, and the output of the last convolution block is used as the input of the sigmoid and softmax activation functions in the output layer. The formula of the sigmoid function is shown in (6); the formula of the softmax function is shown in (7). The sigmoid function is used to distinguish true from false; the softmax function is used for classification, converting it into a multi-classification problem. The input of the discriminator consists of true samples and false samples. The true samples are the oil spill images and oil spill similarities in the first step, and the false samples are the oil spills and oil spill similarities generated by the generator. After the true samples and false samples are input into the discriminator together, the discriminator will not only be able to distinguish between true and false, but also classify the true samples; the discriminator in the first generative adversarial network will learn the characteristics of the oil spill and oil spill similarity images, and learn to automatically distinguish oil spills and oil spill similarities from unknown images; for images with unlabeled label values, the final softmax function in the second step will output the category probability value, and the oil spill and oil spill similarities will be distinguished based on the output value.

4. The SAR image oil spill classification and semantic segmentation method based on a multi-task generative adversarial network according to claim 1 is characterized in that: In step 4.1), the generator network consists of a U-net network, which forms a U-shaped structure by four downsampling and four upsampling; the U-net network is followed by a convolutional layer, and then three convolutional blocks are connected, each of which has a convolutional layer, a normalization layer and a LeakyReLU activation function; finally, there is a convolutional layer and a Tanh activation function; the discriminator is constructed by five convolutional blocks, each of which contains a convolutional layer, a normalization layer and a LeakyReLU activation function; the last convolutional block has only a convolutional layer.

5. The SAR image oil spill classification and semantic segmentation method based on a multi-task generative adversarial network according to claim 1 is characterized in that: The oil spill segmentation label map in step 4.1) is the oil spill area determined by expert interpretation, and then the area is drawn as the oil spill segmentation label map using the labelme tool.

Images