Multi-scale dynamic entrainment water surface oil spill type identification method based on multi-source aerial remote sensing combination
The multi-scale dynamic convolution method integrates hyperspectral, thermal infrared, and SAR images to enhance feature extraction and classification accuracy for marine oil spill type identification, overcoming the limitations of single-sensor systems.
Patent Information
- Application Number
- CN202410973560.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-12-03
AI Technical Summary
Existing remote sensing technologies for marine oil spills, such as SAR, hyperspectral, and thermal infrared, suffer from limitations like high false alarms, inability to identify spill types, and low resolution, leading to inaccurate monitoring and detection.
A multi-scale dynamic convolution method combining hyperspectral, thermal infrared, and SAR images for feature extraction, using main and auxiliary paths to enhance feature information extraction, particularly spatial texture, and improve classification accuracy.
The method significantly improves oil spill type recognition accuracy by integrating multi-source aerial remote sensing, addressing the limitations of single-sensor systems and enhancing feature extraction capabilities.
Smart Images

Figure CN119068327A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of image processing, hyperspectral image classification, thermal infrared image classification and SAR image classification, and in particular to a multi-scale dynamic convolution water surface oil spill type recognition method based on multi-source aerial remote sensing combination. Background Art
[0002] With the development of the economy and society, offshore oil extraction activities are becoming more frequent, and offshore oil transportation is becoming more active, which makes the risk of offshore oil spills increase day by day. The harm of oil spills is huge, and the negative impact on the marine ecological environment and marine biological resources is difficult to eliminate in a short period of time. The identification of marine oil spills is closely related to source tracing and pollution treatment, and is an important basis for the treatment of oil spill pollution. Different types of oil products require different emergency treatment strategies, such as combustion, oil absorption felt adsorption, dispersant spraying, skimmer recovery, etc. Therefore, timely identification of different types of marine oil spill pollution is of great significance to marine oil spill monitoring and emergency response.
[0003] SAR is the most commonly used marine oil spill monitoring technology in microwave remote sensing. Because of its all-weather and all-weather advantages and its ability to penetrate clouds and fog, SAR has become a conventional technical means for relevant business departments to carry out oil spill remote sensing detection. However, it is affected by low wind speed areas, biological oil films and other oil films, resulting in a high false alarm rate for oil spill monitoring, and it cannot identify the type of oil spill and cannot detect the thickness of the oil film. Optical remote sensing contains richer oil spill information than SAR, and has been widely studied and applied in oil spill monitoring. Among them, hyperspectral remote sensing has a high spectral resolution and can extract rich subtle features between different types of oil spills. Therefore, the use of hyperspectral remote sensing for oil spill monitoring has received widespread attention, but it is easily affected by solar flares, resulting in the inability to accurately identify the boundaries of the oil film, and is not suitable for large-scale offshore oil spill monitoring and cannot be observed all day. Due to the different heat capacity and infrared reflectivity of the oil film and seawater, there are different temperature differences between different types of oil films and background seawater. Moreover, thermal infrared remote sensing is almost unaffected by changes in light, can detect oil spills throughout the day, and is sensitive to thicker oil films. However, the resolution of infrared sensors is low and cannot distinguish between phytoplankton and oil spills.
[0004] At present, most of the remote sensing monitoring of marine oil spills is based on a single sensor type, which has certain limitations, and the detection and recognition accuracy cannot meet the needs of accurate monitoring. Based on this, the present invention proposes a multi-scale dynamic convolution method for surface oil spill type recognition based on multi-source aerial remote sensing. Summary of the invention
[0005] 1. Technical issues to be resolved
[0006] The present invention proposes a multi-scale dynamic convolution method for identifying surface oil spill types based on multi-source aerial remote sensing. It mainly adopts the method of main road feature extraction and auxiliary road feature supplementation to fully extract the feature information of the area around the target pixel, alleviate the phenomenon of decreased classification accuracy caused by the loss of local detail features, and improve the feature extraction capability.
[0007] (II) Technical solution
[0008] The present invention comprises the following steps:
[0009] (1) Data registration and cropping: the airborne hyperspectral, airborne thermal infrared, and airborne SAR images acquired approximately simultaneously are registered and cropped to keep the data sizes of the three images consistent;
[0010] (2) Data dimensionality reduction and standardization. First, all bands of the original hyperspectral image are reduced in dimension and standardized using principal component analysis (PCA). Then, the hyperspectral image after dimension reduction and standardization is combined with the SAR HV image and the thermal infrared brightness temperature image to form a multi-source aerial remote sensing image. After that, with the target pixel as the center, a H×W×L cubic information block around the target pixel is selected as input. For the boundary pixels, zero padding is used to ensure that the cube size is the same and the sample label is consistent with the label of the center pixel. Finally, the network divides all samples and their corresponding labels into training set and test set.
[0011] (3) The main road extracts multi-scale spatial spectral features. The main road uses multi-scale dynamic convolution to extract the multi-scale dynamic spatial spectral features of the target pixel. Among them, multi-scale is reflected in two aspects: one is the size of the input space size, and the other is the size of the dynamic convolution kernel.
[0012] (4) Auxiliary paths supplement spatial texture information. In the process of feature extraction of classification networks, multi-layer pooling operations will cause the loss of some fine-grained information such as spatial texture. Therefore, the method of adding auxiliary paths is adopted to supplement the extraction of spatial texture information. After each dynamic convolution and before the next pooling operation, the supplementary extraction of spatial information contained in each different spatial scale is completed. Then, by fusing the information at each scale, the final spatial information supplement block is formed to further supplement some spatial texture and other detailed information, alleviate the loss of local spatial information caused by continuous pooling of the network, and enhance the network's ability to extract information.
[0013] (5) Fusion classification of main road and auxiliary road: The auxiliary road extraction is used as a strategy to supplement spatial information, and is fused with the information extracted by multi-scale dynamic convolution of the main road, and classified through the Softmax function. The entire network continuously improves the information extraction ability through back propagation, and finally obtains the classification results of different oil spill types.
[0014] (III) Beneficial effects
[0015] The advantages of the present invention are:
[0016] Different types of sensors have their own advantages in conducting remote sensing monitoring of marine oil spills. At the same time, the classification method based on multi-scale dynamic convolution uses the method of extracting main road features and supplementing auxiliary road features to fully extract the feature information of the area around the target pixel and alleviate the phenomenon of reduced classification accuracy caused by the loss of local detail features. Different types of oil films are weakly reflective targets. The present invention combines the advantages of multi-source aerial remote sensing images and uses a classification method based on multi-scale dynamic convolution to improve the recognition accuracy of different types of oil spills. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 This is a schematic diagram of the network architecture for identifying oil spill types on water surfaces based on multi-scale dynamic convolution combined with multi-source aerial remote sensing;
[0018] Figure 2 Provide on-site maps, ground truth maps and multi-source aerial remote sensing images of different oil spill types;
[0019] Figure 3 The oil spill type identification results comparing the developed model with other methods. DETAILED DESCRIPTION
[0020] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments:
[0021] Reference Figure 1 , the specific implementation steps of the present invention are:
[0022] (1) Data registration and cropping
[0023] The data used in this embodiment are hyperspectral, brightness temperature and backscattering multi-source remote sensing images of different oil spill types (crude oil, heavy oil, diesel, palm oil) approximately acquired by using the UAV-mounted Cubert S185 hyperspectral imaging system, the UAV-mounted Zenmuse H20T thermal infrared imaging system and the UAV-mounted SAR imaging system. The hyperspectral image, brightness temperature image and SARHV polarization image acquired approximately simultaneously are registered and cropped to keep the data sizes of the three consistent; the cropped image size is 627px×609px.
[0024] (2) Data dimensionality reduction and standardization
[0025] 1) The original hyperspectral image is reduced in dimension and standardized using principal component analysis, and the number of dimensions of the hyperspectral image is set to 8;
[0026] 2) Combine the dimension-reduced and standardized hyperspectral images with SAR HV images and thermal infrared brightness temperature images into multi-source aerial remote sensing images;
[0027] 3) With the target pixel as the center, select the H×W×L cube information block around the target pixel as input. For the boundary pixels, use zero padding to ensure that the cube size is the same and the sample label is consistent with the label of the center pixel;
[0028] 4) The network divides all samples and their corresponding labels into training sets and test sets.
[0029] (3) Extraction of multi-scale spatial spectrum features from the main road model
[0030] The trunk road uses multi-scale dynamic convolution to extract the multi-scale dynamic spatial spectrum features of the target pixel. Among them, multi-scale is reflected in two aspects: one is the size of the input space, and the other is the size of the dynamic convolution kernel. The steps of the entire feature extraction are:
[0031] 1) First, the input is a large spatial size (28×28) neighborhood block of the target pixel, and global feature learning is performed. The convolution kernel size is set to 7×7, and the convolution kernel parameters are adaptively adjusted using dynamic convolution, so that the network can initially extract spatial information and spatial spectrum information that are effective for classification. Finally, the feature block size formed is 28×28×32;
[0032] 2) Based on the first step, the spatial scale is reduced through the global pooling layer and batch normalization layer, and the weight parameters under the 14×14 spatial size are learned. The corresponding dynamic convolution kernel is 5×5. The feature block size formed after this step is 14×14×48;
[0033] 3) Finally, the size is further reduced through the global pooling layer and batch normalization layer to further learn the deep features of the oil spill image in a small area (7×7), and the corresponding dynamic convolution kernel size is 3×3. After this step, the image feature block size is 7×7×64;
[0034] 4) Flatten and concatenate the feature information blocks of different scales formed in the above three steps to form the spatial spectrum information of the oil spill image finally extracted by the main road for subsequent prediction and classification.
[0035] (4) Auxiliary roads supplement spatial texture information
[0036] On the basis of multi-scale dynamic convolution, the auxiliary path is used to supplement the features at different scales. After dynamic convolution learning at each scale, the corresponding features are extracted and fused to form a supplementary block of spatial feature information to further supplement some spatial texture and other detail information, alleviate the loss of local spatial information caused by continuous pooling of the network, and enhance the network's ability to extract information. The main steps are:
[0037] 1) After the input image passes through PCA, the initial size is set to 28×28×10 as the input of dynamic convolution. At this time, the convolution kernel size used is 7×7. Then, the spectral dimension is compressed through the Lambda function to generate a 28×28×1 spatial information area. In order to match the minimum spatial size generated later, the 28×28×1 spatial information area is averaged pooled twice with a step size of 1, and finally, 7×7×1 spatial texture information is obtained;
[0038] 2) After the first dynamic convolution, the network reduces the spatial size through average pooling to generate a 14×14×32 feature block. At this time, a 5×5 dynamic convolution kernel is used. Then, the Lambda function is used again to compress the spectral dimension to generate a 14×14×1 spatial information area, and then the 7×7×1 spatial texture information is obtained through average pooling.
[0039] 3) According to the generated 14×14×32 feature block, dynamic convolution and pooling are performed again to obtain a feature block of size 7×7×48. At this time, the dynamic convolution kernel size used is 3×3. After that, the Lambda function is used again to obtain a 7×7×1 feature area. No pooling operation is performed in this step.
[0040] 4) The spatial information domains generated in the first three steps are fused to form the final 7×7×3 spatial information block, which is used to supplement some of the lost spatial texture and other detail information. The entire network continuously improves its feature extraction capabilities and optimizes network performance through training and learning.
[0041] (5) Classification of trunk roads and auxiliary roads
[0042] The auxiliary road extraction is used as a strategy to supplement the spatial information, and is fused with the information extracted by the multi-scale dynamic convolution of the main road, and classified by the Softmax function. The entire network continuously improves the information extraction ability through back propagation, and finally obtains the classification results of different oil spill types.
[0043] (6) Evaluation of oil spill type identification accuracy
[0044] In order to verify the effectiveness of the proposed method (MDCM), the present invention selected the support vector machine (SVM) algorithm, the random forest (RF) algorithm and the convolutional neural network (CNN) for experimental comparison, and carried out accuracy evaluation based on the true value map of the oil spill type distribution. Figure 3 The recognition accuracy is shown in Table 1.
[0045] Table 1 Identification accuracy of different models for oil spill types
[0046] type SVM RF CNN MDCM crude 70.23 85.69 85.94 89.41 diesel fuel 67.41 82.47 83.20 80.58 Palm Oil 69.86 79.18 83.73 88.63 heavy oil 59.10 76.33 74.62 85.72 seawater 84.52 90.52 94.54 95.29 Overall accuracy (%) 76.13 87.09 91.81 94.37
Claims
1. A multi-scale dynamic convolution method for identifying oil spill types on water surface based on multi-source aerial remote sensing, characterized in that: The following steps are involved: (1) Data registration and cropping: the airborne hyperspectral, airborne thermal infrared, and airborne SAR images acquired approximately simultaneously are registered and cropped to keep the data sizes of the three images consistent; (2) Data dimensionality reduction and standardization. First, all bands of the original hyperspectral image are reduced in dimension and standardized using principal component analysis (PCA). Then, the hyperspectral image after dimension reduction and standardization is combined with the SAR HV image and the thermal infrared brightness temperature image to form a multi-source aerial remote sensing image. After that, with the target pixel as the center, a H×W×L cubic information block around the target pixel is selected as input. For the boundary pixels, zero padding is used to ensure that the cube size is the same and the sample label is consistent with the label of the center pixel. Finally, the network divides all samples and their corresponding labels into training set and test set. (3) The main road extracts multi-scale spatial spectral features. The main road uses multi-scale dynamic convolution to extract the multi-scale dynamic spatial spectral features of the target pixel. Among them, multi-scale is reflected in two aspects: one is the size of the input space size, and the other is the size of the dynamic convolution kernel. (4) Auxiliary paths supplement spatial texture information. In the process of feature extraction of classification networks, multi-layer pooling operations will cause the loss of some fine-grained information such as spatial texture. Therefore, the method of adding auxiliary paths is adopted to supplement the extraction of spatial texture information. After each dynamic convolution and before the next pooling operation, the supplementary extraction of spatial information contained in each different spatial scale is completed. Then, by fusing the information at each scale, the final spatial information supplement block is formed to further supplement some spatial texture and other detailed information, alleviate the loss of local spatial information caused by continuous pooling of the network, and enhance the network's ability to extract information. (5) Fusion classification of main road and auxiliary road: The auxiliary road extraction is used as a strategy to supplement spatial information, and is fused with the information extracted by multi-scale dynamic convolution of the main road, and classified through the Softmax function. The entire network continuously improves the information extraction ability through back propagation, and finally obtains the classification results of different oil spill types.
2. The method according to claim 1, characterized in that: In the step (2), the data is reduced in dimension and standardized, and the number of dimensions of the hyperspectral image is set to 8.
3. The method according to claim 1, characterized in that In step (3), the main road adopts a multi-scale dynamic convolution method to extract the multi-scale dynamic spatial spectrum features of the target pixel, specifically: When the spatial size of the input image is 28×28, the corresponding dynamic convolution kernel size is 7×7; when the spatial size of the input image is 14×14, the corresponding dynamic convolution kernel size is 5×5; when the spatial size of the input image is 7×7, the corresponding dynamic convolution kernel size is 3×3. The network can adopt the appropriate convolution kernel size for feature training and learning according to the size of the input image, making the network more flexible and effective in extracting the spectrum, brightness temperature and backscattering features of multi-source aerial oil spill images.
4. The method according to claim 1, characterized in that: The multi-scale dynamic convolutional surface oil spill type recognition network architecture combined with multi-source aerial remote sensing includes 1 input layer, 3 convolutional layers, 3 attention modules, 3 batch normalization layers, 3 pooling layers, 1 fully connected layer and 1 output layer.
5. The method according to claim 1, characterized in that The number of convolution kernels of the first convolution layer is 32, and the size of the convolution kernel is 7 × 7. The number of convolution kernels of the second convolution layer is 48, and the size of the convolution kernel is 5 × 5. The number of convolution kernels of the third convolution layer is 64, and the size of the convolution kernel is 3 × 3. The convolution layer is activated by a sigmoid function.
6. The method according to claim 1, characterized in that The sampling filter size of the three pooling layers is 2×2, and the pooling adopts maximum pooling.
7. The method according to claim 1, characterized in that The output layer adopts the softmax function.
8. The method according to claim 1, characterized in that The network is optimized using the mini-batch stochastic gradient descent algorithm.
9. The method according to claim 1, characterized in that: The learning rate is 0.
001.
10. The method according to claim 1, characterized in that The batch size for training is 128.
Citation Information
Cited By
Marine oil spill multi-source data fusion monitoring method and system based on deep learning
CN121170599A
