Marine litter intelligent identification and classification method based on multispectral unmanned aerial vehicle remote sensing image

Abstract

The invention discloses a marine litter intelligent identification and classification method based on a multispectral unmanned aerial vehicle remote sensing image, and relates to the technical field of image processing, and the method comprises the steps: collecting and preprocessing a remote sensing image of a preset sea area, constructing a color correction network, and carrying out the color correction of the preprocessed remote sensing image; multiband features and spectral indexes of enhanced garbage detection are obtained, and multispectral features are generated through multi-scale feature pyramid structure fusion; training a conditional generative adversarial network, inputting a remote sensing image and multispectral features, and performing pixel-by-pixel fusion on the enhanced remote sensing image and the remote sensing image after color correction to obtain a multiband fusion feature map; and constructing a lightweight detection network based on the OfficientDet-Lite, taking the multi-band fusion feature map as input, and outputting a bounding box and a garbage category of garbage by adopting a joint detection-classification architecture. According to the method, the problems of color distortion, contour fuzziness, small target missing detection and the like in marine litter identification are solved, and meanwhile, the real-time performance of marine litter identification and classification is ensured.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a method for intelligently identifying and classifying marine debris based on multispectral unmanned aerial vehicle (UAV) remote sensing images. Background Art

[0002] With global marine pollution becoming increasingly severe, the monitoring and management of marine debris (such as plastics, foam, and fishing nets) has become a crucial environmental protection issue. Traditional marine debris monitoring relies primarily on manual inspections or satellite remote sensing. However, manual inspections are inefficient and have limited coverage, while satellite remote sensing is limited by its spatial resolution and revisit period, making it difficult to meet the high-precision, dynamic monitoring requirements. In recent years, drone remote sensing technology, owing to its flexibility, low cost, and high resolution, has gradually become a key tool for marine environmental monitoring.

[0003] Multispectral UAV remote sensing, equipped with multispectral sensors, can obtain spectral information richer than that of visible light, thereby enhancing the ability to identify marine debris. However, marine debris is complex in distribution and diverse in form, and is affected by factors such as light, waves, and suspended matter. Traditional classification methods based on thresholds or manual feature extraction are often inaccurate and difficult to meet the needs of practical applications. In recent years, deep learning technology has made significant progress in the field of image recognition. In particular, models such as convolutional neural networks (CNNs) and Transformers have demonstrated powerful feature learning capabilities in target detection and classification tasks, providing a new technical path for the intelligent identification of marine debris. Therefore, how to effectively utilize multi-band information to enhance the distinction between debris and background, suppress environmental interference, and improve the accuracy of small target detection is an urgent problem that needs to be solved. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a method for intelligent identification and classification of marine debris based on multispectral UAV remote sensing images, which solves the problems of color distortion, blurred outlines, missed detection of small targets, etc. in marine debris identification, while ensuring the real-time performance of marine debris identification and classification.

[0005] A first aspect of the present invention provides a method for intelligently identifying and classifying marine debris based on multispectral drone remote sensing images, comprising the following steps:

[0006] Using a multispectral sensor carried by an unmanned aerial vehicle to collect remote sensing images of a preset sea area, preprocessing the remote sensing images, building a color correction network, and performing color correction on the preprocessed remote sensing images;

[0007] Obtaining multi-band features and spectral indices for enhanced garbage detection, and fusing the multi-band features and spectral indices using a multi-scale feature pyramid structure to generate multispectral features;

[0008] Training a conditional generative adversarial network, inputting a remote sensing image and corresponding multispectral features, obtaining an enhanced remote sensing image, and fusing the enhanced remote sensing image with the color-corrected remote sensing image pixel by pixel to obtain a multi-band fusion feature map;

[0009] A lightweight detection and classification model is constructed based on EfficientDet-Lite, which takes the multi-band fusion feature map as input and adopts a joint detection-classification architecture to output the bounding box and garbage category of the garbage.

[0010] In this solution, a multispectral sensor carried by an unmanned aerial vehicle is used to collect remote sensing images of a preset sea area, and the remote sensing images are preprocessed as follows:

[0011] Use drones equipped with multispectral sensors to collect remote sensing images of a preset ocean area, including visible light bands, near-infrared bands, and short-wave infrared bands. Record metadata during the remote sensing image acquisition process, group the multispectral remote sensing images by band, select the band with the highest spatial resolution as the reference image, and use the remaining bands as images to be registered.

[0012] Performing initial coarse registration of the remote sensing images based on the metadata, using affine transformation to preliminarily align the bands, and using histogram matching to make the brightness distribution of the image to be registered close to that of the reference image;

[0013] In the reference image and the image to be registered, the ORB algorithm is used in combination with the feature pyramid to detect multi-scale feature points, the fast approximate nearest neighbor algorithm is used to perform preliminary matching of the feature points, the Euclidean distance between the descriptors is calculated for registration, and the mismatched points are eliminated based on the registration results;

[0014] The reference image and the image to be registered are divided into grids. The local homography transformation is calculated independently for each grid, and grid-based local registration is performed. The transformation parameters of adjacent grids are smoothed using the moving least squares method to obtain the registered remote sensing image.

[0015] In this solution, a color correction network is constructed to perform color correction on the preprocessed remote sensing images. Specifically:

[0016] A color correction network is constructed based on U-Net as the backbone network. The input channel layer is expanded to the number of bands of multispectral remote sensing images. Grouped convolution is used in the encoder to independently process different bands and extract band features. After each level of downsampling, an attention mechanism is used to dynamically weight important band features.

[0017] A cross-band feature interaction module is used in the bottleneck layer to fuse multispectral information. A multi-scale channel attention mechanism is introduced in the decoder part to adaptively fuse the band features extracted by the corresponding encoder after upsampling in each layer. The output layer is used to generate the corrected multispectral remote sensing image.

[0018] Add a discriminator for adversarial training to optimize the color correction network. The generated multispectral remote sensing image and the real label are cut into image blocks. The discriminator judges the authenticity of each block. Alternating training is used to optimize the local color of the color correction grid based on the judgment results.

[0019] The preprocessed remote sensing image is imported into the trained color correction network, and the color-corrected multi-band remote sensing image is output.

[0020] In this solution, we obtain multi-band features and spectral indices for enhanced garbage detection, and use a multi-scale feature pyramid structure to fuse the multi-band features and spectral indices to generate multispectral features. Specifically,

[0021] Obtain color-corrected multi-band remote sensing images, use lightweight convolutional blocks to independently extract low-level features of each band, import the low-level features of each band as input into the EfficientNet backbone network to extract high-level semantic features, fuse multi-band context information, and output multi-scale feature maps as multi-band features;

[0022] Retrieve and obtain marine debris detection examples, preprocess the marine debris detection examples, interpret the preprocessed example samples using SHAP local attribution analysis, calculate the Shapley value corresponding to each spectral index in the example samples, take the absolute value of the Shapley value corresponding to each spectral index and normalize it to a percentage to quantify the local importance;

[0023] sorting the spectral indices involved in the example samples according to the local importance, selecting a preset number of spectral indices as key spectral indices, acquiring a radiometrically corrected multi-band remote sensing image, and extracting spectral indices for enhanced garbage detection based on the key spectral indices;

[0024] The multi-band features and spectral indices are fused step by step using a multi-scale feature pyramid to generate multispectral features, batch normalization and nonlinear enhancement are performed on the multispectral features, and optimized multispectral features are output.

[0025] In this solution, an enhanced remote sensing image is obtained, and the enhanced remote sensing image is fused pixel by pixel with the color-corrected remote sensing image to obtain a multi-band fusion feature map. Specifically,

[0026] The generator of the conditional generative adversarial network is constructed based on residual dense blocks combined with multi-scale dilated convolution. The preprocessed remote sensing image and the corresponding multispectral features are input to obtain the remote sensing image with enhanced details.

[0027] The detail-enhanced remote sensing image is imported into the discriminator, which determines whether the imported image is real data or generated data. The generator and discriminator of the conditional generative adversarial network are alternately trained using the training data, and the parameters are updated until the discriminator cannot correctly classify the imported image, and the enhanced remote sensing image is output;

[0028] The color-corrected remote sensing image is obtained and Sobel edge extraction is performed to obtain an edge intensity map. A dynamic weight map is generated according to the edge intensity map. Adaptive weights are obtained based on the dynamic weight map to fuse the enhanced remote sensing image with the color-corrected remote sensing image pixel by pixel to obtain a multi-band fusion feature map.

[0029] In this solution, a lightweight detection and classification model is built based on EfficientDet-Lite, specifically:

[0030] A lightweight detection and classification model was built based on the EfficientDet-Lite framework, and training and testing data were obtained using marine debris detection examples.

[0031] Import the training data into the detection and classification model, use the Ghost module to replace the traditional convolution for feature extraction, and perform coordinate attention weighting on the feature map output by the Ghost module to enhance the key position response;

[0032] A detection branch is constructed using a lightweight feature pyramid. The BiFPN structure is adopted, and the convolution in the cross connection is replaced with Ghost convolution. The weighted feature map is imported into the detection branch. After further feature extraction through the lightweight feature pyramid, three Ghost convolution layers with shared weights are used to generate bounding box predictions.

[0033] In the classification branch, the weighted feature maps are spliced, and the features of different bands are dynamically weighted through channel attention. A classifier is constructed using two fully connected layers to generate category predictions. After iterative training, the bounding box predictions and category predictions are verified using the test data. Once the verification passes, the network parameters of the current detection and classification model are retained, and the trained detection and classification model is output.

[0034] In this solution, the multi-band fusion feature map is used as input, and a joint detection-classification architecture is used to output the bounding box and garbage category of the garbage. Specifically:

[0035] The multi-band fusion feature map corresponding to the multi-band remote sensing image is imported into the detection and classification model, and the garbage bounding box prediction results and garbage category prediction results are obtained through parallel calculation of the detection branch and the classification branch;

[0036] Time series processing is performed based on the garbage boundary box prediction results and the garbage category prediction results to obtain the movement path of the marine garbage. Spatiotemporal features are extracted based on the movement path, and the movement path is predicted using the spatiotemporal features. The predicted movement path is sent and displayed in a preset manner.

[0037] The second aspect of the present invention provides a marine debris intelligent identification and classification system based on multispectral UAV remote sensing images, which includes a data acquisition and preprocessing module, a multispectral feature enhancement module, a target detection and classification module, and a post-processing optimization module;

[0038] The data acquisition and preprocessing module acquires remote sensing images of a preset sea area, preprocesses the remote sensing images, and uses a color correction network to perform color correction on the preprocessed remote sensing images;

[0039] The multispectral feature enhancement module is responsible for acquiring multi-band features and spectral indices for enhanced garbage detection. It uses a multi-scale feature pyramid structure to fuse the multi-band features and spectral indices to generate multispectral features. It uses a conditional generative adversarial network to generate enhanced remote sensing images, which are then fused pixel by pixel with the color correction results to construct a multi-band fusion feature map.

[0040] The target detection and classification module is responsible for building a lightweight detection and classification model based on EfficientDet-Lite, taking the multi-band fusion feature map as input, and using the detection branch and classification branch to obtain the garbage detection frame and garbage category;

[0041] The post-processing optimization module is responsible for optimizing the garbage detection results, reducing false detections and missed detections, and compressing and accelerating the detection classification model.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] This invention significantly improves the accuracy, robustness, and real-time performance of marine debris identification by integrating multispectral data, deep learning enhancements, and lightweight detection technologies. Multi-band feature fusion improves the detection rate of plastic, foam, and other debris, particularly under complex lighting and water conditions, significantly reducing false detection rates. A conditional generative adversarial network enhances the resolution of blurred outlines and color distortion, improving the detection accuracy of small objects such as microplastic fragments.

[0044] In addition, a lightweight detection model (EfficientDet-Lite) was selected to meet real-time business needs. It is suitable for online drone monitoring and greatly improves the efficiency of marine debris detection. It solves the pain points of traditional methods such as low efficiency, missed detection, false detection and lack of real-time performance in complex environments, and provides a feasible intelligent solution for marine environmental protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments or exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or exemplary descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained according to these drawings without paying any creative work.

[0046] Figure 1 A flow chart showing a method for intelligent identification and classification of marine debris based on multispectral UAV remote sensing images is shown;

[0047] Figure 2 A flow chart of generating multispectral signatures in an embodiment is shown;

[0048] Figure 3 A flowchart of constructing a lightweight detection and classification model in an embodiment is shown;

[0049] Figure 4 The block diagram of the marine debris intelligent identification and classification system based on multispectral UAV remote sensing imagery is shown. DETAILED DESCRIPTION

[0050] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0052] Figure 1 A flow chart of the method for intelligent identification and classification of marine debris based on multispectral UAV remote sensing images is shown.

[0053] like Figure 1 As shown, this embodiment provides a method for intelligent identification and classification of marine debris based on multispectral drone remote sensing images, including:

[0054] S102, using a multispectral sensor carried by a drone to collect remote sensing images of a preset sea area, preprocessing the remote sensing images, constructing a color correction network, and performing color correction on the preprocessed remote sensing images;

[0055] S104, obtaining multi-band features and spectral indices for enhanced garbage detection, fusing the multi-band features and spectral indices using a multi-scale feature pyramid structure to generate multispectral features;

[0056] S106, training a conditional generative adversarial network, inputting a remote sensing image and corresponding multispectral features, obtaining an enhanced remote sensing image, and fusing the enhanced remote sensing image with the color-corrected remote sensing image pixel by pixel to obtain a multi-band fusion feature map;

[0057] S108: Build a lightweight detection and classification model based on EfficientDet-Lite, take the multi-band fusion feature map as input, and use a joint detection-classification architecture to output the bounding box and garbage category of the garbage.

[0058] It should be noted that drones equipped with multispectral sensors were used to collect remote sensing images of a pre-set ocean area to improve the spectral differentiation of different types of garbage (such as plastic, foam, and fishing nets). Multispectral remote sensing images include visible light bands, near-infrared bands, and short-wave infrared bands. Metadata during the remote sensing image acquisition process, such as flight altitude, lighting conditions, and shooting angle, were recorded. Multispectral remote sensing images were grouped by band, and the band with the highest spatial resolution was selected as the reference image, while the remaining bands were used as the images to be registered. Initial coarse registration of the remote sensing images was performed based on metadata such as GPS and IMU data recorded by the drones. Affine transformation was used to initially align the bands and eliminate significant translation and rotation errors. Histogram matching was used to bring the brightness distribution of the image to be registered close to that of the reference image, reducing contrast deviations caused by lighting or sensor differences. When fog interference was present, dehazing was performed to enhance the extractability of edge features.

[0059] In the reference image and the image to be registered, the ORB algorithm is used in combination with the feature pyramid to detect multi-scale feature points. ORB has higher computational efficiency and is suitable for scenes with high real-time requirements. The multi-scale pyramid strategy enhances adaptability to small targets and weak texture areas. A fast approximate nearest neighbor algorithm is used to perform preliminary matching of feature points, and the Euclidean distance between descriptors is calculated for registration. Based on the registration results, mismatched points are uniformly eliminated through random sampling. The reference image and the image to be registered are divided into grids, and the local homography transformation is calculated independently for each grid to perform gridded local registration to solve the global registration residuals caused by lens distortion or terrain undulations. The moving least squares method is used to smooth the transformation parameters of adjacent grids to avoid obvious seams between blocks, and to obtain the registered remote sensing image. The gridded local registration reduces the amount of calculation for the global transformation and is suitable for real-time processing on the UAV platform. Preferably, the registration quality is verified, and the mutual information (MI) or structural similarity (SSIM) of the registered image is calculated as a quality verification indicator until the accuracy requirements are met.

[0060] It should be noted that the color correction network is constructed using U-Net as the backbone network, leveraging its encoder-decoder structure to preserve spatial details, making it suitable for pixel-level color correction tasks. The input channel layer is expanded to accommodate the number of bands in multispectral remote sensing imagery, such as the visible light band, near-infrared band, and shortwave infrared band, totaling five channels. Grouped convolution is used in the encoder to independently process different bands, extract band features, and reduce cross-band interference. After each level of downsampling, an attention mechanism is used to dynamically weight important band features, such as the shortwave infrared band being more critical for plastic detection. A cross-band feature interaction module is used in the bottleneck layer to fuse multispectral information. This information is then fused through 1×1 convolution to generate global color correction parameters. A multi-scale channel attention mechanism is introduced in the decoder. After upsampling each layer, adaptive feature fusion is performed with the band features extracted by the corresponding encoder to reduce noise transfer. A 3×3 convolution is used in the output layer to generate the corrected multispectral remote sensing image. A discriminator is added for adversarial training to optimize the color correction network. The generated multispectral remote sensing image is then segmented with the ground-truth label image blocks. The discriminator determines authenticity block by block. Alternating training is used to optimize the local color of the color correction grid based on the judgment results, forcing the generator to optimize local color consistency. The preprocessed remote sensing image is then fed into the trained color correction network, outputting the color-corrected multiband remote sensing image.

[0061] Figure 2 A flow chart for generating multi-spectral signatures in an embodiment is shown.

[0062] According to an embodiment of the present invention, multi-band features and spectral indices for enhanced garbage detection are obtained, and a multi-scale feature pyramid structure is used to fuse the multi-band features and spectral indices to generate multispectral features, specifically:

[0063] S202, obtaining a color-corrected multi-band remote sensing image, using a lightweight convolutional block to independently extract low-level features of each band, importing the low-level features of each band as input into the EfficientNet backbone network to extract high-level semantic features, fusing multi-band context information, and outputting a multi-scale feature map as a multi-band feature;

[0064] S204: Retrieve and obtain marine debris detection examples, preprocess the marine debris detection examples, interpret the preprocessed example samples using SHAP local attribution analysis, calculate the Shapley value corresponding to each spectral index in the example samples, take the absolute value of the Shapley value corresponding to each spectral index and normalize it to a percentage to quantify local importance;

[0065] S206, sorting the spectral indices involved in the example samples according to the local importance, selecting a preset number of spectral indices as key spectral indices, acquiring a radiometrically corrected multi-band remote sensing image, and extracting spectral indices for enhanced garbage detection based on the key spectral indices;

[0066] S208 , fusing the multi-band features and spectral indices step by step using a multi-scale feature pyramid to generate multispectral features, performing batch normalization and nonlinear enhancement on the multispectral features, and outputting optimized multispectral features.

[0067] It should be noted that after obtaining color-corrected multi-band remote sensing imagery, lightweight convolutional blocks are used to independently extract low-level features from each band, such as edges and textures, while preserving band specificity. Multi-band low-level feature maps are grouped by band and compressed to the standard input channel number through 1×1 convolution to meet the input requirements of EfficientNet. 3×3 depthwise separable convolution is used to initially fuse the band information. The MBConv module is then used for feature extraction. Through multiple downsampling stages, feature maps of different resolutions are output. Cross-band attention is applied to these feature maps of different resolutions, and the correlation matrix between feature maps of different bands is calculated. Softmax is used to generate fusion weights, and the weighted summation is used to output cross-band enhanced features. Multi-scale feature alignment is performed to output high-level semantic features.

[0068] Retrieve and obtain marine debris detection examples, which include the detection results of a single multispectral image, as well as the multi-band data and spectral index of the corresponding area. Preprocess the marine debris detection examples, calculate the mean band characteristics and spectral index mean for each pixel, generate an interpretation input vector, and interpret the preprocessed example samples using SHAP local attribution analysis. Obtain the Shapley value of each spectral index involved in the marine debris detection example. Positive or negative indicates promotion or inhibition of detection, and the Shapley value reflects the contribution of the feature. Quantify the local importance of the standards, and select the top five spectral indices with the highest importance as the core judgment basis for the example. These indices are marked as key spectral indices. For example, FDI has the highest contribution to plastic detection, and NDVI has an inhibitory effect on falsely detected algae.

[0069] It should be noted that the generator of the conditional generative adversarial network is constructed based on two residual dense blocks and a multi-scale dilated convolution branch located between the residual dense blocks. The residual dense block contains densely connected multi-layer residual modules and removes batch normalization layers to avoid artifacts. The output of each convolutional layer is connected to all subsequent layers to achieve feature reuse and enhance the efficiency of multispectral information transmission. Three parallel dilated convolution branches with different dilation rates are set to process features with different receptive fields in parallel, preserving basic spatial features, capturing mid-scale context, and extracting large-scale edge associations. Feature fusion is performed through dynamically generated weights through 1×1 convolution to enhance edge high-frequency information. The preprocessed remote sensing image and the corresponding multispectral features are used as generator input to obtain a detail-enhanced remote sensing image; the detail-enhanced remote sensing image is imported into a Markov discriminator based on PatchGAN, multi-scale processing is performed in the discriminator to determine whether the imported image belongs to real data or generated data, and the generator and discriminator of the conditional generative adversarial network are alternately trained using training data, and parameters are updated until the discriminator cannot correctly classify the imported image, and the enhanced remote sensing image is output; the color-corrected remote sensing image is obtained for Sobel edge extraction to obtain an edge intensity map, and a dynamic weight map α=σ(β·E+γ) is generated according to the edge intensity map, where α represents an adaptive weight, σ represents a Sigmoid function, E represents an edge intensity map, and β and γ represent learnable parameters; adaptive weights are obtained based on the dynamic weight map, and the enhanced remote sensing image is fused pixel by pixel with the color-corrected remote sensing image to obtain a multi-band fusion feature map.

[0070] Figure 3 A flowchart of constructing a lightweight detection and classification model in an embodiment is shown.

[0071] According to an embodiment of the present invention, a lightweight detection and classification model is constructed based on EfficientDet-Lite, specifically:

[0072] S302: Build a lightweight detection and classification model based on the EfficientDet-Lite framework, and use marine debris detection examples to obtain training data and test data for model training and testing.

[0073] S304: Import the training data into the detection and classification model, use the Ghost module to replace the traditional convolution to extract features, and perform coordinate attention weighting on the feature map output by the Ghost module to enhance the key position response;

[0074] S306: Use a lightweight feature pyramid to build a detection branch. Using the BiFPN structure, replace the convolution in the cross connection with Ghost convolution. Import the weighted feature map into the detection branch. After further feature extraction using the lightweight feature pyramid, use three Ghost convolution layers with shared weights to generate bounding box predictions.

[0075] S308, in the classification branch, the weighted feature maps are spliced, different band features are dynamically weighted through channel attention, a classifier is constructed using two fully connected layers, and category predictions are generated. After iterative training, the bounding box predictions and category predictions are verified using the test data. When the verification passes, the network parameters of the current detection and classification model are retained, and the trained detection and classification model is output.

[0076] It is important to note that the 3×3 standard convolutions in the EfficientNet-Lite backbone network are replaced with Ghost convolutions. 1×1 convolutions are used in the Ghost module to generate a small number of intrinsic feature maps. Depthwise separable convolutions are used to generate ghost feature maps, and the intrinsic and ghost features are concatenated as output, reducing the number of network parameters and computation. A coordinate attention mechanism is inserted into the outputs of the last three stages of the backbone network. Global pooling of the input features in both height and width directions is performed to generate direction-aware features. Spatial attention weights are generated through convolution combined with nonlinear activations and multiplied with the original features to enhance the response to key locations. The detection branch adopts a BiFPN architecture, replacing convolutions in the cross-connections with Ghost convolutions and reducing the number of FPN layers to four, further achieving lightweightness. The classification branch concatenates the feature maps weighted by coordinate attention. The channel attention mechanism dynamically weights the features of different bands to represent the differences in their importance, automatically reducing the weight of affected bands in turbid water bodies and improving the representation of rare waste categories (such as rubber). Two fully connected layers are used to output a Softmax probability distribution. The detection and classification model is trained in two stages: the first stage freezes the backbone network and trains only the detection and classification branches. The second stage fine-tunes all parameters end-to-end.

[0077] The multi-band fusion feature map corresponding to the multi-band remote sensing image is imported into the detection and classification model, and the garbage bounding box prediction results and garbage category prediction results are obtained through the parallel calculation of the detection branch and the classification branch; time series processing is performed based on the garbage bounding box prediction results and the garbage category prediction results, and the current frame detection results are associated and matched with the historical trajectory to obtain the movement path of marine garbage, and a filter is used to smooth the position jump. Based on the motion path, spatiotemporal features are extracted, including short-term motion vector, long-term trend angle, spectral stability and aggregation index, which correspond to the instantaneous movement direction, overall migration trend, material change monitoring and group movement characteristics of the garbage, respectively. The motion path is predicted using the LSTM prediction model based on the spatiotemporal features, and the predicted motion path is sent and visualized in a preset manner to generate a garbage distribution heat map and predicted path arrows, and relevant early warnings are issued.

[0078] Figure 4 The block diagram of the marine debris intelligent identification and classification system based on multispectral UAV remote sensing imagery is shown.

[0079] The second embodiment of the present invention provides a marine debris intelligent identification and classification system 4 based on multispectral UAV remote sensing images, which includes a data acquisition and preprocessing module 401, a multispectral feature enhancement module 402, a target detection and classification module 403, and a post-processing optimization module 404;

[0080] The data acquisition and preprocessing module acquires remote sensing images of a preset sea area, preprocesses the remote sensing images, and uses a color correction network to perform color correction on the preprocessed remote sensing images;

[0081] The multispectral feature enhancement module is responsible for acquiring multi-band features and spectral indices for enhanced garbage detection. It uses a multi-scale feature pyramid structure to fuse the multi-band features and spectral indices to generate multispectral features. It uses a conditional generative adversarial network to generate enhanced remote sensing images, which are then fused pixel by pixel with the color correction results to construct a multi-band fusion feature map.

[0082] The object detection and classification module is responsible for building a lightweight detection and classification model based on EfficientDet-Lite. It takes the multi-band fusion feature map as input, uses the detection branch and the classification branch to obtain the garbage detection frame and garbage category, and uses Focal Loss in the lightweight detection and classification model to alleviate the category imbalance problem.

[0083] The post-processing optimization module is responsible for optimizing garbage detection results, reducing false detections and missed detections, and compressing and accelerating the detection and classification model. For example, it uses non-maximum suppression (NMS) optimization to avoid the incorrect suppression of highly overlapping garbage of the same type. It combines spectral features (such as the reflectivity threshold of plastic in a specific band) to filter out false detection targets such as waves and foam. It uses TensorRT or ONNX Runtime to accelerate model inference to ensure real-time detection on edge computing devices.

[0084] The third aspect of the present invention provides a computer-readable storage medium, which includes a program for a method for intelligently identifying and classifying marine debris based on multispectral drone remote sensing images. When the program for intelligently identifying and classifying marine debris based on multispectral drone remote sensing images is executed by a processor, the steps of the method for intelligently identifying and classifying marine debris based on multispectral drone remote sensing images are implemented.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms. In addition, the functional modules in the various embodiments of the present invention can all be integrated into one processing module, or each module can be a separate module, or two or more modules can be integrated into one module; the above-mentioned integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0086] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0087] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. A method for intelligent identification and classification of marine debris based on multispectral UAV remote sensing images, characterized by: The following steps are involved: Using a multispectral sensor carried by an unmanned aerial vehicle to collect remote sensing images of a preset sea area, preprocessing the remote sensing images, building a color correction network, and performing color correction on the preprocessed remote sensing images; Obtaining multi-band features and spectral indices for enhanced garbage detection, and fusing the multi-band features and spectral indices using a multi-scale feature pyramid structure to generate multispectral features; Training a conditional generative adversarial network, inputting a remote sensing image and corresponding multispectral features, obtaining an enhanced remote sensing image, and fusing the enhanced remote sensing image with the color-corrected remote sensing image pixel by pixel to obtain a multi-band fusion feature map; A lightweight detection and classification model is constructed based on EfficientDet-Lite, which takes the multi-band fusion feature map as input and adopts a joint detection-classification architecture to output the bounding box and garbage category of the garbage.

2. The method for intelligent identification and classification of marine debris based on multispectral UAV remote sensing images according to claim 1 is characterized in that: Use the multispectral sensor carried by the drone to collect remote sensing images of the preset sea area, and pre-process the remote sensing images as follows: Use drones equipped with multispectral sensors to collect remote sensing images of a preset ocean area, including visible light bands, near-infrared bands, and short-wave infrared bands. Record metadata during the remote sensing image acquisition process, group the multispectral remote sensing images by band, select the band with the highest spatial resolution as the reference image, and use the remaining bands as images to be registered. Performing initial coarse registration of the remote sensing images based on the metadata, using affine transformation to preliminarily align the bands, and using histogram matching to make the brightness distribution of the image to be registered close to that of the reference image; In the reference image and the image to be registered, the ORB algorithm is used in combination with the feature pyramid to detect multi-scale feature points, the fast approximate nearest neighbor algorithm is used to perform preliminary matching of the feature points, the Euclidean distance between the descriptors is calculated for registration, and the mismatched points are eliminated based on the registration results; The reference image and the image to be registered are divided into grids. The local homography transformation is calculated independently for each grid, and grid-based local registration is performed. The transformation parameters of adjacent grids are smoothed using the moving least squares method to obtain the registered remote sensing image.

3. The method for intelligent identification and classification of marine debris based on multispectral UAV remote sensing images according to claim 1 is characterized in that: Construct a color correction network to perform color correction on the preprocessed remote sensing image, specifically: A color correction network is constructed based on U-Net as the backbone network. The input channel layer is expanded to the number of bands of multispectral remote sensing images. Grouped convolution is used in the encoder to independently process different bands and extract band features. After each level of downsampling, an attention mechanism is used to dynamically weight important band features. A cross-band feature interaction module is used in the bottleneck layer to fuse multispectral information. A multi-scale channel attention mechanism is introduced in the decoder part to adaptively fuse the band features extracted by the corresponding encoder after upsampling in each layer. The output layer is used to generate the corrected multispectral remote sensing image. Add a discriminator for adversarial training to optimize the color correction network. The generated multispectral remote sensing image and the real label are cut into image blocks. The discriminator judges the authenticity of each block. Alternating training is used to optimize the local color of the color correction grid based on the judgment results. The preprocessed remote sensing image is imported into the trained color correction network, and the color-corrected multi-band remote sensing image is output.

4. The method for intelligent identification and classification of marine debris based on multispectral UAV remote sensing images according to claim 1 is characterized in that: Obtain multi-band features and spectral indices for enhanced garbage detection, and fuse the multi-band features and spectral indices using a multi-scale feature pyramid structure to generate multispectral features, specifically: Obtain color-corrected multi-band remote sensing images, use lightweight convolutional blocks to independently extract low-level features of each band, import the low-level features of each band as input into the EfficientNet backbone network to extract high-level semantic features, fuse multi-band context information, and output multi-scale feature maps as multi-band features; Retrieve and obtain marine debris detection examples, preprocess the marine debris detection examples, interpret the preprocessed example samples using SHAP local attribution analysis, calculate the Shapley value corresponding to each spectral index in the example samples, take the absolute value of the Shapley value corresponding to each spectral index and normalize it to a percentage to quantify the local importance; sorting the spectral indices involved in the example samples according to the local importance, selecting a preset number of spectral indices as key spectral indices, acquiring a radiometrically corrected multi-band remote sensing image, and extracting spectral indices for enhanced garbage detection based on the key spectral indices; The multi-band features and spectral indices are fused step by step using a multi-scale feature pyramid to generate multispectral features, batch normalization and nonlinear enhancement are performed on the multispectral features, and optimized multispectral features are output.

5. The method for intelligent identification and classification of marine debris based on multispectral UAV remote sensing images according to claim 1 is characterized in that: Obtain an enhanced remote sensing image, fuse the enhanced remote sensing image with the color-corrected remote sensing image pixel by pixel, and obtain a multi-band fusion feature map, specifically: The generator of the conditional generative adversarial network is constructed based on residual dense blocks combined with multi-scale dilated convolution. The preprocessed remote sensing image and the corresponding multispectral features are input to obtain the remote sensing image with enhanced details. The detail-enhanced remote sensing image is imported into the discriminator, which determines whether the imported image is real data or generated data. The generator and discriminator of the conditional generative adversarial network are alternately trained using the training data, and the parameters are updated until the discriminator cannot correctly classify the imported image, and the enhanced remote sensing image is output; The color-corrected remote sensing image is obtained and Sobel edge extraction is performed to obtain an edge intensity map. A dynamic weight map is generated according to the edge intensity map. Adaptive weights are obtained based on the dynamic weight map to fuse the enhanced remote sensing image with the color-corrected remote sensing image pixel by pixel to obtain a multi-band fusion feature map.

6. The method for intelligent identification and classification of marine debris based on multispectral UAV remote sensing images according to claim 1 is characterized in that: Build a lightweight detection and classification model based on EfficientDet-Lite, specifically: A lightweight detection and classification model was built based on the EfficientDet-Lite framework, and training and testing data were obtained using marine debris detection examples. Import the training data into the detection and classification model, use the Ghost module to replace the traditional convolution for feature extraction, and perform coordinate attention weighting on the feature map output by the Ghost module to enhance the key position response; A detection branch is constructed using a lightweight feature pyramid. The BiFPN structure is adopted, and the convolution in the cross connection is replaced with Ghost convolution. The weighted feature map is imported into the detection branch. After further feature extraction through the lightweight feature pyramid, three Ghost convolution layers with shared weights are used to generate bounding box predictions. In the classification branch, the weighted feature maps are spliced, and the features of different bands are dynamically weighted through channel attention. A classifier is constructed using two fully connected layers to generate category predictions. After iterative training, the bounding box predictions and category predictions are verified using the test data. Once the verification passes, the network parameters of the current detection and classification model are retained, and the trained detection and classification model is output.

7. The method for intelligent identification and classification of marine debris based on multispectral UAV remote sensing images according to claim 6 is characterized in that: The multi-band fusion feature map is used as input, and a joint detection-classification architecture is used to output the bounding box and garbage category of the garbage. Specifically: The multi-band fusion feature map corresponding to the multi-band remote sensing image is imported into the detection and classification model, and the garbage bounding box prediction results and garbage category prediction results are obtained through parallel calculation of the detection branch and the classification branch; Time series processing is performed based on the garbage boundary box prediction results and the garbage category prediction results to obtain the movement path of the marine garbage. Spatiotemporal features are extracted based on the movement path, and the movement path is predicted using the spatiotemporal features. The predicted movement path is sent and displayed in a preset manner.

8. An intelligent marine debris identification and classification system based on multispectral UAV remote sensing images, characterized by: Implementing the method for intelligent identification and classification of marine debris based on multispectral UAV remote sensing images as described in any one of claims 1 to 7, the system includes a data acquisition and preprocessing module, a multispectral feature enhancement module, a target detection and classification module, and a post-processing optimization module; The data acquisition and preprocessing module acquires remote sensing images of a preset sea area, preprocesses the remote sensing images, and uses a color correction network to perform color correction on the preprocessed remote sensing images; The multispectral feature enhancement module is responsible for acquiring multi-band features and spectral indices for enhanced garbage detection. It uses a multi-scale feature pyramid structure to fuse the multi-band features and spectral indices to generate multispectral features. It uses a conditional generative adversarial network to generate enhanced remote sensing images, which are then fused pixel by pixel with the color correction results to construct a multi-band fusion feature map. The target detection and classification module is responsible for building a lightweight detection and classification model based on EfficientDet-Lite, taking the multi-band fusion feature map as input, and using the detection branch and classification branch to obtain the garbage detection frame and garbage category; The post-processing optimization module is responsible for optimizing the garbage detection results, reducing false detections and missed detections, and compressing and accelerating the detection classification model.

Images