Beach plastic garbage unmanned aerial vehicle monitoring method based on map characteristics

Abstract

The invention discloses a shoal plastic garbage unmanned aerial vehicle monitoring method based on map characteristics. The method is characterized by comprising the steps of carrying out aerial photography modeling on a multispectral unmanned aerial vehicle, preprocessing visible light and multispectral data, training an instance segmentation model to detect garbage targets and generate masks at the same time, and extracting image and spectral features of each garbage target; and classifying and identifying each garbage in combination with the image and spectral features, and establishing a mass-occupied area model of each garbage for stock estimation, and the like. According to the method, the image features reflecting the morphology difference of the plastic products and the spectral features reflecting the material difference are extracted based on the instance segmentation model, the recognition dimension of the plastic garbage is increased, and compared with the prior art, multi-dimensional high-precision plastic garbage type recognition is achieved by combining the image features and the spectral features; and the plastic garbage stock estimation corresponding to the category is realized based on the identification result, and the method has good application prospects and commercial development values.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing monitoring technology, and in particular to a method for monitoring beach plastic waste by drone based on spectral features. Background Art

[0002] With the rapid development of human society and the widespread use of plastic products, marine plastic pollution has become a global environmental problem that needs to be solved urgently. Marine plastic pollution is caused by the leakage of land-based plastic waste into the sea. The beach environment is a key area where land and sea plastic waste migrate and converge. Beach plastic pollution is characterized by unclear spatial distribution, a wide variety of garbage types, and a huge environmental stock. To prevent and control plastic pollution, monitoring comes first. Therefore, it is urgent to develop a beach plastic waste monitoring method that can accurately identify garbage types and scientifically estimate garbage stocks.

[0003] In recent years, drone remote sensing, with its advantages of low cost and high efficiency, supplemented by artificial intelligence technology, has gradually replaced manual survey sampling and become a conventional technical means for plastic waste monitoring. In the existing drone monitoring methods for plastic waste, plastic waste identification strategies can be roughly divided into two categories: semantic segmentation for classifying each pixel and object detection for identifying different targets. Based on the semantic segmentation strategy to monitor plastic waste, the identification dimensions mainly include color and spectral features. Since the objects of semantic segmentation are discrete pixels rather than complete garbage targets, it is difficult to meet the quantitative needs of plastic waste monitoring; while based on the object detection strategy to monitor plastic waste, the classification features are only derived from the visible light band. Since spectral features, an important dimension for plastic waste identification, are not applied, there is a greater risk of misjudgment. In general, in order to carry out more accurate and efficient drone monitoring of plastic waste of various categories and forms, it is urgent to develop a monitoring method that combines image and spectral features. Summary of the invention

[0004] The object of the present invention is to design a method for monitoring beach plastic waste by drones based on atlas features in view of the deficiencies of the prior art. By means of an instance segmentation algorithm, a method combining image and spectral features is adopted to classify and identify all waste monomers in the monitoring area, calculate their areas, and predict their masses, and finally estimate the stock of beach plastic waste. This method uses an instance segmentation model to extract the image and spectral features of plastic waste, realizes the identification of plastic waste types with multi-dimensional high precision, and estimates the stock of plastic waste corresponding to the categories based on the identification results. The instance segmentation algorithm combines the concepts of object detection and semantic segmentation. It can not only identify different types of targets, but also perform pixel-level segmentation on each target to generate a mask covering the surface of the target. The instance segmentation algorithm has been widely applied in fields such as autonomous driving and medical image analysis. In the monitoring of beach plastic waste by drones, using this algorithm can not only obtain the confidence levels of specific targets being predicted as different waste categories based on visible light images, but also further extract the image and spectral features of waste targets based on multi-spectral data and masks. Therefore, it has considerable potential in improving the accuracy of plastic waste identification and quantification, and has good application prospects and commercial development value.

[0005] 1. The specific technical solution for achieving the object of the present invention is: a method for monitoring beach plastic waste by drones based on atlas features, characterized in that this method uses an instance segmentation algorithm to extract atlas features for plastic waste classification and identification, and estimates the mass of each waste monomer one by one based on the identification results. The monitoring of beach plastic waste by drones specifically includes the following steps:

[0006] (I). Multi-spectral UAV aerial photography modeling

[0007] 1-1: Use flight control software to operate a multi-spectral UAV to carry out flight planning and mapping aerial photography for the monitoring area. The sensor bands include a certain number of visible light, near-infrared, and short-wave infrared bands;

[0008] 1-2: Diffuse reflection target boards need to be photographed before and after the aerial photography task for radiometric calibration;

[0009] 1-3: Perform two-dimensional reconstruction on the data collected by the UAV to generate visible light and multi-spectral ortho-images of the monitoring area. The pixel values in the multi-spectral images are presented in the form of reflectance.

[0010] (II). Pretreatment of visible light and multi-spectral data

[0011] 2-1: Geometrically register the ortho-images of all bands, upsample the low-resolution multi-spectral images to the same resolution as the visible light images, and ensure that the area ranges and the number of pixels covered by the visible light and multi-spectral ortho-images are exactly the same;

[0012] 2-2: Grid cropping of visible light and multispectral orthophotos to obtain sub-images with an aspect ratio of 1:1;

[0013] 2-3: Use the LabelMe tool to label the garbage targets in the visible light orthophoto sub-image, indicate the location, outline and category of the garbage targets, and generate a JSON file as the original data set;

[0014] 2-4: Perform data augmentation on the original dataset as the true value for training and evaluating the instance segmentation model.

[0015] (III) Training instance segmentation model to detect garbage targets and generate masks

[0016] 3-1: Mask R-CNN is used as the instance segmentation algorithm, and a deep learning environment is built based on the TensorFlow framework;

[0017] 3-2: Adjust the size of the region proposal network anchor box according to the input image size;

[0018] 3-3: The initial training selects the instance segmentation model weights based on the COCO dataset as the pre-trained weights of the backbone network. Subsequent transfer learning continues to iterate based on the weights of the existing plastic waste instance segmentation model, thereby expanding the applicable plastic waste categories and monitoring scenarios;

[0019] 3-4: Divide the enhanced dataset into training set and test set at a ratio of 9:1 for model training and accuracy evaluation;

[0020] 3-5: During model training, the total loss of each training cycle is calculated in real time to monitor the model's fitting and convergence. The total loss is the sum of the classification and bounding box regression loss of the region proposal network, the classification and bounding box regression loss of the backbone network, and the instance segmentation mask loss. A reasonable maximum training cycle is set to ensure that the model converges while reducing the risk of overfitting.

[0021] 3-6: The trained instance segmentation model is used to preliminarily detect potential junk targets in visible light orthophotos, and generate a mask that depicts its shape and covers its surface.

[0022] 4. Extracting the image and spectral features of each garbage target

[0023] 4-1: Extract image features and spectral features for all potential garbage targets in the monitoring area. The image features include: prediction confidence, projection area, aspect ratio and circularity, etc.; the spectral features include: spectral angle, reflectivity slope, reflectivity intercept and reflectivity mean, etc.;

[0024] 4-2: Prediction confidence ( ) Represents the probability that a certain target is judged to be a specific type of garbage, which is output by the classification branch of the instance segmentation model by detecting each potential garbage target in the visible light image;

[0025] 4-2: For the detected potential garbage targets, further generate their masks using the segmentation branch of the instance segmentation model, and continue to extract image features such as projected area, aspect ratio, and circularity based on the visible light image;

[0026] The projected area ( ) represents the floor area of a certain garbage target, and numerically equals the product of the number of pixels covered by the garbage target, i.e., the mask ( ) and the square of the ground sampling distance of the drone ( ), that is ;

[0027] The aspect ratio ( ) describes the length-width ratio of a certain garbage target, and numerically equals the length-width ratio of the minimum bounding rectangle;

[0028] The circularity ( ) describes the degree of approximation of the contour of a certain garbage target to a circle, and numerically equals the ratio of 4π times the area ( ) to the square of the perimeter ( ), that is ;

[0029] 4-3: Use the masks of each garbage target in the visible light image as the region of interest to crop the multi-spectral image, so as to obtain the reflectance of each garbage target in different bands, and further extract spectral features such as spectral angle, reflectance slope, reflectance intercept, and reflectance mean;

[0030] The spectral angle ( ) represents the vector angle between the measured spectral curve ( ) of a certain garbage target and the reference spectral curve ( ) of a specific type of garbage, that is ;

[0031] The reference spectral curve ( ) of the specific type of garbage is obtained by statistically analyzing a large number of measured data of this type of garbage;

[0032] The reflectance slope ( ) and the reflectance intercept ( ) describe the change trend of the reflectance of a certain garbage target in each band, and are obtained by linearly fitting the reflectance and the corresponding band number;

[0033] The reflectance mean ( )Describing the overall reflectance level of a certain garbage target, which is obtained by calculating the arithmetic mean of the reflectances of the garbage target in all bands.

[0034] (V). Classifying and identifying each garbage target by using image and spectral features

[0035] 5-1: Combining the spectral classification features of the garbage target, based on the supervised classification task, constructing discriminant models for multiple common types of garbage, and the classification features include: image features such as prediction confidence, projected area, aspect ratio, and circularity, and spectral features such as spectral angle, reflectance slope, reflectance intercept, and reflectance mean;

[0036] 5-2: Using the classification algorithm to train the discriminant models for common types of garbage, and verifying its accuracy by using indicators such as precision, recall, and F1-score.

[0037] (VI). Establishing a mass-footprint model for each type of garbage for stockpile estimation

[0038] 6-1: Statistically analyzing the mass and footprint of individual common type plastic garbage. Determining the mass of the garbage monomer by manual weighing, and determining the footprint of the garbage monomer by orthophoto measurement of the unmanned aerial vehicle;

[0039] 6-2: Selecting fitting functions that conform to the physical properties such as the material, density, and geometric shape of different types of garbage, and using the measured data of a large number of individual garbage of the same type to construct multiple mass-footprint regression models corresponding to the garbage types;

[0040] 6-3: Based on the classification results and projected area of the beach plastic garbage, selecting the mass-footprint regression model of the corresponding type of garbage to complete the mass prediction of the garbage monomer;

[0041] 6-4: Conducting classification recognition, area calculation, and mass prediction for all garbage monomers in the monitoring area to achieve the stockpile estimation of beach plastic garbage.

[0042] Compared with the prior art, the present invention realizes multi-dimensional high-precision identification of plastic garbage types by combining image and spectral features, and based on the identification results, realizes the stockpile estimation of plastic garbage corresponding to the types. This method not only extracts image features reflecting the morphological differences of plastic products from visible light images, but also extracts spectral features reflecting material differences from multi-spectral data, and uses machine learning methods to effectively improve the identification and quantitative accuracy of plastic garbage, having good application prospects and commercial development value. Description of the Drawings

[0043] Figure 1 It is a schematic flow chart of the present invention;

[0044] Figure 2Statistical chart of image and spectral features of beach plastic waste;

[0045] Figure 3 Schematic diagram of the accuracy evaluation results of beach plastic waste classification and identification;

[0046] Figure 4 Schematic diagram of the mass-footprint regression model of beach plastic waste;

[0047] Figure 5 Schematic diagram of the monitoring results of beach plastic waste. Detailed implementation manners

[0048] In order to more clearly describe the technical content of the present invention, the following will be further described in conjunction with specific embodiments.

[0049] Embodiment 1

[0050] Refer to Figure 1 , the present invention is used to achieve higher-precision identification of plastic waste types and estimation of the corresponding waste stocks, and specifically includes the following steps:

[0051] (I). Multi-spectral UAV aerial photography modeling

[0052] 1-1: Use flight control software to operate a multi-spectral UAV to carry out flight planning and mapping aerial photography for the monitoring area. The bands of the multi-spectral sensor are green (560±16 nm), red (650±16 nm), red edge (730±16 nm), and near infrared (860±26 nm).

[0053] 1-2: Before and after performing the aerial photography task, a diffuse reflection target board needs to be photographed for radiometric calibration.

[0054] 1-3: Through means such as image matching, orthorectification, and color balancing, perform two-dimensional reconstruction on the data collected by the UAV to generate visible light and multi-spectral orthoimages of the monitoring area. The pixel values in the multi-spectral images are presented in the form of reflectance.

[0055] (II). Visible light and multi-spectral data preprocessing

[0056] 2-1: Geometrically register the orthoimages of all bands, and use the bilinear interpolation algorithm to upsample the low-resolution multi-spectral images to the same resolution as the visible light images, ensuring that the area range and the number of pixels covered by the visible light and multi-spectral orthoimages are exactly the same.

[0057] 2-2: Grid crop the visible light and multi-spectral orthoimages to obtain sub-images with a length and width of 1024 pixels each.

[0058] 2-3: Use the LabelMe tool to label the garbage targets in the visible light orthophoto sub-image, indicate the location, outline and category of the garbage targets, and generate a JSON file as the original data set.

[0059] 2-4: The original dataset is enhanced by image flipping, exposure adjustment, Gaussian blur, and salt and pepper noise, and the expanded dataset is used as the true value for training and evaluating the instance segmentation model.

[0060] (III) Training instance segmentation model to detect garbage targets and generate masks

[0061] 3-1: Mask R-CNN is used as the instance segmentation algorithm, and a deep learning environment is built based on the TensorFlow framework.

[0062] 3-2: Resize the input image to 512 pixels and set the region proposal network anchor box size to [16, 32, 64, 128, 256].

[0063] 3-3: In the initial training, the instance segmentation model weights based on the COCO dataset are selected as the pre-trained weights of the backbone network. Subsequent transfer learning continues to iterate based on the weights of the existing plastic waste instance segmentation model. The initial learning rate is set to 0.0001. The Adam optimizer and Cosine Annealing algorithm are used in the training process to achieve parameter optimization and dynamic adjustment of the learning rate.

[0064] 3-4: Divide the enhanced dataset into training set and test set (division ratio 9:1), which are used for model training and accuracy evaluation respectively.

[0065] 3-5: During the model training process, the total loss of each training cycle is calculated in real time to monitor the fitting and convergence of the model. The total loss is the sum of the classification and bounding box regression loss of the region proposal network, the classification and bounding box regression loss of the backbone network, and the instance segmentation mask loss. The maximum training cycle is set to 100.

[0066] 3-6: The trained instance segmentation model is used to preliminarily detect potential junk targets in visible light orthophotos, and generate a mask that depicts its shape and covers its surface.

[0067] 4. Extracting the image and spectral features of each garbage target

[0068] 4-1: The image features extracted for all potential garbage targets in the monitoring area include prediction confidence, projection area, aspect ratio, and circularity; the spectral features include spectral angle, reflectivity slope, reflectivity intercept, and reflectivity mean.

[0069] 4-2: Prediction confidence ( )The classification branch of the instance segmentation model detects each potential garbage target in the visible light image to complete the output.

[0070] 4-3: The projected area ( ), aspect ratio ( ), circularity ( ) are extracted from the mask of the garbage target, and the mask is generated by the segmentation branch of the instance segmentation model acting on the visible light image.

[0071] 4-4: Use the mask of each garbage target in the visible light image as the region of interest to crop the multi-spectral image, so as to obtain the reflectance of each garbage target in different bands, and further extract the spectral angle ( ), reflectance slope ( ), reflectance intercept ( ), and average reflectance ( ).

[0072] Refer to Figure 2 to extract the image features and spectral features of common beach plastic garbage such as foam plastics, plastic floats, and plastic bottles, and obtain the statistical results of the image and spectral features of the corresponding garbage.

[0073] (V). Classify and identify each garbage target using image and spectral features

[0074] 5-1: Combining the spectral classification features of garbage, based on the supervised classification task, construct discriminant models for multiple common types of garbage to predict whether each potential target belongs to the garbage category. The image and spectral classification features include: prediction confidence, projected area, aspect ratio, circularity, spectral angle, reflectance slope, reflectance intercept, and average reflectance.

[0075] 5-2: Select the K-nearest neighbor classification algorithm to train the discriminant model for common types of garbage, and the model evaluation method uses 10-fold cross-validation. Refer to Figure 3 , the accuracy evaluation (F1 index) of beach plastic garbage classification and identification. The F1 indices for classifying and identifying foam plastics, plastic floats, and plastic bottles using the above discriminant models are 96.0%, 83.8%, and 86.1% respectively. This result shows that classifying and identifying each garbage by combining image and spectral features achieves higher recognition accuracy than using only image or spectral features.

[0076] (VI). Establish a mass-footprint model for various types of garbage for stockpile estimation

[0077] 6-1: Statistically calculate the mass and footprint of individual common types of plastic garbage. Determine the mass of individual garbage by manual weighing, and determine the footprint of individual garbage by measuring the orthophoto image of the drone.

[0078] Refer toFigure 4 , according to the physical properties of foam plastics, plastic floats, and plastic bottles, polynomial functions, power functions, and linear functions are respectively selected as fitting functions. Using the measured data of a large number of single items of the same type of garbage, mass-footprint regression models corresponding to the categories of foam plastics, plastic floats, and plastic bottles are constructed.

[0079] 6-2: Based on the classification results and projected areas of beach plastic garbage, select the mass-footprint regression model of the corresponding category of garbage to complete the mass prediction of single garbage items.

[0080] Refer to Figure 5 , conduct classification and identification, area calculation, and mass prediction for all single garbage items within the monitoring area to complete the stock estimation of beach plastic garbage. A total of 334.7 kg of foam plastics, 47.0 kg of plastic floats, and 43.2 kg of plastic bottles are detected within this monitoring area.

[0081] The above is only a further explanation of the present invention and is not intended to limit this patent. All equivalent implementations of the present invention should be included within the scope of the claims of this patent.

Claims

1. A method for monitoring plastic waste on a beach using drones based on spectral features, characterized in that: The instance segmentation algorithm is used to extract the spectral features for plastic waste classification and identification, and the mass of each waste unit is estimated based on the identification results. The specific steps of the drone monitoring of plastic waste on the beach include:

1. Multispectral UAV aerial photography modeling 1-1: Use flight control software to operate a multispectral drone to carry out flight planning and aerial photography for the monitoring area, wherein the sensor bands of the multispectral drone include visible light, near infrared and short-wave infrared bands; 1-2: Before and after performing aerial photography tasks, diffuse reflection target plates must be photographed for radiation correction; 1-3: Perform two-dimensional reconstruction on the multispectral UAV collected data to generate visible light and multispectral orthophotos of the monitoring area, where the pixel values ​​in the multispectral images are presented in the form of reflectance; 2. Preprocessing of visible light and multispectral data 2-1: Geometrically register orthophotos of all bands, upsample the low-resolution multispectral images to the same resolution as the visible light images, so that the area covered by the visible light and multispectral orthophotos has the same pixel count; 2-2: Grid cropping of visible light and multispectral orthophotos to obtain sub-images with an aspect ratio of 1:1; 2-3: Use the LabelMe tool to label the garbage targets in the visible light orthophoto sub-image, and indicate the location, outline and category of the garbage targets, and generate a JSON file as the original data set; 2-4: Perform data augmentation on the original dataset and use it as the true value for training and evaluating the instance segmentation model; (III) Training instance segmentation model to detect garbage targets and generate masks 3-1: Mask R-CNN is used as the instance segmentation algorithm, and a deep learning environment is built based on the TensorFlow framework; 3-2: Adjust the size of the region proposal network anchor box according to the input image size; 3-3: The initial training selects the instance segmentation model weights based on the COCO dataset as the pre-trained weights of the backbone network. Subsequent transfer learning continues to iterate based on the weights of the existing plastic waste instance segmentation model, thereby expanding the applicable plastic waste categories and monitoring scenarios; 3-4: Divide the enhanced dataset into training set and test set at a ratio of 9:1 for model training and accuracy evaluation; 3-5: During the model training process, the total loss of each training cycle is calculated in real time to monitor the fitting and convergence of the instance segmentation model, and the maximum training cycle is set to ensure that the model reaches convergence while reducing the risk of overfitting. The total loss is the sum of the classification and bounding box regression loss of the region proposal network, the classification and bounding box regression loss of the backbone network, and the instance segmentation mask loss; 3-6: The trained instance segmentation model is used to preliminarily detect potential junk objects in visible light orthophotos, and generate masks that depict their shapes and cover their surfaces; 4. Extracting the image and spectral features of each garbage target 4-1: Extract image features and spectral features for all potential garbage targets in the monitoring area. The image features include: prediction confidence, projection area, aspect ratio and circularity, etc.; the spectral features include: spectral angle, reflectivity slope, reflectivity intercept and reflectivity mean, etc.; 4-2: Prediction confidence indicates the probability that a target is judged as a specific garbage category. The classification branch of the instance segmentation model detects each potential garbage target in the visible light image to complete the output. The segmentation branch of the instance segmentation model generates a mask of the potential garbage target, and continues to extract image features such as projected area, aspect ratio and circularity based on the visible light image. 4-3: Use the masks of each garbage target in the visible light image as the region of interest to crop the multispectral image, obtain the reflectance of each garbage target in different bands, and further extract spectral features such as spectral angle, reflectance slope, reflectance intercept and reflectance mean; 5. Using images and spectral features to classify and identify each garbage target 5-1: Combined with the image classification features of garbage targets, based on the supervised classification task, a discrimination model of multiple common types of garbage is constructed to predict whether each potential target belongs to the garbage category. The classification features include: image features such as prediction confidence, projection area, aspect ratio and circularity, and spectral features such as spectral angle, reflectance slope, reflectance intercept and reflectance mean; 5-2: Use the K nearest neighbor algorithm to train the classifier and obtain a discrimination model for common types of garbage. The accuracy verification indicators are precision, recall and F1 index; 6. Establishing a mass-area model for various types of waste for stock estimation 6-1: Count the mass and floor space of common types of plastic waste monomers, specifically by manually weighing to determine the mass of each monomer and using drone orthophotos to determine the floor space of each monomer; 6-2: Select fitting functions that match the physical properties of different types of garbage, such as material, density, and geometric shape, and use the measured data of multiple similar garbage units to construct multiple mass-occupied area regression models corresponding to the garbage categories; 6-3: Based on the classification results and projected area of ​​plastic waste on the beach, select the mass-occupied area regression model of the corresponding category of waste to complete the mass prediction of the waste monomer; 6-4: Classify and identify all garbage units in the monitoring area, calculate their area and predict their quality to estimate the amount of plastic waste on the shore.

2. The method for monitoring plastic waste on a beach by drone based on the combination of graph features according to claim 1 is characterized in that: The instance segmentation model consists of a classification branch and a segmentation branch; the classification branch detects potential junk objects; and the segmentation branch generates masks for each junk object.

3. The method for monitoring plastic waste on beaches by drone based on the combination of graph features according to claim 1 is characterized in that: The prediction confidence The classification branch of the instance segmentation model detects each potential junk object in the visible light image to complete the output.

4. The method for monitoring plastic waste on a beach by drone based on the combination of graph features according to claim 1 is characterized in that: The projected area The number of pixels covered by the corresponding mask for a garbage target Distance from drone to ground sampling The product of squares is expressed as follows: 。 5. The method for monitoring plastic waste on beaches by drone based on the combination of graph features according to claim 1 is characterized in that: The aspect ratio The aspect ratio of the minimum bounding rectangle of a garbage target.

6. The method for monitoring plastic waste on a beach by drone based on the combination of graph features according to claim 1 is characterized in that: The circularity 4π times the area of ​​a garbage target With perimeter The ratio of the squares is expressed as follows: 。 7. The method for monitoring plastic waste on a beach by drone based on the combination of graph features according to claim 1 is characterized in that: The spectral angle The measured spectrum curve of a garbage target with reference spectral curves for specific categories of waste The vector angle is expressed as follows: 。 8. The method for monitoring plastic waste on a beach by drone based on the combination of graph features according to claim 1 is characterized in that: The reflectivity slope Reflectance Intercept It is obtained by linearly fitting the reflectivity of a garbage target and the corresponding band number.

9. The method for monitoring plastic waste on a beach by drone based on the combination of graph features according to claim 1 is characterized in that: The mean reflectivity It is the arithmetic mean of the reflectivity of a garbage target in all bands.

Images