Floating object real-time identification method and system based on unmanned aerial vehicle

Abstract

The invention belongs to the technical field of water area inspection, and particularly relates to a real-time floater identification method and system based on an unmanned aerial vehicle, and the method comprises the steps: collecting image data of a target area through a camera carried by the unmanned aerial vehicle; inputting the image data into a target detection model for target detection, the target detection model being realized based on a Faster R-CNN or YOLO algorithm and being used for extracting candidate target areas; the candidate target area is input into an image segmentation model for image segmentation, and the image segmentation model is realized based on a ConvNeXt network and is used for carrying out pixel-level segmentation on the candidate target area; according to a result of the image segmentation, classifying the candidate target areas, and identifying the type of the floating object; and outputting the type and the position information of the floating object, so that the method has the effects of quickly responding to the change of the water surface environment, having good stability and reliability and adapting to various complicated water environment conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of water area inspection, and in particular to a real-time floating object identification method and system based on an unmanned aerial vehicle (UAV). Background Art

[0002] Traditional water environment management relies on manual inspections and traditional ground-based observations to monitor floating debris on the surface. These methods suffer from inefficiency and delayed information updates. Particularly in large rivers, lakes, and oceans, manual inspections to identify floating debris are not only time-consuming and labor-intensive, but also increase the difficulty of detecting and addressing them. Furthermore, manual inspections carry unpredictable safety risks.

[0003] In recent years, the development of drone technology has provided a new solution for monitoring floating debris on the water surface. Drones offer advantages such as speed, flexibility, and wide coverage, enabling more efficient scanning and identification of the water surface. Some existing drone recognition systems have achieved a certain degree of automatic identification of floating debris, but these systems often require large amounts of labeled data for training, have limited adaptability to environmental changes, and still have room for improvement in recognition accuracy and processing speed.

[0004] Therefore, a method and system that can accurately and efficiently identify floating objects on the water surface in real time is urgently needed to meet the needs of modern and intelligent water environment management. This method must not only be able to quickly respond to changes in the water surface environment but also possess excellent stability and reliability to adapt to various complex water conditions. The application of such a system will greatly improve the efficiency and quality of water environment management, effectively reduce environmental pollution, and protect water resources.

[0005] In response to the above technical defects, a real-time floating object identification method and system solution based on drone is proposed. Summary of the Invention

[0006] In order to solve the above problems, the present invention provides the following technical solutions:

[0007] A real-time floating object identification method based on a drone, comprising:

[0008] Collect image data of the target area through the camera carried by the drone;

[0009] Inputting the image data into a target detection model for target detection, wherein the target detection model is implemented based on the Faster R-CNN or YOLO algorithm to extract candidate target areas;

[0010] Inputting the candidate target area into an image segmentation model for image segmentation, wherein the image segmentation model is implemented based on a ConvNeXt network and is used to perform pixel-level segmentation on the candidate target area;

[0011] Classifying the candidate target area according to the image segmentation result and identifying the type of floating objects;

[0012] Output the type and location information of the floating object.

[0013] Furthermore, the target detection step includes preprocessing the image data collected by the drone, selecting FasterR-CNN or YOLO as the target detection model according to actual needs, and inputting the preprocessed image data into the selected target detection model;

[0014] Based on Faster R-CNN, a convolutional neural network is used to extract common features of the image. A region generation network is used to generate multiple candidate regions based on the extracted features, and the bounding box of each region is predicted to determine whether it contains the target object.

[0015] Generate multiple bounding boxes for each grid based on the YOLO model, and predict the center coordinates, width, height, confidence level, and category probability of each bounding box;

[0016] Feature extraction is performed on the candidate area to classify the target and fine-tune the bounding box. The category and position of the target object are determined by multiplying the confidence and category probability. The candidate target area is extracted to prepare for subsequent image segmentation and classification. The parameters of the target detection model are adjusted according to the actual detection effect.

[0017] Furthermore, the target detection includes preprocessing the image data before inputting it into the target detection model, reducing noise, building a noise reduction model and calculating as follows:

[0018]

[0019] Where I(x,y) is the value of the smoothed image at position (x,y), I(x',y') is the initial position value of the image, and σ is the standard deviation of the Gaussian kernel, which controls the degree of smoothing.

[0020] Adjust the image size according to the model input requirements and use histogram equalization or adaptive histogram equalization to enhance the contrast. The model construction calculation is as follows:

[0021]

[0022] Among them, H(γk) is the cumulative distribution function, γ i is the contrast of the image data when it is input, and k is the data point in the data set;

[0023] The preprocessed image data is input into the target detection model. FasterR-CNN uses ResNet as the feature extraction network to build a feature extraction calculation model as follows;

[0024] x=x l +F(x l ,{W i}),

[0025] Among them, x l is the input feature map, F(x l ,{W i}) are the convolution kernel parameters in the residual block;

[0026] The target detection model extracts the feature map of the image through CNN, extracts the general features of the image based on ResNet, and builds the convolution operation formula as follows:

[0027]

[0028] Among them, feature_map[i,j] is the feature map, kernel[k,l] is the convolution kernel, b is the bias term coefficient, K is the output channel, and L is the input channel.

[0029] Furthermore, the target detection includes classifying candidate regions and performing bounding box regression to determine the category and precise location of the target, extracting features from the candidate regions, performing classification and regression through a fully connected layer, and simultaneously predicting the category probability and bounding box offset in each candidate frame. The classification model is built and calculated as follows:

[0030] P(y=c|x)=Softmax(W c x+b c ),

[0031] Among them, W c x is the classification weight of category c, b c is the bias of category c;

[0032] Build a regression model and calculate as follows:

[0033]

[0034] Among them, y i is the coordinate of the true bounding box, and N is the number of coordinates of the bounding box;

[0035] After classification and regression, the target detection model outputs candidate target areas and removes overlapping candidate boxes.

[0036] Sort the candidate boxes from high to low according to the confidence, select the candidate box with the highest confidence, mark it as the final detection result, remove other candidate boxes whose overlap with the candidate box exceeds the threshold, and output the category, confidence and position of the bounding box of the detected target.

[0037] Furthermore, the step of inputting the candidate target region into the image segmentation model for image segmentation includes extracting the candidate target region identified by the target detection model from the original image, cropping the candidate region from the original image according to the bounding box coordinates output by the target detection model, adjusting the candidate region to the input size required by the image segmentation model, and normalizing the pixel values of the image to a uniform range;

[0038] The preprocessed candidate region image is input into the image segmentation model based on the ConvNeXt network for segmentation. The ConvNeXt-based image segmentation model is designed and implemented, the preprocessed candidate region image is input into the model, and the image data is converted into a format suitable for model input.

[0039] Furthermore, inputting the candidate target region into the image segmentation model for image segmentation includes selecting a loss function to measure the difference between the model prediction result and the true label. The cross entropy loss function is used, which is suitable for multi-class segmentation tasks and is calculated as follows:

[0040]

[0041] Among them, y i,j,c is the true label, p i,j,c is the probability predicted by the model, H and W are the height and width of the image, and C is the number of categories;

[0042] Select a suitable optimizer to update the model parameters and minimize the loss function. Use Adam optimization and the calculation is as follows:

[0043] m t =β1m1+(1-β1)g t ,

[0044] v t =β2v t-1 +(1-β2)g t 2 ,

[0045]

[0046] Among them, m t and v t are the first-order and second-order moment estimates, α is the learning rate, β1 and β2 are the decay rates, and ε is a small constant to prevent division by zero;

[0047] Through iterative training, model parameters are optimized, and segmentation performance is improved. The candidate target area is input into the trained image segmentation model to obtain pixel-level segmentation results. The segmentation results output by the model are post-processed to generate the final segmented image. The segmentation results from the candidate area are mapped back to the space of the original image to generate a complete segmented image. The segmentation performance of the model on the test set is evaluated. Based on the evaluation results, the model structure and parameters are optimized to improve segmentation performance. The trained image segmentation model is deployed on drones or other real-time monitoring equipment to monitor the images taken by the drone in real time, detect and segment the target area of floating objects, and provide real-time feedback.

[0048] Furthermore, the step of classifying the candidate target areas and identifying the types of floating objects based on the results of the image segmentation includes collecting a large number of images of floating objects, covering different types of floating objects, annotating the collected images, clarifying the type of floating objects corresponding to each candidate target area, dividing the annotated data set into a training set, a validation set, and a test set, adjusting the images to a uniform size required by the model input, increasing the diversity of training data through data enhancement, improving the generalization ability of the model, optimizing model parameters through iterative training, improving classification performance, finding the optimal model configuration through adjusting hyperparameters, reducing the size and computational complexity of the model so that it can run in a resource-constrained environment, and deploying the classification model on a drone or other real-time monitoring equipment to classify candidate target areas in real time.

[0049] According to one aspect of the present invention, a real-time floating object identification system based on a drone is provided, comprising:

[0050] UAV modules for flight control, obstacle avoidance, and image acquisition;

[0051] An image acquisition module for acquiring high-resolution images of the water surface area;

[0052] An image segmentation module, configured to segment the image and identify candidate target areas;

[0053] A classification module, configured to classify the candidate target area and identify the type of floating objects;

[0054] An alarm module is used to trigger a corresponding alarm mechanism according to the classification result;

[0055] The storage module is used to store the image and the processing result thereof.

[0056] According to one aspect of the present invention, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program and the processor implements the steps of the above-mentioned method for real-time identification of floating objects based on a drone when executing the computer program.

[0057] According to one aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for real-time identification of floating objects based on a drone are implemented.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] The present invention discloses a real-time floating object recognition method based on an unmanned aerial vehicle (UAV). The method comprises the following steps: collecting image data of a target area by a camera carried by the UAV; inputting the image data into a target detection model for target detection, wherein the target detection model is implemented based on a Faster R-CNN or YOLO algorithm and is used to extract candidate target areas; inputting the candidate target areas into an image segmentation model for image segmentation, wherein the image segmentation model is implemented based on a ConvNeXt network and is used to perform pixel-level segmentation on the candidate target areas; classifying the candidate target areas according to a result of the image segmentation and identifying the type of floating objects; and outputting the type of the floating objects and their position information. The method has the advantages of rapid response to changes in the water surface environment, good stability and reliability, and adaptability to various complex water environment conditions.

[0060] The present invention discloses a real-time floating object identification system based on a drone. The system comprises a drone module for flight control, obstacle avoidance and image acquisition; an image acquisition module for acquiring a high-resolution image of a water surface area; an image segmentation module for segmenting the image and identifying candidate target areas; a classification module for classifying the candidate target areas and identifying the type of floating objects; an alarm module for triggering a corresponding alarm mechanism according to the classification result; and a storage module for storing the image and its processing result. The system has the effect of efficiently scanning and identifying the water surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0062] Figure 1 It is an overall schematic diagram of the present invention;

[0063] Figure 2 It is a schematic diagram of the framework of the present invention;

[0064] Figure 3 Schematic diagram of the computer structure in the present invention. DETAILED DESCRIPTION

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0066] Example 1

[0067] like Figure 1-Figure 3 As shown, the present application provides a real-time floating object identification method based on a drone, comprising:

[0068] S1: Collect image data of the target area through the camera carried by the drone;

[0069] S2: Inputting the image data into a target detection model for target detection, wherein the target detection model is implemented based on Faster R-CNN or YOLO algorithm to extract candidate target areas;

[0070] S3: Inputting the candidate target area into an image segmentation model for image segmentation, wherein the image segmentation model is implemented based on a ConvNeXt network and is used to perform pixel-level segmentation on the candidate target area;

[0071] S4: Classifying the candidate target area according to the image segmentation result to identify the type of floating objects;

[0072] S5: Output the type and location information of the floating object.

[0073] Furthermore, the target detection step includes preprocessing the image data collected by the drone, selecting FasterR-CNN or YOLO as the target detection model according to actual needs, and inputting the preprocessed image data into the selected target detection model;

[0074] Based on Faster R-CNN, a convolutional neural network is used to extract common features of the image. A region generation network is used to generate multiple candidate regions based on the extracted features, and the bounding box of each region is predicted to determine whether it contains the target object.

[0075] Generate multiple bounding boxes for each grid based on the YOLO model, and predict the center coordinates, width, height, confidence level, and category probability of each bounding box;

[0076] Feature extraction is performed on the candidate area to classify the target and fine-tune the bounding box. The category and position of the target object are determined by multiplying the confidence and category probability. The candidate target area is extracted to prepare for subsequent image segmentation and classification. The parameters of the target detection model are adjusted according to the actual detection effect.

[0077] In one embodiment, the image data collected by the drone is preprocessed, including denoising, adjusting resolution, and enhancing contrast, to improve target detection accuracy. Faster R-CNN or YOLO is selected as the target detection model based on actual needs. Faster R-CNN is suitable for scenarios requiring high detection accuracy, while YOLO is suitable for scenarios requiring real-time detection. The preprocessed image data is input into the selected target detection model.

[0078] For Faster R-CNN: Use a convolutional neural network (CNN) such as ResNet to extract common features of the image. For YOLO: Divide the image into S×S grids, and each grid is responsible for detecting the target object whose center point falls within the area.

[0079] For Faster R-CNN: The Region Proposal Network (RPN) generates multiple candidate regions based on the extracted features, predicts the bounding box of each region and whether it contains the target object. For YOLO: Multiple bounding boxes are generated for each grid, and the center coordinates, width, height, confidence (probability of the existence of the target) and category probability of each bounding box are predicted.

[0080] For Faster R-CNN: Feature extraction is performed on the candidate region, and the target is classified and the bounding box is fine-tuned. For YOLO: The category and location of the target object are determined by multiplying the confidence score and the category probability.

[0081] Extract candidate target regions to prepare for subsequent image segmentation and classification. Based on the actual detection results, adjust the parameters of the target detection model, such as learning rate, batch size, loss function, etc., to improve detection accuracy and real-time performance.

[0082] Through the above steps, the target detection model can efficiently and accurately extract candidate areas that may contain floating objects from the images collected by the drone, laying a solid foundation for subsequent image segmentation and classification, thereby realizing real-time recognition of floating objects based on drones.

[0083] Furthermore, the target detection includes preprocessing the image data before inputting it into the target detection model, reducing noise, building a noise reduction model and calculating as follows:

[0084]

[0085] Where I(x,y) is the value of the smoothed image at position (x,y), I(x',y') is the initial position value of the image, and σ is the standard deviation of the Gaussian kernel, which controls the degree of smoothing.

[0086] Adjust the image size according to the model input requirements and use histogram equalization or adaptive histogram equalization to enhance the contrast. The model construction calculation is as follows:

[0087]

[0088] Among them, H(γk) is the cumulative distribution function, γ i is the contrast of the image data when it is input, and k is the data point in the data set;

[0089] The preprocessed image data is input into the target detection model. FasterR-CNN uses ResNet as the feature extraction network to build a feature extraction calculation model as follows;

[0090] x=x l +F(x l ,{W i}),

[0091] Among them, x l is the input feature map, F(x l ,{W i}) are the convolution kernel parameters in the residual block;

[0092] The target detection model extracts the feature map of the image through CNN, extracts the general features of the image based on ResNet, and builds the convolution operation formula as follows:

[0093]

[0094] Among them, feature_map[i,j] is the feature map, kernel[k,l] is the convolution kernel, b is the bias term coefficient, K is the output channel, and L is the input channel.

[0095] Furthermore, the target detection includes classifying candidate regions and performing bounding box regression to determine the category and precise location of the target, extracting features from the candidate regions, performing classification and regression through a fully connected layer, and simultaneously predicting the category probability and bounding box offset in each candidate frame. The classification model is built and calculated as follows:

[0096] P(y=c|x)=Softmax(W c x+b c ),

[0097] Among them, W c x is the classification weight of category c, b cis the bias of category c;

[0098] Build a regression model and calculate as follows:

[0099]

[0100] Among them, y i is the coordinate of the true bounding box, and N is the number of coordinates of the bounding box;

[0101] After classification and regression, the target detection model outputs candidate target areas and removes overlapping candidate boxes.

[0102] Sort the candidate boxes from high to low according to the confidence, select the candidate box with the highest confidence, mark it as the final detection result, remove other candidate boxes whose overlap with the candidate box exceeds the threshold, and output the category, confidence and position of the bounding box of the detected target.

[0103] In one embodiment, before inputting the image into the target detection model, preprocessing is required to improve the accuracy and efficiency of detection. A Gaussian filter is used to reduce image noise. The image size is adjusted according to the model input requirements. For example, YOLO typically requires an input size of 416x416. Histogram equalization or adaptive histogram equalization (CLAHE) is used to enhance contrast. The preprocessed image data is input into the target detection model. Faster R-CNN uses ResNet as the feature extraction network. The target detection model extracts feature maps of the image through CNN. ResNet extracts common features of the image.

[0104] The object detection model generates candidate regions for subsequent classification and regression. Region proposal networks (RPNs) are used to generate candidate regions. The image is divided into an S×S grid, and multiple candidate boxes are generated for each grid. Classification and bounding box regression are performed on the candidate regions to determine the class and precise location of the object. Feature extraction is performed on the candidate regions, followed by classification and regression using a fully connected layer. For each candidate box, both the class probability and the bounding box offset are predicted. After classification and regression, the object detection model outputs candidate object regions.

[0105] Sort candidate boxes by confidence from high to low. Select the candidate box with the highest confidence and mark it as the final detection result. Remove other candidate boxes whose overlap with this candidate box exceeds the threshold. Repeat steps 2 and 3 until all candidate boxes have been processed.

[0106] Outputs the detected object's category, confidence score, and bounding box location. In object detection models, tuning key parameters significantly impacts performance. This parameter controls the step size for parameter updates during optimization. Common values include 0.001 (default) and 0.0001 (suitable for large datasets).

[0107] The number of samples used in each training run. Common values are 32, 64, and 128. This value measures the difference between the model's predictions and the true values. It balances the weights of classification loss and regression loss. In YOLO, the grid division determines the area each grid is responsible for detecting.

[0108] The number of categories determines the types of objects the model needs to distinguish. Detecting floating objects may require distinguishing between plastic, plants, and trash. This is used to filter out detection results with low confidence. Overlap threshold: Typically set to 0.5. Confidence threshold: Typically set to 0.5.

[0109] Through the detailed steps and mathematical model described above, the object detection model can efficiently and accurately extract candidate regions that may contain floating objects from drone-collected images. These candidate regions are then fed into the image segmentation model for pixel-level segmentation, ultimately enabling real-time drone-based floating object identification. Throughout this process, parameter selection and optimization significantly impact model performance and require adjustment and optimization based on the specific application scenario.

[0110] Furthermore, the step of inputting the candidate target region into the image segmentation model for image segmentation includes extracting the candidate target region identified by the target detection model from the original image, cropping the candidate region from the original image according to the bounding box coordinates output by the target detection model, adjusting the candidate region to the input size required by the image segmentation model, and normalizing the pixel values of the image to a uniform range;

[0111] The preprocessed candidate region image is input into the image segmentation model based on the ConvNeXt network for segmentation. The ConvNeXt-based image segmentation model is designed and implemented, the preprocessed candidate region image is input into the model, and the image data is converted into a format suitable for model input.

[0112] Furthermore, inputting the candidate target region into the image segmentation model for image segmentation includes selecting a loss function to measure the difference between the model prediction result and the true label. The cross entropy loss function is used, which is suitable for multi-class segmentation tasks and is calculated as follows:

[0113]

[0114] Among them, y i,j,c is the true label, p i,j,c is the probability predicted by the model, H and W are the height and width of the image, and C is the number of categories;

[0115] Select a suitable optimizer to update the model parameters and minimize the loss function. Use Adam optimization and the calculation is as follows:

[0116] m t =β1m1+(1-β1)g t ,

[0117] v t =β2v t-1 +(1-β2)g t 2 ,

[0118]

[0119] Among them, m t and v t are the first-order and second-order moment estimates, α is the learning rate, β1 and β2 are the decay rates, and ε is a small constant to prevent division by zero;

[0120] Through iterative training, model parameters are optimized, and segmentation performance is improved. The candidate target area is input into the trained image segmentation model to obtain pixel-level segmentation results. The segmentation results output by the model are post-processed to generate the final segmented image. The segmentation results from the candidate area are mapped back to the space of the original image to generate a complete segmented image. The segmentation performance of the model on the test set is evaluated. Based on the evaluation results, the model structure and parameters are optimized to improve segmentation performance. The trained image segmentation model is deployed on drones or other real-time monitoring equipment to monitor the images taken by the drone in real time, detect and segment the target area of floating objects, and provide real-time feedback.

[0121] In one embodiment, the candidate object regions identified by the object detection model are extracted from the original image. Based on the bounding box coordinates output by the object detection model, the candidate regions are cropped from the original image. The cropped image ensures that the candidate object regions are complete and the background is properly processed.

[0122] Resize the image to fit the candidate regions to the input size required by the image segmentation model. Use bilinear interpolation or other interpolation methods to scale the image to the specified width and height. Avoid introducing excessive distortion when resizing the image, which can affect the segmentation accuracy of the model. Normalize the image's pixel values to a uniform range, typically [-1, 1] or [0, 1]. Normalize the image using the mean and standard deviation, ensuring that all candidate regions have the same normalization parameters to maintain consistency across the model input.

[0123] Increase the diversity of training data through data augmentation to improve the model's generalization ability. Apply transformations such as random flipping, rotation, scaling, and color jittering to generate more diverse candidate region samples. When performing data augmentation, ensure that the labels (i.e., segmentation masks) are synchronized with the image transformations to maintain consistency.

[0124] The preprocessed candidate region images need to be input into a ConvNeXt-based image segmentation model for segmentation. Design and implement a ConvNeXt-based image segmentation model. Use the ConvNeXt network as a feature extractor to extract deep features of the candidate regions. Design a sampling module to gradually downsample the feature maps extracted by the encoder to the original image size, generating pixel-level segmentation results. Introduce skip connections between the encoder and decoder to fuse feature maps at different levels and improve segmentation accuracy. Ensure that the encoder and decoder structures are symmetrical and that the feature maps match in size to avoid information loss.

[0125] Input the preprocessed candidate region image into the model. Convert the image data into a format suitable for model input, such as PyTorch's Tensor format, and load it onto a GPU or TPU for accelerated computation. Ensure that the batch size matches the hardware resources to avoid memory overflows. To ensure that the image segmentation model accurately segments the target region for floating objects, it must be trained and the model parameters must be adjusted to minimize the discrepancy between the predicted results and the true labels.

[0126] Choose a loss function to measure the difference between the model's predictions and the true labels. The cross-entropy loss function is commonly used and is suitable for multi-class segmentation tasks. If the ratio of the target area to the background is uneven, a weighted cross-entropy loss function can be introduced to give a higher weight to the minority class. Choose an appropriate optimizer to update the model parameters and minimize the loss function. Common optimizers include Adam and SGD. For example, for the Adam optimizer, set the learning rate and optimizer parameters appropriately based on the model size and the amount of training data.

[0127] Through iterative training, optimize model parameters and improve segmentation performance. Divide the training dataset into batches and feed each batch into the model for forward propagation. Calculate the loss function, perform backpropagation, and update the model parameters. At the end of each training cycle, evaluate model performance using the validation set, recording the loss and segmentation metrics. Set an appropriate number of training epochs to avoid overfitting; use early stopping to terminate training if the validation loss does not decrease for several consecutive epochs.

[0128] After model training is complete, it needs to be applied to actual candidate object regions for pixel-level segmentation and post-processing to obtain the final segmented image. The candidate object regions are input into the trained image segmentation model to obtain pixel-level segmentation results. Forward propagation is performed using the trained model to output the class probability distribution for each pixel. Ensure that the input format and preprocessing steps during inference are consistent with those during training to avoid errors caused by format differences.

[0129] Post-process the segmentation results output by the model to generate the final segmented image. Convert the class probability distribution of each pixel into a class label. Identify and merge adjacent pixels of the same type to form connected regions and remove noise points. Smooth the boundaries of the segmentation results to reduce jagged artifacts and improve visual quality. Adjust post-processing parameters based on actual needs to ensure the accuracy and aesthetics of the segmentation results.

[0130] Map the segmentation results from the candidate region back to the space of the original image to generate a complete segmented image. Relocate the segmentation results to the corresponding position in the original image based on the bounding box coordinates of the candidate region. Ensure the accuracy of the coordinate transformation to avoid positional offset of the segmentation results. In order to ensure that the performance of the image segmentation model meets expectations, it needs to be comprehensively evaluated and optimized based on the evaluation results. Evaluate the segmentation performance of the model on the test set. Use multiple indicators such as pixel accuracy (PixelAccuracy), mean intersection over union (mIOU), F1 score, etc. to comprehensively evaluate the performance of the model. The proportion of correctly classified pixels to the total number of pixels. Select a representative test set to ensure that the evaluation results can reflect the performance of the model in actual applications.

[0131] Based on the evaluation results, optimize the model's structure and parameters to improve segmentation performance. Adjust hyperparameters such as the learning rate, batch size, and regularization strength to find the optimal parameter combination. Increase or decrease the number of network layers, adjust the size of feature maps, and optimize the network structure. Introduce more diverse data augmentation methods to increase the diversity of training data. Introduce auxiliary loss functions such as boundary loss and Dice loss to improve the model's ability to capture target boundaries. When optimizing the model, make adjustments gradually to avoid changing multiple parameters simultaneously, which makes it difficult to track the optimization results.

[0132] After model optimization is complete, it needs to be deployed in a real-world application environment for real-time floating object monitoring. Deploy the trained image segmentation model to a drone or other real-time monitoring device. Use model compression and optimization techniques, such as model quantization, pruning, and knowledge distillation, to reduce the model size and computational complexity, enabling it to run in resource-constrained environments. Ensure that the hardware resources of the deployment environment match the model's computational requirements to avoid performance bottlenecks caused by insufficient resources. Monitor drone images in real time, detect and segment the target area of floating objects, and provide real-time feedback. Obtain drone images in real time and perform object detection and image segmentation. Visualize the segmentation results and overlay them on the original image to provide intuitive monitoring feedback. Based on the segmentation results, determine the type and number of floating objects and trigger appropriate alarm mechanisms. Ensure real-time and stable data stream processing to avoid poor monitoring results due to delays or interruptions. Based on feedback from real-world applications, continuously optimize and update the image segmentation model to improve its adaptability and performance. Continuously collect and annotate new floating object image data to enrich the training data. Regularly retrain the model using new datasets, updating model parameters to adapt to new environments and scenarios. Monitor the performance of the model in real-time, identifying and resolving potential issues promptly. Establish a comprehensive feedback mechanism and update process to ensure continuous optimization and stable operation of the model.

[0133] Inputting candidate target regions into a ConvNeXt-based image segmentation model for image segmentation is a systematic and complex process. From data preprocessing to model training, inference, and post-processing, each step requires meticulous effort and in-depth consideration. Through rational process design and continuous optimization and adjustment, an efficient and accurate image segmentation model can be achieved to meet the requirements of real-time floating object identification for drone-based systems.

[0134] Furthermore, the step of classifying the candidate target areas and identifying the types of floating objects based on the results of the image segmentation includes collecting a large number of images of floating objects, covering different types of floating objects, annotating the collected images, clarifying the type of floating objects corresponding to each candidate target area, dividing the annotated data set into a training set, a validation set, and a test set, adjusting the images to a uniform size required by the model input, increasing the diversity of training data through data enhancement, improving the generalization ability of the model, optimizing model parameters through iterative training, improving classification performance, finding the optimal model configuration through adjusting hyperparameters, reducing the size and computational complexity of the model so that it can run in a resource-constrained environment, and deploying the classification model on a drone or other real-time monitoring equipment to classify candidate target areas in real time.

[0135] In one embodiment, a large number of images of floating objects are collected, covering different types of floating objects, such as plastic bottles, branches, leaves, garbage bags, etc. Image data is collected by drone photography, network resource acquisition, or field photography. The collected images are labeled to clarify the type of floating objects corresponding to each candidate target area. Image annotation tools (such as Labelme, CVAT, etc.) are used to label the floating objects in the image and generate corresponding category labels. The labeled data set is divided into training set, validation set, and test set, usually in a ratio of 60%, 20%, and 20%. The data is randomly divided to ensure a balanced distribution of each category of floating objects in each data set.

[0136] Choose a model architecture suitable for image classification tasks. Convolutional neural networks (CNNs) are commonly used, such as ResNet, VGG, and EfficientNet. You can also use pretrained models and improve performance through transfer learning. Design or select a model architecture suitable for floating object classification tasks. Use a pretrained CNN model, such as ResNet-50, as a feature extractor. Add fully connected layers to the pretrained model for classification. Adjust the network depth and width based on the amount of data and computing resources to optimize model performance.

[0137] Increase the diversity of training data through data augmentation to improve the model's generalization ability. Apply transformations such as random flipping, rotation, scaling, and color jittering to generate more diverse training samples. Optimize model parameters and improve classification performance through iterative training. Divide the training dataset into batches and feed each batch into the model for forward propagation. Calculate the loss function and perform backpropagation to update the model parameters. At the end of each training cycle, evaluate model performance using the validation set, recording the loss value and classification metrics. Find the optimal model configuration by adjusting hyperparameters. Use grid search or random search to adjust hyperparameters such as learning rate, batch size, and regularization strength to optimize model performance.

[0138] Evaluate the model's classification performance on the test set. Use metrics such as accuracy, precision, recall, and F1 score to comprehensively evaluate the model's performance. Based on the evaluation results, optimize the model's structure and parameters to improve classification performance. Increase or decrease the number of network layers, adjust the size of feature maps, and optimize the network structure. Introduce more diverse data augmentation methods to increase the diversity of training data. Introduce methods to handle class imbalance, such as weighted loss functions, to improve classification performance for minority classes.

[0139] Reduce the model's size and computational complexity to enable operation in resource-constrained environments. Optimize model performance using techniques such as model quantization, pruning, and knowledge distillation. Deploy the classification model on drones or other real-time monitoring equipment to classify candidate target areas in real time. Obtain candidate target areas from image segmentation in real time and input them into the classification model for classification. Visualize the classification results and overlay them on the original image to provide intuitive monitoring feedback. Based on the classification results, determine the type and quantity of floating objects and trigger appropriate alarm mechanisms.

[0140] Based on feedback from real-world applications, we continuously optimize and update the classification model to improve its adaptability and performance. We continuously collect and annotate new floating object image data to enrich the training data. We regularly retrain the model using new datasets, updating model parameters to adapt to new environments and scenarios. We monitor the model's performance in real-world applications in real time to promptly identify and resolve potential issues.

[0141] Classifying candidate target areas and identifying floating object types based on image segmentation results is a systematic and complex process. From data preparation to model training, evaluation, and application, each step requires meticulous effort and in-depth consideration. Through rational process design and continuous optimization and adjustment, an efficient and accurate classification model can be achieved to meet the needs of real-time drone-based floating object identification.

[0142] According to one aspect of the present invention, a real-time floating object identification system based on a drone is provided, comprising:

[0143] UAV modules for flight control, obstacle avoidance, and image acquisition;

[0144] An image acquisition module for acquiring high-resolution images of the water surface area;

[0145] An image segmentation module, configured to segment the image and identify candidate target areas;

[0146] A classification module, configured to classify the candidate target area and identify the type of floating objects;

[0147] An alarm module is used to trigger a corresponding alarm mechanism according to the classification result;

[0148] The storage module is used to store the image and the processing result thereof.

[0149] Example 2

[0150] The system of the present invention includes a drone module for flight control, obstacle avoidance and image acquisition. The flight control system controls the flight path and altitude of the drone. The obstacle avoidance system detects and avoids obstacles in the flight path. The communication module transmits data with the ground station or other equipment. The image acquisition module obtains high-resolution images of the water surface area. The high-resolution camera takes high-definition water surface images. The wide-angle lens expands the shooting range and covers a wider area of water. The image stabilizer reduces image blur caused by shaking during flight. The image segmentation module segments the image and identifies candidate target areas. A deep learning algorithm (such as U-Net) is used for image segmentation to accurately extract candidate areas. The classification module classifies candidate target areas and identifies the type of floating objects. A pre-trained convolutional neural network (such as ResNet-50) is used for classification to support the recognition of multiple types of floating objects.

[0151] The alarm module triggers the corresponding alarm mechanism based on the classification results. Alarm triggering triggers an alarm signal when a specific type of floating object is detected. Alarm methods notify relevant personnel through sound, light, or text message. The storage module stores images and their processing results. Data storage stores the original images and processing results in the cloud or a local database. Data management provides data query, retrieval, and management capabilities.

[0152] Example 3

[0153] The image processing process of the invention includes the following steps: image acquisition, which uses a high-resolution camera mounted on a drone to capture images of the water surface area; image preprocessing, which performs preprocessing such as denoising and enhancement to improve image quality; image segmentation, which uses a deep learning algorithm to segment the image and identify candidate target areas; feature extraction, which extracts features such as color and texture from the candidate areas in preparation for classification; and classification recognition, which inputs the extracted features into a classification model to identify the type of floating object.

[0154] Example 4

[0155] The classification module and the alarm module of the present invention include the following steps:

[0156] Feature extraction extracts visual features from candidate target regions, such as color histograms and texture features. Feature engineering processes the extracted features, such as normalization and dimensionality reduction. Classifier training uses deep learning models (such as ResNet and EfficientNet) to train classifiers, supporting the recognition of multiple types of floating objects. Classification prediction inputs the candidate region to be classified into a trained classifier, which outputs the type of floating object.

[0157] Classification Result Analysis: Analyzes the classification module's output to determine whether an alarm should be triggered. Alarm Trigger: When the type or quantity of floating objects detected exceeds a preset threshold, an alarm signal is triggered. Alarm Mode Selection: Select different alarm methods as needed, such as sound, light, and text message. Alarm Information Transmission: Sends alarm information to relevant personnel or systems via the communication module.

[0158] In order to improve the real-time performance and accuracy of the system, the present invention adopts the following optimization measures: lightweight model design, using lightweight deep learning models (such as MobileNet, ShuffleNet), while ensuring classification accuracy, reducing the consumption of computing resources and improving real-time performance. Parallel processing, in the image processing and classification process, using multi-threading or GPU acceleration technology to improve the processing speed of the system. Data enhancement, when training the classification model, using data enhancement technology (such as rotation, flipping, color jittering, etc.) to increase the diversity of training data and improve the generalization ability and accuracy of the model. Dynamic threshold adjustment, according to the actual application scenario, dynamically adjust the threshold of the alarm trigger to reduce false positives and missed reports.

[0159] The system of this invention has been put to practical use in a large reservoir environmental protection project. By deploying multiple drones equipped with the system, comprehensive monitoring of the reservoir's water surface is achieved. The system can identify and classify floating objects, such as plastic bottles, branches, and leaves, in real time. When large quantities of floating objects are detected, an alarm is triggered, notifying cleanup personnel to address them. In actual operation, the system's recognition accuracy has reached over 95%, meeting the real-time requirements of practical applications and effectively improving the efficiency of reservoir environmental protection and cleanup.

[0160] The present invention discloses a real-time floating object recognition method and system based on a drone. The method comprises the following steps: collecting image data of a target area through a camera carried by the drone; inputting the image data into a target detection model for target detection, wherein the target detection model is implemented based on a FasterR-CNN or YOLO algorithm and is used to extract candidate target areas; inputting the candidate target areas into an image segmentation model for image segmentation, wherein the image segmentation model is implemented based on a ConvNeXt network and is used to perform pixel-level segmentation on the candidate target areas; classifying the candidate target areas according to the image segmentation results and identifying the types of floating objects; and outputting the floating objects. The system can quickly respond to changes in the water surface environment and has good stability and reliability to adapt to various complex water environment conditions; the UAV module is used for flight control, obstacle avoidance and image acquisition; the image acquisition module is used to obtain high-resolution images of the water surface area; the image segmentation module is used to segment the image and identify the candidate target area; the classification module is used to classify the candidate target area and identify the type of floating objects; the alarm module is used to trigger the corresponding alarm mechanism according to the classification result; the storage module is used to store the image and its processing results, which has the effect of efficiently scanning and identifying the water surface.

[0161] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned method for real-time identification of floating objects based on a drone are implemented.

[0162] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned method for real-time identification of floating objects based on a drone are implemented.

[0163] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0164] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0165] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to specific embodiments. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A real-time floating object identification method based on drone, characterized in that: include: Collect image data of the target area through the camera carried by the drone; Inputting the image data into a target detection model for target detection, wherein the target detection model is implemented based on the Faster R-CNN or YOLO algorithm to extract candidate target areas; Inputting the candidate target area into an image segmentation model for image segmentation, wherein the image segmentation model is implemented based on a ConvNeXt network and is used to perform pixel-level segmentation on the candidate target area; Classifying the candidate target area according to the image segmentation result and identifying the type of floating objects; Output the type and location information of the floating object.

2. The method for real-time identification of floating objects based on drones according to claim 1, characterized in that: The target detection step includes preprocessing the image data collected by the drone, selecting FasterR-CNN or YOLO as the target detection model according to actual needs, and inputting the preprocessed image data into the selected target detection model; Based on Faster R-CNN, a convolutional neural network is used to extract common features of the image. A region generation network is used to generate multiple candidate regions based on the extracted features, and the bounding box of each region is predicted to determine whether it contains the target object. Generate multiple bounding boxes for each grid based on the YOLO model, and predict the center coordinates, width, height, confidence level, and category probability of each bounding box; Feature extraction is performed on the candidate area to classify the target and fine-tune the bounding box. The category and position of the target object are determined by multiplying the confidence and category probability. The candidate target area is extracted to prepare for subsequent image segmentation and classification. The parameters of the target detection model are adjusted according to the actual detection effect.

3. The method for real-time identification of floating objects based on drone according to claim 1, characterized in that: The target detection includes preprocessing the image data before inputting it into the target detection model, reducing noise, building a noise reduction model and calculating as follows: Where I(x,y) is the value of the smoothed image at position (x,y), I(x',y') is the initial position value of the image, and σ is the standard deviation of the Gaussian kernel, which controls the degree of smoothing. Adjust the image size according to the model input requirements and use histogram equalization or adaptive histogram equalization to enhance the contrast. The model construction calculation is as follows: Among them, H(γk) is the cumulative distribution function, γ i is the contrast of the image data when it is input, and k is the data point in the data set; The preprocessed image data is input into the target detection model. FasterR-CNN uses ResNet as the feature extraction network to build a feature extraction calculation model as follows; x=x l +F(x l ,{W i }), Among them, x l is the input feature map, F(x l ,{W i }) are the convolution kernel parameters in the residual block; The target detection model extracts the feature map of the image through CNN, extracts the general features of the image based on ResNet, and builds the convolution operation formula as follows: Among them, feature_map[i,j] is the feature map, kernel[k,l] is the convolution kernel, b is the bias term coefficient, K is the output channel, and L is the input channel.

4. The method for real-time identification of floating objects based on drones according to claim 1, characterized in that: The target detection includes classifying candidate regions and performing bounding box regression to determine the target category and precise location, extracting features from the candidate regions, performing classification and regression through a fully connected layer, and simultaneously predicting the category probability and bounding box offset in each candidate frame. The classification model is built and calculated as follows: P(y=c|x)=Softmax(W c x+b c ), Among them, W c x is the classification weight of category c, b c is the bias of category c; Build a regression model and calculate as follows: Among them, y i is the coordinate of the true bounding box, and N is the number of coordinates of the bounding box; After classification and regression, the target detection model outputs candidate target areas and removes overlapping candidate boxes. Sort the candidate boxes from high to low according to the confidence, select the candidate box with the highest confidence, mark it as the final detection result, remove other candidate boxes whose overlap with the candidate box exceeds the threshold, and output the category, confidence and position of the bounding box of the detected target.

5. The method for real-time identification of floating objects based on drone according to claim 1, characterized in that: The step of inputting the candidate target region into the image segmentation model for image segmentation includes extracting the candidate target region identified by the target detection model from the original image, cropping the candidate region from the original image according to the bounding box coordinates output by the target detection model, adjusting the candidate region to the input size required by the image segmentation model, and normalizing the pixel values of the image to a uniform range; The preprocessed candidate region image is input into the image segmentation model based on the ConvNeXt network for segmentation. The ConvNeXt-based image segmentation model is designed and implemented, the preprocessed candidate region image is input into the model, and the image data is converted into a format suitable for model input.

6. The method for real-time identification of floating objects based on drones according to claim 5, characterized in that: Inputting the candidate target region into the image segmentation model for image segmentation includes selecting a loss function to measure the difference between the model prediction result and the true label. The cross entropy loss function is used, which is suitable for multi-class segmentation tasks and is calculated as follows: Among them, y i,j,c is the true label, p i,j,c is the probability predicted by the model, H and W are the height and width of the image, and C is the number of categories; Select a suitable optimizer to update the model parameters and minimize the loss function. Use Adam optimization and the calculation is as follows: m t =β1m1+(1-β1)g t , v t =β2v t-1 +(1-β2)g t 2 , Among them, m t and v t are the first-order and second-order moment estimates, α is the learning rate, β1 and β2 are the decay rates, and ε is a small constant to prevent division by zero; Through iterative training, model parameters are optimized, and segmentation performance is improved. The candidate target area is input into the trained image segmentation model to obtain pixel-level segmentation results. The segmentation results output by the model are post-processed to generate the final segmented image. The segmentation results from the candidate area are mapped back to the space of the original image to generate a complete segmented image. The segmentation performance of the model on the test set is evaluated. Based on the evaluation results, the model structure and parameters are optimized to improve segmentation performance. The trained image segmentation model is deployed on drones or other real-time monitoring equipment to monitor the images taken by the drone in real time, detect and segment the target area of floating objects, and provide real-time feedback.

7. The method for real-time identification of floating objects based on drone according to claim 5, characterized in that: The step of classifying the candidate target areas and identifying the types of floating objects based on the image segmentation results includes collecting a large number of images of floating objects, covering different types of floating objects, annotating the collected images, clarifying the type of floating objects corresponding to each candidate target area, dividing the annotated data set into a training set, a validation set, and a test set, adjusting the images to a uniform size required by the model input, increasing the diversity of training data through data enhancement to improve the generalization ability of the model, optimizing model parameters through iterative training to improve classification performance, finding the optimal model configuration through adjusting hyperparameters to reduce the size and computational complexity of the model so that it can operate in a resource-constrained environment, and deploying the classification model on a drone or other real-time monitoring equipment to classify candidate target areas in real time.

8. A real-time floating object identification system based on drones, characterized in that: include: UAV modules for flight control, obstacle avoidance, and image acquisition; An image acquisition module for acquiring high-resolution images of the water surface area; An image segmentation module, configured to segment the image and identify candidate target areas; A classification module, configured to classify the candidate target area and identify the type of floating objects; An alarm module is used to trigger a corresponding alarm mechanism according to the classification result; The storage module is used to store the image and the processing result thereof.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the real-time floating object identification method based on a drone are implemented as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a real-time floating object identification method based on a drone are implemented as described in any one of claims 1 to 6.

Images