A river channel floating object detection method based on transfer learning and target detection algorithm

Abstract

The present application relates to a kind of river floating object detection methods based on transfer learning and target detection algorithm, comprising the following steps: obtaining real-time video or image data containing river surface floating object, and pre-processing;Select the deep learning target detection model pre-trained based on large-scale image data set, through transfer learning technology, after using data enhancement technology, training is carried out;The preprocessed image is input into the trained deep learning model, the model is extracted by multi-layer convolutional neural network Feature extraction, classification and detection frame positioning, prediction is carried out to floating object class, confidence is respectively comprehensive evaluation to the accuracy of target in detection frame and the accuracy of border prediction;Parameter individualization adjustment;Detection result analysis and application.The problems that the detection of river floating object is difficult to solve floating object variety, easy to be confused with other objects, complex environment, significantly improve the accuracy and real-time of floating object detection.

Description

Technical Field

[0001] This invention relates to the field of aquatic environment monitoring and image processing technology, and in particular to a method for detecting floating objects in rivers based on deep learning algorithms and transfer learning technology. It is suitable for automatically detecting floating objects on the water surface, such as garbage and leaves, and distinguishing them from misidentifications with other objects (such as people, boats, birds, etc.). Background Technology

[0002] Floating debris in waterways not only affects the urban landscape but can also lead to water pollution or obstruct normal drainage. Therefore, accurate and automated detection of floating debris in waterways is of significant importance for environmental protection and water resource management. Traditional methods for detecting floating debris mostly rely on manual inspections or automated detection based on simple image processing. These methods struggle to cope with complex aquatic environments and various disturbances, such as changes in lighting and wave reflections.

[0003] With the development of deep learning technology, object detection algorithms based on convolutional neural networks (CNNs) have made significant progress in image recognition and object detection. However, the detection of floating objects in river channels still faces the following challenges:

[0004] 1. Diverse types of floating objects: Floating objects vary greatly in shape, size, and color;

[0005] 2. Easily confused with other objects: such as ships, birds, etc.;

[0006] 3. Complex environment: Reflection of the water surface and changes in light can affect the accuracy of detection. Summary of the Invention

[0007] To address the challenges of detecting floating objects in rivers, such as the diverse types of objects, their susceptibility to confusion with other objects, and the complexity of the environment, a new floating object detection method based on transfer learning and object detection algorithms is proposed. By fine-tuning a pre-trained model and combining it with real-world data from river scenarios, the method significantly improves the accuracy and real-time performance of floating object detection.

[0008] The technical solution of this invention is as follows:

[0009] A method for detecting floating objects in river channels based on transfer learning and object detection algorithms includes the following steps:

[0010] Step 1: Data Acquisition and Preprocessing

[0011] Acquire real-time video or image data containing floating objects on the river surface, preprocess the acquired images, determine and label the target categories to be detected, and then perform denoising, image enhancement and resizing.

[0012] Step 2: Model Initialization and Transfer Learning

[0013] A deep learning object detection model pre-trained on a large-scale image dataset was selected. Transfer learning techniques were used to fine-tune the pre-trained model using labeled data from a specific river scene. The labeled dataset was divided into training and validation sets, and cross-validation was applied to select suitable parameters for training. Data augmentation techniques were used during training, and the loss function was optimized using cross-entropy loss and L2 regularization. For classes included in the pre-trained model, or classes that had already been trained, transfer training was performed for optimization. The training process was monitored using TensorBoard to obtain changes in the loss function and learning rate, and the average precision (mAP) was output.

[0014] Step 3: Real-time floating object detection:

[0015] The preprocessed image is input into a trained deep learning model. The model uses a multi-layer convolutional neural network to extract features, classify, and locate bounding boxes, predicting floating objects. The confidence scores are used to comprehensively evaluate the accuracy of the target within the bounding box and the accuracy of the bounding box prediction. The target accuracy is expressed by the probability of the target being in the predicted box, while the bounding box prediction accuracy is expressed by the intersection-over-union ratio (IoU). Low-confidence bounding boxes are removed, while high-confidence floating object targets are retained. For potentially overlapping bounding boxes, the NMS algorithm is used to select the optimal bounding box.

[0016] Step 4: Personalized parameter adjustment: Adjust the following parameters flexibly according to project requirements: detection category, confidence threshold, detection box area ratio, intersection-union ratio, and polygon masking;

[0017] Step 5: Analysis and application of test results:

[0018] The detection results include key information about floating objects; the detection system outputs alarm events containing information about floating objects in real time, facilitating the removal or further analysis of floating objects; the detection system can be linked with other IoT sensing devices; based on the real-time detection results, users can continuously adjust parameters to optimize the performance of the detection system and ensure optimal results in different environments.

[0019] Furthermore, the specific steps include:

[0020] Step 1: Data Acquisition and Preprocessing

[0021] Step 1.1 River Channel Image Acquisition

[0022] Use high-definition cameras to acquire real-time video or image data of the river channel, ensuring that the image clarity and coverage are sufficient to capture floating objects on the river surface;

[0023] Step 1.2 Image Preprocessing

[0024] The preprocessing of the acquired images includes the following steps:

[0025] Step 1.2.1 Determine the target category to be detected; the training model has limited training details for target categories, so it can be trained in reverse on non-floating objects to reduce false recognition; floating objects on the water surface in different regions are divided into garbage and non-garbage categories, which can be further subdivided according to shape and material.

[0026] Step 1.2.2 Use the annotation tool to annotate the collected data according to the previously determined target category and assign a category label;

[0027] Step 1.2.3 Denoising: Use filters to remove noise from the image; increase the background image as much as possible; perform convolution on the image, and for points on the image that coincide with the points on the filter, multiply the points on the convolution template in the filter with the points on the image, and then add the products of each point to get the convolution value of that point. By repeatedly calculating, each point is calculated with the surrounding pixels to obtain a new pixel map.

[0028] Step 1.2.4 Image Enhancement: Histogram equalization is performed on the image to enhance the visibility of floating objects; the gray levels of the image are more evenly distributed, and the image contrast is changed;

[0029] Step 1.2.5 Resizing: Resize the image to a standard size of 640×640 pixels, suitable for the input of the deep learning model;

[0030] Step 2: Model Initialization and Transfer Learning

[0031] Step 2.1 Select a pre-trained model

[0032] The YOLO model, a deep learning object detection model pre-trained on a large-scale image dataset, has good generalization ability and can quickly adapt to the task of detecting floating objects in river channels.

[0033] Step 2.2 Model Fine-tuning

[0034] By using transfer learning techniques, the pre-trained model is fine-tuned using labeled data from specific river scenarios, enabling the model to identify specific types of floating objects and effectively distinguish non-floating objects in the water.

[0035] Step 2.3 Training Process

[0036] Step 2.3.1 Divide the labeled dataset into an 80% training set and a 20% validation set for training; apply cross-validation, and try different parameter combinations on each subset, and select the appropriate parameters based on the performance of the validation set.

[0037] Step 2.3.2 Data Augmentation: Data augmentation techniques are used during training to increase data diversity and improve model robustness. Data augmentation methods include geometric augmentation by changing the position, orientation, and size of the target object; color space adjustment; data augmentation algorithms include mosaic augmentation, blending augmentation, random perturbation, and color perturbation.

[0038] Step 2.3.3 Loss Function Optimization: Cross-entropy loss function and L2 regularization are adopted; the loss function is a mixture of the following:

[0039] Step 2.3.3.1 Intersection over Union Loss calculation: IoU is related to the positioning accuracy, calculating the degree of overlap between the predicted result and the actual result;

[0040]

[0041] The intersection-union ratio (IoU) is the ratio of the overlapping region to the union region. truth For the true bounding box, B pred The predicted bounding box is the area where the ground truth bounding box and the predicted bounding box intersect, and the union area is the area where the ground truth bounding box and the predicted bounding box intersect.

[0042]

[0043] L CIoU Let IoU be the loss function, where IoU is the intersection-union ratio, and ρ(B) is the crossover ratio. truth B pred α represents the Euclidean distance between the center points of the two rectangles of the ground truth bounding box and the predicted bounding box, c is the diagonal distance of the smallest closed rectangle that can simultaneously contain both the ground truth bounding box and the predicted bounding box, α is the weight, and v is the similarity function used to measure the aspect ratio. Where w is the width of the actual bounding box and the predicted bounding box, and h is the height of the bounding box;

[0044] Step 2.3.3.2 Classification Loss Calculation: The classification loss is used to process the target class prediction part. In each grid, the probability of each target class is predicted. Cross Entropy Loss is used to measure the difference between the predicted class probability distribution and the true label.

[0045]

[0046] In cross-entropy loss J, N is the number of classes, and y i For the actual class tag, For multi-class classification problems, the probability is obtained through the Softmax function, with the base of the logarithm being e.

[0047]

[0048] In the classification loss L, N is the number of classes, and y i This represents the label of sample i, with 1 for positive class and 0 for negative class, p i This represents the probability that a sample belongs to the positive class.

[0049] Step 2.3.3.3 Bounding Box Loss Calculation: The bounding box loss is used to predict the boundary, including the loss of center point coordinates and side length, which includes coordinate loss and size loss. The mean squared error (MSE) and the absolute error (MAE) are calculated using regression to determine the squared difference and absolute difference between the predicted and true values, respectively.

[0050]

[0051] in, and The y and x coordinates of the predicted bounding box center are given. and These are the y and x coordinates of the true bounding box;

[0052] Step 2.3.3.4 For classes included in the pre-trained model, or classes that have been trained before, optimization is required. Under the premise that there is not much difference between the new training samples and the previous ones, transfer training is performed. By freezing the backbone model, i.e. the early layers of the model, which already contain the general features of the classes, the previous weights are inherited, and the new training set is trained in the later layers, i.e. the subsequent connection layers.

[0053] Step 2.3.3.5 During the training process, the training process is monitored through Tensorboard to obtain the changes in the loss function and learning rate, and the average precision value mAP is output at the same time.

[0054] Step 3: Real-time floating object detection

[0055] Step 3.1 Image Input and Inference

[0056] Step 3.1.1 Input the preprocessed image into the trained deep learning model. The model uses a multi-layer convolutional neural network to extract features, classify, and locate bounding boxes in the image.

[0057] Step 3.1.2 The model has learned all possible classes in the area that needs to be predicted in a specific scenario, and then predicts the class of floating objects.

[0058] Step 3.2 Confidence screening

[0059] The confidence score comprehensively evaluates the accuracy of the target in the detection box and the accuracy of the bounding box prediction. The target accuracy is expressed by the probability of the target being in the predicted box. The accuracy of the bounding box prediction is expressed by the intersection-union ratio. The detection results are filtered by the comprehensive confidence score based on the model output, removing detection boxes with low confidence and retaining floating targets with high confidence.

[0060]

[0061] Confidence is the product of the probability P of predicting the target and the crossover ratio.

[0062] Step 3.3 Nonmaximum suppression

[0063] For potentially overlapping detection boxes, the NMS algorithm is used to select the optimal detection box. First, based on the target detection results, all detected bounding boxes are obtained. The overlap of the detection boxes is calculated using the cross-union ratio (CUP) algorithm. Starting from the bounding box with the highest CUP, all bounding boxes are traversed, and the size of the CUP is compared in turn. If the CUP is greater than the threshold, these highly overlapping boxes are suppressed, and the most representative box is retained.

[0064] Step 4: Personalized parameter adjustment

[0065] Adjust the following parameters flexibly according to actual needs:

[0066] Detection category adjustment: The original categories include classes from pre-training and classes added during later training. However, the categories can be adjusted for different application scenarios. Alternatively, if the recognition effect for a specific scenario is poor due to the limitations of the previous training set, a more targeted training set for that scenario can be added.

[0067] Confidence threshold: By adjusting the confidence level, the reliability of the detection results is controlled;

[0068] Detection frame area ratio: Set the minimum and maximum area of ​​the detection frame according to the size of the floating objects, and remove irrelevant small objects;

[0069] Intersection over Union (IoU): Adjusts the overlap between detection boxes;

[0070] Polygonal masking: Sets a mask for irrelevant areas in complex scenes, ignoring background noise and focusing on detecting specific areas;

[0071] Step 5: Analysis and Application of Test Results

[0072] Step 5.1 Results Output and Analysis

[0073] The detection results include key information such as the type, location, and confidence level of the floating objects; the detection system based on this method outputs alarm events containing this detection information in real time for floating object cleanup or further analysis; the confidence level of the output results can be limited as needed;

[0074] Step 5.2 System Linkage

[0075] The detection system can be linked with other IoT sensing devices;

[0076] Step 5.3 Performance Optimization

[0077] Based on real-time detection results, parameters are continuously adjusted to optimize the performance of the detection system and ensure optimal results in different environments.

[0078] Furthermore, the implementation tools are as follows:

[0079] Hardware implementation: Real-time video of the river channel is acquired using a high-definition camera and connected to a detection system based on the river floating object detection method of this invention;

[0080] Software implementation: Implement the object detection algorithm based on PyTorch and integrate the transfer learning module to fine-tune the model;

[0081] System integration: Integrate the detection system with the river management platform to achieve automatic feedback of detection results and equipment linkage.

[0082] Furthermore, in step 1.2.1, the waste category includes household waste, the non-waste category includes leaves and branches, and other targets include people, animals, and passing ships.

[0083] Furthermore, the data augmentation in step 2.3.2 includes geometric augmentation methods such as changing the position, orientation, and size of the target object, such as random cropping, rotation, scaling, translation, and perspective; color space adjustment, such as adjusting brightness, contrast, saturation, hue shift, and adding Gaussian noise; and data augmentation algorithms such as mosaic enhancement, blending enhancement, random perturbation, and color perturbation.

[0084] Furthermore, in step 3.2, the accuracy of the target is expressed by the probability that the target exists in the prediction box. If only a single class exists, the confidence level is high. The number of target classes in the box is inversely proportional to the target confidence level.

[0085] Furthermore, in step 3.3, the intersection-union ratio is set to 0.7 by default.

[0086] Furthermore, in step 2.3.3.1, if both w and h are in the range of 0 to 1, w 2 +h 2 It will be very small. To avoid gradient problems, 1 / (w) 2+h 2 Replace ) with 1; in step 2.3.3.2, y i The actual class label is represented by one-hot encoding, with the target class encoded as 1 and all other classes encoded as 0.

[0087] Furthermore, the detection category adjustment in step 4 is based on whether or not retrieval is needed. Leaves with only two or three leaves do not need to be retrieved, while leaves in clusters need to be retrieved. At this time, two classes are added to distinguish between non-clustered leaves and clustered leaves. In other scenarios, more different classes can be added according to the characteristics of different regions.

[0088] Furthermore, the feature is that, in step 5.1, the confidence level of the output result can be restricted as needed, for example, only the result with a higher confidence level can be output.

[0089] The beneficial effects of this invention are as follows:

[0090] Compared with existing methods for detecting floating objects, the present invention has the following advantages:

[0091] High detection accuracy: Utilizing deep learning algorithms, it can accurately identify floating objects of various shapes, reducing false detections and missed detections.

[0092] Excellent generalization ability: Through transfer learning, the model can quickly adapt to different river environments, improving detection results.

[0093] Flexible parameter adjustment: The system supports personalized settings for a variety of detection parameters, which can optimize detection performance according to the needs of different scenarios.

[0094] High real-time performance: The system can detect floating objects in the river in real time and provide timely detection feedback, which facilitates rapid response.

[0095] Reference document 1, Application (patent) number: CN202110658902.4; Title: Method for detecting floating objects in river channels based on cascaded convolutional neural networks;

[0096] Technical Comparison: While this technique solves the problem of insufficient data, the complexity of cascaded convolutional neural networks may lead to high computational resource requirements. In contrast, this invention employs transfer learning technology, enabling rapid adaptation using pre-trained models, reducing computational resource requirements, and demonstrating better generalization ability in target detection accuracy, adapting to floating object detection in different environments.

[0097] Reference document 2, Application (patent) number: CN202110760808.X; Title: An algorithm for detecting floating objects in a river under a static camera;

[0098] Technical Comparison: This technology relies on a static camera and background subtraction algorithm, making it suitable for environments with minimal scene changes, but less adaptable to dynamically changing river environments. This invention, through deep learning and object detection technology, can achieve efficient detection not only in static scenes but also adapt to dynamically changing water environments, offering wider applicability and increased detection accuracy.

[0099] Reference document 3, application (patent) number: CN202310738397.3; title: A method for floating object detection and river channel segmentation based on YOLOv5DS;

[0100] Technical Comparison: While YOLOv5DS performs well in floating object detection, its focus on solving the multi-task problem of detection and segmentation increases the model's complexity. This invention focuses on the floating object detection task, employing transfer learning and deep learning object detection algorithms. While maintaining detection speed, it improves the accuracy and real-time performance of floating object detection, and provides flexible parameter adjustment to adapt to different application scenarios.

[0101] In summary, compared to related technologies, this invention utilizes transfer learning and object detection algorithms to achieve efficient and accurate river floating object detection under relatively low computational resource conditions. It possesses strong generalization ability and adaptability, making it suitable for various aquatic environments. Through personalized parameter adjustment, this invention can flexibly address complex requirements in various application scenarios, offering strong real-time performance, high detection accuracy, and significant application advantages. Attached Figure Description

[0102] Figure 1 This is a flowchart of the river floating object detection process based on transfer learning according to the present invention. Detailed Implementation

[0103] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0104] The purpose of this invention is to provide an efficient and accurate method for detecting floating objects in waterways. This method utilizes deep learning algorithms to achieve automated floating object detection, leverages transfer learning to improve the model's generalization ability, and effectively distinguishes floating objects from other surface objects. The method supports flexible adjustment of detection parameters to meet the needs of different application scenarios.

[0105] The method for detecting floating debris in river channels proposed in this invention includes the following main steps:

[0106] Step 1: Data Acquisition and Preprocessing

[0107] Step 1.1 River Channel Image Acquisition

[0108] Use high-definition cameras to acquire real-time video or image data of the river channel, ensuring that the image clarity and coverage are sufficient to capture floating objects on the river surface.

[0109] Step 1.2 Image Preprocessing

[0110] Preprocessing the acquired images mainly includes the following steps:

[0111] Step 1.2.1 Determine the target category to be detected. The training model has limited detail for target classes; therefore, reverse training on non-floating objects can reduce false recognition. Floating objects on the water surface in different regions are divided into garbage and non-garbage categories, which can be further subdivided according to shape and material. Garbage mainly includes household waste, while non-garbage mainly includes leaves, branches, etc. Other targets include people, animals, and passing vessels.

[0112] Step 1.2.2 Use the annotation tool to annotate the collected data according to the previously determined target category and assign a category label.

[0113] Step 1.2.3 Denoising: Use filters to remove noise from the image, reduce the influence of water reflections, waves, and other factors, and minimize interference as much as possible; increase the background image as much as possible. Perform convolution on the image. For points in the image that coincide with points in the filter, multiply the points on the convolution template in the filter with the points in the image, and then add the products of each point to obtain the convolution value of that point. By repeatedly calculating each point with its surrounding pixels, a new pixel image is obtained.

[0114] Step 1.2.4 Image Enhancement: Histogram equalization is performed on the image to enhance the visibility of floating objects. The grayscale of the image is distributed more evenly, and the image contrast is changed to make the details of the target clearer.

[0115] Step 1.2.5 Resizing: Resize the image to a standard size of 640×640 pixels, suitable for the input of the deep learning model.

[0116] Step 2: Model Initialization and Transfer Learning

[0117] Step 2.1 Select a pre-trained model

[0118] The YOLO model, a deep learning object detection model pre-trained on a large-scale image dataset, has good generalization ability and can quickly adapt to the task of detecting floating objects in river channels.

[0119] Step 2.2 Model Fine-tuning

[0120] By using transfer learning techniques, the pre-trained model is fine-tuned using labeled data from specific river scenarios, enabling the model to identify specific types of floating objects and effectively distinguish non-floating objects in the water.

[0121] Step 2.3 Training Process

[0122] Step 2.3.1 divides the labeled dataset into an 80% training set and a 20% validation set. The training set includes 2000 samples across 10 classes. Cross-validation is applied to avoid insufficient training samples for certain classes. By trying different parameter combinations on each subset and selecting relatively suitable parameters based on the validation set performance, the model's generalization ability is improved, avoiding overfitting and underfitting of certain parameters for specific classes.

[0123] Step 2.3.2 Data Augmentation: Data augmentation techniques are used during training to increase data diversity and improve model robustness. Data augmentation methods include geometric enhancements that change the position, orientation, and size of the target object, such as random cropping, rotation, scaling, translation, and perspective adjustment. Color space adjustments include adjusting brightness, contrast, saturation, hue shift, and adding Gaussian noise. Data augmentation algorithms include mosaic enhancement, blending enhancement, random perturbation, and color perturbation.

[0124] Step 2.3.3 Loss Function Optimization: Cross-entropy loss and L2 regularization are used to ensure the model's detection accuracy and generalization ability. The loss function is a mixture of factors, mainly including the following:

[0125] Step 2.3.3.1 Intersection over Union Loss calculation: IOU is related to the positioning accuracy. It calculates the degree of overlap between the predicted result and the actual result. The higher the degree of overlap, the smaller the loss value. Complete overlap results in 0.

[0126]

[0127] The intersection-union ratio (IoU) is the ratio of the overlapping region to the union region. truth For the true bounding box, B pred The predicted bounding box is the area where the ground truth bounding box and the predicted bounding box intersect, and the union area is the area where the ground truth bounding box and the predicted bounding box intersect.

[0128]

[0129] L CIoU Let IoU be the loss function, where IoU is the intersection-union ratio, and ρ(B) is the crossover ratio. truth B predα represents the Euclidean distance between the center points of the two rectangles of the ground truth bounding box and the predicted bounding box, c is the diagonal distance of the smallest closed rectangle that can simultaneously contain both the ground truth bounding box and the predicted bounding box, α is the weight, and v is the similarity function used to measure the aspect ratio. Where w is the width of the actual bounding box and the predicted bounding box, and h is the height of the bounding box; if both w and h are between 0 and 1, w 2 +h 2 It will be very small. To avoid gradient problems, 1 / (w) 2 +h 2 Replace ) with 1.

[0130] Step 2.3.3.2 Classification Loss Calculation: The classification loss is used to process the target class prediction part. In each grid, the probability of each target class is predicted. Cross Entropy Loss is used to measure the difference between the predicted class probability distribution and the true label.

[0131]

[0132] In cross-entropy loss J, N is the number of classes, and y i This is the actual class label (one-hot encoded, the target class is encoded as 1, and all other classes are encoded as 0). For multi-class classification problems, the probability is obtained through the Softmax function, with the base of the logarithm being e.

[0133]

[0134] In the classification loss L, N is the number of classes, and y i This represents the label of sample i, with 1 for positive class and 0 for negative class, p i This represents the probability that a sample belongs to the positive class.

[0135] Step 2.3.3.3 Bounding Box Loss Calculation: The bounding box loss is used to predict the boundary, including the loss of center point coordinates and side length, which includes coordinate loss and size loss. The mean squared error (MSE) and the absolute error (MAE) are calculated using regression to determine the squared difference and absolute difference between the predicted and true values, respectively.

[0136]

[0137] in, and The y and x coordinates of the predicted bounding box center are given. and These are the y and x coordinates of the true bounding box.

[0138] Step 2.3.3.4 For classes included in the pre-trained model, or classes that have already been trained, optimization is required. Under the premise that there will not be much difference from the previous training samples, we perform transfer training here. By freezing the backbone model, that is, the early layers of the model, which already contain the general features of the classes, the previous weights are inherited. The new training set is trained in the later layers, that is, the subsequent connection layers. This makes the model more sensitive to specific tasks while inheriting the previous learning results, preventing the pre-training results from being destroyed, and adapting to the new dataset, saving training time and improving model performance.

[0139] Step 2.3.3.5 During the training process, the training process is monitored through Tensorboard (a visualization tool for Tensorflow), the changes in the loss function and learning rate are obtained, and the average precision value mAP is output.

[0140] Step 3: Real-time floating object detection

[0141] Step 3.1 Image Input and Inference

[0142] Step 3.1.1 Input the preprocessed image into the trained deep learning model. The model uses a multi-layer convolutional neural network to extract features, classify, and locate bounding boxes in the image.

[0143] Step 3.1.2 The model has learned all possible classes in the area that needs to be predicted in a specific scenario, and then predicts the class of floating objects.

[0144] Step 3.2 Confidence screening

[0145] The confidence score comprehensively evaluates the accuracy of both the detected bounding box and the predicted bounding box. Target accuracy is expressed as the probability of a target being present in the predicted bounding box; a higher confidence score indicates the presence of only one class, and the confidence score is inversely proportional to the number of target classes within the bounding box. Bounding box prediction accuracy is expressed using the intersection-over-union ratio (IoU). The detection results are then filtered based on the combined confidence score and the model output, removing low-confidence bounding boxes and retaining high-confidence floating targets.

[0146]

[0147] Confidence is the product of the probability P of predicting the target and the crossover ratio.

[0148] Step 3.3 Non-maximum suppression (NMS)

[0149] For potentially overlapping detection boxes, the NMS algorithm is used to select the optimal detection box to avoid repeatedly identifying the same target. First, based on the target detection results, all detected bounding boxes are obtained. The overlap of the detection boxes is calculated using the intersection-union ratio (IUR) algorithm. Starting from the bounding box with the highest IUR, all bounding boxes are traversed, and the IUR values ​​are compared one by one. If the IUR is greater than the threshold (default is 0.7), these highly overlapping boxes are suppressed, and the most representative box is retained.

[0150] Step 4: Personalized parameter adjustment

[0151] Depending on project requirements, users can flexibly adjust the following parameters:

[0152] Step 4.1 Detection Category Adjustment

[0153] The original categories include classes from pre-training and classes added during later training. However, the categories can be adjusted for different application scenarios. For example, depending on whether retrieval is required, leaves with only a few leaves do not need to be retrieved, while leaves in large patches need to be retrieved. In this case, two classes are added to distinguish between leaves in small patches and leaves in large patches. In other scenarios, more different classes can be added based on the characteristics of different regions. Alternatively, if the recognition effect of certain classes is poor in specific scenarios, such as different regions or different cameras, due to the limitations of the previous training set, a more targeted training set for that scenario can be added to strengthen the training and achieve a relatively better recognition effect for that scenario.

[0154] Step 4.2 Confidence Threshold

[0155] The reliability of the test results can be controlled by adjusting the confidence level.

[0156] Step 4.3 Detection frame area ratio

[0157] Set the minimum and maximum area of ​​the detection frame based on the size of the floating object, and remove irrelevant small objects.

[0158] Step 4.4 Intersection over Union (IOU)

[0159] Adjust the overlap between detection boxes to reduce the problem of multiple detections.

[0160] Step 4.5 Polygon Masking: Set a mask for irrelevant areas in a complex scene, ignore background noise, and focus on detecting specific areas.

[0161] Step 5: Analysis and Application of Test Results

[0162] Step 5.1 Results Output and Analysis

[0163] The detection results include key information such as the type, location, and confidence level of the floating debris. The system outputs alarm events containing this detection information in real time, allowing river managers to clean up the debris or conduct further analysis. The confidence level of the output results can be restricted as needed, for example, only outputting results with high confidence levels.

[0164] Step 5.2 System Linkage

[0165] The detection system can be linked with other IoT sensing devices to improve the efficiency of river management.

[0166] Step 5.3 Performance Optimization

[0167] Based on real-time detection results, users can continuously adjust parameters to optimize the performance of the detection system and ensure optimal results in different environments.

[0168] The tools used to achieve this are as follows:

[0169] Hardware implementation: Real-time video of the river channel is acquired using a high-definition camera and connected to a detection system based on the river floating object detection method of this invention.

[0170] Software implementation: The object detection algorithm is implemented based on PyTorch, and the transfer learning module is integrated to fine-tune the model.

[0171] System integration: Integrate the detection system with the river management platform to achieve automatic feedback of detection results and equipment linkage.

[0172] The above-described embodiments are merely one implementation of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention should be determined by the appended claims.

Claims

1. A method for detecting floating objects in river channels based on transfer learning and object detection algorithms, characterized in that, Includes the following steps: Step 1: Data Acquisition and Preprocessing Acquire real-time video or image data containing floating objects on the river surface, preprocess the acquired images, determine and label the target categories to be detected, and then perform denoising, image enhancement and resizing. Step 2: Model Initialization and Transfer Learning We selected a deep learning object detection model pre-trained on a large-scale image dataset. Through transfer learning, we fine-tuned the pre-trained model using labeled data from a specific river scene. We divided the labeled dataset into a training set and a validation set, and applied cross-validation to select relatively suitable parameters for training. Data augmentation techniques are used during training, and the loss function is optimized using cross-entropy loss function and L2 regularization. For classes included in the pre-trained model, or classes that have been trained before, transfer training is performed to fine-tune the model. The training process is monitored through Tensorboard to obtain changes in the loss function and learning rate, and the average precision value mAP is output. Step 3: Real-time floating object detection: The preprocessed image is input into a trained deep learning model. The model uses a multi-layer convolutional neural network to extract features, classify, and locate bounding boxes, predicting floating objects. The confidence scores are used to comprehensively evaluate the accuracy of the target within the bounding box and the accuracy of the bounding box prediction. The target accuracy is expressed by the probability of the target being in the predicted box, while the bounding box prediction accuracy is expressed by the intersection-over-union ratio (IoU). Low-confidence bounding boxes are removed, while high-confidence floating object targets are retained. For potentially overlapping bounding boxes, the NMS algorithm is used to select the optimal bounding box. Step 4: Personalized parameter adjustment: Adjust the following parameters flexibly according to project requirements: detection category, confidence threshold, detection box area ratio, intersection-union ratio, and polygon masking; Step 5: Analysis and application of test results: The detection results include key information about floating objects; the detection system outputs alarm events containing information about floating objects in real time, facilitating the removal or further analysis of floating objects; the detection system can be linked with other IoT sensing devices; based on the real-time detection results, users can continuously adjust parameters to optimize the performance of the detection system and ensure optimal results in different environments.

2. The method for detecting floating objects in river channels based on transfer learning and target detection algorithms according to claim 1, characterized in that, Specifically, the following steps are included: Step 1: Data Acquisition and Preprocessing Step 1.1 River Channel Image Acquisition Use high-definition cameras to acquire real-time video or image data of the river channel, ensuring that the image clarity and coverage are sufficient to capture floating objects on the river surface; Step 1.2 Image Preprocessing The preprocessing of the acquired images includes the following steps: Step 1.2.1 Determine the target category to be detected; the training model has limited training details for target categories, so it can be trained in reverse on non-floating objects to reduce false recognition; floating objects on the water surface in different regions are divided into garbage and non-garbage categories, which can be further subdivided according to shape and material. Step 1.2.2 Use the annotation tool to annotate the collected data according to the previously determined target category and assign a category label; Step 1.2.3 Denoising: Use filters to remove noise from the image; increase the background image as much as possible; perform convolution on the image, and for points on the image that coincide with the points on the filter, multiply the points on the convolution template in the filter with the points on the image, and then add the products of each point to get the convolution value of that point. By repeatedly calculating, each point is calculated with the surrounding pixels to obtain a new pixel map. Step 1.2.4 Image Enhancement: Histogram equalization is performed on the image to enhance the visibility of floating objects; the gray levels of the image are more evenly distributed, and the image contrast is changed; Step 1.2.5 Resizing: Resize the image to a standard size of 640×640 pixels, suitable for the input of the deep learning model; Step 2: Model Initialization and Transfer Learning Step 2.1 Select a pre-trained model The YOLO model, a deep learning object detection model pre-trained on a large-scale image dataset, has good generalization ability and can quickly adapt to the task of detecting floating objects in river channels. Step 2.2 Model Fine-tuning By using transfer learning techniques, the pre-trained model is fine-tuned using labeled data from specific river scenarios, enabling the model to identify specific types of floating objects and effectively distinguish non-floating objects in the water. Step 2.3 Training Process Step 2.3.1 Divide the labeled dataset into an 80% training set and a 20% validation set for training; and Cross-validation is applied, and different parameter combinations are tried on each subset, with the most suitable parameters selected based on the performance of the validation set. Step 2.3.2 Data Augmentation: Data augmentation techniques are used during training to increase data diversity and improve the robustness of the model; Data augmentation methods include geometric enhancements that alter the position, orientation, and size of the target object; color space adjustments; and data augmentation algorithms such as mosaic enhancement, blending enhancement, random perturbation, and color perturbation. Step 2.3.3 Loss Function Optimization: Cross-entropy loss function and L2 regularization are adopted; the loss function is a mixture of the following: Step 2.3.3.1 Intersection over Union Loss calculation: IoU is related to the positioning accuracy, calculating the degree of overlap between the predicted result and the actual result; The intersection-union ratio (IoU) is the ratio of the overlapping region to the union region. truth For the true bounding box, B pred The predicted bounding box is the area where the ground truth bounding box and the predicted bounding box intersect, and the union area is the area where the ground truth bounding box and the predicted bounding box intersect. L CIoU Let IoU be the loss function, where IoU is the intersection-union ratio, and ρ(B) is the crossover ratio. truth B pred α represents the Euclidean distance between the center points of the two rectangles of the ground truth bounding box and the predicted bounding box, c is the diagonal distance of the smallest closed rectangle that can simultaneously contain both the ground truth bounding box and the predicted bounding box, α is the weight, and v is the similarity function used to measure the aspect ratio. Where w is the width of the actual bounding box and the predicted bounding box, and h is the height of the bounding box; Step 2.3.3.2 Classification Loss Calculation: The classification loss is used to process the target class prediction part. In each grid, the probability of each target class is predicted. Cross Entropy Loss is used to measure the difference between the predicted class probability distribution and the true label. In cross-entropy loss J, N is the number of classes, and y i For the actual class tag, For multi-class classification problems, the probability is obtained through the Softmax function, with the base of the logarithm being e. In the classification loss L, N is the number of classes, and y i This represents the label of sample i, with 1 for positive class and 0 for negative class, p i This represents the probability that a sample belongs to the positive class. Step 2.3.3.3 Bounding Box Loss Calculation: The bounding box loss is used to predict the boundary, including the loss of center point coordinates and side length, which includes coordinate loss and size loss. The mean squared error (MSE) and the absolute error (MAE) are calculated using regression to determine the squared difference and absolute difference between the predicted and true values, respectively. in, and The y and x coordinates of the predicted bounding box center are given. and These are the y and x coordinates of the true bounding box; Step 2.3.3.4 For classes included in the pre-trained model, or classes that have been trained before, optimization is required. Under the premise that there is not much difference between the new training samples and the previous ones, transfer training is performed. By freezing the backbone model, i.e. the early layers of the model, which already contain the general features of the classes, the previous weights are inherited, and the new training set is trained in the later layers, i.e. the subsequent connection layers. Step 2.3.3.5 During the training process, the training process is monitored through Tensorboard to obtain the changes in the loss function and learning rate, and the average precision value mAP is output at the same time. Step 3: Real-time floating object detection Step 3.1 Image Input and Inference Step 3.1.1 Input the preprocessed image into the trained deep learning model. The model uses a multi-layer convolutional neural network to extract features, classify, and locate bounding boxes in the image. Step 3.1.2 The model has learned all possible classes in the area that needs to be predicted in a specific scenario, and then predicts the class of floating objects. Step 3.2 Confidence screening The confidence score comprehensively evaluates the accuracy of the target in the detection box and the accuracy of the bounding box prediction. The target accuracy is expressed by the probability of the target being in the predicted box. The accuracy of the bounding box prediction is expressed by the intersection-union ratio. The detection results are filtered by the comprehensive confidence score based on the model output, removing detection boxes with low confidence and retaining floating targets with high confidence. Confidence is the product of the probability P of predicting the target and the crossover ratio. Step 3.3 Nonmaximum suppression For potentially overlapping detection boxes, the NMS algorithm is used to select the optimal detection box. First, based on the target detection results, all detected bounding boxes are obtained. The overlap of the detection boxes is calculated using the cross-union ratio (CUP) algorithm. Starting from the bounding box with the highest CUP, all bounding boxes are traversed, and the size of the CUP is compared in turn. If the CUP is greater than the threshold, these highly overlapping boxes are suppressed, and the most representative box is retained. Step 4: Personalized parameter adjustment Adjust the following parameters flexibly according to actual needs: Detection category adjustment: The original categories include classes from pre-training and classes added during later training. However, the categories can be adjusted for different application scenarios. Alternatively, if the recognition effect for a specific scenario is poor due to the limitations of the previous training set, a more targeted training set for that scenario can be added. Confidence threshold: By adjusting the confidence level, the reliability of the detection results is controlled; Detection frame area ratio: Set the minimum and maximum area of ​​the detection frame according to the size of the floating objects, and remove irrelevant small objects; Intersection over Union (IoU): Adjusts the overlap between detection boxes; Polygonal masking: Sets a mask for irrelevant areas in complex scenes, ignoring background noise and focusing on detecting specific areas; Step 5: Analysis and Application of Test Results Step 5.1 Results Output and Analysis The detection results include key information such as the type, location, and confidence level of the floating objects; the detection system based on this method outputs alarm events containing this detection information in real time for floating object cleanup or further analysis; the confidence level of the output results can be limited as needed; Step 5.2 System Linkage The detection system can be linked with other IoT sensing devices; Step 5.3 Performance Optimization Based on real-time detection results, parameters are continuously adjusted to optimize the performance of the detection system and ensure optimal results in different environments.

3. The method for detecting floating objects in river channels based on transfer learning and target detection algorithms according to claim 2, characterized in that, The tools used to achieve this are as follows: Hardware implementation: Real-time video of the river channel is captured by a high-definition camera and connected to a detection system based on the method for detecting floating objects in the river channel; Software implementation: Implement the object detection algorithm based on PyTorch and integrate the transfer learning module to fine-tune the model; System integration: Integrate the detection system with the river management platform to achieve automatic feedback of detection results and equipment linkage.

4. The method for detecting floating objects in river channels based on transfer learning and target detection algorithms according to claim 2, characterized in that, In step 1.2.1, the waste category includes household waste, the non-waste category includes leaves and branches, and other targets include people, animals and passing ships.

5. The method for detecting floating objects in river channels based on transfer learning and target detection algorithms according to claim 2, characterized in that, The data augmentation in step 2.3.2 includes geometric augmentation that alters the position, orientation, and size of the target object, such as random cropping, rotation, scaling, translation, and perspective; and color space adjustment, including adjusting brightness, contrast, saturation, hue shift, and adding Gaussian noise. Data augmentation algorithms include mosaic augmentation, hybrid augmentation, random perturbation, and color perturbation.

6. The method for detecting floating objects in river channels based on transfer learning and target detection algorithms according to claim 2, characterized in that, In step 3.2, the accuracy of the target is expressed by the probability that the target exists in the prediction box. If only a single class exists, the confidence level is high. The number of target classes in the box is inversely proportional to the target confidence level.

7. The method for detecting floating objects in river channels based on transfer learning and target detection algorithms according to claim 2, characterized in that, Step 3.3, the intersection-union ratio is set to 0.7 by default.

8. The method for detecting floating objects in river channels based on transfer learning and target detection algorithms according to claim 2, characterized in that, In step 2.3.3.1, if both w and h are in the range of 0 to 1, then w 2 +h 2 It will be very small. To avoid gradient problems, 1 / (w) 2 +h 2 Replace ) with 1; in step 2.3.3.2, y i The actual class label is represented by one-hot encoding, with the target class encoded as 1 and all other classes encoded as 0.

9. The method for detecting floating objects in river channels based on transfer learning and target detection algorithms according to claim 2, characterized in that, In step 4, the detection category is adjusted based on whether or not retrieval is needed. Leaves with only two or three leaves do not need to be retrieved, while leaves in large patches do need to be retrieved. Two new categories are added to distinguish between leaves that are not in large patches and leaves that are in large patches. In other scenarios, more different categories can be added based on the characteristics of different regions.

10. The method for detecting floating objects in river channels based on transfer learning and target detection algorithms according to claim 2, characterized in that, In step 5.1, the confidence level of the output results can be restricted as needed, and only results with higher confidence levels can be output.

Images