River floating object detection method based on transfer learning and target detection algorithm

Abstract

The invention relates to a river channel floating object detection method based on transfer learning and a target detection algorithm, and the method comprises the following steps: obtaining real-time video or image data containing floating objects on the surface of a river channel, and carrying out the preprocessing of the real-time video or image data; selecting a deep learning target detection model pre-trained based on a large-scale image data set, and training by using a data enhancement technology through a transfer learning technology; the preprocessed image is input into a trained deep learning model, the model performs feature extraction, classification and detection frame positioning on the image through a multi-layer convolutional neural network and predicts floating objects, and the confidence degree comprehensively evaluates the accuracy of a target in a detection frame and the accuracy of frame prediction; carrying out parameter personalized adjustment; and analyzing and applying a detection result. The problems that the detection of the river channel floaters is difficult to solve that the floaters are diverse in variety, easy to be confused with other objects and complex in environment are solved, and the accuracy and the real-time performance of the detection of the floaters are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of water environment monitoring and image processing technology, and in particular to a method for detecting floating objects in a river channel based on a deep learning algorithm and transfer learning technology. The method is suitable for automatically detecting floating objects on the water surface, such as garbage, leaves, etc., and distinguishing between false identification and other objects (such as people, ships, birds, etc.). Background Art

[0002] Floating objects in waters not only affect the urban landscape, but may also cause water pollution or hinder the normal drainage of rivers. Therefore, accurate and automated detection of floating objects in rivers is of great significance for environmental protection and water resource management. Traditional floating object detection methods mostly rely on manual inspections or automated detection based on simple image processing, which are difficult to cope with complex water environments and various interferences, such as changes in light and wave reflections.

[0003] With the development of deep learning technology, object detection algorithms based on convolutional neural networks (CNNs) have made significant progress in the fields of image recognition and object detection. However, the detection of floating objects in rivers still faces the following challenges:

[0004] 1. Various types of floating objects: floating objects have great variations in shape, size, color, etc.;

[0005] 2. Easily confused with other objects: such as ships, birds, etc.;

[0006] 3. Complex environment: reflections on the water surface, changes in light, etc. will affect detection accuracy. Summary of the invention

[0007] Aiming at the problem that it is difficult to detect floating objects in rivers, such as the variety of floating objects, easy confusion with other objects, and complex environment, a method for detecting floating objects in rivers based on transfer learning and target detection algorithm is proposed. By fine-tuning the pre-trained model and combining it with actual data in the river scene, the accuracy and real-time performance of floating object detection are significantly improved.

[0008] The technical solution of the present invention is:

[0009] A method for detecting floating objects in a river channel based on transfer learning and target detection algorithm comprises the following steps:

[0010] Step 1: Data collection and preprocessing:

[0011] Acquire real-time video or image data containing floating objects on the river surface, pre-process the collected images, determine the target categories to be detected and mark them, and then perform denoising, image enhancement and size adjustment;

[0012] Step 2: Model initialization and transfer learning:

[0013] Select a deep learning target detection model pre-trained based on a large-scale image dataset, and fine-tune the pre-trained model using the labeled data in a specific river scene through transfer learning technology. The labeled dataset is divided into a training set and a validation set, and cross-validation is used to select relatively suitable parameters for training. During the training process, data enhancement technology is used, and the cross-entropy loss function and L2 regularization technology are used to optimize the loss function. For the classes included in the pre-trained model or the classes that have been trained before, transfer training is performed to tune them. The training process is monitored through Tensorboard to obtain changes in the loss function and learning rate, and the mean average precision value mAP is output at the same time.

[0014] Step 3: Real-time floating object detection:

[0015] The preprocessed image is input into the trained deep learning model. The model uses a multi-layer convolutional neural network to extract features, classify and locate the detection frame of the image, and predict the floating object class. The confidence level comprehensively evaluates the accuracy of the target in the detection frame and the accuracy of the border prediction. The target accuracy is expressed by the probability of the existence of the target in the prediction frame, and the accuracy of the border prediction is expressed by the intersection-over-union ratio. The detection frames with low confidence are removed, and the floating object targets with high confidence are retained. For the detection frames that may overlap, the NMS algorithm is used to filter out the optimal detection frame.

[0016] Step 4: Personalized parameter adjustment: According to project requirements, flexibly adjust the following parameters: detection category, confidence threshold, detection box area ratio, intersection ratio, polygon mask;

[0017] Step 5: Analysis and application of test results:

[0018] The detection results include key information about floating objects; the detection system outputs alarm events containing floating object information in real time, which is convenient for floating object cleaning or further analysis; the detection system is linked with other IoT sensing devices; based on real-time detection results, users can continuously adjust parameters and optimize the performance of the detection system to ensure the best results in different environments.

[0019] Further, the method specifically includes the following steps:

[0020] Step 1: Data collection and preprocessing

[0021] Step 1.1 River channel image acquisition

[0022] Use high-definition cameras to obtain real-time video or image data of the river, ensuring that the image clarity and coverage are sufficient to capture floating objects on the river surface;

[0023] Step 1.2 Image preprocessing

[0024] The collected images are preprocessed, including the following steps:

[0025] Step 1.2.1 Determine the target category to be detected; the training model has limited training details on the target category, and reverse training can be performed on non-floating objects to reduce the effect of misidentification; floating objects on the water surface in different regions are divided into garbage and non-garbage categories, and can be subdivided according to shape and material;

[0026] Step 1.2.2 Use the annotation tool to annotate the collected data according to the previously determined target category and assign a category label;

[0027] Step 1.2.3 Denoising: Use a filter to remove noise from the image; try to increase the background image; perform convolution on the image, and for points on the image that coincide with points on the filter, multiply the points on the convolution template in the filter with the points on the image, and then add the products of each point to get the convolution value of the point. Through repeated calculations, each point is calculated with the surrounding pixels to get a new pixel map;

[0028] Step 1.2.4 Image enhancement: Perform histogram equalization on the image to enhance the visibility of floating objects; distribute the grayscale of the image more evenly and change the image contrast;

[0029] Step 1.2.5 Resizing: Resize the image to a standard size of 640×640 pixels suitable for deep learning model input;

[0030] Step 2: Model initialization and transfer learning

[0031] Step 2.1 Select a pre-trained model

[0032] We selected a deep learning target detection model pre-trained on a large-scale image dataset. The YOLO model has good generalization ability and can quickly adapt to the task of detecting floating objects in rivers.

[0033] Step 2.2 Model fine-tuning

[0034] Through transfer learning technology, the pre-trained model is fine-tuned using labeled data from specific river scenarios, enabling the model to identify specific types of floating objects and effectively distinguish non-floating objects in the water.

[0035] Step 2.3 Training process

[0036] Step 2.3.1 Divide the labeled data set into 80% training set and 20% validation set for training; apply cross-validation, and try different parameter combinations on each subset, and select relatively suitable parameters based on the performance of the validation set;

[0037] Step 2.3.2 Data enhancement: Use data enhancement techniques during training to increase data diversity and improve the robustness of the model; data enhancement methods include geometric enhancement that changes the position, direction, and size of the target object; color space adjustment; data enhancement algorithms include mosaic enhancement, mixed enhancement, random perturbation, and color perturbation;

[0038] Step 2.3.3 Loss function optimization: Use cross entropy loss function and L2 regularization technology; the loss function is a mixed result, including the following:

[0039] Step 2.3.3.1 Intersection over Union Loss calculation, IoU is in the positioning accuracy part, calculate the overlap between the predicted result and the actual result;

[0040]

[0041] The IoU intersection-over-union ratio is the ratio of the overlapping area to the union area. truth is the real frame, B pred is the predicted box, the overlapping area is the intersection of the real box and the predicted box, and the union area is the union of the real box and the predicted box;

[0042]

[0043] L CIoU is the IoU loss function, IoU is the intersection over union ratio, ρ(B truth ,B pred ) represents the Euclidean distance between the center points of the two rectangular boxes of the real box and the predicted box, c is the diagonal distance of the minimum closed rectangular area that can contain both the real box and the predicted box, α is the weight, and v is the similarity function used to measure the aspect ratio. Where w is the width of the actual box and the predicted box, and h is the height of the box;

[0044] Step 2.3.3.2 Classification Loss calculation, the classification loss is to deal with the target category prediction part. In each grid, the probability of each target category will be predicted, and the Cross Entropy Loss is used to measure the difference between the predicted category probability distribution and the true label;

[0045]

[0046] In the cross entropy loss J, N is the number of categories, y i is the actual class label, is the predicted probability. In multi-classification problems, the probability is obtained by the Softmax function, and the base of log is e;

[0047]

[0048] In the classification loss L, N is the number of categories, and y i represents the label of sample i, the positive class is 1, the negative class is 0, and p i Indicates the probability that the sample is a positive class;

[0049] Step 2.3.3.3 Bounding Box Loss calculation, the bounding box loss is used to predict the loss of the boundary, including the loss of the center point coordinates and the side length, including coordinate loss and size loss. The mean square error MSE and absolute error MAE use regression to calculate the square difference and absolute difference between the predicted value and the true value respectively:

[0050]

[0051] in, and are the y and x coordinates of the center of the prediction box, and are the y and x coordinates of the real box;

[0052] Step 2.3.3.4 For the classes included in the pre-trained model or the classes that have been trained before, it is necessary to tune them. Under the premise that there is not much difference between the added training samples and the previous ones, transfer training is performed. By freezing the backbone model, that is, the early layers of the model, the early layers already contain the common features of the class, so the previous weights are inherited, and the new training set is trained in the later layers, that is, the subsequent connection layers;

[0053] Step 2.3.3.5 During the training process, monitor the training process through Tensorboard, obtain the changes in loss function and learning rate, and output the mean average precision value mAP;

[0054] Step 3: Real-time floating object detection

[0055] Step 3.1 Image input and reasoning

[0056] Step 3.1.1 Input the preprocessed image into the trained deep learning model, and the model uses a multi-layer convolutional neural network to extract features, classify, and locate detection frames on the image;

[0057] Step 3.1.2 The model learns all possible classes that may appear in the area to be predicted in a specific scene, and then predicts the floating object class;

[0058] Step 3.2 Confidence screening

[0059] The confidence level comprehensively evaluates the accuracy of the target in the detection frame and the accuracy of the bounding box prediction. The target accuracy is expressed by the probability of the target existing in the prediction frame. The accuracy of the bounding box prediction is expressed by the intersection-over-union ratio. The detection target and the comprehensive confidence level based on the model output are used to filter the detection results, remove the detection frames with low confidence, and retain the floating objects with high confidence.

[0060]

[0061] Confidence is the product of the probability P of the prediction being the target and the intersection and union ratio;

[0062] Step 3.3 Non-maximum suppression

[0063] For the detection frames that may overlap, the NMS algorithm is used to filter out the optimal detection frame. First, all detected bounding boxes are obtained according to the target detection results. The overlap of the detection boxes is calculated by the intersection-and-union algorithm. All bounding boxes are traversed starting from the bounding box with the highest intersection-and-union ratio. The sizes of the intersection-and-union ratios are compared in turn. If the intersection-and-union ratio is greater than the threshold, the boxes with high overlap are suppressed, and the most representative box is retained.

[0064] Step 4: Personalize parameters

[0065] According to actual needs, flexibly adjust the following parameters:

[0066] Detection category adjustment: The original categories include the classes in pre-training and the classes added in post-training. However, the categories can be adjusted for different application scenarios. If the recognition effect of a specific scene is poor due to the limitation of the previous training set, a more targeted training set in the scene can be added.

[0067] Confidence threshold: Control the reliability of the detection results by adjusting the confidence level;

[0068] Detection frame area ratio: Set the minimum and maximum areas of the detection frame according to the size of the floating objects to remove irrelevant small objects;

[0069] Intersection-over-Union: adjusts the overlap between detection frames;

[0070] Polygonal mask: Set a mask for irrelevant areas in complex scenes, ignore background noise, and focus on detecting specific areas;

[0071] Step 5: Analysis and application of test results

[0072] Step 5.1 Result output and analysis

[0073] The detection results include key information such as the type, location and confidence level of floating objects. The detection system implemented based on this method outputs alarm events containing these detection information in real time to clean up floating objects or conduct further analysis. The output results can be limited in terms of confidence level as needed.

[0074] Step 5.2 System linkage

[0075] The detection system can be linked with other IoT sensing devices;

[0076] Step 5.3 Performance Optimization

[0077] Based on real-time detection results, parameters are continuously adjusted to optimize the performance of the detection system to ensure the best results in different environments.

[0078] Furthermore, the implementation tools are as follows:

[0079] Hardware implementation: Real-time river video is collected by high-definition camera equipment and connected to the detection system based on the river floating object detection method of the present invention;

[0080] Software implementation: Implement the target detection algorithm based on PyTorch, integrate the transfer learning module, and fine-tune the model;

[0081] System integration: Integrate the detection system with the river management platform to achieve automatic feedback of detection results and equipment linkage.

[0082] Furthermore, in step 1.2.1, the garbage category includes domestic garbage, the non-garbage category includes leaves and branches, and other targets include people, animals and passing ships.

[0083] Furthermore, the data enhancement in step 2.3.2 includes geometric enhancements that change the position, direction, and size of the target object, such as random cropping, rotation, scaling, translation, and perspective; color space adjustment, such as adjusting brightness, contrast, saturation, hue shift, and adding Gaussian noise; data enhancement algorithms include mosaic enhancement, mixed enhancement, random perturbation, and color perturbation.

[0084] Furthermore, in step 3.2, the probability of the existence of a target in the prediction box is used to express the target accuracy. If only a single class exists, the confidence is higher, and the number of target classes in the box is inversely proportional to the target confidence.

[0085] Furthermore, in step 3.3, the intersection-over-union ratio defaults to 0.7.

[0086] Furthermore, in step 2.3.3.1, if w and h are both between 0 and 1, w 2 +h 2 will be very small. To avoid the gradient problem, 1 / (w 2+h 2 ) is replaced by 1; in step 2.3.3.2, y i is the actual class label, using one-hot encoding, the target class is encoded as 1, and all other classes are encoded as 0.

[0087] Furthermore, the detection category in step 4 is adjusted to determine whether leaves need to be salvaged. If there are only three or two leaves, there is no need to salvage them, but leaves in pieces need to be salvaged. At this time, two categories are added to distinguish between non-pieces of leaves and pieces of leaves. For other scenarios, more different categories can be added according to the characteristics of different regions.

[0088] Furthermore, it is characterized in that, in step 5.1, the confidence level of the output results can be limited as needed, for example, only the results with higher confidence levels are output.

[0089] The beneficial effects of the present invention are:

[0090] Compared with the existing floating object detection methods, the present invention has the following beneficial effects:

[0091] High detection accuracy: Using deep learning algorithms, it can accurately identify floating objects of various forms and reduce false detections and missed detections.

[0092] Good generalization ability: Through transfer learning, the model can quickly adapt to different river environments and improve detection results.

[0093] Flexible parameter adjustment: The system supports personalized settings of multiple detection parameters and can optimize detection performance according to the needs of different scenarios.

[0094] Strong real-time performance: The system can detect floating objects in the river in real time and provide timely detection feedback to facilitate rapid response.

[0095] Comparative Documents 1. Application (Patent) Number: CN202110658902.4; Name: Detection Method of Floating Objects in River Channel Based on Cascaded Convolutional Neural Network;

[0096] Comparison of technical points: Although this technology solves the problem of insufficient data volume, the complexity of the cascaded convolutional neural network may lead to higher computing resource requirements. The present invention adopts transfer learning technology, which can use pre-trained models for rapid adaptation, reducing the requirements for computing resources, and has better generalization ability in target detection accuracy, adapting to floating object detection in different environments.

[0097] Comparative Document 2, Application (Patent) Number: CN202110760808.X; Name: A river floating object detection algorithm under a static camera;

[0098] Comparison of technical points: This technology relies on static cameras and background subtraction algorithms, which is more suitable for environments with less scene changes, but is not very adaptable to river environments with large dynamic changes. Through deep learning and target detection technology, the present invention can not only achieve efficient detection in static scenes, but also adapt to dynamically changing water surface environments, with wider applicability and detection accuracy.

[0099] Comparative Document 3, Application (Patent) Number: CN202310738397.3; Name: A floating object detection and river channel segmentation method based on YOLOv5DS;

[0100] Comparison of technical points: Although YOLOv5DS performs well in floating object detection, it focuses on solving the multi-task problem of detection and segmentation, which increases the complexity of the model. This invention focuses on the floating object detection task, adopts transfer learning and deep learning target detection algorithms, improves the accuracy and real-time performance of floating object detection while maintaining the detection speed, and provides flexible parameter adjustment to meet the needs of different application scenarios.

[0101] In summary, compared with related technologies, the present invention uses transfer learning and target detection algorithms to achieve efficient and accurate river floating object detection under low computing resource conditions, has strong generalization ability and adaptability, and is suitable for different water environments. Through the personalized parameter adjustment function, the present invention can flexibly respond to complex requirements in various application scenarios, has strong real-time performance, high detection accuracy, and has significant application advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Figure 1 This is a flow chart of river floating object detection based on transfer learning in the present invention. DETAILED DESCRIPTION

[0103] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0104] The purpose of the present invention is to provide an efficient and accurate method for detecting floating objects in rivers, which can realize automated detection of floating objects in rivers through deep learning algorithms, improve the generalization ability of the model through transfer learning, and effectively distinguish floating objects from other surface objects. The method supports flexible adjustment of detection parameters to meet the needs of different application scenarios.

[0105] The method for detecting floating objects in a river proposed by the present invention comprises the following main steps:

[0106] Step 1: Data collection and preprocessing

[0107] Step 1.1 River channel image acquisition

[0108] Use high-definition cameras to obtain real-time video or image data of the river, ensuring that the image clarity and coverage are sufficient to capture floating objects on the river surface.

[0109] Step 1.2 Image preprocessing

[0110] The preprocessing of the collected images mainly includes the following steps:

[0111] Step 1.2.1 Determine the target category to be detected. The training model has limited details on the target category. Reverse training can be performed on non-floating objects to reduce misidentification. Floating objects on the water surface in different regions are divided into garbage and non-garbage categories, which can be subdivided according to shape and material. The garbage category mainly includes domestic garbage, and the non-garbage category mainly includes leaves, branches, etc. Other targets include people, animals, and passing ships.

[0112] Step 1.2.2 Use the annotation tool to annotate the collected data according to the previously determined target category and assign a category label.

[0113] Step 1.2.3 Denoising: Use filters to remove noise in the image, reduce the impact of water reflections, waves and other factors, try to suppress and reduce interference; try to increase the background image. Convolve the image, and for points on the image that coincide with the points of the filter, multiply the points on the convolution template in the filter with the points of the image, and then add the products of each point to get the convolution value of the point. Through repeated calculations, each point is calculated with the surrounding pixels to get a new pixel map.

[0114] Step 1.2.4 Image enhancement: Perform histogram equalization on the image to enhance the visibility of floating objects, distribute the grayscale of the image more evenly, and change the image contrast, so that the details of the target are clearer.

[0115] Step 1.2.5 Resizing: Resize the image to a standard size of 640×640 pixels suitable for deep learning model input.

[0116] Step 2: Model initialization and transfer learning

[0117] Step 2.1 Select a pre-trained model

[0118] A deep learning target detection model pre-trained based on a large-scale image dataset was selected. The YOLO model has good generalization ability and can quickly adapt to the task of river floating object detection.

[0119] Step 2.2 Model fine-tuning

[0120] Through transfer learning technology, the pre-trained model is fine-tuned using labeled data in specific river scenarios, enabling the model to identify specific types of floating objects and effectively distinguish non-floating objects in the water.

[0121] Step 2.3 Training process

[0122] Step 2.3.1: Divide the labeled data set into 80% training set and 20% validation set. The training includes 2000 samples for 10 categories. Cross-validation is applied to avoid insufficient samples in some classes during training. Different parameter combinations are tried on each subset, and relatively suitable parameters are selected according to the performance of the validation set to avoid certain parameters for specific classes and avoid overfitting and underfitting, thereby improving the generalization ability of the model.

[0123] Step 2.3.2 Data enhancement: Use data enhancement techniques during training to increase data diversity and improve the robustness of the model. Data enhancement methods include geometric enhancements that change the position, direction, and size of the target object, such as random cropping, rotation, scaling, translation, perspective, etc. Color space adjustment, such as adjusting brightness, contrast, saturation, hue shift, adding Gaussian noise, etc. Data enhancement algorithms include mosaic enhancement, mixed enhancement, random perturbation, and color perturbation.

[0124] Step 2.3.3 Loss function optimization: Use cross entropy loss function and L2 regularization techniques to ensure the detection accuracy and generalization ability of the model. The loss function is a mixed result, mainly including the following:

[0125] Step 2.3.3.1 Intersection over Union Loss calculation, IOU is about the positioning accuracy part, calculate the overlap between the predicted result and the actual result, the higher the overlap, the smaller the loss value, and complete overlap is 0;

[0126]

[0127] The IoU intersection-over-union ratio is the ratio of the overlapping area to the union area. truth is the real frame, B pred is the predicted box, the overlapping area is the intersection of the real box and the predicted box, and the union area is the union of the real box and the predicted box.

[0128]

[0129] L CIoU is the IoU loss function, IoU is the intersection over union ratio, ρ(B truth ,B pred) represents the Euclidean distance between the center points of the two rectangular boxes of the real box and the predicted box, c is the diagonal distance of the minimum closed rectangular area that can contain both the real box and the predicted box, α is the weight, and v is the similarity function used to measure the aspect ratio. Where w is the width of the actual box and the predicted box, and h is the height of the box; if w and h are both between 0 and 1, w 2 +h 2 will be very small. To avoid the gradient problem, 1 / (w 2 +h 2 ) is replaced with 1.

[0130] Step 2.3.3.2 Classification Loss calculation, the classification loss is to process the target category prediction part. In each grid, the probability of each target category will be predicted, and the Cross Entropy Loss is used to measure the difference between the predicted category probability distribution and the true label.

[0131]

[0132] In the cross entropy loss J, N is the number of categories, y i is the actual class label (one-hot encoding, the target class is encoded as 1, and all other classes are encoded as 0), is the predicted probability. In multi-classification problems, the probability is obtained through the Softmax function, and the base of log is e.

[0133]

[0134] In the classification loss L, N is the number of categories, and y i represents the label of sample i, the positive class is 1, the negative class is 0, and p i Represents the probability that the sample is a positive class.

[0135] Step 2.3.3.3 Bounding Box Loss calculation, the bounding box loss is used to predict the loss of the boundary, including the loss of the center point coordinates and the side length, including coordinate loss and size loss. The mean square error MSE and absolute error MAE use regression to calculate the square difference and absolute difference between the predicted value and the true value respectively:

[0136]

[0137] in, and are the y and x coordinates of the center of the prediction box, and are the y and x coordinates of the real box.

[0138] Step 2.3.3.4 For the classes included in the pre-trained model or the classes that have been trained before, they need to be tuned. Under the premise that there will not be much difference between the added training samples and the previous ones, we perform transfer training here by freezing the backbone model, that is, the early layers of the model. The early layers already contain the common features of the class, so the previous weights are inherited, and the new training set is trained in the later layers, that is, the subsequent connection layers. In this way, the model inherits the previous learning results while being more sensitive to specific tasks, preventing the results of pre-training from being destroyed, and being able to adapt to new data sets, saving training time and improving model performance.

[0139] During step 2.3.3.5 training, monitor the training process through Tensorboard (Tensorflow's visualization tool), obtain the changes in loss function and learning rate, and output the average precision value mAP.

[0140] Step 3: Real-time floating object detection

[0141] Step 3.1 Image input and reasoning

[0142] Step 3.1.1 Input the preprocessed image into the trained deep learning model, and the model uses a multi-layer convolutional neural network to extract features, classify, and locate detection frames on the image.

[0143] In step 3.1.2, the model learns all possible classes that may appear in the area that needs to be predicted in a specific scene, and then predicts the floating object class.

[0144] Step 3.2 Confidence screening

[0145] The confidence level comprehensively evaluates the accuracy of the target in the detection frame and the accuracy of the bounding box prediction. The target accuracy is expressed by the probability of the target existing in the prediction frame. If there is only a single class, the confidence level is high, and the number of target classes in the frame is inversely proportional to the target confidence level. The accuracy of the bounding box prediction is expressed by the intersection-over-union ratio. The detection target and the comprehensive confidence level based on the model output are used to filter the detection results, remove the detection frames with low confidence, and retain the floating objects with high confidence.

[0146]

[0147] The confidence is the product of the probability P of the prediction being the target and the intersection-over-intersection ratio.

[0148] Step 3.3 Non-maximum suppression (NMS)

[0149] For detection frames that may overlap, the NMS algorithm is used to filter out the optimal detection frame to avoid repeated recognition of the same target. First, based on the target detection results, all detected bounding boxes are obtained, and the overlap of the detection boxes is calculated using the intersection-over-union algorithm. All bounding boxes are traversed starting from the bounding box with the highest intersection-over-union ratio, and the sizes of the intersection-over-union ratios are compared in turn. If the intersection-over-union ratio is greater than the threshold, which is 0.7 by default, these highly overlapping boxes are suppressed, and the most representative boxes are retained.

[0150] Step 4: Personalize parameters

[0151] According to project requirements, users can flexibly adjust the following parameters:

[0152] Step 4.1 Detection category adjustment

[0153] The original categories include the classes in pre-training and the classes added in later training, but the categories can be adjusted for different application scenarios. For example, for the purpose of whether leaves need to be salvaged, there is no need to salvage when there are only three or two leaves, but leaves in pieces need to be salvaged. At this time, two classes are added to distinguish between non-piece leaves and pieces of leaves. Other scenarios can also add more different classes according to the characteristics of different regions. In addition, for specific scenarios that may be limited by the previous training sets, such as poor recognition of certain classes in different regions or different cameras, more targeted training sets can be added to the scene to strengthen training to achieve relatively better recognition effects for the scene.

[0154] Step 4.2 Confidence Threshold

[0155] By adjusting the confidence level, you can control the reliability of the detection results.

[0156] Step 4.3 Detection box area ratio

[0157] Set the minimum and maximum areas of the detection frame according to the size of the floating objects to remove irrelevant small objects.

[0158] Step 4.4 Intersection over Union (IOU)

[0159] Adjust the overlap between detection boxes to reduce the problem of multiple detections.

[0160] Step 4.5 Polygonal Mask: Set a mask for irrelevant areas in complex scenes, ignore background noise, and focus on detecting specific areas.

[0161] Step 5: Analysis and application of test results

[0162] Step 5.1 Result output and analysis

[0163] The detection results include key information such as the type, location and confidence level of floating objects. The system outputs alarm events containing this detection information in real time for river managers to clean up floating objects or conduct further analysis. The output results can be limited in terms of confidence level as needed, for example, only outputting results with higher confidence levels.

[0164] Step 5.2 System linkage

[0165] The detection system can be linked with other IoT sensing devices to improve river management efficiency.

[0166] Step 5.3 Performance Optimization

[0167] Based on real-time detection results, users can continuously adjust parameters to optimize the performance of the detection system and ensure the best results in different environments.

[0168] The implementation tools are as follows:

[0169] Hardware implementation: The river video is collected in real time by high-definition camera equipment and connected to the detection system based on the river floating object detection method of the present invention.

[0170] Software implementation: Implement the target detection algorithm based on PyTorch, integrate the transfer learning module, and fine-tune the model.

[0171] System integration: Integrate the detection system with the river management platform to achieve automatic feedback of detection results and equipment linkage.

[0172] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A method for detecting floating objects in a river based on transfer learning and target detection algorithm, characterized in that: The following steps are involved: Step 1: Data collection and preprocessing: Acquire real-time video or image data containing floating objects on the river surface, pre-process the collected images, determine the target categories to be detected and mark them, and then perform denoising, image enhancement and size adjustment; Step 2: Model initialization and transfer learning: Select a deep learning target detection model pre-trained based on a large-scale image dataset, use transfer learning technology to fine-tune the pre-trained model using annotated data in a specific river scene, divide the annotated dataset into a training set and a validation set, and use cross-validation to select relatively suitable parameters for training; During the training process, data enhancement technology is used, and the cross entropy loss function and L2 regularization technology are used to optimize the loss function. For the classes included in the pre-trained model or the classes that have been trained before, transfer training is performed to tune them. The training process is monitored through Tensorboard to obtain the changes in the loss function and learning rate, and the mean average precision value mAP is output at the same time. Step 3: Real-time floating object detection: The preprocessed image is input into the trained deep learning model. The model uses a multi-layer convolutional neural network to extract features, classify and locate the detection frame of the image, and predict the floating object class. The confidence level comprehensively evaluates the accuracy of the target in the detection frame and the accuracy of the border prediction. The target accuracy is expressed by the probability of the existence of the target in the prediction frame, and the accuracy of the border prediction is expressed by the intersection-over-union ratio. The detection frames with low confidence are removed, and the floating object targets with high confidence are retained. For the detection frames that may overlap, the NMS algorithm is used to filter out the optimal detection frame. Step 4: Personalized parameter adjustment: According to project requirements, flexibly adjust the following parameters: detection category, confidence threshold, detection box area ratio, intersection ratio, polygon mask; Step 5: Analysis and application of test results: The detection results include key information about floating objects; the detection system outputs alarm events containing floating object information in real time, which is convenient for floating object cleaning or further analysis; the detection system is linked with other IoT sensing devices; based on real-time detection results, users can continuously adjust parameters and optimize the performance of the detection system to ensure the best results in different environments.

2. The method for detecting floating objects in a river channel based on transfer learning and target detection algorithm according to claim 1, characterized in that: The specific steps include: Step 1: Data collection and preprocessing Step 1.1 River channel image acquisition Use high-definition cameras to obtain real-time video or image data of the river, ensuring that the image clarity and coverage are sufficient to capture floating objects on the river surface; Step 1.2 Image preprocessing The collected images are preprocessed, including the following steps: Step 1.2.1 Determine the target category to be detected; the training model has limited training details on the target category, and reverse training can be performed on non-floating objects to reduce the effect of misidentification; floating objects on the water surface in different regions are divided into garbage and non-garbage categories, and can be subdivided according to shape and material; Step 1.2.2 Use the annotation tool to annotate the collected data according to the previously determined target category and assign a category label; Step 1.2.3 Denoising: Use a filter to remove noise from the image; try to increase the background image; perform convolution on the image, and for points on the image that coincide with points on the filter, multiply the points on the convolution template in the filter with the points on the image, and then add the products of each point to get the convolution value of the point. Through repeated calculations, each point is calculated with the surrounding pixels to get a new pixel map; Step 1.2.4 Image enhancement: Perform histogram equalization on the image to enhance the visibility of floating objects; distribute the grayscale of the image more evenly and change the image contrast; Step 1.2.5 Resizing: Resize the image to a standard size of 640×640 pixels suitable for deep learning model input; Step 2: Model initialization and transfer learning Step 2.1 Select a pre-trained model We selected a deep learning target detection model pre-trained on a large-scale image dataset. The YOLO model has good generalization ability and can quickly adapt to the task of detecting floating objects in rivers. Step 2.2 Model fine-tuning Through transfer learning technology, the pre-trained model is fine-tuned using labeled data from specific river scenarios, enabling the model to identify specific types of floating objects and effectively distinguish non-floating objects in the water. Step 2.3 Training process Step 2.3.1 Divide the labeled data set into 80% training set and 20% validation set for training; and Apply cross-validation and try different parameter combinations on each subset and select relatively suitable parameters based on the performance of the validation set; Step 2.3.2 Data enhancement: Use data enhancement techniques during training to increase data diversity and improve the robustness of the model; Data enhancement methods include geometric enhancement that changes the position, direction, and size of the target object; color space adjustment; data enhancement algorithms include mosaic enhancement, mixed enhancement, random perturbation, and color perturbation; Step 2.3.3 Loss function optimization: Use cross entropy loss function and L2 regularization technology; the loss function is a mixed result, including the following: Step 2.3.3.1 Intersection over Union Loss calculation, IoU is in the positioning accuracy part, calculate the overlap between the predicted result and the actual result; The IoU intersection-over-union ratio is the ratio of the overlapping area to the union area. truth is the real frame, B pred is the predicted box, the overlapping area is the intersection of the real box and the predicted box, and the union area is the union of the real box and the predicted box; L CIoU is the IoU loss function, IoU is the intersection over union ratio, ρ(B truth ,B pred ) represents the Euclidean distance between the center points of the two rectangular boxes of the real box and the predicted box, c is the diagonal distance of the minimum closed rectangular area that can contain both the real box and the predicted box, α is the weight, and v is the similarity function used to measure the aspect ratio. Where w is the width of the actual box and the predicted box, and h is the height of the box; Step 2.3.3.2 Classification Loss calculation, the classification loss is to deal with the target category prediction part. In each grid, the probability of each target category will be predicted, and the Cross Entropy Loss is used to measure the difference between the predicted category probability distribution and the true label; In the cross entropy loss J, N is the number of categories, y i is the actual class label, is the predicted probability. In multi-classification problems, the probability is obtained by the Softmax function, and the base of log is e; In the classification loss L, N is the number of categories, and y i represents the label of sample i, the positive class is 1, the negative class is 0, and p i Indicates the probability that the sample is a positive class; Step 2.3.3.3 Bounding Box Loss calculation, the bounding box loss is used to predict the loss of the boundary, including the loss of the center point coordinates and the side length, including coordinate loss and size loss. The mean square error MSE and absolute error MAE use regression to calculate the square difference and absolute difference between the predicted value and the true value respectively: in, and are the y and x coordinates of the center of the prediction box, and are the y and x coordinates of the real box; Step 2.3.3.4 For the classes included in the pre-trained model or the classes that have been trained before, it is necessary to tune them. Under the premise that there is not much difference between the added training samples and the previous ones, transfer training is performed. By freezing the backbone model, that is, the early layers of the model, the early layers already contain the common features of the class, so the previous weights are inherited, and the new training set is trained in the later layers, that is, the subsequent connection layers; Step 2.3.3.5 During the training process, monitor the training process through Tensorboard, obtain the changes in loss function and learning rate, and output the mean average precision value mAP; Step 3: Real-time floating object detection Step 3.1 Image input and reasoning Step 3.1.1 Input the preprocessed image into the trained deep learning model, and the model uses a multi-layer convolutional neural network to extract features, classify, and locate detection frames on the image; Step 3.1.2 The model learns all possible classes that may appear in the area to be predicted in a specific scene, and then predicts the floating object class; Step 3.2 Confidence screening The confidence level comprehensively evaluates the accuracy of the target in the detection frame and the accuracy of the bounding box prediction. The target accuracy is expressed by the probability of the target existing in the prediction frame. The accuracy of the bounding box prediction is expressed by the intersection-over-union ratio. The detection target and the comprehensive confidence level based on the model output are used to filter the detection results, remove the detection frames with low confidence, and retain the floating objects with high confidence. Confidence is the product of the probability P of the prediction being the target and the intersection and union ratio; Step 3.3 Non-maximum suppression For the detection frames that may overlap, the NMS algorithm is used to filter out the optimal detection frame. First, all detected bounding boxes are obtained according to the target detection results. The overlap of the detection boxes is calculated by the intersection-and-union algorithm. All bounding boxes are traversed starting from the bounding box with the highest intersection-and-union ratio. The sizes of the intersection-and-union ratios are compared in turn. If the intersection-and-union ratio is greater than the threshold, the boxes with high overlap are suppressed, and the most representative box is retained. Step 4: Personalize parameters According to actual needs, flexibly adjust the following parameters: Detection category adjustment: The original categories include the classes in pre-training and the classes added in post-training. However, the categories can be adjusted for different application scenarios. If the recognition effect of a specific scene is poor due to the limitation of the previous training set, a more targeted training set in the scene can be added. Confidence threshold: Control the reliability of the detection results by adjusting the confidence level; Detection frame area ratio: Set the minimum and maximum areas of the detection frame according to the size of the floating objects to remove irrelevant small objects; Intersection-over-Union: adjusts the overlap between detection frames; Polygonal mask: Set a mask for irrelevant areas in complex scenes, ignore background noise, and focus on detecting specific areas; Step 5: Analysis and application of test results Step 5.1 Result output and analysis The detection results include key information such as the type, location and confidence level of floating objects. The detection system implemented based on this method outputs alarm events containing these detection information in real time to clean up floating objects or conduct further analysis. The output results can be limited in terms of confidence level as needed. Step 5.2 System linkage The detection system can be linked with other IoT sensing devices; Step 5.3 Performance Optimization Based on real-time detection results, parameters are continuously adjusted to optimize the performance of the detection system to ensure the best results in different environments.

3. The method for detecting floating objects in a river channel based on transfer learning and target detection algorithm according to claim 2, characterized in that: The implementation tools are as follows: Hardware implementation: Real-time river video is collected by high-definition camera equipment and connected to the detection system based on the river floating object detection method of the present invention; Software implementation: Implement the target detection algorithm based on PyTorch, integrate the transfer learning module, and fine-tune the model; System integration: Integrate the detection system with the river management platform to achieve automatic feedback of detection results and equipment linkage.

4. The method for detecting floating objects in a river channel based on transfer learning and target detection algorithm according to claim 2, characterized in that: In step 1.2.1, the garbage category includes domestic garbage, the non-garbage category includes leaves and branches, and other targets include people, animals and passing ships.

5. The method for detecting floating objects in a river channel based on transfer learning and target detection algorithm according to claim 2, characterized in that: Data enhancement in step 2.3.

2. The data enhancement methods include geometric enhancements that change the position, orientation, and size of the target object, such as random cropping, rotation, scaling, translation, and perspective; Color space adjustments, such as adjusting brightness, contrast, saturation, hue shift, and adding Gaussian noise; Data enhancement algorithms include mosaic enhancement, mixed enhancement, random perturbation, and color perturbation.

6. The method for detecting floating objects in a river channel based on transfer learning and target detection algorithm according to claim 2, characterized in that: In step 3.2, the probability of the existence of a target in the prediction box is used to express the target accuracy. If only a single class exists, the confidence is higher, and the number of target classes in the box is inversely proportional to the target confidence.

7. The method for detecting floating objects in a river channel based on transfer learning and target detection algorithm according to claim 2, characterized in that: In step 3.3, the default intersection-union ratio is 0.

7.

8. The method for detecting floating objects in a river channel based on transfer learning and target detection algorithm according to claim 2, characterized in that: In step 2.3.3.1, if w and h are both between 0 and 1, w 2 +h 2 will be very small. To avoid the gradient problem, 1 / (w 2 +h 2 ) is replaced by 1; in step 2.3.3.2, y i is the actual class label, using one-hot encoding, the target class is encoded as 1, and all other classes are encoded as 0.

9. The method for detecting floating objects in a river channel based on transfer learning and target detection algorithm according to claim 2, characterized in that: The detection category adjustment in step 4 is based on the purpose of whether leaves need to be salvaged. If there are only three or two leaves, there is no need to salvage them, but leaves in pieces need to be salvaged. At this time, two classes are added to distinguish between non-pieces of leaves and pieces of leaves. For other scenarios, more different classes can be added according to the characteristics of different regions.

10. The method for detecting floating objects in a river channel based on transfer learning and target detection algorithm according to claim 2, characterized in that: In step 5.1, the confidence level of the output results can be limited as needed, for example, only the results with higher confidence levels are output.

Images