Water surface garbage detection method, device and equipment, storage medium and program product

Abstract

The invention relates to the technical field of computers, and provides a water surface garbage detection method and device, equipment, a storage medium and a program product. The method comprises the following steps: monitoring running states of a plurality of threads according to a preset time interval; based on the running state of each thread, detecting whether an abnormal thread exists or not; if the abnormal thread exists, restarting the abnormal thread; and after the abnormal thread is restarted, inputting the video frames in each target folder into a water surface garbage detection model based on the plurality of threads, and obtaining a water surface garbage detection result corresponding to each video frame. By means of the mode, a multi-thread dynamic keep-alive mechanism is introduced, the possibility of detection interruption caused by thread abnormity in the high-load environment can be avoided, the real-time performance and stability of detection are improved, and therefore water surface garbage can be efficiently and stably detected in the high-load environment.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, equipment, storage medium, and program product for detecting surface debris. Background Technology

[0002] Existing solutions for detecting debris on water surfaces mainly rely on video surveillance systems, machine learning-based video analytics systems, or distributed video processing systems.

[0003] Currently, most video surveillance systems and machine learning-based video analytics systems adopt single-threaded or simple multi-threaded architectures, which have limited computing resources and cannot meet the requirements for real-time video processing. Especially under high load environments, the stability of the system will be severely affected. Distributed video processing systems can process video data through the collaborative work of multiple servers, but their load balancing algorithms are simple and difficult to cope with sudden load changes. In addition, the communication overhead is large, which limits the real-time analysis capability of the system and makes it difficult to meet the sudden high load detection needs in complex scenarios.

[0004] Therefore, existing methods for detecting surface debris on water lack real-time performance and stability, making it difficult to efficiently and stably detect surface debris under high load conditions. Summary of the Invention

[0005] This application provides a method, apparatus, equipment, storage medium, and program product for detecting surface debris, in order to solve the technical problems of poor real-time performance and stability of existing surface debris detection methods, making it difficult to efficiently and stably detect surface debris under high load conditions.

[0006] In a first aspect, embodiments of this application provide a method for detecting surface debris, comprising: monitoring the running status of multiple threads at preset time intervals; each thread processing video frames in a target folder, wherein each video frame in the target folder is obtained based on a video to be detected, which is a video taken of the water surface; detecting whether there is an abnormal thread based on the running status of each thread; if there is an abnormal thread, restarting the abnormal thread; after the abnormal thread has restarted, inputting the video frames in each target folder into a surface debris detection model based on the multiple threads, and obtaining the surface debris detection result corresponding to each video frame output by the surface debris detection model.

[0007] In one embodiment, before monitoring the running status of multiple threads at preset time intervals, the method further includes: obtaining system load information of the video surveillance system; determining the frame extraction frequency based on the system load information; performing frame extraction processing on the video to be detected according to the frame extraction frequency to obtain multiple video frames to be stored; and storing the multiple video frames to be stored in multiple target folders in chronological order based on multiple threads using a polling method.

[0008] In one embodiment, multiple video frames to be stored are stored in multiple target folders in chronological order using a polling method based on multiple threads. This includes: generating multiple temporary files and locking each temporary file; writing multiple video frames to be stored into the multiple temporary files in chronological order using a polling method based on multiple threads; and unlocking each temporary file and renaming each temporary file after all video frames to be stored have been written into the multiple temporary files, thereby obtaining multiple target folders.

[0009] In one embodiment, before inputting video frames from each target folder into the surface debris detection model to obtain the surface debris detection result corresponding to each video frame output by the surface debris detection model, the method further includes: performing refinement enhancement and annotation processing on the sample data to obtain detection result labels corresponding to non-debris samples and non-debris samples, detection result labels corresponding to marine debris samples and marine debris samples, and detection result labels corresponding to surface debris samples and surface debris samples; and training an initial model constructed based on the EfficientDet D2 algorithm based on the detection result labels corresponding to non-debris samples and non-debris samples, detection result labels corresponding to marine debris samples and marine debris samples, and detection result labels corresponding to surface debris samples and surface debris samples to obtain the surface debris detection model.

[0010] In one embodiment, the surface debris detection model includes a backbone network, a bidirectional feature pyramid network, a classification prediction network, and a bounding box prediction network. The backbone network is used to extract features from video frames to obtain feature maps. The bidirectional feature pyramid network is used to perform multi-scale feature fusion on the feature maps to obtain multi-scale features. The classification prediction network and the bounding box prediction network are used to generate surface debris detection results corresponding to the video frames based on the multi-scale features.

[0011] In one embodiment, after inputting video frames from each target folder into a surface debris detection model and obtaining the surface debris detection result corresponding to each video frame output by the surface debris detection model, the method further includes: optimizing the parameters of the Sort target tracking algorithm to determine the maximum tracking age, minimum hit count, and cross-union ratio (CUP) threshold of the Sort target tracking algorithm; performing target tracking on each video frame based on the surface debris detection result, maximum tracking age, minimum hit count, and CUP threshold to obtain the target tracking result; and generating alarm information based on the target tracking result.

[0012] Secondly, embodiments of this application provide a surface debris detection device, comprising: a monitoring module for monitoring the running status of multiple threads at preset time intervals; each thread for processing video frames in a target folder, wherein each video frame in the target folder is obtained based on a video to be detected, which is a video captured of the water surface; a judgment module for detecting whether an abnormal thread exists based on the running status of each thread; a recovery module for restarting the abnormal thread if one exists; and a detection module for inputting the video frames in each target folder into a surface debris detection model based on multiple threads after the abnormal thread has restarted, thereby obtaining the surface debris detection result corresponding to each video frame output by the surface debris detection model.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described methods for detecting surface debris.

[0014] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the above-described methods for detecting surface debris.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the above-described methods for detecting surface debris.

[0016] The surface debris detection method, apparatus, device, storage medium, and program product provided in this application introduce a multi-threaded dynamic keep-alive mechanism. This mechanism periodically monitors the running status of multiple threads and automatically restarts any abnormal threads detected. This ensures that each thread can continuously and normally process video frames in the corresponding target folder. The surface debris detection model is then used to detect surface debris in the video frames. This avoids the possibility of detection interruption due to thread abnormalities under high load conditions, improves the real-time performance and stability of the detection, and enables efficient and stable detection of surface debris under high load conditions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts of the water surface garbage detection method provided in the embodiments of this application.

[0019] Figure 2 This is the second flowchart illustrating the surface debris detection method provided in this application.

[0020] Figure 3 This is a schematic diagram of the workflow of the surface debris detection model provided in the embodiments of this application.

[0021] Figure 4 This is a schematic diagram of the network structure of the surface debris detection model provided in the embodiments of this application.

[0022] Figure 5 This is a schematic diagram of the surface debris detection device provided in the embodiments of this application.

[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Please see Figure 1 and Figure 2 , Figure 1 This is one of the flowcharts illustrating the surface debris detection method provided in this application. Figure 2 This is the second schematic flowchart of the water surface debris detection method provided in the embodiments of this application. Figure 1 As shown in the embodiments of this application, the water surface debris detection method is applied to a video surveillance system or a video analysis system. The water surface debris detection method includes steps S110 to S140, and the specific steps are as follows: S110: Monitor the running status of multiple threads at preset time intervals.

[0026] Each thread processes video frames in a target folder. Each video frame in a target folder is obtained based on the video to be detected, which is a video taken of the water surface.

[0027] To achieve real-time detection and alarm of surface debris, the video surveillance system or video analysis system in this embodiment combines a multi-threaded dynamic keep-alive mechanism, a partitioned buffer balancing strategy, file write protection, uniform frame dropping adjustment for video capture, and an optimized Sort target tracking algorithm to enhance the system's stability and real-time processing capabilities.

[0028] In this embodiment, video frames extracted from the video to be detected are evenly stored in multiple target folders according to chronological order. Each target folder is handled by an independent thread, and each thread processes the video frames in each target folder sequentially in a round-robin manner. This video frame allocation and processing method can improve the system's processing efficiency and reduce the system's waiting time; at the same time, even if some threads are interrupted, the video frames in the target folders corresponding to the remaining threads will not be affected, thereby minimizing the impact of thread interruption on the entire system.

[0029] The number of target folders must be at least two.

[0030] Preferably, the number of target folders is 10.

[0031] Furthermore, the system can monitor the running status of multiple threads simultaneously at preset time intervals.

[0032] Optionally, the preset time interval is 10 seconds.

[0033] S120: Detects whether there are abnormal threads based on the running status of each thread.

[0034] S130: If an abnormal thread exists, restart the abnormal thread.

[0035] Specifically, when a thread crashes or stops, it is considered an abnormal thread, and the system automatically restarts it to ensure the continuity of video processing. This multi-threaded dynamic keep-alive mechanism ensures that the system can quickly recover in abnormal situations, thereby guaranteeing the continuity and real-time performance of video frame processing, reducing the risk of system downtime, ensuring continuous and stable system operation, and improving the overall reliability of the system.

[0036] Preferably, logs should be generated when an abnormal thread restarts, so as to facilitate subsequent problem analysis and troubleshooting.

[0037] S140: After the abnormal thread restarts, based on multiple threads, the video frames in each target folder are input into the surface debris detection model to obtain the surface debris detection result corresponding to each video frame output by the surface debris detection model.

[0038] like Figure 2 As shown, it is understandable that the target folder is used to cache video frames before they are input into the surface debris detection model, therefore... Figure 2 This is referred to as a "buffer". Since each target folder is handled by only one independent thread, the buffer pool contains n (n is a positive integer) buffers, and the thread pool has a corresponding n threads.

[0039] Specifically, after the abnormal thread restarts, that is, when it is ensured that each thread can process video frames normally and there are no abnormal threads in the system, each thread, in a polling manner, sequentially inputs the video frames of each target folder into the surface debris detection model in chronological order, and obtains the surface debris detection result corresponding to each video frame output by the surface debris detection model.

[0040] The surface debris detection method provided in this application introduces a multi-threaded dynamic keep-alive mechanism, which periodically monitors the running status of multiple threads and automatically restarts abnormal threads when abnormal threads are detected. This ensures that each thread can continuously and normally process video frames in the corresponding target folder, and then uses a surface debris detection model to detect surface debris in the video frames. This avoids the possibility of detection interruption due to thread abnormalities under high load conditions, improves the real-time performance and stability of detection, and enables efficient and stable detection of surface debris under high load conditions.

[0041] In some embodiments, before monitoring the running status of multiple threads at preset time intervals, the method further includes: obtaining system load information of the video surveillance system; determining the frame extraction frequency based on the system load information; performing frame extraction processing on the video to be detected according to the frame extraction frequency to obtain multiple video frames to be stored; and storing the multiple video frames to be stored in multiple target folders in chronological order according to the polling method based on the multiple threads.

[0042] Understandably, the system may need to cope with high load or high concurrency. Under high load or high concurrency, processing all video frames of the video to be detected will further increase the system burden, causing the thread to crash or stop. Therefore, this embodiment proposes a video frame extraction adjustment mechanism to adjust the number of extracted video frames under different load conditions in order to adapt to complex detection scenarios and improve the flexibility of video capture and the processing efficiency of the system.

[0043] The system can flexibly adjust the frame extraction frequency according to the needs of the scenario: in normal scenarios, one video frame is saved after each video frame is read, that is, all video frames are extracted; in high-load scenarios or non-critical scenarios, it can be set to save one video frame every four video frames to reduce system pressure.

[0044] Specifically, the system load information of the video surveillance system is obtained. Based on the system load information and the current processing capacity of the system, the frame extraction frequency is dynamically adjusted. The video to be detected is then processed by extracting frames according to the frame extraction frequency to obtain multiple video frames to be stored, so as to ensure that the system can maintain stable operation under high load.

[0045] Furthermore, based on multiple threads, a polling method is used to store multiple video frames to be stored in chronological order to multiple target folders.

[0046] In this embodiment, a partitioned buffer balancing strategy is used to optimize the video frame processing flow.

[0047] Specifically, after the video frames to be stored are extracted (i.e., captured), multiple threads can use a round-robin approach to evenly distribute the multiple video frames to multiple target folders in chronological order. Each target folder can be considered a partition, thus achieving partitioned storage of video frames. This avoids system performance bottlenecks caused by excessive I / O (input / output) operations on a single folder. By distributing video frames across different target folders through partitioned storage, overload of a single folder can be effectively prevented.

[0048] It's important to note that each target folder is handled by a separate thread, which processes the video frames in the target folder sequentially in a polling manner. This allocation method improves system processing efficiency, reduces system waiting time, and minimizes the impact of thread interruptions on the overall system.

[0049] The surface debris detection method provided in this application dynamically adjusts the frame sampling frequency according to the current system load and processing capacity, ensuring stable operation of the system even under high load. At the same time, through a uniform frame sampling adjustment mechanism, the system can achieve up to 4 times the processing capacity, thereby significantly improving the overall efficiency and flexibility of the system without affecting the detection accuracy. In addition, by using a partitioned buffering method, the system performance bottleneck caused by excessive I / O operations on a single folder is avoided, and the upper limit of the system's processing frame rate is increased by 10 times.

[0050] In some embodiments, multiple video frames to be stored are stored in multiple target folders in chronological order using a polling method based on multiple threads. This includes: generating multiple temporary files and locking each temporary file; writing multiple video frames to be stored into multiple temporary files in chronological order using a polling method based on multiple threads; and unlocking each temporary file and renaming each temporary file after all video frames to be stored have been written into multiple temporary files to obtain multiple target folders.

[0051] To ensure data integrity during the reading and processing of video frames, this embodiment employs a file write protection mechanism to guarantee data consistency and reliability.

[0052] The file write protection mechanism consists of three parts: temporary file writing, renaming, and file locking. Before actually writing video frames, they are first saved to a temporary file, and then renamed to the final file after the writing is complete. During the processing, the file is locked to prevent other processes or threads from writing to it, so as to avoid data anomalies caused by read-write conflicts.

[0053] Specifically, for each thread, before writing the video frame to be stored to the target folder, the thread first saves it to a temporary file and locks the temporary file; multiple threads can use a polling method to write multiple video frames to be stored to multiple temporary files in chronological order. After all the video frames to be stored have been written to the corresponding multiple temporary files, each temporary file is unlocked and renamed, thus obtaining multiple target folders.

[0054] For example, suppose there are 100 video frames to be stored, which need to be evenly written to 10 target folders. Before the first thread actually writes the first video frame to the first target folder, it first saves it to a first temporary file and locks it. Similarly, before the second thread actually writes the second video frame to the second target folder, it saves it to a second temporary file and locks it, and so on. The tenth thread saves the tenth video frame to the tenth target folder to the tenth temporary file and locks it. Following a round-robin approach, after the tenth video frame is written to the tenth temporary file, the first thread will save the eleventh video frame to the first temporary file, and so on, until the 100th video frame is stored. Understandably, once 10 video frames have been written to the first temporary file, it can be unlocked and renamed, and the same process can be repeated for the first target folder, ultimately resulting in 10 target folders.

[0055] The surface debris detection method provided in this application adopts a file write protection mechanism based on temporary file writing, renaming, and file locking during the video frame writing process. This mechanism can ensure the consistency and integrity of video frame data, effectively prevent data conflicts in a multi-threaded environment, and effectively avoid data corruption and system anomalies.

[0056] Through the aforementioned multi-threaded dynamic keep-alive mechanism, partitioned buffer balancing strategy, and file write protection mechanism, the water surface debris detection method provided in this application embodiment can reduce the time delay from reading video frames to requesting analysis, and significantly improve the real-time processing capability of the system.

[0057] In some embodiments, before inputting video frames from each target folder into the surface debris detection model to obtain the surface debris detection result corresponding to each video frame output by the surface debris detection model, the method further includes: performing refinement enhancement processing and annotation processing on the sample data to obtain detection result labels corresponding to non-debris samples and non-debris samples, detection result labels corresponding to marine debris samples and marine debris samples, and detection result labels corresponding to surface debris samples and surface debris samples; and training an initial model constructed based on the EfficientDet D2 algorithm based on the detection result labels corresponding to non-debris samples and non-debris samples, detection result labels corresponding to marine debris samples and marine debris samples, and detection result labels corresponding to surface debris samples and surface debris samples to obtain the surface debris detection model.

[0058] Please see Figure 3 , Figure 3 This is a schematic diagram of the workflow of the surface debris detection model provided in the embodiments of this application.

[0059] like Figure 3 As shown, before using the surface debris detection model for surface debris detection and alarm, it is necessary to collect and label sample data for model training.

[0060] The surface debris detection model in this embodiment uses the EfficientDet D2 algorithm's network architecture as its foundation, boasting high detection accuracy and low computational complexity, making it particularly suitable for real-time applications. The EfficientDetD2 model employs EfficientNet as its backbone network, combined with a Bidirectional Feature Pyramid Network (BiFPN) to achieve multi-scale feature fusion, thereby significantly improving the accuracy of surface debris detection. Furthermore, by refining and enhancing the sample data during training, the accuracy and sensitivity of the EfficientDet D2 model in detecting surface debris can be further improved, thereby reducing false positives and false negatives.

[0061] Specifically, before training begins, the sample data undergoes refinement, enhancement, and annotation to ensure the model can accurately identify and detect marine debris and surface litter. Simultaneously, non-debris samples are introduced during the annotation process to improve the model's robustness and enhance recognition accuracy.

[0062] Optionally, non-garbage samples include 500 images of bridges, 500 images of small boats, 500 images of rocks, 500 images of balls, 500 images of water platforms, 500 images of ships, 500 images of masts, 500 images of ports, 500 images of animals, 500 images of buoys, and 500 images of aquatic plants.

[0063] Optionally, the marine debris sample includes 5,500 images, and the surface debris sample (i.e., the river debris sample) includes 5,500 images.

[0064] Optionally, the labeled area covers the rectangular area of ​​the water surface boundary and objects on the water; after the labeling is completed, an XML format label file is generated, which together with the sample image constitutes VOC format labeled data.

[0065] Optionally, the VOC format labeled data is converted to tfrecord file format, and the dataset is randomly divided into training set, validation set and test set in an 8:2:1 ratio to ensure that the ratio of marine debris samples, surface debris samples and non-debris samples in the training set and validation set is 1:1:1.

[0066] Furthermore, using labeled data, the initial model built based on the EfficientDet D2 algorithm was trained to obtain a surface debris detection model.

[0067] It should be noted that before training the model, the initial model built based on the EfficientDet D2 algorithm needs to be adjusted so that it can be used with the labeled dataset. First, some parameters in the configuration file can be changed to suit the labeling use case of this application (e.g., modify the training configuration: reduce the batch size from 128 to 32; increase the initial learning rate in the optimizer to 0.1; change the path and type of fine-tuning checkpoints, changing "classification" to "detection"; update the label mapping path and the input TFRecord file path to reflect the labeled dataset). After the parameters in the configuration file are modified, the model is trained for 300,000 steps to complete the model training.

[0068] Optionally, the trained surface debris detection model can be evaluated using object_detection, and the model performance can be evaluated using a test set.

[0069] Optionally, mAP (mean precision), AR (recall), IoU (intersection ratio and union ratio) can be used as evaluation metrics for model performance.

[0070] mAP (mean precision) refers to the average AP across all classes. The AP for each class is the area under the precision-recall curve for that class. mAP = 0.681921 means that the model has an average precision of 68.19% across all classes, which is a relatively high value, indicating that the model has good performance.

[0071] AR stands for recall, which is the proportion of positive samples correctly identified by the model to the actual number of positive samples.

[0072] Intersection over Union (IoU) is a metric that measures how much the predicted bounding box and the ground truth bounding box overlap. Its value ranges from 0 to 1, with higher values ​​indicating greater overlap. At an IoU threshold of 0.50, the model achieved AP = 0.803 and AR = 0.701, which are excellent results, demonstrating that the model performs well at lower IoU thresholds.

[0073] like Figure 3 As shown, after the model's performance evaluation is qualified, the surface debris detection model needs to be formally deployed and applied.

[0074] Specifically, the trained surface debris detection model is exported as a model file using TensorFlow's exporter_main_v2.py. The model file includes the weights, network structure, and related configuration parameters of the surface debris detection model.

[0075] Furthermore, the exported model file is loaded using Python 3, ready for detection.

[0076] Specifically, a detection interface is written in Python that receives video frames, uses a loaded surface debris detection model to detect surface debris, and then performs a weighted decision based on the detection results of the surface debris detection model and returns it to the system client.

[0077] In some embodiments, the surface debris detection model includes a backbone network, a bidirectional feature pyramid network, a classification prediction network, and a bounding box prediction network; the backbone network is used to extract features from video frames to obtain feature maps; the bidirectional feature pyramid network is used to perform multi-scale feature fusion on the feature maps to obtain multi-scale features; the classification prediction network and the bounding box prediction network are used to generate surface debris detection results corresponding to the video frames based on the multi-scale features.

[0078] Please see Figure 4 , Figure 4 This is a schematic diagram of the network structure of the surface debris detection model provided in the embodiments of this application.

[0079] like Figure 4 As shown, the surface debris detection model includes an EfficientNet backbone, a Bi-FPN layer, a class prediction network, and a box prediction network.

[0080] The backbone network is used to extract features from video frames to obtain feature maps; the bidirectional feature pyramid network is used to perform multi-scale feature fusion on the feature maps to obtain multi-scale features; and the classification prediction network and the bounding box prediction network are used to generate the water surface garbage detection results corresponding to the video frames based on the multi-scale features.

[0081] In some embodiments, after inputting video frames from each target folder into the surface debris detection model and obtaining the surface debris detection result corresponding to each video frame output by the surface debris detection model, the method further includes: optimizing the parameters of the Sort target tracking algorithm to determine the maximum tracking age, minimum hit count, and cross-union ratio (CUP) threshold of the Sort target tracking algorithm; performing target tracking on each video frame based on the surface debris detection result, maximum tracking age, minimum hit count, and CUP threshold corresponding to each video frame to obtain the target tracking result; and generating alarm information based on the target tracking result.

[0082] Understandably, the detection results of the water surface debris detection model are not necessarily completely accurate. The system's direct prediction of warnings based on the water surface debris detection results for each video frame may result in false alarms. Therefore, this embodiment employs the Sort target tracking algorithm and optimizes its parameters to better meet the needs of real-time alerts.

[0083] Specifically, the parameters of the Sort target tracking algorithm are optimized to determine the maximum tracking age, minimum number of hits, and cross-union ratio threshold of the Sort target tracking algorithm.

[0084] Optionally, the maximum tracking age (max_age) is set to 5, which means that tracking stops when no target is detected within 5 video frames, thus avoiding wasting system resources on invalid targets.

[0085] Optionally, the minimum number of hits (min_hits) is set to 7 to ensure that tracking is only performed when the target is detected more than 7 times consecutively, effectively reducing the possibility of false alarms.

[0086] Optionally, the intersection-overlap ratio threshold (iou_threshold) is set to 0.7 to ensure that only targets with high overlap are considered the same target in consecutive video frames, reducing the possibility of false alarms and false negatives.

[0087] By optimizing the parameters of the Sort target tracking algorithm, the system can filter out transient noise and false alarms during target tracking, thereby reducing unnecessary alarms, making more effective use of computing resources, and improving the overall operating efficiency and alarm reliability of the system.

[0088] Furthermore, based on the surface debris detection results, maximum tracking age, minimum hit count, and cross-union threshold corresponding to each video frame, target tracking is performed on the detected targets in each video frame to obtain target tracking results, and alarm information is generated based on the target tracking results.

[0089] For example, if based on the detection results of surface debris in 7 consecutive video frames, it is determined that surface debris exists in all 7 consecutive video frames, and the cross-union threshold of surface debris identified in any two adjacent video frames is greater than 0.7, then the existence of surface debris is officially confirmed, and target tracking is performed; the system will generate corresponding alarm information based on the target tracking results.

[0090] The surface debris detection method provided in this application embodiment utilizes the EfficientDet D2 model, which has been specifically optimized and trained to meet the high real-time and high accuracy requirements of surface debris detection. Combined with a dynamic multi-threaded processing mechanism and a Sort target tracking algorithm, it effectively addresses the shortcomings of traditional methods in terms of real-time performance, adaptability to complex scenarios, and processing efficiency. In particular, it improves detection accuracy under complex surface environments with varying lighting and reflection interference, achieving efficient monitoring and real-time alarming of surface debris. This not only improves the accuracy and real-time performance of debris detection but also enhances the stability and flexibility of the system through optimized architecture design, significantly improving the overall system's application effect in complex surface environments.

[0091] This application's embodiments achieve efficient and stable water monitoring and alarms through a multi-threaded dynamic keep-alive mechanism, a partitioned buffer balancing strategy, file write protection, uniform frame skipping adjustment for video capture, and an optimized Sort target tracking algorithm. Through these technical means, the system achieves video frame processing latency of less than 20ms, full frame rate upper limit compatibility, up to 4x adjustable video processing capability, and precise alarm trigger adjustment, significantly improving the real-time performance and reliability of water monitoring.

[0092] In summary, compared with the prior art, the surface debris detection method provided in this application has at least the following technical advantages: (1) Improved system stability: In the prior art, single-threaded or simple multi-threaded architectures are difficult to guarantee system stability; while this application ensures that the system can recover quickly in abnormal situations and maintain stable operation through a multi-threaded dynamic keep-alive mechanism.

[0093] (2) Improved processing efficiency: Existing distributed video processing systems are complex and have high deployment and maintenance costs; however, this application optimizes system I / O operations through a partitioned buffer balancing strategy, which significantly improves video frame processing efficiency.

[0094] (3) Ensure data integrity and consistency: In traditional systems, file read and write conflicts may lead to data inconsistency; however, this application ensures the integrity and consistency of data during the reading and processing process through a file write protection mechanism.

[0095] (4) Enhanced real-time performance: Traditional target detection algorithms and systems are difficult to meet the high real-time requirements; however, this application significantly improves the real-time processing capability through the Sort target tracking algorithm with uniform frame adjustment and optimization.

[0096] (5) Improved detection accuracy and efficiency: In the existing technology, the traditional target detection algorithm has low accuracy and efficiency; while this application adopts the EfficientDet D2 model, combined with a large amount of finely labeled data, which significantly improves the accuracy and efficiency of garbage detection and achieves efficient and real-time application effect.

[0097] This application also provides a surface debris detection device. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of the structure of the surface debris detection device provided in this application embodiment. In this application embodiment, the surface debris detection device includes a monitoring module 510, a judgment module 520, a recovery module 530, and a detection module 540.

[0098] The monitoring module 510 is used to monitor the running status of multiple threads at preset time intervals.

[0099] Each thread processes video frames in a target folder. Each video frame in a target folder is obtained based on the video to be detected, which is a video taken of the water surface.

[0100] The judgment module 520 is used to detect whether there are abnormal threads based on the running status of each thread.

[0101] Recovery module 530 is used to restart the abnormal thread if an abnormal thread exists.

[0102] The detection module 540 is used to input video frames from each target folder into the surface debris detection model based on multiple threads after the abnormal thread has restarted, and obtain the surface debris detection result corresponding to each video frame output by the surface debris detection model.

[0103] In some embodiments, the surface debris detection device further includes a video acquisition module.

[0104] The video acquisition module is used to acquire system load information of the video surveillance system; determine the frame extraction frequency based on the system load information; perform frame extraction processing on the video to be detected according to the frame extraction frequency to obtain multiple video frames to be stored; and store the multiple video frames to be stored in multiple target folders in chronological order using a polling method based on multiple threads.

[0105] In some embodiments, the video acquisition module is used to generate multiple temporary files and lock each temporary file; based on multiple threads, using a polling method, multiple video frames to be stored are written to the multiple temporary files in chronological order; after all video frames to be stored have been written to the multiple temporary files, each temporary file is unlocked and renamed to obtain multiple target folders.

[0106] In some embodiments, the detection module 540 is used to refine and enhance the sample data and perform annotation processing to obtain detection result labels corresponding to non-garbage samples and non-garbage samples, marine garbage samples and marine garbage samples, and surface garbage samples and surface garbage samples; based on the detection result labels corresponding to non-garbage samples and non-garbage samples, marine garbage samples and marine garbage samples, and surface garbage samples and surface garbage samples, the initial model constructed based on the EfficientDet D2 algorithm is trained to obtain a surface garbage detection model.

[0107] In some embodiments, the surface debris detection model includes a backbone network, a bidirectional feature pyramid network, a classification prediction network, and a bounding box prediction network; the backbone network is used to extract features from video frames to obtain feature maps; the bidirectional feature pyramid network is used to perform multi-scale feature fusion on the feature maps to obtain multi-scale features; the classification prediction network and the bounding box prediction network are used to generate surface debris detection results corresponding to the video frames based on the multi-scale features.

[0108] In some embodiments, the detection module 540 is used to optimize the parameters of the Sort target tracking algorithm, determine the maximum tracking age, minimum number of hits, and cross-union ratio threshold of the Sort target tracking algorithm; perform target tracking on each video frame based on the water surface debris detection result, maximum tracking age, minimum number of hits, and cross-union ratio threshold corresponding to each video frame, and obtain target tracking results; and generate alarm information based on the target tracking results.

[0109] This application also provides an electronic device. Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 6As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions from the memory 630 to execute a surface debris detection method.

[0110] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the water surface debris detection methods provided by the above methods.

[0112] This application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the water surface debris detection methods provided by the above methods.

[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting surface debris on water, characterized in that, include: Monitor the running status of multiple threads at preset time intervals; Each thread is used to process video frames in a target folder, and each video frame in the target folder is obtained based on a video to be detected, which is a video taken of the water surface; Based on the running state of each thread, detect whether there are any abnormal threads; If the abnormal thread exists, then restart the abnormal thread; After the abnormal thread restarts, based on multiple threads, the video frames in each target folder are input into the surface debris detection model to obtain the surface debris detection result corresponding to each video frame output by the surface debris detection model.

2. The method for detecting surface debris according to claim 1, characterized in that, Before monitoring the running status of multiple threads at preset time intervals, the method further includes: Obtain system load information from the video surveillance system; Based on the system load information, the frame extraction frequency is determined; According to the frame extraction frequency, the video to be detected is subjected to frame extraction processing to obtain multiple video frames to be stored. Based on multiple threads, a polling method is used to store multiple video frames to be stored in chronological order into multiple target folders.

3. The method for detecting surface debris according to claim 2, characterized in that, The method of storing multiple video frames to be stored into multiple target folders in chronological order, based on multiple threads and using a polling approach, includes: Generate multiple temporary files and lock each of them; Based on multiple threads, a polling method is used to write multiple video frames to be stored into multiple temporary files in chronological order; After all the video frames to be stored are written to the multiple temporary files, each temporary file is unlocked and renamed to obtain multiple target folders.

4. The method for detecting surface debris according to claim 1, characterized in that, Before inputting the video frames from each of the target folders into the surface debris detection model and obtaining the surface debris detection result corresponding to each video frame output by the surface debris detection model, the method further includes: The sample data is refined, enhanced, and labeled to obtain the detection result labels for non-garbage samples and the corresponding non-garbage samples, the detection result labels for marine garbage samples and the corresponding marine garbage samples, and the detection result labels for surface garbage samples and the corresponding surface garbage samples. Based on the non-garbage samples and their corresponding detection result labels, the marine garbage samples and their corresponding detection result labels, and the surface garbage samples and their corresponding detection result labels, the initial model constructed based on the EfficientDet D2 algorithm is trained to obtain the surface garbage detection model.

5. The method for detecting surface debris according to claim 1, characterized in that, The surface debris detection model includes a backbone network, a bidirectional feature pyramid network, a classification prediction network, and a bounding box prediction network. The backbone network is used to extract features from the video frames to obtain feature maps; The bidirectional feature pyramid network is used to perform multi-scale feature fusion on the feature map to obtain multi-scale features; The classification prediction network and the bounding box prediction network are used to generate the water surface debris detection results corresponding to the video frame based on the multi-scale features.

6. The method for detecting surface debris according to claim 1, characterized in that, After inputting the video frames from each of the target folders into the surface debris detection model and obtaining the surface debris detection result corresponding to each video frame output by the surface debris detection model, the method further includes: The parameters of the Sort target tracking algorithm are optimized to determine the maximum tracking age, minimum number of hits, and cross-union ratio threshold of the Sort target tracking algorithm; Based on the surface debris detection results corresponding to each video frame, the maximum tracking age, the minimum number of hits, and the intersection-union ratio threshold, target tracking is performed on each video frame to obtain target tracking results; Based on the target tracking results, an alarm message is generated.

7. A surface debris detection device, characterized in that, include: The monitoring module is used to monitor the running status of multiple threads at preset time intervals; Each thread is used to process video frames in a target folder, and each video frame in the target folder is obtained based on a video to be detected, which is a video taken of the water surface; The judgment module is used to detect whether there are abnormal threads based on the running status of each thread; A recovery module is used to restart the abnormal thread if the abnormal thread exists. The detection module is used to input video frames from each of the target folders into the surface debris detection model based on multiple threads after the abnormal thread has restarted, and to obtain the surface debris detection result corresponding to each video frame output by the surface debris detection model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the surface debris detection method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the surface debris detection method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the surface debris detection method as described in any one of claims 1 to 6.

Images