Marine benthos detection, identification and tracking method

Abstract

A marine benthic organism detection, identification and tracking method comprises the following steps: collecting, screening and constructing a marine organism benthic data set, performing small target data enhancement, and performing detection and preliminary classification on a target by using a P2-YOLOv11 deep learning model; then, a lightweight ShuffleDeepsort tracking algorithm is used for carrying out tracking; and finally, performing dynamic frame skipping identification in combination with track information obtained by tracking and preliminary classification information, and realizing fine classification under different order element systems by using a multi-level classification network to obtain a multi-level identification result of marine organisms. According to the invention, the processing effect of the small-target marine benthic organism image and the submarine multi-target tracking image actually shot by the underwater vehicle can be obviously improved.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, specifically a method for detecting, identifying and tracking small-target marine benthic organisms from the perspective of an underwater vehicle based on artificial intelligence. Background Art

[0002] Marine benthic organisms are the most widespread and biodiverse ecological group. In recent years, a large amount of video data containing benthic organisms has been collected in deep-sea exploration missions. However, traditional methods rely on professional knowledge and have slow analysis speeds, making it difficult to cope with challenges such as the large number of benthic organisms and small targets.

[0003] Marine benthic organisms are the ecological group with the widest habitat range and the highest biodiversity among marine organisms. In recent years, deep-sea exploration missions using underwater vehicles have collected a large amount of video data about benthic organisms. Traditional manual methods have limited analysis speed and rely on professional knowledge. They are difficult to cope with challenges such as the large number of marine benthic species, small targets under aerial photography, and diverse actual application needs. Although existing deep learning technologies can realize the detection and classification of benthic organisms, there are generally the following unresolved technical problems: they cannot handle multi-type and multi-level recognition and classification tasks, and have weak ability to detect small targets; they are mostly concentrated in laboratory scenarios, with poor applicability and adaptability, and fail to take into account complex actual environments and changeable ocean conditions; they lack an efficient integrated system, cannot take into account target detection, tracking and recognition at the same time, and have failed to form a comprehensive tool that can meet the investigation needs of marine researchers. Summary of the invention

[0001] In view of the shortcomings of the prior art, such as single detection type, weak small target recognition ability and poor practical applicability, the present invention proposes a method for detecting, identifying and tracking marine benthic organisms, which can significantly improve the processing effect of small target marine benthic organism images and seabed multi-target tracking images taken by underwater vehicles.

[0002] The present invention is achieved through the following technical solutions:

[0003] The present invention relates to a method for detecting, identifying and tracking marine benthic organisms. A marine benthic dataset is constructed by collecting and screening. After small target data enhancement, a P2-YOLOv11 deep learning model is used to detect and preliminarily classify the target. A lightweight ShuffleDeepsort tracking algorithm is then used for tracking. Finally, dynamic frame skipping recognition is performed by combining the trajectory information obtained by tracking and the preliminary classification information. A multi-level classification network is used to achieve fine classification under different order element systems, and a multi-level recognition result of marine organisms is obtained.

[0004] The described small target data augmentation means that according to the definition of the relative scale of small targets, objects with a size ratio to the original image size less than 0.3% are considered small targets. Select some images as the base images, consider the complete unoccluded small targets among them, perform random transformations on their wholes, apply Gaussian blur to their edges, and then randomly insert them onto another part of the images while ensuring that they do not overlap with any existing targets and there is a distance from the image boundary to prevent distortion when inserting into the new images.

[0005] The described P2-YOLOv11 deep learning model includes: a backbone network unit, a small target detection head module, a neck unit, and a head unit, where: the backbone network unit performs multi-level feature extraction processing based on the preprocessed image input information to obtain feature maps of different scales; the neck unit performs feature aggregation and fusion processing based on the feature map information of different scales extracted by the backbone network to obtain optimized feature maps; the head unit performs object detection processing based on the optimized feature map information output by the neck unit to obtain detection results such as the category, location, and confidence of the object; the small target detection head module is used to fuse shallow feature information of more scales to enhance the representation ability of the feature pyramid.

[0006] The described P2-YOLOv11 deep learning model is trained in the following way: use the dataset processed by data augmentation to train the model, continuously optimize the model parameters until the preset number of learning iterations is reached, and complete the training process.

[0007] The described lightweight ShuffleDeepsort tracking algorithm specifically includes:

[0008] 1) Initialize the tracker and trajectories: According to the input video stream, initialize the tracker and target trajectory parameters, assign a unique trajectory ID to each detected target, and set the initial state at the same time.

[0009] 2) Extract appearance features: Use ShuffleNetV2 as the feature extraction network to process the detected target boxes and extract the appearance features of the targets. Compared with the traditional DeepSort feature extraction network, ShuffleNetV2 significantly reduces the number of model parameters and network complexity, improves the portability of the system, and is suitable for computing resource-constrained environments such as embedded devices or mobile devices.

[0010] 3) Trajectory prediction: Based on the historical motion information of the targets and the current detection results, use the Kalman filter algorithm to predict the state of the trajectory of each target, generate the predicted position and calculate the error range.

[0011] 4) Re-match the trajectory: After the appearance features are updated, a joint matching strategy of appearance features and motion information is adopted to associate the detection results of the current frame with the predicted trajectory.

[0012] The feature update strategy in step 4) is implemented based on Exponential Moving Average (EMA), that is, by combining the feature information of the current frame and historical frames, dynamically adjusting the update weight according to the confidence of the detector, so as to dynamically optimize the appearance state of the trajectory target. Specifically:

[0013]

[0014] Where: is the appearance feature of trajectory i up to frame t-1, is the appearance feature of the matching detection added to the model, weighted by where confidence is the confidence of the detector and noise is the set threshold for filtering noise.

[0015] 5) Output the tracking results: Output the trajectory information of each target, including trajectory ID, position, speed, and appearance feature status.

[0016] The frame skipping recognition mentioned above refers to: aiming at the problem of long time-consuming for refined classification tasks under different order systems, for the continuously associated trajectories and P2-YOLOv11 detection results, dynamically adjust the frequency of refined classification detection of subtasks. Through frame skipping detection, the detection speed can be greatly improved, and the information generated by detection, tracking, and classification tasks can also be interconnected, enhancing the comprehensiveness of the method. Specifically: for all targets with P2-YOLOv11 detection confidence higher than 0.8, it is considered that their preliminary classification accuracy is relatively high and the refinement difficulty is relatively low, and more detailed category information is confirmed every 10 frames; for targets with detection confidence lower than 0.6, it is considered that their preliminary classification effect is poor and there may be complex situations such as occlusion, and the detection information needs to be confirmed more frequently, and more detailed category information is confirmed every 3 frames; for targets with confidence between 0.6 and 0.8, a refined classification task is performed every 5 frames.

[0017] The refined classification network mentioned above refers to: according to the preliminary classification information and position bounding box of P2-YOLOv11, extract the target area from the corresponding frame, and select classification models for fish and echinoderms that are not limited to being trained specifically for marine organisms to complete accurate classification tasks at the class and below levels, making the overall method have stronger scalability.

[0018] The present invention relates to a system for implementing the above method, comprising: a data preprocessing unit, a detection unit, a tracking unit and an identification unit, wherein: the data preprocessing unit performs small target data enhancement on the constructed data set to improve the performance of small target detection of the model. The detection unit uses the P2-YOLOv11 module to predict the data stream, and obtains the preliminary classification results, bbox position frame and confidence of each target. The tracking unit uses the lightweight ShuffleDeepsort tracking algorithm to associate the existing trajectory information with the detection information provided by P2-YOLOv11, assign a unique Track ID and generate the corresponding trajectory. The identification unit uses a multi-level classification network to perform dynamic frame skipping recognition based on the trajectory information and the detection results of P2-YOLOv11, and obtains specific classification results under different order systems. Technical Effects

[0019] The present invention realizes refined classification under different order-element systems by data enhancement and fusion of multi-scale shallow features, replacement of feature extraction network and optimization of feature update strategy, refinement of classification network and dynamic frame skipping recognition, and can adapt to the diversified needs of practical applications. Compared with the prior art, the present invention significantly improves the detection accuracy of small targets of marine benthic organisms in the image data collected by underwater vehicles in the field and reduces the processing time of the overall method. At the same time, it can realize the refined classification of benthic organisms under multiple order-element systems, meet the needs of diversified target classification in practical applications, and expand the application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the present invention;

[0021] Figure 2 It is the P2-YOLOv11 network model of the target detection module of the present invention. DETAILED DESCRIPTION

[0022] like Figure 1 As shown, this embodiment relates to a method for detecting, identifying and tracking marine benthic organisms, including:

[0023] Step 1: Collect and screen the marine benthic biological dataset, including 7943 images from the ROUD dataset, 3959 images from the Pacstorm dataset, and 503 self-annotated seabed real-shot images, organize them according to the YOLO dataset format, and divide the data into a training set (9362 images) and a test set (3043 images).

[0024] Step 2: Perform small target data augmentation on the images in the training set. Select 200 images as base images, copy the small targets in them, and then randomly paste them onto another 2,000 images.

[0025] Step 3: Construct the Neck and Head parts of the P2-YOLOv11 model, increase its detection scales to 4, and form four branch structures. As Figure 2 shown, the added P2 detection head cascades the upsampled feature layer with the shallow feature layer, and the corresponding detection feature map size is 160X160. Compared with the original model, the P2-YOLOv11 model performs independent detections on the fused feature maps at 4 scales, can learn location features from the shallow feature layer, and improves the detection ability for small targets.

[0026] Step 4: Use the processed marine benthic organism image dataset to train the P2-YOLOv11 model. After multiple debugging and experiments, obtain the optimal result, and export the weights in the model for deployment.

[0027] Step 5: Input the underwater vehicle's obtained seabed real-shot video as continuous frame data into the trained P2-YOLOv11 model for object detection, and obtain the preliminary classification results, bbox position boxes, and confidence levels of each target.

[0028] Step 6: Train the feature extraction module of ShuffleNetV2 in the tracking algorithm. On the premise of maintaining similar tracking indicators, reduce the model weights from the original 43.6M to 1.2M, reduce the number of model parameters, and can adapt to embedded devices or mobile devices with tight computing resources.

[0029] Step 7: Use the lightweight ShuffleDeepSort tracking algorithm to associate the trajectory information with the detection information provided by P2-YOLOv11, assign a unique TrackID, and generate corresponding trajectories. For the continuously matched trajectory information, according to the detection confidence and preliminary classification situation, extract the target area from the corresponding frame, use the multi-level classification network to obtain the refined classification results at the class level, and draw them in the tracking video.

[0030] Based on the above experimental results, the present invention classifies the validation set data according to the definition of the relative scale of small objects, sets the targets with the ratio of the bounding box area to the image area less than 0.3% as small targets, and the targets greater than 0.3% as medium and large targets. Evaluate the P2-YOLO model proposed in this paper according to the above criteria, and the obtained results are shown in Table 1.

[0031] Table 1

[0032] The mAP50 performance of the P2-YOLOv11 model on a dataset dominated by small targets (449 images) is better than that of the original model, and it achieves results comparable to the original model on the complete test set (3043 images). When only considering small targets in the complete dataset, the P2-YOLOv11 model also outperforms the original model in terms of both average precision (AP) and average recall (AR).

[0033] To evaluate the tracking effect, 10 multi-target underwater real-shot videos, totaling 5881 frames, were selected for testing after self-annotation. The video data used was obtained during on-site inspections by an underwater vehicle, moving uniformly parallel to the seabed from an aerial perspective. The widely used MOTA and IDF1 were selected as the evaluation metrics, and the results are shown in Table 2.

[0034] Table 2

[0035] Compared with the prior art, the present invention, through the ShuffleNetV2 and exponential moving average (EMA) feature update strategy, achieves the following effects: while maintaining the performance of the MOTA and IDF1 metrics comparable to the original algorithm, it significantly reduces the inference time of the model. Specifically, the lightweight design of the feature extraction network reduces the number of parameters and computational complexity, and the EMA feature update strategy dynamically optimizes the appearance state using short-term inter-frame feature changes, saving an average of 6.12% in inference time, improving the real-time performance and adaptability of the system, and is particularly suitable for resource-constrained embedded device scenarios.

[0036] The above specific implementation can be locally adjusted by those skilled in the art in different ways without departing from the principles and purposes of the present invention. The protection scope of the present invention is subject to the claims and is not limited by the above specific implementation, and all implementation solutions within its scope are subject to the present invention.

Claims

1. A method for detecting, identifying and tracking marine benthic organisms, characterized in that: By collecting and screening a benthic dataset of marine organisms, the P2-YOLOv11 deep learning model is used to detect and preliminarily classify the targets after small target data enhancement. The lightweight ShuffleDeepsort tracking algorithm is then used for tracking. Finally, dynamic frame skipping recognition is performed by combining the trajectory information obtained from tracking with the preliminary classification information. A multi-level classification network is used to achieve fine classification under different order-based systems, and multi-level recognition results of marine organisms are obtained.

2. The method for detecting, identifying and tracking marine benthic organisms according to claim 1, characterized in that: The small target data enhancement means that according to the definition of the relative scale of small targets, objects whose size ratio to the original image size is less than 0.3% are considered to be small targets, part of the image is selected as the base image, and the complete small targets that are not blocked are considered. After randomly transforming the whole image and Gaussian blurring the edges, it is randomly inserted into another part of the image and ensures that it does not overlap with any existing targets and there are distance pixels from the image boundary.

3. The method for detecting, identifying and tracking marine benthic organisms according to claim 1, characterized in that: The P2-YOLOv11 deep learning model includes: a backbone network unit, a small target detection head module, a neck unit and a head unit, wherein: the backbone network unit performs multi-level feature extraction processing according to the preprocessed image input information to obtain feature maps of different scales; the neck unit performs feature aggregation and fusion processing according to the feature map information of different scales extracted by the backbone network to obtain an optimized feature map; the head unit performs target detection processing according to the optimized feature map information output by the neck unit to obtain the category, position and confidence of the target; the small target detection head module is used to fuse shallow feature information of more scales to enhance the representation ability of the feature pyramid.

4. The method for detecting, identifying and tracking marine benthic organisms according to claim 1 or 3, characterized in that: The P2-YOLOv11 deep learning model is trained in the following manner: the model is trained using a data set that has been processed with data enhancement, and the model parameters are continuously optimized until a preset number of learning iterations is reached to complete the training process.

5. The method for detecting, identifying and tracking marine benthic organisms according to claim 1, characterized in that: The lightweight ShuffleDeepsort tracking algorithm specifically includes: 1) Initialize the tracker and trajectory: Initialize the tracker and target trajectory parameters according to the input video stream, assign a unique trajectory ID to each detected target, and set the initial state; 2) Extract appearance features: Use ShuffleNetV2 as the feature extraction network to process the detected target box and extract the appearance features of the target; 3) Trajectory prediction: Based on the historical motion information of the target and the current detection results, the Kalman filter algorithm is used to predict the state of each target's trajectory, generate the predicted position and calculate the error range; 4) Re-matching trajectory: After the appearance features are updated, a joint matching strategy of appearance features and motion information is used to associate the detection results of the current frame with the predicted trajectory; 5) Output tracking results: Output the track ID, position, speed and appearance feature status of each target.

6. The method for detecting, identifying and tracking marine benthic organisms according to claim 5, characterized in that: The feature update strategy in step 4) is implemented based on the exponential moving average (EMA), that is, combining the feature information of the current frame and the historical frame, dynamically adjusting the update weight according to the confidence of the detector, so as to dynamically optimize the appearance state of the track target, specifically: in: is the appearance feature of trajectory i up to the t-1th frame, To add appearance features to the matching detection to the model, use Weighted, confidence is the confidence of the detector, and noise is the set threshold for filtering noise.

7. The method for detecting, identifying and tracking marine benthic organisms according to claim 1, characterized in that: The frame skipping recognition means: in order to solve the problem that the refinement classification task is time-consuming under different order element systems, for the continuously associated trajectories and P2-YOLOv11 detection results, the frequency of the refinement classification detection of the subtask is dynamically adjusted, specifically: for all targets with a P2-YOLOv11 detection confidence higher than 0.8, more detailed category information is confirmed every 10 frames; for targets with a detection confidence lower than 0.6, more detailed category information is confirmed every 3 frames; for targets with a confidence between 0.6-0.8, a refinement classification task is performed every 5 frames.

8. The method for detecting, identifying and tracking marine benthic organisms according to claim 7, characterized in that: The refined classification network refers to: according to the preliminary classification information and position frame of P2-YOLOv11, the target area is extracted from the corresponding frame, and the fish and echinoderm classification models trained for marine organisms are selected, so as to complete the accurate classification task at the class level and below, so that the overall method has stronger scalability.

9. A system for detecting, identifying and tracking marine benthic organisms for implementing the method described in any one of claims 1 to 8, characterized in that: include: Data preprocessing unit, detection unit, tracking unit and recognition unit, wherein: the data preprocessing unit performs small target data enhancement on the constructed data set to improve the performance of small target detection of the model; the detection unit uses the P2-YOLOv11 module to predict the data stream, obtains the preliminary classification results, bbox position frame and confidence of each target; the tracking unit uses the lightweight ShuffleDeepsort tracking algorithm to associate the existing trajectory information with the detection information provided by P2-YOLOv11, assigns a unique Track ID and generates the corresponding trajectory; the recognition unit uses a multi-level classification network to perform dynamic frame skipping recognition based on the trajectory information and the detection results of P2-YOLOv11, and obtains specific classification results under different order systems.

Images