Marine organism detection method based on lightweight WCSPOmni-DETR with reinforced fine-grained feature expression branch

Abstract

The application discloses a marine organism detection method based on a lightweight WCSPOmni-DETR with reinforced fine-grained feature expression branches. The steps of the application are as follows: obtaining a data set of marine organisms and preprocessing; using wavelet convolution to improve the backbone network of RT-DETR for lightweight processing; constructing a reinforced fine-grained feature expression branch for RT-DETR; improving the AIFI structure of RT-DETR to Efficient Additive attention AIFI; setting training hyperparameters such as training rounds, batch size, learning rate and input image size; inputting the training set and validation set data into the model for training, obtaining a set of optimal weight parameters; and inputting test data using the optimal weight parameters to obtain the predicted visual results.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of underwater target detection, and particularly relates to a marine organism detection method based on a lightweight WCSPOmni-DETR with a reinforced fine-grained feature expression branch. BACKGROUND

[0002] With the increasing tension of land resources, developing marine resources has become a strategic need for many countries and regions. Underwater images are important carriers and forms of marine information. However, due to the specific physical and chemical properties of the underwater environment, underwater images often exhibit uneven illumination, reduced contrast, color distortion and other degradation phenomena, making it difficult to fully extract the features of marine organisms in the images, especially small-sized marine organisms that gather in groups. The spatial information between them is attenuated with the process of deep convolutional neural network feature extraction. Therefore, conventional target detection models cannot utilize efficient features for target positioning and classification. Various feature enhancement and feature fusion modules have been proposed and applied to the neck network of the model to address this situation. Zhuoyan Liu et al. proposed a plug-and-play underwater joint image enhancement module: UnitModule, Hui Zhou et al. proposed an underwater optical detection network (UODN) based on a you only look once version 8 (YOLOv8) framework, which solves the underwater detection problem through a cross-stage multi-branch (CSMB) module and a large kernel space pyramid (LKSP) module, Xia Hua et al. designed a feature enhancement gating module and an adjacent feature fusion mechanism and a dynamic fusion module to prevent the loss of small target information. These modules either still use several layers of mediocre feature maps extracted by the original backbone network, but with some operations for different level feature interactions, or bring a large increase in parameter quantity, which cannot adapt to the limited hardware resources of the underwater target detection platform.

[0003] Real-Time Detection with Transformer(RT-DETR) has been known for its ability to outperform the You Only Look Once(YOLO) series in terms of detection accuracy and speed since its inception. This model efficiently handles multi-scale features through intra-scale interaction and cross-scale fusion, however, it still faces challenges such as high computational complexity and insufficient performance in processing fine targets. Qianfeng Ge et al. implemented Zoom-cat scale stitching and SSF in the CCFM module, enhancing the network's multi-scale feature fusion capability, and replaced the original Multi-Head Self-Attention(MHSA) with Deformable attention. Xiaoxin Li et al. proposed a space-to-depth transformer module to replace the depth convolution in the original backbone network based on the RT-DETR-L size, which promotes the depth interaction between local and global features, and uses a contextual transformer as an encoder to enhance the model's ability to handle overlapping targets. Hailin Feng et al. combined soft thresholding and Cascaded-Group-Attention(CGA) modules with RT-DETR as the main structure, called SC-RTDETR, and achieved significant performance improvement on the pine wilt disease remote sensing image dataset. Although these improved RT-DETRs have made some progress in their respective fields, they either replace modules with larger parameter sizes or fail to effectively enhance the spatial information in the extracted features, making them unable to cope with underwater image distortion and limited hardware resources on the detection platform. SUMMARY

[0004] The present application aims to address the problems of large algorithm size and lack of precision in existing underwater target detection methods. It proposes a WCSPOmni-DETR lightweight marine organism detection method that uses wavelet convolution and an improved Efficient Additive AIFI for lightweight processing, and then uses a reinforced fine-grained feature expression branch to enhance the extraction of detailed information.

[0005] The present application provides a lightweight WCSPOmni-DETR marine organism detection method based on a reinforced fine-grained feature expression branch, including the following steps:

[0006] Step S1: Obtain the marine organism target detection dataset and perform preprocessing; the marine organism target detection dataset contains picture data of marine organisms;

[0007] Step S2: constructing a WCSPOmni-DETR model, the input of which is picture data of marine organisms, and the output is a predicted value of the type of marine organisms in the picture: obtaining an RT-DETR model and a wavelet convolution module, replacing the second convolution module of the BasicBlock module in the backbone network of the RT-DETR model with the wavelet convolution module to obtain an improved backbone extraction network; obtaining an Omni-Kernel module, which is a full kernel module composed of three branches of global branch, large branch and local branch; using a Cross Stage Partial Network architecture to improve the Omni-Kernel module, performing convolution operation on the local part of the input feature map with the Omni-Kernel module, and directly splicing the rest with the convolution result to obtain a CSPOmniKernel module; using the CSPOmniKernel module and the SPDConv module to construct a strengthened fine-grained feature expression branch of the WCSPOmni-DETR model, including: leading out the s2 shallow feature map in the backbone network of the RT-DETR model, performing convolution after SPDConv, and performing global-to-local feature fusion in the CSPOmniKernel module with deep features to strengthen the expression of fine-grained features; obtaining an Efficient Additive Attention module, using it to improve the AIFI module structure of the RT-DETR model to obtain the WCSPOmni-DETR model;

[0008] Step S3: training the WCSPOmni-DETR model using a marine organism target detection data set to obtain a trained WCSPOmni-DETR model;

[0009] Step S4: obtaining a to-be-tested picture, inputting the trained WCSPOmni-DETR model to obtain a prediction result.

[0010] Preferably, the Efficient Additive Attention module is obtained to improve the AIFI module structure of the RT-DETR, specifically including: replacing the original MHSA module of the AIFI module with the Efficient Additive Attention module.

[0011] Preferably, the size of the RT-DETR model used in step S2 is RT-DETR-Resnet18.

[0012] Preferably, in step S2, when the Omni-Kernel module is lightweighted using the idea of ​​Cross Stage Partial Network, the proportion of convolution operations performed on the feature map is 1 / 4.

[0013] Preferably, step S3 further includes the following hyperparameter settings for the WCSPOmni-DETR model: training epochs set to 160, batch size set to 16, optimizer used as AdamW, and learning rate set to 0.0001.

[0014] Preferably, the marine organisms mentioned in step S1 include starfish, coral, sea urchin, sea cucumber, and scallop.

[0015] The beneficial effects of this invention are:

[0016] This invention addresses the specific needs of underwater target detection scenarios by making a series of improvements to the RT-DETR-Resnet18 model as a benchmark. Leveraging its advantages in real-time performance, accuracy, multi-scale detection, and multi-class detection, while simultaneously addressing its limitations in computational complexity and large parameter count, this invention proposes an improved real-time underwater target detection algorithm based on wavelet convolution and the CSPOmniKernel module, termed WCSPOmni-DETR. To enhance the feature representation of fine-grained information in marine biological images, an additional shallow feature processing branch is introduced, and the processed S2 feature layer and deep information are comprehensively represented in the proposed CSPOmniKernel module. Furthermore, to better adapt the algorithm to the hardware resources of underwater platforms, wavelet convolution is introduced into the BasicBlock of the backbone network, reducing the number of parameters while expanding the receptive field. Finally, the encoder of WCSPOmni-DETR employs an efficient additive attention structure to generate faster underwater inference speed and more robust contextual representation. Experiments have shown that the proposed detection algorithm reduces the number of parameters by 25.5% compared to the baseline model. On the URPC2021 dataset, it improves the map50 metric by 0.82% compared to the baseline. The model's size and accuracy are comparable to the current state-of-the-art real-time object detection models, such as the YOLO series. Attached Figure Description

[0017] Figure 1 This is a network structure diagram of the WCSPOmni-DETR model proposed in this invention;

[0018] Figure 2 This is a diagram of the wavelet convolution structure;

[0019] Figure 3 This is a module structure diagram of the main improvement;

[0020] Figure 4 is a network structure diagram of the RT-DETR model;

[0021] Figure 5 is a structure diagram of the SPDConv convolution module;

[0022] Figure 6 is a structure diagram of the CSPOmniKernel module proposed in the application;

[0023] Figure 7 is a structure diagram of the attention module;

[0024] Figure 8 is a curve diagram of the training process;

[0025] Figure 9 is a comparison of the confusion matrices before and after improvement;

[0026] Figure 10 is a comparison of the heat maps before and after improvement of the model;

[0027] Figure 11 is a comparison of the detection results with the YOLO model; DETAILED DESCRIPTION

[0028] The application will be further described below in conjunction with the accompanying drawings.

[0029] As shown in Figure 1 , a marine organism detection method based on a lightweight WCSPOmni-DETR with reinforced fine-grained feature expression branch is provided. The method is an underwater real-time target detection method based on wavelet convolution and the CSPOmniKernel module proposed. The backbone network of the RT-DETR is improved by using cascaded wavelet convolution for lightweight processing, and the encoder adopts Efficient Additive Attention to enhance the real-time inference performance. In addition, the additional S2 feature layer in the backbone network is processed by the SPDConv unit to enhance the fine-grained information expression, and then the global-to-local feature representation is performed in the CSPOmniKernel module and the deep features containing rich semantic information.

[0030] The application comprises the following steps:

[0031] Step 1: Prepare the data set and perform preprocessing

[0032] The URPC2021 dataset was released in the 2021 National Underwater Robot Competition, which contains four categories of sea cucumbers, spiny fish, scallops and starfish, and a total of 7600 images. We randomly divided it into 6080 for training and 1520 for testing model performance. The size of the images in the dataset is mostly 3840x2160 and 720x405. The image size input into the model is 640x640.

[0033] Step 2: Improve the backbone network of RT-DETR using wavelet convolution for lightweight processing

[0034] (2.1) Wavelet convolution

[0035] Shahaf E.Finder uses wavelet transform (WT) to effectively increase the receptive field of convolution without being over-parameterized, thereby proposing wavelet convolution (WTConv), and by cascading, it can also guide the CNN to better respond to low frequencies. WTConv can effectively replace deep convolution (DwConv) to build a lightweight backbone network, while expanding the receptive field and improving the ability to capture fine-grained information in images. Multi-level wavelet convolution is as shown in Figure 2 .

[0036] Two-dimensional Haar wavelet convolution combines operations in two dimensions, using a low-pass filter (f LL ) and three high-pass filters (f LH , f HL , f HH ) as shown in the following formula as the filter of depthwise convolution with a stride of 2.

[0037]

[0038] Then the wavelet transform and inverse wavelet transform in two-dimensional Haar wavelet convolution are as shown in the following two formulas.

[0039] [X LL , X LH , X HL , X HH ] = Conv ([f LL , f LH , f HL , f HH ], X)

[0040] X = Conv-transposed ([f LL , f LH , f HL , f HH ], [X LL , X LH , X HL , XHH ])

[0041] All convolutions in wavelet convolution are depthwise convolutions, and the kernel size is 5x5.

[0042] (2.2) Lightweight backbone

[0043] Considering the real-time requirement of underwater target detection scenarios, the smallest RT-DETR-resnet18 is used as the baseline model, and its backbone network resnet18 is stacked by many standard convolution units and residual edges. As shown in Figure 3 , each BasicBlock still has a large number of parameters (11.17M) and computational consumption. We replace the second standard convolution unit of BasicBlock with cascaded wavelet convolution, as shown in Figure 3 , which effectively reduces the size of the model backbone (5.41M). The first convolution unit still uses standard convolution, which increases the number of channels while reducing the feature map.

[0044] Each level of cascaded wavelet convolution increases the size of the receptive field, while the number of trainable parameters only increases a little. Each convolution focuses on 4 different frequency bands in an increasingly large receptive field, and emphasizes low frequencies in the input, making up for the lack of fine-grained information in deep convolutional networks, forming a feature extraction network that balances speed and performance. Experiments have proved that the model performance is best when the number of cascaded wavelet convolution is 2, and the calculation formula of two-level wavelet convolution is as follows.

[0045]

[0046] where i = 1, 2, is the original input, represents 3 high-frequency components in the i-th level of wavelet convolution, then represents 1 low-frequency component, and DwConv is a depthwise convolution. (i) is the output of the i-th level after DwConv and inverse wavelet transform, and the final output needs to be added to the result of the previous layer after DwConv. The final efficient lightweight backbone network is shown in the following table.

[0047] Lightweight backbone network structure table

[0048]

[0049]

[0050] Step 3: Construct a reinforced fine-grained feature expression branch for RT-DETR

[0051] The conventional deep convolutional feature extraction backbone gradually fades the fine-grained spatial information in the feature map as the standard convolution process proceeds. Although the various types of up and down sampling feature fusion neck network often make a great remedy based on 3-level multi-scale features, there is still room for improvement in the face of marine biology detection of small target aggregation and fuzzy boundary. Instead of optimizing the backbone in this regard, we introduce an S2 feature layer after SPDConv in the proposed CSPOmniKernel module and conduct global-to-local feature fusion with deep features containing rich semantic information.

[0052] (3.1) RT-DETR model

[0053] The main feature of the RT-DETR model is the use of an efficient hybrid encoder based on attention-based intra-feature interaction (AIFI) and CNN-based cross-scale feature fusion (CCFF). It fully combines the ability of convolutional computation to capture local information and the ability of Transformer to build complex long-distance dependencies. The structure of the RT-DETR model is shown in Figure 4

[0054] (3.2) SPDConv

[0055] Marine biology target detection often encounters small groups of small organisms that are mutually occluded and have fuzzy boundaries, which is a major difficulty in the field of marine biology target detection. Standard convolution pursues the expansion of the receptive field of features and fades the fine-grained spatial information during execution. Sunkara and Luo designed a spatial-to-depth convolution (SPDConv) to solve this problem, as shown in Figure 5 After converting the spatial information to the channel dimension and then performing non-strided convolution, the feature information is fully preserved.

[0056] (3.3) Fine-grained feature expression branch

[0057] ​The conventional deep convolutional feature extraction backbone gradually fades the fine-grained spatial information in the feature map as the standard convolution process proceeds. Although the various types of up and down sampling feature fusion neck networks make a great remedy based on the 3-level multi-scale features, there is still room for improvement in the face of marine biology detection of small target aggregation and fuzzy boundary. Instead of optimizing the backbone in this regard, we introduce the S2 feature layer after SPDConv into the proposed CSPOmniKernel module and fuse it with the deep features containing rich semantic information. The fine-grained spatial information in the shallow layer is fully preserved in the SPDConv unit, so that the features fused by the CSPOmniKernel module can more fully express the features of the aggregated organisms and fuzzy boundaries in the image, and can effectively detect details.

[0058] Omni-Kernel Module is a full kernel module composed of global branch, large branch and local branch to effectively learn global to local feature representation. The pre and post processing of the module are both using 1x1 convolution. In the experimental scene of this paper, there is a certain redundancy calculation in the full Omni-Kernel Module, and the model size increased by introducing the module exceeds the demand of real-time target detection model. Therefore, we combine the idea of Cross Stage Partial Network(CSP) to construct the global to local features for part of the input feature map, and the rest still retains its original characteristics, which is processed by the down sampling feature fusion process of the CCFF structure in RT-DETR. The structure of CSPOmniKernel module is shown in Figure 6

[0059] The large branch mainly uses the cheap depth convolution with kernel size KxK to pursue a large receptive field, K=31 in this paper. The local branch is 1x1 depth convolution, and the global branch increases the global modeling function through dual domain processing, including dual domain channel attention module(DCAM) and frequency-based spatial attention module(FSAM), whose calculation formulas are as follows.

[0060]

[0061] Where x is the feature map before input to the branch, F and IF are the fast Fourier transform and its inverse operation respectively, and AvgPool is the global average pooling layer. The derivation process of the proposed CSPOmniKernel module is shown in

[0062] Branch, Identity = Split channel (e, x)

[0063] X​Large = DwConv 31×1 (Branch) + DwConv 31×31 (Branch) + DwConv 1×31 (Branch)

[0064] X out = Concat(DwConv 1×1 (Branch) + X Large + X FSAM (Branch), Identity)

[0065] Where e is the proportion of the original feature along the channel, e = 0.25 in this paper, at this time the parameter quantity and the calculation quantity reduced relative to the original OmniKernel module are 1.14M and 9.9G respectively. Branch is the feature that is segmented out for complex processing, which accounts for 1 / 4 of the original feature. DwConv is deep convolution, X FSAM (Branch) is to replace the input x in the continuous derivation process composed of the above formula (1) (2) (3) with the result of Branch.

[0066] Step 4: Improve the AIFI structure of RT-DETR

[0067] The AIFI module of the benchmark RT-DETR adopts the MHSA mechanism, which calculates the self-attention of the last feature map of the multi-scale feature extracted by the backbone. Although the smallest high semantic information feature map is used, the attention calculation is still a time-consuming operation due to the quadratic complexity with the number of tokens. The hardware resources of the underwater detection platform are limited, so an efficient calculation method is needed. Here we replace MHSA with Efficient Additive attention to form Efficient Additive Intra-Scale Feature Interaction, which effectively reduces the inference time while the generated attention feature map has a more robust context representation.

[0068] (4.1) Efficient Additive attention

[0069] VisionTransformer models are all centered on self-attention mechanisms, such as Figure 7The input is projected into query (Q), key (K) and value (V) matrices by linear transformation, and then the long-range dependencies of features in the input feature map are constructed by computing the attention scores between Q and K tokens and interacting with V. N is the number of tokens, and d is the hidden dimension. With the increase of the number of tokens, the calculation and memory requirements of Q·K increase quadratically, resulting in slow inference speed and high memory occupation, and the computational complexity is O (N 2 d). The calculation formula is:

[0070]

[0071] To solve the above problems, Abdelrahman Shaker et al. introduced Efficient Additive Attention, as shown in Figure 7 Experiments show that the key-value interaction can be replaced by a linear layer without sacrificing any accuracy. This mechanism effectively replaces the quadratic matrix multiplication operation with linear element multiplication, which has a linear complexity relationship with the number of tokens and produces a more robust context representation. The calculation process is as follows:

[0072]

[0073] The query matrix Q∈R N×d is multiplied by a learnable parameter vector w∈R d to obtain the attention weight vector α∈R N of the query.

[0074]

[0075] Then each value of the attention weight vector is multiplied by each column in the query matrix to obtain a single global query vector q∈R d .

[0076]

[0077] Finally, the normalized query matrix is added to the key matrix broadcasted element-wise multiplied by q to obtain the output attention feature map.

[0078] Step 5: Set training hyperparameters

[0079] Our experiments were all completed on a server equipped with a Linux operating system, with an operating system version of Ubuntu 20.04, a CPU configuration of 16vCPU Intel(R) Xeon(R) Platinum 8481C, a GPU configuration of GeForce RTX 4090D GPU, and a Pytorch version of 2.0.0. The specific hyperparameter settings of the experiment are shown in the following table.

[0080] Experimental hyperparameter settings

[0081]

[0082] Step 5: Input the dataset into the model for training

[0083] According to steps 2, 3, and 4, the WCSPOmni-DETR marine organism detection model proposed in the present application is obtained. In step 1, the training dataset is processed, and in step 5, the training hyperparameters are set. Next, the training set and validation set need to be input into the model for training. After 160 rounds of forward derivation, loss value calculation, back propagation, gradient update, and index calculation on the validation set, a set of optimal weight parameters is obtained, which is saved for obtaining the prediction result.

[0084] Figure 8 The change trend of each single index in the training process is shown. After 160 epochs of training, the forward indicators (precision, recall, mAP50, mAP50:95) of the model have converged on both datasets, and the regression box loss (giou loss), classification loss (cls loss), and l1 loss have also converged on the validation set.

[0085] Step 6: Input test data to obtain prediction results

[0086] Using the set of optimal weight parameters obtained in step 5, the new marine organism pictures in the test set are forward-derived to obtain the prediction results and test the final performance of the model. Figure 9 Comparison of confusion matrices of improved and unimproved models. There are a total of 5 classes, namely holothurian, echinus, scallops, starfish, and a background class background. The values on the main diagonal represent the correct classification rate of the model, among which the detection accuracy of holothurian and scallops has improved. The values on the non-main diagonal represent the rate of misclassification of the model as one class into another class. We can see that the main error classification is as background, and the error classification between classes is small, about 0.01. It can be seen that the confusion class possibility of the optimized model is low, and the main error classification is still background.

[0087] In order to verify the enhancement effect of the proposed improved module on the model, we combined different improvement points to establish 7 groups of burning experiments. The experimental results are shown in the following table.

[0088] Ablation experiments on URPC 2021

[0089]

[0090] (1) Precision: The proportion of correct positive samples among all instances identified as positive samples.

[0091]

[0092] (2) Recall: The proportion of correct positive samples among all positive samples.

[0093]

[0094] (3) Mean Average Precision (mAP): A comprehensive measure of the detection performance of the target detection algorithm.

[0095]

[0096] where TP (True Positives) represents the number of positive samples correctly identified as positive samples, FP (False Positives) represents the number of negative samples incorrectly identified as positive samples, FN (False Negatives) represents the number of positive samples incorrectly identified as negative samples, TN (True Negatives) represents the number of negative samples correctly identified as negative samples, and Q R is the number of categories. Among them, the model with the smallest parameter and computational complexity only uses wavelet convolution to improve the backbone, but the performance is relatively poor and still needs to be improved. The model with the best performance is the one that adds a global-to-local feature fusion module branch to the benchmark model and improves the AIFI. However, the computational complexity increases by 14.5% compared to the benchmark model. The proposed WCSPOmni-DETR model achieves a balance between parameter and computational complexity, reducing by 25.5% and 10.2% respectively compared to the benchmark model, and the mAP0.5 index is improved by 0.82 percentage points on the URPC2021 dataset. In this group of burning experiments, the balance between model size and accuracy is achieved. Figure 10 The detection heat map comparison of the benchmark model and WCSPOmni-DETR on 5 test images is shown. From the distribution of significant regions in the heat map, WCSPOmni-DETR detects the clustered targets more subtly, with higher heat values focusing more accurately on the key parts of the organisms, and there is a clear heat value trough between the clustered organisms. In the last row of the detection of a single sea cucumber, it can be seen directly that the proposed model is affected by the additional fine-grained feature branch, focusing more on the detailed feature parts of the organism, and through the multiple spine structures of the sea cucumber, the entire target is judged.

[0097] To verify the superiority of the proposed WCSPOmni-DETR model performance, I compare it with some of the most advanced general object detection models on different datasets, which are mainly divided into three categories: 1) Two-stage models: mainly R-CNN series, among which the relatively new and strong Faster R-CNN is selected. 2) Single-stage models: typical YOLO series, select Yolov5, Yolov8, Yolov9 and Yolov10, as well as other single-stage models including CenterNet and TOOD. 3) Transformer-based models: some models developed based on DETR including Deformable DETR, DAB-DETR, DINO, Lite-DETR and RT-DETR. The comparison models exist in multiple sizes, and the similar or smaller sizes are selected. All training does not use pre-training weights, and all start from zero until the model converges. The comparison results on the URPC2021 dataset are shown in the following table. The black and gray bold respectively represent the optimal and sub-optimal in all comparison models.

[0098] Comparison results of different models on URPC2021 dataset

[0099]

[0100]

[0101] The proposed WCSPOmni-DETR detection method achieves the best or sub-optimal performance in various indicators, proving that our model has high detection accuracy for marine organisms while effectively controlling the size of the model, achieving a balance between model accuracy and size, and can be applied to scenarios where the hardware resources of the underwater target detection platform are limited. Compared with other models, the DAB-DETR with less computational complexity has a larger gap in detection performance on the URPC2021 dataset compared with our WCSPOmni-DETR model, while Lite-DETR is the closest to WCSPOmni-DETR in performance among the Transformer-based models, but its parameter quantity and computational complexity are 2 times or more than WCSPOmni-DETR. Overall, the YOLO series models are still close to the size and performance of our model. The latest YOLOv10m has the optimal parameter quantity and computational complexity among the same level YOLOs, but its performance is relatively poor. The performance of YOLOv9m is closest to our model, with Recall and mAP0.5:0.95 indicators exceeding WCSPOmni-DETR by 0.19 points and 0.56 points, respectively, while Precision and mAP0.5 lag behind WCSPOmni-DETR by 0.29 and 0.23 percentage points, respectively. Although the detection performance is not superior to YOLOv9m, the size of our model has a great advantage over YOLOv9m, with a parameter quantity and computational complexity that is 5.17M and 25.3GFloats less than YOLOv9m, respectively. In summary, the proposed WCSPOmni-DETR model achieves the best performance among the above-mentioned comparison models and is more suitable for marine organism detection.

[0102] The two models YOLOv8m and YOLOv9m with the best performance among the above-mentioned comparison models and WCSPOmni-DETR were compared in actual marine organism detection, and the detection results are as follows Figure 11The black boxes in the leftmost column of the original image represent target instances, the model names beside them are models that do not detect this instance, and the white boxes and model names beside them represent that the model detects the background as a target instance. Several typical marine biological images are selected, including the following types: 1) the boundary between the organism and the seabed environment is blurred and difficult to distinguish, 2) the same or different organisms are densely grouped and mutually occluded, 3) the lens distance is far and the organism size is small, and 4) a single large volume organism. In figure (a), only three sea cucumbers with blurred boundaries are marked with black boxes, YOLOv8 has errors in detecting the background as a target, YOLOv9 and WCSPOmni-DETR correctly detect the three sea cucumbers, but the confidence of our model for the three sea cucumbers is higher than that of YOLOv9. In figure (b), there are different types of biological aggregation, which poses a certain challenge to these detectors, YOLOv8 and YOLOv9 fail to detect the sea cucumber with gradually fading color in the lower left corner, and for the sea cucumber in the middle of the aggregation, our model also has a missed detection, YOLOv8 and YOLOv9 also fail to completely distinguish the two partially overlapping targets. Figure (c) mainly shows the aggregation of the same type of organisms, the target size is generally moderate, except for YOLOv8, which mistakenly detects the background in the upper left corner as a target instance, the performance of other detectors is generally good. In figure (d), there are individual small targets, our model shows good detection performance in this case, correctly detecting all target instances in the image, especially the small sea urchin on the right, which is missed by YOLOv8 and YOLOv9. Figure (e) is a single large target, YOLOv8 fails to completely label the single large target, YOLOv9 has a slight positioning deviation, and only our model accurately identifies it. From the detection of the above several representative marine biological images, it can be seen that the proposed WCSPOmni-DETR has relatively strong control over details, which is due to the introduction of the global-to-local feature fusion branch based on S2 level feature maps. It effectively retains the spatial information of the shallow layer, and after feature fusion through downsampling, the target detection head of RT-DETR can use more efficient features to position the target regression box.

[0103] A marine organism detection model based on lightweight RT-DETR with reinforced fine-grained feature expression branch is proposed in this paper, which can adapt to the hardware conditions of underwater target detection platform and has high marine organism detection performance. The model uses a lightweight backbone improved by two-level wavelet convolution as the feature extraction backbone network, which reduces the parameter amount of the original backbone by 51.7%. Secondly, the S2 shallow features are converted to channel dimension after SPDConv, and then the shallow and deep features are fused in the proposed CSPOmniKernel module, which enhances the fine-grained features in the fused features and effectively deals with the situation of small marine organisms in groups and the blurred boundary between organisms and environment caused by reduced contrast and color distortion in underwater scenes. Finally, the improved Efficient Additive AIFI can further increase the inference efficiency of the model and the generated attention feature map has a more robust context representation, which to some extent improves the ability of MHSA to construct global feature representation. Through experiments, the effectiveness of the proposed model is proved by various indicators and instance detection. The parameter amount and computational complexity of the model are effectively reduced compared with the baseline model(25.5%, 10.2%), while the detection accuracy of the model is also improved. Although WCSPOmni-DETR has been improved, it still needs to pursue accurate detection when facing small organisms in groups and with blurred boundaries.

Claims

1. A marine organism detection method based on a lightweight WCSPOmni-DETR with reinforced fine-grained feature expression branches, characterized in that: The method comprises the following steps: Step S1: Obtain a marine organism target detection data set and perform preprocessing; the marine organism target detection data set comprises picture data of marine organisms; Step S2: Construct a WCSPOmni-DETR model, wherein the input of the WCSPOmni-DETR model is the picture data of marine organisms, and the output is a predicted value of the type of marine organisms in the picture: Obtain an RT-DETR model and a wavelet convolution module, replace the second convolution module of the BasicBlock module in the backbone network of the RT-DETR model with the wavelet convolution module, and obtain an improved backbone extraction network; Obtain an Omni-Kernel module, which is a full kernel module composed of a global branch, a large branch and a local branch; Lightweight improve the Omni-Kernel module by using a Cross Stage Partial Network architecture, perform convolution operation on the local part of the input feature map by using the Omni-Kernel module, and directly splice the result of the convolution operation with the remaining part, to obtain a CSPOmniKernel module; Construct a strengthened fine-grained feature expression branch of the WCSPOmni-DETR model by using the CSPOmniKernel module and the SPDConv module, including: extracting the s2 shallow feature map in the backbone network of the RT-DETR model, performing convolution by using the SPDConv, and performing global-to-local feature fusion in the CSPOmniKernel module with the deep feature, to strengthen the expression of fine-grained features; Obtain an Efficient Additive Attention module, improve the AIFI module structure of the RT-DETR model by using the Efficient Additive Attention module, and obtain the WCSPOmni-DETR model; Step S3: Train the WCSPOmni-DETR model by using the marine organism target detection data set, and obtain a trained WCSPOmni-DETR model; Step S4: Obtain a to-be-detected picture, input the trained WCSPOmni-DETR model, and obtain a prediction result.

2. The marine organism detection method of the lightweight WCSPOmni-DETR based on a strengthened fine-grained feature expression branch according to claim 1, wherein: The Efficient Additive Attention module is obtained, and the AIFI module structure of the RT-DETR is improved by using the Efficient Additive Attention module, specifically including: replacing the original MHSA module of the AIFI module with the Efficient Additive Attention module.

3. The marine organism detection method based on the lightweight WCSPOmni-DETR with reinforced fine-grained feature expression branch according to claim 1, wherein: The size of the RT-DETR model used in step S2 is RT-DETR-Resnet18.

4. The marine organism detection method based on the lightweight WCSPOmni-DETR reinforced by fine-grained feature expression branches according to claim 1, wherein: When the Omni-Kernel module is lightweight improved by using the Cross Stage Partial Network, the proportion of the convolution operation on the feature map is 1 / 4.

5. The marine organism detection method based on the lightweight WCSPOmni-DETR reinforced by fine-grained feature expression branches according to claim 1, wherein: The step S3 further includes the following hyperparameter settings for the WCSPOmni-DETR model: the training round is set to 160, the batch size is set to 16, the optimizer is AdamW, and the learning rate is set to 0.0001.

6. The marine organism detection method of claim 1, wherein the WCSPOmni-DETR is a lightweight WCSPOmni-DETR based on a reinforced fine-grained feature expression branch. characterized in that The marine organisms in the step S1 include starfish, corals, sea urchins, sea cucumbers, and scallops.

Images