A method, device, equipment and storage medium for target detection of underwater fish

Abstract

The present application relates to a method, device, equipment and storage medium for underwater fish target detection. This method uses YOLOv8 as the baseline model and improves the backbone network of YOLOv8 to construct a target detection model, that is, multiple groups of cascaded first modules and second modules are set between the Conv layer and the SPPF layer. The input features are downsampled by the first module and then input into the second module. The second module retains a rich gradient flow and also adds deformable convolutions with a multi-scale channel attention mechanism. The deformable convolutions with the multi-scale channel attention mechanism enhance the feature extraction ability on the one hand and enable the deformable convolutions to have a larger and more accurate receptive field on the other hand, so that the target detection model can overcome the influence of the underwater environment and improve the accuracy of the recognized underwater fish.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular, to a method, device, equipment and storage medium for target detection of underwater fish. Background Art

[0002] In large rivers, reservoirs and the aquaculture industry, it is essential to monitor and analyze the growth status and behavior of fish, as well as to identify and count the species of fish.

[0003] With the development of machine vision and deep learning technology, the target detection technology based on deep learning has been widely applied in fish statistics, and even in the field of aquatic ecology, it provides a faster, cheaper and more accurate alternative to the manual data analysis method for monitoring and evaluating animal abundance.

[0004] The task of target detection is to find all the targets of interest in an image, so as to determine their categories and locations. Due to the complexity of the underwater environment, there are still many gaps in current research. The main reason is that existing general target detection models rarely pay attention to the underwater situation, because the relevant convolutional geometric structures in the convolutional modules used to construct convolutional neural networks are fixed, and their ability to model geometric transformations is essentially limited. This also leads to poor performance of target detection models using ordinary convolutions when detecting objects with rich geometric transformations, such as fish swimming underwater. Moreover, due to the complexity of the underwater situation, such as turbid water quality, light refraction in water, dark environment, geometric deformation, etc., these all result in limited performance of general target detection models underwater. Summary of the Invention

[0005] The following is an overview of the subject matter described in detail in this article. This overview is not intended to limit the scope of protection of the claims.

[0006] The main purpose of the embodiments of the present invention is to propose a method, device, equipment and storage medium for target detection of underwater fish, which can enable the target detection model to overcome the influence of the underwater environment and improve the accuracy of the detected underwater fish.

[0007] To achieve the above object, a first aspect of the embodiments of the present invention provides a method for target detection of underwater fish, and the method for target detection of underwater fish includes:

[0008] Obtain an underwater image;

[0009] Input the underwater image into the target detection model to obtain the fish detection result output by the target detection model. The target detection model is a model obtained by improving the backbone network part of the benchmark model based on YOLOv8. The backbone network part of the target detection model includes a Conv layer, an SPPF layer, and multiple sets of cascaded first modules and second modules arranged between the Conv layer and the SPPF layer, and there is a corresponding second module connected to the Neck part of the target detection model. The first module is used to perform downsampling on the input features of the first module. The second module is a module obtained by adding deformable convolution with multi-scale channel attention to the Bottleneck layer in the benchmark module based on the C2f module of the backbone network part of YOLOv8.

[0010] In some embodiments, the deformable convolution in the deformable convolution with multi-scale channel attention is DCNv2.

[0011] In some embodiments, the deformable convolution with multi-scale channel attention sets multi-scale channel attention during the generation of the offset in the DCNv2 convolution.

[0012] In some embodiments, the multi-scale channel attention processes features in the following manner:

[0013] Use CA attention to decompose and pool the input features of the multi-scale channel attention into features in two spatial directions;

[0014] Convolve the features after CA attention convolves the input features of the multi-scale channel attention without decomposition to obtain a first convolution result, and separate the weights in two spatial directions from the first convolution result through an activation function. Fuse the two spatial direction weights with the corresponding features in the two spatial directions to obtain a first fusion result;

[0015] Perform global pooling and convolution on the input features of the multi-scale channel attention to obtain a second convolution result. Multiply the second convolution result by the mean of the first convolution result to obtain a multiplication result. Finally, fuse the multiplication result with the first fusion result to obtain the output features of the multi-scale attention.

[0016] In some embodiments, the target detection model is a model obtained by improving the backbone network part and the Neck part of the benchmark model based on YOLOv8. Among them, the improvement of the Neck part of the benchmark model includes: replacing the downsampling operation in the Neck part of the benchmark model with the first module.

[0017] In some embodiments, the first module is further configured to extract local features and surrounding context features from the downsampled features, combine the local features and the surrounding context features to form a joint feature, and improve the joint feature through global context features.

[0018] In some embodiments, extracting local features and surrounding context features from the downsampled features, combining the local features and the surrounding context features to form a joint feature, and improving the joint feature through global context features includes:

[0019] Extract local features from the downsampled features through a 3×3 convolution;

[0020] Extract surrounding context features from the downsampled features through a 3×3 dilated convolution;

[0021] Concatenate the local features and the surrounding context features to obtain a joint feature;

[0022] Input the joint feature into a combined module including a global pooling layer and two fully connected layers to obtain a weight vector output by the combined module;

[0023] Combine the weight vector with the joint feature as the output feature of the first module.

[0024] To achieve the above object, a second aspect of the embodiments of the present invention provides an underwater fish target detection device, and the underwater fish target detection device includes:

[0025] An image acquisition unit for acquiring an underwater image;

[0026] A fish detection unit for inputting the underwater image into a target detection model to obtain a fish detection result output by the target detection model; wherein the target detection model is a model obtained by improving the backbone network part of the benchmark model based on YOLOv8, and the backbone network part of the target detection model includes a Conv layer, an SPPF layer, and multiple groups of cascaded first modules and second modules arranged between the Conv layer and the SPPF layer, and there is a corresponding second module connected to the Neck part of the target detection model; the first module is used for downsampling the input features of the first module, and the second module is a module obtained by adding deformable convolutions with multi-scale channel attention in the Bottleneck layer of the benchmark module based on the C2f module of the backbone network part of YOLOv8.

[0027] To achieve the above object, an electronic device provided in the third aspect of the embodiments of the present invention includes: at least one control processor and a memory communicatively connected to the at least one control processor; the memory stores instructions executable by the at least one control processor, and when the instructions are executed by the at least one control processor, the at least one control processor is enabled to execute a target detection method for underwater fish in the first aspect above.

[0028] To achieve the above object, a fourth aspect of the embodiments of the present invention provides a computer-readable storage medium storing computer-executable instructions for causing a computer to execute a target detection method for underwater fish in the first aspect above.

[0029] An embodiment of the present application provides a target detection method for underwater fish. In this embodiment, YOLOv8 is used as a benchmark model, and the backbone network of YOLOv8 is improved to construct a target detection model. That is, multiple groups of cascaded first modules and second modules are provided between the Conv layer and the SPPF layer. The input features are downsampled by the first module and then input into the second module. The second module retains a rich gradient flow, and a deformable convolution with a multi-scale channel attention mechanism is also added. The deformable convolution with the multi-scale channel attention mechanism enhances the feature extraction ability on the one hand and enables the deformable convolution to have a larger and more accurate receptive field on the other hand. As a result, the target detection model can overcome the influence of the underwater environment and improve the accuracy of the identified underwater fish.

[0030] It can be understood that the beneficial effects of the above second aspect to the fourth aspect compared with the related art are the same as those of the first aspect compared with the related art. For details, please refer to the relevant descriptions in the first aspect and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the related art descriptions. Obviously, the drawings in the following descriptions are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0032] Figure 1 is a flowchart of a target detection method for underwater fish provided by an embodiment of the present application;

[0033] Figure 2 is a schematic structural diagram of the backbone network part of a target detection model provided by an embodiment of the present application;

[0034] Figure 3 It is a schematic structural diagram of the C2f module provided by an embodiment of the present application;

[0035] Figure 4 It is a schematic structural diagram of the Bottleneck layer in the C2f module provided by an embodiment of the present application;

[0036] Figure 5 It is a schematic structural diagram of DCNv2 provided by an embodiment of the present application;

[0037] Figure 6 It is a schematic diagram of the working process of the multi-scale channel attention provided by an embodiment of the present application;

[0038] Figure 7 It is a processing block diagram of the CA attention for features provided by an embodiment of the present application;

[0039] Figure 8 It is a processing block diagram of DCNv2 with a multi-scale channel attention mechanism added for features provided by an embodiment of the present application;

[0040] Figure 9 It is a schematic structural diagram of DCNv2 with a multi-scale channel attention mechanism added provided by an embodiment of the present application;

[0041] Figure 10 It is a schematic structural diagram of the second module provided by an embodiment of the present application;

[0042] Figure 11 It is a schematic diagram of the processing flow of the first module for features provided by an embodiment of the present application;

[0043] Figure 12 It is a processing block diagram of the first module for features provided by an embodiment of the present application;

[0044] Figure 13 It is a schematic structural diagram of the object detection model provided by an embodiment of the present application;

[0045] Figure 14 It is a schematic diagram of the experimental results provided by an embodiment of the present application;

[0046] Figure 15 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0047] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are for the purpose of describing embodiments of this application only and are not intended to limit this application.

[0049] Referring to Figure 1 , an embodiment of this application provides a method for detecting underwater fish targets. The method includes the following steps S110 - S120:

[0050] Step S110: Obtain an underwater image.

[0051] Step S120: Input the underwater image into the target detection model to obtain the fish detection result output by the target detection model; where the target detection model is a model obtained by improving the backbone network part of the benchmark model based on YOLOv8. The backbone network part of the target detection model includes a Conv layer, an SPPF layer, and multiple groups of cascaded first modules and second modules set between the Conv layer and the SPPF layer, and there is a corresponding second module connected to the Neck part of the target detection model; the first module is used to perform downsampling on the input features of the first module, and the second module is a module obtained by taking the C2f module in the backbone network part of YOLOv8 as the benchmark module and adding deformable convolution with multi-scale channel attention in the Bottleneck layer of the benchmark module.

[0052] In step S110, the underwater image can be captured by an underwater camera.

[0053] In step S120, in this embodiment, YOLOv8 is used as the benchmark model, and the modules between the Conv layer and the SPPF layer in the backbone network part of YOLOv8 are set as the first module and the second module. Referring to Figure 2 ( Figure 2 For a part of the subsequent Figure 13 structure), specifically, the original C2f module is replaced with the second module, where the arrangement and combination of the first module and the second module are the same as the arrangement and combination of the original downsampling operation and C2f. From top to bottom, the second second module and the third second module are connected to the Neck part of the target detection model (the Neck part Figure 2 is not shown). It should be noted that since the YOLOv8 model is well-known common sense in this field, its structure will not be described in detail here. In the target detection model obtained after improving YOLOv8, the first module is used to perform downsampling on its input features. It should be noted that in the subsequent description of the embodiments, compared with the downsampling operation, the first module can not only perform the downsampling operation but also integrate information in a timely manner after the downsampling operation, which will not be described in detail here.

[0054] In this implementation, the second module is mainly introduced. The second module uses the C2f module in the backbone network of YOLOv8 as the reference module, referring to Figure 3 and Figure 4 , Figure 3 is a structural schematic diagram of the C2f module (the C2f module of the backbone network part shown in the figure), Figure 4 is Figure 3 the Bottleneck layer structure in the shown structure. In this embodiment, deformable convolution with multi-scale channel attention is added to the Bottleneck layer, and the ordinary convolution of the original Bottleneck layer is replaced with deformable convolution with multi-scale channel attention (it should be noted that here it is not just deformable convolution, but a multi-scale channel attention mechanism is set in the deformable convolution). This not only retains the rich gradient flow of the original C2f module, but also can improve the feature extraction ability, thereby improving the detection performance of the object detection model. Because the ability of the Bottleneck layer to model geometric transformations is essentially limited, the detection model using ordinary convolution has poor performance when detecting objects with rich geometric transformations (such as underwater environments), such as swimming fish underwater, refraction of light in water, etc. In this embodiment, deformable convolution (DCN) is used to replace the convolution therein to enhance the feature extraction ability; moreover, since the offset of DCN is only learned by one convolution, its learning ability is not strong, resulting in its receptive field still not being very accurate and still introducing irrelevant regions. Therefore, in this embodiment, on the basis of introducing DCN, multi-scale channel attention is further added to DCN, which can enrich the multi-scale gradient flow, is better than the receptive field of the ordinary convolution module, and can reduce information loss more.

[0055] In some embodiments, the deformable convolution is DCNv2. Compared with DCNv3, DCNv2 is more suitable for small models. The following details DCNv2:

[0056] Referring to Figure 5, where Input feature map represents the input feature, Output feature map is the output feature, Deformable convolution represents deformable convolution, and Conv, offset field, and Sigmoid are attempts in the field which will not be introduced here. The working principle of DCNv2 is as follows: For the input feature map, assuming the original convolution operation is 3×3, in order to learn the offset, another 3×3 convolution layer is defined. The output dimension is actually the size of the original feature map, and the number of channels is equal to 2N (representing the offsets in the x and y directions respectively). Then, a weight coefficient mask is added to guide the offset. The following deformable convolution can be regarded as first performing an interpolation operation based on the offset generated in the above part, and then performing a normal convolution. Therefore, compared with the ordinary convolution, DCNv2 is equivalent to adding an offset (Δp n ) and a weight coefficient Δm k to each convolution sampling point x. The calculation formula is formula (1), where y(p) represents the value after the convolution operation, and k represents the number of the sampling point.

[0057]

[0058] A multi-scale channel attention mechanism is added to DCNv2 to increase the attention for the offsets in the X and Y directions. The deformable convolution with multi-scale channel attention sets the multi-scale channel attention when generating the offsets in the DCNv2 convolution. Referring to Figure 6 , the multi-scale channel attention processes the features through the following steps S210 to S230:

[0059] Step S210: Use the CA attention to decompose and pool the input features of the multi-scale channel attention into features in two spatial directions.

[0060] Step S220: Convolve the features obtained by convolving the input features of the multi-scale channel attention by the CA attention without decomposition, and directly convolve them to obtain the first convolution result. Then, separate the weights in two spatial directions from the first convolution result through an activation function, and fuse the weights in two spatial directions with the corresponding features in two spatial directions to obtain the first fusion result.

[0061] Step S230: Perform global pooling and convolution on the input features of the multi-scale channel attention to obtain the second convolution result. Multiply the second convolution result by the mean of the first convolution result to obtain the multiplication result. Finally, fuse the multiplication result with the first fusion result to obtain the output features of the multi-scale attention.

[0062] The following provides a detailed introduction to steps S210 to S230:

[0063] Referring to Figure 7 , the CA attention not only considers the relationship between channels but also the position information in the feature space. To enable the attention to capture long-range spatial interactions with precise position information, the CA attention decomposes the global pooling into one-dimensional feature encoding operations in two spatial directions, and then aggregates the features to obtain a pair of direction-aware feature maps. These two transformations of the CA attention allow the module to capture long-term dependencies along one spatial direction and preserve the precise position information along the other spatial direction, which helps the network to more accurately locate the target of interest. The CA attention directly uses two simple global pooling operations to decompose the information in two directions, but this transformation is rough. After the two directions are separated, the information exchange through only a concatenation and convolution operation is far from enough, and the insufficiently rich information interaction causes varying degrees of information loss.

[0064] The proposed multi-scale channel attention mechanism added in DCNv2 in this embodiment is to fuse multi-scale information on the basis of the decomposed pooling operation, extract the information in the intermediate process as weights and fuse it with the outputs in two directions to enhance the information interaction finally. Then, use the global information of the global pooling to guide the two directions, to a certain extent, avoiding information loss.

[0065] Referring to Figure 8 , the added multi-scale channel attention mechanism is mainly divided into three branches, which fuse multi-scale at three different gradients:

[0066] The first branch still converts the global pooling operation into one-dimensional encoding for each of the x and y directions. This is the ordinary CA attention mechanism part.

[0067] The second branch is to directly perform convolution without separation after the convolution of the first branch, and then pass through the activation function Sigmod, and then separate the two directions x_weight and y_weight, which are the weight coefficients of x and y. The advantage of doing this is that it can supplement the information of x and y, and can be more accurate after fusion.

[0068] The third branch is to perform ordinary global pooling on the initial input information, and then multiply the mean value of the data before the second branch is separated after convolution. The purpose of doing this is to fuse the global information to guide the encoding information in two directions. Finally, the information of the three branches is fused to obtain the output. The schematic diagram of the added multi-scale channel attention is as Figure 9 shown.

[0069] In the added multi-scale channel attention, the decomposition of the global pooling into one-dimensional encoding operations for each can be expressed by formula (2). Where z cIndicates the output in the height or width direction, where H and W represent height and width respectively, and i and j represent specific channels thereof, and x c Indicates the input feature map.

[0070]

[0071] Specifically, given the input x c , first, a pooling operation with a size of (H, 1) or (1, W) is used to encode each channel along the horizontal and vertical coordinates respectively. Therefore, the outputs in the height and width directions can be expressed as shown in formulas (3) and (4), where is the output of height h and width w on a certain channel.

[0072]

[0073]

[0074] After concatenating the obtained outputs in the width and height directions to obtain a tensor with the same number of channels and appending it to the input, it is then convolved and passed through an activation function. Then, on the one hand, it is separated into the outputs in the x and y directions respectively ( represents height, width direction), and on the other hand, after passing it through convolution and an activation function, it is separated into the weight coefficients in the x and y directions This weight coefficient can compensate for the information loss in the intermediate process and is used to guide the outputs in the x and y directions. Specifically, it can be expressed by formulas (5) and (6), where represent the outputs in the height and width directions after fusing the weight coefficients respectively.

[0075]

[0076]

[0077] Therefore, the output of the added multi-scale channel attention can be expressed as formula (7), where x c represents the feature map of the original input, and a c is the global information supplement under global pooling.

[0078]

[0079] For the DCNv2 convolution of the added multi-scale channel attention mechanism, a multi-scale channel attention mechanism is mainly added during the generation of the offset in the ordinary DCNv2 convolution, thereby enhancing the accuracy of the offset. This makes the receptive field larger than that of the ordinary convolution and reduces the introduced irrelevant regions. Therefore, the information extraction ability of the convolution is stronger, and it can extract more accurate context regions.

[0080] Referring to Figure 2 and Figure 10 (where the DCNv2 with added multi-scale channel attention is denoted as DCN_MPCA), in this embodiment, the second module replaces the C2f module of YOLOv8, retaining the rich gradient flow of the C2f module. A large amount of spatial feature information is retained in these gradient flows. After introducing the DCNv2 convolution with the added multi-scale channel attention mechanism in the Bottleneck, the object detection model can have a larger and more accurate receptive field, thereby extracting more accurate and rich feature information. Because the second module can make the feature extraction ability of the model stronger, thus improving the model performance.

[0081] The following introduces the first module:

[0082] In some embodiments of the present application, the first module is further configured to extract local features and surrounding context features from the downsampled features, combine the local features and the surrounding context features into a joint feature, and improve the joint feature through the global context feature. Among them, referring to Figure 11 , extracting local features and surrounding context features from the downsampled features, combining the local features and the surrounding context features into a joint feature, and improving the joint feature through the global context feature includes the following steps S310 - S350:

[0083] Step S310: Extract local features from the downsampled features through a 3×3 convolution.

[0084] Step S320: Extract surrounding context features from the downsampled features through a 3×3 dilated convolution.

[0085] Step S330: Concatenate the local features and the surrounding context features to obtain a joint feature.

[0086] Step S340: Input the joint feature into a combined module including a global pooling layer and two fully connected layers to obtain a weight vector output by the combined module.

[0087] Step S350: Combine the weight vector with the joint feature as the output feature of the first module.

[0088] Referring to Figure 12 , the first module consists of 5 parts:

[0089] (1) Downsampling operation f down (*) Implemented by a 3×3 convolution;

[0090] (2) Local feature extractor f loc (*) Implemented by a conventional convolution;

[0091] (3) Surrounding context feature extractor f sur (*) Implemented with dilated convolution;

[0092] (4) Joint feature extractor f joi (*) It is a simple concatenation layer, followed by a BN layer and a SiLU activation function;

[0093] (5) Global feature extractor f glo (*) It is a global pooling layer followed by two fully connected layers to extract features, obtaining a weight vector, and using this vector to guide joint feature fusion.

[0094] The above only introduced that there is a first module in the backbone network setting of the object detection model. In this embodiment, the Neck part of the baseline model is also improved; that is, the downsampling operation in the Neck part of the baseline model is replaced by the first module.

[0095] The first module is mainly responsible for downsampling operations and information integration, and can integrate information in a timely manner after the downsampling operation. The first module improves the joint features through global context features by introducing the joint features of local features and surrounding context, and integrates the information after downsampling, thereby improving the information utilization efficiency. Since the first module can be applied throughout the network of the object detection model, from the spatial level to the semantic level, and is mostly set before the second module, the more accurate features will be sent to the second module for further processing in the first time, avoiding information waste. Compared with the baseline model, the first module improves the AP of the object detection model on the COCO dataset by 1.14%, reaching 38.44% (compared based on the YOLOv8-N version).

[0096] Refer to Figure 13 , an embodiment of the present application provides a method for detecting underwater fish targets. This method mainly uses an object detection model to identify underwater images.

[0097] YOLOv8 is selected as the baseline model. After improving the baseline model, an object detection model is obtained (for easy comparison, the object detection model is named YOLO-DC). The improvement points include:

[0098] The first improvement: Replace the C2f module in the backbone network part of the baseline model with the second module. The second module takes the C2f module as the baseline module, and then in the Bottleneck layer with the C2f module as the baseline module, replace the ordinary convolution with deformable convolution with multi-scale channel attention, that is, the DCN_MPCA in the above embodiment, which will not be elaborated here.

[0099] Second improvement: Replace the downsampling operation of the baseline model with the first module. For details of the first module, please refer to the above-mentioned embodiment introduction, which will not be elaborated here.

[0100] Benefiting from the joint context information of the first module to strengthen the feature information, combined with the powerful feature extraction ability of the second module and the receptive field superior to that of the ordinary convolution module, the performance of the model is greatly improved, thus improving the accuracy of fish recognition in underwater images. YOLO-DC has three versions: N, S, and M, with dimension scaling factors w being {0.25, 0.5, 0.75}, quantity scaling factors d being {0.33, 0.33, 0.67}, and ratio scaling factors r being {2, 2, 1.5} respectively. Compared with YOLOv8-N, YOLO-DC-N (i.e., the N version of YOLO-DC) has an AP increase of 3.5% on the COCO dataset, reaching 40.8%.

[0101] It should be noted that after building the object detection model, it also needs to be trained and optimized, which will not be elaborated here.

[0102] The following provides experimental conclusions to prove the effectiveness of the object detection model (YOLO-DC):

[0103] Experimental environment: Two GPUs (NVIDIA RTX3090) with 24GB video memory, operating system Ubuntu22.04.3, CUDA version 11.8, programming language Python (Python version 3.9), and deep learning framework Pytorch version 2.1.

[0104] Data used in the experiment: The publicly available dataset Microsoft COCO2017 for object detection, the underwater object detection dataset RUOD, and the combined dataset of PASCAL VOC (07 + 12). Among them, the Microsoft COCO2017 dataset is one of the most widely used public datasets in the field of object detection, with 80 categories including pedestrians, vehicles, animals, daily necessities, public facilities, etc. The training set contains 118,287 images, and the test set contains 5,000 images. The training set of the combined PASCAL VOC dataset (07 + 12) is composed of the training sets and validation sets of PASCAL VOC2007 and PASCAL VOC2012, and the test set is the test set of PASCAL VOC2007. The training set includes 16,551 images, and the test set includes 4,952 images. The RUOD dataset covers general underwater scenarios and various underwater detection challenges. The training set includes 9,800 images, and the test set includes 4,200 images. In addition to the regular training and test sets, the RUOD dataset also includes test sets for 3 environmental challenges, namely fog effect, color bias, and light interference. This can comprehensively evaluate the performance of the detector.

[0105]

[0106] Table 1

[0107] In the experiment, YOLOv8 was selected as the baseline model, and at the same time, other advanced models in the field were selected for comparative experiments. The number of training epochs of YOLO-DC is 500, and the other hyperparameters all adopt the default settings of YOLOv8. No pre-trained weights are loaded for all experiments, which means the model learns from scratch, so as to more accurately evaluate the performance of the model. The evaluation metrics in the experiment uniformly adopt the mainstream COCO evaluation metrics in the field. The experimental results show that YOLO-DC has achieved good results on these datasets and is better than other advanced models.

[0108] Under the same experimental environment, YOLOv5, YOLOv6, YOLOv7, YOLOX, Gold-YOLO, the baseline model YOLOv8, and the YOLO-DC model were selected for comparative experiments on the Microsoft COCO2017 dataset.

[0109] Experiments have shown that on the Microsoft COCO2017 dataset, each version of YOLO-DC outperforms other models, achieving significant improvements. With comparable computational requirements and number of parameters, compared to YOLOv8-N, the AP of YOLO-DC-N has increased by 3.5%, reaching 40.8%. Compared to the S and M versions of YOLOv8, the AP has increased by 1.71% and 0.2% respectively. Compared to YOLOv5-N, YOLOv6-N, YOLOv7-Tiny, YOLOX-N, and Gold-YOLO-N, the AP has increased by 12.8%, 3.8%, 3.4%, 8%, and 0.9% respectively. The S and M versions of YOLO-DC also show significant improvements compared to other advanced models. The specific data of the comparative experiments are shown in Table 1. In terms of the number of parameters and computational requirements, YOLO-DC also performs well, with its comprehensive performance being superior to other advanced models, achieving the best balance between performance and computation.

[0110] Refer to Figure 14 , comparison with other advanced models on the Microsoft COCO dataset: (1) AP performance vs computational requirements (2) AP performance vs number of parameters.

[0111] Comparative experiments (PACAL VOC (07+12)) To comprehensively evaluate the various performances of the models, comparative experiments were also conducted on the PASCAL VOC combined dataset (07+12) and the underwater dataset RUOD, selecting YOLOv5 (N, S), YOLOv6-N, YOLOv7-Tiny, and the baseline model YOLOv8-N.

[0112] Experiments on the PASCAL VOC combined dataset (07+12) have shown that compared to YOLOv8-N, the AP of YOLO-DC-N has increased by 9.89%, reaching 82.34%, which is higher than all other selected advanced models. The best-performing YOLOv6-N has an AP that is 0.39% lower than that of YOLO-DC-N, and the computational requirements and number of parameters of YOLO-DC-N are much lower than those of YOLOv6-N. The detailed data are shown in Table 2 below.

[0113]

[0114] Table 2

[0115] Benefiting from the excellent performance of deformable convolution in dealing with geometric deformations, YOLO-DC still outperforms all other advanced models on the underwater dataset RUOD. Compared with YOLOv8-N, the AP of YOLO-DC-N has increased by 0.35%, reaching 82.34%. Amazingly, compared with YOLOv5-s with nearly twice the number of parameters, the AP of YOLO-DC-N is still 6.05% higher. The detailed data is shown in Table 3 below:

[0116]

[0117]

[0118] Table 3

[0119] Experiments have proved that compared with multiple advanced algorithms on multiple datasets, the YOLO-DC model has better performance than other models in terms of the number of parameters, computational volume, and detection accuracy.

[0120] To verify the effectiveness of the multi-scale channel attention, the second module, and the first module in DCN_MPCA, ablation experiments were conducted on each module on the Microsoft COCO2017 dataset.

[0121]

[0122] Table 4

[0123] YOLOv8-N was selected as the baseline model with an AP of 37.3%. When only the convolution in the C2f module was replaced with DCNv2 convolution, the AP of the model increased by 1.33%, reaching 38.63%. This proves that DCNv2 convolution can improve the information extraction ability, expand the receptive field, and make the model's feature extraction ability stronger and more accurate. When the second module was used alone, that is, the improved deformable convolution, namely the deformable convolution with multi-scale channel attention (DCN_MPCA), the AP of the model increased by 1.73% compared with the baseline model, reaching 39.03%. This proves that DCN_MPCA has stronger information extraction ability than ordinary DCNv2 convolution, thus verifying the effectiveness of the multi-scale channel attention in DCN_MPCA. When the first module was used alone, the AP increased by 1.14% compared with the baseline model, reaching 38.44%. This proves that after the downsampling operation, timely integrating the feature information of the surrounding context and the global context into the main convolution module is beneficial to improving the performance of the model.

[0124] When the second module and the first module are used simultaneously, that is, the YOLO-DC model, the AP of the model is improved by 3.5% compared with the baseline model and reaches 40.8%. The first module can highlight the key information of the feature map by using local features and surrounding context information after downsampling, and then send it to the second module with strong feature extraction ability. The two cooperate with each other in the whole network and complement each other, achieving better results than using these two modules separately. This also proves that the setting of integrating effective information after downsampling in the first module and then sending it to the main convolutional module, that is, the second module, can further improve the performance of the model. The detailed experimental data are shown in Table 4.

[0125] Some embodiments of the present application provide an underwater fish target detection device, which includes:

[0126] An image acquisition unit for acquiring underwater images.

[0127] A fish detection unit for inputting the underwater image into the target detection model to obtain the fish detection result output by the target detection model; where the target detection model is based on YOLOv8 as the baseline model and is obtained by at least improving the backbone network part of the baseline model. The backbone network part of the target detection model includes a Conv layer, an SPPF layer, and multiple groups of cascaded first modules and second modules arranged between the Conv layer and the SPPF layer, and there is a corresponding second module connected to the Neck part of the target detection model; the first module is used for downsampling the input features of the first module, and the second module is based on the C2f module of the backbone network part of YOLOv8 as the baseline module and is obtained by adding deformable convolutions with multi-scale channel attention in the Bottleneck layer of the baseline module.

[0128] It should be noted that the underwater fish target detection device provided in this embodiment and the above-mentioned underwater fish target detection method are based on the same inventive concept. Therefore, the relevant content of the above-mentioned underwater fish target detection method also applies to the content of the underwater fish target detection method. Therefore, it will not be elaborated here.

[0129] As Figure 15 , the embodiments of the present application also provide an electronic device, which includes:

[0130] At least one memory;

[0131] At least one processor;

[0132] At least one program;

[0133] The program is stored in the memory, and the processor executes at least one program to implement the above-mentioned underwater fish target detection method of the present disclosure.

[0134] The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), an in-vehicle computer, etc.

[0135] The following provides a detailed introduction to the electronic device according to the embodiments of the present application.

[0136] The processor 1600 can be implemented by using a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention;

[0137] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1700 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the program code is stored in the memory 1700 and is called by the processor 1600 to execute the underwater fish target detection method of the embodiments of the present invention.

[0138] The input / output interface 1800 is used to implement information input and output;

[0139] The communication interface 1900 is used to implement communication interaction between this device and other devices, and can achieve communication through a wired method (such as USB, network cable, etc.) or can also achieve communication through a wireless method (such as a mobile network, WIFI, Bluetooth, etc.);

[0140] The bus 2000 transmits information between the various components of the device (such as the processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900);

[0141] Among them, the processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900 achieve communication connections with each other inside the device through the bus 2000.

[0142] The embodiments of the present invention also provide a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute the above-mentioned underwater fish target detection method.

[0143] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include memories remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The embodiments described in the present invention are for more clearly illustrating the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0145] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0146] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0148] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of this application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0149] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the relationship between associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item) of the following" or its similar expressions refer to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0150] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0151] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, in each embodiment of this application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0153] When an integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes: various media that can store programs, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0154] The above is a specific description of the preferred implementation of the embodiments of this application. However, the embodiments of this application are not limited to the above-mentioned implementation manners. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the embodiments of this application. These equivalent deformations or substitutions are all included within the scope defined by the claims of the embodiments of this application.

Claims

1. A method for target detection of underwater fish, characterized in that, The target detection method for underwater fish includes: Obtaining an underwater image; Inputting the underwater image into a target detection model to obtain a fish detection result output by the target detection model. The target detection model is a model obtained by improving the backbone network part of the benchmark model based on YOLOv8. The backbone network part of the target detection model includes a Conv layer, an SPPF layer, and multiple sets of cascaded first modules and second modules arranged between the Conv layer and the SPPF layer, and there is a corresponding second module connected to the Neck part of the target detection model. The first module is used to downsample the input features of the first module. The second module is a module obtained by adding deformable convolutions with multi-scale channel attention to the Bottleneck layer in the benchmark module based on the C2f module of the backbone network part of YOLOv8. The deformable convolution in the deformable convolution with multi-scale channel attention is DCNv2. The deformable convolution with multi-scale channel attention sets multi-scale channel attention during the generation of offsets in the DCNv2 convolution. Moreover, the multi-scale channel attention processes features in the following manner: Using CA attention to decompose and pool the input features of the multi-scale channel attention into features in two spatial directions; Directly convolving the features after convolution of the input features of the multi-scale channel attention by CA attention without decomposition to obtain a first convolution result, separating the weights in two spatial directions from the first convolution result through an activation function, and fusing the weights in the two spatial directions with the corresponding features in the two spatial directions to obtain a first fusion result; Globally pooling and convolving the input features of the multi-scale channel attention to obtain a second convolution result, multiplying the second convolution result by the mean of the first convolution result to obtain a multiplication result, and finally fusing the multiplication result with the first fusion result to obtain the output features of the multi-scale channel attention.

2. The target detection method for underwater fish according to claim 1, characterized in that, The target detection model is a model obtained by improving the backbone network part and the Neck part of the benchmark model based on YOLOv8. Among them, the improvement of the Neck part of the benchmark model includes: replacing the downsampling operation of the Neck part of the benchmark model with the first module.

3. The target detection method for underwater fish according to claim 1, wherein The first module is also used to extract local features and surrounding context features from the downsampled features, form a joint feature from the local features and the surrounding context features, and improve the joint feature through global context features.

4. The target detection method for underwater fish according to claim 3, wherein, The extraction of local features and surrounding context features from the downsampled features, the formation of a joint feature from the local features and the surrounding context features, and the improvement of the joint feature through global context features include: Extracting local features from the downsampled features through 3×3 convolution; Extracting surrounding context features from the downsampled features through 3×3 dilated convolution; Concatenate the local feature and the surrounding context feature to obtain a combined feature; Input the combined feature into a combined module including a global pooling layer and two fully connected layers to obtain a weight vector output by the combined module; Combine the weight vector with the combined feature as the output feature of the first module.

5. An underwater fish target detection device, characterized in that, The underwater fish target detection device includes: An image acquisition unit for acquiring an underwater image; A fish detection unit for inputting the underwater image into a target detection model to obtain a fish detection result output by the target detection model; wherein the target detection model is a model obtained by improving the backbone network part of the benchmark model YOLOv8. The backbone network part of the target detection model includes a Conv layer, an SPPF layer, and multiple groups of cascaded first modules and second modules arranged between the Conv layer and the SPPF layer, and there is a corresponding second module connected to the Neck part of the target detection model; the first module is used for downsampling the input feature of the first module, and the second module is a module obtained by adding a deformable convolution with multi-scale channel attention to the Bottleneck layer in the benchmark module of the backbone network part of YOLOv8; The deformable convolution in the deformable convolution with multi-scale channel attention is DCNv2; The deformable convolution with multi-scale channel attention sets multi-scale channel attention during the generation of the offset in the DCNv2 convolution; Moreover, the multi-scale channel attention processes features in the following manner: Use CA attention to decompose and pool the input feature of the multi-scale channel attention into features in two spatial directions; Directly convolve the feature obtained by convolving the input feature of the multi-scale channel attention by CA attention without decomposition to obtain a first convolution result, and separate the weights in two spatial directions from the first convolution result through an activation function, and fuse the weights in the two spatial directions with the features corresponding to the two spatial directions to obtain a first fusion result; Globally pool and convolve the input feature of the multi-scale channel attention to obtain a second convolution result, multiply the second convolution result by the mean of the first convolution result to obtain a multiplication result, and finally fuse the multiplication result with the first fusion result to obtain the output feature of the multi-scale channel attention.

6. An electronic device, characterized in that, Including: At least one control processor and a memory for communicatively connecting with the at least one control processor; The memory stores instructions executable by the at least one control processor. The instructions are executed by the at least one control processor to enable the at least one control processor to execute the underwater fish target detection method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to execute the underwater fish target detection method according to any one of claims 1 to 4.

Images