Real-time target detection method applied to underwater fishing robot

Abstract

The invention provides a real-time target detection method applied to an underwater fishing robot, and the method specifically comprises the steps: obtaining an underwater image through an underwater robot, carrying out the preprocessing of the obtained image, arranging the image into a data set, and randomly dividing the data set into a training set, a verification set, and a test set; constructing an improved backbone network by using a MobileNetV4 network, and extracting a feature map of the underwater image; fasterNet is used to optimize a main branch and an auxiliary branch of feature fusion, and feature fusion is carried out on the extracted feature map; training the constructed network model, verifying the trained network model by using the verification set, and judging whether the network model reaches an expected performance index or not; and deploying the network model to an underwater fishing robot system, and performing target detection on an underwater video image obtained in real time. According to the method, the parameter quantity and the calculation quantity are remarkably reduced, the precision of the model is improved, and the requirement of the underwater fishing robot for a target detection algorithm with better performance can be met.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater target searching, and in particular to a real-time target detection method applied to an underwater fishing robot. Background Art

[0002] Using underwater fishing robots to obtain seafood and estimate the output of marine fisheries can effectively promote the development of deep-sea aquaculture.

[0003] Generally speaking, when using underwater fishing robots to complete the above tasks, the first thing to solve is how to detect the target seafood faster and more accurately to facilitate the next work; this requires the target detection model carried by the underwater fishing robot system to have better real-time performance, faster detection speed and higher detection accuracy. The YOLO series algorithm realizes real-time target detection by converting the target detection task into a single forward pass regression problem, and can achieve a high frame rate under GPU acceleration to meet the needs of real-time applications; the YOLO series algorithm has several versions, among which YOLOv9, as the latest version, has further improved its performance compared to YOLOv8, but YOLOv9 adds auxiliary branches in the feature fusion stage to minimize the loss of information in the image. Therefore, compared with YOLOv8, YOLOv9 is slower than YOLOv8 in inference speed and slower than YOLOv8 in real-time target detection. Summary of the invention

[0004] According to the technical problems raised above, a real-time target detection method for underwater fishing robots is provided. The present invention discloses a real-time underwater target detection method for underwater fishing robots, which further reduces the number of parameters and calculations in the YOLOv9-c network, while improving the accuracy of the model. The improved network is applied to the underwater fishing robot system to meet the underwater fishing robot's requirements for a target detection algorithm with better performance.

[0005] The technical means adopted by the present invention are as follows:

[0006] A real-time target detection method applied to an underwater fishing robot, the specific steps comprising:

[0007] S1. Use an underwater robot to obtain underwater images, pre-process the obtained images, organize them into a data set, and randomly divide the data set into a training set, a validation set, and a test set;

[0008] S2. Use the MobileNetV4 network to build an improved backbone network and extract feature maps of underwater images;

[0009] S3, using FasterNet to optimize the main branch and auxiliary branch of feature fusion, and perform feature fusion on the feature map extracted in S2;

[0010] S4, train the constructed network model, and use the verification set to verify the trained network model to determine whether the network model reaches the expected performance indicators;

[0011] S5. Deploy the network model in S4 to the underwater fishing robot system to perform target detection on the underwater video images acquired in real time.

[0012] Furthermore, the step S1 specifically includes:

[0013] An underwater robot is used to acquire underwater images, and the images are preprocessed. The images include at least one of four organisms, namely, sea cucumbers, sea urchins, scallops and starfish. The sea cucumbers, sea urchins, scallops and starfish in the images are annotated with four different labels. A data set is generated using the annotated images, and the data set is simply randomly divided to obtain a training set, a validation set and a test set.

[0014] Furthermore, the step S2 specifically includes:

[0015] Use the MobileNetV4 network to build and improve the YOLOv9-based backbone network; use the images of the training set in S1 as input, and the process of extracting feature maps is as follows:

[0016] Unify the size of the input image and input the unified image into the backbone network. The input image passes through the Silence layer of the MobileNetV4 network and is directly output to the next layer of the backbone network, MNV4ConvSmall.

[0017] The MNV4ConvSmall includes a single convolution layer and five other network layers consisting of convolution modules. After the input image is processed by MNV4ConvSmall, feature maps of different sizes are output.

[0018] Furthermore, the step S3 specifically includes:

[0019] FasterNet, GhostConv and dynamic upsampling methods are used to optimize the main branch and auxiliary branch used for feature fusion in YOLOv9, and feature fusion is performed on the feature maps output in S2. The feature maps at all levels after feature fusion are sent to the improved detection head to detect the category and position of the target;

[0020] In the main branch, a dynamic upsampling method based on point sampling is adopted. In the auxiliary branch, GhostConv is used to optimize the convolution module in the auxiliary branch. The FasterNet Block in the FasterNet optimizes the RepNCSPELAN4 module existing in both the main branch and the auxiliary branch. The FasterNet Block includes some convolution modules, convolution modules, BN layers and ReLU activation functions. The RepNCSPELAN4 module is used for feature extraction and fusion in YOLOv9. Each RepNCSPELAN4 module includes a convolution module and a RepNCSP module. The Bottleneck in the RepNCSP module is optimized by FasterNet Block. The GhostConv improves and optimizes the convolution module in the detection head, sends the feature map after feature fusion to the detection head, and generates the category and detection frame of the corresponding target in the image.

[0021] Furthermore, the detecting of the type and position of the target includes:

[0022] S31. Use FasterNet to build FasterRepNCSPELAN4 and optimize the main branch and auxiliary branch, that is, optimize BottleNeck in the RepNCSPELAN4 submodule.

[0023] S32, optimizing the upsampling operation in the backbone network using a dynamic upsampling method;

[0024] S33, using GhostConv to optimize the auxiliary branch and the convolutional layers in the detection head;

[0025] S34, after the network construction is completed, the feature maps of different sizes output in S2 are respectively input into three different network layers in the main branch and the auxiliary branch; the feature maps of different sizes are processed by pooling, convolution, upsampling, splicing, downsampling and segmentation in the main branch to complete feature fusion, and the main feature map is output; the feature maps of different sizes are processed by convolution, segmentation, downsampling and splicing in the auxiliary branch to complete feature fusion, and the auxiliary feature map is output;

[0026] S35, output the main feature map and the auxiliary feature map output in S34 to the detection head to detect the category and position of the target.

[0027] Furthermore, the step S4 specifically includes:

[0028] S41. Calculate the loss function, and use the gradient of the loss function to calculate the gradient of each parameter relative to the loss function through the back propagation algorithm to update the network parameters. The loss function for the classification problem in the model is BCE Loss, which is used to calculate the difference between the predicted category distribution and the true category distribution. The loss function for the regression problem is DFL Loss and CIoU Loss, which is the difference between the position and size of the predicted bounding box and the true bounding box. The calculation method is as follows:

[0029]

[0030] DFL(S i , S i+1 )=-((y i+1 -y)log S i +(yy i )log S i+1 )

[0031]

[0032] In BCE Loss, N is the batch size; y i represents the true category of the target; p i is the predicted category of the target. In DFL Loss, S i ,S i+1 are the predicted value and the near-predicted value respectively; y, y i ,y i+1 is the actual value, label integral value, and adjacent label integral value; ρ 2 (b,b gt ) is the Euclidean distance between the center points of the predicted box and the true box; c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box; α is a weight parameter used to balance the influence of proportion consistency; v is used to measure the consistency of aspect ratio; IoU is the intersection over union ratio of the predicted box and the true box, which is used to measure the overlap degree of the two boxes; S42, iterative training is performed until the preset number of training rounds is reached, the training is completed, and the weight file for the target detection task is obtained;

[0033] S43. Use a pre-built validation set to validate the trained model to determine whether the model reaches the expected performance indicators; export the model that reaches the expected performance indicators.

[0034] Furthermore, the step S5 specifically includes:

[0035] The exported model is deployed on the underwater fishing robot system; the underwater fishing robot obtains real-time video stream, and the ground station performs underwater image enhancement on the video stream; the ACE algorithm is used for image enhancement, and the algorithm flow is as follows:

[0036] Adjust the color of the image and correct the chromatic aberration of the image as follows:

[0037]

[0038] Among them, R c It is the intermediate result of the algorithm, which calculates the brightness difference between each point and other points and performs weighted summation according to the distance; c (i)-I c (j) is the brightness difference between two pixels, r(*) is the brightness representation function;

[0039] Dynamically expand the color-adjusted image and process each channel of the image separately through linear expansion.

[0040] O c (p) = round (127.5 + s c R c )

[0041] Among them, O c (p) is a linear expansion method, s c is [(m c ,0),(M c ,255)], M c =max i (R c (i)), m c =min i (R c (i));

[0042] The intermediate result R c Stretch and map to [0,255] to complete image enhancement as follows:

[0043]

[0044] Among them, the result R calculated by L(x) in the first stage c Perform normalization processing;

[0045] After image enhancement, the target detection model deployed in the system detects the targets in the video stream and marks their locations and categories.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] The real-time target detection method applied to underwater fishing robots provided by the present invention utilizes the underwater fishing robot to obtain an underwater image data set, expands the number of underwater images, is beneficial to the training of the model, and can be better applied to real-time underwater target detection; at the same time, it is improved based on the YOLOv9 target detection method, improves the detection speed and accuracy, reduces the number of parameters in the network, and is combined with the underwater image enhancement algorithm, so that it is more effectively applied to underwater real-time detection tasks, and still has effectiveness, real-timeness and accuracy in complex marine environments.

[0048] Based on the above reasons, the present invention can be widely promoted in the field of underwater target search technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 The present invention is a flow chart of the real-time target detection method applied to an underwater fishing robot.

[0051] Figure 2 This is the overall network structure diagram after the present invention is improved based on YOLOv9.

[0052] Figure 3 This is the improved trunk-branch network structure of the present invention.

[0053] Figure 4 This is the improved auxiliary branch network structure of the present invention.

[0054] Figure 5 This is the optimized feature extraction-fusion module of the present invention. DETAILED DESCRIPTION

[0055] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0056] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] like Figure 1 As shown, the present invention provides a real-time target detection method applied to an underwater fishing robot, and the specific steps include:

[0058] S1. Use an underwater robot to obtain underwater images, pre-process the obtained images, organize them into a data set, and randomly divide the data set into a training set, a validation set, and a test set;

[0059] In specific implementation, as a preferred embodiment of the present invention, the step S1 specifically includes:

[0060] Using an underwater robot to acquire underwater images, preprocessing the images, and selecting clear images for annotation, wherein the images include at least one of four organisms: sea cucumbers, sea urchins, scallops, and starfish, and the sea cucumbers, sea urchins, scallops, and starfish in the images are annotated with four different labels;

[0061] In the implementation, LabelImg is used to annotate the sea cucumbers, sea urchins, scallops, and starfish in the image. The four labels in the model are as follows: 0:'starfish', 1:'holothurian', 2:'echinus', 3:'scallop';

[0062] Generate a dataset using the annotated images. After annotating the images, generate an xml file for each image to describe the size and location information of the annotated detection box. Use Python to batch process the xml file corresponding to each image and convert it into the txt format required by the YOLO algorithm to complete the initial establishment of the dataset. Perform a simple random division of the dataset to obtain the training set, validation set, and test set. The division ratio of the training set, validation set, and test set is 7:2:1.

[0063] S2. Use the MobileNetV4 network to build an improved backbone network and extract feature maps of underwater images;

[0064] In specific implementation, as a preferred embodiment of the present invention, step S2 specifically includes:

[0065] The MobileNetV4 network is used to build and improve the YOLOv9-based backbone network; the improved backbone network has 7 network layers, namely Silence, conv0, layer1, layer2, layer3, layer4, and layer5; Figure 2 shown.

[0066] During implementation, the input image first passes through the Silence module of the backbone network. It does not perform any operation on the input, but simply returns the input directly and passes the information of all channels downward. MobileNetV4 is used to optimize the remaining backbone network of YOLOv9. The optimized network structure is mainly composed of a network layer consisting of a convolutional layer and multiple UIB modules. Each UIB module includes a convolutional layer, a batch normalization layer, and a ReLU6 activation function to maximize computational utilization, obtain three feature maps, and send them as input to the next stage.

[0067] Taking the image of the training set in S1 as input, the process of extracting the feature map is as follows:

[0068] The input image size is unified to 640*640, and the unified image is input into the backbone network with an input tensor size of 1*3*640*640. The input image passes through the Silence layer of the MobileNetV4 network and is directly output to the next layer of the backbone network, MNV4ConvSmall.

[0069] The MNV4ConvSmall includes a single convolution layer conv0 and five other network layers layer1-layer5 composed of convolution modules. After the input image is processed by MNV4ConvSmall, layer2, layer3 and layer5 respectively output three feature maps of different sizes, which are 80*80, 40*40 and 20*20, respectively. Figure 2 shown.

[0070] S3, using FasterNet to optimize the main branch and auxiliary branch of feature fusion, and perform feature fusion on the feature map extracted in S2;

[0071] In specific implementation, as a preferred embodiment of the present invention, step S3 specifically includes:

[0072] FasterNet, GhostConv and dynamic upsampling methods are used to optimize the main branch and auxiliary branch used for feature fusion in YOLOv9, and feature fusion is performed on the feature maps output in S2. The feature maps at all levels after feature fusion are sent to the improved detection head to detect the category and position of the target;

[0073] A feature extraction-fusion module optimized by FasterNet is constructed, and the submodule of RepNCSP in the feature extraction-fusion module is optimized. The Bottleneck is improved based on a partial convolution method in order to reduce redundant calculations and significantly improve computational efficiency. The improved Bottleneck operates on the image as follows: the image is processed by FasterBlock and Conv in turn. If the shortcut in the network layer is True, the input x is added to the result of the two convolutional layers, otherwise the result of the two convolutional layers is directly returned, where x is the data input to the network layer and shortcut is a Boolean variable initialized in the network.

[0074] In the main branch, a dynamic upsampling method based on point sampling is used to effectively reduce the number of parameters in the network. In the auxiliary branch, GhostConv is used to optimize the convolution module in the auxiliary branch, and the number of parameters and the amount of calculation are further reduced in the auxiliary branch to improve the efficiency and performance of the model in the present invention. The FasterNet Block in the FasterNet optimizes the RepNCSPELAN4 module that exists in both the main branch and the auxiliary branch. The FasterNet Block includes a partial convolution module, a convolution module, a BN layer and a ReLU activation function; the RepNCSPELAN4 module is used for feature extraction and fusion in YOLOv9, and each RepNCSPELAN4 module includes a convolution module and a RepNCSP module. The Bottleneck in the RepNCSP module is optimized by FasterNet Block; the main branch and auxiliary branch used for feature fusion in YOLOv9 are optimized by FasterNet. This optimization utilizes the characteristics that the PConv module only processes part of the input channel information to reduce the amount of calculation and memory access, and effectively reduces the number of parameters of the main branch and auxiliary branch used for feature fusion. The GhostConv improves and optimizes the convolution module in the detection head, sends the feature map after feature fusion to the detection head, and generates the category and detection frame of the corresponding target in the image.

[0075] In specific implementation, as a preferred embodiment of the present invention, the detection of the type and position of the target includes:

[0076] S31. Use FasterNet to build FasterRepNCSPELAN4, optimize the main branch and auxiliary branch, speed up the calculation, and reduce the number of parameters, that is, optimize the BottleNeck in the RepNCSPELAN4 submodule; the detection head in the original network consists of two groups of identical Sequentials, each Sequential is composed of two Conv layers and a Conv2d connected in sequence, the improved detection head has the same overall network structure as the original detection head, each Sequential is composed of two GhostConv layers and a Conv2d connected in sequence; the two identical Sequentials are used to predict the position information and category information of each anchor point, and output the box and class in the form of tensors.

[0077] S32. Use dynamic upsampling method to optimize the upsampling operation in the backbone network; while saving computing resources, it can effectively improve the efficiency and quality of image processing.

[0078] S33, using GhostConv to optimize the auxiliary branch and the convolutional layers in the detection head;

[0079] In implementation, GhostConv is a plug-and-play convolution module, which is characterized by extracting the feature map of the image with less computational effort. The specific steps of GhostConv are as follows: first, a feature map with a smaller number of channels is generated with less computational effort through traditional convolution operations, and then, based on the obtained feature map, a convolution operation is performed again to generate a new feature map, and finally, the two sets of feature maps are spliced ​​together to obtain the final feature map.

[0080] S34. After the network construction is completed, the feature maps of different sizes output in S2 are respectively input into three different network layers in the main branch and the auxiliary branch; the feature maps of different sizes are processed by pooling, convolution, upsampling, splicing, downsampling and segmentation in the main branch to complete feature fusion, and the main feature map is output as three feature maps P3, P4, and P5; the feature maps of different sizes are processed by convolution, segmentation, downsampling and splicing in the auxiliary branch to complete feature fusion, and the auxiliary feature map is output as three feature maps A3, A4, and A5. Figure 2 As shown;

[0081] The tensors output by each detection head are concatenated to obtain a tensor of size 1*8*8400 from the main branch and the auxiliary branch, which contains the predicted category information and prediction box parameters, where batch_size = 1, the number of prediction boxes = 8400, and each prediction box corresponds to 8 dimensions, namely 4 target categories plus 4 parameters x, y, h, w of the prediction box.

[0082] S35. Output the main feature map and auxiliary feature maps P3, P4, P5, A3, A4, and A5 output in S34 to the detection head to detect the category and position of the target.

[0083] S4, train the constructed network model, and use the verification set to verify the trained network model to determine whether the network model reaches the expected performance indicators;

[0084] In specific implementation, as a preferred embodiment of the present invention, step S4 specifically includes:

[0085] Set the batch size, training rounds, and learning rate in the network parameters and start training the model.

[0086] S41. Set the parameters in the training process, where size = 640, batch_size = 8. Fix the image size to 640*640 to ensure that the output feature maps of each layer of the network have the same size. The unified input size is conducive to efficient batch processing and parallel computing, thereby improving the efficiency of training and reasoning.

[0087] Calculate the loss function, and use the gradient of the loss function to calculate the gradient of each parameter relative to the loss function through the back-propagation algorithm, update the network parameters; and perform iterative training until the pre-set number of training rounds is reached, complete the training, and obtain a weight file that can be used for target detection tasks.

[0088] The loss function for the classification problem in the model is BCE Loss, which is used to calculate the difference between the predicted category distribution and the true category distribution. The loss function for the regression problem is DFL Loss and CIoU Loss, which is the difference between the position and size of the predicted bounding box and the true bounding box. The calculation method is as follows:

[0089]

[0090] DFL(S i , S i+1 )=-((y i+1 -y)log S i +(yy i )log S i+1 )

[0091]

[0092] In BCE Loss, N is the batch size; Y i represents the true category of the target; p i is the predicted category of the target, with a probability value between 0 and 1; in DFL Loss, S i ,Si+1 are the predicted value and the near-predicted value respectively; y, y i ,y i+1 is the actual value, label integral value, and adjacent label integral value; ρ 2 (b,b gt ) is the Euclidean distance between the center points of the predicted box and the real box; c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the real box; α is a weight parameter used to balance the impact of proportional consistency; ν is used to measure the consistency of aspect ratio; IoU is the intersection over union ratio of the predicted box and the real box, which is used to measure the overlap of the two boxes;

[0093] S42, performing iterative training until a preset number of training rounds is reached, completing the training, and obtaining a weight file for the target detection task;

[0094] S43. Use the pre-built validation set to verify the trained model to determine whether the model reaches the expected performance indicators; export the model that reaches the expected performance indicators, and the format of the exported model is ONNX to facilitate model deployment in subsequent steps.

[0095] S5. Deploy the network model in S4 to the underwater fishing robot system to perform target detection on the underwater video images acquired in real time.

[0096] In specific implementation, as a preferred embodiment of the present invention, step S5 specifically includes:

[0097] Export the model in ONNX format and deploy the model on the underwater fishing robot system; use TensorRT to deploy the model to accelerate the model reasoning speed and hardware acceleration;

[0098] Real-time video images are obtained by underwater fishing robots. Since the quality of real-time underwater images is limited by light conditions, color decay, low contrast, and blurred details occur. Therefore, before performing target detection on the real-time video stream, the video stream is first enhanced, and then the target category and position in the video image are detected; the underwater fishing robot obtains the real-time video stream, and the ground station performs underwater image enhancement on the video stream; the ACE algorithm is used for image enhancement to reduce computational complexity, which can well solve the color distortion problem caused by the underwater environment and better adapt to the image characteristics of different underwater environments. The algorithm flow is as follows:

[0099] Adjust the color of the image and correct the chromatic aberration of the image as follows:

[0100]

[0101] Among them, R cIt is the intermediate result of the algorithm, which calculates the brightness difference between each point and other points and performs weighted summation according to the distance; c (i)-I c (j) is the brightness difference between two pixels, r(*) is the brightness representation function, which must be an odd function, used to amplify smaller differences and enrich larger differences, expanding or compressing the dynamic range according to local content;

[0102] Dynamically expand the color-adjusted image and process each channel of the image separately through linear expansion.

[0103] O c (p) = round (127.5 + s c R c )

[0104] Among them, O c (p) is a linear expansion method, s c is [(m c ,0),(M c ,255)], M c =max i (R c (i)), m c =min i (R c (i));

[0105] The intermediate result R c Stretch and map to [0,255] to complete image enhancement as follows:

[0106]

[0107] Among them, the result R calculated by L(x) in the first stage c Perform normalization processing to make the brightness distribution of the image more uniform, thereby achieving global white balance adjustment and contrast enhancement of the image;

[0108] After image enhancement, the target detection model deployed in the system detects the targets in the video stream and marks their locations and categories.

[0109] During implementation, the underwater fishing robot system includes a ground station and an underwater fishing robot. The system installed on the ground station is Ubuntu 22.04, which is used to deploy the underwater target detection system and communicate with the underwater fishing robot.

[0110] Example 1

[0111] like Figure 1As shown, the present invention provides a real-time target detection method for an underwater fishing robot. This embodiment uses the MobileNetV4 network to build an improved backbone network and extract the feature map of the underwater image:

[0112] Step 1: After the image of size 640*640 is input into the backbone network, it is transformed into a tensor of size batch_size*3*640*640 according to the batch size. The input tensor first passes through the Silence layer. The Silence module does not process the input in any way and directly outputs it to the next layer of the backbone network and the auxiliary branch.

[0113] Step 2. In the network layer conv0, a tensor of size batch_size*3*640*640 is convolved with kernel_size=3, stride=2, pads=1, and the output tensor of size batch_size*32*320*320 is used as the input of layer1.

[0114] Step 3. In the network layer layer1, the tensor of size batch_size*32*320*320 is convolved twice, where convolution operation 1 in the layer1 network layer, kernal_size=3, stride=2, pads=1; convolution operation 2, kernal_size=1, stride=1, pads=0; the output size is batch_size*32*160*160 tensor as the input of layer2;

[0115] Step 4. In the network layer layer2, the tensor of size batch_size*32*160*160 is processed by two convolutions, where convolution operation 1 in the layer2 network layer, kernal_size=3, stride=2, pads=1; convolution operation 2, kernal_size=1, stride=1, pads=0; the output tensor of size batch_size*64*80*80 is used as the input of layer3, and the tensor of size batch_size*64*80*80 is used as the feature Figure 1 They are sent to the main branches and auxiliary branches respectively;

[0116] Step 5. In the network layer layer3, the batch_size*64*80*80 tensor is convolved to obtain a batch_size*96*40*40 tensor as the input of layer3, and the batch_size*96*40*40 tensor is used as the feature Figure 2They are sent to the main branches and auxiliary branches respectively;

[0117] Step 6. In the network layers layer4 and layer5, the tensor of size batch_size*96*40*40 is convolved to obtain a tensor of size batch_size*1280*20*20, and the tensor of size batch_size*1280*20*20 is used as the feature Figure 3 They are sent to the main branches and auxiliary branches respectively;

[0118] Step 7: The three feature maps of different sizes are sent to three different network layers in the main branch and the auxiliary branch in steps 4, 5, and 6 for feature fusion;

[0119] Step 8: In the main branch, feature Figure 3 First, it undergoes convolution processing with kernel kernal_size=1 and stride=1. Then it undergoes maximum pooling processing three times in sequence with pooling window kernal_size=5, stride=1 and pads=1. The feature maps obtained from each processing are concatenated to complete the first step of feature fusion.

[0120] Step 9: Dynamically upsample the feature map obtained in step 8 and compare the processed feature map with the feature map output from layer 3 of the backbone network. Figure 2 To splice;

[0121] Step 10, send the feature map obtained by splicing in step 9 to FasterRepNCSPELAN4 for further feature fusion;

[0122] Step 11: Dynamically upsample the feature map obtained in step 10 and compare the processed feature map with the feature map output from layer 2 of the backbone network. Figure 1 To splice;

[0123] Step 12: Send the feature map obtained by splicing in step 11 to FasterRepNCSPELAN4 for further feature fusion, and output the feature map P3 to the Ghost detection head;

[0124] Step 13, down-sampling the feature map output by step 12, the process is as follows: first, average pooling, pooling window kernal_size = 2, stride = 1, then split the feature map into two parts on the first dimension, convolve the first feature map after segmentation, convolution kernel kernal_size = 3, perform maximum pooling on the second feature map after segmentation, pooling window kernal_size = 3, stride = 2, then perform convolution operation, convolution kernel kernal_size = 1, stride = 1, finally, splice the two processed feature maps to complete downsampling, and splice with the feature map output by step 10;

[0125] Step 14, send the feature map obtained by splicing in step 13 to FasterRepNCSPELAN4 for further feature fusion, and output the feature map P3 to the Ghost detection head;

[0126] Step 15, repeat the operation in step 13, and concatenate the processed feature map with the feature map output in step 8, and output the feature map P5 to the Ghost detection head.

[0127] Step 16, as shown in the figure, in the auxiliary branch, the original image is first directly input by the Silence module in the backbone network, and then processed by GhostConv, GhostConv, FasterRepNCSPELAN4 and downsampling modules in the auxiliary branch;

[0128] Step 17, such as Figure 4 As shown in the figure, the three feature maps output by the backbone network are respectively input into three different CBLinears in the auxiliary branch to split the feature maps;

[0129] Step 18: Compare the feature map obtained in step 16 with the feature map processed by CBLinear Figure 1 ,feature Figure 2 ,feature Figure 3 The images are stacked and processed by FasterRepNCSPELAN4 to obtain feature map A3, which is then output to the detection head.

[0130] Step 19: Downsample feature map A3 and compare it with the feature map processed by CBLinear. Figure 2 ,feature Figure 3 The images are stacked and processed by FasterRepNCSPELAN4 to obtain feature map A4, which is then output to the detection head.

[0131] Step 20: Downsample feature map A4 and compare it with the feature map processed by CBLinear. Figure 3The images are stacked and processed by FasterRepNCSPELAN4 to obtain feature map A5, which is then output to the detection head.

[0132] Step 21, after feature fusion, the feature maps P3, P4, P5 and A3, A4, A5 are output to the optimized detection head respectively, and two tensors of the same size are obtained. The tensor size is batch_size*8*8400, which contains the predicted category information and prediction box parameters, where batch_size=1, the number of prediction boxes=8400, and each prediction box corresponds to 8 dimensions, that is, 4 target categories plus 4 parameters x, y, h, w of the prediction box.

[0133] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0134] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0137] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0138] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time target detection method for an underwater fishing robot, characterized in that: The specific steps include: S1. Use an underwater robot to obtain underwater images, pre-process the obtained images, organize them into a data set, and randomly divide the data set into a training set, a validation set, and a test set; S2. Use the MobileNetV4 network to build an improved backbone network and extract feature maps of underwater images; S3, using FasterNet to optimize the main branch and auxiliary branch of feature fusion, and perform feature fusion on the feature map extracted in S2; S4, train the constructed network model, and use the verification set to verify the trained network model to determine whether the network model reaches the expected performance indicators; S5. Deploy the network model in S4 to the underwater fishing robot system to perform target detection on the underwater video images acquired in real time.

2. The real-time target detection method for underwater fishing robots according to claim 1, characterized in that: The step S1 specifically includes: An underwater robot is used to acquire underwater images, and the images are preprocessed. The images include at least one of four organisms, namely, sea cucumbers, sea urchins, scallops and starfish. The sea cucumbers, sea urchins, scallops and starfish in the images are annotated with four different labels. A data set is generated using the annotated images, and the data set is simply randomly divided to obtain a training set, a validation set and a test set.

3. The real-time target detection method for underwater fishing robots according to claim 1, characterized in that: The step S2 specifically includes: Use the MobileNetV4 network to build and improve the YOLOv9-based backbone network; use the images of the training set in S1 as input, and the process of extracting feature maps is as follows: Unify the size of the input image and input the unified image into the backbone network. The input image passes through the Silence layer of the MobileNetV4 network and is directly output to the next layer of the backbone network, MNV4ConvSmall. The MNV4ConvSmall includes a single convolution layer and five other network layers consisting of convolution modules. After the input image is processed by MNV4ConvSmall, feature maps of different sizes are output.

4. The real-time target detection method for underwater fishing robots according to claim 1, characterized in that: The step S3 specifically includes: FasterNet, GhostConv and dynamic upsampling methods are used to optimize the main branch and auxiliary branch used for feature fusion in YOLOv9, and feature fusion is performed on the feature maps output in S2. The feature maps at all levels after feature fusion are sent to the improved detection head to detect the category and position of the target; In the main branch, a dynamic upsampling method based on point sampling is adopted. In the auxiliary branch, GhostConv is used to optimize the convolution module in the auxiliary branch. The FasterNet Block in the FasterNet optimizes the RepNCSPELAN4 module existing in both the main branch and the auxiliary branch. The FasterNet Block includes some convolution modules, convolution modules, BN layers and ReLU activation functions. The RepNCSPELAN4 module is used for feature extraction and fusion in YOLOv9. Each RepNCSPELAN4 module includes a convolution module and a RepNCSP module. The Bottleneck in the RepNCSP module is optimized by FasterNet Block. The GhostConv improves and optimizes the convolution module in the detection head, sends the feature map after feature fusion to the detection head, and generates the category and detection frame of the corresponding target in the image.

5. The real-time target detection method for underwater fishing robots according to claim 4, characterized in that: The detecting of the type and position of the target includes: S31. Use FasterNet to build FasterRepNCSPELAN4 and optimize the main branch and auxiliary branch, that is, optimize BottleNeck in the RepNCSPELAN4 submodule. S32, optimizing the upsampling operation in the backbone network using a dynamic upsampling method; S33, using GhostConv to optimize the auxiliary branch and the convolutional layers in the detection head; S34, after the network construction is completed, the feature maps of different sizes output in S2 are respectively input into three different network layers in the main branch and the auxiliary branch; the feature maps of different sizes are processed by pooling, convolution, upsampling, splicing, downsampling and segmentation in the main branch to complete feature fusion, and the main feature map is output; the feature maps of different sizes are processed by convolution, segmentation, downsampling and splicing in the auxiliary branch to complete feature fusion, and the auxiliary feature map is output; S35, output the main feature map and the auxiliary feature map output in S34 to the detection head to detect the category and position of the target.

6. The real-time target detection method for underwater fishing robots according to claim 1, characterized in that: The step S4 specifically includes: S41. Calculate the loss function, and use the gradient of the loss function to calculate the gradient of each parameter relative to the loss function through the back propagation algorithm to update the network parameters. The loss function for the classification problem in the model is BCE Loss, which is used to calculate the difference between the predicted category distribution and the true category distribution. The loss function for the regression problem is DFL Loss and CIoU Loss, which is the difference between the position and size of the predicted bounding box and the true bounding box. The calculation method is as follows: DFL(S i ,S i+1 )=-((y i+1 -y)logS i +(y-y i )log S i+1 ) In BCE Loss, N is the batch size; y i represents the true category of the target; p i is the predicted category of the target. In DFL Loss, S i ,S i+1 are the predicted value and the near-predicted value respectively; y, y i ,y i+1 is the actual value, label integral value, and adjacent label integral value; ρ 2 (b,b gt ) is the Euclidean distance between the center points of the predicted box and the real box; c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the real box; α is a weight parameter used to balance the impact of proportional consistency; v is used to measure the consistency of aspect ratio; IoU is the intersection over union ratio of the predicted box and the real box, which is used to measure the overlap of the two boxes; S42, performing iterative training until a preset number of training rounds is reached, completing the training, and obtaining a weight file for the target detection task; S43. Use a pre-built validation set to validate the trained model to determine whether the model reaches the expected performance indicators; export the model that reaches the expected performance indicators.

7. The real-time target detection method for underwater fishing robots according to claim 1, characterized in that: The step S5 specifically includes: The exported model is deployed on the underwater fishing robot system; the underwater fishing robot obtains real-time video stream, and the ground station performs underwater image enhancement on the video stream; the ACE algorithm is used for image enhancement, and the algorithm flow is as follows: Adjust the color of the image and correct the chromatic aberration of the image as follows: Among them, R c It is the intermediate result of the algorithm, which calculates the brightness difference between each point and other points and performs weighted summation according to the distance; c (i)-I c (j) is the brightness difference between two pixels, r(*) is the brightness representation function; Dynamically expand the color-adjusted image, processing each channel of the image separately through linear expansion: O c (p)=round(127.5+s c R c ) Among them, O c (p) is a linear expansion method, s c is [(m c ,0),(M c ,255)], M c =max i (R c (i)), m c =max i (R c (i)); The intermediate result R c Stretch and map to [0,255] to complete image enhancement as follows: Among them, the result R calculated by L(x) in the first stage c Perform normalization processing; After image enhancement, the target detection model deployed in the system detects the targets in the video stream and marks their locations and categories.

Images