A low-resolution underwater image target detection method based on multi-task joint training

Abstract

The application provides a low-resolution underwater image target detection method based on multi-task joint training, comprising the following steps: obtaining a real underwater scene high-resolution image containing an expected underwater target, and labeling the expected underwater target in the image data; performing blurring processing on the obtained real underwater scene high-resolution image; constructing a target detection network for low-resolution underwater images based on multi-task joint training; the target detection network is a convolutional neural network; performing hyperparameter setting and training on the target detection network; after the network training is completed, running the trained target detection network to perform target detection on the underwater image to be detected. The application can effectively realize accurate detection and recognition of low-resolution underwater image data.

Description

Technical Field

[0001] This invention relates to the field of underwater image target detection, and more particularly to a low-resolution underwater image target detection method based on multi-task joint training. Background Technology

[0002] With the development of the times and the advancement of science and technology, exploring and researching the marine environment and resources has become an inevitable trend. Underwater images are one of the most direct means of reflecting the internal conditions of the ocean. Underwater target detection refers to the process of determining the location, size, and category of underwater targets in an input image. With the rise of artificial intelligence and deep learning, underwater target detection has broad application prospects in marine exploration. Currently, target detection based on deep convolutional neural networks can be divided into two-stage and one-stage methods according to its detection stage. The two-stage method first selectively searches for regions of interest in the image, and then classifies and regresses those regions; the one-stage method abandons the extraction of regions of interest in the image, directly extracting image features and then performing classification and regression. The one-stage method is an end-to-end target detection method with a simplified process, lower computational load while ensuring accuracy, and lower equipment requirements, thus making it more applicable in underwater target detection scenarios. Existing underwater image target detection technologies suffer from problems such as low underwater image quality and low algorithm accuracy.

[0003] Reference 1 (Yeh CH, Lin CH, Kang LW, Huang CH, Wang C. "Lightweight deep neural network for joint learning of underwater object detection and color conversion", IEEE Trans. NeuralNetw.Learn.Syst.2021, 1-15.) is a lightweight deep neural network method for joint learning of underwater object detection and color conversion. This method addresses image degradation problems such as color distortion, attenuation, and noise in underwater images. However, it only addresses the degradation of underwater environment images and does not consider the low-resolution imaging caused by the underwater environment. The effectiveness of the lightweight joint learning model in underwater object detection is demonstrated on the Raspberry Pi platform.

[0004] Reference 2 (Lin WH, Zhong JX, Liu S., Li T., Li G. “ROIMIX: Proposal-Fusion Among Multiple Images for Underwater Object Detection”, In Proceedings of the ICASSP 2020: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, ​​Spain, 4-8 May 2020, pp. 2588-2592.) is an underwater target detection method based on feature fusion. This method targets occluded and overlapping targets in underwater scenes, but does not focus on multi-scale targets in the underwater environment. It only uses candidate boxes to fuse different targets to generate new feature information and simulates underwater target training samples, which has strong robustness and high underwater target detection accuracy.

[0005] Reference 3 (Ge Z., Liu ST, Wang F., Li ZM, Sun J. “YOLOX: Exceeding YOLO series in 2021”, arXiv:2107.08430, 2021.) proposes a lightweight one-stage target detection method, YOLOX (You Only Look Once X). This method combines decoupling head, data augmentation, and anchorless detection with the traditional YOLO method. It can be applied to target detection in different scenarios and has high detection accuracy. However, for target detection in special environments, it is still necessary to study the impact of the environment.

[0006] Reference 4 (Zhang M., Xu S., Song W., He Q., Wei Q. “Lightweight underwater object detection based on YOLOv4 and multi-Scale Attentional Feature Fusion”, RemoteSens. 2021, 13, 4706.) introduces an improved attentional feature fusion mechanism and proposes a lightweight object detection method based on MobileNetv2 and YOLOv4 algorithms. It achieves a good trade-off between algorithm accuracy and detection time on various datasets. However, the impact of the underwater environment on images still exists, but this method does not directly solve the problem; it only allows for fine-tuning of the model and cannot handle the degradation of underwater images.

[0007] References:

[0008] [1] Yeh C.H., Lin C.H., Kang L.W., Huang C.H., Wang C. “Lightweight deep neural network for joint learning of underwater object detection and color conversion”, IEEE Trans. Neural Netw. Learn. Syst. 2021, 1 - 15.

[0009] [2] Lin W.H., Zhong J.X., Liu S., Li T., Li G. “ROIMIX: Proposal - fusion among multiple images for underwater object detection”, In Proceedings of the ICASSP 2020: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 4 - 8 May 2020, pp. 2588 - 2592. [3] Ge Z., Liu S.T., Wang F., Li Z.M., Sun J. “YOLOX: Exceeding YOLO series in 2021”, arXiv:2107.08430, 2021.

[0010] [4] Zhang M., Xu S., Song W., He Q., Wei, Q. “Lightweight underwater object detection based on YOLOv4 and multi - scale attentional feature fusion”, Remote Sens. 2021, 13, 4706. Summary of the Invention

[0011] To overcome the problems of color distortion and low resolution in underwater images, as well as the low accuracy and high computational cost of existing underwater target detection methods based on deep convolutional neural networks, this paper proposes a low-resolution underwater image target detection method based on multi-task joint training. This method utilizes underwater image enhancement and super-resolution to process underwater images, and uses a classic feature extraction network and a feature pyramid network with fusion scale to effectively fuse contextual information, enabling the detection probe to effectively extract underwater target feature information and effectively detect underwater targets at multiple scales.

[0012] The technical means employed in this invention are as follows:

[0013] A low-resolution underwater image target detection method based on multi-task joint training includes the following steps:

[0014] Step 1: Acquire a high-resolution image of the real underwater scene containing the desired underwater target, and annotate the desired underwater target in the image data;

[0015] Step 2: Blur the high-resolution image of the real underwater scene obtained in Step 1;

[0016] Step 3: Construct a target detection network for low-resolution underwater images based on multi-task joint training; the target detection network is a convolutional neural network, and the target detection network includes: a color space transformation enhancement module, a shallow residual super-resolution module, and an underwater target detection module;

[0017] Step 4: Train the target detection network from Step 3;

[0018] Step 5: After the network training is complete, run the trained target detection network to perform target detection on the underwater image to be detected.

[0019] Furthermore, in step 2, all the high-resolution images of the real underwater scene acquired are blurred, including the following steps:

[0020] Step 2-1: Perform downsampling on the high-resolution image; for the pixel values ​​of the high-resolution image, sample the pixel values ​​at intervals in the rows and columns, and discard other pixels between the sampled points to reduce the resolution of the high-resolution image;

[0021] Step 2-2: Process the downsampled image using a bicubic upsampling algorithm.

[0022] Furthermore, step 2-2 also includes the following steps:

[0023] Step 2-2-1: Construct the bicubic difference function, as follows:

[0024]

[0025] Where W(x) represents the interpolation of the function variable x, and a represents the kernel of the bicubic operator;

[0026] Step 2-2-2: For the pixel (x, y) to be interpolated, take all neighboring pixels (x, y) within a 4×4 pixel range. i ,y j ); where i,j = 0,1,2,3, for the neighboring pixel (x i ,y j Perform interpolation calculations:

[0027]

[0028] Among them, (x i ,y j ), i,j=0,1,2,3 represent all neighborhood points near pixel (x,y); F represents the pixel value of pixel (x,y); W represents the interpolation of pixel (x,y);

[0029] Step 2-2-3: Obtain the low-resolution image obtained after processing by the bicubic upsampling algorithm, wherein the size of the low-resolution image is the same as the size of the high-resolution image.

[0030] Furthermore, the process of processing the low-resolution underwater image using the color space conversion module in step 3-1 is as follows:

[0031] I low-hsv =RGB2HSV(I low-rgb )

[0032] f rgb =Conv3(Res(SE1(Res(Conv3(Input)))))

[0033] f hsv =HSV2RGB(Conv3(Res(SE2(Res(Conv3(Input1))))))

[0034] Head uie =CS(f rgb +f hsv );

[0035] Among them, I low-rgb Represents a low-resolution underwater image in the RGB color space; I low-hsv This represents a low-resolution underwater image represented in the HSV color space; f rgbThis represents the enhanced output feature map in the RGB color space; f hsv The first line represents the enhanced output feature map under the HSV color space representation; the second line represents the conversion operation of a given image from the RGB color space representation to the HSV color space representation; the third line represents the conversion operation of a given image from the HSV color space representation to the RGB color space representation; the fourth line represents the convolution operation with a kernel size of 3×3 and a stride of 1, followed by batch normalization (BN) and the modified linear unit (ReLU) activation function; the fifth line represents the convolution operation with a kernel size of 1×1 and a stride of 1, followed by batch normalization (BN) and the modified linear unit (ReLU) activation function; the sixth line represents the convolution operation with a kernel size of 3×3 and a stride of 1 and the superposition operation of the double bending activation function; the seventh line represents the head. uie This represents the output image after contrast enhancement of a low-resolution underwater image; Res and SE represent the residual structure operation and channel attention operation, respectively, and are expressed as follows:

[0036] Res(z)=Conv1(z)+Conv3(Conv1(z))

[0037] SE(z)=z×Globalpool(FC(ReLU(FC(Sigmoid(z))));

[0038] Where z represents the function variable, Globalpool represents the global average pooling operation, FC represents the fully connected operation, and ReLU and Sigmoid represent the modified linear unit ReLU activation function and the double-bending Sigmoid activation function, respectively.

[0039] Furthermore, the process of processing the low-resolution underwater image using a shallow residual super-resolution network in step 3-2 is as follows:

[0040] f sr =Conv1(Input)+Res(Res(Res(Res(Input))))

[0041] Head sr =UCS(f sr );

[0042] Among them, f sr Represents the super-resolution output feature map; UCS represents the superposition operation of upsampling with a 3×3 kernel and a stride of 1 convolution and a double-bending activation function; Head sr This represents the output image after resolution enhancement of a low-resolution underwater image.

[0043] Furthermore, step 4 includes the following process:

[0044] Step 4-1: Freeze the underwater target detection module, that is, during the early training of the network, keep all parameters of the underwater target detection module unchanged;

[0045] Step 4-2: The complete low-resolution underwater image target detection network based on multi-task joint training includes two inputs: one is the high-resolution image I of the real underwater scene acquired in step 1. high Another approach involves optimizing image contrast and resolution using a color space conversion enhancement module and a shallow residual super-resolution module, respectively, and then obtaining an enhanced image I with the same size as the high-resolution image through element-wise superposition. en ,Right now:

[0046] I en =f rgb +f hsv +f sr ;

[0047] Step 4-3: Unfreeze the underwater target detection module and extract high-resolution images of real underwater scenes using an open-source feature extraction network. high and enhanced image I en Three feature maps at different scales;

[0048] Step 4-4: The extracted feature images at the three scales are fused by introducing a feature pyramid network with fusion scale, upsampling module, and downsampling module; the fusion process includes: setting the sum of the outputs of the color space transformation network and the shallow residual super-resolution network as the input to the three output scales of the target detection network, denoted as f from largest to smallest. low1 f low2 f low3 The three output scales of the high-resolution image input to the object detection network are denoted as f from largest to smallest. high1 f high2 f high3 The three feature images after upsampling and fusion are designated as F1, F2, and F3 from largest to smallest; the three feature images after downsampling and fusion are designated as F6, F5, and F4 from largest to smallest; and the three output feature images are designated as Head1, Head2, and Head3 from largest to smallest. The principle is as follows:

[0049] F3 = Conv3(Conv1(Conv3(f high3 -f low3 ))

[0050] F2=Conv3(Conv1(Conv3(Contact(upsample(F3),(f high2 -f low2 )))))

[0051] F1=Conv3(Conv1(Conv3(Contact(upsample(F2),(f high1 -f low1 )))))

[0052] F6 = Conv3(Conv1(Conv3(F1)))

[0053] F5=Conv3(Conv1(Conv3(Contact(downsample(F4),F2))))

[0054] F4=Conv3(Conv1(Conv3(Contact(downsample(F5),F3))))

[0055] Head3 = Conv head (F4)

[0056] Head2=Conv head (F5)

[0057] Head1 = Conv head (F6);

[0058] Where upsample represents an upsampling operation with a stride of 2; downsample represents a downsampling convolution operation with a kernel size of 3×3 and a stride of 2; Contact represents a one-dimensional channel concatenation operation; Conv head This refers to the Conv3 and the convolution stacking operation with a kernel size of 3×3 and a stride of 1;

[0059] Steps 4-5: Input the three fused feature images into the classification and regression layer for prediction; input the output feature images Head1, Head2, and Head3 after processing by the feature pyramid network into the three classification and regression layers respectively to obtain the prediction results of the classification and regression layers; then compare the average confidence of the prediction results of the three classification and regression layers, and select the prediction result with the highest average confidence to calculate the information representing the underwater target category and location.

[0060] Steps 4-6: Through iterative training, the underwater target category and location information obtained in Steps 4-5 are compared with the information in the ground truth image to calculate the network loss. The parameters in the underwater target detection module are adjusted autonomously using weights to reduce this loss. When this loss converges to a fixed value, the entire low-resolution underwater image target detection network based on multi-task joint training is trained. The current network now has the ability to autonomously detect underwater targets.

[0061] Furthermore, the annotation involves locating targets in the underwater image using a rectangular bounding box, and labeling the category name of the targets within the bounding box.

[0062] Furthermore, in step 5, after the network training is complete, the target detection network is run to perform target detection on the underwater image to be detected. Specifically, the network automatically identifies the position and size of the target in the underwater image and marks a rectangular box that can completely surround the detected target in the underwater image. At the same time, it predicts the confidence level of the target category within the rectangular box, and the generated confidence level label is placed above the rectangular box. Compared with the prior art, the present invention has the following advantages:

[0063] This invention targets low-resolution underwater images by preprocessing them to obtain clearer underwater images. The preprocessing process includes: a color space conversion network using convolutional modules, residual modules, and attention mechanisms to enhance the low-resolution underwater image; and a shallow residual super-resolution network using continuous convolutional modules to perform super-resolution image processing on the low-resolution underwater image.

[0064] This invention employs a classic network to extract feature images at three scales from underwater images in the target detection network. It introduces a feature pyramid network with fusion scale, convolution module, and upsampling module to process and fuse the selected feature images at the three scales.

[0065] The process of training a multi-task joint training network model in this invention is as follows: First, train the color space network and the shallow residual super-resolution network, making full use of the frozen training; then directly train the object detection network, input it into the three classification and regression layers, and obtain the prediction results of the classification and regression layers. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a diagram showing the overall structure of the multi-task joint training network of this invention.

[0068] Figure 2 This is a structural diagram of the color space conversion network and shallow residual super-resolution network of the present invention.

[0069] Figure 3 This is a structural diagram of the modules of the present invention. (a) is a channel attention structure diagram; (b) is a residual module structure diagram; and (c) is a convolution stacking structure diagram.

[0070] Figure 4 This is a flowchart illustrating the overall training process of the multi-task joint training network of the present invention. Detailed Implementation

[0071] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0072] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover, or otherwise exclusively include, a process, method, system, product, or apparatus that comprises a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0073] like Figure 1-4 As shown, this invention provides a low-resolution underwater image target detection method based on multi-task joint training, comprising the following steps:

[0074] Step 1: Obtain a high-resolution image of the actual underwater scene containing the desired underwater target, and label the desired underwater target in the image data; the labeling is to locate the target in the underwater image with a rectangle and label the category name of the target within the rectangle.

[0075] Step 2: Blur the high-resolution image of the real underwater scene obtained in Step 1.

[0076] Step 3: Construct a target detection network for low-resolution underwater images based on multi-task joint training; the target detection network is a convolutional neural network, and the target detection network includes: a color space transformation enhancement module, a shallow residual super-resolution module, and an underwater target detection module;

[0077] Step 4: Set hyperparameters and train the target detection network described in Step 3 to enable the network to autonomously detect underwater targets; set the relevant hyperparameters and conventional optimization algorithms required for training the target detection network, the shallow residual super-resolution network, and the color space conversion enhancement network; save the weight parameter file when the loss function converges after training (generate a pth file and save it in the project file directory). Preferably, the relevant hyperparameters and conventional optimization algorithms include: hyperparameters such as learning rate, iteration period, decay rate, and batch size; the optimization algorithm refers to data augmentation, which can be directly changed to a data augmentation method. Step 5: After the network training is completed, run the trained target detection network to perform target detection on the underwater image to be detected. As a preferred embodiment, in this application, Step 2...

[0078] All acquired high-resolution images of real underwater scenes were blurred, including the following steps:

[0079] Step 2-1: Perform downsampling on the high-resolution image; for the pixel values ​​of the high-resolution image, sample the pixel values ​​at intervals in the rows and columns, while discarding other pixels between the sampled points to reduce the resolution of the high-resolution image; for example, set the sampling interval to 1, which means that sampling is performed once every 1 pixel, which is ultimately a 2x downsampling operation on the image.

[0080] Step 2-2: Process the downsampled image using a bicubic upsampling algorithm. Step 2-2 further includes the following steps:

[0081] Step 2-2-1: Construct the bicubic difference function, as follows:

[0082]

[0083] Where W(x) represents the interpolation of the function variable x, and a represents the kernel of the bicubic operator;

[0084] Step 2-2-2: For the pixel (x, y) to be interpolated, take all neighboring pixels (x, y) within a 4×4 pixel range. i ,y j ); where i,j = 0,1,2,3, for the neighboring pixel (x i ,y j Perform interpolation calculations:

[0085]

[0086] Where (xi,yj),i,j=0,1,2,3 represent all neighborhood points near pixel (x,y); F represents the pixel value of pixel (x,y); and W represents the interpolation of pixel (x,y).

[0087] Step 2-2-3: Obtain the low-resolution image obtained after processing by the bicubic upsampling algorithm, wherein the size of the low-resolution image is the same as the size of the high-resolution image.

[0088] In a preferred embodiment, the color space conversion enhancement module in step 3 is located in the preprocessing part of the target detection network, and is specifically composed of a residual convolutional network and a channel attention mechanism. Its purpose is to enhance the low-resolution underwater image and improve the contrast of the underwater image so that the target detection network can more easily identify the outline of the target to be detected.

[0089] The shallow residual super-resolution module is located in the preprocessing part of the target detection network. Specifically, it consists of four cascaded residual convolution modules. Its purpose is to perform super-resolution on low-resolution underwater images to improve the clarity of underwater images, so that the target detection network can more easily identify the location of the target to be detected.

[0090] The underwater target detection module, together with the color space transformation enhancement module and the shallow residual super-resolution module, forms a complete low-resolution underwater image target detection network based on multi-task joint training. When a low-resolution underwater image is input into the network, the color space transformation enhancement module and the shallow residual super-resolution module first optimize the image contrast and resolution respectively, and output the enhanced image by element-wise superposition. Then, the underwater target detection module performs target detection on the enhanced image. Specifically, it uses classic feature extraction networks (such as VGG-16, ResNet-50, MobileNetV3, etc.) to extract feature images at three different scales. Then, it uses a feature pyramid network that introduces fusion scale, upsampling module, and downsampling module to fuse the extracted feature images at the three scales. Finally, the three fused feature images are input into the classification and regression layer for prediction.

[0091] The color space conversion module processes low-resolution underwater images as follows:

[0092] I low-hsv =RGB2HSV(I low-rgb )

[0093] f rgb =Conv3(Res(SE1(Res(Conv3(Input)))))

[0094] f hsv=HSV2RGB(Conv3(Res(SE2(Res(Conv3(Input1))))))

[0095] Head uie =CS(f rgb +f hsv )

[0096] Among them, I low-rgb Represents a low-resolution underwater image in the RGB color space; I low-hsv This represents a low-resolution underwater image represented in the HSV color space; f rgb An enhanced output feature map represented in the RGB color space; f hsv This represents the enhanced output feature map in the HSV color space representation; RGB2HSV represents the conversion operation from RGB color space representation to HSV color space representation of a given image; HSV2RGB represents the conversion operation from HSV color space representation to RGB color space representation of a given image; Conv3 represents a convolution with a kernel size of 3×3 and a stride of 1 (followed by batch normalization (BN) and modified linear unit ReLU activation function); Conv1 represents a convolution with a kernel size of 1×1 and a stride of 1 (followed by batch normalization (BN) and modified linear unit ReLU activation function); CS represents the superposition operation of a convolution with a kernel size of 3×3 and a stride of 1 and a double bending activation function; Head uie This represents the output image after contrast enhancement of a low-resolution underwater image, which can be used to visually observe the image generation effect; Res and SE represent the residual structure operation and the channel attention operation, respectively, and are expressed as follows:

[0097] Res(z)=Conv1(z)+Conv3(Conv1(z))

[0098] SE(z)=z×Globalpool(FC(ReLU(FC(Sigmoid(z))))

[0099] Where z represents the function variable, Globalpool represents the global average pooling operation, FC represents the fully connected operation, and ReLU and Sigmoid represent the modified linear unit ReLU activation function and the double-bending Sigmoid activation function, respectively.

[0100] The process of processing low-resolution underwater images using a shallow residual super-resolution network is as follows:

[0101] f sr =Conv1(Input)+Res(Res(Res(Res(Input))))

[0102] Head sr =UCS(f sr );

[0103] Among them, f sr Represents the super-resolution output feature map; UCS represents the superposition operation of upsampling with a 3×3 kernel and a stride of 1 convolution and a double-bending activation function; Head sr This represents the output image after resolution enhancement of a low-resolution underwater image.

[0104] In a preferred embodiment, step 4 includes the following process:

[0105] Step 4-1: Freeze the underwater target detection module, that is, during the early training of the network, keep all parameters of the underwater target detection module unchanged;

[0106] Step 4-2: The complete low-resolution underwater image target detection network based on multi-task joint training includes two inputs: one is the high-resolution image I of the real underwater scene acquired in step 1. high Another approach involves optimizing image contrast and resolution using a color space conversion enhancement module and a shallow residual super-resolution module, respectively, and then obtaining an enhanced image I with the same size as the high-resolution image through element-wise superposition. en ,Right now:

[0107] I en =f rgb +f hsv +f sr ;

[0108] Step 4-3: Unfreeze the underwater target detection module and extract high-resolution images of real underwater scenes using an open-source feature extraction network. high and enhanced image I en Three feature maps at different scales; (e.g., VGG-16, ResNet-50, MobileNetV3, etc., which can be selected arbitrarily according to actual needs).

[0109] Step 4-4: The extracted feature images at the three scales are fused by introducing a feature pyramid network with fusion scale, upsampling module, and downsampling module; the fusion process includes: setting the sum of the outputs of the color space transformation network and the shallow residual super-resolution network as the input to the three output scales of the target detection network, denoted as f from largest to smallest. low1 f low2 f low3 The three output scales of the high-resolution image input to the object detection network are denoted as f from largest to smallest. high1 f high2 fhigh3 The three feature images after upsampling and fusion are designated as F1, F2, and F3 from largest to smallest; the three feature images after downsampling and fusion are designated as F6, F5, and F4 from largest to smallest; and the three output feature images are designated as Head1, Head2, and Head3 from largest to smallest. The principle is as follows:

[0110] F3 = Conv3(Conv1(Conv3(f high3 -f low3 ))

[0111] F2=Conv3(Conv1(Conv3(Contact(upsample(F3),(f high2 -f low2 )))))

[0112] F1=Conv3(Conv1(Conv3(Contact(upsample(F2),(f high1 -f low1 )))))

[0113] F6 = Conv3(Conv1(Conv3(F1)))

[0114] F5=Conv3(Conv1(Conv3(Contact(downsample(F4),F2))))

[0115] F4=Conv3(Conv1(Conv3(Contact(downsample(F5),F3))))

[0116] Head3 = Conv head (F4)

[0117] Head2=Conv head (F5)

[0118] Head1 = Conv head (F6);

[0119] Where upsample represents an upsampling operation with a stride of 2; downsample represents a downsampling convolution operation with a kernel size of 3×3 and a stride of 2; Contact represents a one-dimensional channel concatenation operation; Conv head This refers to Conv3 and the convolution stacking operation with a kernel size of 3×3 and a stride of 1; assuming I high with I enIf the dimensions are all H×W×C, then the corresponding feature image dimensions at three different scales are H / 8×W / 8×128, H / 16×W / 16×256, and H / 32×W / 32×512, respectively.

[0120] Steps 4-5: Input the three fused feature images into the classification and regression layer for prediction; input the output feature images Head1, Head2, and Head3 after processing by the feature pyramid network into the three classification and regression layers respectively to obtain the prediction results of the classification and regression layers; then compare the average confidence of the prediction results of the three classification and regression layers, and select the prediction result with the highest average confidence to calculate the information representing the underwater target category and location.

[0121] Steps 4-6: Through iterative training, the underwater target category and location information obtained in Steps 4-5 are compared with the information in the ground truth image to calculate the network loss. The parameters in the underwater target detection module will be adjusted through autonomous weights to reduce this loss. When the value of this loss converges to a certain fixed value, the entire low-resolution underwater image target detection network based on multi-task joint training is trained.

[0122] In step 5, after the network training is completed, the target detection network is run to detect targets in the underwater image to be detected. Specifically, the network automatically identifies the position and size of the target to be detected in the underwater image, marks a rectangular box that can completely surround the detected target in the underwater image, and predicts the confidence level of the target category within the rectangular box. The generated confidence level is marked above the rectangular box.

[0123] In a preferred embodiment, the underwater target in this application includes scallops, sea cucumbers, starfish, and sea urchins.

[0124] Example 1

[0125] In this embodiment, the underwater image training set is derived from a real underwater scene dataset. This dataset contains 7,543 labeled real underwater optical images, including color shifts and weak contrasts caused by various lighting conditions. It includes four types of targets: sea urchins, starfish, scallops, and sea cucumbers, and also features complex detection scenarios such as target overlap and occlusion.

[0126] The dataset was divided into a training set and test images in a ratio of 8:2.

[0127] Table 1 shows the comparison results of the underwater target detection accuracy and model size of the YOLOV5-Large method and the method of the present invention in the test images.

[0128] Table 1. Statistics on underwater target detection accuracy, model size, and parameter quantity in simulation experiments.

[0129]

[0130] As shown in Table 1, the method of this invention can effectively detect underwater targets. Compared with the YOLOv5-Middle method, when the feature extraction network is ResNet50, the detection effect of starfish, scallops and sea urchins is significantly better, and the average accuracy is improved by 1.09%. When the feature extraction network is MobileNetV3, the detection model is reduced by 41.96% and the number of parameters is reduced by 35.5% while keeping the accuracy basically unchanged.

[0131] The simulation results above fully demonstrate that the method of the present invention can efficiently detect targets in underwater images and perform type identification while ensuring the lightweight nature of the underwater target detection network. The method of the present invention has lower hardware requirements and better overall performance, demonstrating the effectiveness and advancement of the present invention.

[0132] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions of the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A low-resolution underwater image target detection method based on multi-task joint training, characterized in that, Includes the following steps: Step 1: Acquire a high-resolution image of the real underwater scene containing the desired underwater target, and annotate the desired underwater target in the image data; Step 2: Blur the high-resolution image of the real underwater scene obtained in Step 1; Step 3: Construct a target detection network based on multi-task joint training of low-resolution underwater images; the target detection network is a convolutional neural network, and the target detection network includes: a color space transformation enhancement module, a shallow residual super-resolution module, and an underwater target detection module; Step 3 includes the following steps: Step 3-1: Process the low-resolution underwater image using the color space conversion module: ； ； ； )； in, This represents a low-resolution underwater image represented in the RGB color space. This represents a low-resolution underwater image represented in the HSV color space. This represents an enhanced output feature map in the RGB color space. This represents the enhanced output feature map under the HSV color space representation; This represents the conversion operation of a given image from RGB color space representation to HSV color space representation; This represents the conversion operation of a given image from HSV color space representation to RGB color space representation; Indicates the kernel size as Convolution with a stride of 1, followed by batch normalization (BN) and modified linear unit (ReLU) activation function operations; This indicates a convolution with a kernel size of 1×1 and a stride of 1, followed by batch normalization (BN) and modified linear unit (ReLU) activation function operations; CS This represents the superposition operation of a convolution with a kernel size of 3×3 and a stride of 1 and a double-bending activation function; This represents the output image after contrast enhancement of a low-resolution underwater image. Res and SE Residual structure operation and channel attention operation are represented as follows: ； ； in, Represents function variables, This indicates a global average pooling operation. This indicates a fully connected operation. and These represent the modified linear unit ReLU activation function and the double-bending Sigmoid activation function, respectively. Step 3-2: Process low-resolution underwater images using a shallow residual super-resolution network: ； ； in, This represents the super-resolution output feature map; This represents the superposition operation of upsampling with a convolution of kernel size 3×3 and stride 1 and a double-bending activation function; This represents the output image after resolution enhancement of a low-resolution underwater image. Step 4: Train the object detection network from Step 3; Step 4 includes the following process: Step 4-1: Freeze the underwater target detection module, that is, during the early training of the network, keep all parameters of the underwater target detection module unchanged; Step 4-2: The low-resolution underwater image target detection network based on multi-task joint training includes two types of inputs: one is the high-resolution image of the real underwater scene acquired in step 1. I high Another approach involves optimizing image contrast and resolution using a color space conversion enhancement module and a shallow residual super-resolution module, respectively, and then obtaining an enhanced image of the same size as the high-resolution image through element-wise superposition. I en ,Right now: ； Step 4-3: Unfreeze the underwater target detection module and extract high-resolution images of real underwater scenes using an open-source feature extraction network. I high and image enhancement I en Three feature maps at different scales; Step 4-4: The extracted feature images at the three scales are fused by introducing a feature pyramid network with fusion scale, upsampling module, and downsampling module; the fusion process includes: setting the sum of the outputs of the color space transformation network and the shallow residual super-resolution network as inputs to the three output scales of the target detection network, arranged from largest to smallest. f low1 , f low2 , f low3 The three output scales of the high-resolution image input to the object detection network are set from largest to smallest as follows: , , The three feature images after upsampling and fusion are set from largest to smallest as follows: F 1. F 2. F 3. The three feature images after downsampling and fusion are set from largest to smallest as follows: F 6. F 5. F 4. The three output feature images are set from largest to smallest as follows: Head 1. Head 2. Head 3. The principle is as follows: ； ； ； ； ； ； ； ； ； in, This indicates an upsampling operation with a step size of 2; This indicates a downsampling convolution operation with a kernel size of 3×3 and a stride of 2. This indicates a channel concatenation operation with a dimension of one; for and the kernel size is Convolution stacking operation with stride of 1; Steps 4-5: Input the three fused feature images into the classification and regression layer for prediction; output the feature image after processing by the feature pyramid network. , , The data is input into three classification and regression layers to obtain the prediction results of the classification and regression layers. Then, the average confidence scores of the prediction results of the three classification and regression layers are compared, and the prediction result with the highest average confidence score is selected to calculate information representing the underwater target category and location. Steps 4-6: Through iterative training, the underwater target category and location information obtained in Steps 4-5 are compared with the information in the ground truth image to calculate the network loss. The parameters in the underwater target detection module will be adjusted through autonomous weights to reduce this loss. When the value of this loss converges to a certain fixed value, the entire low-resolution underwater image target detection network based on multi-task joint training is trained. Step 5: After the network training is complete, run the trained target detection network to perform target detection on the underwater image to be detected.

2. The low-resolution underwater image target detection method based on multi-task joint training according to claim 1, characterized in that, Step 2 involves blurring all the high-resolution images of the real underwater scene that have been acquired, including the following steps: Step 2-1: Perform downsampling on the high-resolution image; for the pixel values ​​of the high-resolution image, sample the pixel values ​​at intervals in the rows and columns, and discard other pixels between the sampled points to reduce the resolution of the high-resolution image; Step 2-2: Process the downsampled image using a bicubic upsampling algorithm.

3. The low-resolution underwater image target detection method based on multi-task joint training according to claim 2, characterized in that, Step 2-2 also includes the following steps: Step 2-2-1: Construct the bicubic difference function, as follows: ； in, This indicates that for function variables interpolation, Represents the kernel of a bicubic operator; Step 2-2-2: For the pixels to be interpolated... Take the pixel point nearby All neighboring pixels within the pixel range ;in, For the neighboring pixels Perform interpolation calculations: ； in, , Represents pixel ( x , y All neighboring points; Represents pixels Pixel values; Represents pixels Interpolation; Step 2-2-3: Obtain the low-resolution image obtained after processing by the bicubic upsampling algorithm, wherein the size of the low-resolution image is the same as the size of the high-resolution image.

4. The low-resolution underwater image target detection method based on multi-task joint training according to claim 1, characterized in that, The annotation involves using a rectangle to locate targets in the underwater image and labeling the category name of the targets within the rectangle.

5. The low-resolution underwater image target detection method based on multi-task joint training according to claim 1, characterized in that, In step 5, after the network training is completed, the target detection network is run to detect targets in the underwater image to be detected. Specifically, the network automatically identifies the position and size of the target to be detected in the underwater image, marks a rectangular box that can completely surround the detected target in the underwater image, and predicts the confidence level of the target category within the rectangular box. The generated confidence level is marked above the rectangular box.

Images