An end-to-end lightweight underwater target detection method and system

Abstract

The application provides an end-to-end lightweight underwater target detection method and system, belonging to the field of target detection, comprising: frame sampling of underwater video streams at fixed intervals to obtain original images as to-be-detected images; the original images are sent into a differentiable brightness enhancement module and a lightweight texture enhancement module in parallel to generate color correction images and texture enhancement images in sequence, and the original images, the color correction images and the texture enhancement images are input into a spatial correlation gate attention fusion module to obtain a fusion enhancement image; the fusion enhancement image is input into a backbone network composed of a plurality of feature extraction layers connected in series and then connected in series with a spatial pyramid pooling layer to output a multi-scale feature map; the multi-scale feature map is sent into a path aggregation network to obtain a semantic consistent fusion feature map; the fusion feature map is input into a multi-branch detection head together with the output of the spatial pyramid pooling layer to complete class, bounding box and confidence prediction, and then output the final detection result through non-maximum suppression.

Description

Technical Field

[0001] This invention belongs to the field of target detection, and in particular relates to an end-to-end lightweight underwater target detection method and system. Background Technology

[0002] With the advancement of the net-zero development concept, the green development and automated exploration of seabed resources have gradually become a hot topic of research and industry attention in recent years. As a key component supporting the technological realization of this field, underwater image and video target detection technology has received widespread attention. Underwater target detection not only has significant application value in marine energy exploration, pipeline inspection, and ecological monitoring, but also provides a technological foundation for building intelligent and unmanned seabed operation systems. However, due to the unique underwater imaging environment, detection tasks face numerous challenges. The turbidity of seawater, interference from suspended particles, and the strong attenuation and scattering of light in water often result in severe visual distortion problems in the acquired raw images, such as uneven color difference, low contrast, blurring, and loss of detail. These problems significantly reduce the effectiveness of traditional image processing and detection algorithms, limiting their application performance in real-world complex environments.

[0003] To address these issues, researchers have attempted to propose improvements in areas such as image enhancement, feature extraction, and model structure design. Although some methods have achieved good performance on standard test sets, balancing high accuracy with detection speed and lightweight end-to-end system deployment remains a significant technical challenge in current underwater target detection research.

[0004] Based on current research findings in underwater target detection, significant progress has been made in detection accuracy, model structure optimization, and image preprocessing, which can improve the model's ability to process complex underwater images to a certain extent. However, many challenges remain in practical applications. Especially when dealing with massive underwater video streams or high-frequency image data, existing methods often suffer from response delays, increased recognition error rates, and even system stuttering, frame drops, or crashes due to factors such as excessive model parameters, high computational resource consumption, slow inference speed, and limited device processing capabilities. These issues fail to meet the technical requirements of underwater operations for high real-time performance, lightweight deployment, and stable recognition, and also affect the system's application scalability and practicality. Summary of the Invention

[0005] To address the above technical problems, this invention provides an end-to-end lightweight underwater target detection method and system, the specific technical solution of which is as follows:

[0006] An end-to-end lightweight underwater target detection method includes the following steps:

[0007] Step 1: Sample the underwater video stream at fixed intervals to obtain the original image as the image to be detected;

[0008] Step 2: The original image is fed in parallel into the differentiable brightness enhancement module and the lightweight texture enhancement module to generate a color correction map and a texture enhancement map in sequence. The original image, color correction map, and texture enhancement map are then input into the spatially correlated gated attention fusion module to obtain a fused enhancement map.

[0009] Step 3: Input the fusion enhancement map into a backbone network consisting of several feature extraction layers connected in series and then connected in series with spatial pyramid pooling layers, and output a multi-scale feature map.

[0010] Step 4: Feed the multi-scale feature map into the path aggregation network to obtain a semantically consistent fused feature map;

[0011] Step 5: Input the fused feature map and the output of the spatial pyramid pooling layer into the multi-branch detection head to complete the prediction of category, bounding box and confidence, and then output the final detection result after non-maximum suppression.

[0012] An end-to-end lightweight underwater target detection system includes the following modules:

[0013] The sampling module samples the underwater video stream frame by frame at fixed intervals to obtain the original image as the image to be detected.

[0014] The fusion enhancement map generation module feeds the original image into the differentiable brightness enhancement module and the lightweight texture enhancement module in parallel to generate a color correction map and a texture enhancement map in sequence. The original image, color correction map, and texture enhancement map are then input into the spatially correlated gating attention fusion module to obtain the fusion enhancement map.

[0015] The multi-scale feature map generation module takes the fused enhancement map as input to a backbone network consisting of several feature extraction layers connected in series and then connected in series with spatial pyramid pooling layers, and outputs a multi-scale feature map.

[0016] The fusion feature map generation module feeds the multi-scale feature map into the path aggregation network to obtain a semantically consistent fusion feature map.

[0017] The results generation module inputs the fused feature map and the output of the spatial pyramid pooling layer into the multi-branch detection head to complete the prediction of category, bounding box and confidence, and then outputs the final detection result after non-maximum suppression.

[0018] An electronic device includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method.

[0019] A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the method described thereon.

[0020] The present invention has the following beneficial effects:

[0021] This invention addresses the problems of heavy models, slow processing, and complex deployment in traditional underwater target detection systems by proposing an end-to-end lightweight underwater target detection optimization method. This method achieves efficient processing of underwater images from input to detection output through joint optimization of multiple dimensions such as model architecture, data preprocessing, and inference process, ensuring the stability and real-time performance of the system in complex scenarios.

[0022] This invention employs a lightweight image enhancement module in the image input stage, fully considering the characteristics of underwater images and combining a differentiable enhancement mechanism to achieve real-time correction of color distortion, blur noise, and contrast enhancement. Secondly, a low-parameter, high-efficiency convolutional structure is introduced into the backbone network, significantly reducing model complexity while maintaining feature extraction capabilities. Simultaneously, a multi-scale target perception mechanism is integrated into the detection head to enhance the recognition accuracy of small and blurred targets. Finally, end-to-end model compression and quantization techniques are used in the system deployment phase to reduce resource consumption during model runtime, and edge computing devices are combined to achieve localized model inference, reducing data transmission latency. Through these optimization strategies, this invention constructs a lightweight target detection system suitable for complex underwater environments. It not only effectively improves the overall system processing speed and detection accuracy but also significantly enhances the model's operating efficiency and deployment flexibility on resource-constrained platforms, ultimately achieving rapid, stable, and accurate underwater target detection, providing strong technical support for underwater intelligent operation systems.

[0023] The lightweight image enhancement module proposed in this invention includes differentiable brightness enhancement, lightweight texture enhancement, and spatial correlation gated attention fusion network, which can be directly integrated into the front end of the detection network to achieve end-to-end training, image enhancement and target detection in one. Attached Figure Description

[0024] Figure 1 This is a flowchart of the end-to-end target detection method of the present invention.

[0025] Figure 2 These are the original image, the optimized image, and the detection results of this invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above objectives, this invention adopts the following technical solution.

[0027] like Figure 1 As shown, the present invention provides an end-to-end lightweight underwater target detection method, comprising the following steps:

[0028] Step 1: First, sample frames from the continuously acquired video stream by the optical camera at intervals of 15 frames, and extract key frame images as images to be detected.

[0029] Furthermore, the sampled image to be detected is input into the lightweight image enhancement module to perform operations such as color deviation correction, noise suppression and contrast enhancement, and output an enhanced image with higher visual quality.

[0030] Specifically, the lightweight image enhancement module consists of a differentiable brightness enhancement module, a lightweight texture enhancement module, and a spatially correlated gated attention fusion module. First, the image to be detected is input into two parallel processing paths: the differentiable brightness enhancement module and the lightweight texture enhancement module. On one hand, the differentiable brightness enhancement module performs brightness separation and color correction on the image based on Retinex theory, compensating for color deviation and low-light issues in underwater imaging, and outputting a color-corrected image with natural colors and enhanced contrast (denoted as Image 1). On the other hand, the image is simultaneously input into the lightweight texture enhancement module. This module extracts texture and edge features through a low-complexity guided filtering strategy and depthwise separable convolution, effectively alleviating image blur and suppressing background noise, outputting a texture-enhanced image with clear texture (denoted as Image 2). Next, the original image, Image 1, and Image 2 are input into the spatially correlated gated attention fusion module for fusion processing. This module constructs spatially sensitive attention weights to fuse features from the three input images, dynamically adjusting the contribution weights of each path feature according to the importance of the image region, and finally outputting the enhanced image as input to the subsequent object detection network.

[0031] Specifically, the aforementioned differentiable brightness enhancement module aims to simulate the traditional Retinex image enhancement principle. By decomposing the image into a reflection component that reflects the inherent properties of objects and an illumination component that reflects the influence of light, it improves the brightness and color quality of underwater images. This module employs a fully differentiable neural network structure, enabling its integration into the entire system for end-to-end joint training. First, the input image is normalized and transformed into logarithmic space. Then, a shallow convolutional network consisting of a few lightweight convolutional layers is used to extract features from the image and estimate its illumination component. Next, the reflection component is extracted by subtracting the estimated illumination component from the original image in logarithmic space, thereby eliminating the influence of uneven underwater illumination. Subsequently, the reflection map is transformed back into the original image domain, and learnable nonlinear enhancement parameters are introduced to further improve the image's brightness dynamic range and visual contrast. Finally, a color-corrected image with enhanced brightness is output (denoted as Image 1).

[0032] The shallow convolutional network includes a depthwise separable convolution layer with a ReLU function, a 1×1 point convolution layer, and a Sigmoid activation function to control the brightness range.

[0033] Furthermore, the aforementioned lightweight texture enhancement module borrows the idea of ​​guided filtering, using the original image as a guide map to enhance the texture details of the input image. First, the original image is input into the guide map feature convolutional network to extract multi-scale structural information, resulting in an intermediate feature map. Next, low-frequency structural information is extracted from the intermediate feature map, serving as the RGB guide map. Then, based on the local consistency relationship between the RGB guide map and the original image, a texture extraction network separates the smooth structural components and texture detail components. The extracted texture detail components are then subjected to nonlinear mapping and detail enhancement to improve local image contrast and detail, resulting in a high-frequency texture map, i.e., the enhanced texture map. Finally, through a structure-texture fusion module, the RGB guide map and the enhanced high-frequency texture map are fused spatially, outputting an image with clearer texture and stronger detail representation.

[0034] The guide map feature convolutional network includes a 3×3 depthwise separable convolution with ReLU activation, a GhostConv layer for compressing the number of channels, a 5×5 depthwise separable convolutional network with a large kernel and ReLU activation to enhance the receptive field, a 1×1 point convolution layer, and a Sigmoid activation function to control texture details. Finally, it outputs a three-channel RGB guide map.

[0035] The texture extraction network includes a GhostConv layer that increases the number of channels, a 3×3 depthwise separable convolutional network with ReLU activation function, a 3×3 depthwise separable convolutional network with residual connections, a 1×1 point convolutional network for compressing the number of channels, and finally outputs a high-frequency texture map.

[0036] The aforementioned detail enhancement refers to scaling and adjusting the pixel values ​​of high-frequency texture images by multiplying them by weights, adding offsets, and then applying the ReLU function, thereby enhancing local contrast.

[0037] The structure-texture fusion module first combines the original image, the RGB guide map extracted by the guide map feature convolutional network, and the high-frequency texture map after texture extraction and detail enhancement processing, according to the channel dimension. Then, the concatenated result is passed through a lightweight convolutional network to generate a fusion weight map. Finally, the structure map and texture map are weighted and added together according to this fusion weight. The lightweight convolutional network includes one 1×1 convolution layer, one 3×3 convolution layer, and one sigmoid activation function layer.

[0038] Furthermore, the aforementioned spatially correlated gated attention fusion module controls the weights of feature fusion at different spatial locations by combining spatial attention mechanisms with a gating network. First, the original image, color correction image, and texture enhancement image are transformed and normalized using 1×1 convolutions. Then, using the original image as a reference, the spatial correlation between the three feature maps is calculated in the spatial dimension using the self-attention concept, resulting in the spatial correlation weight matrix for color and texture features. Finally, the fused image is fused using a gating network and used as input for object detection.

[0039] The gated network first stitches three feature maps together at the channel level, then inputs the weights of the three branches generated in the gated network, and finally broadcasts them by channel to participate in the weighting. The fused features are then convolved, integrated, and normalized to output the enhanced image.

[0040] The gated network includes a 1×1 convolutional network with ReLU activation function for channel compression, a 1×1 convolutional network for mapping to 3 gated channels, and a Sigmoid activation function layer for normalization operations.

[0041] Step 2: Input the enhanced image into the backbone network for multi-scale feature extraction to obtain feature maps at different scales.

[0042] Specifically, the backbone network includes several feature extraction layers and spatial pyramid pooling layers. The feature extraction layers are connected in series, and the spatial pyramid pooling layers are connected in series at the end of the series-connected feature extraction layers. Each feature extraction layer contains several sequentially connected convolutional modules. Each convolutional module includes a 1×1 convolution layer, a depthwise separable convolution layer, a non-linear activation function layer, and a batch normalization layer. The spatial pyramid pooling layer comprises a first lightweight depthwise separable convolutional network, several cascaded pooling convolutional kernels, a concatenation layer, and a second lightweight depthwise separable convolutional network. The first lightweight depthwise separable convolutional network extracts features from the input feature map and splits the extracted features into two paths. One path passes through several parallel pooling convolutional kernels for progressive pooling and is then fed to the concatenation layer. The cascaded pooling convolutional kernels include 1×1 dilated convolutions, 3×3 dilated convolutions, and 5×5 dilated convolutions. The other path is directly fed to the concatenation layer. The concatenation layer concatenates all received features and then feeds them to the second lightweight depthwise separable convolutional network for further feature extraction to obtain the feature map. Both the first and second lightweight depthwise separable convolutional networks are sub-networks composed of multiple lightweight convolutional modules stacked sequentially. Each lightweight convolutional module includes a cascaded 1×1 convolutional layer, a depthwise separable convolutional layer, a non-linear activation function layer (RelU6), and a batch normalization layer.

[0043] Step 3: Input the feature map output from Step 2 into the path aggregation network for feature fusion. The path aggregation network adopts a feature pyramid structure, including several 1×1 convolutional layers and a top-down multi-scale fusion connection module. The 1×1 convolutional layers are used to compress the channels of feature maps output at different scales in the backbone network to unify their channel dimensions, facilitating subsequent fusion processing. The top-down connection module is used to progressively upsample high-semantic, low-resolution feature maps and fuse them with adjacent low-level feature maps, forming a top-down multi-scale feature propagation path, following the principle of high-level features guiding low-level features. During the fusion process, the output fused feature map at each level is obtained by element-wise summation of the corresponding low-level feature map and the upsampled high-level feature map, followed by edge refinement and semantic enhancement through subsequent 3×3 convolutions, resulting in a multi-scale, semantically consistent fused feature map, improving the detection model's ability to recognize targets at different scales.

[0044] Step 4: Input the fused feature map obtained in Step 3 and the feature map output from the spatial pyramid pooling layer in Step 2 into the detection head network for detection, obtaining a result map containing the target category and location. Then, process these result maps using a non-maximum suppression method to obtain the target detection result map. The detection head network includes several parallel detection branches, each corresponding to a fused feature map input at a specific scale, to achieve accurate detection of targets at different scales. Each branch includes several cascaded 1×1 convolutional layers and 3×3 convolutional layers, used for category prediction, bounding box regression, and confidence prediction, respectively.

[0045] Step 5: Perform structural optimization on the trained object detection network, which includes two stages: model pruning and model quantization, to reduce model complexity and improve inference speed, thereby meeting the deployment requirements on resource-constrained devices.

[0046] The model pruning operation, while maintaining the continuity of the backbone network structure, performs channel-level pruning on several convolutional layers in the backbone network, path aggregation network, and detection head network based on channel importance evaluation results, removing redundant channels with low contribution. This pruning process significantly reduces the number of model parameters and computational overhead, while preserving the original detection accuracy to the greatest extent possible.

[0047] The model quantization operation is performed after pruning, employing a perceptual quantization training strategy to convert floating-point weight parameters and intermediate activation values ​​in the model into low-bit representations. This quantization process includes linear mapping of convolutional layer weights to INT8 representation, and setting a fixed range for activation values ​​and inserting quantization pseudo-operations to replace integer operations during network inference, thereby further reducing model size and improving execution efficiency.

[0048] After the above pruning and quantization processes, the target detection network can achieve lightweight deployment while maintaining high detection accuracy, meeting the requirements of edge devices for real-time performance, low power consumption, and low latency.

[0049] Step 6: Deploy the pruned and quantized model on an embedded or edge computing platform to achieve real-time inference in a low-power, high-efficiency environment. The deployment process includes steps such as model export, inference engine adaptation, hardware loading, and system integration.

[0050] In the model export step, the trained target detection model is converted into a standard intermediate representation format suitable for deployment, preferably the ONNX format, to ensure the model's universality and portability across different platforms.

[0051] In the inference engine adaptation step, based on the computing resources and instruction set structure of the target hardware platform, an adapted deep learning inference framework is selected, including but not limited to TensorRT, TFLite, OpenVINO, or NCNN. The model structure is then processed by operator fusion, memory optimization, and quantization scheduling to improve the execution efficiency and resource utilization of the inference process.

[0052] In the hardware loading step, the inference engine and the target model are deployed together to the edge device. The edge device may include Jetson Nano, Jetson Xavier, Raspberry Pi, RK3588, Cambricon MLU, Ascend Atlas NPU, etc., to meet the performance and power consumption balance requirements in different scenarios.

[0053] In the system integration step, the target detection module is integrated and encapsulated with the image input module, image preprocessing module, post-processing module and control command interface module to form a complete underwater target detection system. It provides a unified interface calling standard, which facilitates deployment and linkage with platforms such as underwater robots, remotely operated buoys and AUVs, thereby improving the overall stability and real-time response capability of the system.

[0054] More specifically, the present invention provides an end-to-end lightweight underwater target detection method, comprising the following steps:

[0055] Step 1: Input the original underwater image into the lightweight image enhancement module for color correction and texture enhancement to obtain the image to be detected.

[0056] Specifically, assuming the original underwater image Includes RGB three channels, size is The original image is first input into a differentiable brightness enhancement module to obtain a color-corrected image. Simultaneously, a lightweight texture enhancement module is used to obtain a texture enhancement map. Finally, the original image Color-corrected images Texture enhancement map The input is a spatially correlated gated attention fusion module, and the fusion output is the image to be detected. .

[0057] Specifically, the differentiable brightness enhancement module first normalizes the image to [0,1] and transforms it to logarithmic space according to the following formula, with the result being:

[0058] ;

[0059] in, , >0 is used to avoid logarithmic zero. Next, Input is fed into the illumination component estimation network, and the output is the estimated illumination map. :

[0060] ;

[0061] ;

[0062] in, This represents depthwise separable convolution. Represents point convolution. This represents the Sigmoid activation function.

[0063] Then, estimate the reflection component according to the following formula:

[0064] ;

[0065] Finally, the reflection map is nonlinearly enhanced and remapped according to the following formula to obtain the color-corrected image. :

[0066] ;

[0067] in, , For learnable parameters, The weighting parameters control the scaling and translation of the input R. The offset parameter controls the amount of translation. The initial value is 1. The initial value is 0, and it will be automatically adjusted through backpropagation during training. exp represents an exponential function with the natural constant as the base, and ReLU represents the ReLU6 activation function.

[0068] Furthermore, the lightweight texture enhancement module first inputs the original image into the guide map feature extraction network to extract multi-scale structures and outputs a structural guide map. :

[0069] ;

[0070] ;

[0071] Next, the original image I and the guiding image G are input into the texture extraction network to extract texture detail maps. :

[0072] ;

[0073] ;

[0074] Then, a non-linear adjustment is performed on the texture detail map T to obtain... :

[0075] ;

[0076] in, , These are learnable parameters. The weighting parameters control the scaling and translation of the input R. The offset parameter controls the amount of translation. The initial value is 1. The initial value is 0, and it will be automatically adjusted through backpropagation during training.

[0077] Finally, the original image I, the structure guidance map G, and the enhanced texture map are combined. spliced ​​as The input to the fusion network yields the texture enhancement map. :

[0078] ;

[0079] ;

[0080] ;

[0081] Specifically, the spatially related gating attention fusion module first combines the original image I and the color-enhanced image according to the following formula. Texture enhancement images Perform feature channel alignment and normalization:

[0082] ;

[0083] ;

[0084] ;

[0085] Next, the normalized original image Based on this, calculate the color enhancement image separately. Texture enhancement images And the original image The greater the similarity between the two positions, the more credible the position is.

[0086] ;

[0087] ;

[0088] in, This refers to the spatial coordinates of the image; ε is a stability parameter, taking the smallest positive number. This is used for numerical stabilization; finally, the weight W for the fusion of the three image features is obtained according to the following formula, where... Finally, the fused graph is output according to the weights. :

[0089] ;

[0090] ;

[0091] Step 2: Combine the fused image obtained in Step 1. Multi-scale feature extraction is performed on the input image using a lightweight backbone network. This lightweight backbone network employs MobileNetV3_Small as its backbone, which features small parameter count and high computational efficiency, making it suitable for image processing tasks in resource-constrained environments. Specifically, the MobileNetV3_Small network first processes the input image... Perform initial convolution operations to compress spatial dimensions and extract low-level texture features, generating feature maps. ,in This is the initial number of channels. Subsequently, image features are fused. Semantic features are extracted sequentially through multiple inverse residual structure modules with different scales and receptive fields. Each inverse residual structure module consists of three parts: extended convolution, depthwise separable convolution, and projective convolution. Some modules embed SE attention mechanisms to enhance channel expressiveness, outputting feature maps at multiple scales. , , The above feature map { } contains spatial structure and semantic information at different resolutions, in The post-stitching spatial pyramid pooling layer enhances the multi-scale receptive field. First, the feature map is input into the first lightweight depthwise separable convolutional network to obtain basic features. Then, it is split into two paths: one path is directly fed to the stitching layer; the other path sequentially passes through parallel 1×1 dilated convolutions, 3×3 dilated convolutions, and 5×5 dilated convolutions to extract features under different receptive fields, and then feeds them into the stitching layer to fuse all features. Finally, the second lightweight depthwise separable convolutional network further extracts and compresses the features to obtain the enhanced high-level feature map. Subsequent input path aggregation networks perform feature fusion and enhancement.

[0092] Step 3: Combine the feature maps of different scales output by the lightweight backbone network MobileNetV3_Small in Step 2. The inputs are sequentially fed into the path aggregation network to perform feature fusion operations, thereby further improving the model's ability to detect targets at different scales.

[0093] The path aggregation network is constructed using a feature pyramid network structure, which mainly includes multiple 1×1 convolutional layers and a top-down multi-scale fusion connection module. The multiple 1×1 convolutional layers are used to process the input feature maps at different scales. Channel compression is performed to unify the number of channels to a preset channel dimension for subsequent step-by-step fusion.

[0094] Next, the top-down multi-scale fusion connection module propagates from high-level features with low resolution and strong semantics to low-level features with high resolution and strong structure. Layer by layer, the high-level feature maps are restored to the size of the adjacent low-level feature maps through upsampling, and element-wise addition and fusion operations are performed to form a fused feature map.

[0095] The fusion process is continued in the manner described above to obtain the final three-layer fused feature map. Each layer of fused features retains both high-level semantic information and low-level structural details, effectively improving the model's ability to express and perceive multi-scale targets.

[0096] Step 4: Input the multi-scale fusion feature map obtained from the path aggregation network in Step 3 and the feature map output from the spatial pyramid pooling layer in Step 2 into the detection head network to perform target detection. Output a prediction result map containing target category information and its spatial location information in the image. After processing the prediction result map using the non-maximum suppression method, the final target detection result map is obtained.

[0097] Specifically, the detection head network includes several parallel detection branches, each corresponding to an input feature map of a specific scale, used to achieve accurate detection of multi-scale targets. Each detection branch includes several cascaded convolutional structures, each consisting of a set of 1×1 convolutional layers and a set of 3×3 convolutional layers, used for target category prediction, bounding box regression, and confidence score estimation, respectively.

[0098] The 1×1 convolutional layer is used to perform channel compression and feature mapping transformation on the input feature map;

[0099] The 3×3 convolutional layer is used to enhance spatial context awareness and further extract the structural information required for localization and classification.

[0100] The category prediction sub-branch is used to output the probability value of the category to which each detection box belongs;

[0101] The bounding box regression sub-branch is used to predict the center point coordinates (x, y), width (w), and height (h) of the target box;

[0102] The confidence prediction sub-branch combines the classification probability and the target existence probability to output the target score.

[0103] The outputs of each detection branch are aggregated during the inference phase, and redundant boxes with high overlap and low confidence are removed by a non-maximum suppression algorithm, ultimately retaining the target detection result image with the highest confidence.

[0104] Step 5: Perform structural optimization on the target detection network model trained in Step 4. The structural optimization includes two operations: model pruning and model quantization.

[0105] The model pruning process refers to trimming several convolutional layers in the image enhancement network, backbone network, path aggregation network, and detector head network according to channel importance to remove redundant computations.

[0106] The model quantization process refers to the use of a perceptual quantization training strategy, which involves inserting simulated quantization operations into the floating-point weights and activation values ​​of the model during the training phase. Weight quantization employs a linear symmetric quantization method, while activation value quantization uses an asymmetric quantization method.

[0107] Step 6: Deploy the pruned and quantized model from Step 5 onto the RK3588 edge computing platform. Specifically, first, export the model to ONNX format, and then use the RKNN Toolkit for model conversion and quantization adaptation to generate an RKNN format model that can run on the RK3588 NPU. Subsequently, load this model and inference engine onto the embedded device, and integrate it with the video input module, preprocessing module, post-processing module, and control command module to build an underwater target detection system that can run on the RK3588 platform.

[0108] Another aspect of the present invention provides an end-to-end lightweight underwater target detection system, comprising the following modules:

[0109] The sampling module samples the underwater video stream frame by frame at fixed intervals to obtain the original image as the image to be detected.

[0110] The fusion enhancement map generation module feeds the original image into the differentiable brightness enhancement module and the lightweight texture enhancement module in parallel to generate a color correction map and a texture enhancement map in sequence. The original image, color correction map, and texture enhancement map are then input into the spatially correlated gating attention fusion module to obtain the fusion enhancement map.

[0111] The multi-scale feature map generation module takes the fused enhancement map as input to a backbone network consisting of several feature extraction layers connected in series and then connected in series with spatial pyramid pooling layers, and outputs a multi-scale feature map.

[0112] The fusion feature map generation module feeds the multi-scale feature map into the path aggregation network to obtain a semantically consistent fusion feature map.

[0113] The results generation module inputs the fused feature map and the output of the spatial pyramid pooling layer into the multi-branch detection head to complete the prediction of category, bounding box and confidence, and then outputs the final detection result after non-maximum suppression.

[0114] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method.

[0115] Another aspect of the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the method described thereon.

[0116] The training process of the end-to-end lightweight underwater target detection model is described below:

[0117] Step 1: Create a dataset.

[0118] Step 1.1: Data Collection. Collect typical underwater datasets of RUIE as sample data, covering the main scenarios of low brightness, occlusion, and high blur.

[0119] Step 1.2: Data Partitioning. To demonstrate the effectiveness of this invention, the data in this embodiment is randomly divided into a training set, a validation set, and a test set in a 6:2:2 ratio. The training set and validation set are used to train the model parameters, and the test set is used to test the model performance, ensuring that the data from the three sets do not overlap.

[0120] Step 1.3: Data Preprocessing. To improve detection efficiency, the model input data is resampled to 640×640. Simultaneously, random cropping, flipping, mixing, and mosaic techniques are used to expand the training dataset, reduce class imbalance, and improve the robustness of the lightweight object detection model.

[0121] Step 2: Train and validate the model. Input the training dataset from Step 1 into the constructed lightweight object detection model for training. The training parameters are set as follows: input image size is 640×640, batch size is 16, optimizer is SGD, initial learning rate is 0.01, and number of iterations is 300, resulting in the trained lightweight object detection model.

[0122] Step 3: Model Export. Export the pt format weight file of the trained lightweight object detection model as an ONNX format lightweight object detection model. After compilation by the toolchain, convert it into an rknn format lightweight object detection model suitable for NPU devices.

[0123] Step 4: Deploy the trained lightweight object detection model to the underwater platform. The underwater platform is equipped with an RK3588 chip. The trained lightweight object detection model is deployed onto a board equipped with the RK3588 chip, and inference is performed by accessing multiple video streams. After processing with a non-maximum suppression algorithm, the target coordinates and class confidence scores are returned.

[0124] Table 1 Performance comparison of the lightweight target detection model of the present invention with YOLOv8-s

[0125] ,

[0126] As shown in Table 1, the end-to-end lightweight underwater target detection method is simple in principle and easy to implement. The lightweight target detection model of this invention is compared with YOLOv8s:

[0127] The number of parameters was reduced from 11.1M to 8.9M, the computational cost was reduced from 28.5GFLOPS to 24.6GFLOPS, and the inference speed was increased from 131FPS to 139FPS.

[0128] like Figure 2 As shown, the original image, color correction image, texture enhancement image, fusion enhancement image, and detection results of this invention are presented. It can be seen that the detection results have high accuracy.

Claims

1. An end-to-end lightweight underwater target detection method, characterized in that, The method comprises the following steps: Step 1, frame sampling is performed on an underwater video stream at a fixed interval to obtain an original image as a to-be-detected image; Step 2, the original image is sequentially sent into a differentiable brightness enhancement module and a lightweight texture enhancement module to generate a color correction image and a texture enhancement image, and the original image, the color correction image and the texture enhancement image are input into a spatial correlation gate attention fusion module to obtain a fusion enhancement image; Step 3, the fusion enhancement image is input into a backbone network composed of a plurality of feature extraction layers connected in series and then connected in series with a spatial pyramid pooling layer to output a multi-scale feature map; Step 4, the multi-scale feature map is input into a path aggregation network to obtain a semantic consistent fusion feature map; Step 5, the fusion feature map and the multi-scale feature map output by the spatial pyramid pooling layer are input into a multi-branch detection head to complete class, bounding box and confidence prediction, and finally output the final detection result after non-maximum suppression; The differentiable brightness enhancement module in step 2 is executed in the following order: the original image is normalized and logarithmized to obtain a logarithmic image; a 3*3 depth separable convolution, a 1*1 point convolution and a Sigmoid function are used to estimate an illumination component; the logarithmic image is subtracted from the reflection component; the reflection component is enhanced by a learnable parameter nonlinearity and exponentially mapped to output a color correction image; the lightweight texture enhancement module in step 2 is executed in the following order: The original image is used as a guide, and a 3*3 depth separable convolution, a GhostConv, a 5*5 depth separable convolution, a 1*1 convolution and a Sigmoid function are used to generate a structure guide image; The original image and the structure guide image are spliced and then sequentially input into a GhostConv, a 3*3 depth separable convolution, a residual 3*3 depth separable convolution and a 1*1 convolution to extract a high-frequency texture image; The high-frequency texture image is adjusted by a learnable parameter nonlinearity to obtain an adjusted high-frequency texture image; The original image, the structure guide image and the adjusted high-frequency texture image are spliced in the channel dimension and then sequentially input into a 1*1 convolution, a 3*3 convolution and a Sigmoid function to generate a fusion weight, and the fusion weight is used to output a texture enhancement image; The spatial correlation gate attention fusion module in step 2 is executed in the following sub-order: The original image, the color correction image and the texture enhancement image are respectively input into a 1*1 convolution and a normalization to obtain an original enhancement image F0, a color enhancement image F1 and a texture enhancement image F2; The original enhancement image F0 is taken as a reference to calculate spatial cosine similarity images Sim1 and Sim2 between the original enhancement image F0 and the color enhancement image F1 and between the original enhancement image F0 and the texture enhancement image F2; The original enhancement image F0, the color enhancement image F1, the texture enhancement image F2, the spatial cosine similarity images Sim1 and Sim2 are spliced and then sequentially input into a 1*1 convolution and a Sigmoid function to generate a three-channel weight, and the fusion enhancement image is output according to the three-channel weight.

2. The end-to-end lightweight underwater target detection method according to claim 1, wherein, The spatial pyramid pooling layer in step 3 is executed in the following order: First, a first lightweight depth separable convolution is used to extract features and divide them into two paths; One path is directly sent to the concatenation layer, and the other path is sent to the concatenation layer after 1x1, 3x3 and 5x5 hollow convolution in parallel; The concatenation result is further fused by a second lightweight depth separable convolution, and a multi-scale feature map is output.

3. The end-to-end lightweight underwater target detection method according to claim 1, wherein, The feature extraction layer includes a plurality of convolution modules connected in sequence, and each convolution module includes a 1x1 convolution layer, a depth separable convolution layer, a nonlinear activation function layer and a batch normalization layer.

4. The end-to-end lightweight underwater target detection method according to claim 1, wherein, The detection head network includes a plurality of parallel detection branches, each branch corresponding to a specific scale of fusion feature map input, realizing detection of targets of different scales.

5. An end-to-end lightweight underwater target detection system, which implements the method of any one of claims 1-4, characterized in that, The method comprises the following steps: A sampling module samples frames at fixed intervals from the underwater video stream to obtain original images as detection images; A fusion enhancement image generation module inputs the original images into a differentiable brightness enhancement module and a lightweight texture enhancement module in parallel to generate color correction images and texture enhancement images in sequence, and inputs the original images, the color correction images and the texture enhancement images into a spatial correlation gate attention fusion module to obtain a fusion enhancement image; A multi-scale feature map generation module inputs the fusion enhancement image into a backbone network composed of a plurality of feature extraction layers connected in series and a spatial pyramid pooling layer connected in series to output a multi-scale feature map; A fusion feature map generation module inputs the multi-scale feature map into a path aggregation network to obtain a semantic consistent fusion feature map; A result generation module inputs the fusion feature map and the spatial pyramid pooling layer output together into a multi-branch detection head to complete class, bounding box and confidence prediction, and then outputs the final detection result through non-maximum suppression.

6. An electronic device, comprising: It comprises: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 4.

7. A computer readable storage medium characterized by Executable instructions are stored thereon, which are executed by a processor to make the processor implement the method of any one of claims 1 to 4.

Images