Underwater multi-task target recognition method and related device

Abstract

This application discloses an underwater multi-task target recognition method and related apparatus, comprising: inputting an acquired underwater image into a pre-trained target recognition model; performing multi-layer feature extraction on the underwater image to obtain multiple enhanced feature maps of different scales; the enhanced feature maps of different scales correspond to target detection ranges of different sizes; performing feature fusion processing on the multiple enhanced feature maps to obtain a fused feature map corresponding to candidate targets in the underwater image; the candidate targets are regions in the image where objects exist, including detection targets and segmentation targets; performing decoupled prediction processing on the fused feature maps to output the category prediction results of the candidate targets, the bounding box position parameters corresponding to the candidate targets, and the instance segmentation mask information corresponding to the candidate targets; classifying and outputting the prediction results according to the candidate target category; wherein, for detection targets, the target bounding box and category information are output, and for segmentation targets, the instance segmentation mask and category information are output.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to underwater multi-task target recognition methods and related devices. Background Technology

[0002] In underwater cleaning scenarios, analyzing underwater obstacles and cleaning areas typically requires simultaneous object detection and instance segmentation to guide the robot's obstacle avoidance and cleaning operations. In related technologies, object detection and instance segmentation are usually performed by two independent models cascaded together: the detection model outputs the bounding boxes of discrete objects, while the segmentation model outputs masks of continuous regions. This results in high computational overhead and inference latency, consequently impacting deployment and operational efficiency on embedded devices. Summary of the Invention

[0003] In view of the above problems, this application provides an underwater multi-task target recognition method and related apparatus to achieve simultaneous target detection and instance segmentation within a single model framework, reducing computational overhead and inference latency, thereby improving real-time processing capabilities on embedded devices. The specific solution is as follows:

[0004] The first aspect of this application provides an underwater multi-task target identification method, including:

[0005] The acquired underwater image is input into a pre-trained target recognition model, and multi-layer feature extraction is performed on the underwater image to obtain multiple enhanced feature maps of different scales; wherein, the scale represents the downsampling factor of the enhanced feature map relative to the underwater image, and the enhanced feature maps of different scales correspond to different target detection ranges;

[0006] Multiple enhanced feature maps are fused to obtain fused feature maps corresponding to candidate targets in the underwater image; the candidate targets are regions in the image where objects exist, including detected targets and segmented targets;

[0007] The fused feature map is subjected to decoupled prediction processing, and the category prediction result of the candidate target, the bounding box position parameters corresponding to the candidate target, and the instance segmentation mask information corresponding to the candidate target are output.

[0008] The prediction results are classified and output according to the candidate target categories; for detected targets, the target bounding box and category information are output, and for segmented targets, the instance segmentation mask and category information are output.

[0009] Optionally, the step of performing multi-layer feature extraction on the underwater image to obtain multiple enhanced feature maps at different scales includes:

[0010] The underwater image is sequentially subjected to convolution, batch normalization, and nonlinear mapping activation to obtain a first initial feature map.

[0011] The first initial feature map is subjected to convolution, batch normalization and nonlinear mapping activation processing at least three times to obtain the second initial feature map, the third initial feature map and the fourth initial feature map, respectively.

[0012] Feature enhancement processing is performed on the first initial feature map, the second initial feature map, the third initial feature map, and the fourth initial feature map to obtain multiple enhanced feature maps of different scales.

[0013] Optionally, the step of performing feature enhancement processing on the first initial feature map, the second initial feature map, the third initial feature map, and the fourth initial feature map to obtain multiple enhanced feature maps of different scales includes:

[0014] The fourth initial feature map is input into the attention enhancement module for attention enhancement processing to obtain the first enhanced feature map;

[0015] The third initial feature map, the second initial feature map, and the first initial feature map are respectively input into the convolution module for convolutional feature extraction to obtain the second enhanced feature map, the third enhanced feature map, and the fourth enhanced feature map.

[0016] Optionally, the step of performing feature fusion processing on multiple enhanced feature maps to obtain a fused feature map corresponding to the candidate target in the underwater image includes:

[0017] The fourth enhanced feature map is upsampled for the first time, and the first upsampling result is concatenated and fused with the third enhanced feature map to obtain the first intermediate fused feature map.

[0018] The first intermediate fused feature map is upsampled a second time, and the second upsampled result is concatenated and fused with the second enhanced feature map to obtain the second intermediate fused feature map;

[0019] The second intermediate fused feature map is upsampled a third time, and the result of the third upsampling is concatenated and fused with the first enhanced feature map to obtain the third intermediate fused feature map.

[0020] The third intermediate fused feature map is concatenated and convolved with the first enhanced feature map to obtain the first scale fused feature map;

[0021] The first scale fusion feature map is downsampled for the first time, and the first downsampling result is concatenated and fused with the second intermediate fusion feature map to obtain the second scale fusion feature map.

[0022] The second scale fusion feature map is downsampled a second time, and the second downsampling result is concatenated and convolved with the first intermediate fusion feature map to obtain the third scale fusion feature map.

[0023] The first-scale fused feature map, the second-scale fused feature map, and the third-scale fused feature map are used together as the fused feature map corresponding to the candidate target.

[0024] Optionally, the target recognition model is obtained through the following steps:

[0025] Construct a training dataset, which includes underwater images and corresponding annotation information, including bounding box annotation information for detected targets and mask annotation information for segmented targets;

[0026] The underwater image is subjected to image processing operations, including at least one of image stitching, image blending, image flipping, and image color perturbation.

[0027] The processed training dataset is input into the target recognition model for forward computation to obtain the target category prediction result, bounding box prediction result, and mask prediction result.

[0028] Based on the target category prediction result, the bounding box prediction result, the mask prediction result, and the annotation information corresponding to the underwater image, a loss function is calculated, wherein classification loss and bounding box regression loss are calculated for detected targets; and classification loss and mask loss are calculated for segmented targets.

[0029] The model parameters are updated according to the loss function until the preset training termination condition is met, thus obtaining the trained target recognition model.

[0030] Optionally, the step of decoupling the prediction process on the fused feature map and outputting the category prediction result of the candidate target, the bounding box position parameters corresponding to the candidate target, and the instance segmentation mask information corresponding to the candidate target includes:

[0031] The fused feature map is input into multiple prediction branches, and the prediction branches include at least a classification branch, a bounding box regression branch, and a segmentation branch.

[0032] The classification branch is used to perform convolution operations on the fused feature map to output the category prediction result of the candidate target;

[0033] The bounding box regression branch is used to perform convolution operations on the fused feature map to output the bounding box position parameters corresponding to the candidate targets.

[0034] The fused feature map is convolved through the segmentation branch to output the mask coefficients or mask features corresponding to the candidate targets;

[0035] Instance segmentation mask information corresponding to the candidate target is generated based on the mask coefficients or the mask features.

[0036] Optionally, the output category prediction results of the candidate targets also include the category confidence score corresponding to each category prediction result; before classifying and outputting the prediction results according to the candidate target categories, the method further includes post-processing of the candidate targets, the post-processing including:

[0037] Based on the bounding box position parameters corresponding to the candidate target, the boundary region of the candidate target is determined;

[0038] Based on the category prediction results of the candidate targets and the category confidence scores corresponding to the category prediction results, the candidate targets are filtered to obtain a first set of candidate targets;

[0039] Based on the boundary regions corresponding to each candidate target in the first candidate target set, non-maximum suppression processing is performed according to the candidate target category to remove redundant candidate targets with overlapping spatial locations, thereby obtaining the second candidate target set;

[0040] The step of classifying and outputting the prediction results based on the candidate target category includes:

[0041] The classification output is based on the candidate target categories in the second candidate target set after nonmaximum suppression processing.

[0042] Optionally, processing the bounding box of the candidate target based on the bounding box position parameters corresponding to the candidate target to obtain the boundary region of the candidate target includes:

[0043] Based on the bounding box position parameters corresponding to the candidate target, the bounding box of the candidate target is decoded to obtain the predicted bounding box of the candidate target;

[0044] For the detected targets among the candidate targets, the predicted bounding box corresponding to the detected target is used as the boundary region of the detected target;

[0045] For segmented targets among the candidate targets, the boundary region of the segmented target is determined based on the instance segmentation mask corresponding to the segmented target, or the predicted bounding box corresponding to the segmented target is used as the boundary region of the segmented target.

[0046] A second aspect of this application provides an underwater multi-task target identification device, comprising:

[0047] The feature extraction unit is used to input the acquired underwater image into a pre-trained target recognition model, perform multi-layer feature extraction on the underwater image, and obtain multiple enhanced feature maps of different scales; wherein, the scale represents the downsampling factor of the enhanced feature map relative to the underwater image, and the enhanced feature maps of different scales correspond to different target detection ranges.

[0048] The feature fusion unit is used to perform feature fusion processing on the multiple enhanced feature maps to obtain a fused feature map corresponding to the candidate target in the underwater image; the candidate target is a region in the image where an object exists, including detected targets and segmented targets;

[0049] The decoupled prediction unit is used to perform decoupled prediction processing on the fused feature map and output the category prediction result of the candidate target, the bounding box position parameters corresponding to the candidate target, and the instance segmentation mask information corresponding to the candidate target.

[0050] The result output unit is used to classify and output the prediction results according to the candidate target categories. Specifically, it outputs the target bounding box and category information for detected targets, and the instance segmentation mask and category information for segmented targets.

[0051] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the underwater multi-task target recognition method described in the first aspect or any implementation thereof.

[0052] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0053] The memory is used to store computer programs;

[0054] The processor is used to execute the computer program so that the electronic device can implement the underwater multi-task target recognition method of the first aspect or any implementation thereof.

[0055] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform the underwater multi-task target identification method described in the first aspect or any implementation thereof.

[0056] The sixth aspect of this application provides an underwater robot, including the electronic equipment described in the fourth aspect above.

[0057] By employing the aforementioned technical solution, the underwater multi-task target recognition method provided in this application obtains multiple enhanced feature maps of different scales through multi-layer feature extraction. This enables the model to simultaneously capture the global semantic information of large-scale targets and the local detail information of small-scale targets, thereby solving the detection difficulty caused by the large differences in target scale in the underwater environment. Then, by performing feature fusion processing on multiple enhanced feature maps, deep semantics and shallow details are combined, enhancing the model's ability to express features of blurred and stacked targets. Furthermore, by decoupling the prediction process, target category, bounding box, and instance segmentation mask information are output simultaneously, allowing a single model to complete the dual tasks that originally required a detection model and a segmentation model in a single forward propagation, significantly reducing computational overhead and inference latency. Finally, classification output is performed based on candidate target categories, outputting bounding boxes for detected targets and masks for segmented targets. In summary, this application replaces the traditional dual-model concatenation scheme with a single-model architecture, significantly reducing the number of parameters and computational load while ensuring detection and segmentation accuracy. This effectively improves the deployment and operation efficiency on embedded devices, meeting the real-time and low-power requirements of embedded devices such as underwater robots. Attached Figure Description

[0058] Figure 1 This is a schematic flowchart of an underwater multi-task target recognition method provided in an embodiment of this application;

[0059] Figure 2 This is a schematic diagram of the structure of an underwater multi-task target identification device provided in an embodiment of this application;

[0060] Figure 3 This is a schematic diagram of a computer device structure provided in an embodiment of this application. Detailed Implementation

[0061] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. Those skilled in the art will understand that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems. The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be interchanged where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or device that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to these processes, methods, products, or devices.

[0062] With the popularization of smart homes and automation technologies, underwater robots have been widely used in routine pool bottom maintenance. Underwater robots are usually equipped with vision modules, which help them avoid obstacles by recognizing debris (such as leaves and pebbles) on the pool bottom, or actively clean continuous areas such as steps and walls, thereby improving cleaning efficiency and intelligence.

[0063] However, the environment at the bottom of a pool is extremely complex, presenting challenges such as extremely small target size (e.g., small stones, broken leaves), poor underwater imaging quality (uneven lighting, turbidity), target stacking and occlusion, and limited computing power on the robot side. Among existing target detection algorithms, YOLOv5 uses CSPDarknet (Cross Stage Partial Darknet) as its backbone network, with C3 (CSP Bottleneck with 3 convolutions) as its core module. However, its gradient flow is only transmitted through the last Bottleneck (a residual module where the channel is first compressed and then restored, usually consisting of two convolutional layers), resulting in severe loss of shallow detail information during deep transmission and insufficient ability to locate small targets. YOLOv8 introduced the C2f module (CSP Bottleneck with 2 convolutions and feature fusion, a cross-stage local module containing two convolutions and feature fusion), which enhances gradient flow through multi-branch connections, reducing the number of parameters by approximately 22% compared to YOLOv5. However, its feature extraction capability for blurred underwater targets remains limited, especially in distinguishing sediments with similar gray levels to the background. While YOLOv10 achieved end-to-end inference without NMS (Non-Maximum Suppression), its accuracy in detecting small targets is still unsatisfactory, and it cannot simultaneously output instance segmentation masks. Furthermore, traditional solutions typically deploy detection and segmentation models separately, doubling the computing power and memory consumption of edge devices, making it difficult to meet real-time requirements. Therefore, the main technical problem this application aims to solve is: how to achieve high-precision target detection and instance segmentation simultaneously with a single model in the complex underwater environment at the bottom of a swimming pool, reducing the number of model parameters and computational load, improving the ability to identify extremely small and blurred targets, and adapting to the real-time inference needs of edge devices.

[0064] To address the aforementioned problems, this application provides an underwater multi-task target recognition method. The underwater multi-task target recognition method of this application will be described in detail below with reference to the accompanying drawings. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating an underwater multi-task target identification method provided in an embodiment of this application. The underwater multi-task target identification method provided in this application may include steps 101 to 103, which are described in detail below.

[0065] 101. Input the acquired underwater images into the pre-trained target recognition model, perform multi-layer feature extraction on the underwater images, and obtain multiple enhanced feature maps of different scales;

[0066] This embodiment first acquires an underwater image of the pool bottom using an underwater camera device, and then feeds this underwater image as input data into a pre-trained target recognition model. The model's backbone network performs multi-level feature extraction operations on the underwater image. Each level progressively reduces the spatial resolution of the feature map and increases the number of channels through basic operations such as convolution, normalization, and nonlinear activation, thereby generating multiple enhanced feature maps at different scales. Here, scale refers to the downsampling factor of each enhanced feature map relative to the spatial size of the original input underwater image, such as 4x, 8x, 16x, or 32x. Enhanced feature maps at different scales have different receptive field sizes. Lower resolution (i.e., higher downsampling factor) enhanced feature maps have larger receptive fields, suitable for detecting larger targets, while higher resolution (i.e., lower downsampling factor) enhanced feature maps have smaller receptive fields, which can retain more detailed spatial location information and are suitable for detecting smaller targets.

[0067] 102. Perform feature fusion processing on multiple enhanced feature maps to obtain fused feature maps corresponding to candidate targets in the underwater image; candidate targets are regions in the image where objects exist, including detected targets and segmented targets;

[0068] Feature fusion employs a bidirectional path combining top-down and bottom-up approaches. First, the lowest-resolution (i.e., largest receptive field) augmented feature map is upsampled to gradually restore spatial resolution, and then concatenated with a higher-resolution augmented feature map at the corresponding scale along the channel dimension. This concatenation then generates an intermediate fused feature map with rich semantic information. Subsequently, this intermediate fused feature map is downsampled and concatenated again with the previously retained augmented feature maps at other scales, resulting in at least two fused feature maps of different resolutions. Each pixel position (i.e., anchor point) in the fused feature map corresponds to a candidate target region in the original underwater image. A candidate target region refers to a local area in the image where an object may exist, specifically including discrete objects that need to be located using bounding boxes (called detection targets) and continuous surfaces that need to be described using pixel-level masks (called segmentation targets).

[0069] 103. Perform decoupled prediction processing on the fused feature map, and output the category prediction result of the candidate target, the bounding box position parameters corresponding to the candidate target, and the instance segmentation mask information corresponding to the candidate target;

[0070] In this embodiment, the multi-task decoupled detection head of the model processes the fused feature maps at different resolutions in parallel. Each detection head contains three independent prediction branches: a classification branch, a bounding box regression branch, and a segmentation branch. The classification branch outputs the probability distribution of each anchor point's position belonging to each preset target category through convolution operations, thereby obtaining the category prediction result of the candidate target. The bounding box regression branch outputs the bounding box position parameters corresponding to each anchor point. The bounding box position parameters are usually represented in the form of offsets relative to the preset anchor boxes, specifically including the horizontal and vertical coordinate offsets of the center point and the scaling of the width and height. The segmentation branch outputs the mask coefficient vector corresponding to each anchor point. The dimension of the mask coefficient vector is the same as the number of preset prototype masks.

[0071] 104. Classify and output the prediction results according to the candidate target categories;

[0072] If a candidate target is predicted to be a detection target (e.g., a discrete object such as a leaf, pebble, drain, step, or escalator), the model only outputs the bounding box coordinates and category information of the target, without generating mask data. If a candidate target is predicted to be a segmentation target (e.g., a continuous area at the bottom of a swimming pool), the model first linearly combines the predicted mask coefficients with K pre-generated prototype masks to generate a pixel-level precision instance segmentation mask, and then outputs the instance segmentation mask and category information, without outputting the bounding box. Through the above classification output strategy, a single model can provide both the location bounding box of a detection target and the fine contour of a segmentation target in the same inference process, avoiding the additional computational overhead of deploying separate detection and segmentation models, and achieving lightweight and highly efficient underwater multi-task target recognition.

[0073] By employing the aforementioned technical solution, the underwater multi-task target recognition method provided in this application obtains multiple enhanced feature maps of different scales through multi-layer feature extraction. This enables the model to simultaneously capture the global semantic information of large-scale targets and the local detail information of small-scale targets, thereby solving the detection difficulty caused by the large differences in target scale in the underwater environment. Then, by performing feature fusion processing on multiple enhanced feature maps, deep semantics and shallow details are combined, enhancing the model's feature representation ability for blurred and stacked targets. Furthermore, by decoupling the prediction process, target category, bounding box, and instance segmentation mask information are output simultaneously, allowing a single model to complete the dual tasks that originally required a detection model and a segmentation model in a single forward propagation, significantly reducing computational overhead and inference latency. Finally, classification output is performed based on candidate target categories, outputting bounding boxes for detected targets and masks for segmented targets. In summary, this application replaces the traditional dual-model concatenation scheme with a single-model architecture, significantly reducing the number of parameters and computational load while ensuring detection and segmentation accuracy. This effectively improves the deployment and operation efficiency on embedded devices, meeting the real-time and low-power requirements of embedded devices such as underwater robots.

[0074] In one possible implementation, multi-layer feature extraction is performed on the underwater image to obtain multiple enhanced feature maps at different scales, including: sequentially performing convolution, batch normalization, and nonlinear mapping activation processing on the underwater image to obtain a first initial feature map; repeating the convolution, batch normalization, and nonlinear mapping activation processing on the first initial feature map at least three times to obtain a second initial feature map, a third initial feature map, and a fourth initial feature map, respectively; and performing feature enhancement processing on the first initial feature map, the second initial feature map, the third initial feature map, and the fourth initial feature map to obtain multiple enhanced feature maps at different scales.

[0075] In this embodiment, the backbone network of the model first preprocesses and performs preliminary feature extraction on the input underwater image. For example, assuming the input image size is 640×640×3 (height×width×number of channels), the backbone network first slices the image using the Focus module, dividing it into a 2×2 grid. The four pixels within each grid are then stitched together along the channel dimension, reducing the spatial resolution to half (320×320) while increasing the number of channels to four times (12), resulting in a 320×320×12 intermediate feature map. Subsequently, a standard convolutional layer performs a convolution operation on this intermediate feature map, outputting a first initial feature map with 64 channels and a spatial resolution of 160×160. During this process, each convolution operation is followed by batch normalization to accelerate model convergence and stabilize the training process. Then, a nonlinear mapping activation function (e.g., SiLU or ReLU) is used to perform a nonlinear transformation on the features, enabling the model to extract richer image feature information.

[0076] The backbone network then continues to progressively downsample and abstract the first initial feature map through multiple cascaded stages. Specifically, the basic processing module, which includes convolution, batch normalization, and nonlinear mapping activation, is repeatedly executed on the first initial feature map (size 160×160×64) at least three times. Each processing reduces the spatial resolution of the feature map to half of its original value (i.e., successively becoming 80×80, 40×40, and 20×20), while the number of channels increases accordingly (e.g., successively becoming 128, 256, and 512), thereby generating the second initial feature map (80×80×128), the third initial feature map (40×40×256), and the fourth initial feature map (20×20×512), respectively. In the actual network architecture, each stage consists of multiple stacked C3k2 modules. Each C3k2 module uses dynamic convolution, allowing the convolution kernel weights to adaptively adjust according to the content of the input feature map, better adapting to the scale changes of small underwater targets. Simultaneously, all standard 3×3 convolutions in the backbone network are replaced with depthwise separable convolutions. It should be noted that the formula for calculating the number of parameters is... .when =3、 =128、 When the coefficient is 128, the standard 3×3 convolution has 147,456 parameters, while the depthwise separable convolution has 17,536 parameters, which is about 11.9% of the former. Therefore, the replaced target recognition model can maintain feature extraction capability while reducing the computational burden.

[0077] After obtaining four initial feature maps at different scales, feature enhancement processing is further performed on each feature map. For the fourth initial feature map (20×20×512) with the lowest resolution and richest semantic information, a C2PSA attention module is inserted into the output path. The C2PSA attention module combines channel attention and spatial attention, which can capture global contextual information through multi-scale pooling, and weight important features along the channel dimension and enhance key regions along the spatial dimension, thereby outputting the first enhanced feature map. For the higher-resolution third, second, and first initial feature maps (40×40×256, 80×80×128, and 160×160×64), the model performs convolutional enhancement processing through the C3k2 module. Specifically, each C3k2 module first splits the input feature map into two branches along the channel dimension. One branch undergoes convolution, normalization, and activation operations, and an attention mechanism is introduced for feature weighting. This is then fused with the other branch, and finally, the enhanced feature maps are output through channel integration, denoted as the second, third, and fourth enhanced feature maps, respectively. Through this differentiated enhancement processing, the model ensures that deep features possess global semantic awareness while enhancing the sensitivity of shallow features to spatial details, thereby obtaining four higher-quality enhanced feature maps.

[0078] In one possible implementation, feature enhancement processing is performed on multiple initial feature maps to obtain multiple enhanced feature maps, including: inputting the fourth initial feature map into the attention enhancement module for attention enhancement processing to obtain the first enhanced feature map; and inputting the third, second, and first initial feature maps into the convolution module for convolutional feature extraction to obtain the second, third, and fourth enhanced feature maps.

[0079] In this embodiment, the feature enhancement process first targets the fourth initial feature map, which has the lowest resolution and the richest semantic information. The fourth initial feature map typically has a spatial resolution of 20×20 and 512 channels. Each pixel in this map corresponds to a large receptive field in the original image, making it suitable for capturing global contextual information. The model inputs this fourth initial feature map into an attention enhancement module (e.g., a C2PSA module) for attention enhancement processing. The attention enhancement module internally includes a multi-scale pooling structure, a channel attention submodule, and a spatial attention submodule. The module first performs parallel pooling operations on the input feature map using multiple pooling kernels of different sizes (e.g., 5×5, 9×9, 13×13) to obtain global contextual features for different receptive fields. Then, these multi-scale pooling results are concatenated along the channel dimension and sequentially passed through the channel attention submodule and the spatial attention submodule to calculate the importance weight of each feature channel and the importance weight of each spatial location, respectively. Finally, the calculated attention weights are multiplied element-wise with the original input feature map to perform weighted adjustments, ensuring the model focuses on channels and regions that contribute significantly to target recognition while suppressing interference from background noise and water turbidity. After the aforementioned attention enhancement processing, the module outputs an enhanced feature map, denoted as the first enhanced feature map. This first enhanced feature map maintains the same spatial resolution and number of channels as the fourth initial feature map, making the features more conducive to representing the overall information in the image.

[0080] For the higher-resolution third initial feature map (e.g., 40×40×256), second initial feature map (e.g., 80×80×128), and first initial feature map (e.g., 160×160×64), different enhancement strategies are applied to input them into the convolutional module for convolutional feature extraction. The convolutional module here can be a dynamic convolutional module (such as the C3k2 module), which automatically adjusts the convolutional parameters based on the input feature map to adapt to different features. Taking the first initial feature map as an example, this feature map retains rich spatial detail information and is suitable for detecting extremely small targets (such as leaf fragments or pebbles with an area smaller than 32×32 pixels). The dynamic convolution module first splits the input feature map into two branches along the channel dimension. One branch undergoes convolution, batch normalization, and non-linear activation operations sequentially, with the kernel weights predicted in real-time by a lightweight sub-network based on the input features, thus adaptively adjusting the receptive field and feature extraction mode. The other branch undergoes an identity mapping or lightweight transformation. The output feature maps from the two branches are then fused (e.g., element-wise addition or channel concatenation), and finally output after channel integration. The model processes the third, second, and first initial feature maps in this way, resulting in the second, third, and fourth enhanced feature maps, respectively. The spatial resolutions of these three enhanced feature maps increase sequentially (40×40, 80×80, and 160×160), making them suitable for detecting large, medium, and very small targets, respectively.

[0081] In one possible implementation, feature fusion processing is performed on multiple enhanced feature maps to obtain a fused feature map corresponding to the candidate target in the underwater image. This includes: performing a first upsampling on the fourth enhanced feature map, concatenating and convolving the first upsampling result with the third enhanced feature map to obtain a first intermediate fused feature map; performing a second upsampling on the first intermediate fused feature map, concatenating and convolving the second upsampling result with the second enhanced feature map to obtain a second intermediate fused feature map; and performing a third upsampling on the second intermediate fused feature map, concatenating and convolving the third upsampling result with the first enhanced feature map to obtain a third... Intermediate fusion feature map; the third intermediate fusion feature map is concatenated and convolved with the first enhanced feature map to obtain the first scale fusion feature map; the first scale fusion feature map is downsampled for the first time, and the first downsampling result is concatenated and convolved with the second intermediate fusion feature map to obtain the second scale fusion feature map; the second scale fusion feature map is downsampled for the second time, and the second downsampling result is concatenated and convolved with the first intermediate fusion feature map to obtain the third scale fusion feature map; the first scale fusion feature map, the second scale fusion feature map, and the third scale fusion feature map are used together as the fusion feature map corresponding to the candidate target.

[0082] The model first performs an upsampling operation on the fourth augmented feature map, which has the lowest resolution and the richest semantic information. For example, the spatial resolution of the fourth augmented feature map is 20×20, while the resolution of the third augmented feature map is 40×40. The fourth augmented feature map is upsampled to 40×40 using methods such as bilinear interpolation or transposed convolution to match the spatial size of the third augmented feature map. Then, the upsampled feature map and the third augmented feature map are concatenated along the channel dimension (Concat), with the number of channels in the concatenated feature map being the sum of the number of channels in both maps. Next, convolutional fusion is performed on this concatenated feature map (e.g., through a convolutional fusion module (CBS, convolutional layer (Conv) + batch normalization layer (BatchNormalization) + non-linear activation function) to compress and integrate the high-dimensional channel information using learnable convolutional kernels, generating a first intermediate fused feature map. Therefore, this first intermediate fused feature map retains the high-level semantic information from the fourth augmented feature map while incorporating the medium-scale details from the third augmented feature map.

[0083] After obtaining the first intermediate fused feature map, the model performs a second upsampling operation on it. The spatial resolution of the first intermediate fused feature map is 40×40, and the resolution of the second enhanced feature map is 80×80. The first intermediate fused feature map is upsampled to 80×80 to match the size of the second enhanced feature map, and then concatenated along the channel dimension. The concatenated feature map passes through another CBS module, where convolution operations compress the number of channels to a preset dimension to obtain the second intermediate fused feature map. This feature map has a spatial resolution of 80×80. The feature information of the second intermediate fused feature map comes from the fusion of deep and mid-level features, while also incorporating higher-resolution shallow details, which helps in detecting medium-scale targets.

[0084] The model then performs a third upsampling operation on the second intermediate fused feature map. The resolution of the second intermediate fused feature map is 80×80, while the resolution of the first enhanced feature map is 160×160. The second intermediate fused feature map is upsampled to 160×160, concatenated with the first enhanced feature map along the channel dimension, and then processed by a convolutional fusion module to obtain the third intermediate fused feature map. This feature map has a spatial resolution of 160×160, belonging to a high-resolution feature layer. It contains fusion information at all scales from deep to shallow layers, preserving rich spatial location details, and is suitable for detecting extremely small targets (such as leaf fragments or pebbles with an area smaller than 32×32 pixels).

[0085] After completing the three top-down upsampling fusion operations described above, the model further performs an additional feature fusion operation: directly concatenating and convolving the third intermediate fused feature map with the original first enhanced feature map. Specifically, the third intermediate fused feature map (160×160) and the first enhanced feature map (160×160) have the same spatial resolution, but the former has undergone multiple upsampling and fusion, resulting in richer semantic information; the latter, as the shallowest original enhanced feature map, retains the purest detail information. After concatenating the two, they are passed through the convolutional fusion module again to obtain the first-scale fused feature map (denoted as P2). The resulting first-scale fused feature map integrates the original shallow details and deep semantics, and can be used to detect extremely small targets.

[0086] After completing the top-down path, the model executes a bottom-up path to further fuse multi-scale information and generate detection feature layers suitable for targets of different scales. First, the first-scale fused feature map (P2, resolution 160×160) undergoes a downsampling operation, for example, by reducing the resolution to 80×80 through a convolution with a stride of 2, making it the same size as the second intermediate fused feature map (resolution 80×80). Then, the downsampling result is concatenated with the second intermediate fused feature map along the channel dimension and processed by a convolutional fusion module to obtain the second-scale fused feature map (denoted as P3). This second-scale fused feature map integrates information from high-resolution details and medium-resolution semantics, making it suitable for detecting medium-scale targets.

[0087] The model then performs a second downsampling operation on the second-scale fused feature map (P3, resolution 80×80), reducing its resolution to 40×40 to match the size of the first intermediate fused feature map (resolution 40×40). The downsampled result is then concatenated with the first intermediate fused feature map and processed by a convolutional fusion module to obtain the third-scale fused feature map (denoted as P4). This third-scale fused feature map has a resolution of 40×40 and a large receptive field, making it suitable for detecting large-scale targets (such as steps, escalators, etc.). Thus, the model generates three fused feature maps of different resolutions through two bottom-up downsampling fusion operations.

[0088] In one possible implementation, the target recognition model is trained through the following steps: Constructing a training dataset, which includes underwater images and corresponding annotations, including bounding box annotations for detected targets and mask annotations for segmented targets; performing image processing operations on the underwater images, including at least one of image stitching, image blending, image flipping, and image color perturbation; inputting the processed training dataset into the target recognition model for forward computation to obtain target category prediction results, bounding box prediction results, and mask prediction results; calculating a loss function based on the target category prediction results, bounding box prediction results, and mask prediction results and the corresponding annotations of the underwater images, wherein classification loss and bounding box regression loss are calculated for detected targets; classification loss and mask loss are calculated for segmented targets; updating the model parameters according to the loss function until a preset training termination condition is met, thus obtaining the trained target recognition model.

[0089] This application embodiment uses an underwater high-definition camera deployed at the bottom of a swimming pool to capture raw videos in various real-world scenarios and extracts image frames from the videos as training images. Furthermore, to enhance the model's generalization ability, the captured scenarios need to cover different environmental conditions, specifically including: Regarding light intensity, scenarios are categorized as strong light (pool surface light intensity greater than 1000 lux, such as sunny midday), weak light (200 to 500 lux, such as cloudy days or evening), and dark light (less than 50 lux, such as nighttime or deep water); regarding water turbidity, scenarios are categorized as clear water (turbidity value less than 1 NTU), slightly turbid (1 to 5 NTU), and heavily turbid (5 to 10 NTU); regarding target density, scenarios are categorized as sparse (0 to 2 targets per image), medium-density (3 to 5 targets), and dense (6 or more targets). For each acquired underwater image, a hybrid annotation strategy is used to generate corresponding annotation information: for detection-type targets (such as discrete objects like leaves, drains, pebbles, steps, and ladders), their bounding boxes are annotated and recorded as normalized center coordinates, width, and height; for segmentation-type targets (such as continuous areas at the bottom of a swimming pool), pixel-level instance segmentation masks are annotated and recorded as polygon point sets. The annotation file for the same image contains both bounding box and mask information, stored in different fields or distinguished by category IDs.

[0090] Before inputting training images into the model, online data augmentation processing is required for the original images and their annotation information to expand the diversity of training samples. Specifically, data augmentation operations include, but are not limited to, the following: image stitching (Mosaic), which involves randomly selecting four training images, scaling them according to a certain ratio, and stitching them into a new image, while adjusting the bounding boxes and mask coordinates of targets in each image accordingly; image mixing (MixUp), which involves linearly superimposing two images at the pixel level according to a certain weighting coefficient and fusing the corresponding annotation information with soft labels; image flipping, including horizontal and vertical flipping, to enhance the model's robustness to changes in orientation; and image color perturbation, which involves randomly shifting brightness, saturation, and hue to simulate changes in underwater lighting and water color shift. In actual training, one or more of the above operations can be selected as needed and randomly applied to the images in each training batch with a preset probability.

[0091] The enhanced training images are then input into a pre-built target recognition model for forward propagation. The model's backbone and neck network, following the aforementioned multi-layer feature extraction and fusion processes, generate multi-scale fused feature maps. Subsequently, each detection head decouples and predicts the fused feature maps, outputting results from three branches: the classification branch outputs the probability distribution of each anchor point's position belonging to each preset category, i.e., the target category prediction result; the regression branch outputs the bounding box position parameters corresponding to each anchor point (usually the center coordinate offset and width / height scaling relative to the anchor box); and the segmentation branch outputs the mask coefficient vector corresponding to each anchor point. In this way, through a single forward computation, the model simultaneously obtains the category, bounding box, and mask information of all candidate targets in the entire image, achieving end-to-end multi-task prediction.

[0092] Then, the predicted results are compared with the actual image annotations to calculate the loss function and guide the update of the model parameters. Since this model handles both detection and segmentation tasks, the loss function adopts a task-branch design: for detected targets in the annotations, classification loss and bounding box regression loss are calculated. The classification loss uses Focal loss, and the classification loss function expression is as follows: ,in =2、 =0.25, This represents the probability that the model predicts the target belongs to the true class (positive class), and its value ranges from 0 to 1. The closer the probability is to 1, the more accurate the prediction. The classification loss function can reduce the contribution of a large number of easily classified negative samples (background) to the loss, making the model pay more attention to difficult-to-classify positive samples (foreground), thereby alleviating the problem of a severe imbalance between the number of foreground and background samples in the detected target class. The bounding box regression loss uses CIoU loss (Complete IoU Loss), and the CIoU loss function expression is: ,in This is the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. This is the Euclidean distance between the center point of the predicted bounding box and the center point of the ground truth bounding box. Indicates the center coordinates of the prediction box; Represents the center coordinates of the true bounding box; It is the diagonal length of the smallest closure region that simultaneously contains both the predicted bounding box and the ground truth bounding box; A parameter used to measure aspect ratio consistency; The weights are preset. CIoU loss adds centroid distance and aspect ratio consistency penalties to the traditional IoU loss, enabling the predicted bounding box to more accurately fit the real target. For segmented targets in the labeled information, classification loss and masking loss are calculated. The classification loss also uses Focal loss, while the masking loss is a weighted combination of Dice loss and Binary Cross-Entropy (BCE) loss. The Dice loss measures the overlap between the predicted mask and the ground truth mask. The specific expression for the Dice loss is: Where A and B represent the pixel sets of the predicted mask and the ground truth mask, respectively; the binary cross-entropy loss (BCE) calculates the classification error independently for each pixel, and its expression is: ,in Total number of pixels For true labels (0 or 1). This is for predicting probabilities. Combining both allows the model to simultaneously consider overall region overlap and local boundary accuracy, resulting in smoother and more accurate mask boundaries. For detection-type targets, the mask loss coefficient... Setting it to 0 means that mask loss is not calculated. The total loss function is... ,in This represents the sum of classification losses for all targets (including detection and segmentation classes); This represents the bounding box regression loss only for detected targets; This represents the mask loss only for segmented targets; and Preset weighting coefficients, typically set to 0.05 and 0.1, are used to balance the contributions of each loss term. Through this loss calculation method, the model can simultaneously optimize both detection and segmentation tasks during the same training process without interference. No loss is calculated for background regions that are neither detection nor segmentation categories.

[0093] After calculating the total loss function, an optimizer is used to backpropagate and iteratively update the model parameters. In this embodiment, the AdamW optimizer can be used, with an initial learning rate set to 0.01. A cosine annealing scheduling strategy is employed to dynamically adjust the learning rate, enabling the model to converge quickly in the early stages of training and fine-tune it in the later stages. Each training batch size is set to 32, and the model iterates for a total of 300 epochs on the training dataset. After each iteration, the model's detection accuracy (e.g., mAP) and segmentation accuracy (e.g., mask IoU) are evaluated on the validation set. When the validation accuracy no longer improves or reaches the preset maximum number of epochs for several consecutive epochs, training is stopped, and the current model parameters are saved as the final trained object recognition model.

[0094] In one possible implementation, the fused feature map is decoupled for prediction processing, outputting the candidate target's category prediction result, the bounding box position parameters corresponding to the candidate target, and the instance segmentation mask information corresponding to the candidate target. This includes: inputting the fused feature map into multiple prediction branches, each including at least a classification branch, a bounding box regression branch, and a segmentation branch; performing a convolution operation on the fused feature map through the classification branch to output the candidate target's category prediction result; performing a convolution operation on the fused feature map through the bounding box regression branch to output the bounding box position parameters corresponding to the candidate target; performing a convolution operation on the fused feature map through the segmentation branch to output the mask coefficients or mask features corresponding to the candidate target; and generating instance segmentation mask information corresponding to the candidate target based on the mask coefficients or mask features.

[0095] The model performs decoupled prediction processing on multiple fused feature maps at different resolutions. Taking the previously generated first-scale fused feature map (P2), second-scale fused feature map (P3), and third-scale fused feature map (P4) as examples, an independent multi-task detection head is connected to each resolution fused feature map. Each detection head adopts a decoupled structure, that is, it is divided into three parallel and independent prediction branches: a classification branch, a bounding box regression branch, and a segmentation branch. These three branches share the same input fused feature map, but each uses different convolutional layers for feature transformation, thereby learning the relevant features of the target category, target location, and target mask respectively. Each branch can focus on its own task, avoiding feature interference between multiple tasks and improving the accuracy and efficiency of prediction.

[0096] The classification branch first performs several convolutional operations on the input fused feature map, typically including one or more 1×1 convolutional layers, to map the number of channels in the feature map to the dimension of the number of classes. Taking the P2 detector as an example, the input feature map size is 160×160×C (C is the number of channels). After the convolution by the classification branch, the output is a feature map with a dimension of 160×160×C. The characteristic tensor, where The preset total number of target categories (including detection targets, segmentation targets, and background) is used. Each spatial location (i.e., each anchor point) in the output feature map corresponds to a candidate region in the original image. The values ​​on the channel dimension of the output feature map represent the unnormalized scores (logits) of the candidate region belonging to each category. In practical applications, these scores are usually converted into probability values ​​using the sigmoid function in subsequent processing to obtain the category prediction result for each candidate target. For detection heads with different resolutions, the structure of the classification branch is the same, but due to the different feature map sizes, the corresponding anchor point densities are also different. The high-resolution P2 detection head has more anchor points and is more suitable for detecting densely distributed small targets.

[0097] The bounding box regression branch also performs convolution operations on the input fused feature map. Its output channel count is typically 4×A, where A is the number of anchor boxes preset for each anchor point, and 4 represents the four positional parameters that need to be predicted for each bounding box (i.e., the center point x-coordinate offset, the center point y-coordinate offset, the width scaling factor, and the height scaling factor). Taking the P3 detector as an example, the feature map size is 80×80, and the preset anchor box size set for each anchor point is [10, 14, 16, 22, 28, 36] pixels (represented by width and height), with a number of anchor boxes A=6. Therefore, the feature map dimension output by the bounding box regression branch is 80×80×24. Each set of four values ​​in the output represents the offset of the predicted bounding box at the corresponding anchor point relative to the preset anchor boxes. In the subsequent decoding process, by applying these offsets to the initial position and size of the anchor boxes, the true bounding box coordinates of the candidate target in the original image coordinate system can be restored. For different resolution detection heads, the anchor frame size is set to match the receptive field of the feature map: the P2 detection head is responsible for very small targets (160×160, area less than 32×32 pixels), with the smallest anchor frame size; the P3 detection head is responsible for medium targets (80×80), with anchor frame sizes of [10, 14, 16, 22, 28, 36] pixels, and the anchor frame size is centered; the P4 detection head is responsible for large targets (40×40), with the largest anchor frame size (anchor frame size can be [30, 45, 60, 80, 100, 120] pixels), so that the model can effectively detect targets of various scales at the bottom of the pool, from tiny pebbles to large steps.

[0098] The segmentation branch performs convolution operations on the input fused feature map, outputting A×K channels, where A is the number of anchor boxes at each anchor point, and K is the number of pre-defined prototype masks. Prototype masks are a set of pre-learned, fixed-size mask primitives, typically obtained through model training, and the number K can be selected between 16 and 64 (e.g., K=32). Taking the P4 detector as an example, with a feature map size of 40×40, an anchor box count of A=6, and K=32, the feature map output by the segmentation branch has a dimension of 40×40×192. Each set of K values ​​in the output (i.e., the mask coefficient vector) represents the combined weights of the candidate target at the corresponding anchor point and each prototype mask. The segmentation branch does not directly output the complete instance mask, but instead outputs these lightweight coefficients, significantly reducing computational cost. For detection classes that do not require segmentation, the output of the segmentation branch is zeroed out by the corresponding loss function during training but ignored during inference.

[0099] After obtaining the mask coefficients output by the segmentation branch, the model generates the final instance segmentation mask through linear combination. Specifically, for each candidate target, its corresponding mask coefficient vector (of length K) is summed element-wise with K pre-stored prototype masks to obtain an intermediate result. The generated intermediate result is a two-dimensional matrix of continuous values. The Sigmoid function is then used to map the value of each pixel to the [0, 1] interval, and finally binarized according to a preset threshold (e.g., 0.5) to obtain the final instance segmentation mask. The resolution of this instance segmentation mask corresponds to the resolution of the input feature map. For example, for the mask generated by the P2 detector, its size is 160×160, which can be aligned with the original image through upsampling or downsampling. In this way, the model can output pixel-level precision contours for each segmented target (e.g., the bottom region of a swimming pool), while no mask is generated for detected targets, only bounding boxes are output.

[0100] In one possible implementation, the output of the candidate target category prediction results also includes the category confidence score corresponding to each category prediction result; before classifying and outputting the prediction results according to the candidate target category, the post-processing of the candidate targets is also included, which includes: determining the boundary region of the candidate targets based on the bounding box position parameters corresponding to the candidate targets; filtering the candidate targets based on the category prediction results and the category confidence scores corresponding to the category prediction results to obtain a first candidate target set; performing non-maximum suppression processing on the boundary regions corresponding to each candidate target in the first candidate target set according to the candidate target category to remove redundant candidate targets with overlapping spatial locations to obtain a second candidate target set; classifying and outputting the prediction results according to the candidate target category, including: classifying and outputting the candidate target categories in the second candidate target set after non-maximum suppression processing.

[0101] After the model completes forward inference, the bounding box position parameters output by each detector head are typically represented as offsets relative to preset anchor frames, which cannot directly correspond to the actual positions in the original image. Therefore, the first step in post-processing is to decode these bounding box position parameters to determine the true boundary region of each candidate target in the original underwater image. Specifically, for each anchor point and its corresponding anchor frame (the size and center position of the anchor frame are preset), the bounding box position parameters output by the model include the center point horizontal coordinate offset (Δx), the center point vertical coordinate offset (Δy), the width scaling factor (Δw), and the height scaling factor (Δh). During decoding, Δx and Δy are first multiplied by the width and height of the anchor frame, and then added to the center coordinates of the anchor frame to obtain the center coordinates of the predicted bounding box; then Δw and Δh are exponentially multiplied by the width and height of the anchor frame to obtain the width and height of the predicted bounding box. For detector heads with different resolutions, the decoded bounding box coordinates need to be restored to the original image coordinate system according to the scaling ratio between the feature map and the original image.

[0102] After obtaining the predicted bounding box and class prediction results for each candidate target, the model further uses the class confidence score output by the classification branch to perform preliminary screening of the candidate targets. Specifically, the class prediction result for each candidate target is a vector, where each dimension represents the unnormalized score of the target belonging to a certain preset class. This score is converted into a probability value between 0 and 1 using the Sigmoid function, i.e., the class confidence score. The model selects the class with the highest confidence score as its predicted class for each candidate target and records the corresponding confidence score value. Then, based on a preset confidence score threshold (e.g., 0.5), all candidate targets are filtered: candidate targets with a confidence score greater than or equal to the threshold are retained, while candidate targets with a confidence score lower than the threshold are removed. It can be understood that low-confidence predictions are usually false detections or background noise. After the above screening process, the remaining high-confidence candidate targets constitute the first candidate target set, which can effectively reduce the computational cost of subsequent non-maximum suppression processing and reduce the impact of erroneous samples on the final detection results.

[0103] Even after initial screening, the first set of candidate targets may still contain multiple targets corresponding to the same real object, meaning their predicted bounding boxes highly overlap spatially. To address this, the model performs Non-Maximum Suppression (NMS) on the first set of candidate targets by category. Specifically, the first set of candidate targets is first grouped according to prediction categories, and each category is processed independently. For candidate targets within the same category, they are sorted from highest to lowest confidence level, and the candidate target with the highest confidence level is selected as the baseline. Then, the overlap between the bounding boxes of other candidate targets and the baseline is calculated (usually using the Intersection over Union (IoU), which is the ratio of the intersection area to the union area of ​​two bounding boxes). If the IoU of a candidate target is greater than a preset threshold (e.g., 0.45), it is considered to have excessive overlap with the baseline target and is considered a redundant prediction, so it is discarded; otherwise, it is retained. Next, the next highest-confidence target is selected from the remaining candidate targets as the new baseline, and the above process is repeated until all candidate targets within the category have been processed. For segmentation targets, the boundary region can be provided by the bounding rectangle of the predicted instance segmentation mask, or the decoded predicted bounding box can be used directly. Both methods can be used as the location basis for NMS. By performing NMS separately for each category, the model can effectively remove duplicate detection boxes on the same object, while avoiding incorrect suppression of targets of different categories (such as leaves and pebbles) due to overlapping positions.

[0104] The candidate targets retained after NMS processing constitute the second candidate target set. The model then performs classification output based on the category type of each candidate target in the second candidate target set. The specific rules are as follows: If a candidate target is predicted to be a detection class (e.g., leaves, pebbles, drains, steps, or escalators), the model only outputs the predicted bounding box coordinates and its category information (including category ID and confidence score), without outputting any mask data. If a candidate target is predicted to be a segmentation class (e.g., a continuous area at the bottom of a pool), the model first linearly combines the mask coefficients output from the segmentation branch with a pre-generated prototype mask to generate an instance segmentation mask. This mask (usually represented as a binary image or contour point set) and its category information are then output, without outputting the bounding box. For example, for obstacles to be avoided, bounding box information is output; for pool bottom areas to be cleaned, pixel-level masks are directly obtained. Thus, the robot can directly control its movement based on these results without further parsing of the model output.

[0105] In one possible implementation, the bounding boxes of candidate targets are processed based on the bounding box position parameters corresponding to the candidate targets to obtain the boundary regions of the candidate targets. This includes: decoding the bounding boxes of candidate targets based on the bounding box position parameters corresponding to the candidate targets to obtain the predicted bounding boxes of the candidate targets; for detection targets among the candidate targets, the predicted bounding boxes corresponding to the detection targets are used as the boundary regions of the detection targets; for segmentation targets among the candidate targets, the boundary regions of the segmentation targets are determined based on the instance segmentation mask corresponding to the segmentation targets, or the predicted bounding boxes corresponding to the segmentation targets are used as the boundary regions of the segmentation targets.

[0106] Since the bounding box position parameters output by each detector head of the model are given in the form of offsets relative to preset anchor boxes, they cannot be directly mapped to the pixel coordinate system of the original underwater image. Therefore, the primary task of post-processing is to decode these bounding box position parameters to obtain the predicted bounding box of each candidate target in the original image. The specific operations of the decoding process are as follows: First, obtain the center coordinates, width, and height of the preset anchor box corresponding to each anchor point; then, extract the four offsets corresponding to the anchor point from the bounding box regression branch output by the model, namely, the center point x-coordinate offset, the center point y-coordinate offset, the width scaling factor, and the height scaling factor. During decoding, the center coordinates of the predicted bounding box are obtained by adding the corresponding offset to the anchor box center coordinates and multiplying them by the anchor box size. The center point x-coordinate is equal to the anchor box center x-coordinate plus the x-coordinate offset multiplied by the anchor box width, and the center point y-coordinate is equal to the anchor box center y-coordinate plus the y-coordinate offset multiplied by the anchor box height. The width and height of the predicted bounding box are obtained by multiplying the anchor box width by the exponential function value of the width scaling factor and the anchor box height by the exponential function value of the height scaling factor. Since the feature maps output by different detection heads have different resolutions, the coordinates of the decoded bounding boxes are located in the feature map coordinate system. They need to be further multiplied by the downsampling factor of the feature map relative to the original image (for example, the downsampling factor for the P2 detection head is 4, for P3 it is 8, and for P4 it is 16) to restore the coordinates to the pixel coordinate system of the original image.

[0107] After obtaining the predicted bounding boxes for all candidate targets, the boundary regions used for subsequent non-maximum suppression processing need to be determined according to the target category. For detected targets, the annotation information during model training only includes bounding boxes (see the dataset integration step), and the predicted bounding boxes can accurately represent the position and extent of the target in the image. For example, detected targets include discrete objects such as leaves, drains, pebbles, steps, and escalators. These targets usually have relatively regular shapes, and rectangular bounding boxes are sufficient for localization. Therefore, for detected targets, the predicted bounding boxes obtained from the previous decoding step are directly used as their boundary regions without any additional transformations.

[0108] For segmentation targets, the output is a pixel-level instance segmentation mask rather than a bounding box. However, in non-maximum suppression processing, a boundary region is still needed for overlap comparison with other candidate targets. Therefore, this application provides two optional boundary region determination methods. The first method: Determine the boundary region based on the instance segmentation mask corresponding to the segmentation target. Specifically, after the model generates the instance segmentation mask (usually a binary image), calculate the minimum bounding rectangle of the mask (i.e., the minimum axis-aligned rectangle that can contain all mask pixels), and use this bounding rectangle as the boundary region of the segmentation target. The first method can more accurately reflect the actual contour range of the target, especially when the target shape is irregular (such as local fragments in the pool bottom area), the bounding rectangle fits the target better than the predicted bounding box. The second method: Directly use the predicted bounding box (i.e., the decoded rectangle) corresponding to the segmentation target as its boundary region. This method has less computational cost, and when the segmentation target shape is relatively regular (such as a complete pool bottom area), the difference between the predicted bounding box and the bounding rectangle is not significant. In practical applications, one method can be selected according to the specific scenario, or the two methods can be combined (for example, the mask bounding rectangle is used by default, and the predicted bounding box is used when the mask generation fails).

[0109] In this embodiment, the proposed multi-task target recognition model is compared with the current mainstream lightweight model YOLOv8n and separately deployed detection and segmentation models (YOLOv11n-det and YOLOv11n-seg) on ​​a self-built pool bottom dataset. In terms of detection accuracy, the overall detection accuracy (measured by mean average precision, mAP) of the proposed model is 4.7 percentage points higher than YOLOv8n. Especially for extremely small targets with an area less than 32×32 pixels, the detection AP reaches 80.3%, significantly outperforming other comparative models. Based on the aforementioned embodiments, the performance improvement of the proposed model mainly stems from several improvements in the model structure, including the high-resolution P2 fusion feature map that preserves rich spatial details, the C3k2 dynamic convolution module that adaptively extracts local features at different scales, and the Focal Loss used during training to effectively alleviate the imbalance between positive and negative samples. In instance segmentation, the model in this application achieves a segmentation accuracy of 89.5%, which is essentially on par with the YOLOv11n-seg model trained alone (accuracy of 90.2%), with a difference of less than 1 percentage point. However, in terms of model resource consumption, the model in this application has only 4.5 million parameters (M, Million), while the total number of parameters for deploying YOLOv11n-det (4.5M) and YOLOv11n-seg (4.0M) separately is 8.5M, representing a reduction of approximately 46% in the number of parameters. In terms of computation, the model in this application requires 10.5 billion floating-point operations (10.5 Giga Floating Point Operations, or 10.5 G FLOPs), while the sum of the computation of the two separate models is 21.2 G FLOPs, representing a reduction of approximately 50% in the computation of the model in this application. Based on the above comparison, it can be seen that by designing a multi-task shared backbone network, this application can complete target detection and instance segmentation simultaneously with a single model. It can significantly reduce the number of parameters and computational overhead while maintaining the segmentation accuracy without a significant decrease, thus meeting the requirements of such devices to balance real-time performance and low power consumption in actual use.

[0110] The above describes an underwater multi-task target recognition method provided by embodiments of this application. The following will describe the apparatus for implementing the above-described underwater multi-task target recognition method. Please refer to... Figure 2 , Figure 2 This is a schematic diagram of the structure of an underwater multi-task target identification device provided in an embodiment of this application. Figure 2 As shown, the underwater multi-task target identification device 200 includes:

[0111] The feature extraction unit 201 is used to input the acquired underwater image into a pre-trained target recognition model, perform multi-layer feature extraction on the underwater image, and obtain multiple enhanced feature maps of different scales; wherein, the scale represents the downsampling factor of the enhanced feature map relative to the underwater image, and the enhanced feature maps of different scales correspond to different target detection ranges.

[0112] The feature fusion unit 202 is used to perform feature fusion processing on the multiple enhanced feature maps to obtain a fused feature map corresponding to the candidate target in the underwater image; the candidate target is a region in the image where an object exists, including detected targets and segmented targets.

[0113] The decoupling prediction unit 203 is used to perform decoupling prediction processing on the fused feature map and output the category prediction result of the candidate target, the bounding box position parameters corresponding to the candidate target, and the instance segmentation mask information corresponding to the candidate target.

[0114] The result output unit 204 is used to classify and output the prediction results according to the candidate target categories. Specifically, it outputs the target bounding box and category information for detected targets, and the instance segmentation mask and category information for segmented targets.

[0115] Optionally, the feature extraction unit 201 is specifically used to: sequentially perform convolution, batch normalization, and nonlinear mapping activation processing on the underwater image to obtain a first initial feature map; repeat the convolution, batch normalization, and nonlinear mapping activation processing on the first initial feature map at least three times to obtain a second initial feature map, a third initial feature map, and a fourth initial feature map, respectively; and perform feature enhancement processing on the first initial feature map, the second initial feature map, the third initial feature map, and the fourth initial feature map to obtain multiple enhanced feature maps of different scales.

[0116] Optionally, the feature extraction unit 201 is specifically used to: input the fourth initial feature map into the attention enhancement module for attention enhancement processing to obtain the first enhanced feature map; and input the third initial feature map, the second initial feature map and the first initial feature map into the convolution module for convolution feature extraction to obtain the second enhanced feature map, the third enhanced feature map and the fourth enhanced feature map.

[0117] Optionally, the feature fusion unit 202 is specifically configured to: perform a first upsampling on the fourth enhanced feature map, concatenate and convolve the first upsampling result with the third enhanced feature map to obtain a first intermediate fused feature map; perform a second upsampling on the first intermediate fused feature map, concatenate and convolve the second upsampling result with the second enhanced feature map to obtain a second intermediate fused feature map; perform a third upsampling on the second intermediate fused feature map, concatenate and convolve the third upsampling result with the first enhanced feature map to obtain a third intermediate fused feature map; and then perform a third upsampling on the third intermediate fused feature map. The first enhanced feature map is concatenated and convolved with the first enhanced feature map to obtain a first-scale fused feature map. The first-scale fused feature map is downsampled for the first time, and the first downsampling result is concatenated and convolved with the second intermediate fused feature map to obtain a second-scale fused feature map. The second-scale fused feature map is downsampled for the second time, and the second downsampling result is concatenated and convolved with the first intermediate fused feature map to obtain a third-scale fused feature map. The first-scale fused feature map, the second-scale fused feature map, and the third-scale fused feature map are used together as the fused feature map corresponding to the candidate target.

[0118] Optionally, the device further includes a model training unit; the model training unit is specifically used for: constructing a training dataset, the training dataset including underwater images and corresponding annotation information, the annotation information including bounding box annotation information for detected targets and mask annotation information for segmented targets; performing image processing operations on the underwater images, the image processing operations including at least one of image stitching, image blending, image flipping, and image color perturbation; inputting the processed training dataset into the target recognition model for forward computation to obtain target category prediction results, bounding box prediction results, and mask prediction results; calculating a loss function based on the target category prediction results, the bounding box prediction results, the mask prediction results, and the annotation information corresponding to the underwater images, wherein classification loss and bounding box regression loss are calculated for detected targets; classification loss and mask loss are calculated for segmented targets; updating the model parameters according to the loss function until a preset training termination condition is met to obtain a trained target recognition model.

[0119] Optionally, the decoupled prediction unit 203 is specifically configured to: input the fused feature map into multiple prediction branches, each prediction branch including at least a classification branch, a bounding box regression branch, and a segmentation branch; perform convolution operations on the fused feature map through the classification branch to output the category prediction result of the candidate target; perform convolution operations on the fused feature map through the bounding box regression branch to output the bounding box position parameters corresponding to the candidate target; perform convolution operations on the fused feature map through the segmentation branch to output the mask coefficients or mask features corresponding to the candidate target; and generate instance segmentation mask information corresponding to the candidate target based on the mask coefficients or mask features.

[0120] Optionally, the device further includes: a candidate target post-processing unit; the candidate target post-processing unit is specifically used to: determine the boundary region of the candidate target based on the bounding box position parameters corresponding to the candidate target; filter the candidate target based on the category prediction result of the candidate target and the category confidence corresponding to the category prediction result to obtain a first candidate target set; perform non-maximum suppression processing according to the candidate target category based on the boundary region corresponding to each candidate target in the first candidate target set to remove redundant candidate targets with overlapping spatial locations to obtain a second candidate target set; the result output unit 204 is specifically used to: classify and output the candidate target categories in the second candidate target set after non-maximum suppression processing.

[0121] Optionally, the candidate target post-processing unit is specifically configured to: decode the bounding box of the candidate target based on the bounding box position parameters corresponding to the candidate target to obtain the predicted bounding box of the candidate target; for detection targets among the candidate targets, use the predicted bounding box corresponding to the detection target as the boundary region of the detection target; for segmentation targets among the candidate targets, determine the boundary region of the segmentation target based on the instance segmentation mask corresponding to the segmentation target, or use the predicted bounding box corresponding to the segmentation target as the boundary region of the segmentation target.

[0122] This application also provides an electronic device in its embodiments. (See reference...) Figure 3 The diagram illustrates a structural schematic of an electronic device suitable for implementing the underwater multi-task target identification method in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, personal digital assistants (PDAs), tablet computers (PADs), desktop computers, etc. Figure 3The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0123] like Figure 3 As shown, the electronic device may include a processing unit (e.g., a central processing unit (CPU), graphics processing unit (GPU), etc.) 301, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 302 or a program loaded from storage device 308 into random access memory (RAM) 303. When the electronic device is powered on, RAM 303 also stores various programs and data required for the operation of the electronic device. The processing unit 301, ROM 302, and RAM 303 are interconnected via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0124] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, memory cards, hard drives, etc.; and communication devices 309. Communication device 309 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0125] This application also provides an underwater robot, including the electronic device provided in the aforementioned embodiments of this application.

[0126] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the underwater multi-task target recognition methods provided in this application.

[0127] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the underwater multi-task target recognition methods provided in this application.

[0128] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause an electronic device (which may be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in the various embodiments of this application.

[0130] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0131] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An underwater multi-task target recognition method, characterized in that, The method comprises the following steps: input the obtained underwater image into a pre-trained target recognition model, perform multi-layer feature extraction on the underwater image, and obtain multiple enhanced feature maps with different scales; wherein the scale represents the down-sampling multiple of the enhanced feature map relative to the underwater image, and the enhanced feature maps with different scales correspond to different sizes of target detection range; perform feature fusion processing on the multiple enhanced feature maps to obtain a fusion feature map corresponding to a candidate target in the underwater image; the candidate target is a region with an object in the image, including a detection class target and a segmentation class target; perform decoupling prediction processing on the fusion feature map to output a class prediction result of the candidate target, a bounding box position parameter corresponding to the candidate target, and instance segmentation mask information corresponding to the candidate target; classify and output the prediction result according to the candidate target class; wherein the target bounding box and class information are output for the detection class target, and the instance segmentation mask and class information are output for the segmentation class target.

2. The method of claim 1, wherein, The method comprises the following steps: perform convolution, batch normalization and nonlinear mapping activation processing on the underwater image in sequence to obtain a first initial feature map; repeat the convolution, batch normalization and nonlinear mapping activation processing on the first initial feature map at least three times to obtain a second initial feature map, a third initial feature map and a fourth initial feature map, respectively; perform feature enhancement processing on the first initial feature map, the second initial feature map, the third initial feature map and the fourth initial feature map to obtain multiple enhanced feature maps with different scales.

3. The method of claim 2, wherein, The method comprises the following steps: input the fourth initial feature map into an attention enhancement module for attention enhancement processing to obtain a first enhanced feature map; input the third initial feature map, the second initial feature map and the first initial feature map into convolution modules respectively for convolution feature extraction to obtain a second enhanced feature map, a third enhanced feature map and a fourth enhanced feature map.

4. The method of claim 3, wherein, The method comprises the following steps: perform first upsampling on the fourth enhanced feature map, and perform splicing and convolution fusion on the first upsampling result and the third enhanced feature map to obtain a first intermediate fusion feature map; perform second upsampling on the first intermediate fusion feature map, and perform splicing and convolution fusion on the second upsampling result and the second enhanced feature map to obtain a second intermediate fusion feature map; perform third upsampling on the second intermediate fusion feature map, and perform splicing and convolution fusion on the third upsampling result and the first enhanced feature map to obtain a third intermediate fusion feature map; perform splicing and convolution fusion on the third intermediate fusion feature map and the first enhanced feature map to obtain a first scale fusion feature map; performing first down-sampling on the first scale fusion feature map, splicing and convolution fusing the first down-sampling result and the second intermediate fusion feature map to obtain a second scale fusion feature map; performing second down-sampling on the second scale fusion feature map, splicing and convolution fusing the second down-sampling result and the first intermediate fusion feature map to obtain a third scale fusion feature map; the first scale fusion feature map, the second scale fusion feature map and the third scale fusion feature map are taken as the fusion feature map corresponding to the candidate target together.

5. The method of claim 1, wherein, The target recognition model is obtained by the following steps: constructing a training data set, the training data set including underwater images and annotation information corresponding to the underwater images, the annotation information including boundary box annotation information of detection class targets and mask annotation information of segmentation class targets; performing image processing operations on the underwater images, the image processing operations including at least one of image splicing processing, image mixing processing, image flipping processing and image color disturbance processing; inputting the processed training data set into the target recognition model for forward calculation to obtain target class prediction results, boundary box prediction results and mask prediction results; calculating a loss function based on the target class prediction results, the boundary box prediction results and the mask prediction results and the annotation information corresponding to the underwater images, wherein the classification loss and the boundary box regression loss are calculated for the detection class targets; the classification loss and the mask loss are calculated for the segmentation class targets; updating the model parameters according to the loss function until the preset training termination condition is met, and obtaining the trained target recognition model.

6. The method of claim 1, wherein, The decoupling prediction processing of the fusion feature map outputs the class prediction results of the candidate target, the boundary box position parameters corresponding to the candidate target and the instance segmentation mask information corresponding to the candidate target, including: inputting the fusion feature map into a plurality of prediction branches respectively, the prediction branches including at least a classification branch, a boundary box regression branch and a segmentation branch; performing convolution operation on the fusion feature map through the classification branch to output the class prediction results of the candidate target; performing convolution operation on the fusion feature map through the boundary box regression branch to output the boundary box position parameters corresponding to the candidate target; performing convolution operation on the fusion feature map through the segmentation branch to output the mask coefficient or the mask feature corresponding to the candidate target; generating the instance segmentation mask information corresponding to the candidate target based on the mask coefficient or the mask feature.

7. The method of claim 1, wherein, The output class prediction results of the candidate target also include the class confidence of each class prediction result; before the classification output of the prediction results according to the candidate target class, the candidate target is also subjected to post-processing, and the post-processing includes: determining the boundary region of the candidate target based on the boundary box position parameters corresponding to the candidate target; performing screening processing on the candidate target based on the class prediction results of the candidate target and the class confidence corresponding to the class prediction results to obtain a first candidate target set; Based on the boundary regions corresponding to each candidate target in the first candidate target set, non-maximum suppression processing is performed according to the candidate target category to remove redundant candidate targets with overlapping spatial locations, thereby obtaining the second candidate target set; The step of classifying and outputting the prediction results based on the candidate target category includes: The classification output is based on the candidate target categories in the second candidate target set after nonmaximum suppression processing.

8. The method of claim 7, wherein, Determining the boundary region of the candidate target based on the bounding box position parameters corresponding to the candidate target includes: Based on the bounding box position parameters corresponding to the candidate target, the bounding box of the candidate target is decoded to obtain the predicted bounding box of the candidate target; For the detected targets among the candidate targets, the predicted bounding box corresponding to the detected target is used as the boundary region of the detected target; For segmented targets among the candidate targets, the boundary region of the segmented target is determined based on the instance segmentation mask corresponding to the segmented target, or the predicted bounding box corresponding to the segmented target is used as the boundary region of the segmented target.

9. An electronic device, comprising: It includes at least one processor and a memory connected to the processor, wherein the memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the underwater multi-task target identification method as described in any one of claims 1 to 8.

10. An underwater robot, characterized in that, include: The electronic device as described in claim 9.

Images