Underwater salient target detection method and system based on double-flow fusion network

Abstract

The invention discloses an underwater salient target detection method and system based on a double-flow fusion network, and belongs to the technical field of computer vision. The method comprises the following steps: respectively extracting multi-scale features of an RGB image and a depth image through a double-flow encoder; in the shallow layer, fusing and enhancing the edge and detail information of the bimodal features through an edge fusion module; in a deep layer, content-adaptive cross-modal semantic fusion is realized in a frequency domain through a dynamic filtering module; fusing the multi-scale features through a cross-layer aggregation decoder to generate a rough saliency map; extracting detail features from the original RGB image through a global detail purification network; and finally, fusing the rough saliency map and the detail features, and outputting an underwater saliency target prediction map. The objective of the invention is to improve the precision and boundary definition of salient target detection in an underwater complex scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, specifically to a method and system for detecting salient underwater targets based on a dual-stream fusion network. Background Technology

[0002] Salient object detection aims to identify the most conspicuous object regions in an image. Underwater environments, due to light absorption and scattering, often result in color distortion, low contrast, and blurred details, degrading the performance of general salient object detection methods. Incorporating depth information is one approach to improve the robustness of underwater detection.

[0003] However, underwater depth maps are mostly estimated by algorithms, resulting in high noise levels and inaccurate boundaries. Existing depth information fusion methods often lack effective mechanisms to suppress noise in low-quality depth maps and struggle to maintain both high-level semantic consistency and low-level detail integrity during the fusion process, leading to blurred contours and loss of detail in the detection results. Therefore, a solution for detecting salient underwater targets that can suppress modal noise and collaboratively optimize semantics and details is needed. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an underwater salient target detection method and system based on a dual-stream fusion network, thereby alleviating the problems of poor detection accuracy and boundary clarity caused by modal noise and loss of detail in existing technologies.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: On the one hand, a method for detecting salient underwater targets based on a dual-stream fusion network (hereinafter referred to as the method) is provided, including the following steps: S1: Acquire the RGB image of the underwater scene and its corresponding depth map, and perform preprocessing; S2: Extract multi-scale features from RGB images and depth maps using a dual-stream encoder; S3: In the shallow features of the dual-stream encoder, the RGB features and depth features are fused and enhanced through the edge fusion module; S4: In the deep features of the dual-stream encoder, semantic-level fusion of RGB features and deep features is performed through a dynamic filtering module; S5: By using a cross-layer aggregation decoder, multi-scale features from the edge fusion module and the dynamic filtering module are fused to generate a coarse saliency prediction map; S6: Extract detailed features directly from the original RGB image through a global detail refinement network; S7: Integrate coarse saliency prediction map features with detailed features to generate the final underwater salient target prediction map.

[0006] Preferably, in step S3, the edge blending module's processing procedure includes: Intramodal spatial attention is applied to the input RGB features and depth features respectively to obtain the enhanced features; The enhanced RGB features are input into the first detail enhancement block, which splits the features into two branches. One branch introduces spatial attention guided by deep features for enhancement, while the other branch retains the original RGB information. Then, feature recalibration is performed on the two branches, and cross-fusion is carried out. Differential edge sensing is performed on the cross-fused features of the two branches to obtain edge enhancement features; The edge enhancement features are fused to obtain enhanced RGB features; Symmetrically, the enhanced depth features are input into the second detail enhancement block, and the same operation is performed guided by the RGB features to output the enhanced depth features.

[0007] Preferably, in step S4, the processing procedure of the dynamic filtering module includes: Intramodal spatial attention is applied to the input RGB features and depth features respectively for enhancement; In the first frequency domain filtering block, a set of dynamic frequency domain filters is generated using depth features as the guiding branch. Meanwhile, RGB features are used as the target branch, which is preprocessed and transformed to the frequency domain by fast Fourier transform. Then, they are multiplied element-wise with the dynamic frequency domain filter to achieve frequency domain transformation and filtering. The filtered frequency domain features are transformed back to the spatial domain by inverse Fourier transform, and then the filtered RGB features are obtained by convolution projection. Symmetrically, in the second frequency domain filtering block, guided by the RGB features, the same operation is performed on the depth features to obtain the filtered depth features; The filtered RGB features and depth features are fused with the original input features through residual connections to output the final deep fused features.

[0008] Preferably, in step S5, each decoding stage of the cross-layer aggregation decoder performs feature fusion through a pair of symmetrical cross-layer fusion blocks; wherein, the first cross-layer fusion block uses deep semantic features to enhance shallow detail features, and the second cross-layer fusion block uses shallow detail features to enrich deep semantic features; the cross-layer fusion block achieves dual attention-guided feature modulation by combining channel attention of global average pooling and global max pooling, and spatial attention of cross-channel average pooling and cross-channel max pooling.

[0009] Preferably, in step S6, the global detail refinement network includes parallel multi-scale context paths and feature recalibration paths; The multi-scale context path adopts a spatial pyramid pooling structure to capture multi-scale global context information; The feature recalibration path sequentially recalibrates the input features through the channel attention module and the spatial attention module, and performs detail enhancement and noise suppression.

[0010] On the other hand, an underwater salient target detection system (hereinafter referred to as the system) based on a dual-stream fusion network for implementing the method is provided, characterized in that the system includes: The data acquisition and preprocessing module is used to acquire and preprocess underwater RGB images and depth maps; A dual-stream coding module is used to extract multi-scale features from RGB and depth images respectively; The edge blending module is used to perform cross-modal edge enhancement and blending in the shallow layer of the encoder. The dynamic filtering module is used to perform content-adaptive semantic fusion in the frequency domain at a deep level of the encoder; A cross-layer aggregation decoding module is used to fuse multi-scale features and generate a coarse saliency map; The global detail enhancement module is used to extract and enhance detail information from the original RGB image; The prediction map generation module is used to fuse coarse saliency map features and detailed features to output the final saliency map.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention enhances edge details in the shallow layer and achieves adaptive semantic fusion in the deep layer through a layered fusion strategy, and combined with an independent detail purification path, which helps to improve the overall detection accuracy, improve the quality of target boundaries, and enhance the robustness of the model to complex underwater interference. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the overall network architecture of the dual-stream fusion network of the present invention. Figure 2 This is a schematic diagram of the edge blending module of the present invention; Figure 3 This is a schematic diagram of the dynamic filtering module of the present invention; Figure 4 This is a schematic diagram of the cross-layer aggregation decoder of the present invention; Figure 5 This is a schematic diagram of the global detail purification network of the present invention. Detailed Implementation

[0013] Specific embodiments of the present invention are given below. These specific embodiments are only used to further illustrate the present invention in detail and do not limit the scope of protection of the present invention.

[0014] Reference Appendix Figure 1The overall architecture of the dual-stream fusion network proposed in this invention takes the RGB image of the underwater scene and its corresponding depth map as input, and outputs a high-resolution underwater salient target prediction map through hierarchical and collaborative multimodal information processing.

[0015] The network mainly consists of the following core components connected in sequence: Dual-stream encoder: used to extract multi-scale features from RGB images and depth maps respectively.

[0016] Edge fusion module: Deployed in the shallow layer of the encoder, it is used to perform edge enhancement and cross-modal fusion of RGB and depth features.

[0017] Dynamic filtering module: Deployed deep within the encoder, it is used to perform content-adaptive semantic fusion of RGB and depth features in the frequency domain.

[0018] Cross-layer aggregation decoder: used to fuse features from various layers of the encoder to generate a preliminary saliency map.

[0019] Global Detail Enhancement Network: Used to directly extract and enhance detail features from raw RGB images.

[0020] The network outputs a predicted map of salient underwater targets. In a preferred embodiment, the network can generate multiple intermediate-scale predicted maps during training for auxiliary supervision, but only the final predicted map is output during final application.

[0021] The input to the method of this invention is a pair of underwater RGB images and a depth map. The depth map can be estimated from the RGB images by a monocular depth estimation algorithm, or it can be directly acquired by a depth sensor.

[0022] Before inputting data into the network, the following preprocessing steps are required to ensure data format consistency and network training stability: Size normalization: Scaling the RGB image and depth map to a uniform fixed size of 256×256 pixels.

[0023] Value range normalization: linearly normalize the pixel values ​​of the depth map to the [0,1] interval; normalize the pixel values ​​of the RGB image to the [0,1] interval or perform standardization processing.

[0024] Data augmentation: During the training phase, augmentation operations such as random horizontal flipping can be applied to the training samples. The RGB images, depth maps and corresponding ground truth labels need to be transformed synchronously.

[0025] After the above preprocessing, tensor data that can be directly processed by the network is obtained.

[0026] The dual-stream encoder consists of two identical but parameter-independent feature extraction backbone networks, referred to as the RGB encoder stream and the depth encoder stream, respectively.

[0027] In a preferred embodiment, the backbone network adopts a PVTv2 architecture, which can naturally generate a multi-scale feature pyramid through downsampling in multiple stages.

[0028] Specifically, each encoder stream receives a preprocessed image (RGB or depth map) and outputs four feature maps at different scales, denoted as: Stage 1 characteristics: highest spatial resolution, fewer channels, rich in detailed information.

[0029] Stage 2 characteristics: spatial resolution is secondary, and the number of channels increases.

[0030] Stage 3 characteristics: lower spatial resolution, enhanced semantic information.

[0031] Stage 4 characteristics: lowest spatial resolution, most channels, and contains the highest level of semantic information.

[0032] Let the four features output by the RGB encoder stream be {R1, R2, R3, R4}, and the four features output by the depth encoder stream be {D1, D2, D3, D4}. These features form the basis for subsequent module processing.

[0033] The edge fusion module is deployed in the shallow layer of the dual-stream encoder, specifically acting on the feature pairs (R1, D1) and (R2, D2) output from stages 1 and 2. Its core function is to enhance edge and detail information between the RGB and depth modalities, achieving cross-modal complementarity and noise suppression, and generating enhanced shallow-layer fused features. The structure of this module is shown in the attached figure. Figure 2 As shown.

[0034] Let the current processing be in the i-th (i=1,2) stage, and the input be the RGB feature r_i and the depth feature d_i.

[0035] First, independent intramodal spatial attention is applied to r_i and d_i respectively to initially focus on the salient regions within their respective modes.

[0036] This operation is achieved through a basic convolutional block (containing 3×3 convolution, batch normalization, and ReLU activation function) and subsequent spatial weight generation convolutional layers, generating weighted spatial attention features f_r and f_d.

[0037] Subsequently, features f_r and f_d are fed into a pair of symmetrical detail enhancement blocks for interactive processing. Taking the detail enhancement block for RGB feature f_r as an example, the process is as follows: Feature splitting and cross-modal guidance: split f_r into two branches.

[0038] One of the paths remains unchanged, focusing on RGB information; Another approach involves modulating a spatial attention map generated by deep features f_d, thereby incorporating spatial priors that emphasize depth information.

[0039] Feature recalibration: Channel attention and spatial attention recalibration are performed on the features of the two branches mentioned above to adaptively enhance important channels and spatial locations.

[0040] Cross-fusion: The spatial attention maps generated after the recalibration of the two branches are exchanged and used to modulate the features of the other branch, thereby realizing deep interaction of information between the two branches.

[0041] Differential edge perception: Average pooling and max pooling are performed on the features of the two interacting branches respectively, and the absolute value of the difference between the two is calculated. This difference result is used as the edge enhancement feature.

[0042] Feature aggregation: The edge enhancement features are concatenated with the features of the two branches, and then fused and reduced in dimensionality through convolutional layers to obtain the processing result of the block.

[0043] Symmetrically, another detail enhancement block uses f_d as its backbone and introduces f_r as a guide to perform the same operation.

[0044] Finally, the outputs of the two detail enhancement blocks are added to the original f_r and f_d respectively through residual connections to obtain the enhanced RGB features {r}_i and depth features {d}_i of this module, which are then used as inputs to the subsequent decoder.

[0045] The dynamic filtering module is deployed deep within the dual-stream encoder, specifically acting on the feature pairs (R3, D3) and (R4, D4) output from stages 3 and 4. Its core function is to achieve cross-modal high-level semantic information fusion in the frequency domain using a content-adaptive approach. The structure of this module is shown in the attached figure. Figure 3 As shown.

[0046] Let the current processing be in the i-th (i=3,4) stage, with the input being the RGB feature r_i and the depth feature d_i.

[0047] Similar to the edge fusion module, intramodal spatial attention enhancement is first applied to r_i and d_i respectively to obtain features f_r and f_d.

[0048] The enhanced features f_r and f_d are fed into a pair of symmetrical frequency domain filter blocks. Taking a filter block that uses depth features to guide RGB features as an example, the operation is performed in the frequency domain: Dynamic filter generation (guide branch): Global average pooling is performed on the deep features f_d to obtain a global descriptor, and then a set of weight coefficients are generated through a lightweight multilayer perceptron (MLP) and the Softmax function.

[0049] This set of weights is used to blend a predefined set of learnable complex fundamental filters to synthesize a dynamic frequency domain filter that is relevant to the current content.

[0050] Frequency domain transformation and filtering (target branch): After preprocessing the RGB features f_r through point convolution and activation functions, they are transformed to the frequency domain through Fast Fourier Transform (FFT).

[0051] In the frequency domain, it is multiplied channel by channel with the aforementioned generated dynamic frequency domain filter to complete the filtering operation.

[0052] Spatial domain reconstruction: The filtered frequency domain features are transformed back to the spatial domain through inverse Fourier transform (IFFT), and then projected through point convolution to obtain the filtered RGB features.

[0053] Symmetrically, another frequency domain filter block, guided by the RGB feature f_r, performs the same filtering operation on the depth feature f_d.

[0054] Finally, the outputs of the two frequency domain filter blocks are added to the original r_i and d_i respectively through residual connection to obtain the enhanced RGB feature {r}_i and depth feature {d}_i of this module, which are used as the input of the subsequent decoder.

[0055] The cross-layer aggregation decoder is used to systematically fuse features from various levels of the encoder to generate an initial saliency prediction map.

[0056] The global detail extraction network works in parallel, directly extracting fine details and global structural information from the original RGB image to compensate for spatial details lost during encoding. The decoder adopts a deep-to-shallow, coarse-to-fine approach, and its structure is shown in the attached figure. Figure 4 As shown.

[0057] The decoding process is divided into four stages, corresponding to the four levels of the encoder.

[0058] Each decoding stage receives two types of input: The fused feature x_i is derived from the encoder at the same layer and processed by the corresponding module (edge ​​fusion module or dynamic filtering module).

[0059] The output feature x_{i-1} from the previous decoding stage.

[0060] Within each stage, the above inputs are processed through a pair of symmetrical cross-level fusion blocks: The first fusion block utilizes the rich semantic information of deep features x_{i-1} to enhance and guide shallow detail features x_i.

[0061] The second fusion block uses the detailed information of the shallow feature x_i to enrich and refine the deep semantic feature x_{i-1}.

[0062] The core operations of each cross-level fusion block include: Context extraction: Global average pooling and global max pooling are performed on the input features simultaneously. The resulting statistics are then processed by a small neural network and fused to generate a channel attention vector.

[0063] Structure extraction: The input features are subjected to cross-channel average pooling and max pooling, the results are concatenated and then processed by a large convolutional kernel to generate a spatial attention map.

[0064] Feature modulation: The channel attention and spatial attention mentioned above are applied to another input feature to achieve dual attention-guided feature enhancement.

[0065] The outputs of the two fusion blocks are concatenated and fused through a convolutional layer to form the output of the current decoding stage. This output is then upsampled and used as the deeper input for the next shallower decoding stage.

[0066] The decoder eventually outputs a coarse saliency feature map S_coarse at the shallowest layer (stage 1), with its resolution initially restored.

[0067] The Global Detail Enhancement Network directly processes the original RGB input image, and its structure is shown in the attached figure. Figure 5 As shown.

[0068] It contains two parallel paths: Multi-scale context path: A spatial pyramid pooling structure is adopted to capture multi-scale context information from global to local in parallel through average pooling operations of multiple different grid scales, and the results are fused.

[0069] Feature recalibration path: First, the initial features are recalibrated sequentially using the channel attention module and the spatial attention module to focus on important information.

[0070] Subsequently, the details are further enhanced through a residual difference structure: the smoothed (estimated noisy / redundant) version of the main feature is subtracted, and the remaining detail components are enhanced with adaptive weights.

[0071] The outputs of the two paths are fused to generate a detail-refined feature F_DP. This feature contains high-resolution details and a clear global structure.

[0072] The coarse salient features S_coarse output from the decoder are concatenated and fused with the detailed features F_DP output from the global detail refinement network.

[0073] Subsequently, the image is processed through a prediction head (a 1×1 convolution followed by a Sigmoid activation function) to generate the final underwater salient target prediction map S_final.

[0074] The pixel value of S_final represents the probability that the pixel belongs to a salient target, thus completing the entire process from the input image to the salient map.

[0075] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A method for detecting salient underwater targets based on a dual-stream fusion network, characterized in that, Includes the following steps: S1: Acquire the RGB image of the underwater scene and its corresponding depth map, and perform preprocessing; S2: Extract multi-scale features from RGB images and depth maps using a dual-stream encoder; S3: In the shallow features of the dual-stream encoder, the RGB features and depth features are fused and enhanced through the edge fusion module; S4: In the deep features of the dual-stream encoder, semantic-level fusion of RGB features and deep features is performed through a dynamic filtering module; S5: By using a cross-layer aggregation decoder, multi-scale features from the edge fusion module and the dynamic filtering module are fused to generate a coarse saliency prediction map; S6: Extract detailed features directly from the original RGB image through a global detail refinement network; S7: Integrate coarse saliency prediction map features with detailed features to generate the final underwater salient target prediction map.

2. The underwater salient target detection method based on a dual-stream fusion network according to claim 1, characterized in that, In step S3, the edge blending module's processing includes: Intramodal spatial attention is applied to the input RGB features and depth features respectively to obtain the enhanced features; The enhanced RGB features are input into the first detail enhancement block, which splits the features into two branches. One branch introduces spatial attention guided by deep features for enhancement, while the other branch retains the original RGB information. Then, feature recalibration is performed on the two branches, and cross-fusion is carried out. Differential edge sensing is performed on the cross-fused features of the two branches to obtain edge enhancement features; The edge enhancement features are fused to obtain enhanced RGB features; Symmetrically, the enhanced depth features are input into the second detail enhancement block, and the same operation is performed guided by the RGB features to output the enhanced depth features.

3. The underwater salient target detection method based on a dual-stream fusion network according to claim 1, characterized in that, In step S4, the processing procedure of the dynamic filtering module includes: Intramodal spatial attention is applied to the input RGB features and depth features respectively for enhancement; In the first frequency domain filtering block, a set of dynamic frequency domain filters is generated using depth features as the guiding branch; Meanwhile, RGB features are used as the target branch, which is preprocessed and transformed to the frequency domain by fast Fourier transform. Then, they are multiplied element-wise with the dynamic frequency domain filter to achieve frequency domain transformation and filtering. The filtered frequency domain features are transformed back to the spatial domain by inverse Fourier transform, and then the filtered RGB features are obtained by convolution projection. Symmetrically, in the second frequency domain filtering block, guided by RGB features, the same operation is performed on the depth features to obtain the filtered depth features; The filtered RGB features and depth features are fused with the original input features through residual connections to output the final deep fused features.

4. The underwater salient target detection method based on a dual-stream fusion network according to claim 1, characterized in that, In step S5, each decoding stage of the cross-layer aggregation decoder performs feature fusion through a pair of symmetrical cross-layer fusion blocks; wherein, the first cross-layer fusion block uses deep semantic features to enhance shallow detail features, and the second cross-layer fusion block uses shallow detail features to enrich deep semantic features; the cross-layer fusion block achieves dual attention-guided feature modulation by combining channel attention of global average pooling and global max pooling, and spatial attention of cross-channel average pooling and cross-channel max pooling.

5. The underwater salient target detection method based on a dual-stream fusion network according to claim 1, characterized in that, In step S6, the global detail refinement network includes parallel multi-scale context paths and feature recalibration paths; The multi-scale context path adopts a spatial pyramid pooling structure to capture multi-scale global context information; The feature recalibration path sequentially recalibrates the input features through the channel attention module and the spatial attention module, and performs detail enhancement and noise suppression.

6. An underwater salient target detection system based on a dual-stream fusion network that implements the method as described in any one of claims 1-5, characterized in that, The system includes: The data acquisition and preprocessing module is used to acquire and preprocess underwater RGB images and depth maps; A dual-stream coding module is used to extract multi-scale features from RGB and depth images respectively; The edge blending module is used to perform cross-modal edge enhancement and blending in the shallow layer of the encoder. The dynamic filtering module is used to perform content-adaptive semantic fusion in the frequency domain at a deep level of the encoder; A cross-layer aggregation decoding module is used to fuse multi-scale features and generate a coarse saliency map; The global detail enhancement module is used to extract and enhance detail information from the original RGB image; The prediction map generation module is used to fuse coarse saliency map features and detailed features to output the final saliency map.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes a computer program to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images