Efficient RGB-D saliency detection method using multiple information complementation
By adopting multiple information complementarity and attention mechanism methods in RGB-D significance detection, combined with the enhancement of global and local feature coordination depth, the unbalanced efficiency and accuracy of RGB-D detection methods on mobile devices in the prior art is solved. Problem, achieving more efficient and accurate significance target detection.
Patent Information
- Application Number
- CN202411107618.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-19
Smart Images

Figure CN118982655A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of salient object detection, and in particular to an efficient RGB-D saliency detection method utilizing multiple information complementarity. Background Art
[0002] Salient Object Detection (SOD) can locate and segment the areas that attract the most human visual attention. As an image preprocessing technology, salient object detection has been widely used in computer vision tasks such as image semantic segmentation, quality assessment, and image retrieval. Early SOD methods mainly studied red, green, and blue (RGB) images, and it was difficult to detect salient objects in scenes with complex backgrounds or changing lighting conditions.
[0003] In recent years, researchers have introduced depth images to supplement the missing spatial information of RGB images, which helps detect salient targets from complex scenes such as cluttered backgrounds and lighting changes. Therefore, SOD methods based on RGB depth images (RGB Depth, RGB-D) have received special attention. Compared with traditional methods that rely on manual features, RGB-D SOD methods based on deep learning can further improve detection accuracy. However, the RGB-D SOD networks of these methods are difficult to balance efficiency and accuracy, and are not easy to apply to mobile devices.
[0004] At present, scholars have built a variety of RGB-D SOD deep learning network models based on Convolutional Neural Networks (CNNs). The applicant found that these networks mainly use strategies such as feature extraction, depth map enhancement and feature fusion to improve detection accuracy. While improving accuracy, they also greatly increase the number of model parameters and significantly reduce the calculation speed, making it difficult to deploy on mobile devices with limited computing and storage. Therefore, an efficient RGB-D SOD network is crucial, and its core is a lightweight backbone network. With the rise of lightweight networks such as SqueezeNet and MobileNet, the RGB-D SOD method using these networks as the backbone can greatly reduce the number of parameters and significantly improve the calculation speed, but its detection accuracy is still far behind that of non-lightweight advanced models, and its efficiency needs to be further improved. In short, RGB-DSOD still faces huge challenges in balancing detection efficiency and accuracy.
[0005] The applicant found that there is complementary information between the RGB image and the depth image, which helps to improve the quality of the depth image and thus improve the final SOD detection accuracy. Therefore, how to efficiently use the complementary information between the RGB image and the depth image to improve the accuracy and efficiency of RGB-D SOD is a problem that those skilled in the art need to solve urgently. Summary of the invention
[0006] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is: how to provide an efficient RGB-D saliency detection method using multiple information complementarity, and to fuse and enhance RGB features and depth features through the information complementarity between the deep feature map and the RGB feature map and the multiple attention mechanism, so as to improve and balance the accuracy and efficiency of RGB-D saliency detection.
[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0008] An efficient RGB-D saliency detection method using multiple complementary information, including:
[0009] S1: Obtain an RGB image and a depth image containing the target to be detected;
[0010] S2: Input the RGB image and the depth image into the trained saliency detection model and output the saliency prediction image of the target to be detected;
[0011] The training steps of the saliency detection model are as follows:
[0012] S201: Taking the RGB image and the depth image as training data and the corresponding truth image as input;
[0013] S202: Perform multi-scale feature extraction on the RGB image and the depth image through the backbone network to generate RGB feature maps and depth feature maps of several scales;
[0014] When extracting high-level scale deep feature maps: use the RGB feature map of the corresponding scale as an auxiliary to the deep feature map, combine the information complementarity between the deep feature map and the RGB feature map and the multiple attention mechanism to interactively enhance the RGB features and the deep features, and generate the corresponding scale deep feature map and enhanced RGB feature map;
[0015] S203: Fusing the RGB feature map of the high-level scale with the enhanced RGB feature map of the corresponding scale to generate a fused feature map;
[0016] S204: Utilize the global and local information of the low-level RGB feature map and the fusion feature map to perform collaborative depth enhancement to generate an enhanced depth image; calculate the training loss of the backbone network by the enhanced depth image and the input depth image to optimize the parameters of the backbone network;
[0017] S205: performing feature concatenation and fusion of several stages on the low-level RGB feature map and the fused feature map through the decoder to generate saliency prediction images at each stage; calculating the decoder training loss at each stage through the saliency prediction images at each stage and the corresponding true value maps to optimize the decoder parameters;
[0018] S206: repeating steps S201 to S205 until the saliency detection model converges or reaches a preset number of iterations;
[0019] S3: Taking the saliency prediction image of the target to be detected as its saliency detection result.
[0020] Preferably, in step S202, the backbone network performs five consecutive stages of RGB feature extraction on the RGB image to sequentially generate RGB feature maps of the first to fifth scales; wherein the RGB feature extraction of each stage includes a plurality of inverted residual blocks connected end to end in sequence;
[0021] The inverted residual block includes the first 1×1 convolution layer, the first BN layer, the first ReLu function layer, the 3×3 depth-separable convolution layer, the second BN layer, the second ReLu function layer, the second 1×1 convolution and the third BN layer, which are connected end to end in sequence; the input of the first 1×1 convolution layer and the output of the third BN layer are residually connected as the output of the inverted residual block.
[0022] Preferably, in step S202, the backbone network performs three consecutive stages of depth feature extraction on the depth image to sequentially generate depth feature maps of the first to third scales; wherein the depth feature extraction of each stage includes a plurality of inverted residual blocks connected end to end in sequence; the depth feature map of the third scale is input into two auxiliary extraction modules connected end to end in sequence, and the depth feature maps of the fourth and fifth scales are sequentially extracted and generated;
[0023] The auxiliary extraction module includes an inverted residual block and a two-stream complementary residual module connected end to end in sequence; the two-stream complementary residual module assists in the extraction and generation of a depth feature map by introducing an RGB feature map of a corresponding scale; in the auxiliary extraction module, the depth feature map of the N-1th scale is input into the inverted residual block, and an initial depth feature map of the Nth scale is output; the initial depth feature map and the RGB feature map of the Nth scale are input into the two-stream complementary residual module, and a depth feature map and an enhanced RGB feature map of the Nth scale are output;
[0024] The calculation formula of the auxiliary extraction module is expressed as:
[0025]
[0026] Where: D nDenote the enhanced RGB feature map and depth feature map of the nth scale output by the auxiliary extraction module; D n-1 Represents the depth feature map of the N-1th scale; R n Represents the RGB feature map of the Nth scale; IRB represents the inverted residual block, the superscript of IRB represents the channel expansion multiple r, and the subscript represents the step size; DCR represents the dual-stream complementary residual module; n represents the nth stage of the backbone network, corresponding to the nth scale.
[0027] Preferably, in step S202, the dual-stream complementary residual module includes multiple cross-modal complementary modules, a first BN layer, a first ReLu function layer, a 3×3 depth-separable convolution layer, a second BN layer, a second ReLu function layer, a 1×1 convolution, and a third BN layer, which are sequentially connected end to end; wherein the input of the multiple cross-modal complementary module is residually connected with the output of the third BN layer as the output of the dual-stream complementary residual module;
[0028] The multi-cross-modal complementary module utilizes the complementary information between the deep feature map and the RGB feature map and combines the multi-attention mechanism to enhance the interaction between RGB features and deep features.
[0029] Preferably, in step S202, the processing steps of the multiple cross-modal complementary modules are as follows:
[0030] S2021: RGB feature map F RGB ∈R H×W×C and the deep feature map F D ∈R H×W×C As the input of the multi-modal complementary module, W represents width, H represents height, and C represents the number of channels;
[0031] S2022: Calculate F RGB and F D Attention weights for correlation between channel and width dimensions Will Split into F along the channel dimension RGB and F D The attention weight and pass and Combined with F RGB and F D calculate and Will and Rotate 90° clockwise along the W axis;
[0032] Calculating attention weights When: F RGB and F DRotate 90° counterclockwise along the W axis to get the tensor and Will and Splice to get Will Input attention generation module generates attention weights
[0033] Calculated by the following formula and
[0034]
[0035] Where: represents the element-wise multiplication operator;
[0036] S2023: Calculate F RGB and F D Attention weights for correlation between channel and height dimensions Will Split into F along the height dimension RGB and F D The attention weight and pass and Combined with F RGB and F D calculate and Will and Rotate 90° clockwise along the H axis;
[0037] Calculating attention weights When: F RGB and F D Rotate 90° counterclockwise along the H axis to get the tensor and Will and Splice to get Will Input attention generation module generates attention weights
[0038] Calculated by the following formula and
[0039]
[0040] S2024: Calculate F RGB and F D The spatial attention weight Will Split into F along the spatial dimension RGB and F D The spatial attention weight and pass and Combined with F RGB and F D calculate and Will and Rotate 90° clockwise along the C axis;
[0041] Calculating spatial attention weights When: F RGB and F D Concatenate into tensors Will Input attention generation module generates attention weights The shape is (H×W×2);
[0042] Calculated by the following formula and
[0043]
[0044] S2025: Through F RGB , and Calculate the enhanced RGB feature map F RGB-D ;
[0045] The formula is:
[0046]
[0047] Where: cat represents the splicing in the channel dimension;
[0048] S2026: Pass F D , and Calculate the deep feature map F D-RGB ;
[0049] The formula is:
[0050]
[0051] Preferably, the calculation formula of the dual-stream complementary residual module is as follows:
[0052]
[0053] Where: represents the enhanced RGB feature map and depth feature map output by the dual-stream complementary residual module; F RGB 、F D Represents the RGB feature map and depth feature map of the dual-stream complementary residual module input; BN represents the batch normalization layer; Dw_Conv represents a 3×3 depth-separable convolution; Conv 1×1 Represents 1×1 convolution; They represent the F of multiple cross-modal complementary modules based on the input RGB and F D Output enhanced RGB feature map and depth feature map.
[0054] Preferably, in step S203, the depth feature map of the third scale is input into the inverted residual block, and the initial depth feature map of the fourth scale is output; the initial depth feature map and the RGB feature map of the fourth scale are input into the dual-stream complementary residual module to generate the depth feature map and the enhanced RGB feature map of the fourth scale; the enhanced RGB feature map of the fourth scale and the RGB feature map of the fourth scale are multiplied element by element, and then input into the inverted residual block to output the fused feature map of the fourth scale;
[0055] The depth feature map of the fourth scale is input into the inverted residual block, and the initial depth feature map of the fifth scale is output; the initial depth feature map and the RGB feature map of the fifth scale are input into the dual-stream complementary residual module to generate the depth feature map and the enhanced RGB feature map of the fifth scale; the enhanced RGB feature map of the fifth scale and the RGB feature map of the fifth scale are multiplied element by element, and then input into the inverted residual block to output the fused feature map of the fifth scale;
[0056] The calculation formula of the fused feature map is:
[0057]
[0058] Where: F n Represents the fused feature map of the nth scale; IRB represents the inverted residual block, the superscript of IRB represents the channel expansion multiple r, and the subscript represents the step size; Represents the corresponding element multiplication operation; R n Represents the RGB feature map of the nth scale; Represents the enhanced RGB feature map of the nth scale.
[0059] Preferably, in step S204, the enhanced depth image is generated by the following steps:
[0060] S2041: taking the RGB feature maps R1, R2 and R3 of the first to third scales, and the fused feature maps F4 and F5 of the fourth and fifth scales as input; wherein R1, R2 and R3 are taken as a global feature group, and R3, F4 and F5 are taken as a local feature group;
[0061] S2042: For the global feature group: splice the features of R3 after pixel rearrangement to R2; splice the features of R2 after pixel rearrangement to R1; expand the number of channels of R3, R2 and R1 to 256, and adjust the resolution of R2 and R1 to be the same as R3; splice the adjusted R1, R2 and R3 to obtain the global combined feature F G ; Through 1×1 convolution, F G The number of channels is reduced to 256;
[0062] S2043: For the local feature group: splice the feature after pixel rearrangement of F5 to F4; expand the number of feature channels of F4, F5 and R3 to 256, and adjust the resolution of F4 and F5 to be the same as R3; splice the adjusted R3, F4 and F5 along the channel dimension to obtain the local combined feature F L ; Through 1×1 convolution, F L The number of channels is reduced to 256;
[0063] S2044: F L Input two inverted residual blocks connected end to end; the output of the second inverted residual block is combined with F G The two two-stream complementary residual modules are connected end to end to generate a global lift graph T. G and the local lift graph T L ; The global lifting graph T is transformed through 1×1 convolution G and the local lift graph T L Convert to single channel;
[0064] S2045: Use sigmoid function and bilinear upsampling to globally improve graph T G and the local lift graph T L The size of is adjusted to be consistent with the input depth image; the adjusted global lift map T G and the local lift graph T L Add and average to generate an enhanced depth image;
[0065] The formula is:
[0066] T G ,T L =σ(upsampling(Conv 1×1 (2DCRB(F G ,2IRB(F L)))));
[0067]
[0068] Where: D i represents the enhanced depth image; T G , T L Represent the global lift graph and the local lift graph respectively; F G 、F L Represents global combined features and local combined features; IRB represents inverted residual block; DCR represents dual stream complementary residual module; Conv 1×1 represents 1×1 convolution; upsampling represents bilinear upsampling; σ represents the sigmoid function;
[0069] S2046: Calculate the training loss of the backbone network by the following formula to optimize the parameters of the backbone network;
[0070] S GDCDR =1-SSIM(D i ,D g );
[0071] Where: S GDCDR represents the backbone network training loss; SSIM represents structural similarity; D i Denotes enhanced depth image; D g Represents the input depth image.
[0072] Preferably, step S205 specifically includes the following steps:
[0073] S2051: Use the RGB feature maps R1, R2 and R3 of the first to third scales, and the fusion feature maps F4 and F5 of the fourth and fifth scales as inputs of the decoder;
[0074] S2052: input F5 into the compact pyramid refinement module and the convolutional attention mechanism module connected end to end in sequence, and output the feature fusion image of the fifth stage; the feature fusion image of the fifth stage is sequentially passed through a single channel 1×1 convolution, a sigmoid function and bilinear upsampling to obtain a saliency prediction image T5 of the fifth scale; the decoder training loss of the fifth stage is calculated through T5 and the corresponding truth map to optimize the decoder parameters;
[0075] In the compact pyramid refinement module, the input feature map is first expanded four times along the channel dimension through 1×1 convolution, and then passes through three parallel 3×3 depth-wise separable convolution branches; then the output features of each branch are element-wise added, and the added features are reduced in number of channels through the ReLu function layer, BN layer and 1×1 convolution. The obtained output is residually connected with the input feature map as the output of the compact pyramid refinement module;
[0076] S2053: After splicing the feature fusion image of the fifth stage and F4, the concatenation is input into the compact pyramid refinement module and the convolution attention mechanism module which are connected end to end in sequence, and the feature fusion image of the fourth stage is output; after the feature fusion image of the fourth stage is sequentially passed through a single channel 1×1 convolution, a sigmoid function and bilinear upsampling, a saliency prediction image T4 of the fourth scale is obtained; the decoder training loss of the fourth stage is calculated through T4 and the corresponding truth map to optimize the decoder parameters;
[0077] S2054: After splicing the feature fusion image of the fourth stage and R3, input them into the compact pyramid refinement module, and output the feature fusion image of the third stage; after the feature fusion image of the third stage passes through the 1×1 convolution of a single channel, the sigmoid function and the bilinear upsampling in sequence, a saliency prediction image T3 of the third scale is obtained; the decoder training loss of the third stage is calculated by T3 and the corresponding truth map to optimize the decoder parameters;
[0078] S2055: After splicing the feature fusion image of the third stage and R2, input them into the compact pyramid refinement module, and output the feature fusion image of the second stage; after the feature fusion image of the second stage passes through the 1×1 convolution, sigmoid function and bilinear upsampling of a single channel in sequence, a saliency prediction image T2 of the second scale is obtained; the decoder training loss of the second stage is calculated by T2 and the corresponding truth map to optimize the decoder parameters;
[0079] S2056: After splicing the feature fusion image of the second stage and R1, input them into the compact pyramid refinement module to output the feature fusion image of the first stage; the feature fusion image of the first stage is sequentially passed through a single channel 1×1 convolution, S-shaped function and bilinear upsampling to obtain a saliency prediction image T1 of the first scale; the decoder training loss of the first stage is calculated by T1 and the corresponding truth map to optimize the parameters of the decoder; the saliency prediction image T1 of the first scale is used as the saliency prediction image finally output by the saliency detection model.
[0080] Preferably, in step S205, the decoder training loss is calculated by the following formula:
[0081] L=L BCE +L Dice ;
[0082]
[0083] Where: L represents the decoder training loss; L BCE , L DiceRepresent the binary cross entropy loss function and Dice loss function respectively; N represents the saliency prediction image T n and the number of pixels of the true value map G; g i Represents the label. If a pixel point is predicted correctly, g i If the prediction fails, g i is 0; represents the probability of correct prediction of the pixel at position i in the saliency prediction map of the nth stage; represents the dot product operation; |T n ∩G| represents the number of corresponding pixel values in the saliency prediction map of the nth stage and the true value map that are the same.
[0084] Compared with the prior art, the efficient RGB-D saliency detection method using multiple information complementarity in the present invention has the following advantages:
[0085] Beneficial effects:
[0086] The present invention uses a backbone network to perform multi-scale feature extraction on RGB images and depth images respectively. At the same time, when extracting the depth feature map of the high-level scale, the RGB feature map of the corresponding scale is used as an auxiliary, and the information complementarity between the depth feature map and the RGB feature map and the multiple attention mechanism are combined for fusion enhancement. First, the RGB image provides rich visual information such as color and texture, while the depth image reveals the geometric structure in the scene and the distance relationship between objects. By fusing RGB features and depth features, the model can understand the scene more comprehensively, capture the spatial layout information that is difficult to directly observe in the RGB image and the detailed texture that may be missing in the depth image, and realize the information complementarity between the depth feature map and the RGB feature map, thereby improving the recognition ability of significant targets. Secondly, the multiple attention mechanism is used in the fusion process, which can automatically focus on the feature area that is most critical to saliency detection, ignoring irrelevant or noise information, which not only enhances the robustness of feature representation, but also enables the model to more effectively capture the key features of the target object, and enhances the feature representation ability and fusion ability of the model. Finally, by combining the complementary information of deep features and RGB features and the fusion process guided by the attention mechanism, a richer and more accurate feature representation can be generated. The enhanced feature map can provide more comprehensive clues in subsequent detection tasks and help the model locate salient targets more accurately.
[0087] On the basis of enhancing RGB features and depth features, the present invention further fuses the enhanced RGB feature map of the high-level scale with its RGB feature map to generate a fused feature map, and performs global and local feature collaborative depth enhancement on the low-level scale RGB feature map and the fused feature map, and then optimizes the backbone network parameters by calculating the backbone network training loss through the enhanced depth image and the input depth image. First, a deep enhancement strategy of collaborative global and local features is adopted, which not only considers the overall structural information of the depth image, but also pays attention to the detailed changes in the local area, so that the generated enhanced depth image has higher local accuracy while maintaining global consistency, thereby achieving the quality of the collaboratively enhanced depth map. Secondly, by further fusing the enhanced RGB feature map of the high-level scale with the original RGB feature map, a richer and more accurate fused feature map is generated. These feature maps fully integrate the visual information of the RGB image and the geometric information of the depth image, and can more comprehensively describe the objects in the scene and the relationship between them, thereby improving the accuracy of salient target detection. Finally, the difference between the enhanced depth image and the input depth image is used to calculate the training loss, which can more effectively guide the optimization process of the backbone network. The difference-based training strategy can force the network to pay more attention to the key information in the depth image, thereby improving the backbone network's ability to extract and represent deep features.
[0088] On the basis of generating a fused feature map, the present invention further performs several stages of feature splicing and fusion of several low-level scale RGB feature maps and fused feature maps through a decoder, generates saliency prediction images at each stage, and optimizes decoder parameters by calculating training loss by comparison with the true value map. First, the decoder can gradually combine high-level semantic information with low-level detail information through a multi-stage feature splicing and fusion process, thereby generating a more refined saliency prediction image. The multi-scale fusion method helps to capture the feature performance of saliency targets at different scales, thereby improving the accuracy and robustness of saliency detection. Secondly, the feature splicing and fusion operations in the decoder enable the features of different levels to be effectively integrated, generating more representative feature representations, and the enhanced feature fusion capability helps the model better understand and describe the objects in the scene and the relationship between them, thereby improving the performance of saliency detection. Finally, by calculating the decoder training loss at each stage by comparison with the true value map, the optimization process of the decoder can be accurately guided. This training strategy based on supervised learning can force the decoder to pay more attention to the key features of the saliency target, thereby improving its ability to generate saliency prediction images. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:
[0090] Figure 1 This is the overall network structure diagram of the saliency detection model;
[0091] Figure 2 This is the network structure diagram of the multi-modal complementarity (MCC) module;
[0092] Figure 3 The network structure diagram of the global / local feature collaborative deep enhancement (GLCDE) module;
[0093] Figure 4 This is a graph showing the prediction results of the model and other methods on three datasets in the experiment. DETAILED DESCRIPTION
[0094] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.
[0095] The following is a further detailed description through specific implementation methods:
[0096] Example:
[0097] This embodiment discloses an efficient RGB-D saliency detection method using multiple information complementarity.
[0098] like Figure 1 As shown, an efficient RGB-D saliency detection method using multiple complementary information includes:
[0099] S1: Obtain an RGB image and a depth image containing the target to be detected;
[0100] S2: Input the RGB image and the depth image into the trained saliency detection model and output the saliency prediction image of the target to be detected;
[0101] The training steps of the saliency detection model are as follows:
[0102] S201: Taking the RGB image and the depth image as training data and the corresponding truth image as input;
[0103] S202: Perform multi-scale feature extraction on the RGB image and the depth image through the backbone network (RGB Assisted Extraction and Fusion Stream of Depth, EFSD) to generate RGB feature maps and depth feature maps of several scales;
[0104] Specifically, in the RGB feature map and the depth feature map, the feature map with a larger size (smaller scale) is defined as a low-scale feature map; and the feature map with a smaller size (larger scale) is defined as a high-scale feature map.
[0105] In this embodiment, the RGB feature maps and depth feature maps of the first to third scales are defined as low-scale RGB feature maps and depth feature maps; the RGB feature maps and depth feature maps of the fourth to fifth scales are defined as high-scale RGB feature maps and depth feature maps.
[0106] When extracting high-level scale deep feature maps: use the RGB feature map of the corresponding scale as an auxiliary to the deep feature map, combine the information complementarity between the deep feature map and the RGB feature map and the multiple attention mechanism to interactively enhance the RGB features and the deep features, and generate the corresponding scale deep feature map and enhanced RGB feature map;
[0107] S203: Fusing the RGB feature map of the high-level scale with the enhanced RGB feature map of the corresponding scale to generate a fused feature map;
[0108] S204: Utilize the global and local information of the low-level RGB feature map and the fusion feature map to perform collaborative depth enhancement to generate an enhanced depth image; calculate the training loss of the backbone network by the enhanced depth image and the input depth image to optimize the parameters of the backbone network;
[0109] S205: performing feature concatenation and fusion of several stages on the low-level RGB feature map and the fused feature map through the decoder to generate saliency prediction images at each stage; calculating the decoder training loss at each stage through the saliency prediction images at each stage and the corresponding true value maps to optimize the decoder parameters;
[0110] S206: repeating steps S201 to S205 until the saliency detection model converges or reaches a preset number of iterations;
[0111] S3: Taking the saliency prediction image of the target to be detected as its saliency detection result.
[0112] The present invention uses a backbone network to perform multi-scale feature extraction on RGB images and depth images respectively. At the same time, when extracting the depth feature map of the high-level scale, the RGB feature map of the corresponding scale is used as an auxiliary, and the information complementarity between the depth feature map and the RGB feature map and the multiple attention mechanism are combined for fusion enhancement. First, the RGB image provides rich visual information such as color and texture, while the depth image reveals the geometric structure in the scene and the distance relationship between objects. By fusing RGB features and depth features, the model can understand the scene more comprehensively, capture the spatial layout information that is difficult to directly observe in the RGB image and the detailed texture that may be missing in the depth image, and realize the information complementarity between the depth feature map and the RGB feature map, thereby improving the recognition ability of significant targets. Secondly, the multiple attention mechanism is used in the fusion process, which can automatically focus on the feature area that is most critical to saliency detection, ignoring irrelevant or noise information, which not only enhances the robustness of feature representation, but also enables the model to more effectively capture the key features of the target object, and enhances the feature representation ability and fusion ability of the model. Finally, by combining the complementary information of deep features and RGB features and the fusion process guided by the attention mechanism, a richer and more accurate feature representation can be generated. The enhanced feature map can provide more comprehensive clues in subsequent detection tasks and help the model locate salient targets more accurately.
[0113] On the basis of enhancing RGB features and depth features, the present invention further fuses the enhanced RGB feature map of the high-level scale with its RGB feature map to generate a fused feature map, and performs global and local feature collaborative depth enhancement on the low-level scale RGB feature map and the fused feature map, and then optimizes the backbone network parameters by calculating the backbone network training loss through the enhanced depth image and the input depth image. First, a deep enhancement strategy of collaborative global and local features is adopted, which not only considers the overall structural information of the depth image, but also pays attention to the detailed changes in the local area, so that the generated enhanced depth image has higher local accuracy while maintaining global consistency, thereby achieving the quality of the collaboratively enhanced depth map. Secondly, by further fusing the enhanced RGB feature map of the high-level scale with the original RGB feature map, a richer and more accurate fused feature map is generated. These feature maps fully integrate the visual information of the RGB image and the geometric information of the depth image, and can more comprehensively describe the objects in the scene and the relationship between them, thereby improving the accuracy of salient target detection. Finally, the difference between the enhanced depth image and the input depth image is used to calculate the training loss, which can more effectively guide the optimization process of the backbone network. The difference-based training strategy can force the network to pay more attention to the key information in the depth image, thereby improving the backbone network's ability to extract and represent deep features.
[0114] On the basis of generating a fused feature map, the present invention further performs several stages of feature splicing and fusion of several low-level scale RGB feature maps and fused feature maps through a decoder, generates saliency prediction images at each stage, and optimizes decoder parameters by calculating training loss by comparison with the true value map. First, the decoder can gradually combine high-level semantic information with low-level detail information through a multi-stage feature splicing and fusion process, thereby generating a more refined saliency prediction image. The multi-scale fusion method helps to capture the feature performance of saliency targets at different scales, thereby improving the accuracy and robustness of saliency detection. Secondly, the feature splicing and fusion operations in the decoder enable the features of different levels to be effectively integrated, generating more representative feature representations, and the enhanced feature fusion capability helps the model better understand and describe the objects in the scene and the relationship between them, thereby improving the performance of saliency detection. Finally, by calculating the decoder training loss at each stage by comparison with the true value map, the optimization process of the decoder can be accurately guided. This training strategy based on supervised learning can force the decoder to pay more attention to the key features of the saliency target, thereby improving its ability to generate saliency prediction images.
[0115] In order to better introduce the technical solution of the present invention, this embodiment is described through the following parts.
[0116] 1. RGB feature map extraction
[0117] In this embodiment, MobileNetV2 is used as the backbone network for extracting RGB features. In order to adapt to the saliency detection task, the present invention removes the global average pooling layer and the last fully connected layer of MobileNetV2. In the MobileNetV2 backbone network, five consecutive stages of RGB feature extraction are performed on the RGB image to generate RGB feature maps of the first to fifth scales, which are denoted as R1, R2, R3, R4, and R5 respectively; the sizes of the RGB feature maps of the first to fifth scales are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input RGB image respectively. The step size of the first IRB in each stage is set to 2, and the channel expansion multiple r is set to 4.
[0118] The RGB feature extraction at each stage includes several inverted residual blocks connected end to end in sequence; combined with Figure 1 As shown in (d), the inverted residual block includes the first 1×1 convolution layer, the first BN layer, the first ReLu function layer, the 3×3 depth-separable convolution layer, the second BN layer, the second ReLu function layer, the second 1×1 convolution and the third BN layer, which are connected end to end in sequence; the input of the first 1×1 convolution layer and the output of the third BN layer are residually connected as the output of the inverted residual block.
[0119] 2. Deep Feature Map Extraction
[0120] In this embodiment, the backbone network performs three consecutive stages of depth feature extraction on the depth image, and sequentially generates depth feature maps of the first to third scales, i.e., low-level scale depth feature maps D1, D2, and d3. The high resolution of these depth feature maps leads to high computational cost. The depth feature extraction of each stage includes a number of inverted residual blocks connected end to end in sequence. The depth feature map of the third scale is then input into two auxiliary extraction modules connected end to end in sequence, and the depth feature maps of the fourth and fifth scales, i.e., high-level scale depth feature maps D4 and D5, are sequentially extracted and generated. Since the semantic information of D4 and D5 is relatively rich and the computational cost is low, in order to reduce the computational overhead, the present invention chooses to introduce RGB features in the deep stage of EFSD to assist in depth feature extraction and fusion.
[0121] The auxiliary extraction module includes an inverted residual block and a dual-flow complementary residual (DCR) module connected end to end in sequence; the dual-flow complementary residual module assists in the extraction and generation of the depth feature map by introducing the RGB feature map of the corresponding scale; in the auxiliary extraction module, the depth feature map of the N-1th scale is input into the inverted residual block, and the initial depth feature map of the Nth scale is output; the initial depth feature map and RGB feature map of the Nth scale are input into the dual-flow complementary residual module, and the depth feature map and enhanced RGB feature map of the Nth scale are output;
[0122] The calculation formula of the auxiliary extraction module is expressed as:
[0123]
[0124] Where: D n Denote the enhanced RGB feature map and depth feature map of the nth scale output by the auxiliary extraction module; D n-1 Represents the depth feature map of the N-1th scale; R n Represents the RGB feature map of the Nth scale; IRB represents the inverted residual block, the superscript of IRB represents the channel expansion multiple r, and the subscript represents the step size; DCR represents the dual-stream complementary residual module; n represents the nth stage of the backbone network, corresponding to the nth scale.
[0125] 1. Dual-stream complementary residual module (DCR)
[0126] Combination Figure 1As shown in (d), the dual-stream complementary residual module includes a multiple cross-modal complementation (MCC) module, a first BN layer, a first ReLu function layer, a 3×3 depth-separable convolution layer, a second BN layer, a second ReLu function layer, a 1×1 convolution, and a third BN layer, which are sequentially connected end to end; wherein the input of the multiple cross-modal complementary module is residually connected with the output of the third BN layer as the output of the dual-stream complementary residual module;
[0127] The multi-cross-modal complementary module utilizes the complementary information between the deep feature map and the RGB feature map and combines the multi-attention mechanism to enhance the interaction between RGB features and deep features.
[0128] The calculation formula of the dual-stream complementary residual module is as follows:
[0129]
[0130] Where: represents the enhanced RGB feature map and depth feature map output by the dual-stream complementary residual module; F RGB 、F D Represents the RGB feature map and depth feature map of the dual-stream complementary residual module input; BN represents the batch normalization layer; Dw_Conv represents a 3×3 depth-separable convolution; Conv 1×1 Represents 1×1 convolution; They represent the F of multiple cross-modal complementary modules based on the input RGB and F D Output enhanced RGB feature map and depth feature map.
[0131] The present invention designs a dual-stream complementary residual (DCR) module based on the MCC module to extract interactive feature map information. And based on the MCC and DCR modules, an RGB-assisted deep feature extraction and fusion backbone (backbone network) is proposed, so that RGB features can fully participate in the extraction and fusion of low-computational cost deep features, thereby ensuring the high efficiency of the entire network.
[0132] 2. Multi-modal complementary module (MCC)
[0133] Combination Figure 2 As shown, according to the information complementarity between the RGB image and the depth image, the present invention designs multiple cross-modal complementary modules to achieve cross-modal feature interaction between the RGB image and the depth image.
[0134] MCC is a module that can use the complementary information between RGB and depth images to enhance the interaction between RGB features and depth features. Based on the complementary characteristics of the color texture information of RGB images and the spatial information of depth images, the present invention designs this module under the framework of the triple attention mechanism. Figure 2 As shown in (c), in the MCC module, the depth feature map and the RGB feature map are input into two parallel streams respectively. To achieve information complementarity between cross-modal inputs, the MCC module calculates and exchanges the attention weights representing the channel and width, channel and height, and the dependency between spatial dimensions of the input features to achieve information complementarity. The specific process is as follows:
[0135] S2021: RGB feature map F RGB ∈R H×W×C and the deep feature map F D ∈R H×W×C As the input of the multi-cross-modal complementary module, W represents width, H represents height, and C represents the number of channels;
[0136] S2022: Calculate F RGB and F D Attention weights for correlation between channel and width dimensions Contains representative F RGB and F D Attention weights for correlation between channel and width dimensions and Therefore, the present invention splits along the channel dimension To obtain a shape of (C×W×1) and pass and Combined with F RGB and F D calculate and Will and Rotate 90° clockwise along the W axis to maintain the shape of the original input tensor;
[0137] Calculating attention weights When: F RGB and F D Rotate 90° counterclockwise along the W axis (such as Figure 2 (a) shows the tensor and Will and Splice to get This operation can simultaneously calculate F RGB and FD The attention weight of Input the Attention Generation (AG) module to generate attention weights
[0138] The calculation formula is as follows:
[0139] W RGB-D =σ(BN(Conv k×k (Z-pool(x))));
[0140] Where: σ represents the sigmoid function; Conv represents the standard convolution operation; BN represents the batch normalization layer; Z-pool represents the Z pooling layer; k represents the size of the convolution kernel; when calculating the attention weights representing the channel and width, channel and height, and the dependency between the spatial dimensions of the input features, replace x in the formula with Then W RGB-D Respectively expressed as and
[0141] and Exchange and fuse each other's attention weights to complete the information exchange between the channel and width dimensions. The calculation formula is as follows:
[0142]
[0143] Where: represents the element-wise multiplication operator;
[0144] S2023: Calculate F RGB and F D Attention weights for correlation between channel and height dimensions The shape is (H×C×2); Split into F along the height dimension RGB and F D The attention weight and (Shape is (H×C×1)); by and Combined with F RGB and F D calculate and Will and Rotate 90° clockwise along the H axis to maintain the shape of the original input tensor;
[0145] Calculating attention weights When: F RGB and FD Rotate 90° counterclockwise along the H axis to get the tensor and Will and Splice to get Will Input attention generation module generates attention weights
[0146] and Exchange and fuse each other's attention weights to complete the information exchange between the channel and height dimensions. The calculation formula is as follows:
[0147]
[0148] S2024: Calculate F RGB and F D The spatial attention weight Will Split into F along the spatial dimension RGB and F D The spatial attention weight and (Shape is (H×C×1)); through and Combined with F RGB and F D calculate and Will and Rotate 90° clockwise along the C axis to maintain the shape of the original input tensor;
[0149] Calculating spatial attention weights When: F RGB and F D Concatenate into tensors Will Input attention generation module generates attention weights The shape is (H×W×2);
[0150] Calculated by the following formula and
[0151]
[0152] S2025: Through F RGB , and Calculate the enhanced RGB feature map F RGB-D ;
[0153] In this embodiment, the input tensor FRGB and F D The corresponding outputs after information exchange between different dimensions are added, and the obtained tensors are concatenated in the channel dimension. After multiple interactions, the final output F is obtained in the MCC module. RGB-D and F D-RGB , which are the corresponding outputs after multiple interactions between RGB features and depth features. The formula is expressed as:
[0154]
[0155] Where: cat represents concatenation (concatenation) in the channel dimension;
[0156] S2026: Pass F D , and Calculate the deep feature map F D-RGB ;
[0157] The formula is:
[0158]
[0159] Based on the information complementarity between deep feature maps and RGB feature maps and the triple attention mechanism, this paper proposes a multiple cross-modal complementary module to enhance the feature representation and fusion capabilities of lightweight neural networks.
[0160] 3. Fusion Feature Map Generation
[0161] In this embodiment, the structure of the deep stage of EFSD is as follows Figure 2 As shown in (b), each stage consists of two IRB modules and one DCR module. The deep features from the previous stage first pass through IRB, and the r of IRB is set to 4 and the step size is 2. After the deep features output by IRB and the RGB features of the same size are input into the DCR module, the extracted deep features D and RGB features R are obtained respectively. DCR Since the depth feature plays an auxiliary role in the fusion process and the RGB feature has richer semantic information, the present invention fuses R by multiplying the corresponding elements while allowing D to enter the next stage. DCR Finally, the multiplied features are further extracted by IRB with r=6 to obtain the fused feature map.
[0162] The calculation formula of the fused feature map is:
[0163]
[0164] Where: F n Represents the fused feature map of the nth scale; IRB represents the inverted residual block, the superscript of IRB represents the channel expansion multiple r, and the subscript represents the step size; Represents the corresponding element multiplication operation; R n Represents the RGB feature map of the nth scale; Represents the enhanced RGB feature map of the nth scale.
[0165] In this embodiment, the depth feature map of the third scale is input into the inverted residual block, and the initial depth feature map of the fourth scale is output; the initial depth feature map and the RGB feature map of the fourth scale are input into the dual-stream complementary residual module to generate a depth feature map and an enhanced RGB feature map of the fourth scale; the enhanced RGB feature map of the fourth scale and the RGB feature map of the fourth scale are multiplied element by element, input into the inverted residual block, and the fused feature map of the fourth scale is output.
[0166] The depth feature map of the fourth scale is input into the inverted residual block, and the initial depth feature map of the fifth scale is output; the initial depth feature map and the RGB feature map of the fifth scale are input into the dual-stream complementary residual module to generate the depth feature map and the enhanced RGB feature map of the fifth scale; the enhanced RGB feature map of the fifth scale and the RGB feature map of the fifth scale are multiplied element by element, input into the inverted residual block, and the fused feature map of the fifth scale is output.
[0167] 4. Enhanced Depth Image Generation
[0168] Due to the simplicity of structure, lightweight backbone networks are usually not as good as other backbone networks in feature extraction, which in turn affects the accuracy of the network. Therefore, the quality of the depth map is more critical for efficient saliency detection methods. The present invention finds that there is more noise in low-quality depth maps, and the depth difference between the detected object and the smooth background is not obvious. In view of the above analysis, the present invention designs a global / local features cooperative depth enhancement (Global and Local Features Cooperation Depth Enhancement, GLCDE) module, which divides RGB features and fusion features of different sizes into global / local feature groups, and uses the DCR module to extract the details and semantic information of the two feature groups respectively, so as to help the depth map suppress noise and enhance the depth contrast between the detected object and the smooth background.
[0169] Combination Figure 3 As shown in Figure 2, GLCDE generates an enhanced depth image through the following steps:
[0170] S2041: Take the RGB feature maps R1, R2 and R3 of the first to third scales, and the fusion feature maps F4 and F5 of the fourth and fifth scales as input; R1, R2 and R3 are used as the global feature group (F global ), R3, F4 and F5 are used as local feature groups (Flocal ), their pixel receptive fields have a smaller overlapping area, so they can capture more detailed information, including the information of the pixel itself and its neighborhood. In this embodiment, R3 is added to both feature groups.
[0171] S2042: To enrich the features with fewer channels, especially F global The present invention adopts sub-pixel convolution to convolve and reconstruct features with sufficient information but low resolution by rearranging pixels to generate high-resolution features. Although this method will lead to a reduction in the number of channels, it will not introduce useless information.
[0172] For the global feature group: splice the features of R3 after pixel rearrangement to R2; splice the features of R2 after pixel rearrangement to R1; expand the number of channels of R3, R2 and R1 to 256, and adjust the resolution of R2 and R1 to be the same as R3; splice the adjusted R1, R2 and R3 to obtain the global combined feature F G ; Through 1×1 convolution, F G The number of channels is reduced to 256;
[0173] S2043: For the local feature group: splice the feature after pixel rearrangement of F5 to F4; expand the number of feature channels of F4, F5 and R3 to 256, and adjust the resolution of F4 and F5 to be the same as R3; splice the adjusted R3, F4 and F5 along the channel dimension to obtain the local combined feature F L ; Through 1×1 convolution, F L The number of channels is reduced to 256;
[0174] S2044: Since the size of R3 is 1 / 8 of the original image, it cannot show complete details, so F G As an auxiliary.
[0175] F L Input two inverted residual blocks connected end to end; the output of the second inverted residual block is combined with F G The two two-stream complementary residual modules are connected end to end to generate a global lift graph T. G and the local lift graph T L ; The global lifting graph T is transformed through 1×1 convolution G and the local lift graph T L Convert to single channel;
[0176] S2045: Use sigmoid function and bilinear upsampling to globally improve graph T G and the local lift graph T L The size of is adjusted to be consistent with the input depth image; the adjusted global lift map T Gand the local lift graph T L Add and average to generate an enhanced (quality improved) depth image;
[0177] The formula is:
[0178] T G ,T L =σ(upsampling(Conv 1×1 (2DCRB(F G ,2IRB(F L )))));
[0179]
[0180] Where: D i represents the enhanced depth image; T G 、T L Represent the global lift graph and the local lift graph respectively; F G 、F L Represents global combined features and local combined features; IRB represents inverted residual block; DCR represents dual stream complementary residual module; Conv 1×1 represents 1×1 convolution; upsampling represents bilinear upsampling; σ represents the sigmoid function;
[0181] S2046: GLCDE training loss uses SSIM metric to supervise D i With the input depth map D g The structural similarity between them. The training loss of the backbone network is calculated by the following formula to optimize the parameters of the backbone network;
[0182] S GDCDR =1-SSIM(D i ,D g );
[0183] Where: S GDCDR represents the training loss of the backbone network; SSIM represents the structural similarity, i.e., SSIM metric, and SSIM uses the default settings; D i Denotes the enhanced depth image; D g Represents the input depth image. GLCDE is only used to enhance the depth map during the training phase and is omitted during testing and is not counted as a model parameter.
[0184] The present invention designs a global / local feature collaborative depth map enhancement (GLCDE) module, which increases the depth contrast between the smooth area and the target object in the image and suppresses the depth map noise. The GLCDE module improves the detection accuracy of the network by improving the quality of the depth map.
[0185] 5. Decoder
[0186] The input of the decoder is the features derived from the encoder (i.e., the backbone network), including RGB feature maps R1, R2, R3 and fused feature maps F4, F5. In order to achieve a balance between network accuracy and lightweight, the design of the decoder considers both effectiveness and efficiency. The decoder is mainly composed of five Compact Pyramid Refinements (CPRs). In each CPR, the input features are first expanded four times along the channel dimension through a 1×1 convolution, followed by three parallel 3×3 depth-separable volumes with expansion rates of 1, 2, and 3, respectively. Then, the output features of each branch are element-wise added, and the summed features are passed through the ReLu layer, BN layer, and 1×1 convolution to reduce the number of channels, and the obtained output is residually connected with the initial input. Finally, the attention mechanism is used to correct the fused features. Features of different receptive fields can be obtained through the CPR module, which can help the decoder to efficiently complete multi-level feature fusion. In order to make full use of the semantic information contained in the fused features, the present invention adds a convolutional block attention module (CBAM) to the two CPR inputs F4 and F5 to help the prediction graph further focus on the detection target.
[0187] Knot Figure 2 As shown in (b), the structure of the decoder in this embodiment is as follows: There are five stages in the decoding end. First, the feature map of this stage is concatenated with the feature map from the previous stage by reducing the number of channels by half through 1×1 convolution, and then feature fusion is performed through ordinary CPR or CPR with CBAM. The fused feature map is sequentially passed through 1×1 convolution with a single channel, sigmoid function and bilinear upsampling to obtain the saliency prediction map T of this stage. n At the same time, the output also flows into the next stage encoding end to be spliced with the next stage input. In this way, the decoder of the present invention efficiently aggregates multi-level features from the last stage to the first stage.
[0188] Specifically, step S205 includes the following steps:
[0189] S2051: Use the RGB feature maps R1, R2 and R3 of the first to third scales, and the fusion feature maps F4 and F5 of the fourth and fifth scales as inputs of the decoder;
[0190] S2052: input F5 into the compact pyramid refinement module and the convolutional attention mechanism module connected end to end in sequence, and output the feature fusion image of the fifth stage; the feature fusion image of the fifth stage is sequentially passed through a single channel 1×1 convolution, a sigmoid function and bilinear upsampling to obtain a saliency prediction image T5 of the fifth scale; the decoder training loss of the fifth stage is calculated through T5 and the corresponding truth map to optimize the decoder parameters;
[0191] In the compact pyramid refinement module, the input feature map is first expanded four times along the channel dimension through 1×1 convolution; then it passes through three parallel 3×3 depthwise separable convolution branches (the expansion rates of the convolution are 1, 2, and 3 respectively); then the output features of each branch are element-wise added, and the added features are reduced in number of channels through the ReLu function layer, BN layer, and 1×1 convolution, and finally the obtained output is residually connected with the input feature map as the output of the compact pyramid refinement module;
[0192] In this embodiment, the existing convolutional attention mechanism module is used. The convolutional attention mechanism module consists of two key parts: a channel attention module and a spatial attention module. The working logic of the convolutional attention mechanism module is not repeated in this embodiment.
[0193] S2053: After splicing the feature fusion image of the fifth stage and F4, the concatenation is input into the compact pyramid refinement module and the convolution attention mechanism module which are connected end to end in sequence, and the feature fusion image of the fourth stage is output; after the feature fusion image of the fourth stage is sequentially passed through a single channel 1×1 convolution, a sigmoid function and bilinear upsampling, a saliency prediction image T4 of the fourth scale is obtained; the decoder training loss of the fourth stage is calculated through T4 and the corresponding truth map to optimize the decoder parameters;
[0194] S2054: After splicing the feature fusion image of the fourth stage and R3, input them into the compact pyramid refinement module, and output the feature fusion image of the third stage; after the feature fusion image of the third stage passes through the 1×1 convolution of a single channel, the sigmoid function and the bilinear upsampling in sequence, a saliency prediction image T3 of the third scale is obtained; the decoder training loss of the third stage is calculated by T3 and the corresponding truth map to optimize the decoder parameters;
[0195] S2055: After splicing the feature fusion image of the third stage and R2, input them into the compact pyramid refinement module, and output the feature fusion image of the second stage; after the feature fusion image of the second stage passes through the 1×1 convolution, sigmoid function and bilinear upsampling of a single channel in sequence, a saliency prediction image T2 of the second scale is obtained; the decoder training loss of the second stage is calculated by T2 and the corresponding truth map to optimize the decoder parameters;
[0196] S2056: After splicing the feature fusion image of the second stage and R1, input them into the compact pyramid refinement module to output the feature fusion image of the first stage; the feature fusion image of the first stage is sequentially passed through a single channel 1×1 convolution, S-shaped function and bilinear upsampling to obtain a saliency prediction image T1 of the first scale; the decoder training loss of the first stage is calculated by T1 and the corresponding truth map to optimize the parameters of the decoder; the saliency prediction image T1 of the first scale is used as the saliency prediction image finally output by the saliency detection model.
[0197] In this embodiment, the significance prediction map T of the five stages n The loss function between (n=1, 2, 3, 4, 5) and the significance truth graph G is obtained by combining the binary cross-entropy loss function (BCE) and the Dice loss function. The Dice loss function is a set similarity measurement function.
[0198] Specifically, the decoder training loss is calculated by the following formula:
[0199] L=L BCE +L Dice ;
[0200]
[0201]
[0202] Where: L represents the decoder training loss; L BCE , L Dice Represent the binary cross entropy loss function and Dice loss function respectively; N represents the saliency prediction image T n and the number of pixels of the true value map G; g i Represents the label. If a pixel point is predicted correctly, g i If the prediction fails, g i is 0; represents the probability of correct prediction of the pixel at position i in the saliency prediction map of the nth stage; represents the dot product operation; |T n∩G| represents the number of corresponding pixel values in the saliency prediction map of the nth stage and the true value map that are the same.
[0203] 6. Experimental Description
[0204] In order to better illustrate the advantages of the technical solution of the present invention, this embodiment discloses the following experiment.
[0205] 1. Experimental detailed parameter settings and data sets
[0206] This experiment implements the proposed network on the PyTorch framework. The network consists of an encoder based on MobileNetV2 and EFSD, GLCDE, and a decoder. In order to ensure the consistency of the depth map and RGB image, the resolution of both images is adjusted to 320×320. Data augmentation operations such as horizontal flipping and random cropping are performed to improve the generalization performance of the model.
[0207] This experiment uses an RTX 4090 GPU for training and testing, and the following hyperparameter configurations are used: the initial learning rate is set to 0.0001, the batch size is 10, and the training cycle is 60 epochs. This experiment uses the Adam optimizer to optimize the network of the present invention, in which the parameters β1 and β2 are set to 0.9 and 0.99 respectively, and the weight decay rate is 0.0001. This experiment was conducted on five datasets, including NLPR, DUTLF-D, SSD, STERE, and NJU2K. This experiment selected 700 samples from NLPR and 1500 samples from NJU2K for training, and then selected other datasets from NLPR and NJU2K and the remaining datasets for testing. This experiment also evaluated the proposed network on five widely used RGB-DSOD datasets, including NLPR, NJU2K, SSD, and STERE. For the DUTLF-D dataset, this experiment used 800 images for training and another 400 images for testing. These experimental parameter settings and data set selections ensured the repeatability of the experiments and the comparability of the results.
[0208] 2. Evaluation indicators
[0209] This experiment uses four widely used evaluation indicators to comprehensively evaluate the performance of the model. The first is F-measure (F β ), this indicator can comprehensively measure the accuracy of the model's prediction results, and its calculation method is as follows:
[0210]
[0211] Among them, Precision represents the precision rate, Recall represents the recall rate, and usually β 2It is set to 0.3, which is based on the experience of previous salient target detection. This experiment calculates the maximum F under different thresholds β As The higher the value, the better the model's performance. The second is the mean absolute error (MAE), which measures the average absolute error between the predicted value and the true value. The lower the MAE value, the more accurate the model's prediction. The last two indicators are S-measure (S ∝ ) and maximum E- These two indicators are also one of the important criteria for evaluating model performance. Taking these four indicators into consideration, this experiment ranks the prediction performance of each method on each dataset. The selection of these evaluation indicators is intended to comprehensively and accurately evaluate the performance of the model so as to make objective comparison and analysis.
[0212] 3. Experimental comparison
[0213] On five challenging datasets (NLPR, SSD, NJU2K, DUTLF-D and STERE), the method of the present invention is compared with 12 most recent and reproducible methods, and the results are shown in Table 1. The prediction graphs of the comparison methods are all from official releases or tested using officially released models. Among the RGB-D SOD methods compared in this experiment, ACSD and DCMC are based on manual feature extraction; DFM-Net, MFF, MobileSal and MoADNet are lightweight SOD methods based on lightweight backbone networks; other methods are CNN-based methods, most of which use VGG as the network backbone. From the perspective of prediction accuracy, on the NLPR dataset, the various indicators of the method of the present invention are ranked in the top two. On the SSD dataset, all indicators are the best. In order to more intuitively show the comprehensive prediction results of each method in the three datasets, on the one hand, Table 1 gives the average values of the evaluation indicators of each method in the five datasets. Considering the evaluation indicators of the three datasets, the prediction results of the method of the present invention are the best. On the other hand, Figure 4 The prediction results of the method of the present invention and other six representative methods on five data sets are shown. By comparison, it can be found that the method of the present invention can effectively extract salient targets when processing complex scenes and depth information containing noise, and has better performance than other methods.
[0214] In addition, Table 1 shows the network parameter size of the present invention and the comparative method as well as the calculation speed on the RTX 4090 GPU. It can be found that the parameters of the network of the present invention are 5.5M, which is the third smallest among all methods. The calculation speed of the network of the present invention is the third fastest among all methods. Moreover, compared with other networks based on traditional CNN, the network parameters of the present invention are much smaller and the calculation speed is much faster. For example, the parameters of S2MA are 16 times larger than those of the network of the present invention, and the calculation speed is less than 1 / 10 of that of the method of the present invention; the parameters of DMRA are 10 times larger than those of the network of the present invention, and the calculation speed is 9 times slower. These results show that the lightweight SOD network of the present invention has both high precision and small model parameters and fast calculation speed.
[0215] Table 1 Comparison of prediction performance on five challenging datasets
[0216]
[0217]
[0218] In Table 1, “bold+underlined” indicates the best, “underlined” indicates the second best, and “bold” indicates the third best.
[0219] 4. Ablation experiment
[0220] This experiment evaluates the prediction results of each group of experiments on the NLPR dataset. and MAE are used as the main indicators. In addition, since there are many experimental groups in Table 2, in order to more accurately show the effects of different fusion strategies and modules, this experiment evaluates the prediction results of each group on the NLPR and SSD datasets.
[0221] In order to verify the fusion effect at different stages of EFSD and the effectiveness of the GLCDE module, the present invention conducted the ablation experiments in Table 2. Experiments No. 1-6 mainly studied the impact of fusing RGB and depth features at different stages on the final prediction results without GLCDE. Experiments No. 7-12 focused on the impact of fusing RGB features and depth features at different stages on the final prediction results with GLCDE. Experiment No. 13 shows the prediction effect of the network model without GLCDE without fusing features. The indicators in Table 2 show that the prediction results of Experiments No. 5 and Experiment No. 11 are the best. Since the features extracted in the deep stage contain more semantic information, the present invention sets a DCR module in the deep stage of EFSD to introduce RGB features to efficiently extract and fuse features. Such a deployment most significantly improves the final prediction results while saving computational costs, which is consistent with the original design intention of EFSD. By comparing Experiments No. 1-6 and Experiments No. 7-12, it is found that in almost all cases, the prediction effect of the network with GLCDE is better than that of the module without GLCDE, which confirms the effectiveness of GLCDE proposed in the present invention.
[0222] Finally, in Experiment No. 14, after the depth and RGB features complement each other through the MCC module, this experiment uses the output corresponding to the RGB feature as the fusion feature for prediction. By comparing the results of Experiment No. 13 and Experiment No. 14, it can be found that in the MCC module, the RGB feature obtains the position information by interacting with the depth feature, which improves the prediction effect of the final model. This further verifies the effectiveness of the MCC module proposed in this invention.
[0223] Table 2 Ablation study of EFSD fusion part and GLCDE
[0224]
[0225] In Table 2, Represents the output corresponding to R after the depth feature D and RGB feature R are input into the MCC module.
[0226] In order to verify the effectiveness of EFSD and its structural deployment, this experiment conducted the ablation experiment in Table 3. In this experiment, the deep features output by the backbone network are fused with the RGB features of the same size by element-by-element multiplication. It is worth noting that the backbone network replaces the DCR module in EFSD with IRB. In the No.3 experiment, the r of IRB in the deep stage of EFSD is set to 4. The deployment of experiment No.4 is consistent with the structural deployment of EFSD proposed in the present invention. The No.2 and No.4 experiments are compared, and the results show the effectiveness of the DCR module in EFSD. Because it introduces RGB features when extracting deep features, on the one hand, it helps the deep features to suppress noise; on the other hand, the deep features complement the spatial information of the detection target to the RGB features, completing the preliminary fusion. Comparing the results of No.3 and No.4 experiments, it can be verified that the present invention changes the r of IRB for extracting fusion features in EFSD from 4 to 6, so that more features can be involved in the extraction of fusion features. Finally, by comparing the No.1 and No.4 experiments, it can be more clearly demonstrated that EFSD significantly improves the network detection performance proposed in the present invention.
[0227] Table 3 Efficacy of deep feature extraction and fusion backbone (EFSD) assisted by RGB features
[0228]
[0229] In Table 3, “RGB” represents the RGB backbone network and its input. “Depth” represents the backbone network with DCR in EFSD replaced by IRB. The representative fusion feature is obtained directly through element multiplication. The r of IRB in the fourth and fifth stages (deep networks) of EFSD is set to 4.
[0230] The GLCDE module can improve the quality of the depth map, which in turn affects the prediction accuracy of the entire model. As shown in Table 4, this experiment conducted a series of analytical experiments on the deployment of the GLCDE structure. It aims to study the impact of the DCR module and pixel shuffle operation in GLCDE on the prediction performance of GLCDE and the entire model. From the comparison of the three groups of experimental results using GLCDE, it can be seen that the experiments using GLCDE are better than the two experiments using GLCDE without the DCR module or pixel shuffle operation in terms of prediction accuracy. This proves the effectiveness of the DCR module and pixel shuffle operation in GLCDE. Because the pixel shuffle operation uses low-resolution features for convolution and reconstruction between multiple channels to generate high-resolution features, it enriches local feature groups with fewer channels. Adding the DCR module to GLCDE can make the local combination feature G, where the initial features are insufficient, more accurate.L Participate in the extraction process to enrich the detail information of the depth enhancement map.
[0231] Comparing the experiments without GLCDE and with GLCDE, we can see that the GLCDE module has a significant improvement on the final prediction effect. The fundamental reason is that GLCDE uses the semantic information and detail information of the two feature groups to effectively reduce the impact of noise on the depth features and enhance the depth contrast between the detected object and the smooth background.
[0232] Table 4 GLCDE structure deployment
[0233]
[0234] In Table 4, '×DCR' means that DCR in GLCDE is replaced by IRB, and the local combination feature F mentioned in 3.3 is replaced by IRB. L and the global combined feature F G After concatenation, the concatenation is input into a continuous IRB. '×Pixel Shuffle' means canceling the sub-pixel convolution pixel shuffle operation of GLCDE.
[0235] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.
Claims
1. An efficient RGB-D saliency detection method using multiple complementary information, characterized in that: include: S1: Obtain an RGB image and a depth image containing the target to be detected; S2: Input the RGB image and the depth image into the trained saliency detection model and output the saliency prediction image of the target to be detected; The training steps of the saliency detection model are as follows: S201: Taking the RGB image and the depth image as training data and the corresponding truth map as input; S202: Perform multi-scale feature extraction on the RGB image and the depth image through the backbone network to generate RGB feature maps and depth feature maps of several scales; When extracting high-level scale deep feature maps: use the RGB feature map of the corresponding scale as an auxiliary to the deep feature map, combine the information complementarity between the deep feature map and the RGB feature map and the multiple attention mechanism to interactively enhance the RGB features and the deep features, and generate the corresponding scale deep feature map and enhanced RGB feature map; S203: Fusing the RGB feature map of the high-level scale with the enhanced RGB feature map of the corresponding scale to generate a fused feature map; S204: Utilize the global and local information of the low-level RGB feature map and the fusion feature map to collaboratively perform depth enhancement to generate an enhanced depth image; Optimize the parameters of the backbone network by calculating the backbone network training loss through the enhanced depth image and the input depth image; S205: performing feature concatenation and fusion of several stages on the low-level RGB feature map and the fused feature map through the decoder to generate saliency prediction images at each stage; calculating the decoder training loss at each stage through the saliency prediction images at each stage and the corresponding true value maps to optimize the decoder parameters; S206: repeating steps S201 to S205 until the saliency detection model converges or reaches a preset number of iterations; S3: Taking the saliency prediction image of the target to be detected as its saliency detection result.
2. The efficient RGB-D saliency detection method using multiple information complementation as claimed in claim 1, characterized in that: In step S202, the backbone network performs five consecutive stages of RGB feature extraction on the RGB image, and generates RGB feature maps of the first to fifth scales in sequence; wherein the RGB feature extraction of each stage includes a plurality of inverted residual blocks connected end to end in sequence; The inverted residual block includes the first 1×1 convolution layer, the first BN layer, the first ReLu function layer, the 3×3 depth-separable convolution layer, the second BN layer, the second ReLu function layer, the second 1×1 convolution and the third BN layer, which are connected end to end in sequence; the input of the first 1×1 convolution layer and the output of the third BN layer are residually connected as the output of the inverted residual block.
3. The efficient RGB-D saliency detection method using multiple information complementation as claimed in claim 2, characterized in that: In step S202, the backbone network performs three consecutive stages of depth feature extraction on the depth image to sequentially generate depth feature maps of the first to third scales; wherein the depth feature extraction of each stage includes a plurality of inverted residual blocks connected end to end in sequence; the depth feature map of the third scale is input into two auxiliary extraction modules connected end to end in sequence to sequentially extract and generate depth feature maps of the fourth and fifth scales; The auxiliary extraction module includes an inverted residual block and a two-stream complementary residual module connected end to end in sequence; the two-stream complementary residual module assists in the extraction and generation of the depth feature map by introducing the RGB feature map of the corresponding scale; in the auxiliary extraction module, the depth feature map of the N-1th scale is input into the inverted residual block, and the initial depth feature map of the Nth scale is output; The initial depth feature map and RGB feature map of the Nth scale are input into the dual-stream complementary residual module, and the depth feature map and enhanced RGB feature map of the Nth scale are output; The calculation formula of the auxiliary extraction module is expressed as: Where: D n Denote the enhanced RGB feature map and depth feature map of the nth scale output by the auxiliary extraction module; D n-1 Represents the depth feature map of the N-1th scale; R n Represents the RGB feature map of the Nth scale; IRB represents the inverted residual block, the superscript of IRB represents the channel expansion multiple r, and the subscript represents the step size; DCR represents the dual-stream complementary residual module; n represents the nth stage of the backbone network, corresponding to the nth scale.
4. The efficient RGB-D saliency detection method using multiple information complementation as claimed in claim 3, characterized in that: In step S202, the dual-stream complementary residual module includes a multiple cross-modal complementary module, a first BN layer, a first ReLu function layer, a 3×3 depth-separable convolution layer, a second BN layer, a second ReLu function layer, a 1×1 convolution, and a third BN layer, which are sequentially connected end to end; wherein the input of the multiple cross-modal complementary module and the output of the third BN layer are residually connected as the output of the dual-stream complementary residual module; The multi-cross-modal complementary module utilizes the complementary information between the deep feature map and the RGB feature map and combines the multi-attention mechanism to enhance the interaction between RGB features and deep features.
5. The efficient RGB-D saliency detection method using multiple information complementation as claimed in claim 4, characterized in that: In step S202, the processing steps of the multiple cross-modal complementary module are as follows: S2021: RGB feature map F RGB ∈R H×W×C and the deep feature map F D ∈R H×W×C As the input of the multi-modal complementary module, W represents width, H represents height, and C represents the number of channels; S2022: Calculate F RGB and F D Attention weights for correlation between channel and width dimensions Will Split into F along the channel dimension RGB and F D The attention weight and pass and Combined with F RGB and F D calculate and Will and Rotate 90° clockwise along the W axis; Calculating attention weights When: F RGB and F D Rotate 90° counterclockwise along the W axis to get the tensor and Will and Splice to get Will Input attention generation module generates attention weights Calculated by the following formula and Where: represents the element-wise multiplication operator; S2023: Calculate F RGB and F D Attention weights for correlation between channel and height dimensions Will Split into F along the height dimension RGB and F D The attention weight and pass and Combined with F RGB and F D calculate and Will and Rotate 90° clockwise along the H axis; Calculating attention weights When: F RGB and F D Rotate 90° counterclockwise along the H axis to get the tensor and Will and Splice to get Will Input attention generation module generates attention weights Calculated by the following formula and S2024: Calculate F RGB and F D The spatial attention weight Will Split into F along the spatial dimension RGB and F D The spatial attention weight and pass and Combined with F RGB and F D calculate and Will and Rotate 90° clockwise along the C axis; Calculating spatial attention weights When: F RGB and F D Concatenate into tensors Will Input attention generation module generates attention weights Calculated by the following formula and S2025: Through F RGB , and Calculate the enhanced RGB feature map F RGB-D ; The formula is: Where: cat represents the splicing in the channel dimension; S2026: Pass F D , and Calculate the deep feature map F D-RGB ; The formula is:
6. The efficient RGB-D saliency detection method using multiple information complementation as claimed in claim 5, characterized in that: The calculation formula of the dual-stream complementary residual module is as follows: Where: represents the enhanced RGB feature map and depth feature map output by the dual-stream complementary residual module; F RGB 、F D Represents the RGB feature map and depth feature map of the dual-stream complementary residual module input; BN represents the batch normalization layer; Dw_Conv represents a 3×3 depth-separable convolution; Conv 1×1 Represents 1×1 convolution; They represent the F of multiple cross-modal complementary modules based on the input RGB and F D Output enhanced RGB feature map and depth feature map.
7. The efficient RGB-D saliency detection method using multiple information complementation as claimed in claim 3, characterized in that: In step S203, the depth feature map of the third scale is input into the inverted residual block, and an initial depth feature map of the fourth scale is output; The initial depth feature map and RGB feature map of the fourth scale are input into the dual-stream complementary residual module to generate a depth feature map and an enhanced RGB feature map of the fourth scale; the enhanced RGB feature map of the fourth scale and the RGB feature map of the fourth scale are element-wise multiplied and input into the inverted residual block to output a fused feature map of the fourth scale; Input the depth feature map of the fourth scale into the inverted residual block and output the initial depth feature map of the fifth scale; The initial depth feature map and RGB feature map of the fifth scale are input into the dual-stream complementary residual module to generate a depth feature map and an enhanced RGB feature map of the fifth scale; the enhanced RGB feature map of the fifth scale and the RGB feature map of the fifth scale are multiplied element by element, and then input into the inverted residual block to output a fused feature map of the fifth scale; The calculation formula of the fused feature map is: (n=4,5); Where: F n Represents the fused feature map of the nth scale; IRB represents the inverted residual block, the superscript of IRB represents the channel expansion multiple r, and the subscript represents the step size; Represents the corresponding element multiplication operation; R n Represents the RGB feature map of the nth scale; Represents the enhanced RGB feature map of the nth scale.
8. The efficient RGB-D saliency detection method using multiple information complementation as claimed in claim 7, characterized in that: In step S204, an enhanced depth image is generated by the following steps: S2041: taking the RGB feature maps R1, R2 and R3 of the first to third scales, and the fused feature maps F4 and F5 of the fourth and fifth scales as input; wherein R1, R2 and R3 are taken as a global feature group, and R3, F4 and F5 are taken as a local feature group; S2042: For the global feature group: splice the features of R3 after pixel rearrangement to R2; splice the features of R2 after pixel rearrangement to R1; expand the number of channels of R3, R2 and R1 to 256, and adjust the resolution of R2 and R1 to be the same as R3; splice the adjusted R1, R2 and R3 to obtain the global combined feature F G ; Through 1×1 convolution, F G The number of channels is reduced to 256; S2043: For the local feature group: splice the feature after pixel rearrangement of F5 to F4; expand the number of feature channels of F4, F5 and R3 to 256, and adjust the resolution of F4 and F5 to be the same as R3; splice the adjusted R3, F4 and F5 along the channel dimension to obtain the local combined feature F L ; Through 1×1 convolution, F L The number of channels is reduced to 256; S2044: F L Input two inverted residual blocks connected end to end; the output of the second inverted residual block is combined with F G The two two-stream complementary residual modules are connected end to end to generate a global lift graph T. G and the local lift graph T L ; The global lifting graph T is transformed through 1×1 convolution G and the local lift graph T L Convert to single channel; S2045: Use sigmoid function and bilinear upsampling to globally improve graph T G and the local lift graph T L The size of is adjusted to be consistent with the input depth image; the adjusted global lift map T G and the local lift graph T L Add and average to generate an enhanced depth image; The formula is: T G ,T L =σ(upsampling(Conv 1×1 (2DCRB(F G ,2IRB(F L ))))); Where: D i represents the enhanced depth image; T G , T L Represent the global lift graph and the local lift graph respectively; F G 、F L Represents global combined features and local combined features; IRB represents inverted residual block; DCR represents dual stream complementary residual module; Conv 1×1 represents 1×1 convolution; upsampling represents bilinear upsampling; σ represents the sigmoid function; S2046: Calculate the training loss of the backbone network by the following formula to optimize the parameters of the backbone network; S GDCDR =1-SSIM(D i ,D g ); Where: S GDCDR Represents the backbone network training loss; SSIM stands for structural similarity; D i Denotes enhanced depth image; D g Represents the input depth image.
9. The efficient RGB-D saliency detection method using multiple information complementation as claimed in claim 1, characterized in that: Step S205 specifically includes the following steps: S2051: Use the RGB feature maps R1, R2 and R3 of the first to third scales, and the fusion feature maps F4 and F5 of the fourth and fifth scales as inputs of the decoder; S2052: input F5 into the compact pyramid refinement module and the convolutional attention mechanism module connected end to end in sequence, and output the feature fusion image of the fifth stage; the feature fusion image of the fifth stage is sequentially passed through a single channel 1×1 convolution, a sigmoid function and bilinear upsampling to obtain a saliency prediction image T5 of the fifth scale; the decoder training loss of the fifth stage is calculated through T5 and the corresponding truth map to optimize the decoder parameters; In the compact pyramid refinement module, the input feature map is first expanded four times along the channel dimension through 1×1 convolution; then it passes through three parallel 3×3 depthwise separable convolution branches; then the output features of each branch are element-wise added, and the added features are reduced in number of channels through the ReLu function layer, BN layer and 1×1 convolution, and finally the obtained output is residually connected with the input feature map as the output of the compact pyramid refinement module; S2053: After splicing the feature fusion image of the fifth stage and F4, the concatenation is input into the compact pyramid refinement module and the convolution attention mechanism module which are connected end to end in sequence, and the feature fusion image of the fourth stage is output; after the feature fusion image of the fourth stage is sequentially passed through a single channel 1×1 convolution, a sigmoid function and bilinear upsampling, a saliency prediction image T4 of the fourth scale is obtained; the decoder training loss of the fourth stage is calculated through T4 and the corresponding truth map to optimize the decoder parameters; S2054: After splicing the feature fusion image of the fourth stage and R3, input them into the compact pyramid refinement module, and output the feature fusion image of the third stage; after the feature fusion image of the third stage passes through the 1×1 convolution of a single channel, the sigmoid function and the bilinear upsampling in sequence, a saliency prediction image T3 of the third scale is obtained; the decoder training loss of the third stage is calculated by T3 and the corresponding truth map to optimize the decoder parameters; S2055: After splicing the feature fusion image of the third stage and R2, input them into the compact pyramid refinement module, and output the feature fusion image of the second stage; after the feature fusion image of the second stage passes through the 1×1 convolution, sigmoid function and bilinear upsampling of a single channel in sequence, a saliency prediction image T2 of the second scale is obtained; the decoder training loss of the second stage is calculated by T2 and the corresponding truth map to optimize the decoder parameters; S2056: After the feature fusion image of the second stage is spliced with R1, the spliced image is input into the compact pyramid refinement module, and the feature fusion image of the first stage is output; The feature fusion image of the first stage is sequentially passed through a single channel 1×1 convolution, sigmoid function and bilinear upsampling to obtain the saliency prediction image T1 of the first scale. The decoder training loss of the first stage is calculated by T1 and the corresponding truth map to optimize the decoder parameters. The saliency prediction image T1 of the first scale is used as the saliency prediction image finally output by the saliency detection model.
10. The efficient RGB-D saliency detection method using multiple information complementation as claimed in claim 9, characterized in that: In step S205, the decoder training loss is calculated by the following formula: L=L BCE +L Dice ; Where: L represents the decoder training loss; L BCE , L Dice Represent the binary cross entropy loss function and Dice loss function respectively; N represents the saliency prediction image T n and the number of pixels of the true value map G; g i Represents the label. If a pixel point is predicted correctly, g i If the prediction fails, g i is 0; represents the probability of correct prediction of the pixel at position i in the saliency prediction map of the nth stage; represents the dot product operation; |T n ∩G| represents the number of corresponding pixel values in the saliency prediction map of the nth stage and the true value map that are the same.
Citation Information
Cited By
Food nutrition assessment method based on adaptive fusion and feature enhancement
CN119495094A
Light field salient target detection method based on multi-modal feature fusion
CN119942289A
RGB-D-oriented fusion model and target detection method thereof
CN119963807A
Food nutrition evaluation method and system based on multi-modal fusion
CN120356204A
