An RGB-D saliency object detection device and method based on a depth-adaptive inverse thinning network architecture.

Abstract

This invention discloses an RGB-D saliency object detection device and method based on a depth-adaptive inverse thinning network architecture. It employs a dual-stream architecture (RGB stream and depth stream) to extract RGB-related features and depth features respectively, and uses a cross-fusion method to achieve multimodal feature fusion. It determines whether the fused features are high-level features; if so, it obtains intermediate saliency detection results; otherwise, it classifies them as low-level features and waits for the generated intermediate saliency results to be processed by an inverse thinning module for inverse thinning of low-level features. Finally, it generates saliency prediction results. This invention addresses the problem of how to achieve cross-level multimodal learning and fully explore the correlation and complementarity between RGB images and depth maps.

Description

Technical Field

[0001] This invention relates to the field of reverse thinning network architecture, and more specifically to an RGB-D saliency object detection device and method based on a depth-adaptive reverse thinning network architecture. Background Technology

[0002] With the continuous development of semiconductor technology, image sensors have been evolving towards greater human-likeness and refinement. To better reproduce and approximate the world perceived by the human eye, image sensors are also developing towards three-dimensionality. 3D sensors have expanded the application scope of image sensors and are currently widely used in scientific research, security, and industry to compensate for the lack of depth information in traditional 2D image sensing. The rapid development of 3D sensing technology makes it easier to acquire 3D images (RGB-D images) containing depth information, greatly promoting the development of the human visual system towards a more human-like direction.

[0003] Human vision relies heavily on depth information because it contains semantic category features of objects that attract human attention. Acquired 3D images require processing before recognition and judgment can be achieved (simulating the human brain's recognition and processing) to better simulate the human visual system's perception of the real world. Therefore, effectively identifying the regions with the most important semantic information in a scene is widely used to improve the perception capabilities of smart devices and ADAS in 3D scenes, overcoming the challenges faced by RGB image visual saliency detection algorithms. In conclusion, research on visually saliency object detection algorithms based on RGB-D images will have significant theoretical and practical value, and will be an indispensable research direction in the future field of computer vision.

[0004] Depth maps can provide rich spatial structure, object boundaries, and supplementary clues for saliency prediction. Researchers have already demonstrated the discriminative power of depth information in determining location and spatial structure. However, these methods primarily focus on utilizing rich depth geometric information, neglecting the interactive fusion design of multimodal features. In addition to exploring the fundamental research question of rich depth information, how to achieve cross-modal fusion research is another fundamental research question in the field of RGB-D saliency detection.

[0005] Research on RGB-D saliency detection fusion strategies has made some progress, and existing fusion strategies can be divided into: early fusion strategies, outcome fusion strategies, and feature fusion strategies. Early fusion strategies directly use RGB images and depth maps as inputs and generate features in the same way, clearly ignoring the distributional differences between the two modalities. Unlike early fusion, outcome fusion methods use two sub-components to process the saliency maps of the RGB image and depth map respectively, and then weight the two sub-saliency maps to obtain the final saliency prediction result. The first two fusion strategies belong to limited multimodal fusion, while feature fusion strategies are the key strategies for fusing intermediate information from multiple modalities to predict saliency. Many existing models combine visual color features and depth features in multiple layers and input them into the decoder to predict saliency. Although these models can explore the structural relationships between multimodalities, they still lack cross-layer feature learning exploration. Therefore, how to achieve cross-layer multimodal learning and fully explore the correlation and complementarity between RGB image and depth map multimodalities has become an urgent problem to be solved in the industry. Summary of the Invention

[0006] This invention provides an RGB-D saliency object detection device and method based on a depth-adaptive inverse thinning network architecture, which solves the problem of how to achieve cross-level multimodal learning and fully explore the correlation and complementarity between RGB images and depth map multimodals.

[0007] This invention is achieved through the following technical solution:

[0008] An RGB-D saliency object detection device based on a depth-adaptive inverse thinning network architecture, the detection device comprising a depth-adaptive weighting module, an inverse thinning stream module, and an integrated decoding module;

[0009] The depth adaptive weighting module is used to adaptively adjust the impact of depth map quality on saliency detection performance;

[0010] The reverse refinement stream module is used to capture semantic information of missing salient objects and to refine the edges and integrity of salient objects.

[0011] The integrated decoding module is used to aggregate saliency prediction results with cross-level multimodal features.

[0012] Furthermore, the detection method specifically includes,

[0013] Step 1: Using a dual-stream architecture, the input RGB image and depth map are distinguished, and RGB related features and depth features are extracted from the RGB image and depth map;

[0014] Step 2: The RGB-related features and depth features extracted in Step 1 are fused using a cross-fusion method to achieve multimodal feature fusion;

[0015] Step 3: Based on the multimodal feature fusion in Step 2, determine whether the fused feature is a high-level feature. If it is a high-level feature, proceed to Step 4; otherwise, classify it as a low-level feature and proceed to Step 5.

[0016] Step 4: High-level features are input into the ensemble decoder to obtain intermediate saliency detection results;

[0017] Step 5: Wait for the generated intermediate saliency results to be processed by the reverse refinement module to refine the low-level features.

[0018] Step 6: The intermediate saliency detection results from Step 4 are used by the reverse refinement module to refine the low-level features from Step 5. The obtained reverse refined features are then input into the integrated decoder to generate the saliency prediction results.

[0019] Furthermore, step 1 specifically involves the following steps: the depth map, after obtaining depth enhancement features through a depth adaptive weighting module, undergoes depth feature extraction via a depth stream; the RGB image is directly processed by an RGB stream for feature extraction.

[0020] Furthermore, step 6 specifically involves the following steps: after the intermediate saliency detection result from step 4 undergoes reverse refinement, the high-level features are refined again in a loop. When the refinement reaches the last feature of the high-level features, the low-level features undergo reverse refinement and are refined in a loop. When the refinement reaches the last feature of the low-level features, the refinement results of the low-level and high-level features are input into the integrated decoding module, and the integrated decoding module outputs the saliency prediction result.

[0021] Furthermore, the depth adaptive weighting module specifically includes the following processing steps:

[0022] Step 1.1: Calculate the dimension-reduced depth feature D by passing the input depth map through the bottleneck convolution unit formula (2);

[0023] Step 1.2: The channel dimensions of the reduced depth features are divided into G groups by grouped convolutional units. According to formula (3), the channel dimensions can be obtained. Then D g Divided into two branches, the channel dimension is represented as

[0024] Step 1.3: For D g The two branches perform channel attention and spatial attention calculations respectively, specifically calculated by equations (4) and (5) to further enhance D. g Its expressive power adapts to changes in the quality of depth maps;

[0025] Step 1.4: Integrate the information flow of each group through channel conversion operation.

[0026] Furthermore, the features of different levels of RGB flow and depth flow are first represented as f i rgb ,(i=1,2…,5) and f i depth , (i = 1, 2, ..., 5), where each layer of the depth flow will pass through a depth-weighted network, i.e., f i D (i = 1, 2, ..., 5), defined as:

[0027] f i D =mul(D(f) i depth ),f i depth );i=1,2,…,5 (1)

[0028] Where D is the importance calculation of adaptive depth information for each subgroup of features;

[0029] Depth map D int (w,h,c) is obtained through a 1×1 bottleneck convolution to achieve a computationally efficient lower dimension, defined as...

[0030] D = Conv(D int (2)

[0031] Among them, the bottleneck convolution Conv uses a 1×1 convolution kernel, D int The input is the depth map; then, it is divided into G groups along the channel dimensions of the reduced depth map, with the specific formula defined as:

[0032] D = D1, D2, ..., D g ;g=1,2,....c / G (3)

[0033] Each group The detailed semantic information was fully explored, and then D g Divided into two branches, the channel dimension is represented as After group regularization, its channel statistics are updated to... The detailed definition is as follows:

[0034] CA=δ(W1·GN(D g1 )+b1)·D g1 (4)

[0035] in, and It is Dg1 Scale and translation weighting coefficients; another branch is spatial statistical features, which further enhance D. g2 The expressive power of the depth map and its ability to adapt to changes in depth map quality are defined as follows:

[0036] SA=δ(W2·GN(D g2 )+b2)·D g2 (5)

[0037] in, and The feature weights of the two branches are concatenated as follows: Finally, the information flow of each group is integrated through the channel shuffle mechanism.

[0038] Furthermore, the reverse refinement stream specifically includes the following processing steps:

[0039] Step 5.1: The input intermediate saliency map is upsampled and then inversely processed;

[0040] Step 5.2: The result of the inverse operation is multiplied element-wise with the fusion feature of the next layer to achieve reverse refinement of the features of the next layer, retaining more salient clues from the previous layer;

[0041] Step 5.3: After obtaining the reverse operation features, perform convolution operations to further refine the features;

[0042] Step 5.4: The reverse-refined features are concatenated with the input features to form the reverse-refined features for the next layer and the intermediate saliency map output by this layer of the network.

[0043] Furthermore, the detailed decoding process of the integrated decoding block in step 6 is as follows:

[0044] Step 6.1: The input consists of multiple cross-layer multimodal features. First, the features are calculated by three parallel global context prediction modules, then the intermediate saliency map of the high-level feature input is imported, and the global context information of the cross-layer multimodal features is obtained.

[0045] Step 6.2: Introduce residual blocks to avoid the loss of original information and concatenate them with the obtained global prediction information;

[0046] Step 6.3: Obtain the final significance prediction results through a lightweight aggregation method.

[0047] Furthermore, the dual-stream architecture employs a ResNet-50 pre-trained model to explore the features of the RGB image and depth map, obtaining a high-level intermediate saliency prediction result S. intAs input to the reverse refinement network, it guides the propagation and learning of multimodal features across layers, refining low-level features through high-level semantic structural information, thereby establishing the relationship between location and boundary spatial cues.

[0048] The beneficial effects of this invention are:

[0049] The network structure of this invention can not only efficiently guide the propagation and learning of multimodal features across levels, but also fully focus on salient objects containing deeper semantic information.

[0050] This invention designs a depth adaptive weighting module for each depth extraction layer of the depth branch. Specifically, it explores the semantic and spatial information of each group by dividing the channel dimension of the depth features, thereby further improving the expressive power of depth information and adapting to changes in the quality of the depth map.

[0051] This invention fully mines the deep semantic information in cross-level multimodal features, efficiently guides the propagation of cross-level multimodal features, and improves the performance of RGB-D saliency object detection. Attached Figure Description

[0052] Figure 1 This is the quantitative result of the present invention on four evaluation metrics across five test datasets.

[0053] Figure 2 This is a visual comparison example of the present invention with 11 state-of-the-art methods on the ReDWeb-S dataset.

[0054] Figure 3 This is an example image of a magnified visualization of a complex scene in the ReDWeb-S dataset of this invention.

[0055] Figure 4 This is a structural schematic diagram of a specific embodiment of the present invention.

[0056] Figure 5 This is a schematic diagram of the depth adaptive weighted module structure of the present invention.

[0057] Figure 6 This is a schematic diagram of the reverse refinement structure of the present invention.

[0058] Figure 7 This is a schematic diagram of the integrated decoding module structure of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] An RGB-D saliency object detection device based on a depth-adaptive inverse thinning network architecture, the detection device comprising a depth-adaptive weighting module, an inverse thinning stream module, and an integrated decoding module;

[0061] The depth adaptive weighting module is used to adaptively adjust the impact of depth map quality on saliency detection performance;

[0062] The reverse refinement stream module is used to capture semantic information of missing salient objects and to refine the edges and integrity of salient objects.

[0063] The integrated decoding module is used to aggregate saliency prediction results with cross-level multimodal features.

[0064] Furthermore, the detection method specifically includes,

[0065] Step 1: Employ a dual-stream architecture, namely RGB stream and depth stream, to extract features from the RGB image and depth map respectively;

[0066] Step 2: The RGB-related features and depth features extracted in Step 1 are fused using a cross-fusion method to achieve multimodal feature fusion;

[0067] Step 3: Based on the multimodal feature fusion in Step 2, determine whether the fused feature is a high-level feature. If it is a high-level feature, proceed to Step 4; otherwise, classify it as a low-level feature and proceed to Step 5.

[0068] Step 4: High-level features are input into the ensemble decoder to obtain intermediate saliency detection results;

[0069] Step 5: Wait for the generated intermediate saliency results to be processed by the reverse refinement module to refine the low-level features.

[0070] Step 6: The intermediate saliency detection results from Step 4 are used by the reverse refinement module to refine the low-level features from Step 5. The obtained reverse refined features are then input into the integrated decoder to generate the saliency prediction results.

[0071] Furthermore, step 1 specifically involves the following steps: the depth map, after obtaining depth enhancement features through a depth adaptive weighting module, undergoes depth feature extraction via a depth stream; the RGB image is directly processed by an RGB stream for feature extraction.

[0072] Furthermore, step 6 specifically involves the following steps: after the intermediate saliency detection result from step 4 undergoes reverse refinement, the high-level features are refined again in a loop. When the refinement reaches the last feature of the high-level features, the low-level features undergo reverse refinement and are refined in a loop. When the refinement reaches the last feature of the low-level features, the refinement results of the low-level and high-level features are input into the integrated decoding module, and the integrated decoding module outputs the saliency prediction result.

[0073] Furthermore, although depth maps provide more effective and complementary cues than RGB images, low-quality depth maps still negatively impact saliency prediction results. To adaptively adjust for the impact of depth map quality on saliency detection performance, a group normalization unit is used to simultaneously capture pixel-level spatial and channel-level pairing relationships. The specific workflow is as follows: Figure 5 As shown, a group regularization block is embedded before the attention output of each channel for residual learning.

[0074] The depth-adaptive weighting module specifically includes the following processing steps:

[0075] Step 1.1: Calculate the dimension-reduced depth feature D by passing the input depth map through the bottleneck convolution unit formula (2);

[0076] Step 1.2: The channel dimensions of the reduced depth features are divided into G groups by grouped convolutional units. According to formula (3), the channel dimensions can be obtained. Then D g Divided into two branches, the channel dimension is represented as

[0077] Step 1.3: For D g The two branches perform channel attention and spatial attention calculations respectively, specifically calculated by equations (4) and (5) to further enhance D. g Its expressive power adapts to changes in the quality of depth maps;

[0078] Step 1.4: Integrate the information flow of each group through channel conversion operation.

[0079] Furthermore, the features of different levels of RGB flow and depth flow are first represented as f i rgb ,(i=1,2…,5) and f i depth , (i = 1, 2, ..., 5), where each layer of the depth flow will pass through a depth-weighted network, i.e., f i D (i = 1, 2, ..., 5), defined as:

[0080] f iD =mul(D(f) i depth ),f i depth );i=1,2,…,5 (1)

[0081] Where D is the importance calculation of adaptive depth information for each subgroup of features;

[0082] Depth map D int (w,h,c) is obtained through a 1×1 bottleneck convolution to achieve a computationally efficient lower dimension, defined as...

[0083] D = Conv(D int (2)

[0084] Among them, the bottleneck convolution Conv uses a 1×1 convolution kernel, D int The input is the depth map; then, it is divided into G groups along the channel dimensions of the reduced depth map, with the specific formula defined as:

[0085] D = D1, D2, ..., D g ;g=1,2,....c / G (3)

[0086] Each group The detailed semantic information was fully explored, and then D g Divided into two branches, the channel dimension is represented as After group regularization, its channel statistics are updated to... The detailed definition is as follows:

[0087] CA=δ(W1·GN(D g1 )+b1)·D g1 (4)

[0088] in, and It is D g1 Scale and translation weighting coefficients; another branch is spatial statistical features, which further enhance D. g2 The expressive power of the depth map and its ability to adapt to changes in depth map quality are defined as follows:

[0089] SA=δ(W2·GN(D g2 )+b2)·D g2 (5)

[0090] in, and The feature weights of the two branches are concatenated as follows: Finally, the information flow of each group is integrated through the channel shuffle mechanism.

[0091] Furthermore, the reverse refinement stream specifically includes the following processing steps:

[0092] Step 5.1: The input intermediate saliency map is upsampled and then inversely processed;

[0093] Step 5.2: The result of the inverse operation is multiplied element-wise with the fusion feature of the next layer to achieve reverse refinement of the features of the next layer, retaining more salient clues from the previous layer;

[0094] Step 5.3: After obtaining the reverse operation features, perform convolution operations to further refine the features;

[0095] Step 5.4: The reverse-refined features are concatenated with the input features to form the reverse-refined features for the next layer and the intermediate saliency map output by this layer of the network.

[0096] Furthermore, the detailed decoding process of the integrated decoding block in step 6 is as follows:

[0097] Step 6.1: The input consists of multiple cross-layer multimodal features. First, the features are calculated by three parallel global context prediction modules, then the intermediate saliency map of the high-level feature input is imported, and the global context information of the cross-layer multimodal features is obtained.

[0098] Step 6.2: Introduce residual blocks to avoid the loss of original information and concatenate them with the obtained global prediction information;

[0099] Step 6.3: Obtain the final significance prediction results through a lightweight aggregation method.

[0100] Furthermore, the dual-stream architecture employs a ResNet-50 pre-trained model to explore the features of the RGB image and depth map, obtaining a high-level intermediate saliency prediction result S. int As input to the reverse refinement network, it guides the propagation and learning of multimodal features across layers, refining low-level features through high-level semantic structural information, thereby establishing the relationship between location and boundary spatial cues.

[0101] Step 5, reversing the network to implement the inference process, includes the following steps:

[0102] First, the features of different levels of RGB flow and depth flow are represented as f. i rgb ,(i=1,2…,5) and f i depth , (i = 1, 2, ..., 5), where each layer of the depth flow will pass through a depth-weighted network, i.e., f i D (i = 1, 2, ..., 5), defined as:

[0103] fi D =mul(D(f) i depth ),f i depth );i=1,2,…,5 (1)

[0104] Where D is the importance calculation of adaptive depth information for each subgroup of features.

[0105] Depth map D int (w,h,c) is obtained through a 1×1 bottleneck convolution to achieve a computationally efficient lower dimension, defined as...

[0106] D = Conv(D int (2)

[0107] Among them, the bottleneck convolution Conv uses a 1×1 convolution kernel, D int The input is the depth map. Then, it is divided into G groups along the channel dimensions of the reduced depth map, with the specific formula defined as:

[0108] D = D1, D2, ..., D g ;g=1,2,....c / G (3)

[0109] Each group The detailed semantic information was fully explored, and then D g Divided into two branches, the channel dimension is represented as After group regularization, its channel statistics are updated to... The detailed definition is as follows:

[0110] CA=δ(W1·GN(D g1 )+b1)·D g1 (4)

[0111] in, and It is D g1 Scale and translation weighting coefficients. Another branch is spatial statistical features, which further enhance D. g2 The expressive power of the depth map and its ability to adapt to changes in depth map quality are defined as follows:

[0112] SA=δ(W2·GN(D g2 )+b2)·D g2 (5)

[0113] in and The feature weights of the two branches are concatenated as follows: Finally, the information flow of each group is integrated through the channel shuffle mechanism.

[0114] The fusion operation between the RGB stream and the depth stream is denoted as f. i sum (i = 1, 2, ..., 5), as follows:

[0115] f i sum =∑(f i D ,f i rgb );i=1,2,…,5 (6)

[0116] After cross-modal feature fusion, the features from layers 3-5 are input into the ensemble decoder to obtain the intermediate saliency prediction result S. int Defined as:

[0117] S int =Agg(Dec(f3) sum f4 sum f5 sum (7)

[0118] Here, Agg and Dec are decoding integration operations, specifically including a dense combination of four parallel initial blocks and residual blocks, and the specific calculation process is as follows:

[0119] Dec=Conv3(Con(d1,d2,d3,d4))+Conv1(f i sum (8)

[0120] Where d1, d2, d3, d4 represent the features of each parallel branch, i = 1, 2, ..., 5, and Conv1 and Conv3 represent 1*1 and 3*3 convolution operations, respectively.

[0121] Agg=ConvB(Con(M1,Con(M2,Conv3(U p (Dec5))))) (9)

[0122] Where M1 and M2 represent the product of decoder features, Con is the concatenation operation, Conv3 is the 3x3 convolution operation, and ConvB represents the operation of convolutional blocks composed of different convolutional kernels.

[0123] The high-level significance prediction result S int As input to the inverse refinement network to guide cross-level multimodal learning, the inverse refinement yields three intermediate saliency prediction results, which can be defined as:

[0124]

[0125]

[0126] in This represents the reverse refinement operation, and the specific calculation is shown in the following formula:

[0127]

[0128] Among them, I i The anti-attention weights are defined as shown in Equation 8:

[0129]

[0130]

[0131] in, This represents the anti-attention refinement subtraction operation, where Σ is the Sigmoid function, and U p It is an upsampling operation.

[0132] Low-level features are refined through high-level semantic structure information, effectively suppressing cross-modal information features at the low level. Low-level features are further refined through a reverse refinement process, and the low-level feature refinement formula is defined as:

[0133] f i ref =f i ref +(U p (S3)⊙f i ref ); i = 1, 2 (13)

[0134] Where ⊙ represents element-wise multiplication, S3 is the intermediate significance prediction value of level 3, and U p This is for upsampling operations.

[0135] The final saliency detection result is obtained by inputting the refined features of the lower layers and the saliency detection of the third layer into the ensemble decoder, and is defined as follows:

[0136] S out =Agg(Dec(f1) ref f2 ref ,S3)) (14)

[0137] Joint optimization is used to refine the output through back attention, further improving the saliency results. Specifically, weighted IoU loss and binary cross-entropy (BCE) loss are applied to deeply supervise the intermediate saliency results and the final saliency output results, with adaptive weights. The total loss is defined as follows:

[0138]

[0139] Where λ∈[0,1] is a trade-off parameter balancing cross-level details, empirically set to 0.4, G is the significance truth value, and S i (i = 1, 2, ..., 5) Specific The calculation is as follows:

[0140]

[0141] Where S represents the significance prediction result, s n It is an element in S.

[0142] Performance testing was conducted on five test datasets: NJU2K, NLPR, SIP, ReDWeb-S, and COME15K (i.e., COME-E: 4600 simpler test samples; COME-H: 3000 more challenging test samples). To ensure fairness in comparison with existing RGB-D SOD models, both the comprehensive evaluation method and the proposed method were trained on training datasets consisting of NJU2K and NLPR, and all competing methods were generated from published source code with default parameters or obtained from publicly available results. The 12 RGB-D saliency object detection models compared include DF, PCF, TANet, DMRA, PGAR, D3Net, BBSNet, ATSA, DANet, CDNet, DSA2F, and DCF.

[0143] E ξ S α F β The comparison results with the four evaluation indicators of MAE are as follows: Figure 1 As shown.

[0144] like Figure 1 As can be clearly seen, the proposed model DAIR significantly outperforms the comparison models on all performance metrics across the five datasets, especially in E. ξ The MAE values ​​demonstrate that DAIR possesses excellent region structure similarity and accuracy. For example, compared to the DSA2F model on the SIP dataset, DAIR achieves higher E... ξ The values ​​of DAIR and MAE were improved by 2.2% and 1.0%, respectively. Although the performance of DAIR is close to that of the second-ranked model (CDNet), the overall performance of DAIR is still better, especially in terms of mean absolute error (MAE). This further illustrates that the depth adaptive weighted network structure can flexibly cope with changes in depth map quality and has higher accuracy in significant detection.

[0145] In addition to the quantitative analysis experiments mentioned above, a visual saliency detection visualization comparison experiment was conducted on some representative test samples selected on the ReDWeb-S test dataset to more intuitively demonstrate the performance comparison between the DAIR model and the state-of-the-art (SOTA) model. Figure 2 The image shows the visual saliency comparison results of the DAIR model and 11 state-of-the-art (SOTA) models on the ReDWeb-S test dataset. These include several highly challenging complex scenes: i) simple scenes (single saliency, strong foreground and background contrast, and salient objects typically located in the image center, contrasting with complex scenes); ii) small salient objects; iii) multiple salient objects; iv) background diversity; v) low contrast; and vi) complex shapes. Furthermore, to further evaluate the boundary detection capabilities of different RGB-D saliency detection models, some sample examples have been enlarged, such as... Figure 3 As shown.

[0146] Depend on Figure 2 As can be seen, compared to the proposed DAIR model, most state-of-the-art (SOTA) models fail to correctly segment salient objects and exhibit the highest number of detection errors in complex scenes. However, the DAIR model accurately predicts the most salient objects and coherent details in the most challenging scenes, such as small salient objects (row 3), background diversity (rows 6-7), and low contrast (rows 8-9). Furthermore, it is clear that the model improves upon zooming in at boundaries. Figure 3 In this study, the proposed model outperforms other models in boundary preservation capabilities across challenging scenarios with multiple salient objects, low contrast, and complex shapes. These results further confirm the model's strong ability to detect salient objects and sharp boundaries. This benefits from the DAIR model's robustness to challenging variations in depth map quality and its powerful ability to mine deep semantic cues across multiple modalities. In summary, these salient results validate the effectiveness and generalization ability of the proposed model on various challenging datasets.

[0147] The proposed model is developed based on the popular PyTorch library and optimized using the RAdam optimizer. The model loads a publicly available pre-trained ResNet-50 model as its backbone. It is trained on a TITAN RTX GPU using 1785 pairs of RGB-D images from the NJU2K dataset and 700 pairs of RGB-D images from the NLPR dataset as training samples. Hyperparameters that need to be adjusted during model training include the initial learning rate, learning latency per epoch, batch size, number of epochs, and number of groups. The depth maps used for training and testing are normalized using the min-max normalization method. Performance testing is conducted on five test datasets: in addition to the remaining test data from NJU2K and NLPR, the newly released and challenging representative test datasets SIP, ReDWeb-S, and COME15K (i.e., COME-E: 4600 simpler test samples; COME-H: 3000 more challenging test samples) for both quantitative and qualitative experimental analysis. To ensure fairness in comparison with existing RGB-D SOD models, both the comprehensive evaluation and comparison methods and the proposed methods were trained on the NJU2K and NLPR training datasets, and all competing methods were generated from published source code with default parameters or obtained from publicly available results.

Claims

1. A detection method for an RGB-D saliency object detection device based on a depth-adaptive inverse thinning network architecture, characterized in that, The detection device includes a depth adaptive weighting module, a reverse thinning stream module, and an integrated decoding module; The depth adaptive weighting module is used to adaptively adjust the impact of depth map quality on saliency detection performance; The reverse refinement stream module is used to capture semantic information of missing salient objects and to refine the edges and integrity of salient objects. The integrated decoding module is used to aggregate saliency prediction results with cross-level multimodal features; The detection method is specifically as follows: Step 1: Using a dual-stream architecture, the input RGB image and depth map are distinguished, and RGB related features and depth features are extracted from the RGB image and depth map; Step 2: The RGB-related features and depth features extracted in Step 1 are fused using a cross-fusion method to achieve multimodal feature fusion; Step 3: Based on the multimodal feature fusion in Step 2, determine whether the fused feature is a high-level feature. If it is a high-level feature, proceed to Step 4; otherwise, classify it as a low-level feature and proceed to Step 5. Step 4: High-level features are input into the ensemble decoder to obtain intermediate saliency detection results; Step 5: Wait for the generated intermediate saliency results to be processed by the reverse refinement module to refine the low-level features. Step 6: The intermediate saliency detection results from Step 4 are used by the reverse refinement module to refine the low-level features from Step 5. The obtained reverse refined features are then input into the ensemble decoder to generate the saliency prediction results. The depth-adaptive weighting module specifically includes the following processing steps: Step 1.1: Calculate the dimension-reduced depth feature D by passing the input depth map through the bottleneck convolution unit formula (2); Step 1.2: The channel dimensions of the reduced depth features are divided into G groups by grouped convolutional units. According to formula (3), the channel dimensions can be obtained. Then D g Divided into two branches, the channel dimension is represented as Step 1.3: For D g The two branches perform channel attention and spatial attention calculations respectively, specifically calculated by equations (4) and (5) to further enhance D. g Its expressive power adapts to changes in the quality of depth maps; Step 1.4: Integrate the information flow of each group through channel conversion operation.

2. The detection method of the RGB-D saliency object detection device based on a depth-adaptive reverse thinning network architecture according to claim 1, characterized in that, Specifically, step 1 involves the following steps: the depth map is processed by a depth adaptive weighting module to obtain depth enhancement features, and then depth features are extracted using a depth stream; the RGB image is directly processed by an RGB stream to extract features.

3. The detection method of the RGB-D saliency object detection device based on a depth-adaptive reverse thinning network architecture according to claim 1, characterized in that, Specifically, step 6 involves reversing the refinement flow of the intermediate saliency detection results from step 4, then refining the high-level features in a loop. After refining to the last feature of the high-level features, reversing the refinement flow of the low-level features and refining the low-level features in a loop are performed. After refining to the last feature of the low-level features, the refinement results of the low-level and high-level features are input into the integrated decoding module, and the integrated decoding module outputs the saliency prediction results.

4. The detection method of the RGB-D saliency object detection device based on a depth-adaptive inverse thinning network architecture according to claim 1, characterized in that, First, the features of different levels of RGB flow and depth flow are represented as f. i rgb ,(i=1,2…,5) and f i depth , (i = 1, 2, ..., 5), where each layer of the depth flow will pass through a depth-weighted network, i.e., f i D (i = 1, 2, ..., 5), defined as: f i D =mul(D(f i depth ),f i depth );i=1,2,…,5 (1) Where D is the importance calculation of adaptive depth information for each subgroup of features; Depth map D int (w,h,c) is obtained through a 1×1 bottleneck convolution to achieve a computationally efficient lower dimension, defined as... D=Conv(D int ) (2) Among them, the bottleneck convolution Conv uses a 1×1 convolution kernel, D int The input is the depth map; then, it is divided into G groups along the channel dimensions of the reduced depth map, with the specific formula defined as: D=D1,D2,...,D g ;g=1,2,....c / G (3) Among them, each group The detailed semantic information was fully explored, and then D g Divided into two branches, the channel dimension is represented as After group regularization, its channel statistics are updated to... The detailed definition is as follows: CA=δ(W1·GN(D g1 )+b1)·D g1 (4) in, and It is D g1 Scale and translation weighting coefficients; another branch is spatial statistical features, which further enhance D. g2 The expressive power of the depth map and its ability to adapt to changes in depth map quality are defined as follows: SA=δ(W2·GN(D g2 )+b2)·D g2 (5) in, and The feature weights of the two branches are concatenated as follows: Finally, the information flow of each group is integrated through the channel shuffle mechanism.

5. The detection method of the RGB-D saliency object detection device based on a depth-adaptive inverse thinning network architecture according to claim 1, characterized in that, The reverse refinement stream specifically includes the following processing steps: Step 5.1: The input intermediate saliency map is upsampled and then inversely processed; Step 5.2: The result of the inverse operation is multiplied element-wise with the fusion feature of the next layer to achieve reverse refinement of the features of the next layer, retaining more salient clues from the previous layer; Step 5.3: After obtaining the reverse operation features, perform convolution operations to further refine the features; Step 5.4: The reverse-refined features are concatenated with the input features to form the reverse-refined features for the next layer and the intermediate saliency map output by this layer of the network.

6. The detection method of the RGB-D saliency object detection device based on a depth-adaptive inverse thinning network architecture according to claim 1, characterized in that, The detailed decoding process of the integrated decoding block in step 6 is as follows: Step 6.1: The input consists of multiple cross-layer multimodal features, which are calculated by three parallel global context prediction modules. The intermediate saliency map of the high-level feature input is imported, and the global context information of cross-layer multimodal features is obtained. Step 6.2: Introduce residual blocks to avoid the loss of original information and concatenate them with the obtained global prediction information; Step 6.3: Obtain the final significance prediction results through a lightweight aggregation method.

7. The detection method of the RGB-D saliency object detection device based on a depth-adaptive reverse thinning network architecture according to claim 1, characterized in that, The dual-stream architecture uses a ResNet-50 pre-trained model to explore features of RGB images and depth maps, obtaining a high-level intermediate saliency prediction result S. int As input to the reverse refinement network, it guides the propagation and learning of multimodal features across layers, refining low-level features through high-level semantic structural information, thereby establishing the relationship between location and boundary spatial cues.

Images