Semantic Feature-Based and Bio-Inspired RGB-D Salient Object Detection Methods

Abstract

This invention relates to the field of computer vision technology, specifically to an RGB-D salient object detection method based on semantic features and bio-inspired principles. The method extracts features from RGB-D image samples using an encoder to obtain low-level, mid-level, and high-level semantic features of the RGB and depth images. A multi-stage fusion module fuses the low-level semantic features of the RGB and depth images to obtain low-level fused features; it then fuses the mid-level semantic features to obtain mid-level fused features; finally, it fuses the high-level semantic features of the RGB and depth images to obtain high-level fused features. A cortical decoder decodes the low-level, mid-level, and high-level fused features to obtain a saliency prediction map. This method improves the efficiency, robustness, and generalization ability of RGB-D salient object detection models.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to an RGB-D salient object detection method based on semantic features and bio-inspired principles. Background Technology

[0002] RGB-D salient object detection is a key research direction in the field of computer vision. Its core task is to achieve accurate localization and segmentation of the most salient objects in an image by fusing the color and texture information of an RGB image with the three-dimensional geometric features of a depth image. This technology has demonstrated significant application value in multiple fields such as image understanding, object recognition, intelligent robot navigation, medical image analysis, and video surveillance.

[0003] In existing technologies, Transformer-based models have attracted significant attention due to their superior global context modeling capabilities and long-range dependency capture abilities. These methods significantly improve the ability to identify salient regions by integrating hierarchical feature extraction mechanisms with self-attention mechanisms. For example, some methods utilize pure Transformer structures for salient detection, enhancing the identification of salient regions; others strengthen local structure modeling through token transformations; and still others combine encoder-decoder architectures and multi-head attention mechanisms to achieve more accurate salient predictions.

[0004] However, current Transformer-based methods still face several unresolved issues. First, the quadratic computational complexity of the Transformer limits its efficiency in practical applications. Many methods attempt to improve efficiency by pruning redundant tokens, but these methods often lead to a significant decrease in accuracy and face difficulties when applied to local vision Transformers, thus limiting the model's detection accuracy and generalization ability. Second, cross-modal feature fusion methods typically rely on simple concatenation or weighted fusion, failing to fully consider the independence of RGB and depth modalities, resulting in feature redundancy and an inability to fully utilize the complementary information between the two. Finally, multi-scale feature processing capabilities are insufficient. Existing methods mostly employ basic feature aggregation techniques, which struggle to effectively capture features at different scales, limiting the model's adaptability to complex scenes. Summary of the Invention

[0005] The purpose of this invention is to provide an RGB-D salient target detection method based on semantic features and bio-inspired principles, in order to solve at least one technical problem existing in the prior art.

[0006] The first aspect of this invention provides a method for detecting RGB-D salient targets based on semantic features and bio-inspired methods, comprising the following steps:

[0007] The RGB-D image to be detected is input into the salient object detection model, which outputs the salient object detection result. The training method of the salient object detection model includes the following steps:

[0008] Construct a neural network that includes an encoder, a multi-stage fusion module, and a cortical decoder;

[0009] By extracting features from RGB-D image samples using an encoder, low-level, mid-level, and high-level semantic features of RGB and depth images are obtained.

[0010] The multi-stage fusion module fuses the low-level semantic features of RGB images and depth images to obtain low-level fusion features; it fuses the mid-level semantic features of RGB images and depth images to obtain mid-level fusion features; and it fuses the high-level semantic features of RGB images and depth images to obtain high-level fusion features.

[0011] The saliency prediction map is obtained by decoding the low-level, mid-level and high-level fused features through the cortical decoder.

[0012] The parameters of the neural network are updated based on the loss function to obtain a salient object detection model.

[0013] In this context, the RGB-D image to be detected refers to the RGB-D image for which salient object detection needs to be performed, and the RGB-D image sample refers to the RGB-D image used to train the salient object detection model. RGB-D images contain RGB information and depth information; the RGB information constitutes the RGB image, and the depth information constitutes the depth image. The salient object detection results are typically represented by a salientity prediction map.

[0014] Furthermore, the encoder includes a DSSE module. In the DSSE module, the input features are segmented into multiple patches, and adjacent patches are fused to obtain local cluster tokens. Multi-head attention interaction is performed on the local cluster tokens and the original image tokens to generate preliminary semantic tokens. Spatial pooling is performed on the preliminary semantic tokens to capture global semantic information and generate a global center token. The global center token and the preliminary semantic tokens are fused through a heterogeneous attention mechanism to generate the final semantic token.

[0015] Furthermore, the above-mentioned fusion of low-level semantic features of RGB images and depth images includes:

[0016] The low-level semantic features of the RGB image and the depth image are multiplied and concatenated element-wise. The two results are then concatenated again through a 1×1 convolution to obtain the feature map. ;

[0017] Feature map The results are sequentially combined using a combination of 3×3 convolutions and three 3×3 dilated convolutions, then concatenated and processed by a 1×1 convolution to obtain low-level fusion features.

[0018] Furthermore, the above-mentioned fusion of intermediate semantic features from RGB and depth images includes:

[0019] The intermediate semantic features of the RGB and depth images are divided into two branches. One branch performs element-wise multiplication, and then a selective region attention mechanism is used to enhance salient features to obtain the feature map. and feature map The other branch first performs a concatenation operation, then a 1×1 convolution, and finally merges it with the feature map. and feature map Fusion yields intermediate fusion characteristics.

[0020] Furthermore, the above-mentioned fusion of high-level semantic features of RGB images and depth images includes:

[0021] The high-level semantic features of the RGB and depth images are first multiplied element-wise, and then concatenated with the concatenation result of the high-level semantic features of the RGB and depth images. After dimensionality reduction optimization and channel fusion, spatial features are enhanced using a spatial attention mechanism to obtain high-level fused features.

[0022] Furthermore, the decoding of low-level, mid-level, and high-level fusion features described above includes:

[0023] Dilated convolutions are applied to the intermediate and high-level fusion features, which are then merged through element-wise multiplication. The merged features are then fed into three 3×3 convolutional layers for parallel processing. Finally, the three parallel-processed features are concatenated into a single feature. ;

[0024] Features Channel fusion is performed to obtain features. Then, dimensionality optimization is performed, and the optimized features are concatenated with the low-level fusion features. The concatenated result is then subjected to channel fusion and dimensionality optimization again. The processed result is compared with... The data is then stitched together to obtain a significance prediction map.

[0025] Furthermore, the loss function described above is:

[0026]

[0027] in, This represents the total loss of the salient object detection model; , , , These represent the weighting coefficients of the four loss terms; , , , These respectively represent the features fused into a single feature in the cortical decoder. The losses incurred in the stages of channel fusion, dimension optimization, and saliency prediction map generation.

[0028] A second aspect of the present invention provides an electronic device comprising:

[0029] At least one processor;

[0030] and memory that is communicatively connected to at least one processor;

[0031] The memory stores instructions that, when executed by at least one processor, implement the aforementioned RGB-D salient target detection method based on semantic features and bio-inspired principles.

[0032] A third aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the above-described RGB-D salient target detection method based on semantic features and bio-inspired principles.

[0033] The technical solutions of the embodiments of the present invention have the following beneficial effects:

[0034] (1) By using a two-stage sparse semantic enhancement mechanism, feature redundancy is effectively reduced, feature extraction efficiency and model generalization ability are improved, and computational cost is reduced.

[0035] (2) The multi-stage fusion module achieves efficient fusion of RGB and depth features, integrates information from different modalities at different scales, and optimizes the performance of multimodal salient object detection;

[0036] (3) The cortical decoder can simulate the brain’s hierarchical visual processing mechanism, achieve efficient hierarchical feature refinement and computational optimization, and improve the robustness and generalization ability of salient object detection. Attached Figure Description

[0037] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of the structure of the two-stage sparse semantic enhancement module in an embodiment of the present invention.

[0039] Figure 3 This is a schematic diagram of the structure of the multi-stage fusion module in an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0041] According to an embodiment of the present invention, a salient object detection model is first trained, and then the trained salient object detection model is used to perform RGB-D salient object detection. For ease of representation, the salient object detection model of the present invention is labeled as the OSFBiF-Net model.

[0042] like Figure 1 As shown, the OSFBiF-Net model includes an encoder, a multi-stage fusion module, and a cortical decoder. The encoder is based on the Swin Transformer encoder and incorporates a two-stage sparse semantic enhancement (DSSE) module. Specifically, the encoder consists of four coding stages, with the third stage being the DSSE module.

[0043] In the encoder, feature extraction is performed on RGB-D image samples to obtain high-level semantic features of the RGB images and high-level semantic features of the depth images. Specifically, the encoder performs feature extraction in multiple stages on the RGB and depth images of the RGB-D image samples to obtain semantic feature representations at different levels. For example, performing four stages of feature extraction can yield the semantic features of the RGB images. , , and and semantic features of depth images , , and ;in, This can be referred to as the low-level semantic features of RGB images. , This can be referred to as the intermediate semantic feature of an RGB image. This can be referred to as the high-level semantic features of RGB images; This can be referred to as the low-level semantic features of depth images. , This can be referred to as the intermediate semantic feature of a depth image. This can be referred to as the high-level semantic features of depth images.

[0044] For RGB images, the image is segmented into multiple patches, and each patch is converted into a token, forming the original image token (IT). The IT is processed by the encoding module in the first stage of the encoder to generate semantic features. . Semantic features are generated through the second-stage encoding module of the encoder. In the third stage of the encoder, semantic enhancement of the features is performed through the DSSE module. Within the DSSE module, the features are... The data is divided into multiple patches, and adjacent patches are merged to obtain Local Clustering Tokens (LCTs). LCTs and Information Interaction Tokens (ITs) interact with each other through multi-head attention to generate Preliminary Semantic Tokens (ISTs). ISTs undergo spatial pooling to capture global semantic information, generating Global Central Tokens (GCTs). GCTs and ISTs are then fused through a heterogeneous attention mechanism to generate semantic features. The fourth stage of the encoder's encoding module uses a self-attention mechanism to... By combining with IT, spatial details of images can be restored and high-level semantic features can be generated. .

[0045] For depth images, the image is segmented into multiple patches, and each patch is converted into a token, forming the original image token (IT). The IT is then processed by the encoding module in the first stage of the encoder to generate semantic features. . Semantic features are generated through the second stage of encoder processing. In the third stage of the encoder, semantic enhancement of the features is performed through a two-stage sparse semantic enhancement (DSSE) module. In the DSSE module, the features are... The data is divided into multiple patches, and adjacent patches are merged to obtain Local Clustering Tokens (LCTs). LCTs and Information Interaction Tokens (ITs) interact with each other through multi-head attention to generate Preliminary Semantic Tokens (ISTs). ISTs undergo spatial pooling to capture global semantic information, generating Global Central Tokens (GCTs). GCTs and ISTs are then fused through a heterogeneous attention mechanism to generate semantic features. The fourth stage of the encoder's encoding module uses a self-attention mechanism to... By combining with IT, spatial details of images can be restored and high-level semantic features can be generated. .

[0046] The specific structure of the DSSE module used in this embodiment of the invention is as follows: Figure 2As shown, "IT" represents the tokenized representation of the original input image (RGB image token or depth image token), "LCT" represents the local clustering token, "IST" represents the initial semantic token, "GCT" represents the global center token, and "FST" represents the final semantic token (i.e., semantic features). or semantic features ).

[0047] In the DSSE module, the second-layer feature map of the input RGB image or depth image is segmented into multiple patches. Adjacent patches are fused to obtain Local Clustering Tokens (LCTs). Then, using a multi-head attention mechanism, the LCTs interact with the original image token sequence IT to generate preliminary semantic tokens (ISTs), as shown in the following formula:

[0048]

[0049] in, Indicates a feedforward network. This represents the initial semantic token after processing by the feedforward network. This represents the initial semantic token.

[0050] The initial semantic token (IST) is then spatially pooled to generate a global central token (GCT) to capture global semantic information. Using a heterogeneous attention mechanism, the GCT and IST are fused to generate the final semantic token (FST), as shown in the following equation:

[0051]

[0052] ;

[0053] in, Indicates the initial semantic token. Indicates the global central token. This indicates a multi-head attention mechanism. This represents a key used in multi-head attention mechanisms. This represents the value used in multi-head attention mechanisms. This indicates the features after processing by the multi-head attention mechanism. Indicates a feedforward network. This represents the final semantic token.

[0054] Next, by utilizing the self-attention layer of the Swing Transformer encoder, combined with IT and FST, the spatial details of the image are recovered, yielding high-level semantic features of the processed RGB image. High-level semantic features of depth images .

[0055] like Figure 3 As shown, semantic features at different levels ( , , , , , , , The input is fed into a multi-stage fusion module, where semantic features at different levels are processed in parallel across three stages.

[0056] Phase 1: Input consists of low-level semantic features and ,Will and Perform element-wise multiplication and concatenation separately, then pass the two results together through a 1x1 convolution and concatenate them again to obtain the feature map. ;

[0057] The results are sequentially processed through a combination of 3×3 convolutions and three 3×3 dilated convolutions, then concatenated and output as the final low-level fusion feature through a 1×1 convolution. The specific formula is as follows:

[0058]

[0059]

[0060] in, This represents a 1×1 convolution operation. This represents element-wise addition. This represents element-wise multiplication. This represents the low-level feature map after initial fusion. This represents a 3×3 convolution operation. Indicates low-level fusion features, This represents the feature map after dilated convolution, where i equals 3, 5, or 7.

[0061] Second stage: Input is intermediate semantic features , and , The process is divided into two branches. One branch performs element-wise multiplication, and then feeds the result into a selective region attention (SRA) module to enhance salient features, resulting in a feature map. and feature map The other branch first performs a concatenation operation, then a 1×1 convolution, and finally merges it with the feature map. and feature map The fusion process yields the final intermediate fusion characteristics. and The specific formula is as follows:

[0062]

[0063]

[0064] in, This represents the output of the selective region attention module. This represents the output of the spatial attention branch. This represents the output of the channel attention branch. This represents the output feature map after SRA processing. This represents the convolution operation. Representing intermediate semantic feature maps from RGB paths or , Represents intermediate semantic feature maps from deep paths or , This represents element-wise addition. This represents element-wise multiplication. This represents a selective region attention mechanism.

[0065] Third stage: Input is high-level semantic features and First perform element-wise multiplication, then combine with... and The concatenated results are concatenated again; after passing through a dimensionality reduction optimization (DO) module, this module first uses 3×3 convolution + batch normalization (BN) + Gaussian error linear unit activation function (GELU) to compress the fused high-dimensional features to low dimensions and remove redundant information; then, 1×1 convolution is used to refine the channels again, significantly reducing the number of parameters and computational cost, while retaining key semantics, providing lighter and more efficient input features for subsequent channel fusion (CF) and spatial attention (SA); after passing through the channel fusion (CF) module, the spatial features are enhanced by the spatial attention (SA) module, finally outputting high-level fused features. The specific formula is as follows:

[0066]

[0067] in, Indicates advanced fusion features, This represents element-wise addition. This represents element-wise multiplication. This represents the spatial attention mechanism. Indicates channel merging operation. This indicates a dimensionality reduction optimization operation.

[0068] In the cortical decoder, the first step is to fuse them into a single feature. Phase 1; This phase involves two stages of decoding, inputting intermediate fusion features. , and advanced fusion features For each fusion feature , , The features are processed separately, using different dilated convolutions to extract multi-scale features. The outputs are then merged through element-wise multiplication. The merged features are fed into three 3×3 convolutional layers for parallel processing to further extract features. Finally, the three parallel-processed features are concatenated into a single feature. The specific formula is as follows:

[0069]

[0070] in, This indicates a feature that incorporates multiple layers of high-level visual features. This indicates the advanced visual features after initial fusion and processing. Indicates the features after upsampling Perform dilated convolution operation. Represents the upsampled feature map Perform a 3×3 convolution operation. Indicates intermediate fusion features Features after dilated convolution This indicates a feature concatenation operation. This indicates an element-wise multiplication operation.

[0071] The next step is to perform channel fusion again on the spliced ​​results: fusion features First, channel fusion is performed to obtain features. Then, the concatenated results undergo another dimensionality optimization stage: dimensionality optimization is performed again, combining the optimized features with the low-level fused features. The data is then stitched together, and the stitched results undergo further channel fusion and dimensional optimization. Finally, a saliency prediction map is generated. The processed results are compared with... By concatenating the data, we obtain the final significance prediction map output by the model. The specific formula is as follows:

[0072] ;

[0073] in, This represents the significance prediction plot of the final output. Indicates channel merging operation. This indicates a dimension optimization operation. Indicates will Feature map after upsampling Indicates low-level fusion features, This indicates a feature splicing operation.

[0074] When training the OSFBiF-Net model, the saliency prediction map and boundary prediction map are compared with the corresponding ground truth map, the loss function is calculated, and these loss signals are used for backpropagation to optimize the model parameters.

[0075] Specifically, the binary cross-entropy (CE) loss is calculated at multiple scales, and a corresponding weighting coefficient is applied to the loss at each scale to comprehensively consider the significance and accuracy of boundary prediction at different scales. The specific expression of the multi-scale CE loss function is as follows:

[0076]

[0077] in, Represents cross-entropy loss, This represents the i-th element of the actual label. This represents the probability value of the i-th element in the probability distribution predicted by the model. This represents the total number of categories, and i represents the category index.

[0078] Furthermore, the Intersection over Union (IOU) loss is calculated at multiple scales, and corresponding weighting coefficients are applied to the loss at each scale to comprehensively consider the significance and accuracy of boundary prediction at different scales, as shown in the following formula:

[0079]

[0080] in, Indicates the loss by intersection and union. The crossover union ratio (CUNR) is the calculated value. A represents the predicted salient object region, and B represents the actual salient object region.

[0081] The total loss function is a weighted sum of four prediction losses, each of which simultaneously calculates the binary cross-entropy (CE) loss and the intersection-over-union (IOU) loss. The loss for each prediction is given by the following formula:

[0082]

[0083] in, Indicates the first Total loss in each prediction phase Represents the overall prediction map With truth value Binary cross-entropy loss between them Indicates the first Intermediate prediction chart With truth value The crossover loss between them.

[0084] The final loss function is a weighted sum of the four predicted losses, as shown in the formula:

[0085]

[0086] in, This represents the total loss of the model; , , , These represent the weighting coefficients of the four loss terms; , , , These respectively represent the features fused into a single feature in the cortical decoder. The losses incurred in the stages of channel fusion, dimension optimization, and saliency prediction map generation.

[0087] Finally, the OSFBiF-Net model trained using the above steps is used for RGB-D salient object detection. The RGB-D image to be detected is input into the OSFBiF-Net model, which generates a salient prediction map and a boundary prediction map.

[0088] The following describes experiments using the OSFBiF-Net model, and illustrates the technical effects of the embodiments of the present invention with specific experimental data.

[0089] In the experiment, four commonly used image description evaluation metrics were used to evaluate the performance of the model, namely, structural similarity measure ( ), maximum F value ( ), Maximum Enhanced Alignment Metric ( Mean absolute error ( The detailed description of the evaluation indicators is as follows:

[0090] (1) It is used to evaluate the structural similarity between the predicted map and the real map, combining region-level and object-level structural similarity, and also incorporating comparisons of brightness, contrast, and structure. The calculation formula is as follows:

[0091]

[0092] in, and These represent the predicted image and the actual image, respectively. and They represent and The mean, and They represent and variance express and covariance, and These represent the constants used in the stabilization formula.

[0093] (2) Precision and recall are calculated at different thresholds, and the highest score at the optimal threshold is reported to evaluate the model's accuracy in the object detection task. The calculation formula is as follows:

[0094]

[0095] in, Indicates accuracy; Indicates recall rate; This parameter is used to balance precision and recall, and is usually set to 1.

[0096] (3) The robustness of the model in complex scenes is evaluated by combining local pixel values ​​and image-level means, while also considering global statistics and local pixel matching. The calculation formula is as follows:

[0097]

[0098] in, This represents the prediction graph. Represents a real image. Represents a similarity measure. This represents the parameter used to adjust the steepness of the curve.

[0099] (4) Calculating the pixel-level mean absolute error between the predicted saliency map and the true value provides an accurate assessment of the model's error range, which is particularly valuable for fine-grained evaluation. The calculation formula is as follows:

[0100]

[0101] in, Indicates the total number of pixels. and Let represent the values ​​of the predicted image and the ground truth image at the i-th pixel, respectively.

[0102] To verify the effectiveness of the two-stage sparse semantic augmentation module, experiments were conducted on the LFSD, NLPR, and ReDweb-S datasets. The experimental results are shown in Tables 1-3. `w / o DSSE` represents the model variant without the DSSE module; `Swin-DESS(S)` represents the model variant using a smaller-scale Swin Transformer as the backbone network and integrating the DSSE module; `Swin-DESS(B)` represents the model variant using a larger-scale version of the Swin Transformer as the backbone network and integrating the DSSE module. The backbone network refers to the core part of the model, used to extract features from the input image.

[0103] Table 1. Performance metrics of the DSSE module on different model variants and the LFSD dataset.

[0104]

[0105] Table 2. Performance metrics of the DSSE module on different model variants and the NLPR dataset.

[0106]

[0107] Table 3. Performance metrics of the DSSE module on different model variants and the ReDweb-S dataset.

[0108]

[0109] It can be seen that, compared with w / o DSSE, the flops of Swin-DESS(S) are reduced by 33% without sacrificing accuracy; this demonstrates that integrating the DESS module into the third stage of the Swin Transformer architecture significantly reduces the computational cost of the backbone architecture.

[0110] To verify the effectiveness of the multi-stage fusion module, experiments were conducted on the NLPR and DUT-Depth datasets. The experimental results are shown in Tables 4 and 5. Baseline represents a model variant using a simple additive fusion method; +Stage1 represents a model variant with the first stage of the multi-stage fusion module added to Baseline; +Stage1+Stage2 represents a model variant with the second stage of the multi-stage fusion module further added to +Stage1; and +Stage1+Stage2+Stage3 represents a complete model variant with all three stages of the multi-stage fusion module added.

[0111] Table 4 Performance metrics of the multi-stage fusion module on the NLPR dataset

[0112]

[0113] Table 5 Performance metrics of the multi-stage fusion module on the DUT-Depth dataset

[0114]

[0115] It can be seen that the multi-stage fusion module significantly improves performance on the NLPR and DUT-Depth datasets. , and The metrics also reduced the number of erroneous metrics. Experimental results show that the introduction of a multi-stage fusion module helps to enhance feature representation and reduce noise, thereby optimizing the overall performance of the multi-stage fusion module.

[0116] To verify the effectiveness of the cortical decoder, experiments were conducted on the NLPR and DUT-Depth datasets, and the results are shown in Tables 6 and 7. Baseline represents a model variant using a simple hierarchical decoding and summation method, and w / oHighLevelVisual indicates the removal of high-level fusion features from the base model. The variant, w / o lowLevelVisual, indicates the exclusion of low-level fusion features from the base model. The variant HighLevelVisual+LowLevelVisual represents a model variant that uses the full cortical decoder.

[0117] Table 6 Performance metrics of the cortical decoder on the NLPR dataset

[0118]

[0119] Table 7 Performance metrics of the cortical decoder on the DUT-Depth dataset

[0120]

[0121] As can be seen, the cortical decoder performs excellently on both the NLPR and DUT-Depth datasets. Compared to variants without a cortical decoder, the cortical decoder significantly improves the evaluation metrics. , , and reduced Compared to the w / oHighLevelVisual and w / olowLevelVisual variants, the full model achieves better optimization of parameters and computational complexity, demonstrating the effectiveness of the cortical decoder.

[0122] To demonstrate the effectiveness and generalization of the embodiments of the present invention, the OSFBiF-Net model of the present invention was evaluated and compared with existing RGB-D salient object detection models on five commonly used datasets, and the experimental results are shown in Tables 8-12.

[0123] Table 8. Performance metrics of different models on the RGBD135 dataset

[0124]

[0125] Table 9. Performance metrics of different models on the SIP dataset

[0126]

[0127] Table 10 Performance metrics of different models on the LSFD dataset

[0128]

[0129] Table 11 Performance metrics of different models on the ReDweb-S dataset

[0130]

[0131] Table 12 Performance metrics of different models on the DUTLE-Depth dataset

[0132]

[0133] As can be seen, OSFBiF-Net demonstrates superior performance, significantly outperforming other models on the RGBD135, SIP, LFSD, ReDWeb-S, and DUTLF-Depth datasets. Compared to existing models, OSFBiF-Net achieves significant improvements across multiple evaluation metrics, particularly excelling in handling complex scenes and multimodal data. This is likely because OSFBiF-Net effectively fuses multimodal features and bidirectional information flow, better capturing the features of salient targets and thus improving overall detection accuracy.

[0134] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any form or substance. It should be noted that those skilled in the art can make various improvements and additions without departing from the method of the present invention, and these improvements and additions should also be considered within the scope of protection of the present invention. Any modifications, alterations, and equivalent changes made by those skilled in the art based on the above-disclosed technical content without departing from the spirit and scope of the present invention are equivalent embodiments of the present invention. Furthermore, any modifications, alterations, and evolutions made to the above embodiments based on the essential technology of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A semantic feature- and bio-inspired RGB-D salient target detection method, characterized in that, Includes the following steps: The RGB-D image to be detected is input into the salient object detection model, which outputs the salient object detection result. The training method of the salient object detection model includes the following steps: Construct a neural network that includes an encoder, a multi-stage fusion module, and a cortical decoder; By extracting features from RGB-D image samples using an encoder, low-level, mid-level, and high-level semantic features of RGB and depth images are obtained. The multi-stage fusion module fuses the low-level semantic features of RGB images and depth images to obtain low-level fusion features; it fuses the mid-level semantic features of RGB images and depth images to obtain mid-level fusion features; and it fuses the high-level semantic features of RGB images and depth images to obtain high-level fusion features. The saliency prediction map is obtained by decoding the low-level, mid-level and high-level fused features through the cortical decoder. The parameters of the neural network are updated based on the loss function to obtain a salient object detection model.

2. The method according to claim 1, characterized in that, The encoder includes a DSSE module, in which the input features are segmented into multiple patches, and adjacent patches are fused to obtain local clustering tokens. Multi-head attention interaction is performed between local clustering tokens and original image tokens to generate preliminary semantic tokens; Spatial pooling is performed on the initial semantic tokens to capture global semantic information and generate a global central token; The global central token and the initial semantic token are fused together using a heterogeneous attention mechanism to generate the final semantic token.

3. The method according to claim 1, characterized in that, The fusion of low-level semantic features of RGB images and depth images includes: The low-level semantic features of the RGB image and the depth image are multiplied and concatenated element-wise. The two results are then concatenated again through a 1×1 convolution to obtain the feature map. ; Feature map The results are sequentially combined using a combination of 3×3 convolutions and three 3×3 dilated convolutions, then concatenated and processed by a 1×1 convolution to obtain low-level fusion features.

4. The method according to claim 1, characterized in that, The fusion of intermediate semantic features from RGB and depth images includes: The intermediate semantic features of the RGB and depth images are divided into two branches. One branch performs element-wise multiplication, and then a selective region attention mechanism is used to enhance salient features to obtain the feature map. and feature map The other branch first performs a concatenation operation, then a 1×1 convolution, and finally merges it with the feature map. and feature map Fusion yields intermediate fusion characteristics.

5. The method according to claim 1, characterized in that, The fusion of high-level semantic features from RGB and depth images includes: The high-level semantic features of the RGB and depth images are first multiplied element-wise, and then concatenated with the concatenation result of the high-level semantic features of the RGB and depth images. After dimensionality reduction optimization and channel fusion, spatial features are enhanced using a spatial attention mechanism to obtain high-level fused features.

6. The method according to claim 1, characterized in that, The decoding of low-level, mid-level, and high-level fused features includes: Dilated convolutions are applied to the intermediate and high-level fusion features, which are then merged through element-wise multiplication. The merged features are then fed into three 3×3 convolutional layers for parallel processing. Finally, the three parallel-processed features are concatenated into a single feature. ; Features Channel fusion is performed to obtain features. Then, dimensionality optimization is performed, and the optimized features are concatenated with the low-level fusion features. The concatenated result is then subjected to channel fusion and dimensionality optimization again. The processed result is compared with... The data is then stitched together to obtain a significance prediction map.

7. The method according to claim 1, characterized in that, The loss function is: in, This represents the total loss of the salient object detection model; , , , These represent the weighting coefficients of the four loss terms; , , , These respectively represent the features fused into a single feature in the cortical decoder. The losses incurred in the stages of channel fusion, dimension optimization, and saliency prediction map generation.

8. An electronic device, characterized in that, include: At least one processor; and memory that is communicatively connected to at least one processor; The memory stores instructions that, when executed by at least one processor, implement the semantic feature- and bio-inspired RGB-D salient target detection method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The device stores instructions that, when executed by a processor, implement the RGB-D salient object detection method based on semantic features and bio-inspired principles as described in any of claims 1-7.

Images