RGB-D saliency target detection method based on hierarchical feature propagation and scale perception

Abstract

The invention relates to the technical field of computer vision and image processing, in particular to an RGB-D saliency target detection method based on hierarchical feature propagation and scale perception. The method comprises the following steps: acquiring a to-be-detected RGB image and a corresponding depth map, and preprocessing input data; a double-flow pyramid visual Transform is used as an encoder, and multi-level feature extraction is carried out on the RGB image and the depth map; performing weighted aggregation on adjacent hierarchical features through an adaptive feature aggregation module to enhance complementarity between high-level semantic information and low-level space detail information; adopting a hierarchical cross-modal fusion strategy to carry out detail positioning fusion on low-level features, and carrying out global semantic fusion on high-level features; and a scale perception decoding structure composed of a multi-scale context convolution block and a perception refining unit is used to decode the fusion features step by step, and a final saliency target detection result is generated.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, specifically to an RGB-D salient target detection method based on hierarchical feature propagation and scale awareness, which is suitable for fusing RGB appearance information and depth structure information to achieve accurate localization of salient target regions. Background Technology

[0002] Salient object detection (SOD) aims to automatically identify the most visually appealing regions in complex scenes, serving as a crucial foundation for various visual tasks such as image segmentation, object detection, scene understanding, and human-computer interaction. With the development of deep learning technology, RGB image-based salient object detection methods have achieved significant performance improvements. However, single RGB images often lack reliable geometric and structural information in complex backgrounds, low-contrast regions, and occluded scenes, leading to unstable detection results. Therefore, RGB-D salient object detection methods incorporating depth information have gradually gained attention. Depth maps provide spatial structure and geometric cues within a scene, effectively supplementing salient object localization. RGB-D salient object detection (SOD) aims to accurately locate and segment the most attractive regions in an image by fusing color (RGB) and depth images. However, existing technologies still face the following challenges:

[0003] (1) There is a lack of effective information transmission mechanism between features at different levels, and high-level semantic information is difficult to guide low-level detailed features; (2) Cross-modal fusion strategies generally adopt rigid designs and do not fully consider the differences between RGB and deep features at different semantic levels; (3) Insufficient modeling of multi-scale structural relationships in the decoding stage can easily lead to incomplete or blurred boundaries in significant regions.

[0004] Therefore, there is a need for an RGB-D salient object detection method that can simultaneously model multi-level feature dependencies, achieve cross-modal adaptive fusion, and possess scale-aware capabilities. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes an RGB-D salient target detection method based on hierarchical feature propagation and scale awareness. By constructing a hierarchical feature propagation mechanism, a hierarchical cross-modal fusion strategy, and a scale-aware decoding structure, high-precision detection of salient target regions can be achieved.

[0006] This invention provides the following technical solution: an RGB-D salient target detection method based on hierarchical feature propagation and scale awareness, comprising: 1. Acquire and input RGB-D image data; 1.1) Collect publicly available datasets in the field of RGB-D salient object detection, including the LFSD dataset, NJU2K dataset, NLPR dataset, DUT-RGBD dataset, SIP dataset, and STERE dataset.

[0007] 1.2) The RGB image and its corresponding depth map are used as bimodal input data of the network, and the corresponding salient target annotation map is obtained as supervision information.

[0008] 2. Multi-level feature coding; 2.1) Input the RGB image and depth map into the shared structure encoding network respectively, and use the Pyramid Vision Transformer (PVT) to extract multi-level features from the input data.

[0009] 2.2) During the encoding process, RGB features and depth features are obtained at different levels, denoted as follows: and ,in These correspond to different semantic levels of features, from low to high.

[0010] 3. Adaptive feature aggregation; 3.1) Input the multi-level RGB features and depth features obtained in step 2 into the Adaptive Feature Aggregation Module (AFAM) to perform spatial scale alignment on the features at different levels.

[0011] 3.2) The aligned features are mapped by convolution operation, and adjacent layer features are spliced ​​and aggregated to enhance the complementary relationship between high-level semantic information and low-level spatial detail information.

[0012] 3.3) Adaptive weighting of aggregated features is performed through a channel attention mechanism to suppress redundant feature channels, resulting in enhanced multi-level RGB features. with depth features .

[0013] 4. Layered cross-modal fusion; 4.1) Targeting low-level features ( The RGB features and depth features obtained in step 3 are input into the Detail Localization Fusion Module (DLFM). The low-level features are spatially aligned and noise is suppressed through a multi-head self-attention mechanism, thereby achieving cross-modal fusion of fine structural information.

[0014] 4.2) Targeting high-level characteristics ( The RGB features and deep features obtained in step 3 are input into the Global Semantic Fusion Module (GSFM), which models the global semantic dependency between RGB and deep features through a grouped channel interaction attention mechanism.

[0015] 4.3) Output the low-level fusion result and the high-level fusion result separately as input features for the subsequent decoding stage.

[0016] 5. Scale-aware decoding; 5.1) Input the multi-level fusion features obtained in step 4 into the scale perception decoder (SP Decoder).

[0017] 5.2) Model features at different scales by using multi-scale contextual convolutional blocks, and refine salient regions step by step by combining perceptual refining units.

[0018] 5.3) The decoded features are fused to output the final saliency prediction result.

[0019] 5.4) Generate a salient target detection map through the output layer to locate the salient target region in the input RGB-D image.

[0020] In addition, during the training phase, a deep supervision strategy is adopted to supervise the learning of output results at different levels and real annotations, so as to improve the stability of model training and detection accuracy.

[0021] Based on the above steps, this invention constructs an RGB-D salient object detection method with hierarchical feature propagation as its core, scale-aware decoding as its support, and hierarchical cross-modal fusion as its key component. Specifically, an adaptive feature aggregation module enhances information transfer between features at different levels, a detail localization fusion module aligns low-level structures, and a global semantic fusion module models high-level semantic consistency, thereby effectively improving the completeness and boundary accuracy of salient object detection in complex scenes.

[0022] This invention fully utilizes the complementarity of RGB images and depth maps in terms of appearance and structural information through hierarchical feature propagation and hierarchical cross-modal fusion, which can effectively improve the accuracy, completeness and boundary localization precision of salient target detection in complex scenes. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall structure of the RGB-D salient target detection model based on hierarchical feature propagation and scale awareness of the present invention. Figure 2This is a schematic diagram of the feature aggregation module in this invention; Figure 3 This is a schematic diagram of the detailed positioning and fusion module in this invention; Figure 4 This is a schematic diagram of the global semantic fusion module in this invention; Figure 5 This is a schematic diagram of the structure of the multi-scale contextual convolution block in this invention; Figure 6 This is a schematic diagram of the sensing and refining unit in this invention. Detailed Implementation

[0024] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Example 1 The embodiments of this invention are based on an RGB-D salient target detection model based on hierarchical feature propagation and scale awareness. A schematic diagram of the overall structure of this model is shown below. Figure 1 As shown. In one embodiment of the present invention, an RGB-D salient target detection method based on hierarchical feature propagation and scale awareness is provided, comprising the following steps: S1. Obtain the RGB image to be detected and its corresponding depth map, and preprocess the input data; Preferably, the specific method of step S1 is as follows: Obtain an RGB-D salient object detection dataset, wherein the samples in the dataset include RGB images, spatially aligned depth maps, and corresponding salient object annotation maps; The RGB image and its depth map are used as bimodal input data for the network, and the corresponding salient target annotation map is obtained as supervision information.

[0026] In one implementation, since saliency target detection based solely on RGB lacks stable geometric constraints in complex backgrounds, occlusions, or low-light scenes, this invention introduces depth images as a supplementary modality and employs an RGB-D dual-modal input method for saliency target detection to provide spatial structure cues and improve the robustness of saliency prediction.

[0027] First, publicly available datasets in the field of RGB-D salient object detection were collected, including the LFSD dataset, NJU2K dataset, NLPR dataset, DUT-RGBD dataset, SIP dataset, and STEREO dataset. Each sample in these datasets contains an RGB image, a spatially aligned depth image, and a corresponding salient object annotation map.

[0028] The RGB image and its corresponding depth image are used as bimodal input data for the network, while the corresponding salient object annotation map is used as supervision information. Let the input RGB image and depth image be represented as: ; ; in, and These represent the height and width of the image, respectively. The RGB image and depth image are normalized and spatially aligned before input to ensure the accuracy of subsequent feature extraction and fusion.

[0029] S2. Perform multi-level feature encoding on the RGB image and depth map to extract RGB features and depth features at different semantic levels; Preferably, the specific method of step S2 is as follows: The RGB image and depth map are respectively input into a shared-structure coding network for feature extraction; By performing stepwise downsampling and feature mapping operations, RGB features and depth features with different spatial resolutions and semantic levels are obtained.

[0030] In one implementation, since salient targets typically contain both local edge details and global semantic structure information, a single-level feature is insufficient to balance target integrity and boundary clarity. Therefore, this invention employs a multi-level feature encoding method to obtain feature representations at different semantic levels, providing a foundation for subsequent cross-level propagation and cross-modal fusion.

[0031] The RGB and depth images obtained in step 1 are respectively input into a shared-structure encoding network for feature extraction. The encoding network uses a Pyramid Vision Transformer (PVT) as its basic encoding structure to perform multi-level feature modeling on the input data. The encoding network extracts feature representations at different spatial resolutions and semantic levels through progressive downsampling and feature mapping operations.

[0032] During the encoding process, RGB features and depth features are obtained at different levels, as follows: ; in, and Representing the first RGB and depth features extracted from the layers, and They represent the first Layer RGB and depth feature extraction operators. These are low-level features, mainly containing spatial structure and edge detail information; These are high-level features, primarily containing abstract semantic information.

[0033] S3. Adaptive feature aggregation is performed on RGB features and deep features at different semantic levels to generate aggregated features; Preferably, the specific method of step S3 is as follows: Spatial scale alignment is performed on RGB features and depth features at different levels; The scale-aligned features of adjacent layers are concatenated, and cross-layer feature aggregation is performed through convolutional mapping. A channel attention mechanism is introduced to aggregated features to generate channel weights, and the aggregated features are weighted to obtain enhanced features.

[0034] In one implementation, because low-level features have strong spatial detail representation capabilities but are susceptible to noise interference, while high-level features have strong semantic representation capabilities but lack precise boundary information, a cross-level information interaction mechanism is needed. This invention utilizes an adaptive feature aggregation module, such as... Figure 2 As shown, feature aggregation is performed between adjacent levels, and redundant channels are suppressed by attention weighting, thereby enhancing the complementarity of multi-layer features.

[0035] The multi-level RGB features and depth features obtained in step 2 are input into the adaptive feature aggregation module to perform spatial scale alignment on the features at different levels. The process is as follows: ; in, Indicates an upsampling or downsampling operation. Indicates the first RGB features and depth features at different levels Indicates the aligned first Hierarchical RGB features and depth features.

[0036] The scale-aligned features from adjacent layers are concatenated and convolved to achieve cross-layer feature aggregation, generating aggregated features. Its expression is: ; A channel attention mechanism is introduced for aggregated features, generating channel weights through global average pooling and nonlinear mapping: ; The features are then adaptively weighted to obtain enhanced multi-level features: ; S4. Perform hierarchical cross-modal fusion on the aggregated features to generate fused features; Preferably, the specific method of step S4 is as follows: For low-level features, RGB features and depth features are input into the detail localization and fusion module, and cross-modal fusion is performed through an attention mechanism; For high-level features, RGB features and deep features are input into the global semantic fusion module and fused through bidirectional cross-modal semantic interaction.

[0037] In one implementation, since the RGB modality emphasizes appearance and texture information while the depth modality emphasizes geometric structure information, features at different semantic levels exhibit significant differences in noise sensitivity and semantic abstraction. Adopting a unified fusion strategy could easily lead to alignment errors and noise propagation. Therefore, this invention designs separate fusion methods for low-level and high-level features: low-level features emphasize structural alignment and noise suppression, while high-level features emphasize global semantic consistency modeling.

[0038] Targeting low-level features The RGB features obtained in step 3 with depth features The input is fed into the detail localization and fusion module for cross-modal fusion. This module employs a multi-head self-attention mechanism to model low-level features, enhancing spatial alignment and suppressing noise interference. Its structure is as follows: Figure 3 As shown. First, a linear mapping is performed between RGB features and deep features to construct query, key, and value vectors: ; in, , and This is a learnable mapping matrix. Subsequently, the cross-modal attention response is computed using a scaled dot product attention mechanism. : ; in, This represents the feature dimension of the attention head. To enhance feature representation and stabilize the training process, the attention output is linearly mapped and residually fused with the input features to obtain low-level fused features: ; in, To output the projection matrix, This indicates an element-wise addition operation.

[0039] Targeting high-level characteristics The RGB features and depth features obtained in step 3 are input into the global semantic fusion module for cross-modal semantic interaction, and its structure is as follows: Figure 4 As shown, this module enhances high-level semantic consistency by modeling the global semantic dependencies between RGB and deep features.

[0040] First, to expand the receptive field and capture long-range contextual information, dilated depthwise convolutions are applied to both the RGB features and the depth features: ; Subsequently, using RGB features as the query source and depth features as the key and value, a cross-modal attention relationship is constructed: ; ; Similarly, deep features can be used as the query source, and RGB features as the key and value, to obtain the reverse attention result. The bidirectional cross-modal attention results are concatenated using multiple heads and linearly mapped to obtain high-level fusion features: ; in, Indicates the number of attention heads. This is the output mapping matrix.

[0041] Low-level fusion features Features of integration with high-level leaders The output serves as the input features for the scale-aware decoding stage.

[0042] S5. Scale-aware decoding is performed on the fused features to generate salient target detection results.

[0043] Preferably, the specific method of step S5 is as follows: Multi-scale context modeling is performed on the fused features to obtain multi-scale feature representations; The multi-scale feature representation is enhanced step by step by perceptual refining units to generate channel attention weights and spatial attention weights. Then, the features are weighted and refined features are obtained through residual connections. The refined features are fused step by step to obtain the decoded features; The decoded features are mapped through the output layer to generate the final salient target detection map.

[0044] In one implementation, since salient targets exhibit significant scale variations, and relying solely on simple upsampling can easily lead to incomplete target regions or blurred boundaries, this invention employs a scale-aware decoding approach. Through multi-scale contextual modeling and progressive refinement enhancement, it explicitly models cross-scale relationships, thereby improving the completeness and boundary clarity of the prediction results. Its structure is as follows: Figure 5 and Figure 6 As shown.

[0045] Multi-level fused features are input into multi-scale contextual convolutional blocks, and multi-scale contextual information is extracted through convolution operations with different dilation rates: ; Features obtained at different scales are concatenated and compressed to obtain context-enhanced features: ; To further enhance the response in salient regions and suppress background interference, a perceptual refinement unit is introduced to enhance contextual features, and its structure is as follows: Figure 6 As shown. First, channel attention weights and spatial attention weights are generated: ; The features are then weighted and refined features are obtained through residual connections: ; in, This represents element-wise multiplication. This indicates an element addition operation.

[0046] The refined features at different levels are fused step by step to obtain the final decoded feature representation.

[0047] After completing multi-layer feature fusion and scale-aware decoding, this invention maps the final features into a saliency prediction map, and then maps the decoded features through the output layer to generate the final saliency target detection map:

[0048] Preferably, during the training phase, the method employs a deep supervision strategy to jointly supervise the prediction results at different levels.

[0049] In one implementation, to improve the feature learning performance of intermediate layers and accelerate convergence, this invention applies deep supervision constraints to the multi-layer outputs during the training phase. During training, a deep supervision strategy is employed to jointly supervise the prediction results of different layers, and its loss function is defined as: ; in, Indicates the first The saliency prediction results of the layer This represents the corresponding true saliency annotation map.

[0050] Example 2 Drones need to accurately identify insulator damage or tower defects in varied environments such as mountains, fog, and strong sunlight reflections. Because the background and target can be visually highly similar, and the shooting angle fluctuates significantly due to wind, a single RGB image often struggles to distinguish target edges, leading to unstable detection results. To overcome these shortcomings, this invention employs an RGB-D salient target detection method based on hierarchical feature propagation and scale awareness. The implementation process of this method is as follows: S1. The visible light camera and infrared depth sensor onboard the drone simultaneously acquire RGB images and depth maps. The system performs image size normalization and spatial alignment to eliminate physical deviations caused by sensor displacement.

[0051] S2. Using the dual-stream pyramid visual Transformer (PVT) as an encoder, low-level features containing edge details and high-level features containing tower semantics are extracted.

[0052] S3. Aggregate features from adjacent levels and use channel attention mechanism to suppress background noise from high-altitude clouds and fog, and enhance the feature response of small targets such as insulators.

[0053] S4. For low-level features, the detail localization and fusion module is used to enhance edge alignment; for high-level features, the global semantic fusion module is used to model the global differences between the overall tower structure and the background.

[0054] S5. Scale-aware decoding: Utilizes multi-scale contextual convolutional blocks to model the differences in object scale caused by changes in UAV altitude during inspection, ultimately generating a high-precision defect saliency map.

[0055] Table 1 Performance Comparison of the State-of-the-Art (SOTA) Models in Power Scenarios Based on the Invention

[0056] Table 1 shows a performance comparison of this invention with existing state-of-the-art (SOTA) models in this field. The table reveals that this invention significantly outperforms existing SOTA models in all core evaluation metrics. Specifically: Significantly improved detection accuracy: The present invention has the lowest mean absolute error (MAE) at 0.035, which is about 16.7% lower than SPNet. This proves that the present invention has higher pixel-level accuracy in complex background suppression and precise localization of significant regions, and can effectively filter out environmental noise such as high-altitude clouds and fog.

[0057] Superior structural preservation and integrity: This invention (F-measure) and The structural similarity index reached the highest value, at 0.941 and 0.932 respectively, which indicates that the method can maintain extremely high target integrity and boundary clarity when dealing with small targets such as insulator damage and complex tower topology.

[0058] This invention effectively enhances the complementarity between high-level semantics and low-level details by weighted aggregation of features from adjacent layers through an adaptive feature aggregation module, thus solving the problem of blurred target edges in complex backgrounds for models such as BBS-Net. Addressing the differences between RGB and depth information in inspection scenarios, it employs a strategy of detail localization fusion and global semantic fusion, avoiding the loss of details caused by the rigid cross-modal fusion design of traditional models. Because the scale-aware decoding structure introduces multi-scale context modeling and perceptual refinement units, this invention can accurately capture targets at different scales due to changes in drone altitude, resulting in a significant lead in structural similarity metrics.

[0059] In summary, the method described in this invention exhibits strong robustness and detection accuracy in complex industrial application scenarios such as power line inspection, demonstrating significant technological advancement.

[0060] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0061] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0062] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An RGB-D salient target detection method based on hierarchical feature propagation and scale awareness, characterized in that, include: S1. Obtain the RGB image to be detected and its corresponding depth map, and preprocess the input data; S2. Perform multi-level feature encoding on the RGB image and depth map to extract RGB features and depth features at different semantic levels; S3. Adaptive feature aggregation is performed on RGB features and deep features at different semantic levels to generate aggregated features; S4. Perform hierarchical cross-modal fusion on the aggregated features to generate fused features; S5. Scale-aware decoding is performed on the fused features to generate salient target detection results.

2. The RGB-D salient target detection method based on hierarchical feature propagation and scale awareness according to claim 1, characterized in that, The specific method for step S1 is as follows: Obtain an RGB-D salient object detection dataset, wherein the samples in the dataset include RGB images, spatially aligned depth maps, and corresponding salient object annotation maps; The RGB image and its depth map are used as bimodal input data for the network, and the corresponding salient target annotation map is obtained as supervision information.

3. The RGB-D saliency target detection method based on hierarchical feature propagation and scale awareness according to claim 1, characterized in that, The specific method for step S2 is as follows: The RGB image and depth map are respectively input into a shared-structure coding network for feature extraction; By performing stepwise downsampling and feature mapping operations, RGB features and depth features with different spatial resolutions and semantic levels are obtained.

4. The RGB-D saliency target detection method based on hierarchical feature propagation and scale awareness according to claim 1, characterized in that, The specific method for step S3 is as follows: Spatial scale alignment is performed on RGB features and depth features at different levels; The scale-aligned features of adjacent layers are concatenated, and cross-layer feature aggregation is performed through convolutional mapping. A channel attention mechanism is introduced to aggregated features to generate channel weights, and the aggregated features are weighted to obtain enhanced features.

5. The RGB-D saliency target detection method based on hierarchical feature propagation and scale awareness according to claim 1, characterized in that, The specific method for step S4 is as follows: For low-level features, RGB features and depth features are input into the detail localization and fusion module, and cross-modal fusion is performed through an attention mechanism; For high-level features, RGB features and deep features are input into the global semantic fusion module and fused through bidirectional cross-modal semantic interaction.

6. The RGB-D saliency target detection method based on hierarchical feature propagation and scale awareness according to claim 1, characterized in that, The specific method for step S4 is as follows: Multi-scale context modeling is performed on the fused features to obtain multi-scale feature representations; The multi-scale feature representation is enhanced step by step by perceptual refining units to generate channel attention weights and spatial attention weights. Then, the features are weighted and refined features are obtained through residual connections. The refined features are fused step by step to obtain the decoded features; The decoded features are mapped through the output layer to generate the final salient target detection map.

7. The RGB-D salient target detection method based on hierarchical feature propagation and scale awareness according to claim 1, characterized in that, During the training phase, the method employs a deep supervision strategy to jointly supervise the prediction results at different levels.

Images