Fusion frequency domain guided rgb-d lightweight semantic segmentation method

Abstract

The application provides a fusion frequency domain guided RGB-D lightweight semantic segmentation method, relates to the technical field of image processing, and improves the consistency and propagation efficiency of cross-layer semantic features by designing a frequency domain guided prompt adapter. Secondly, a spectrum guided dynamic convolution module is proposed, which realizes efficient multi-scale feature modeling while fusing spatial domain and frequency domain features. Finally, a multi-scale frequency domain proxy attention module is constructed to enhance the semantic interaction and global modeling capability between different scale features in a low overhead manner. The method of the application can realize excellent segmentation performance with low parameter amount on multiple RGB-D and RGB-L semantic segmentation datasets, and shows good generalization ability on 5 datasets in the RGB-D significant target detection task. The research results show that the method of the application is more accurate in the segmentation result in a complex structure scene.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a lightweight semantic segmentation method for RGB-D with frequency domain guidance. Background Technology

[0002] Semantic segmentation, a core task in computer vision, aims to assign semantic labels to each pixel in an image, thereby achieving detailed scene analysis. With the rapid development of deep learning, semantic segmentation technology has demonstrated enormous potential and broad prospects in practical applications such as autonomous driving, intelligent monitoring, and medical image analysis.

[0003] In complex indoor scenes, challenges such as low lighting, frequent occlusion, and diverse object shapes make it difficult to capture deep structure and geometric information when relying solely on RGB images for semantic segmentation, leading to a significant decrease in segmentation accuracy. To address these issues, researchers have gradually introduced auxiliary modality fusion strategies, such as the RGB-D method that fuses RGB and depth maps, and the RGB-L method that fuses RGB and LiDAR. These methods compensate for the limitations of RGB single-modality representation by leveraging the complementary advantages between different modalities. Among them, RGB-D semantic segmentation, by combining visual and geometric information, significantly enhances the model's ability to model fine-grained features and has become a major direction in current multimodal semantic segmentation research.

[0004] Early RGB-D methods typically concatenated depth maps and RGB images directly into a four-channel input fed into shallow convolutional networks, validating the feasibility of multimodal input. However, they failed to consider the fundamental differences in feature distribution between RGB and depth maps. To enhance the expressive power of the depth modality, single-channel depth maps were converted into three-channel geometric representations, and HHA encoding improved segmentation performance. However, with limited training data, the model might tend to learn feature patterns from RGB images, neglecting the uniqueness of depth information. Further research proposed a dual-branch encoder-decoder structure to extract hierarchical features from RGB and depth modalities separately and fuse them across multiple layers, thus more effectively integrating complementary information between modalities. By extending traditional RNNs to multimodal inputs and utilizing a shared "memory" mechanism to dynamically fuse color and depth features, the ability to distinguish object categories and geometric relationships was enhanced. Mapping RGB-D pixels to 3D space to construct a K-nearest neighbor graph structure and aggregating local geometric and semantic information through graph neural networks significantly improved segmentation performance in indoor scenes. However, these methods largely rely on local receptive fields, making it difficult to model global context and limiting their adaptability to unstructured data.

[0005] In recent years, to overcome the limitations of traditional methods in global modeling, attention mechanisms have been introduced into the field of computer vision and have rapidly become a research hotspot. For example, dual ResNet branches are used to extract RGB and depth image features separately, and these features are fused into a third branch by introducing a complementary attention module. Channel weights are dynamically adjusted based on feature information at different levels, thereby enhancing the network's responsiveness to key regions. Based on the Transformer architecture, a cross-modal feature correction and fusion module is proposed to calibrate the features of another modality in both channel and spatial dimensions, and achieve effective fusion through a cross-attention mechanism. Currently, a progressively guided fusion strategy has also been proposed, and a depth enhancement module optimized for geometric structure has been designed. A content-enhanced mask attention mechanism and a multi-resolution feature fusion strategy are introduced to learn mask queries from the original image and its downsampled version, capturing rich contextual information and improving the model's semantic understanding ability. A self-attention mechanism is used to optimize single-modal features, and semantically aware positional encoding is introduced. By spatially constraining the computational range of localized attention, redundant computation is reduced while maintaining structural continuity. A multi-modal alignment mechanism based on additive attention and a dot-product attention fusion strategy are designed to effectively improve fusion quality and suppress noise interference. While most methods focus on improving segmentation accuracy, lightweight network design and inference efficiency are increasingly valued. For example, asymmetric hybrid structures combining Transformers and convolutional neural networks (CNNs) effectively reduce computational burden while maintaining performance. Dual-branch networks built on the Mamba architecture can capture long-range dependencies and enhance intermodal information interaction. However, most current dual-encoder structures still face significant redundant computation and inference latency. Although these methods have made significant progress in improving performance, they generally rely on multi-encoder architectures and large-scale attention computations, making it difficult to achieve an ideal balance between model performance and computational efficiency.

[0006] Existing RGB-D semantic segmentation methods typically employ single-branch, dual-branch, and triple-branch architectures in their network structure design. Single-branch models use depth information as auxiliary input to the RGB branch, fusing it into a unified path. Dual-branch models extract features from both RGB and depth modalities separately, integrating information through an interaction module. Triple-branch models introduce additional fusion pathways to enhance intermodal collaboration, but also introduce higher model complexity. Current attempts aim to improve inference efficiency while maintaining performance through joint RGB-D pre-training strategies; however, further model compression without significantly sacrificing accuracy remains a critical challenge. Current mainstream methods generally rely on spatial domain convolution or Transformer modules to extract RGB and depth features separately, subsequently fusing them in shallow layers or during decoding. This approach lacks global modeling capabilities in the feature integration path, making it difficult to capture long-distance semantic relationships, leading to insufficient semantic alignment between modalities and affecting overall perceptual consistency. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to propose a lightweight semantic segmentation method for RGB-D with frequency domain guidance, comprising:

[0008] Acquire RGB and depth images, and convert the RGB image into I RGB and depth image I Depth The RGB features are obtained by inputting them into the embedding layer. and depth features ;

[0009] RGB features and depth features Input the hierarchical encoder to obtain the RGB features and depth features output by the four stages of the hierarchical encoder. Each stage of the hierarchical encoder contains multiple RGB-D blocks.

[0010] The RGB features output from the four stages are processed through a multi-scale frequency domain proxy and then concatenated to obtain the target features. The target features are then processed through a lightweight decoder head and an MLP to obtain a semantic segmentation prediction map.

[0011] Optionally, the RGB image I RGB and depth image I Depth The RGB features are obtained by inputting them into the embedding layer. and depth features Specifically, this is achieved through the following formula:

[0012] ;

[0013] ;

[0014] Where I is the RGB image I RGB and depth image I Depth , For convolution operations with a kernel size of 1, This is a convolution operation with a kernel size of 3, BN is a batch normalization layer, Y is the intermediate feature, and Z is the RGB feature. or depth features GELU represents the Gaussian error linear unit activation function.

[0015] Optionally, RGB features and depth features Input the hierarchical encoder to obtain the RGB features and depth features output from the four stages of the hierarchical encoder, including:

[0016] Will and The first stage of the hierarchical encoder is input to obtain the first stage RGB features. and the first stage Depth features ,right and Perform downsampling, and then... and downsampling The second stage of the input layer encoder is used to obtain the second stage RGB features. Second-stage Depth features ,right and Perform downsampling, and then... and downsampling Input the third stage of the hierarchical encoder to obtain the third stage RGB features. and the third stage Depth features ,right and Perform downsampling, and then... and downsampling Input the fourth stage of the hierarchical encoder to obtain the fourth stage RGB features. and the fourth stage Depth features .

[0017] Optionally, will and The first stage of the hierarchical encoder is input to obtain the first stage RGB features. and the first stage Depth features ,include:

[0018] Will and After passing through multiple RGB-D blocks in sequence, the result is... and In the i-th RGB-D block, let the input representation of the RGB-D block be... and , and After attention branching, attention features are obtained. , and After convolutional branching, convolutional features are obtained. Attention features After upsampling, it is combined with convolutional features By splicing, block aggregation features are obtained. Specifically, it is expressed by the following formula:

[0019] ;

[0020] Projecting the block aggregation features yields the output of the i-th RGB-D block. and ,Will and As the input to the (i+1)th RGB-D block, the output of the last RGB-D block in the first stage is: and ;

[0021] The processing procedures for the second, third, and fourth stages of the RGB-D block in the layered encoder are the same as those for the first stage, except that the objects being processed are different.

[0022] Optional, and After attention branching, attention features are obtained. ,include:

[0023] Based on the input RGB-D block and Generate query Q using the following formula:

[0024] ;

[0025] in, Indicates channel-dimensional splicing. Representation layer normalization, Indicates average pooling. Represents a linear transformation;

[0026] Based on the input RGB-D block Generate key K and value V, specifically through the following formula:

[0027] ;

[0028] Where GELU represents the Gaussian error linear unit activation function;

[0029] Calculate global attention based on query Q, key K, and value V. Specifically, this is achieved through the following formula:

[0030] ;

[0031] in, The query vector dimension is T, where T represents transpose and Softmax represents the activation function.

[0032] The hint token for the i-th RGB-D block And query the Q-input frequency domain cue adapter to obtain cue attention. ;

[0033] global attention and prompt attention By summing the features, we obtain the attention features. Specifically, it is expressed by the following formula:

[0034] ;

[0035] Will draw attention Reorganize to obtain the hint token for the (i+1)th RGB-D block. .

[0036] Optionally, the hint token for the i-th RGB-D block. And query the Q-input frequency domain cue adapter to obtain cue attention. ,include:

[0037] According to the prompt token Generate suggestion branch key and hint branch value Specifically, this is achieved through the following formula:

[0038] ;

[0039] in, Indicates the prompt token Reorganization This indicates that the operation segments the features along the spatial dimension;

[0040] Will The spectral energy is extracted using Fourier transform to generate a cue vector. Specifically, this is achieved through the following formula:

[0041] ;

[0042] in, denoted as Fourier transform, and MLP represents multilayer perceptron;

[0043] Based on the prompt vector Calculate attention prompts Specifically, this is achieved through the following formula:

[0044] ;

[0045] Where Softmax represents the activation function and T represents the transpose.

[0046] Optional, and After convolutional branching, convolutional features are obtained. Specifically, this is achieved through the following formula:

[0047] ;

[0048] ;

[0049] Based on the spectrum-guided dynamic convolution module, and Processing is performed to obtain convolutional features. Specifically, this is achieved through the following formula:

[0050] ;

[0051] in, This represents the intermediate features of Depth. Represents intermediate features of RGB. This represents a spectrum-guided dynamic convolution module. This represents the Hadamard product.

[0052] Optionally, let the input features of the spectrum-guided dynamic convolution module be represented as: , for or The spectrum-guided dynamic convolution module processes the input features. The processing includes:

[0053] For input features Convolutional and batch normalization layers are applied to obtain spatial domain features. Specifically, it is expressed by the following formula:

[0054] ;

[0055] in, For convolution operations with kernel size k, BN is a batch normalization layer;

[0056] Extracting input features The spectral characteristics F of are specifically expressed by the following formula:

[0057] ;

[0058] The spectral feature F is divided into low-frequency features. Mid-frequency characteristics and high frequency characteristics Specifically, it is expressed by the following formula:

[0059] ;

[0060] in, This indicates that the spectral feature F is divided into regions based on a set frequency threshold;

[0061] Low-frequency features Mid-frequency characteristics and high frequency characteristics Perform convolution to obtain low-frequency convolution features. Mid-frequency convolution features and high-frequency convolution features Specifically, this is achieved through the following formula:

[0062] ;

[0063] ;

[0064] ;

[0065] Low-frequency convolution features Mid-frequency convolution features and high-frequency convolution features By integrating the data, the fused spectral characteristics are obtained. Specifically, it is expressed by the following formula:

[0066] ;

[0067] fusion spectral features After inverse Fourier transform, the frequency domain features are obtained. Specifically, it is expressed by the following formula:

[0068] ;

[0069] spatial domain features and frequency domain features Element-wise summation yields the final output of the spectrum-guided dynamic convolution module. Specifically, it is expressed by the following formula:

[0070] ;

[0071] in, for or .

[0072] Optionally, the RGB features output from the four stages are subjected to multi-scale frequency domain proxy and feature concatenation to obtain the target features, including:

[0073] Will and After passing through the multi-scale frequency domain proxy attention module, the first fused feature is obtained, and the first fused feature is combined with... After passing through the multi-scale frequency domain proxy attention module, the second fusion feature is obtained, and the second fusion feature is combined with... After passing through the multi-scale frequency domain proxy attention module, the third fusion feature is obtained. The first fusion feature, the second fusion feature and the third fusion feature are then concatenated to obtain the target feature.

[0074] Optionally, will and After passing through the multi-scale frequency domain proxy attention module, the first fusion feature is obtained, including:

[0075] Will After Fourier transform and average pooling, the spectral energy vector is obtained. Specifically, it is expressed by the following formula:

[0076] ;

[0077] Using a perceptron MLP, the spectral energy vector is... The mapping is represented by a multi-scale dynamic agent vector A, specifically by the following formula:

[0078] ;

[0079] Will After passing through a linear layer, we obtain The key K a Sum V a Using the multi-scale dynamic agent vector A as the query, intermediate context features are calculated. Specifically, this is achieved through the following formula:

[0080] ;

[0081] Atten represents the attention mechanism;

[0082] Will The query vector is obtained after passing through the linear layer. The query vector, the multi-scale dynamic proxy vector A, and the intermediate context features Attention calculation is performed to obtain the first fusion feature Y, which is achieved through the following formula:

[0083] ;

[0084] Similarly, the first fusion feature and After passing through the multi-scale frequency domain proxy attention module, the second fusion feature is obtained, including:

[0085] according to Calculate the spectral energy vector and multi-scale dynamic surrogate vector, calculate the key and value of the first fusion feature, based on... The intermediate context features are calculated from the multi-scale dynamic proxy vector and the keys and values ​​of the first fusion feature, and then the second fusion feature is calculated.

[0086] Similarly, the second fusion feature and After passing through the multi-scale frequency domain proxy attention module, the third fusion feature is obtained, including:

[0087] according to Calculate the spectral energy vector and multi-scale dynamic surrogate vector, calculate the key and value of the second fusion feature, based on... The intermediate context features are calculated from the multi-scale dynamic proxy vector and the keys and values ​​of the second fusion feature, and then the third fusion feature is calculated.

[0088] The beneficial effects of adopting the above technical solution are as follows:

[0089] This invention designs a frequency-domain guided cue adapter to improve the consistency and propagation efficiency of cross-layer semantic features. Secondly, it proposes a spectrum-guided dynamic convolution module, which achieves efficient multi-scale feature modeling while fusing spatial and frequency domain features. Finally, it constructs a multi-scale frequency-domain proxy attention module to enhance semantic interaction and global modeling capabilities between features of different scales in a low-overhead manner. The method of this invention achieves excellent segmentation performance with a low parameter count on multiple RGB-D and RGB-L semantic segmentation datasets and demonstrates good generalization ability on five datasets in the RGB-D salient object detection task. The results show that, compared with related methods, the method of this invention provides more accurate segmentation results in structurally complex scenes, offering new ideas and practical basis for the fusion and lightweight design of multimodal perception networks. Attached Figure Description

[0090] Figure 1Figure 1 shows the visualization results of the RGB-D lightweight semantic segmentation method guided by the fusion frequency domain in this embodiment of the invention. Figure 2 shows the experimental results based on the NYU Depth V2 dataset, and Figure 3 shows the experimental results based on the SUN-RGBD dataset.

[0091] Figure 2 This is a schematic diagram of the structure of the RGB-D lightweight semantic segmentation method guided by the fusion frequency domain in an embodiment of the present invention;

[0092] Figure 3 This is a schematic diagram of the RGB-D block architecture in an embodiment of the present invention;

[0093] Figure 4 This is a schematic diagram of the frequency domain guidance prompt adapter architecture in an embodiment of the present invention;

[0094] Figure 5 This is a schematic diagram of the spectrum-guided dynamic convolution module in an embodiment of the present invention;

[0095] Figure 6 These are visualization comparison images on the NYU Depth V2 dataset in this embodiment of the invention. (a1), (a2), (a3), (a4), (a5), and (a6) represent the RGB image, depth image, semantic segmentation image of the DFormer network, semantic segmentation image of CMNext, semantic segmentation image of the present invention, and real semantic segmentation image of the first scene, respectively. (b1), (b2), (b3), (b4), (b5), and (b6) represent the RGB image, depth image, semantic segmentation image of the DFormer network, semantic segmentation image of CMNext, and semantic segmentation image of the present invention of the second scene, respectively. The real semantic segmentation maps (c1), (c2), (c3), (c4), (c5), and (c6) are respectively the RGB image, depth image, semantic segmentation map of DFormer network, semantic segmentation map of CMNext, semantic segmentation map of the present invention, and real semantic segmentation map of the third scene; (d1), (d2), (d3), (d4), (d5), and (d6) are respectively the RGB image, depth image, semantic segmentation map of DFormer network, semantic segmentation map of CMNext, semantic segmentation map of the present invention, and real semantic segmentation map of the fourth scene.

[0096] Figure 7These are visualization comparison images on the NJU2K dataset in this embodiment of the invention. (a1), (a2), (a3), (a4), (a5), (a6), and (a7) represent the RGB image, depth map, semantic segmentation map of the DFormer network, semantic segmentation map of HIDANet, semantic segmentation map of PICR-Net, semantic segmentation map of the present invention, and real semantic segmentation map of the first scene, respectively. (b1), (b2), (b3), (b4), (b5), (b6), and (b7) represent the RGB image, depth map, semantic segmentation map of the DFormer network, semantic segmentation map of HIDANet, semantic segmentation map of PICR-Net, and semantic segmentation map of the present invention, respectively. Semantic segmentation map, real semantic segmentation map, (c1), (c2), (c3), (c4), (c5), (c6), (c7) are respectively the RGB image, depth map, semantic segmentation map of DFormer network, semantic segmentation map of HIDANet, semantic segmentation map of PICR-Net, semantic segmentation map of the present invention, and real semantic segmentation map of the third scene; (d1), (d2), (d3), (d4), (d5), (d6), (d7) are respectively the RGB image, depth map, semantic segmentation map of DFormer network, semantic segmentation map of HIDANet, semantic segmentation map of PICR-Net, semantic segmentation map of the present invention, and real semantic segmentation map of the fourth scene.

[0097] Figure 8 For the visualization comparison of complex scenes in the embodiments of the present invention, (a1), (a2), (a3), (a4), and (a5) are respectively the RGB image, depth map, and true semantic segmentation map of the first complex scene (i.e., Figure 8 The true value in the image), the semantic segmentation map of HIDANet, and the present invention (i.e., Figure 8 The semantic segmentation maps of the present invention are as follows: (b1), (b2), (b3), (b4), and (b5) are the RGB image, depth map, real semantic segmentation map, HIDANet semantic segmentation map, and the semantic segmentation map of the present invention for the second complex scene, respectively; (c1), (c2), (c3), (c4), and (c5) are the RGB image, depth map, real semantic segmentation map, HIDANet semantic segmentation map, and the semantic segmentation map of the present invention for the third complex scene, respectively; and (d1), (d2), (d3), (d4), and (d5) are the RGB image, depth map, real semantic segmentation map, HIDANet semantic segmentation map, and the semantic segmentation map of the present invention for the fourth complex scene, respectively. Detailed Implementation

[0098] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0099] RGB-D semantic segmentation, by fusing complementary information from RGB and depth images, can effectively improve the accuracy of semantic understanding in complex scenes. However, due to the fundamental differences in feature representation between RGB and depth modalities, existing methods often rely on a dual-encoder architecture, resulting in low feature alignment efficiency and high computational overhead, making it difficult to achieve a good balance between model performance and lightweight design. To address this, we propose a frequency-domain guided RGB-D lightweight semantic segmentation method, aiming to improve cross-modal feature fusion performance and reduce model complexity.

[0100] First, an RGB-D Block integrating Transformer and CNN structures is constructed, employing a parallel design of global attention branches and local convolution branches to balance long-range dependency modeling and detail extraction. RGB-D salient object detection (SOD) aims to accurately identify salient regions in an image by jointly utilizing RGB image and depth map information. In recent years, numerous methods have been proposed to improve the detection performance of this task. For example, introducing multi-level guided attention mechanisms, combined with fusion and refinement modules, effectively enhances the collaborative representation ability of multimodal features. Although the above methods have made continuous breakthroughs in accuracy, most RGB-D SOD frameworks still suffer from problems such as large parameter scale and slow inference speed. To address this, this invention designs a simple and efficient combination of frequency domain perception modules, which can significantly reduce computational overhead while maintaining detection accuracy, achieving a better balance between accuracy and efficiency.

[0101] Secondly, a frequency-domain guided cue adapter is designed to extract spectral energy through Fourier transform to dynamically generate cue vectors, which are then fused with query features to enhance cross-layer semantic alignment. Thirdly, a spectrum-guided dynamic convolution module is designed, dividing features into low, mid, and high frequency bands. A gating mechanism is used to fuse multi-frequency features with spatial domain extracted features, improving multi-scale modeling capabilities. Finally, a multi-scale frequency-domain proxy attention module is constructed, utilizing the spectral features from the previous stage to generate dynamic proxy vectors, achieving efficient cross-layer global modeling at low cost.

[0102] Specifically, this invention proposes a lightweight RGB-D semantic segmentation method guided by the frequency domain. Starting from a frequency domain modeling perspective, it constructs an end-to-end multimodal fusion mechanism to achieve more efficient and robust semantic representation capabilities. To fully verify the effectiveness of the proposed method, experiments are conducted on two core tasks: RGB-D semantic segmentation and RGB-L semantic segmentation. Furthermore, to evaluate the model's cross-task generalization ability, extended testing is performed on the RGB-D salient object detection task to further verify its transferability and adaptability. The main contributions of this invention are as follows:

[0103] 1) A frequency-guided prompt adapter (FPA) is proposed, which extracts spectral energy by performing Fourier transform on the input features and guides the dynamic generation process of the prompt vector, thereby achieving a deep fusion of cross-layer semantic perception and prompt optimization.

[0104] 2) Design a spectrum-guided dynamic convolution (SDC) module to divide features into three frequency bands: low frequency, mid frequency and high frequency, to model global structure, edge information and detail texture respectively, and to fuse multi-frequency band and spatial domain features through a gating mechanism to effectively improve multi-scale modeling capabilities.

[0105] 3) Construct a multi-scale frequency-aware agent attention (MFAA) module, which dynamically generates agent vectors based on the spectral features of the previous stage, and constructs an efficient cross-layer attention path, which improves semantic consistency and context modeling capabilities while reducing computational costs.

[0106] like Figure 1 As shown, Figure (a) presents the experimental results based on the NYU Depth V2 dataset, and Figure (b) presents the experimental results based on the SUN-RGBD dataset. The method proposed in this invention achieves state-of-the-art performance on multiple benchmark datasets. In the figures, the vertical axis represents the mean intersection over union (mIoU), and the horizontal axis represents the number of model parameters. Figure 1 As shown, on the NYU Depth V2 and SUN-RGBD datasets for RGB-D semantic segmentation, the model achieves average intersection-union ratios (AUCs) of 57.6% and 52.8%, respectively, with less than half the number of parameters as mainstream methods. Experiments on the KITTI-360 dataset for RGB-L semantic segmentation further validate its robustness to other cross-modal scenarios. In generalization tests, the model outperforms existing methods across all four metrics (F-measure, E-measure, S-measure, and MAE) on five RGB-D salient object detection datasets. Furthermore, ablation experiments on NYU Depth V2 also validate the independent value and synergistic benefits of each module.

[0107] Combination Figure 2This is a schematic diagram of the RGB-D lightweight semantic segmentation method with frequency domain guidance provided by the present invention. The overall structure follows a typical encoder-decoder design, in which the encoder integrates Transformer and CNN modules, and performs joint feature extraction and fusion of RGB image and depth map through RGB-D Block; the multi-scale frequency domain proxy attention module performs feature optimization; and the decoder is used to restore the encoded features to pixel-level semantic prediction.

[0108] Specifically, acquire the RGB image and the depth image, and then convert the RGB image into an I... RGB and depth image I Depth The RGB features are obtained by inputting them into the embedding layer. and depth features This is to achieve initial feature extraction and dimensionality reduction. The embedding layer consists of a two-stage hybrid module: Stage 1 employs a multi-branch structure to enhance modality fusion capabilities, while Stage 2 improves the robustness of feature representation through residual enhancement, specifically implemented through the following formula:

[0109] ;

[0110] ;

[0111] Where I is the RGB image I RGB and depth image I Depth , For convolution operations with a kernel size of 1, This is a convolution operation with a kernel size of 3, BN is a batch normalization layer, Y is the intermediate feature, and Z is the RGB feature. or depth features GELU represents the Gaussian error linear unit activation function.

[0112] After processing by the embedding layer, RGB and depth features enter the hierarchical encoder, where they are encoded at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image. Each stage of the hierarchical encoder contains multiple RGB-D blocks, with RGB and depth features performing cross-modal global modeling and spectral guided convolutional feature extraction, respectively. Simultaneously, a frequency-domain guided cue adapter dynamically generates cue vectors based on Fourier spectral energy and fuses them with the backbone attention, thereby improving cross-layer semantic consistency. Furthermore, a multi-scale frequency-domain surrogate attention module introduced between the encoder and decoder generates surrogate vectors based on the spectral features from the previous stage, guiding the decoder to obtain more context-consistent fused features. Finally, the integrated features enter the decoder for multi-scale reconstruction, generating the final semantic prediction result.

[0113] To address the complementarity and inconsistency between RGB and depth modal features in feature representation, a structurally efficient RGB-D Block is designed in the encoder for multimodal feature fusion and calibration. Its structure is as follows: Figure 3 As shown, this module consists of two parallel sub-branches: a global attention branch and a CNN branch, which are responsible for cross-modal global modeling and local detail modeling, respectively.

[0114] The global attention branch employs a frequency-domain guided cue adapter, enhancing cross-layer information transfer through a cue-guided mechanism while simultaneously modeling long-range semantic dependencies using the global attention module. The CNN branch utilizes spectral-guided dynamic convolution, decomposing the input features into low-frequency, mid-frequency, and high-frequency channels. Information from each frequency band is extracted and then fused with spatial features through a gating mechanism to achieve multi-scale fine-grained modeling. The outputs of the two branches are concatenated along the channel dimension and transformed linearly to generate updated RGB and depth features, which are then input to the next stage.

[0115] Specifically, RGB features and depth features The input is a hierarchical encoder, which outputs RGB features and depth features from four stages. Each stage of the hierarchical encoder contains multiple RGB-D blocks. In specific implementation, the first stage (i.e., Figure 2 The number of RGB-D blocks N1 in Stage 1 is 3, and the second stage (i.e. Figure 2 The number of RGB-D blocks N2 in Stage 2 is 3, and the third stage (i.e. Figure 2 The number of RGB-D blocks N3 in Stage 3 is 12, and the fourth stage (i.e. Figure 2 The number of RGB-D blocks N4 in Stage 4 is 3.

[0116] Specifically, will and The first stage of the hierarchical encoder is input to obtain the first stage RGB features. and the first stage Depth features ,in, and The dimensions are H×W×C, where H is... and The height, W is and The width, C is and The number of channels, and The size is C1 represents and The number of channels.

[0117] Will and After passing through multiple RGB-D blocks in sequence, the result is... and Specifically, in the first RGB-D block, the input is... and The output of the first RGB-D block serves as the input of the second RGB-D block, the output of the second RGB-D block serves as the input of the third RGB-D block, and the output of the third RGB-D block is... and ;

[0118] Combination Figure 3 In the i-th RGB-D block, let the input representation of the RGB-D block be... and , and After attention branch (i.e.) Figure 2 Global attention in, and also Figure 3 (in the Transformer branch), to obtain attention features Specifically, based on the input RGB-D block and Generate query Q using the following formula:

[0119] ;

[0120] in, Indicates channel-dimensional splicing. Representation layer normalization, Indicates average pooling. Represents a linear transformation;

[0121] Based on the input RGB-D block Generate key K and value V, specifically through the following formula:

[0122] ;

[0123] Where GELU represents the Gaussian error linear unit activation function;

[0124] Calculate global attention based on query Q, key K, and value V. Specifically, this is achieved through the following formula:

[0125] ;

[0126] in, The query vector dimension is T, where T represents transpose and Softmax represents the activation function.

[0127] Furthermore, to enhance global information transmission capabilities, a learnable cue token is introduced and input together with the query vector into the frequency domain guided cue adapter (i.e., the frequency domain cue adapter), which takes the cue token of the i-th RGB-D block as input. And query the Q-input frequency domain cue adapter to obtain cue attention. ;

[0128] Specifically, in combination Figure 4 According to the prompt token Generate suggestion branch key and hint branch value Specifically, this is achieved through the following formula:

[0129] ;

[0130] in, Indicates the prompt token Reorganization This indicates that the operation segments the features along the spatial dimension;

[0131] Will The spectral energy is extracted using Fourier transform to generate a cue vector. Specifically, this is achieved through the following formula:

[0132] ;

[0133] in, denoted as Fourier transform, and MLP represents multilayer perceptron;

[0134] Based on the prompt vector Calculate attention prompts Specifically, this is achieved through the following formula:

[0135] ;

[0136] Where Softmax represents the activation function and T represents the transpose.

[0137] global attention and prompt attention By summing the features, we obtain the attention features. Specifically, it is expressed by the following formula:

[0138] ;

[0139] Will draw attention Reorganize to obtain the hint token for the (i+1)th RGB-D block. .

[0140] It should be noted that the hint token for the first RGB-D block in the first stage can be preset, and the hint token for each subsequent RGB-D block is calculated from the previous RGB-D block. The hint token between two RGB-D blocks is also the hint token calculated by the current RGB-D block using the previous RGB-D block.

[0141] and After convolutional branch (i.e.) Figure 2 Spectral dynamic convolution in [the context of] is also Figure 3 (CNN branch in the process) to obtain convolutional features Specifically, this is achieved through the following formula:

[0142] ;

[0143] ;

[0144] Based on the spectrum-guided dynamic convolution module, and Processing is performed to obtain convolutional features. Specifically, this is achieved through the following formula:

[0145] ;

[0146] in, This represents the intermediate features of Depth. Represents intermediate features of RGB. This represents a spectrum-guided dynamic convolution module. This represents the Hadamard product.

[0147] In multimodal perception tasks, local detail modeling and global structure perception are equally important. Therefore, a spectrum-guided dynamic convolution module is designed, such as... Figure 5 As shown, by jointly modeling spatial and frequency domain information, key information at different scales and frequencies can be fully captured, while controlling computational complexity and improving the local receptive field limitation of traditional spatial domain convolution operations, thereby enhancing the overall perception capability of downstream semantic segmentation tasks.

[0148] Specifically, the spectrum-guided dynamic convolution module comprises two parts: a spatial domain modeling branch and a frequency domain modeling branch. The spatial domain branch focuses on extracting local neighborhood information and modeling local context, while the frequency domain branch addresses global feature representation and multi-scale fusion of different frequency components. The two branches process input features independently, performing element-wise fusion at the output stage to balance local details with global structural features.

[0149] Specifically, in combination Figure 5 Let the input features of the spectrum-guided dynamic convolution module be represented as... , for or The spectrum-guided dynamic convolution module processes the input features. The processing includes:

[0150] Spatial branching adopts Large convolutional kernels directly model local details and mesoscale contextual relationships in the original spatial domain. By expanding the receptive field through larger kernels, the dependence on feature propagation from deep stacking is reduced, thus improving the ability to model local textures. Specifically, for input features... Convolutional and batch normalization layers are applied to obtain spatial domain features. Specifically, it is expressed by the following formula:

[0151] ;

[0152] in, For convolution operations with kernel size k, BN is a batch normalization layer;

[0153] The frequency domain branch extracts input features by applying a two-dimensional Fourier transform to the input features, mapping them from the spatial domain to the frequency domain. The spectral characteristics F of are specifically expressed by the following formula:

[0154] ;

[0155] In the frequency domain, different frequency components correspond to different structural information in an image. Low-frequency components mainly reflect the overall contour and global background information, mid-frequency components contain local region boundaries and coarse texture information, while high-frequency components are related to detailed textures, noise, and other high-frequency characteristics. Therefore, the spectral feature F is divided into low-frequency features. Mid-frequency characteristics and high frequency characteristics Specifically, it is expressed by the following formula:

[0156] ;

[0157] in, This indicates that the spectral feature F is divided into regions based on a set frequency threshold;

[0158] After completing the frequency band allocation, for each frequency band, apply directly to the frequency domain. Small convolutions are used to enhance local structure modulation capabilities. Low-frequency features are addressed separately. Mid-frequency characteristics and high frequency characteristics Perform convolution to obtain low-frequency convolution features. Mid-frequency convolution features and high-frequency convolution features Specifically, this is achieved through the following formula:

[0159] ;

[0160] ;

[0161] ;

[0162] Through the above processing, the network can capture local frequency variations and enhance the fine-grained representation of features. Unlike convolutions in the spatial domain, small convolutions in the frequency domain can effectively suppress interference from irrelevant frequency components and strengthen information in the frequency bands of interest. Specifically, low-frequency convolutional features... Mid-frequency convolution features and high-frequency convolution features By integrating the data, the fused spectral characteristics are obtained. Specifically, it is expressed by the following formula:

[0163] ;

[0164] fusion spectral features After inverse Fourier transform, the frequency domain features are obtained. Specifically, it is expressed by the following formula:

[0165] ;

[0166] The inverse transform ensures that the features processed in the frequency domain can be effectively aligned and fused with the features processed in the spatial domain in the same domain.

[0167] spatial domain features and frequency domain features Element-wise summation yields the final output of the spectrum-guided dynamic convolution module. Specifically, it is expressed by the following formula:

[0168] ;

[0169] in, for or .

[0170] By combining spatial and frequency domain features, the spectrum-guided dynamic convolution module not only retains the strong modeling ability of spatial convolution for local information, but also introduces multi-band frequency domain features to supplement the blind spots of traditional convolution in perceiving global and structural information. Without significantly increasing computational costs, it can effectively improve overall performance and robustness.

[0171] Attention characteristics After upsampling, it is combined with convolutional features By splicing, block aggregation features are obtained. Specifically, it is expressed by the following formula:

[0172] ;

[0173] Projecting the block aggregation features yields the output of the i-th RGB-D block. and ,Will and As the input to the (i+1)th RGB-D block, the output of the last RGB-D block in the first stage is: and ;

[0174] The processing procedures for the second, third, and fourth stages of the RGB-D block in the layered encoder are the same as those for the first stage, except that the objects being processed are different.

[0175] right and Perform downsampling, and then... and downsampling The second stage of the input layer encoder is used to obtain the second stage RGB features. Second-stage Depth features Specifically, in the first RGB-D block, the input is the downsampled... and downsampling The output of the first RGB-D block serves as the input of the second RGB-D block, the output of the second RGB-D block serves as the input of the third RGB-D block, and the output of the third RGB-D block is... and .

[0176] in, and The size is C2 represents and The number of channels.

[0177] right and Perform downsampling, and then... and downsampling Input the third stage of the hierarchical encoder to obtain the third stage RGB features. and the third stage Depth features Similarly, in the first RGB-D block, the input is the downsampled... and downsampling The output of the previous RGB-D block is used as the input of the current RGB-D block, and the output of the 12th RGB-D block is... and .

[0178] in, and The size is C3 indicates and The number of channels.

[0179] right and Perform downsampling, and then... and downsampling Input the fourth stage of the hierarchical encoder to obtain the fourth stage RGB features. and the fourth stage Depth features Similarly, in the first RGB-D block, the input is the downsampled... and downsampling The output of the first RGB-D block serves as the input of the second RGB-D block, the output of the second RGB-D block serves as the input of the third RGB-D block, and the output of the third RGB-D block is... and .

[0180] Feature extraction from a hierarchical encoder transmits each layer's features to the decoder. During deep feature modeling, features at different scales exhibit significant differences in spatial resolution and semantic granularity. To address this, a multi-scale frequency-domain proxy attention module is proposed. This module dynamically generates proxy vectors through a frequency-domain guidance mechanism, enabling efficient global information interaction and contextual modeling between features at different scales.

[0181] Specifically, the RGB features output from the four stages are processed through multi-scale frequency domain proxy and then concatenated to obtain the target features. and The first fused feature is obtained after passing through a multi-scale frequency domain proxy attention module.

[0182] The multi-scale frequency domain proxy attention module first performs spectral analysis using the RGB features from the previous stage. It then maps the spatial features to the frequency domain using a two-dimensional Fourier transform, extracting the global energy distribution. In the spectral space, different frequency components correspond to image structural information at different scales. To extract a generalized representation of the overall features, spatial mean pooling is further performed on the spectral amplitudes to obtain the spectral energy vector. Specifically, the module will... After Fourier transform and average pooling, the spectral energy vector is obtained. Specifically, it is expressed by the following formula:

[0183] ;

[0184] Spectral energy vectors can effectively encode the importance distribution of features at a higher scale in the frequency domain. A perceptron MLP is used to map spectral energy into a set of multi-scale dynamic surrogate vectors, dynamically generating surrogates through frequency domain information, thus improving the perception and adaptation capabilities between cross-scale features. Specifically, a perceptron MLP is used to map spectral energy vectors... The mapping is represented by a multi-scale dynamic agent vector A, specifically by the following formula:

[0185] ;

[0186] Will After passing through a linear layer, we obtain The key K a Sum V a Using the multi-scale dynamic agent vector A as the query, intermediate context features are calculated. Specifically, this is achieved through the following formula:

[0187] ;

[0188] Atten represents the attention mechanism;

[0189] Will The query vector is obtained after passing through the linear layer. The query vector, the multi-scale dynamic proxy vector A, and the intermediate context features Attention calculation is performed to obtain the first fusion feature Y, which is achieved through the following formula:

[0190] ;

[0191] In summary, by introducing a frequency-domain information-driven proxy mechanism, the MFAA module can improve deep semantic consistency modeling and context feature completion capabilities with extremely low parameter and computational overhead. As a feature bridging path between the encoder and decoder, this module not only enhances global modeling capabilities but also possesses good pluggability, making it widely applicable in various segmentation networks.

[0192] The first fusion feature and After passing through the multi-scale frequency domain proxy attention module, the second fused feature is obtained. Similarly, according to... Calculate the spectral energy vector and multi-scale dynamic surrogate vector, calculate the key and value of the first fusion feature, based on... The intermediate context features are calculated from the multi-scale dynamic proxy vector and the keys and values ​​of the first fusion feature, and then the second fusion feature is calculated.

[0193] The second fusion feature and After passing through the multi-scale frequency domain proxy attention module, the third fusion feature is obtained. Similarly, according to... Calculate the spectral energy vector and multi-scale dynamic surrogate vector, calculate the key and value of the second fusion feature, based on... The intermediate context features are calculated from the multi-scale dynamic proxy vector and the keys and values ​​of the second fusion feature, and then the third fusion feature is calculated.

[0194] The first, second, and third fused features are concatenated to obtain the target features. The target features are then processed through a lightweight decoder head and an MLP to obtain a semantic segmentation prediction map.

[0195] It should be noted that the method provided by this invention can achieve different results with different datasets. Specifically, it can achieve semantic segmentation and salient object detection, such as... Figure 2 The I on the right side of Seg Pred For the semantic segmentation prediction map, the I on the right side of SOD Pred This is the prediction map for salient target detection, also known as the saliency map.

[0196] For RGB-D tasks, RGB modalities tend to have higher energy distribution in high-frequency regions, exhibiting rich texture and edge information; while depth maps are concentrated in low-frequency bands, reflecting geometric structure and spatial continuity. This fixed frequency domain complementarity provides clear guiding cues for fusion modeling. The frequency domain cueing mechanism proposed in this invention constructs semantic consistency cues through spectral transformation, while the spectral dynamic convolution module achieves group perception and scale control in different frequency bands, thereby avoiding the aliasing problem of modal coupling features caused by spatial convolution. Compared to direct interaction in the spatial domain, the frequency domain guidance mechanism can improve cross-modal alignment efficiency without increasing too many parameters, and has stronger stability and robustness.

[0197] To comprehensively evaluate the performance of this invention in different RGB-D tasks, various mainstream evaluation metrics were used to evaluate the performance of different networks for semantic segmentation and salient object detection tasks.

[0198] For semantic segmentation tasks, the mean intersection over union (mIoU) is primarily used as a performance evaluation metric. mIoU is the ratio of the area of ​​intersection to the area of ​​union between the predicted segmented region and the ground truth labeled region, used to measure the model's segmentation accuracy across different categories. For salient object detection tasks, to comprehensively evaluate the consistency between the predicted salient map and the ground truth label from multiple perspectives, four mainstream metrics are used: harmonic mean (F-measure), enhancement evaluation metric (E-measure), structural similarity metric (S-measure), and mean absolute error (MAE) to display comprehensive results. The F-measure measures the consistency between the detection results and the ground truth label by combining accuracy and recall, emphasizing the model's overall accuracy in salient region extraction. The E-measure integrates region hierarchy and pixel-level alignment characteristics, taking into account both overall structure and local details, reflecting the visual consistency between the salient map and the ground truth mask. The S-measure evaluates the structural similarity between the predicted and ground truth maps from both object perception and region perception perspectives, reflecting the model's ability to model the integrity and layout consistency of salient objects. MAE directly calculates the pixel-level average error between the predicted saliency map and the real mask, quantifying the overall prediction bias. The smaller the value, the closer the prediction result is to the real label.

[0199] In the RGB-D semantic segmentation task, two representative RGB-D indoor scene semantic segmentation datasets, NYU Depth V2 and SUN-RGBD, are used. NYU Depth V2 contains 1449 pairs of RGB-D sample images with a resolution of 640×480, including 795 training images and 654 test images, with 40 semantic categories. SUN-RGBD contains 10335 RGB-D images, with 5285 images used as the training set and 5050 images used as the test set, containing 37 semantic categories, with the input size adjusted to 480×480.

[0200] In the RGB-L semantic segmentation task, the KITTI-360 dataset was used for evaluation. KITTI-360 is a suburban urban driving scene dataset containing 49,004 training images and 12,276 validation images, with an image resolution of 1408×376, covering 19 semantic categories. In the RGB-D salient object detection task, five mainstream datasets—NJU2K, NLPR, SIP, STERE, and DES—were selected for evaluation. NJU2K contains 1985 RGB-D images; NLPR consists of 1000 pairs of RGB-D images acquired by Kinect, with a resolution of 640×480, covering various indoor and outdoor scenes; the SIP dataset contains 929 images with human pose changes, with a resolution of 992×774; STERE contains 1000 RGB-D images; and the DES dataset is relatively small, containing only 135 indoor scene images. Following the mainstream setup, 700 samples from NLPR-train and 1485 samples from NJU2K-train were used as the training set, and tests were conducted uniformly on five datasets: NJU2K-test with 500 images, NLPR-test with 300 images, and SIP, STERE, and DES with 929, 1000, and 135 images, respectively.

[0201] All experiments were conducted on a single NVIDIA 3090 GPU using PyTorch 2.1.2 as the deep learning framework. During training, random horizontal flipping and random scaling were employed as data augmentation strategies to enhance the model's generalization ability. For the global attention branch of the RGB-D module, the GAA module from the DFormer network pre-trained on ImageNet was used for initialization, and fine-tuning was performed on this basis. For the network decoder, the lightweight Hamburger module was used as the basic structure, referencing the SegNeXt design. For the optimizer, AdamW was chosen, with a weight decay coefficient of 0.01 and a multinomial decay learning rate. The standard cross-entropy loss function was used to supervise pixel-level semantic prediction. In the RGB-D semantic segmentation task, the NYU Depth V2 dataset was trained for 400 epochs with a batch size of 8 and an initial learning rate of 6e-5. The SUN RGB-D dataset was trained for 300 epochs with a batch size of 16 and an initial learning rate of 8e-5. Referring to other recent related methods, a multi-scale flip inference strategy was adopted in the evaluation phase, with scaling factors set to [0.5, 0.75, 1, 1.25, 1.5] to improve prediction stability and accuracy. In the RGB-D salient object detection task, the training epochs were set to 200, the batch size to 16, and the initial learning rate to 2e-4. The learning rate was updated at a decay rate of 0.1 after every 60 epochs to ensure stable convergence during training.

[0202] For the experimental comparison and analysis of RGB-D semantic segmentation, this invention systematically compared the proposed network method with 16 current state-of-the-art methods on two mainstream RGB-D indoor semantic segmentation datasets: NYU Depth V2 and SUN RGB-D. The results are shown in Table 1. The model of this invention achieves the best overall performance while maintaining a low number of parameters.

[0203] Table 1. Quantitative Comparison Experiments of RGB-D Semantic Segmentation Tasks

[0204]

[0205] In Table 1, bold and underlined text represent the best and second-best results for each column, respectively; "—" indicates that no corresponding experiment was conducted; and "†" indicates that the results of DFormer were referenced.

[0206] On the NYU Depth V2 dataset, leveraging a multi-scale inference strategy, the proposed Large model achieves a 57.6% mIoU with 40.8M training parameters, significantly outperforming lightweight networks of similar size. Compared to the representative method CMX, this invention improves the mIoU metric by approximately 0.7% while reducing training parameters by approximately 77%, fully validating the advantage of maintaining expressive power while compressing model size. Furthermore, the Small version model achieves a 54.6% mIoU with only 20.3M training parameters, with training overhead only half that of most mainstream methods, further validating the applicability of the proposed spectrum-guided dynamic convolution and frequency-domain proxy attention mechanism in lightweight deployment. Experimental results on the SUN-RGBD dataset also demonstrate the strong cross-dataset generalization ability of the proposed method. In particular, compared to the recent state-of-the-art method Sigma, this invention achieves higher segmentation performance with less than 40% of its parameter count, reaching a 52.8% mIoU. These results further validate the effectiveness of the frequency domain cueing tuning mechanism and the cross-layer semantic alignment strategy in modeling complementary information between RGB and deep modalities. Overall, the network of this invention exhibits high semantic modeling capability and stable cross-modal fusion performance while maintaining low computational overhead and parameter count, achieving a good trade-off between performance and efficiency.

[0207] To more intuitively evaluate the performance of different RGB-D semantic segmentation methods, Figure 6 A visual comparison of segmentation results on the NYU Depth V2 dataset is provided. Figure 6In the image, (a1), (a2), (a3), (a4), (a5), and (a6) represent the RGB image, depth image, semantic segmentation image of the DFormer network, semantic segmentation image of CMNext, and the semantic segmentation image of the present invention (i.e., the image of the present invention). Figure 6 The semantic segmentation graph of the text, the real semantic segmentation graph (i.e., the text) Figure 6 (b1), (b2), (b3), (b4), (b5), and (b6) represent the RGB image, depth image, semantic segmentation image of the DFormer network, semantic segmentation image of CMNext, semantic segmentation image of the present invention, and real semantic segmentation image of the second scene, respectively. (c1), (c2), (c3), (c4), (c5), and (c6) represent the RGB image, depth image, semantic segmentation image of the DFormer network, semantic segmentation image of CMNext, semantic segmentation image of the present invention, and real semantic segmentation image of the third scene, respectively. (d1), (d2), (d3), (d4), (d5), and (d6) represent the RGB image, depth image, semantic segmentation image of the DFormer network, semantic segmentation image of CMNext, semantic segmentation image of the present invention, and real semantic segmentation image of the fourth scene, respectively.

[0208] Experimental results show that the method of this invention exhibits significant advantages in boundary perception, small target recognition, and complex material understanding. In the first column of the scene, other methods exhibit blurring in the boundary area between the whiteboard and the chair, easily leading to category confusion and often misclassifying the whiteboard area as surrounding furniture or wall structures; the method of this invention can accurately identify the boundary between the two, significantly enhancing the perception ability of edge structures. In the second column of the samples, the scene contains targets such as beds, floors, and pillows. Due to the complex texture of the bed sheet surface, some methods have significant missed detections and label confusion when distinguishing between beds and pillows. The frequency domain and spatial domain feature fusion strategy proposed in this invention effectively enhances the model's ability to model structural boundaries and detailed textures, accurately preserving the complete outline of the target object and achieving more refined pixel-level segmentation. In the third column of the kitchen scene, there are small targets such as paper attached to the surface of the refrigerator. Most methods easily misidentify them as background or surrounding objects, resulting in a loss of detail; in contrast, the network of this invention, through cross-modal cueing optimization and multi-scale feature enhancement mechanisms, successfully separates the boundary information between the paper and the refrigerator, demonstrating that the method of this invention has a high perception ability for low-contrast small targets. Furthermore, in living room scenarios with factors such as mirror reflection and complex floor materials, other methods often face problems of semantic inconsistency or texture disturbance. However, the method of this invention, with its frequency domain-guided semantic alignment mechanism and context modeling capabilities, can more accurately distinguish the semantic boundaries between reflective tabletops and surrounding objects, maintaining the coherence and integrity of details in each area.

[0209] To further verify the adaptability and generalization ability of the proposed method under different modal combinations, this invention conducted RGB-L semantic segmentation experiments on the KITTI-360 dataset and systematically compared it with current mainstream methods. The results are shown in Table 2. It can be seen that although the proposed method was initially designed for RGB-D scenes, it still exhibits excellent performance and outstanding generalization ability in RGB-L scenes. The proposed model, with a mIoU of 66.3%, surpasses recent representative methods such as DFormer (66.1%) while maintaining a lightweight parameter scale. Thanks to the intervention of the frequency domain guidance mechanism, the distribution offset problem between the RGB image and the LiDAR depth map in feature representation is effectively alleviated, improving the robustness of cross-modal feature fusion and demonstrating good task transfer adaptability. This further verifies the stable modeling ability of the frequency domain perception mechanism in modal change environments.

[0210] Table 2 Experimental results of RGB-L segmentation

[0211]

[0212] To fully verify the generalization ability of the network of this invention, it was compared with a number of representative state-of-the-art methods on five mainstream RGB-D salient object detection datasets: NJU2K, NLPR, SIP, STERE, and DES. These methods include DCF, CMINet, DCMF, SSLSOD, CIRNet, CAVER, MMRNet, PICR-Net, HIDANet, AirSOD, GTransNet, and MambaSOD. The relevant quantitative comparison results are shown in Table 3.

[0213] Table 3 Quantitative Comparison Experiment of RGB-D SOD Task

[0214]

[0215] Overall, the frequency-domain guided lightweight sensing network proposed in this invention achieves state-of-the-art performance across all datasets on all four evaluation metrics (F-measure, E-measure, S-measure, and MAE). Specifically, the Large model presented in this invention outperforms or is equivalent to the existing state-of-the-art methods on all datasets, as illustrated by the visualization comparison. Figure 7As shown, (a1), (a2), (a3), (a4), (a5), (a6), and (a7) represent the RGB image, depth map, semantic segmentation map of the DFormer network, semantic segmentation map of HIDANet, semantic segmentation map of PICR-Net, semantic segmentation map of the present invention, and real semantic segmentation map of the first scene, respectively. (b1), (b2), (b3), (b4), (b5), (b6), and (b7) represent the RGB image, depth map, semantic segmentation map of the DFormer network, semantic segmentation map of HIDANet, semantic segmentation map of PICR-Net, semantic segmentation map of the present invention, and real semantic segmentation map of the second scene, respectively. The cut images (c1), (c2), (c3), (c4), (c5), (c6), and (c7) are respectively the RGB image, depth image, semantic segmentation image of DFormer network, semantic segmentation image of HIDANet, semantic segmentation image of PICR-Net, semantic segmentation image of the present invention, and real semantic segmentation image of the third scene. (d1), (d2), (d3), (d4), (d5), (d6), and (d7) are respectively the RGB image, depth image, semantic segmentation image of DFormer network, semantic segmentation image of HIDANet, semantic segmentation image of PICR-Net, semantic segmentation image of the present invention, and real semantic segmentation image of the fourth scene.

[0216] In the NJU2K dataset, the network of this invention achieves a MAE of 0.024, outperforming other methods, and further improves on the F-measure and S-measure metrics, indicating that the network of this invention has a stronger ability to identify target regions. In the NLPR dataset, the network of this invention also demonstrates leading performance, with F-measure reaching 0.941, E-measure reaching 0.967, S-measure reaching 0.941, and MAE only 0.017, all surpassing existing methods. Especially in background suppression and boundary preservation, the advantages of the method of this invention are more obvious, reflecting the effective promotion of fine-grained feature capture by frequency domain modeling. In the STERE dataset experiments, the network of this invention achieves excellent saliency detection results through deep feature optimization and frequency domain guided prompting tuning mechanism, with an F-measure of 0.932, which is better than the comparison methods, and also has certain advantages in S-measure and MAE.

[0217] On the relatively small DES dataset, the network of this invention also performed stably, outperforming all other methods in three of the four metrics, fully demonstrating that the network maintains high robustness even with a limited number of samples. On the SIP dataset, the network of this invention achieved F-measure of 0.934, E-measure of 0.943, S-measure of 0.910, and MAE reduced to 0.034, all improvements over methods such as MambaSOD, further validating the network's excellent cross-scene generalization ability. In summary, thanks to the synergistic optimization of frequency domain guided cueing tuning, spectrum-guided dynamic convolution, and multi-scale frequency domain proxy attention mechanism, the method of this invention achieves a balance between accuracy and efficiency in the RGB-D salient object detection task, demonstrating high performance advantages and broad adaptability.

[0218] Figure 8 Four sets of representative complex scene samples are presented. Among them, (a1), (a2), (a3), (a4), and (a5) are the RGB image, depth map, and ground truth semantic segmentation map of the first complex scene, respectively. Figure 8 The true value in the image), the semantic segmentation map of HIDANet, and the present invention (i.e., Figure 8 The semantic segmentation maps of the present invention are as follows: (b1), (b2), (b3), (b4), and (b5) are the RGB image, depth map, real semantic segmentation map, HIDANet semantic segmentation map, and the semantic segmentation map of the present invention for the second complex scene, respectively; (c1), (c2), (c3), (c4), and (c5) are the RGB image, depth map, real semantic segmentation map, HIDANet semantic segmentation map, and the semantic segmentation map of the present invention for the third complex scene, respectively; and (d1), (d2), (d3), (d4), and (d5) are the RGB image, depth map, real semantic segmentation map, HIDANet semantic segmentation map, and the semantic segmentation map of the present invention for the fourth complex scene, respectively.

[0219] Figure 8 Covering challenging scenarios such as blurred boundaries, occlusion interference, and complex backgrounds, the proposed method demonstrates its superior performance in RGB-D salient object detection. For example, in the second group of horse images, severe fence occlusion significantly impacted the coherence of the foreground target, leading to large-scale misclassification by HIDANet. In contrast, the proposed model suppressed occlusion interference through frequency domain modeling, better preserving the target shape. In the third group of fighter jet samples, due to the similar brightness between the tail fin and the sky, HIDANet segmentation regions exhibited adhesion and contour shift. The proposed method, however, effectively extracted structural edge information through a spectrum-guided mechanism, achieving more refined semantic segmentation. Overall, the proposed method demonstrates superior boundary preservation and detail representation capabilities across multiple complex scenarios.

[0220] As shown in Table 4, to comprehensively evaluate the performance advantages of the proposed network in terms of lightweight design and inference efficiency, several lightweight RGB-D semantic segmentation models with FPS exceeding 30 were selected for comparison, covering the number of trainable parameters, the number of inference parameters, and the inference speed (FPS) on actual devices. PGDENet's design incorporates a progressive guidance strategy and a geometric enhancement module, emphasizing the utilization of deep structural information during the fusion process; however, the module stacking is complex, resulting in high decoder computational overhead. AsymFormer achieves lightweight modeling through an asymmetric backbone structure, allocating modality processing tasks through CNN and Transformer, thus limiting its fusion capabilities. In contrast, the proposed method employs a unified fusion backbone, combining frequency domain cueing, spectral convolution, and proxy attention mechanisms, effectively improving feature alignment and multi-scale semantic modeling capabilities. All experiments were conducted on a single NVIDIA 3090 GPU. The results show that the proposed method can achieve an inference speed of 46 FPS while maintaining a low number of inference parameters, demonstrating its good applicability in resource-constrained application scenarios.

[0221] Table 4 Model Parameters and FPS Analysis

[0222]

[0223] This invention proposes a frequency-domain guided lightweight RGB-D perceptual network, aiming to improve the efficiency of multimodal feature alignment and semantic modeling, and address the issues of parameter redundancy and insufficient fusion capabilities in existing methods. First, a frequency-domain guided cue adapter is designed to improve the consistency and propagation efficiency of cross-layer semantic features. Second, a spectrum-guided dynamic convolution module is proposed, achieving efficient multi-scale feature modeling while fusing spatial and frequency domain features. Finally, a multi-scale frequency-domain proxy attention module is constructed to enhance semantic interaction and global modeling capabilities between features of different scales in a low-overhead manner. The proposed method achieves excellent segmentation performance with a low parameter count on multiple RGB-D and RGB-L semantic segmentation datasets, and demonstrates good generalization ability on five datasets in the RGB-D salient object detection task. The results show that, compared with related methods, the proposed method achieves more accurate segmentation results in structurally complex scenes, providing new ideas and practical basis for the fusion and lightweight design of multimodal perceptual networks.

[0224] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A lightweight semantic segmentation method for RGB-D guided by frequency domain integration, characterized in that, include: Obtain RGB and depth images, and convert the RGB image... I RGB and depth images I Depth The RGB features are obtained by inputting them into the embedding layer. and depth features ; RGB features and depth features Input the hierarchical encoder to obtain the RGB features and depth features output by the four stages of the hierarchical encoder. Each stage of the hierarchical encoder contains multiple RGB-D blocks. Among them, RGB features and depth features Input the hierarchical encoder to obtain the RGB features and depth features output from the four stages of the hierarchical encoder, including: Will and The first stage of the hierarchical encoder is input to obtain the first stage RGB features. and the first stage Depth features ,right and Perform downsampling, and then... and downsampling The second stage of the input layer encoder is used to obtain the second stage RGB features. Second-stage Depth features ,right and Perform downsampling, and then... and downsampling Input the third stage of the hierarchical encoder to obtain the third stage RGB features. and the third stage Depth features ,right and Perform downsampling, and then... and downsampling Input the fourth stage of the hierarchical encoder to obtain the fourth stage RGB features. and the fourth stage Depth features ; Among them, and The first stage of the hierarchical encoder is input to obtain the first stage RGB features. and the first stage Depth features ,include: Will and After passing through multiple RGB-D blocks in sequence, the result is... and ; in the i In a given RGB-D block, let the input representation of the RGB-D block be... and , and After attention branching, attention features are obtained. , and After convolutional branching, convolutional features are obtained. Attention features After upsampling, it is combined with convolutional features By splicing, block aggregation features are obtained. Specifically, it is expressed by the following formula: ； Projecting the block aggregation features yields the first... i Output of RGB-D blocks and ,Will and As the first i+ The input is one RGB-D block, and the output of the last RGB-D block in the first stage is... and ; The processing procedures for the second, third, and fourth stages of the RGB-D block in the layered encoder are the same as those for the first stage of the RGB-D block, except that the objects being processed are different. The RGB features output from the four stages are processed through a multi-scale frequency domain proxy and then concatenated to obtain the target features. The target features are then processed through a lightweight decoder head and an MLP to obtain a semantic segmentation prediction map.

2. The RGB-D lightweight semantic segmentation method guided by frequency domain integration according to claim 1, characterized in that, RGB image I RGB and depth images I Depth The RGB features are obtained by inputting them into the embedding layer. and depth features Specifically, this is achieved through the following formula: ； ； in, I RGB image I RGB and depth images I Depth , For convolution operations with a kernel size of 1, For convolution operations with a kernel size of 3, BN For batch normalization layer, Y As an intermediate feature, Z RGB features or depth features , GELU This represents the Gaussian error linear unit activation function.

3. The RGB-D lightweight semantic segmentation method guided by frequency domain integration according to claim 1, characterized in that, and After attention branching, attention features are obtained. ,include: Based on the input RGB-D block and Generate query Q Specifically, this is achieved through the following formula: ； in, Indicates channel-dimensional splicing. Representation layer normalization, Indicates average pooling. Represents a linear transformation; Based on the input RGB-D block Generate key K Sum V Specifically, this is achieved through the following formula: ； in, GELU Represents the activation function of the Gaussian error linear unit; According to the query Q ,key K Sum V Calculate global attention Specifically, this is achieved through the following formula: ； in, For query vector dimensions, T Indicates transpose. Softmax Indicates the activation function; The first i A prompt token for an RGB-D block And query the Q-input frequency domain cue adapter to obtain cue attention. ; global attention and prompt attention By summing the features, we obtain the attention features. Specifically, it is expressed by the following formula: ； Will draw attention Reorganization was carried out to obtain the first i +1 RGB-D block hint token .

4. The RGB-D lightweight semantic segmentation method guided by frequency domain integration according to claim 3, characterized in that, The first i A prompt token for an RGB-D block And query the Q-input frequency domain cue adapter to obtain cue attention. ,include: According to the prompt token Generate suggestion branch key and hint branch value Specifically, this is achieved through the following formula: ； in, Indicates the prompt token Reorganization This indicates that the operation segments the features along the spatial dimension; Will The spectral energy is extracted using Fourier transform to generate a cue vector. Specifically, this is achieved through the following formula: ； in, Indicates Fourier transform, MLP This represents a multilayer perceptron; Based on the prompt vector Calculate attention prompts Specifically, this is achieved through the following formula: ； in, Softmax This represents the activation function. T This indicates transpose.

5. The RGB-D lightweight semantic segmentation method guided by frequency domain integration according to claim 1, characterized in that, and After convolutional branching, convolutional features are obtained. Specifically, this is achieved through the following formula: ； ； Based on the spectrum-guided dynamic convolution module, and Processing is performed to obtain convolutional features. Specifically, this is achieved through the following formula: ； in, This represents the intermediate features of Depth. Represents intermediate features of RGB. This represents a spectrum-guided dynamic convolution module. This represents the Hadamard product.

6. The RGB-D lightweight semantic segmentation method guided by frequency domain integration according to claim 5, characterized in that, Let the input features of the spectrum-guided dynamic convolution module be represented as follows: , for or The spectrum-guided dynamic convolution module processes the input features. The processing includes: For input features Convolutional and batch normalization layers are applied to obtain spatial domain features. Specifically, it is expressed by the following formula: ； in, For the kernel size is k Convolution operation, BN For batch normalization layer; Extracting input features spectral characteristics F Specifically, it is expressed by the following formula: ； spectral characteristics F Classified as low-frequency features Mid-frequency characteristics and high frequency characteristics Specifically, it is expressed by the following formula: ； in, This indicates the spectral characteristics based on a set frequency threshold. F Divide the area into regions; Low-frequency features Mid-frequency characteristics and high frequency characteristics Perform convolution to obtain low-frequency convolution features. Mid-frequency convolution features and high-frequency convolution features Specifically, this is achieved through the following formula: ； ； ； Low-frequency convolution features Mid-frequency convolution features and high-frequency convolution features By integrating the data, the fused spectral characteristics are obtained. Specifically, it is expressed by the following formula: ； fusion spectral features After inverse Fourier transform, the frequency domain features are obtained. Specifically, it is expressed by the following formula: ； spatial domain features and frequency domain features Element-wise summation yields the final output of the spectrum-guided dynamic convolution module. Specifically, it is expressed by the following formula: ； in, for or .

7. The RGB-D lightweight semantic segmentation method guided by frequency domain integration according to claim 1, characterized in that, The RGB features output from the four stages are processed through multi-scale frequency domain proxy and then concatenated to obtain the target features, including: Will and After passing through the multi-scale frequency domain proxy attention module, the first fused feature is obtained, and the first fused feature is combined with... After passing through the multi-scale frequency domain proxy attention module, the second fusion feature is obtained, and the second fusion feature is combined with... After passing through the multi-scale frequency domain proxy attention module, the third fusion feature is obtained. The first fusion feature, the second fusion feature and the third fusion feature are then concatenated to obtain the target feature.

8. The RGB-D lightweight semantic segmentation method guided by frequency domain integration according to claim 7, characterized in that, Will and After passing through the multi-scale frequency domain proxy attention module, the first fusion feature is obtained, including: Will After Fourier transform and average pooling, the spectral energy vector is obtained. Specifically, it is expressed by the following formula: ； in, Indicates average pooling. Indicates Fourier transform; Using a multilayer perceptron (MLP), the spectral energy vector is... Mapped to multi-scale dynamic agent vectors A Specifically, it is expressed by the following formula: ； in, MLP This represents a multilayer perceptron; Will After passing through a linear layer, we obtain The key K a Sum V a Multi-scale dynamic proxy vectors A As a query, calculate intermediate context features. Specifically, this is achieved through the following formula: ； in, Atten This represents the attention mechanism; Will The query vector is obtained after passing through the linear layer. Query vectors, multi-scale dynamic proxy vectors A and intermediate context features Attention calculations are performed to obtain the first fusion feature. Y Specifically, this is achieved through the following formula: ； Similarly, the first fusion feature and After passing through the multi-scale frequency domain proxy attention module, the second fusion feature is obtained, including: according to Calculate the spectral energy vector and multi-scale dynamic surrogate vector, calculate the key and value of the first fusion feature, based on... The intermediate context features are calculated from the multi-scale dynamic proxy vector and the keys and values ​​of the first fusion feature, and then the second fusion feature is calculated. Similarly, the second fusion feature and After passing through the multi-scale frequency domain proxy attention module, the third fusion feature is obtained, including: according to Calculate the spectral energy vector and multi-scale dynamic surrogate vector, calculate the key and value of the second fusion feature, based on... The intermediate context features are calculated from the multi-scale dynamic proxy vector and the keys and values ​​of the second fusion feature, and then the third fusion feature is calculated.

Images