A structure adaptive decoding method for DFormerv2 encoder
By designing a multi-branch adaptive refining and feature enhancement module that adapts to DFormerv2 encoders of different sizes, the problem of feature distribution differences caused by a fixed decoder structure is solved, thereby improving the performance and efficiency of RGB-D semantic segmentation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2026-03-16
- Publication Date
- 2026-06-05
AI Technical Summary
The existing DFormerv2 encoder suffers from performance bottlenecks due to the fixed structure of the decoder, which is difficult to adapt to differences in feature distribution under different scale configurations. This affects the efficiency and accuracy of RGB-D semantic segmentation.
A structure-adaptive decoding method is adopted, and a configurable decoder structure is designed through a multi-branch adaptive refinement module and a feature enhancement module to adapt to the feature distribution differences of encoders of different sizes, thereby achieving feature fusion and semantic recovery.
It improves the overall performance and computational efficiency of RGB-D semantic segmentation, alleviates the structural mismatch problem between encoders of different sizes, and improves the accuracy and speed of segmentation results.
Smart Images
Figure CN122156635A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision and pattern recognition technology, specifically to a structure-adaptive decoding method for DFormerv2 encoders. Background Technology
[0002] Semantic segmentation, a fundamental task in computer vision, aims to assign semantic labels to each pixel in an input image and is widely used in fields such as robot vision and indoor scene understanding. In recent years, with the development of deep learning technology, the encoder-decoder structure has become the mainstream framework for semantic segmentation models. In this structure, the encoder is responsible for extracting multi-scale semantic features, while the decoder fuses and spatially reconstructs high-level semantics.
[0003] In RGB-D semantic segmentation tasks, encoders typically enhance their understanding of geometric structure and semantic context by jointly modeling color images and depth information. With the introduction of the Transformer architecture and its variants into multimodal modeling, RGB-D encoders, represented by DFormerv2, have made significant progress in global modeling and geometrically perceptive feature fusion. However, in Transformer-type RGB-D encoding frameworks like DFormerv2, existing research largely focuses on encoder structure design, while the decoding stage often uses a uniform and fixed lightweight decoding structure, failing to fully consider the feature differences arising from the same encoder at different scale configurations. In practical applications, the same RGB-D encoder system (such as DFormerv2) often has multiple scale configurations; for example, basic and high-capacity encoders differ significantly in feature levels, semantic density, and redundancy. If the decoder still uses a fixed structure to treat encoder outputs of different scales, it can easily lead to insufficient feature refinement in lightweight encoders or ineffective control of redundant features in high-capacity encoders, thus creating new performance bottlenecks.
[0004] Therefore, how to design a decoding method with structural adaptability to address the differences in feature distribution presented by the same RGB-D encoder under different scale configurations, so as to achieve more reasonable feature fusion and semantic recovery, has become a key problem that urgently needs to be solved in the RGB-D semantic segmentation model based on DFormerv2. Summary of the Invention
[0005] This invention provides a structure-adaptive decoding method for DFormerv2 encoders, which addresses the problem in semantic segmentation models based on DFormerv2 encoders where the decoding structure is fixed and difficult to adapt to feature distribution differences caused by different scale configurations of the same encoder. This improves the overall performance of RGB-D semantic segmentation while ensuring computational efficiency.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] A structure-adaptive decoding method for DFormerv2 encoders includes the following steps:
[0008] S1. Based on the semantic segmentation task of the DFormerV2 encoder, select multi-scale feature maps from the output features of the DFormerV2 encoder and upsample them to generate fused features.
[0009] S2. Input the fused features into the multi-branch adaptive refinement module to extract features and obtain a set of branch feature maps. The multi-branch adaptive refinement module includes multiple parallel convolutional branches, and each convolutional branch includes a configurable convolutional module for feature extraction.
[0010] S3. Input the feature map set into the adaptive branch gating module of the multi-branch adaptive refining module, calculate the spatially related gating weights for each branch, and generate the spatially weighted features of each branch.
[0011] S4. Based on the learnable static weights, the spatial weighted features of each branch are weighted and fused to obtain unified and refined features;
[0012] S5. Input the unified and refined features into the feature enhancement module and then into the semantic segmentation prediction head to generate the final segmentation result.
[0013] Preferably, in S1, the multi-scale feature maps of the latter three stages are selected as follows:
[0014]
[0015] in, Indicates the number of channels. , Representing the spatial dimensions of the feature map; upsampling of multi-scale feature maps:
[0016]
[0017] in, Represents the bilinear interpolation upsampling function, which will and Adjust to Same space size , recorded as and ; upsampled feature map and along Channel dimensions are spliced together to generate fused features. :
[0018]
[0019] in, This indicates a channel direction splicing operation.
[0020] Preferably, in S2, the convolution operation of the convolution branch is represented as follows:
[0021]
[0022] in, Indicates the kernel size as The number of groups is The convolution operation; the convolution kernels for each branch are adjusted according to the encoder size. and group configuration Targeted configurations were performed, and feature information was extracted from different receptive field ranges to obtain a set of branch feature maps. .
[0023] As a preferred embodiment, in S3, the spatially relevant gating weights are calculated for each branch output, generating a spatial weight map for each branch as follows:
[0024]
[0025] in, This represents a 1×1 convolution operation. Using the Sigmoid function, the output is constrained to the range [0,1] for single-channel weighted graphs. Perform channel broadcast operation to obtain ,use For the corresponding branch feature map Perform element-wise multiplication to obtain the spatially weighted features. :
[0026]
[0027] in, This indicates an element-wise multiplication operation.
[0028] Preferably, S4 includes:
[0029] Learnable static weight parameters Spatial weighted features Perform linear fusion and satisfy:
[0030]
[0031] The fusion formula is as follows:
[0032]
[0033] in, This represents the fusion coefficient of each branch. This is a unified and refined feature output by the MABR module.
[0034] Preferably, S5 includes: a feature enhancement module that generates intermediate features based on unified refined features; and... The input channel attention module generates channel weights; the channel weights are applied to intermediate features to obtain channel-weighted features; the channel-weighted features are input to the spatial attention module to generate spatial weights; the final enhanced features are obtained based on the channel-weighted features and spatial weights.
[0035] As a preferred embodiment, S5 includes:
[0036] For unified refining characteristics Perform a 1×1 convolution operation to non-linearly calibrate and integrate the features, and generate intermediate features. :
[0037]
[0038] in, This represents a 1×1 convolution operation with the number of channels remaining constant.
[0039] Will The input channel attention module first... Global average pooling and global max pooling are performed separately to extract channel statistics. The results of these two pooling operations are then input into a multilayer perceptron structure with shared parameters. The two outputs of the multilayer perceptron structure are summed element-wise, and finally, the channel weights are obtained through a sigmoid activation function. :
[0040]
[0041] in, and These are global average pooling and global max pooling operations, respectively. This represents a 1×1 convolution operation used to map statistical information to channel responses; This is the Sigmoid activation function.
[0042] Preferably, S5 also includes: channel weights Acting on Channel weighted features are obtained. :
[0043]
[0044] Channel weighting features The input spatial attention module performs average pooling and max pooling along the channel dimension to obtain 2-channel feature maps. These two feature maps are then concatenated along the channel dimension, and then processed through a single layer. Convolution captures spatial context information, and finally, spatial weights are obtained through the Sigmoid activation function.
[0045]
[0046] in, and Channel weighted features respectively Perform average and max pooling operations along the channel dimension; This indicates splicing along the channel dimension; This represents a 7×7 convolution.
[0047] As a preferred approach, spatial weights are applied to channel-weighted features to obtain enhanced features. :
[0048]
[0049] Enhance features Input the segmentation prediction head and output the final segmentation result.
[0050] Compared with the prior art, the beneficial effects of the present invention are reflected in:
[0051] 1. Unlike traditional encoder-decoder architectures that use fixed-structure decoders to uniformly process output features from encoders of different sizes, this invention proposes a structure-adaptive coupled decoding method. This method differs from traditional techniques that use multi-branch convolutions or attention mechanisms independently or simply in series. Instead, this method uses the configurable multi-branch structure of the MABR module to structurally refine local feature differences. Based on this, it combines the cross-dimensional joint enhancement of the FEM module to achieve collaborative optimization of local spatial details and global semantic information. This ensures that the decoder can achieve structural matching and semantic alignment with DFormerv2 encoders of different sizes, improving the performance and efficiency of RGB-D semantic segmentation in complex indoor scenes.
[0052] 2. This invention is applicable to encoder-decoder type RGB-D semantic segmentation tasks based on DFormerv2 encoders, and is used to solve the problem that in semantic segmentation models based on DFormerv2 encoders, the decoding structure is fixed and it is difficult to adapt to the feature distribution differences caused by different scale configurations of the same encoder.
[0053] The decoder of this invention consists of a MABR and an FEM, which work synergistically. Structurally, it combines static configuration and dynamic adaptation, allowing the decoder to be flexibly configured to better adapt to the output of DFormerv2 encoders of different sizes, thereby further improving the upper limit of segmentation performance.
[0054] 3. This invention introduces a multi-branch adaptive refinement module and a feature enhancement module during the decoding stage, achieving hierarchical processing and collaborative enhancement of features output by the DFormerv2 encoder at different scale configurations. Specifically, the multi-branch adaptive refinement module, through a configurable multi-branch convolutional structure and adaptive gating mechanism, performs fine-grained modeling and hierarchical refinement of the fused features, addressing the local structural differences generated by the DFormerv2 encoder under different scale configurations. Meanwhile, the feature enhancement module, based on a channel and spatial joint attention mechanism, performs cross-dimensional filtering and enhancement of the refined features to improve the consistency of high-level semantic expression and help preserve spatial detail information.
[0055] 4. Through the aforementioned adaptive coupling design, this invention effectively alleviates the mismatch problem caused by structural differences between the DFormerv2 encoder and decoder under different scale configurations, enabling the decoder to more rationally utilize the feature information output by encoders of different scales. In different scale implementations represented by the DFormerv2 encoder, this method helps improve the overall performance of semantic segmentation results while ensuring computational efficiency, thereby achieving a more balanced performance gain between accuracy and efficiency. Attached Figure Description
[0056] Figure 1 This is a schematic diagram of the method framework of Embodiment 1 of the present invention;
[0057] Figure 2 This is a schematic diagram of the adaptive gating weight distribution in the multi-branch adaptive refining module of Embodiment 1 of the present invention;
[0058] Figure 3 This is a schematic diagram comparing the semantic segmentation performance of the present invention with other model methods under the Dformerv2 encoder in Embodiment 1 of the present invention. Detailed Implementation
[0059] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.
[0060] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.
[0061] The Structure-Adaptive Coupled Decoder (SACD) method proposed in this invention is applied to semantic segmentation tasks based on the DFormerV2 encoder. By introducing a Multi-Branch Adaptive Refinement (MABR) module and a Feature Enhancement Module (FEM) in the decoding stage, the structure of the output features of encoders of different sizes is adaptively optimized, thereby improving the feature fusion and semantic reconstruction effects in the decoding stage.
[0062] Existing multi-branch convolutional structures typically aim to enhance multi-scale representation capabilities by expanding the receptive field through parallel convolutional branches to improve the richness of feature extraction. The MABR module in this invention is not simply for enhancing multi-scale representation; rather, it addresses the differences in feature distribution among the output features of DFormerv2 encoders of different sizes by designing an MABR that incorporates static configuration and dynamic adaptation, achieving adaptive adaptation to encoded features of different scales. CCM is a component of MABR, and this invention modifies the convolutional kernel and the number of groups in the CCM, making it a crucial element in enabling the decoder to adapt to encoders of different sizes. Traditional convolutional methods, however, are fixed and do not change with the size of the encoder.
[0063] Unlike existing technologies that treat feature enhancement as a general, standalone plugin, the feature enhancement module in this invention forms a local-global coupled collaborative mechanism with MABR. Its technical essence lies in the fact that MABR performs structured local refinement on encoders of different sizes, while FEM introduces global contextual dependencies based on the refined local features through cross-dimensional joint enhancement. This design aims to compensate for the local receptive field limitations imposed by MABR through global enhancement, achieving simultaneous optimization of local geometric details and global semantic consistency when dealing with feature differences in encoders of different sizes, thus achieving a performance gain greater than the sum of its parts (1+1>2).
[0064] Example 1:
[0065] like Figure 1The structure-adaptive decoding method for DFormerv2 encoders, as shown, includes the following steps:
[0066] S1. Based on the semantic segmentation task of the DFormerV2 encoder, select multi-scale feature maps from the output features of the DFormerV2 encoder and upsample them to generate fused features.
[0067] Multi-scale feature selection and alignment. Several high-level semantic feature maps are selected from the multi-layer features output by the encoder. These selected feature maps are upsampled to ensure consistent spatial resolution and then concatenated along the channel dimension to obtain fused features. This embodiment specifically includes:
[0068] First, select the multi-scale feature maps of the last three stages from the output of the DFormerV2 encoder, denoted as follows: .
[0069]
[0070] in, Indicates the number of channels. This indicates the spatial dimensions of the feature map.
[0071] Specific value explanation: Assume the input image size is... ,but The spatial dimensions correspond to the input image size respectively. . The size is , The size is , The size is The DFormerV2 encoder has different channel count configurations depending on its scale. For DFormerV2-B, the channel count is as follows: For DFormerV2-L, the number of channels is respectively .
[0072] The selected feature map will then be... and Upsampling to unify resolution Space dimensions:
[0073]
[0074] in Represents the bilinear interpolation upsampling function, which will and Adjust to Same space size , recorded as and Subsequently, the upsampled feature map and Stitching along the channel dimension of F2:
[0075]
[0076] in This indicates a channel-direction splicing operation to generate fused features. This fusion feature Total number of channels Taking DFormerV2-B as an example, Taking DFormerV2-L as an example, The fused features are then used as input to the multi-branch adaptive refining module.
[0077] S2. Input the fused features into the multi-branch adaptive refinement module to extract features and obtain a set of branch feature maps. The multi-branch adaptive refinement module includes multiple parallel convolutional branches, and each convolutional branch includes a configurable convolutional module for feature extraction.
[0078] The fused features are input into a multi-branch convolutional structure, and different receptive field configurations are used in different branches to process the features in parallel to extract diverse local structural information. In this embodiment, the obtained fused features... The input is fed into the MABR module of the decoder for refinement. MABR contains... There are 10 parallel convolutional branches, each branch Both methods employ a configurable convolutional module (CCM) for feature extraction, with the CCM having the following input channels: Number of output channels in the branch Adaptive settings based on encoder size: DFormerV2-B ;DFormerV2-L The convolution operation for each branch is represented as follows:
[0079]
[0080] in The core of MABR lies in its configuration. The adaptive nature of this configuration, which matches the encoder size, was determined experimentally when the encoder was DFormerV2-B. , Perform grouped convolution. That is, all three branches use... The convolutional kernels were all set to 4 groups; when the encoder was DFormerV2-L, the configuration was determined experimentally. , That is, the three branches each adopt... The convolution kernel, where Convolution kernel uses the number of groups , Convolution kernel uses the number of groups ,and Convolution kernel uses the number of groups After multi-branch refinement, a set of branch feature maps is obtained. .
[0081] S3. Input the feature map set into the adaptive branch gating module of the multi-branch adaptive refining module, calculate the spatially related gating weights for each branch, and generate the spatially weighted features of each branch.
[0082] Adaptive gated modulation. Spatially relevant gate weights are calculated for the output of each branch, and the feature responses are adaptively modulated to balance the contributions of different branches in the spatial dimension. Specifically, this includes:
[0083] feature map set Input to the adaptive branch gating module (corresponding appendix) Figure 1 Spatial weighting is performed within the ABG module inside the MABR. Specifically, for the output features of each branch... First through a Convolution is used to reduce the dimensionality to a single channel, and then the Sigmoid function is applied to constrain the output to... Within the range, thus obtaining the spatial weight map. The calculation formula is as follows:
[0084]
[0085] Next, the single-channel weight map is analyzed. Extend the channel broadcast operation to One channel, obtain This makes its dimension match the corresponding branch output features. Keep it consistent, then use Output features for the corresponding branches Perform element-wise multiplication to obtain the spatially weighted features. ,Right now:
[0086]
[0087] After adaptive gating modulation, a spatially weighted feature set is obtained. .
[0088] S4. Based on the learnable static weights, the spatial weighted features of each branch are weighted and fused to obtain unified and refined features; specifically including:
[0089] Features after adaptive gating Linear fusion is performed, an operation that is handled by a globally weighted fusion operation within the MABR module, such as... Figure 1 The internal structure of the MABR module is shown below. Complete. To ensure the normalization of the fusion, learnable static weight parameters are introduced. And satisfy:
[0090]
[0091] The fusion formula is as follows:
[0092]
[0093] in, This is a unified and refined feature for the final output of the MABR module. The fusion coefficients of each branch are parameters learned during model training. This step outputs unified, refined features. It is passed as input to the feature enhancement module.
[0094] S5. Input the unified and refined features into the feature enhancement module and then into the semantic segmentation prediction head to generate the final segmentation result.
[0095] The refined features are input into the feature enhancement module, where feature reshaping, channel enhancement, and spatial enhancement are performed sequentially. The final segmentation result is then output through the semantic segmentation prediction head, specifically including:
[0096] Unified Refining Characteristics Input Feature Enhancement Module (FEM Module), FEM refines unified features Perform convolution, channel, and spatial attention operations.
[0097] First, FEM on input features Perform a 1×1 convolution operation to non-linearly calibrate and integrate the features, and generate intermediate features. .
[0098]
[0099] Next, calculate the channel attention weights. .
[0100] Will Input channel attention module (corresponding appendix) Figure 1 The CAB module in [the document / platform]). This module first [addresses / compare]... Global average pooling and global max pooling are performed separately to extract channel statistics. The results of these two pooling operations are then input into a multilayer perceptron (MLP) structure that shares parameters. The two outputs of the MLP are summed element-wise, and the channel weights are obtained by applying a sigmoid activation function. ,Right now:
[0101]
[0102] in, and These are global average pooling and global max pooling operations, respectively. This represents a 1×1 convolution operation used to map statistical information to channel responses;
[0103] This is the Sigmoid activation function.
[0104] The calculated channel weights Acting on Channel weighted features are obtained. .
[0105]
[0106] Furthermore, spatial attention weights are calculated. .
[0107] Will Input Spatial Attention Module (corresponding appendix) Figure 1 The SAB module in the middle). This module first follows along Average pooling and max pooling are performed on each dimension to obtain 2-channel feature maps, which are then concatenated along the channel dimension. A single-layer 7×7 convolutional kernel is then used to capture spatial context information, and finally, a sigmoid activation function is applied to obtain spatial weights. .
[0108]
[0109] in, and To each Perform average and max pooling operations along the channel dimension; This indicates splicing along the channel dimension; express Convolution. Then, spatial weights are applied to the channel-weighted features to obtain the final enhanced features. :
[0110]
[0111] The enhanced features obtained in this step The predicted semantic segmentation result is sent to the segmentation head.
[0112] To further illustrate the feasibility and effectiveness of this invention, this embodiment provides a specific training and evaluation configuration for semantic segmentation using the SACD decoder and DFormerv2-B encoder based on the NYUDepthV2 dataset. The model uses the cross-entropy loss function. The training process employs the AdamW optimizer, with a total training period of 450 epochs and a batch size set to... The model is initialized using pre-trained weights with an initial learning rate of [value missing]. The learning rate is used for a linear warm-up in the first 10 epochs, then increased exponentially. A polynomial decay strategy is used for attenuation. Multi-scale scaling enhancement is employed during training, with a scale range of [0.5, 0.75, 1.0, 1.25, 1.5]. During model evaluation, the input image is cropped to a size of [missing value]. Furthermore, a horizontal flipping strategy is employed to improve accuracy.
[0113] To further illustrate the feasibility and effectiveness of the present invention, a systematic experimental verification of the proposed structure-adaptive coupled decoding method was conducted on the typical indoor RGB-D semantic segmentation dataset NYU Depth V2. During the experiments, pixel accuracy (PA), mean pixel accuracy (MA), and mean intersection over union (mIoU) were used as the main evaluation metrics for model segmentation performance. mIoU comprehensively measures the segmentation quality of semantic regions of each category, while PA and MA reflect the prediction accuracy of the model at the overall pixel level and the category balance level, respectively. Furthermore, to comprehensively evaluate the present invention's ability to balance performance and efficiency, the model's parameter count, floating-point operations (FLOPs), inference latency, and mean intersection over union (mIoU) were statistically analyzed and compared to understand the impact of the proposed decoding structure on computational overhead and segmentation accuracy under different encoder configurations.
[0114] Table 1 shows the comparison results of the proposed method with the latest state-of-the-art (SOTA) models in various evaluation indicators.
[0115] method Parameters Floating-point computation Inference latency Average crossover ratio CMX-B2 66.6M 65.6G 71.5ms 54.4 DFormer-L 39.0M 69.3G 44.5ms 57.2 GeminiFusion-B3 75.8M 138.2G 68.2ms 56.8 DFormerv2-B 53.9M 67.2G 50.7ms 57.7 CMX-B5 181.1M 167.8G 114.9ms 56.9 CMNext-B4 119.6M 131.9G 98.5ms 56.9 MultiMAE
[46] 95.2M 267.9G 76.9ms 56.0 GeminiFusion-B5 137.2M 256.1G 108.7ms 57.7 DFormerv2-L 95.5M 124.1G 79.9ms 58.4 This invention 56.14M 77.8G 50.9ms 58.5
[0116] As shown in Table 1, real-time inference speed, mIoU, and computational overhead are key metrics for evaluating the feasibility of deploying RGB-D models in real-world downstream applications. To fully evaluate the effectiveness of the structure-adaptive coupling decoding method, it was compared with other mainstream methods in terms of Params, FLOPs, Latency, and mIoU. To ensure fairness, all experiments were conducted on the same hardware environment using a single RTX 3090 GPU, with input image resolution of 480×640. The experimental results are shown in Table 1, demonstrating that the structure-adaptive coupling decoding method combined with the DFormerv2-B encoder achieves a good balance between inference speed and segmentation accuracy.
[0117] Table 2 Comparison results of the present invention with other mainstream decoding methods under multiple evaluation indicators.
[0118] Decoding method Parameters Floating-point computation Pixel accuracy Average accuracy Average crossover ratio FCN 54.69M 61.41G 79.0 69.3 56.1 UPerNet 82.27M 306.97G 79.4 68.9 56.2 DeepLabV3+ 57.46M 77.39G 79.7 70.2 56.8 NL 53.90M 67.20G 80.1 70.6 57.5 Segformer 62.59M 243.99G 79.9 70.0 57.2 HAM 53.90M 67.20G 79.8 70.9 57.0 This invention 56.14M 77.8G 80.5 71.5 58.5
[0119] Table 2 shows a comparison of segmentation accuracy and computational cost for various representative decoders on the NYUv2 dataset when using DFormerV2-B as the fixed backbone network. It can be seen that, with similar parameter sizes, our proposed SACD achieves the best result of 58.5 mIoU on DFormerV2-B while maintaining a low computational cost (77.86 GFLOPs). This result demonstrates that SACD exhibits superior performance gains within the DFormerV2 framework compared to existing decoding structures.
[0120] Experimental results show that the structure-adaptive coupling decoding method proposed in this invention can effectively improve the accuracy of RGB-D semantic segmentation while maintaining reasonable computational complexity, achieving a good balance between accuracy and computational efficiency, and verifying the effectiveness and practical value of the method in real-world application scenarios.
[0121] Figure 2 The results show that the adaptive gating mechanism generates significantly different attention distributions across the three branches. Different branches spatially focus on different regions and structural patterns within the scene, demonstrating complementary capabilities in capturing overall layout, geometric contours, and local details. This indicates that MABR can automatically allocate adaptive feature contributions through multi-branch gating, thereby improving the modeling effect of multi-level information in the decoding stage. The experiments fully verify that even when human joints are severely occluded, the method of this invention can accurately detect the location of occluded human joints, demonstrating the advantages of this invention.
[0122] Figure 3This paper presents a comparison of semantic segmentation results between the SACD-DFormerV2-L and the original DFormerv2-L decoders on the NYUv2 dataset. Under the same backbone network configuration (both based on the DFormerv2-L encoder), SACD-DFormerV2-L can achieve better prediction results while maintaining inference speed.
[0123] The above description is merely a detailed explanation of the method provided by this invention and is not intended to limit the scope of protection of this invention. Those skilled in the art can make various improvements and changes without departing from the spirit and scope of this invention. Therefore, any technical solutions obtained by equivalent substitution or equivalent transformation of this invention should be included within the scope of protection of this invention.
Claims
1. A structure-adaptive decoding method for DFormerv2 encoders, characterized in that, Includes the following steps: S1. Based on the semantic segmentation task of the DFormerV2 encoder, select multi-scale feature maps from the output features of the DFormerV2 encoder and upsample them to generate fused features. S2. Input the fused features into the multi-branch adaptive refinement module to extract features and obtain a set of branch feature maps. The multi-branch adaptive refinement module includes multiple parallel convolutional branches, and each convolutional branch includes a configurable convolutional module for feature extraction. S3. Input the feature map set into the adaptive branch gating module of the multi-branch adaptive refining module, calculate the spatially related gating weights for each branch, and generate the spatially weighted features of each branch. S4. Based on the learnable static weights, the spatial weighted features of each branch are weighted and fused to obtain unified and refined features; S5. Input the unified and refined features into the feature enhancement module and then into the semantic segmentation prediction head to generate the final segmentation result.
2. The structure-adaptive decoding method for a DFormerv2 encoder according to claim 1, characterized in that, In S1, the multi-scale feature maps of the last three stages are selected as follows: ， in, Indicates the number of channels. , Representing the spatial dimensions of the feature map; upsampling of multi-scale feature maps: ， in, Represents the bilinear interpolation upsampling function, which will and Adjust to Same space size , recorded as and ; upsampled feature map and along Channel dimensions are spliced together to generate fused features. : ， in, This indicates a channel direction splicing operation.
3. The structure-adaptive decoding method for a DFormerv2 encoder according to claim 1, characterized in that, In S2, the convolution operation of the convolution branch is represented as: ， in, Indicates the kernel size as The number of groups is The convolution operation; the convolution kernels for each branch are adjusted according to the encoder size. and group configuration Targeted configurations were performed, and feature information was extracted from different receptive field ranges to obtain a set of branch feature maps. .
4. The structure-adaptive decoding method for a DFormerv2 encoder according to claim 3, characterized in that, In S3, the spatially relevant gating weights are calculated for each branch, generating the spatial weight graph for each branch as follows: ， in, This represents a 1×1 convolution operation. Using the Sigmoid function, the output is constrained to the range [0,1] for single-channel weighted graphs. Perform channel broadcast operation to obtain ,use For the corresponding branch feature map Perform element-wise multiplication to obtain the spatially weighted features. : ， in, This indicates an element-wise multiplication operation.
5. The structure-adaptive decoding method for a DFormerv2 encoder according to claim 1, characterized in that, S4 include: Learnable static weight parameters Spatial weighted features Perform linear fusion and satisfy: ， The fusion formula is as follows: ， in, This represents the fusion coefficient of each branch. This is the unified refining feature output by the multi-branch adaptive refining module.
6. The structure-adaptive decoding method for a DFormerv2 encoder according to claim 1, characterized in that, S5 includes: a feature enhancement module that generates intermediate features based on unified refined features; and... The input channel attention module generates channel weights; the channel weights are applied to intermediate features to obtain channel-weighted features; the channel-weighted features are input to the spatial attention module to generate spatial weights; the final enhanced features are obtained based on the channel-weighted features and spatial weights.
7. The structure-adaptive decoding method for a DFormerv2 encoder according to claim 6, characterized in that, S5 include: For unified refining characteristics Perform a 1×1 convolution operation to non-linearly calibrate and integrate the features, and generate intermediate features. : ， in, This represents a 1×1 convolution operation with the number of channels remaining constant. Will The input channel attention module first... Global average pooling and global max pooling are performed separately to extract channel statistics. The results of these two pooling operations are then input into a multilayer perceptron structure with shared parameters. The two outputs of the multilayer perceptron structure are summed element-wise, and finally, the channel weights are obtained through a sigmoid activation function. : ， in, and These are global average pooling and global max pooling operations, respectively. This represents a 1×1 convolution operation used to map statistical information to channel responses; This is the Sigmoid activation function.
8. The structure-adaptive decoding method for a DFormerv2 encoder according to claim 7, characterized in that, S5 also includes: channel weights Acting on Channel weighted features are obtained. :
9. Channel weighted features The input spatial attention module performs average pooling and max pooling along the channel dimension to obtain 2-channel feature maps. These two feature maps are then concatenated along the channel dimension, and then processed through a single layer. Convolution captures spatial context information, and finally, spatial weights are obtained through the Sigmoid activation function. ， in, and Channel weighted features respectively Perform average and max pooling operations along the channel dimension; This indicates splicing along the channel dimension; This represents a 7×7 convolution.
10. A structure-adaptive decoding method for a DFormerv2 encoder according to claim 8, characterized in that, The enhanced feature is obtained by applying spatial weights to the channel-weighted feature. : ， Enhance features Input the segmentation prediction head and output the final segmentation result.