DiffFusion-Seg: multi-scale feature fusion semantic segmentation framework based on diffusion model and Transform

Abstract

The invention discloses a multi-scale feature fusion semantic segmentation method and a multi-scale feature fusion semantic segmentation system based on a diffusion model and Transform. The method comprises the following steps: carrying out normalization and data enhancement preprocessing on an input image; extracting multi-scale detail features in a plurality of time steps through a pre-trained diffusion model, and forming a diffusion feature set through channel compression and normalization; a visual Transform module is utilized to extract global semantic features; the method comprises the following steps: by taking a Transform feature as a query and a diffusion feature as a key value (Key / Value), realizing semantic-guided detail fusion through a trans-attention mechanism; performing context enhancement in combination with an enhanced local attention (ELA) and scale awareness pyramid pooling (SFA-PPM) module, and constructing a multi-scale feature pyramid; and finally, outputting a pixel-level semantic segmentation result through the lightweight MLP decoder. According to the method, the fine-grained texture representation of the diffusion model and the global semantic modeling capability of Transform are effectively combined, and the segmentation precision, the boundary fidelity and the small target recognition performance are improved. The method is high in calculation efficiency, can realize real-time reasoning on the middle-end GPU, and is suitable for the fields of automatic driving, remote sensing analysis, medical image segmentation and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and artificial intelligence, and relates to an image semantic segmentation method and system based on diffusion model and Transformer fusion, which is especially suitable for multi-scale feature fusion and pixel-level semantic recognition tasks. The method can be applied to automatic driving scene understanding, medical image assisted diagnosis, remote sensing feature extraction, industrial quality detection, intelligent security and video behavior analysis, and belongs to the technical field of deep learning image processing and intelligent perception. BACKGROUND

[0002] Semantic segmentation is an important basic task in the field of computer vision, which aims to assign semantic class labels to each pixel in an image, realizing pixel-level semantic recognition. Semantic segmentation technology plays a key role in automatic driving, remote sensing mapping, medical image analysis, industrial defect detection and video understanding, such as: automatic driving systems rely on accurate segmentation of roads, vehicles and pedestrians to ensure safety; remote sensing image analysis needs to distinguish cultivated land, buildings and water bodies for land planning; accurate segmentation of lesions in medical images directly affects the diagnosis effect. Therefore, improving the performance of semantic segmentation models in boundary details, complex scenes and small targets has significant engineering value.

[0003] The existing mainstream semantic segmentation methods are mainly based on convolutional neural networks (CNN). From FCN, U-Net to DeepLab series, convolutional structures significantly improve the segmentation accuracy through mechanisms such as skip connection, dilated convolution and multi-scale context extraction. However, due to the fixed local receptive field of convolutional operators, it is difficult to effectively model long-range dependencies, which can cause global semantic misclassification, blurred fine boundaries and insufficient small target recognition. To make up for the lack of global information modeling ability, the Transformer structure is introduced into the semantic segmentation task, such as SegFormer, Mask2Former and other models rely on self-attention mechanism to enhance global information association. However, such methods often weaken local detail expression while obtaining long-range dependencies, accompanied by high computational complexity and memory consumption, which limits their practical deployment on resource-constrained devices (such as single-card RTX 4060 and embedded devices).

[0004] Diffusion Models, which have been rapidly developed in recent years, have made breakthroughs in image generation tasks. They learn the multi-scale structural information of images through a step-by-step noise adding and inverse diffusion process, forming rich details and global structural features at different noise time steps. Studies have shown that the intermediate features of the U-Net within the diffusion model have the advantages of edge detail expression and noise robustness, and have been successfully used in image restoration, super-resolution and other tasks. However, when the diffusion model is used for discriminant tasks (such as semantic segmentation), it still faces key challenges: the feature distribution at different time steps is very different, making it difficult to align with semantic features; the multi-time step features have redundancy, making direct use of them costly; and there is a lack of efficient cross-modal fusion mechanism to achieve adaptive complementation of details and semantic information.

[0005] Therefore, existing semantic segmentation techniques still lack a fusion architecture that can both leverage the advantages of diffusion models in fine-grained expression and combine the global semantic understanding capabilities of Transformers, while also being efficient and deployable. In practical needs, how to further improve the boundary fidelity, semantic consistency, and small target recognition capability while ensuring inference efficiency is a technical problem that needs to be solved. SUMMARY

[0006] I. Invention Objectives

[0007] The main objective of the present invention is to propose a multi-scale feature semantic segmentation method and system based on the fusion of diffusion models and Transformers, to solve the problems of existing semantic segmentation models, such as insufficient boundary detail recovery, limited global semantic modeling capability, low inference efficiency due to feature redundancy, and weak small target recognition capability. Existing semantic segmentation methods based on convolutional neural networks or Transformers often face the dilemma of balancing between details and global semantics. While diffusion models can provide high-quality detail features, direct application to discriminant tasks can result in modal differences and feature redundancy. The present invention takes full advantage of the multi-scale detail advantages of diffusion models in the multi-time step denoising process, and efficiently fuses the global semantic expression of Transformers across scales, achieving complementary enhancement of detail features and semantic features.

[0008] The present invention designs a cross-modal and cross-scale feature interaction structure, guides the adaptive selection and integration of semantic features to diffusion model detail information, preserves high-resolution boundary structures, and maintains overall semantic consistency and contextual understanding capabilities. In resource-constrained device configurations (such as RTX 4060, embedded terminals), it still maintains high inference speed and deployability. Compared with existing methods, the present invention can significantly improve the accuracy, boundary fidelity, complex scene understanding capability, and small target recognition effect of semantic segmentation, and has wide engineering application value and industrial promotion prospects.

[0009] II. Technical solutions

[0010] To achieve the above-mentioned purposes, the present application provides a multi-scale semantic segmentation method and system based on diffusion model and Transformer fusion. The method comprises an input preprocessing module, a diffusion branch feature extraction module, a Transformer branch feature extraction module, a cross-scale fusion module, a decoding and prediction module, and a training optimization module. By effectively aligning and fusing the multi-time step feature representation of the diffusion model and the Transformer feature representation, the segmentation quality can be improved while ensuring the inference efficiency. The overall structure of the system is shown in the figure (see attached Figure 1 ).

[0011] The method of the present application comprises the following steps:

[0012] (1) Input preprocessing module

[0013] This module is used for standardization processing and data enhancement of the original input image I to improve the stability and generalization ability of model training. Specifically, it includes normalization, random cropping, random horizontal flipping, color jittering and other operations to obtain standardized image input suitable for training and inference stage. The preprocessed image is input into the diffusion model feature extraction branch and the Transformer feature extraction branch to realize parallel feature modeling.

[0014] (2) Diffusion model feature extraction branch (Diffusion Branch)

[0015] This branch uses a pre-trained diffusion model (such as DDIM or Latent Diffusion) to extract multi-time step detail features of the image, and the processing steps are as follows:

[0016] 1. Select several representative time steps (e.g. t0, t1, t2) in the inverse diffusion process, and extract the corresponding multi-scale intermediate feature set {F1, F2,..., F5} from the U-Net decoder.

[0017] 2. For different time step features at the same scale, concatenate along the channel direction to form comprehensive features D i containing multi-noise level information.

[0018] 3. Perform channel compression (compression ratio can be 1 / 4) on D i by 1x1 convolution + BatchNorm + ReLU, and unify the channel dimension.

[0019] 4. The diffusion features are normalized and scaled by the adapter layer activated by LayerNorm+GELU, obtaining stable diffusion feature output {D'1, D'2, D'3, D'4, D'5}. These features retain the texture details of the low-noise layer and the structural information of the high-noise layer, providing high-fidelity detail support for subsequent fusion.

[0020] (3) Transformer feature extraction branch (Transformer Branch)

[0021] This branch extracts global semantic features based on the visual Transformer (such as the SegFormer framework), and the specific steps are as follows:

[0022] 1. The overlapping Patch Embedding module is used to divide the input image into an overlapping patch sequence, forming an initial token representation.

[0023] 2. The features are extracted by multiple Transformer Blocks (including multi-head self-attention MSA and mixed feed-forward network Mix-FFN).

[0024] 3. The model consists of four stages (Stage 1-4), which gradually reduce the resolution and extract different semantic level features {T1, T2, T3, T4}.

[0025] 4. The tokens output by each stage are subjected to 1x1 convolution + LayerNorm transformation to obtain Transformer features with uniform channel dimensions.

[0026] (4) Cross-scale fusion module (Cross-Attention+MSFE-FPN)

[0027] This module is the core innovative part of the present application, responsible for efficiently fusing diffusion features and Transformer features at multiple scales to enhance semantic consistency and detail expression. This module includes the following substructures:

[0028] (a) Cross-attention fusion (Cross-Attention Fusion)

[0029] 1. Take the Transformer feature T i as the Query, and the diffusion feature D' i as the Key and Value;

[0030] 2. Through the multi-head cross-attention mechanism (Multi-Head Cross-Attention), semantic-guided detail selection and feature injection are realized:

[0031]

[0032] 3. The output feature M i is connected with residual, normalized and linearly mapped to stabilize training and enhance semantic structure.

[0033] (b) Enhanced Local Attention Module (ELA)

[0034] After the fusion of the feature M i , the ELA (Efficient Local Attention) module is introduced, which uses lightweight depthwise-separable convolution combined with local window attention to highlight the response area of boundaries and small targets.

[0035] (c) Multi-scale Feature Enhancement Pyramid (MSFE-FPN+SFA-PPM)

[0036] 1. Introduce SFA-PPM (Selective Feature Aggregation + Pyramid Pooling Module) on the deepest layer of fusion feature M4, enhance the context information through multi-scale adaptive pooling and gated aggregation.

[0037] 2. From bottom to top and from top to bottom, the multi-scale feature pyramid {M5, M6, M7, M8} is formed by step-by-step fusion, and the resolution is unified (such as H / 4×W / 4).

[0038] This structure not only ensures the consistency of high-level semantics, but also retains the boundary and texture details of the bottom layer, realizing the global coordination of multi-level features.

[0039] (5) Decoding and prediction module

[0040] 1. The fused multi-scale features {M5, M6, M7, M8} are upsampled to the same spatial resolution and then concatenated.

[0041] 2. The concatenated features are processed by a lightweight MLP decoder (multi-layer linear layer + GELU + Dropout) and a 3×3 depthwise separable convolution to output per-pixel class logits.

[0042] 3. Finally, the semantic segmentation mask is obtained through softmax or argmax operation.

[0043] (6) Training and optimization strategy

[0044] 1. Loss function: joint loss

[0045] L = L CE + αLauxCE + beta L Dice + gamma L boundary

[0046] where L CE is the main cross-entropy loss, L Dice improves class balance and small target performance, L boundary optimizes boundary fidelity.

[0047] 2. Optimizer and scheduling strategy: AdamW (learning rate 1x10 -4 , weight decay 1x10 -5 ) is used, combined with CosineAnnealingLR scheduler and linear warm-up.

[0048] 3. The training stage is divided into two steps:

[0049] First step: freeze diffusion and Transformer backbone, only train fusion module and decoder;

[0050] Second step: unfreeze the whole network for joint fine-tuning.

[0051] 4. Training tips: use mixed precision training (AMP), gradient clipping, and step-by-step unfreezing strategy to prevent gradient instability.

[0052] (7) Inference and post-processing

[0053] 1. In the inference stage, the input image automatically completes feature extraction, fusion and prediction, supporting sliding window (SlidingWindow) splicing to process high-resolution images.

[0054] 2. Support FP16 half-precision inference, with an average inference time of about 17ms under 512x512 input, meeting the real-time application requirements.

[0055] 3. Optional CRF (Conditional Random Field) or boundary refinement algorithm for post-processing to further optimize boundary quality.

[0056] III. Technical effects

[0057] The multi-scale semantic segmentation method and system based on diffusion model and Transformer fusion proposed by the present application can effectively collaborate between fine-grained feature expression and global semantic understanding, and has significant performance advantages and engineering application value compared with the prior art. The technical effects mainly reflect as follows:

[0058] 1. Significantly improve boundary fidelity and detail recovery ability: By introducing diffusion model in the middle of the multi-time step inverse diffusion process, the invention makes full use of the details and structural information formed at different noise levels, enabling the model to better identify boundary areas, weak structures and small targets in images. Experimental results show that compared with traditional convolution or pure Transformer method, the invention has obvious optimization in complex occlusion, boundary blur and texture detail expression, effectively solving the problems of edge blur and small target missing detection of existing models.

[0059] 2. Improve the accuracy of semantic segmentation of complex scenes and small targets: A cross-modal and cross-scale feature fusion mechanism is adopted, and Cross-Attention is used to complete the adaptive interaction between diffusion detail features and Transformer global semantic features. Combined with the multi-scale feature enhancement pyramid (MSFE-FPN) and the local attention enhancement module (ELA), the semantic consistency and spatial structure differentiation ability in multi-object scenes are improved, making the model more stable and accurate in complex background, class-intensive and scale-difference scenes.

[0060] 3. Lightweight design considering high precision and real-time inference: The fusion network constructed by the invention adopts modular and lightweight decoding structure, which significantly improves the segmentation performance while keeping low computational cost. Under the condition of typical input size 512x512, the average inference time of the model is about 17ms, which can meet the real-time or quasi-real-time application requirements. Compared with traditional single Transformer segmentation model (such as SegFormer series), the mIoU on Cityscapes dataset is improved by about 1.5%~3.5%, and the Boundary F1 is improved by about 3%~6%, showing excellent performance and efficiency balance.

[0061] 4. Good generalization ability and application potential: Due to the ability to utilize both the detail expression advantage of diffusion model and the global semantic modeling ability of Transformer, the invention has strong task adaptability and scene generalization ability. This method can be widely applied to automatic driving scene understanding, remote sensing feature extraction, medical image assisted diagnosis, industrial surface defect detection, intelligent security and video behavior analysis, etc., providing high-precision, low-latency semantic understanding ability for multi-industry intelligent perception systems.

[0062] IV. CONCLUSION

[0063] In summary, the application proposes a multi-scale feature fusion semantic segmentation method and system based on a diffusion model and a Transformer, which fully utilizes the multi-time step feature fine-grained expression ability of the diffusion model and the global semantic modeling advantage of the Transformer, and realizes efficient balance between semantic consistency and detail fidelity.

[0064] The core innovation points of the application mainly include:

[0065] 1. Cross-scale fusion mechanism of multi-time step diffusion features: utilize the texture and structure information of the diffusion model in different noise levels to form high-fidelity detail features, and dynamically fuse with semantic features;

[0066] 2. Cross-attention feature alignment strategy based on Transformer: adaptively select key detail information through semantic guided Cross-Attention, realize cross-modal information interaction and global consistent semantic expression;

[0067] 3. SFA-PPM and ELA boundary and context enhancement module: enhance the multi-scale context ability of the feature pyramid, effectively improve the complex scene analysis and boundary quality;

[0068] 4. Multi-scale supervision and two-stage progressive training strategy: improve the stability and generalization ability of feature learning, and obtain better performance at lower computational cost.

[0069] Through the above technologies, the application has excellent segmentation accuracy, real-time performance and deployment feasibility in various tasks such as urban roads, remote sensing images, medical images and industrial detection, and has significant engineering application value and industrial promotion potential compared with the prior art, providing a reliable and efficient solution for high-precision visual understanding in complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 is the overall structure diagram of the application;

[0071] Figure 2 is a diffusion model feature extraction flowchart;

[0072] Figure 3 is a Transformer feature extraction module structure diagram;

[0073] Figure 4 is a cross-scale fusion module (Cross-Attention+MSFE-FPN) structure diagram;

[0074] Figure 5 Figure 1 is a structural diagram of an SFA-PPM (Selective Feature Aggregation + Pyramid Pooling Module) sub-module structure.

[0075] Figure 6 Figure 2 is a structural diagram of an ELA (Efficient Local Attention) sub-module structure.

[0076] Figure 7 Figure 3 is a structural diagram of a decoding and prediction module structure. DETAILED DESCRIPTION

[0077] The application will be further described below with reference to the drawings. The embodiments are used to explain the principles of the application and are not intended to limit the scope of protection of the application. Those skilled in the art can make equivalent replacements and improvements without departing from the concept of the application.

[0078] Embodiment 1: Process step implementation (S101-S110)

[0079] S101: Input data preprocessing

[0080] The original input image is subjected to data enhancement and standardization processing, including normalization, random cropping, random horizontal flipping and color jittering, to obtain a standardized input image, which is input to the diffusion model feature extraction branch and the Transformer feature extraction branch, respectively.

[0081] S102: Diffusion model multi-time step feature extraction

[0082] A plurality of noise time step sets T = (t0, t1, t2} are selected in the inverse diffusion process of the diffusion model, and a set of intermediate features is extracted from the five scale layers of the pre-trained Stable Diffusion / DDIM U-Net decoder:

[0083] Scale layer extraction of intermediate feature set:

[0084] Multi-time step feature splicing and channel compression

[0085] The same scale features at different time steps are spliced along the channel direction:

[0086]

[0087] And through 1x1 convolution and BatchNorm, the compressed diffusion features D i .

[0088] S104: Diffusion feature normalization adaptation

[0089] The Adapter (LayerNorm + GELU) module is used to normalize and scale the features, obtaining stable diffusion feature output:

[0090] D′ i = Adapter (D i ) S105: Transformer global semantic feature extraction

[0091] Use SegFormer-like encoder to extract four-scale semantic features:

[0092]

[0093] Model global semantic relationship through multi-head self-attention mechanism.

[0094] S106: Cross-attention feature fusion (Cross-Attention)

[0095] Use Transformer features as Query, and diffusion features as Key and Value, to perform cross-modal semantic fusion:

[0096]

[0097] Realize the details selection and compensation of global semantic guidance.

[0098] S107: Local detail enhancement (ELA module)

[0099] Input the fused feature M i to the Efficient Local Attention module, and enhance the boundary and small target area response through local window attention and depth separable convolution to obtain enhanced features:

[0100] M′ i = ELA (M i ) S108: Multi-scale feature fusion pyramid construction (MSFE-FPN + SFA-PPM)

[0101] Add pyramid pooling and scale factor attention module to the deepest layer of features to realize context supplement and scale selection, and form a unified scale output feature set through top-down and bottom-up fusion:

[0102] (M5, M6, M7, M8} S109: Decoding and prediction output

[0103] Through upsampling and splicing fusion features, use lightweight MLP-style decoder to complete pixel-by-pixel classification:

[0104] Y = Decoder([M5↑, M6↑, M7↑, M8↑])

[0105] And use Softmax or Argmax to output the final semantic segmentation result.

[0106] S110: auxiliary training and inference optimization

[0107] Training with joint loss function:

[0108] L = L cE + αL aux + βL Dice + γL boundary

[0109] The inference stage supports FP16 half-precision and sliding window inference, and optional CRF post-processing to further optimize boundary quality and improve real-time inference performance.

[0110] Note: The above steps S101-S110 constitute the complete implementation process of the present application. Those skilled in the art can adjust or replace the above steps based on different computing resources or application scenarios, but all should be considered to fall within the scope of the present application.

[0111] Example 2: Resource-constrained device acceleration strategy

[0112] In a resource-constrained environment (e.g. single-card RTX 4060, embedded terminal, real-time inference application), to further improve inference efficiency and reduce computation and memory consumption, this embodiment optimizes the model structure and training strategy based on Example 1, including:

[0113] (1) Freeze more backbone layers during the training phase, only retain part of the adaptation layer and cross-attention fusion module for training, and reduce the parameter size of the adapter;

[0114] (2) Reduce the base input resolution during training and inference (e.g. from 512x512 to 384x384), and use sliding window inference or multi-scale fusion during inference to maintain segmentation quality;

[0115] (3) Enable EMA (Exponential Moving Average) strategy during training to obtain more stable model parameters for inference;

[0116] (4) Use FP16 mixed precision inference to further reduce memory and latency.

[0117] Beneficial technical effects of implementing the present application under resource-constrained conditions

[0118] Compared with the semantic segmentation method using only the Transformer or only the CNN / U-Net, the embodiment has the following advantages:

[0119] 1. Enhanced detail expression capability: By introducing the intermediate representation of the diffusion model (DDIM / UNet) under different noise intensities, the preservation and enhancement of fine textures and boundary pixels are realized;

[0120] 2. Improved semantic consistency and small target segmentation performance: Through the cross-attention fusion mechanism, the diffusion detail features and the Transformer global semantic features are complementary, effectively improving the recognition ability in complex scenes, small targets and dense scenes;

[0121] 3. Balance between computational efficiency and accuracy: By using cross-scale MSFE-FPN fusion and low-resolution input optimization, the computational overhead and memory cost are greatly reduced while maintaining high prediction quality, which is suitable for high-resolution remote sensing images, urban scenes and real-time inference tasks.

Claims

1. A semantic segmentation method based on multi-scale feature fusion using a diffusion model and Transformer, characterized in that, The method includes the following steps: S1: Input preprocessing steps: Normalize, randomly crop, horizontally flip, and color dithering are performed on the input image to obtain a standardized input image; S2: Diffusion feature extraction steps: Using a pre-trained diffusion model, select multiple representative time steps during the reverse diffusion process, extract multi-scale feature maps from each layer of the U-Net structure, and form a diffusion feature set through channel compression, normalization and scale alignment; S3: Transformer feature extraction steps: The input image is divided into overlapping image patches using an overlapping Patch Embedding module, and global semantic features are extracted through a multi-layer Transformer Block (including a multi-head self-attention module and a hybrid feedforward network); S4: Cross-scale fusion step: Using Transformer features as queries and diffused features as keys / values, semantically guided feature fusion is performed through a multi-head cross-attention mechanism; S5: Context Enhancement Step: Introduce Enhanced Local Attention (ELA) and Scale-Aware Pyramid Pooling (SFA-PPM) modules to the fused features to achieve detail compensation and context feature aggregation; S6: Decoding and prediction steps: Upsample and concatenate the fused multi-scale features, and output pixel-level semantic segmentation results through a lightweight multilayer perceptron (MLP) decoder; S7: Training and optimization steps: A joint optimization objective of cross-entropy loss, Dice loss and boundary loss is adopted. The AdamW optimizer and CosineAnnealing learning rate scheduling strategy are combined to carry out two-stage training to improve the semantic consistency and boundary accuracy of the model.

2. The method according to claim 1, characterized in that, The diffusion model is the DDIM (Denoising Diffusion Implicit Model), which extracts features at different time steps in the reverse diffusion process to capture multi-level texture and structural information.

3. The method according to claim 1, characterized in that, The Transformer feature extraction module adopts a four-stage hierarchical structure (Stage1-Stage4), with each stage containing a multi-head self-attention layer and a hybrid feedforward layer to achieve progressive downsampling and semantic level extraction.

4. The method according to claim 1, characterized in that, In the cross-scale fusion step, the multi-head cross-attention result is calculated using the following formula: Where Q is the Transformer feature, K and V are the diffusion features, and d k The dimension of the key vector.

5. The method according to claim 1, characterized in that, The Enhanced Local Attention (ELA) module employs a structure that combines depthwise separable convolution with local window attention to enhance the response of boundaries and small target regions.

6. The method according to claim 1, characterized in that, The Scale-Aware Pyramid Pooling Module (SFA-PPM) integrates contextual information at different scales through multi-scale adaptive pooling and gated aggregation mechanisms to improve global semantic consistency.

7. The method according to claim 1, characterized in that, The MLP decoder in the decoding and prediction steps consists of multiple linear layers, a GELU activation function, and Dropout, combined with a 3×3 depthwise separable convolution to reduce computational complexity.

8. The method according to claim 1, characterized in that, The training process employs a two-stage optimization strategy: The first stage freezes the diffusion model and the Transformer backbone, and only trains the fusion module and decoder; The second phase involves unfreezing the entire network and conducting joint fine-tuning to obtain better feature representation capabilities.

9. A semantic segmentation system based on the method of any one of claims 1 to 8, characterized in that, The system includes: The system comprises an input preprocessing module, a diffusion feature extraction module, a Transformer feature extraction module, a cross-scale fusion module, a context enhancement module, and a decoding and prediction module, wherein the connections and functions between the modules correspond to the order and function of the steps described in claim 1.

Images