Image semantic segmentation method based on dynamic fusion and context awareness

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to an image semantic segmentation method based on dynamic fusion and context awareness, which comprises the following steps of: inputting an input image into a lightweight multi-scale feature extraction backbone network, and outputting shallow-layer, middle-layer and deep-layer features; inputting the shallow-layer, middle-layer and deep-layer features into an MRFM module, and outputting a first enhanced semantic feature, a second enhanced semantic feature and a third enhanced semantic feature; inputting the third enhanced semantic features into the LACP, and outputting enhanced context features; inputting the first enhanced semantic feature, the second enhanced semantic feature and the enhanced context feature into a DWFM, and outputting a fusion feature; inputting the enhanced context features into the double-branch prediction structure, and outputting a final segmentation prediction map; constructing a joint loss function, and performing end-to-end training and optimization on the network formed in the steps S1 to S5; the problems that in an existing image semantic segmentation method, a fusion strategy is rigid, the context sensing capacity is weak, and model parameters are redundant are solved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an image semantic segmentation method based on dynamic fusion and context awareness. Background Technology

[0002] Image semantic segmentation, as a pixel-level classification task, has wide applications in engineering scenarios such as infrastructure crack detection and road defect identification. Existing deep learning methods mainly improve segmentation accuracy by constructing complex networks, but this leads to a significant increase in computational cost, making it difficult to deploy on mobile or embedded devices.

[0003] Current semantic segmentation methods face a structural contradiction between maintaining resolution and context awareness. On the one hand, maintaining high resolution to preserve detailed information can lead to insufficient receptive field, making it difficult to model the global semantic structure. On the other hand, improving semantic expressiveness through large-scale downsampling or deep structures can easily result in the loss of crack details. Furthermore, static feature fusion strategies are insufficient in judging the importance of information at different scales, further limiting fusion efficiency and segmentation performance.

[0004] Therefore, it is necessary to invent an image semantic segmentation method that is compact, has strong context awareness, and has a dynamic and efficient fusion mechanism, so as to achieve a balance between segmentation accuracy and inference efficiency and adapt to actual deployment needs. Summary of the Invention

[0005] To address the problems of rigid fusion strategies, weak context awareness, and redundant model parameters in existing image semantic segmentation methods, this invention provides an image semantic segmentation method based on dynamic fusion and context awareness, aiming to achieve a good balance between segmentation accuracy and inference efficiency, and is particularly suitable for fine-grained detection tasks of crack-type targets.

[0006] This invention is achieved using the following technical solution: An image semantic segmentation method based on dynamic fusion and context-awareness includes the following steps: S1: Construct a lightweight multi-scale feature extraction backbone network, input the input image to be segmented into the backbone network, and extract and output shallow features, mid-level features and deep features; S2: Construct a multi-resolution feature weaving module (MRFM), input the shallow, medium and deep features into the MRFM respectively, and generate and output the first, second and third enhanced semantic features through the step-by-step interaction and fusion between the parallel high-resolution path and the low-resolution path in the MRFM. S3: Construct a lightweight context aggregation module LACP, input the third enhanced semantic feature into LACP, and perform context feature enhancement processing through depthwise separable dilated convolution with different dilation rates and global context pooling operation in LACP, and output enhanced context features; S4: Construct a dynamic weighted feature fusion module (DWFM), input the first and second enhanced semantic features and the enhanced context features into DWFM, and perform adaptive weighted fusion of the three enhanced semantic features through the global context awareness mechanism in DWFM to generate and output the fused features; S5: Construct a two-branch prediction structure including a main branch and an auxiliary branch. Input the fused features into the two-branch prediction structure. The main branch performs upsampling and convolution operations on the fused features and outputs the final segmentation prediction map. S6: Construct a joint loss function to perform end-to-end training and optimization of the network constructed in steps S1 to S5.

[0007] Furthermore, in step S1, the lightweight multi-scale feature extraction backbone network includes an initial downsampling module Stem and a cascaded feature extraction stage. The feature extraction stage processes and outputs the shallow features, mid-level features and deep features in sequence. The output of the corresponding shallow features maintains the original resolution of the input image, while the output of the corresponding mid-level features and deep features reduces the resolution to 1 / 2 and 1 / 4, respectively. The features output by each stage have a uniform number of channels.

[0008] Further, in step S2, the multi-resolution feature weaving module MRFM comprises three cascaded sub-modules, each sub-module including: High-resolution paths are used to preserve the spatial resolution of input features; A low-resolution path with a fixed number of output channels and performs downsampling. The high-resolution path and the low-resolution path are subjected to feature interaction and fusion, and a channel attention mechanism is introduced after fusion.

[0009] Further, in step S4, the Dynamic Weighted Feature Fusion (DWFM) module sequentially includes: The dynamic weight calculation unit generates spatial and channel joint attention weights based on global context information, which are used to weight the three input features. The feature fusion unit uses convolution operations to integrate and reduce the dimensionality of the weighted features to generate the fused features.

[0010] Further, in step S3, the lightweight context aggregation module LACP includes: Three parallel depthwise separable dilated convolution branches, each with a different dilation rate; A global context branch extracts global context information through adaptive average pooling and convolution operations; The outputs of the three depthwise separable dilated convolutional branches and one global context branch are concatenated along the channel dimension and then integrated into the enhanced context feature through a convolution operation.

[0011] Furthermore, in step S5, the auxiliary branch of the dual-branch prediction structure receives the mid-layer features from the backbone network as input and outputs an auxiliary prediction result. The auxiliary prediction result is used to provide additional supervision signals during training to promote gradient propagation and accelerate convergence.

[0012] Furthermore, in step S6, the joint loss function is a weighted sum of the cross-entropy loss function and the Dice loss function.

[0013] Furthermore, the method is deployed on mobile or embedded devices for detecting and segmenting surface cracks in roads, buildings, or bridges.

[0014] This invention provides an image semantic segmentation method based on dynamic fusion and context-awareness, which has the following advantages compared with existing technologies: Firstly, the Multi-Resolution Feature Weaving Module (MRFM) proposed in this invention enhances the multi-scale feature interaction capability through a dual-path structure and channel attention mechanism.

[0015] Secondly, the Dynamic Weighted Fusion Module (DWFM) proposed in this invention introduces a context-aware mechanism, which effectively solves the problem of unreasonable static fusion weights.

[0016] Secondly, the LACP module proposed in this invention uses depth-separable dilated convolution and global context path, balancing receptive field and computational efficiency.

[0017] Fourth, the overall network constructed by this invention has the significant characteristics of lightweight structure and high computational efficiency, which can meet the stringent deployment requirements of mobile or embedded devices for low power consumption and real-time performance while ensuring high segmentation accuracy.

[0018] In summary, the image semantic segmentation method proposed in this invention can effectively solve the problems of rigid fusion, insufficient context and bloated model in existing methods, and is suitable for practical scenarios such as intelligent detection, mobile inspection, and engineering monitoring, which require both segmentation accuracy and inference efficiency. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the present invention.

[0020] Figure 2A comparison diagram of the original image, ground truth image, and final segmentation image of the crack segmentation image in an embodiment of the present invention. Figure 1 .

[0021] Figure 3 A comparison diagram of the original image, ground truth image, and final segmentation image of the crack segmentation image in an embodiment of the present invention. Figure 2 . Detailed Implementation

[0022] The present invention will be further explained and described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some embodiments, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0023] An image semantic segmentation method based on dynamic fusion and context-awareness, such as Figure 1 As shown, it includes the following steps: S1: Construct a lightweight multi-scale feature extraction backbone network, input the input image to be segmented into the backbone network, and extract and output shallow feature 1, mid-level feature 2 and deep feature 3.

[0024] The lightweight multi-scale feature extraction backbone network employs a high-resolution-preserving structure to achieve efficient multi-scale feature extraction. First, an initial downsampling module (Stem) performs preliminary feature extraction and dimensionality reduction on the input image. Subsequently, the features are fed into a cascaded feature extraction stage for further processing. The first stage output retains the features of the original resolution of the input image, serving as shallow features (feature1) containing rich textures and edge details.

[0025] The second stage involves downsampling the features, and the output spatial size is the original size. Figure 1 The feature of / 2 is used as the mid-level feature feature2, which contains medium-range structural information.

[0026] The third stage further downsamples, and the output space size is the original... Figure 1 The feature of / 4 is used as the deep feature 3, which contains high-level semantic information.

[0027] To control model complexity, the feature maps output at each stage have the same number of channels, which is uniformly set to 64 channels in this embodiment. This backbone network, through shared convolutional parameters and a lightweight design, effectively balances detail preservation and semantic extraction capabilities with only 0.64M parameters.

[0028] S2: Construct a multi-resolution feature weaving module (MRFM), input the shallow feature feature1, the middle feature feature2, and the deep feature feature3 into the MRFM respectively, and generate and output three enhanced semantic features feature4, feature5, and feature6 through the step-by-step interaction and fusion between the parallel high-resolution path and the low-resolution path in the MRFM.

[0029] The Multi-Resolution Feature Weaving Module (MRFM) consists of three sequentially connected sub-modules. Each sub-module employs a dual-path structure, including: High-resolution branch: Receives and preserves the spatial resolution of the input features, and performs channel adjustment through 1×1 convolution to enhance expressive power.

[0030] Low-resolution branch: The number of output channels is fixed at 64, and downsampling is performed through convolution with a stride of 2 (the maximum downsampling depth of the entire MRFM module is limited to 1 / 16 of the original image).

[0031] Within each submodule, high-resolution and low-resolution branches interact and fuse features sequentially. Specifically, semantic information from the low-resolution branch is integrated with detailed information from the high-resolution branch through feature addition or concatenation operations. To further improve the fusion effect, a lightweight channel attention mechanism (ECA) is introduced after each interaction to adaptively enhance the response to key channels in the segmentation task. After sequential processing by the three submodules, three multi-scale features with enhanced semantic expressive power are finally output, denoted as feature4, feature5, and feature6.

[0032] S3: Construct a lightweight context aggregation module LACP, input the third enhanced semantic feature feature6 into LACP, and perform context feature enhancement processing through depthwise separable dilated convolution with different dilation rates and global context pooling operation in LACP, and output enhanced context feature feature7. The lightweight context aggregation module LACP employs a parallel branching structure to efficiently capture multi-scale context information, specifically including: Three parallel depthwise separable dilated convolution branches: each branch employs depthwise separable convolution to reduce computational cost and has a different dilation rate. In this embodiment, the dilation rates of the three branches are set to 1, 2, and 3, respectively, corresponding to different receptive field sizes, enabling the capture of contextual information within different ranges.

[0033] A global context branch: This branch first performs adaptive average pooling on the input features to compress them into a global feature vector; then it is transformed through a 1×1 convolutional layer to extract image-level global context information.

[0034] The output features from the four branches are concatenated along the channel dimension to form a feature map that integrates multi-scale information from local to global. Finally, a 1×1 convolutional layer is used to integrate and adjust the channels of this concatenated feature, outputting the final enhanced contextual feature (feature7). This module expands the effective receptive field of the model while maintaining low computational complexity.

[0035] S4: Construct a dynamic weighted feature fusion module (DWFM), input the first enhanced semantic feature (feature4), the second enhanced semantic feature (feature5), and the enhanced context feature (feature7) into DWFM, and perform adaptive weighted fusion of the three enhanced semantic features through the global context awareness mechanism in DWFM to generate and output the fused feature (feature8). The Dynamic Weighted Feature Fusion (DWFM) module performs the following operations in sequence: Dynamic weight calculation: First, the three input features are captured using global average pooling (GAP) to capture their respective global contextual information. Then, a set of spatial and channel joint attention weights corresponding to the dimensions of the input features are generated through a lightweight fully connected layer (or 1×1 convolution) and a sigmoid activation function. These weights are learnable and can dynamically evaluate and assign importance to each feature layer level, each spatial location, and each channel based on the overall semantic content of the input image.

[0036] Feature fusion: Then, the generated attention weights are used to weight the three input features element by element. The weighted features are then concatenated along the channel dimension and integrated and dimensionality reduced through a 1×1 convolutional layer to generate a unified fusion feature rich in multi-scale information, feature8.

[0037] S5: Construct a dual-branch prediction structure including a main branch and an auxiliary branch. Input the fused feature 8 into the dual-branch prediction structure containing the main branch and the auxiliary branch. The main branch performs upsampling and convolution operations on feature 8 to gradually restore the spatial size of the feature map. Finally, output a pixel-level segmentation prediction map feature 9 with the same resolution as the input image.

[0038] The auxiliary branch receives the mid-layer feature feature2 from the backbone network as input and outputs an auxiliary prediction result. The auxiliary prediction result is used to provide additional supervision signals during training to promote gradient propagation and accelerate convergence.

[0039] S6: Construct a joint loss function for end-to-end training and optimization of the network constructed in steps S1 to S5. The joint loss function is a weighted sum of the cross-entropy loss function and the Dice loss function.

[0040] The cross-entropy loss function is used to optimize the classification accuracy of each pixel.

[0041] The Dice loss function is used to optimize the regional continuity and integrity of foreground target segmentation and has good robustness to class imbalance problems.

[0042] Using this joint loss function to guide network training can simultaneously improve the overall segmentation accuracy mIoU and the ability to model edge continuity.

[0043] Performance evaluation: The network constructed using the above method was trained and tested using the publicly available crack segmentation dataset CrackSeg9k. After training, the network achieved an average crossover union (mIoU) of 81.26% on the test set with only about 0.64M parameters and a computational cost of approximately 4.6 GFLOPs. Figure 2 and Figure 3 As shown, the final segmentation map generated by the method of this invention closely matches the ground truth map, accurately and completely segmenting the fine cracks on the road surface while effectively suppressing background noise. This fully demonstrates that the method of this invention, while maintaining an extremely lightweight model size, possesses excellent fine-grained segmentation performance, making it particularly suitable for deployment on mobile or embedded devices (such as drones, inspection robots, and smartphones) with strict limitations on real-time performance and computing resources. It can be used for efficient and accurate automated detection and segmentation of surface cracks in infrastructure such as roads, bridges, and buildings.

[0044] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An image semantic segmentation method based on dynamic fusion and context-awareness, characterized in that: Includes the following steps: S1: Construct a lightweight multi-scale feature extraction backbone network, input the input image to be segmented into the backbone network, and extract and output shallow features, mid-level features and deep features; S2: Construct a multi-resolution feature weaving module (MRFM), input the shallow, medium and deep features into the MRFM respectively, and generate and output the first, second and third enhanced semantic features through the step-by-step interaction and fusion between the parallel high-resolution path and the low-resolution path in the MRFM. S3: Construct a lightweight context aggregation module LACP, input the third enhanced semantic feature into LACP, and perform context feature enhancement processing through depthwise separable dilated convolution with different dilation rates and global context pooling operation in LACP, and output enhanced context features; S4: Construct a dynamic weighted feature fusion module (DWFM), input the first and second enhanced semantic features and the enhanced context features into DWFM, and perform adaptive weighted fusion of the three enhanced semantic features through the global context awareness mechanism in DWFM to generate and output the fused features; S5: Construct a two-branch prediction structure including a main branch and an auxiliary branch. Input the fused features into the two-branch prediction structure. The main branch performs upsampling and convolution operations on the fused features and outputs the final segmentation prediction map. S6: Construct a joint loss function to perform end-to-end training and optimization of the network constructed in steps S1 to S5.

2. The image semantic segmentation method based on dynamic fusion and context-awareness according to claim 1, characterized in that: In step S1, the lightweight multi-scale feature extraction backbone network includes an initial downsampling module Stem and a cascaded feature extraction stage; The feature extraction stage processes and outputs the shallow features, mid-level features and deep features in sequence. The output of the shallow features maintains the original resolution of the input image, while the output of the mid-level features and deep features reduces the resolution to 1 / 2 and 1 / 4, respectively. The features output by each stage have the same number of channels.

3. The image semantic segmentation method based on dynamic fusion and context-awareness according to claim 1, characterized in that: In step S2, the multi-resolution feature weaving module (MRFM) comprises three cascaded sub-modules, each sub-module including: High-resolution paths are used to preserve the spatial resolution of input features; A low-resolution path with a fixed number of output channels and performs downsampling. The high-resolution path and the low-resolution path are subjected to feature interaction and fusion, and a channel attention mechanism is introduced after fusion.

4. The image semantic segmentation method based on dynamic fusion and context-awareness according to claim 1, characterized in that: In step S4, the Dynamic Weighted Feature Fusion (DWFM) module sequentially includes: The dynamic weight calculation unit generates spatial and channel joint attention weights based on global context information, which are used to weight the three input features. The feature fusion unit uses convolution operations to integrate and reduce the dimensionality of the weighted features to generate the fused features.

5. The image semantic segmentation method based on dynamic fusion and context-awareness according to claim 1, characterized in that: In step S3, the lightweight context aggregation module LACP includes: Three parallel depthwise separable dilated convolution branches, each with a different dilation rate; A global context branch extracts global context information through adaptive average pooling and convolution operations; The outputs of the three depthwise separable dilated convolutional branches and one global context branch are concatenated along the channel dimension and then integrated into the enhanced context feature through a convolution operation.

6. The image semantic segmentation method based on dynamic fusion and context-awareness according to claim 1, characterized in that: In step S5, the auxiliary branch of the dual-branch prediction structure receives the mid-layer features from the backbone network as input and outputs the auxiliary prediction result. The auxiliary prediction result is used to provide additional supervision signals during training to promote gradient propagation and accelerate convergence.

7. The image semantic segmentation method based on dynamic fusion and context-awareness according to claim 1, characterized in that: In step S6, the joint loss function is a weighted sum of the cross-entropy loss function and the Dice loss function.

8. An image semantic segmentation method based on dynamic fusion and context-awareness according to any one of claims 1-7, characterized in that: The method is deployed on mobile or embedded devices for detecting and segmenting surface cracks in roads, buildings, or bridges.

Images