Semantic segmentation method based on double-focus attention and adaptive frequency domain fusion

Abstract

The invention relates to the technical field of image processing, in particular to a semantic segmentation method based on double-focus attention and adaptive frequency domain fusion, comprising: constructing a semantic segmentation network based on double-focus attention and adaptive frequency domain fusion, including a double-focus attention module, an adaptive frequency domain fusion module and a hierarchical feature fusion architecture; inputting the image data set into the double-focus attention module for preliminary enhancement and global context modeling to obtain a first output result; the first output result is input to the adaptive frequency domain fusion module for alignment and complementation of high-level semantic features and bottom-level details, and a second output result is obtained; and processing the second output result through the hierarchical feature fusion architecture to generate a pixel-level classification result. Collaborative design of the DFA module and the ASF module is realized, excellent segmentation precision is maintained while calculation overhead is effectively reduced, and optimal balance between calculation efficiency and segmentation precision is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a semantic segmentation method based on double focus attention and adaptive frequency domain fusion. BACKGROUND

[0002] In recent years, with the development of deep learning, semantic segmentation has made certain progress in the fields of autonomous driving, medical image analysis and remote sensing image interpretation. The breakthrough of Vision Transformer (ViT) in image classification tasks has promoted the development of semantic segmentation methods based on Transformer. However, the mainstream methods based on Transformer are difficult to deploy in resource-constrained environments due to high computational complexity.

[0003] In order to balance the expression ability and computational efficiency of the model, researchers have proposed various lightweight semantic segmentation schemes in recent years. For example, SeaFormer reduces computational complexity through axial attention mechanism, and SegNeXt uses multi-scale convolution to construct spatial attention mechanism, which has made certain progress in efficiency. However, these methods still have performance bottlenecks when dealing with complex scenes rich in high-frequency details, especially in small target segmentation and complex texture region performance.

[0004] In order to solve the technical problems that the improvement of traditional methods for the encoder cannot solve the computational bottleneck of processing high-resolution input images, and the existing research on the decoder is relatively less, the purpose of the present application is to provide a semantic segmentation method based on double focus attention and adaptive frequency domain fusion, the technical scheme adopted is as follows: SUMMARY

[0005] In order to solve the technical problems that the improvement of traditional methods for the encoder cannot solve the computational bottleneck of processing high-resolution input images, and the existing research on the decoder is relatively less, the purpose of the present application is to provide a semantic segmentation method based on double focus attention and adaptive frequency domain fusion, the technical scheme adopted is as follows:

[0006] An image dataset is acquired, and a semantic segmentation network based on a dual-focus attention and adaptive frequency domain fusion is constructed, including a dual-focus attention module, an adaptive frequency domain fusion module, and a hierarchical feature fusion architecture;

[0007] The image dataset is input into the dual-focus attention module for preliminary enhancement and global context modeling, to obtain a first output result;

[0008] The first output result is input into the adaptive frequency domain fusion module for alignment and complementarity of high-level semantic features and low-level details, to obtain a second output result;

[0009] The second output result is processed by the hierarchical feature fusion architecture to generate a pixel-level classification result.

[0010] Preferably, the dual-focus attention module includes a space-channel strip attention module and parallel attention paths, which are dynamic square attention paths and cubic strip attention paths, respectively.

[0011] Preferably, the image dataset is input into the dual-focus attention module for preliminary enhancement and global context modeling, to obtain a first output result, including:

[0012] The space-channel strip attention module fuses a space strip attention mechanism and a channel self-attention mechanism to perform feature enhancement on spatial features and channel features through two-stage parallel processing of the image dataset;

[0013] The enhanced features are input into the attention paths, the dynamic square attention paths process the enhanced features through low-frequency and high-frequency paths in parallel and fuse to obtain first output components, and the cubic strip attention paths process the enhanced features in parallel to integrate the spatial dependency in the horizontal and vertical directions to obtain second output components;

[0014] The first output components and the second output components are fused to determine the first output result.

[0015] Preferably, the two-stage parallel processing of the image dataset is performed to enhance the spatial features and the channel features, including:

[0016] Based on the space strip attention mechanism, the image dataset is averaged and pooled in the spatial dimension to obtain spatial features, including feature vectors in the height direction and the width direction, and is divided into several groups, the feature vectors in the height direction and the width direction are processed by convolution kernels, and are sequentially subjected to splicing, group normalization, and a gating function to generate a spatial attention map;

[0017] The channel self-attention mechanism is used to process the image dataset by a down-sampling strategy, Query, Key and Value vectors are generated through group normalization and convolution kernel, and a channel weight vector is obtained;

[0018] The spatial attention map and the channel weight vector are fused to complete feature enhancement.

[0019] Preferably, the cubic strip attention path includes at least two spatial strip attention units.

[0020] Preferably, obtaining the first output component and the second output component includes:

[0021] The low-frequency features in the enhanced features are dynamically filtered to obtain smooth features through a low-frequency channel, the high-frequency features in the enhanced features are reserved through a high-frequency channel, and the smooth features and the high-frequency features are adaptively fused to obtain the first output component.

[0022] Both of the two spatial strip attention units use strip convolution kernel and reflection padding technology to process the spatial dependence relationship in the horizontal direction and the vertical direction respectively to obtain the second output component.

[0023] Preferably, the first output result is input into the adaptive frequency domain fusion module to align and complement the high-level semantic features and the low-level details to obtain a second output result, including:

[0024] The first output result is input into the adaptive frequency domain fusion module for channel compression.

[0025] The compressed first output result is parallelly passed through a low-frequency channel and a high-frequency channel to correspondingly output a low-frequency mask and a high-frequency mask.

[0026] Based on the low-frequency mask and the high-frequency mask, hierarchical fusion is performed, and a content-aware up-sampling algorithm and a residual compensation mechanism are correspondingly used to optimize the features.

[0027] The adjusted and fused optimized features are obtained as a second output result.

[0028] Preferably, the compressed first output result is parallelly passed through a low-frequency channel and a high-frequency channel to correspondingly output a low-frequency mask and a high-frequency mask, including:

[0029] The low-frequency channel is designed to learn a low-pass filter convolution kernel to extract global structure information to generate a low-frequency mask.

[0030] The high-frequency channel uses residual high-pass filtering to generate a high-frequency mask through residual calculation.

[0031] Preferably, the low-pass filter convolution kernel is designed to learn, and the corresponding calculation formula is:

[0032]

[0033] wherein, denotes the output of the learnable low-pass filter; denotes the convolutional neural network operation; denotes the input of the learnable low-pass filter; denotes the learnable convolution kernel parameter.

[0034] Preferably, the pixel-level classification result is generated by processing the second output result through the hierarchical feature fusion architecture, specifically:

[0035] The pixel-level classification result is generated by adjusting the channel and compressing the feature of the second output result through 1x1 convolution based on the hierarchical feature fusion architecture.

[0036] The present application has the following beneficial effects:

[0037] The semantic segmentation network based on the double-focus attention and adaptive frequency domain fusion is constructed, that is, in the decoder, the collaborative design of the encoder and the decoder is realized, the calculation overhead is effectively reduced, and excellent segmentation precision is maintained; the double-focus attention module is used for preliminary enhancement and global context modeling, and the calculation efficiency is effectively optimized; compared with the traditional self-attention mechanism, the double-focus attention module reduces the dimension redundancy of the query and the key through the channel dimension compression technology, moderately reduces the calculation burden, and with the help of the efficient local-global feature interaction mechanism, the calculation overhead is greatly reduced while the global receptive field is maintained, a multi-scale attention fusion strategy is introduced, and efficient collaborative modeling of different scale features is realized; and the module can effectively capture long-range dependencies even in extreme scenarios, providing a new technical path for the research of lightweight attention mechanism.

[0038] The adaptive frequency domain fusion module aligns and complements the high-level semantic features and the low-level details, that is, the first output result is adaptively decomposed in the frequency domain through a learnable low-pass and high-pass filter, and efficient fusion of low-frequency semantic information and high-frequency detail features is realized; unlike the traditional fixed parameter method, the adaptive frequency domain fusion module uses a learnable low-frequency filter, and uses multi-scale convolution to generate a mask of low-frequency and high-frequency features; through a dynamic weight adjustment mechanism, the module can adaptively balance the weights of different frequency band features according to the input features, improve the model performance, and optimize the calculation efficiency.

[0039] Finally, the pixel-level classification result is generated through the hierarchical feature fusion architecture, which integrates the cross-level features while reducing the calculation overhead, and guarantees the lightweight characteristics of the model. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, and the advantages thereof, below briefly introduces the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can be obtained based on these drawings without creative labor.

[0041] Figure 1 An architecture diagram of a semantic segmentation network of a semantic segmentation method based on double-focus attention and adaptive frequency domain fusion provided by an embodiment of the present application;

[0042] Figure 2 An architecture diagram of a double-focus attention module of a semantic segmentation method based on double-focus attention and adaptive frequency domain fusion provided by an embodiment of the present application;

[0043] Figure 3 An architecture diagram of a space-channel strip attention module of a semantic segmentation method based on double-focus attention and adaptive frequency domain fusion provided by an embodiment of the present application;

[0044] Figure 4 An architecture diagram of a dynamic square attention path and a cubic strip attention path of a semantic segmentation method based on double-focus attention and adaptive frequency domain fusion provided by an embodiment of the present application;

[0045] Figure 5 An architecture diagram of an adaptive frequency domain fusion module of a semantic segmentation method based on double-focus attention and adaptive frequency domain fusion provided by an embodiment of the present application;

[0046] Figure 6 An architecture diagram of a hierarchical feature fusion architecture of a semantic segmentation method based on double-focus attention and adaptive frequency domain fusion provided by an embodiment of the present application;

[0047] Figure 7 A visualization result of an ADE20K dataset of a semantic segmentation method based on double-focus attention and adaptive frequency domain fusion provided by an embodiment of the present application;

[0048] Figure 8 A visualization result of a Cityscapes dataset of a semantic segmentation method based on double-focus attention and adaptive frequency domain fusion provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to further clarify the technical means and effects taken by the present application to achieve the predetermined inventive purpose, the specific implementation, structure, features and effects of a semantic segmentation method based on double-focus attention and adaptive frequency domain fusion according to the present application are described in detail as follows in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0051] The specific scheme of the semantic segmentation method based on double-focus attention and adaptive frequency domain fusion provided by the present application is described in detail below in combination with the drawings.

[0052] For better illustration, based on actual situation, the existing technology is mainly to improve the encoder, and in the background of lightweight semantic segmentation, the optimization of the decoder has its unique and irreplaceable advantages compared with the improvement of the encoder, which is the key to achieve the best balance between precision and efficiency.

[0053] Specifically, first, the transfer of the calculation bottleneck; in the semantic segmentation task of high-resolution input, the encoder usually performs down-sampling on the feature map to reduce the calculation amount. The decoder needs to up-sample the low-resolution and high-semantic features to restore the original resolution for pixel-level prediction. In the up-sampling process, especially when the high-resolution encoder features are fused by jump connection, there is a calculation bottleneck for actual deployment, and thus the optimization of the decoder can more directly and effectively reduce the overall calculation overhead and memory occupation.

[0054] Second, the core role of detail recovery; the encoder is mainly responsible for extracting abstract semantics, and its design trend is to sacrifice spatial details to obtain a larger receptive field and higher semantic representation. The final output of the segmentation task is extremely dependent on accurate object boundaries and other spatial details. The decoder is the last pass responsible for recovering and refining these details, so directly optimizing the feature fusion and up-sampling strategy of the decoder is more efficient and specialized than trying to make the encoder consider both abstract semantics and detail preservation, which can more directly improve the spatial quality of the final segmentation map.

[0055] Thirdly, the differentiated advantage of architecture innovation; the current mainstream research is highly concentrated on the improvement of the encoder, especially the backbone network, and its performance improvement has gradually entered the platform period. The decoder design such as simple MLP (Multilayer Perceptron, i.e. multilayer perceptron) or bilinear upsampling has not made breakthrough for many years, and there is great optimization potential. Therefore, innovation is carried out for the decoder, which can avoid incremental improvement on the encoder design in excessive competition, and provides a differentiated technical path for achieving significant performance improvement.

[0056] In view of this, the application focuses on optimizing the decoder, improving the segmentation performance through innovative design of the decoder, and realizing cooperative optimization of the encoder and the decoder based on multiple modules. While optimizing the encoder, the innovative improvement of the decoder is assisted.

[0057] Please refer to Figure 1 , which shows an architecture diagram of a semantic segmentation network of a semantic segmentation method based on dual-focus attention and adaptive frequency domain fusion provided by the first embodiment of the application. The method comprises:

[0058] Step S1: Obtain an image data set, and construct a semantic segmentation network based on dual-focus attention and adaptive frequency domain fusion, including a dual-focus attention module, an adaptive frequency domain fusion module and a hierarchical feature fusion architecture;

[0059] Step S2: input the image data set into the dual-focus attention module for preliminary enhancement and global context modeling, to obtain a first output result;

[0060] Step S3: input the first output result into the adaptive frequency domain fusion module for alignment and complementation of high-level semantic features and low-level details, to obtain a second output result;

[0061] Step S4: process the second output result through the hierarchical feature fusion architecture to generate a pixel-level classification result.

[0062] It can be explained that through the cooperation of the dual-focus attention module, the adaptive frequency domain fusion module and the hierarchical feature fusion architecture, a semantic segmentation network DFFSeg based on dual-focus attention and adaptive frequency domain fusion is constructed, which takes a feedforward encoder-decoder as a basic framework, and respectively introduces a dual-focus attention module (DFA, DualFocus Attention) and an adaptive frequency domain fusion module (ASF, AdaptiSpectra Fusion) two efficient modules.

[0063] DFFSeg adopts a progressive multi-scale feature processing strategy, which cooperatively utilizes spatial-channel attention mechanism and frequency domain analysis method to achieve accurate segmentation performance while reducing computational complexity. In the encoder stage, the DFA module is used to enhance feature representation, and in the decoder stage, the ASF module is used to efficiently fuse multi-level features.

[0064] For illustration, the image dataset is ADE20K and Cityscapes, two mainstream public datasets for semantic segmentation. ADE20K dataset is a large-scale scene parsing dataset released by MIT, covering 150 finely labeled semantic categories, including common objects, scenes, and materials. It contains 20,210 training images, 2,000 validation images, and 3,000 test images, with diverse image resolutions and complex scenes, suitable for testing the generalization ability of models in diverse environments. Cityscapes dataset focuses on urban scene understanding, containing 5,000 high-resolution street view images, labeled with 19 common objects. The dense semantic information of fine labeling is particularly suitable for tasks that require accurate boundary prediction. Preferably, in the subsequent processing, 2,975 are used for training, 500 for verification, and 1,525 for testing.

[0065] As an optional implementation, before the image dataset enters the semantic segmentation network based on double-focus attention and adaptive frequency domain fusion, the image dataset is preprocessed by grayscale, denoising, edge detection, etc. to obtain representative feature maps, so as to more efficiently cope with the subsequent processing process.

[0066] In particular, based on the drawings, H represents the height of the feature map, and its resolution increases by 4, 8, 16, and 32, indicating that the size of the feature map gradually decreases as the number of network layers increases. 、 、 、 C1, C2, C3, and C4 represent the number of channels corresponding to the respective levels, where the first layer does not participate in the subsequent processing process. C1, C2, C3, and C4 represent the number of channels corresponding to the respective levels, where the first layer does not participate in the subsequent processing process.

[0067] C1, C2, C3, and C4 represent the number of channels corresponding to the respective levels, where the first layer does not participate in the subsequent processing process. C1, C2, C3, and C4 represent the number of channels corresponding to the respective levels, where the first layer does not participate in the subsequent processing process. Resolution fusion is performed to balance the calculation amount and detail preservation, facilitate up-sampling to the original image size through lightweight bilinear interpolation, and significantly improve the inference speed at the end of the network.

[0068] Referring to Figure 2 Further, the double-focus attention module includes a space-channel strip attention module and parallel attention paths, and the attention paths are respectively a dynamic square attention path and a cubic strip attention path.

[0069] It can be explained that the double-focus attention module, i.e., the DFA module, adopts a serial-parallel hybrid architecture design, i.e., using DFA as a new MetaFormer module of the token mixer, integrating three mechanisms of space-channel strip attention, dynamic square attention and cubic attention, effectively improving the calculation efficiency while maintaining segmentation accuracy. Through the space-channel strip attention module, preliminary enhancement and global context modeling are performed, and after outputting the corresponding results, the parallel input is input to the dynamic square attention path and the cubic strip attention path, two complementary attention paths, and finally the outputs of each path are fused, reducing the calculation complexity while realizing multi-level feature focusing.

[0070] Further, in step S2, the following steps are included:

[0071] Referring to Figure 3 In step S21, the space-channel strip attention module fuses the space strip attention mechanism and the channel self-attention mechanism, and performs feature enhancement on the spatial features and channel features of the image dataset through two-stage parallel processing.

[0072] It is explained that the space-channel strip attention module includes two attention mechanisms, and feature enhancement is realized through two-stage processing.

[0073] Further, in step S21, the spatial features and channel features of the image dataset are enhanced through two-stage parallel processing, including:

[0074] In step S211, the spatial features of the image dataset are obtained based on the space strip attention mechanism, including the feature vectors in the height direction and the width direction, and are divided into several groups, and the feature vectors in the height direction and the width direction are processed through convolution kernels, and are sequentially subjected to splicing, group normalization and a gating function to generate a spatial attention map.

[0075] Specifically, the feature map is obtained by processing the image data set as described above, and then in this step, the feature map is explained based on the feature map. In the spatial attention stage, the module calculates the attention weight along the height and width directions respectively. First, the input feature map is average-pooled in the spatial dimension to obtain the spatial features, i.e., the feature vectors in the height direction and the width direction. Then, in this embodiment, the feature vectors are divided into four groups, and each group is processed by a 1D depth convolution kernel with a corresponding kernel size of 3, 5, 7, and 9 respectively to capture multi-scale context information. The processed features are then spliced, group-normalized (GroupNorm), and passed through a gating function, wherein the gating function includes a Softmax function for multi-classification problems or a Sigmoid function for binary classification problems, and then a final spatial attention map is generated. The corresponding calculation formula is:

[0076]

[0077]

[0078]

[0079] wherein, represents the spatial attention map; represents the input feature map; represents the feature vector in the height direction, represents the height direction; represents the feature vector in the width direction, represents the width direction; represents the convolution kernel; represents the splicing operation; represents the normalization processing; represents the gating function; , , , represents the four groups of feature vectors in the height direction. , , , represents the four groups of feature vectors in the width direction.

[0080] It can be explained that the upper indexes and respectively represent the low-value and high-value groups of the feature vectors in the four groups, wherein in the high-value group, the lower indexes , , Respectively represent small, medium and large corresponding spatial dimensions, that is, respectively represent smaller, medium size and larger size or spatial range in high value grouping, distinguish data processing methods and results in different dimensions by marking, ensure the accuracy and effectiveness of data.

[0081] Step S212: Based on the channel self-attention mechanism, the image data set is processed by adopting a down-sampling strategy, Query, Key and Value vectors are generated through group normalization and convolution kernel, and a channel weight vector is obtained.

[0082] Specifically, the down-sampling strategy is adopted to reduce the computational complexity, and the feature map resolution is reduced through adaptive average pooling or feature recombination operation; then Query, Key and Value vectors are generated through group normalization and 1x1 convolution, and the self-attention weight between channels, i.e. the channel weight vector, is calculated to accurately identify and enhance important channel features.

[0083] Step S213: Fusion of spatial attention map and channel weight vector to complete feature enhancement.

[0084] It is explained that the spatial attention map accurately captures the key areas in the image based on the height direction and width direction, highlights important visual information, and improves the model's ability to identify details; while the channel weight vector weights the important channels for enhancement, optimizes the feature extraction process, and efficiently processes multi-dimensional data. The fusion of the two enhances the expression ability of the features and improves the overall performance, making it perform better in complex scenes.

[0085] Please refer to Figure 4 , wherein figure (a) represents the architecture diagram of dynamic square attention path; figure (b) represents the architecture diagram of cubic strip attention path.

[0086] Step S22: input the enhanced features into the attention path, the dynamic square attention path processes the enhanced features in parallel through the low-frequency path and the high-frequency path, and fuses to obtain the first output component, and the cubic strip attention path processes the enhanced features in parallel in the horizontal direction and the vertical direction. The spatial dependence is integrated to obtain the second output component.

[0087] It can be explained that the dynamic square attention path and the cubic strip attention path both process the feature map for feature enhancement, so as to retain key information while improving the resolution of detail analysis.

[0088] Further, in step S22, obtaining the first output component and the second output component includes:

[0089] Step S221: dynamically filtering the low-frequency features in the enhanced features through a low-frequency channel to obtain smoothed features, retaining the high-frequency features in the enhanced features through a high-frequency channel, and adaptively fusing the smoothed features and the high-frequency features to obtain a first output component.

[0090] Specifically, the dynamic square attention path combines dynamic convolution with a gating mechanism; first, the low-frequency channel extracts global context information based on the input enhanced features through adaptive average pooling, and generates dynamic convolution kernel weights related to the input content using a 1x1 convolution; wherein, a grouping convolution strategy is adopted to divide the enhanced features into multiple groups and perform dynamic filtering processing on each group, effectively capturing local spatial features. In combination with the high-frequency channel, the smoothed features and the high-frequency features after filtering are adaptively fused using learnable parameters to achieve the effect of high-frequency and low-frequency component separation and enhancement. The corresponding calculation formula is:

[0091]

[0092]

[0093]

[0094] wherein, represents the first output component; , respectively represent the outputs of the low-frequency channel and the high-frequency channel; represents the enhanced features output by the spatial-channel strip attention module; represents the average pooling operation; represents the dynamic filtering function; , both represent learnable parameters; represents the identity matrix.

[0095] Further, the cubic strip attention path includes at least two spatial strip attention units; wherein, the horizontal strip attention unit and the vertical strip attention unit are cascaded respectively, corresponding to processing the spatial dependency relationship in the horizontal direction and the vertical direction, i.e., using a bidirectional strip processing strategy to enhance the spatial feature expression ability.

[0096] Step S222: both spatial strip attention units use strip convolution kernels and reflection padding techniques to process the spatial dependency relationship in the horizontal direction and the vertical direction respectively to obtain a second output component.

[0097] Specifically, the corresponding calculation formula is:

[0098]

[0099]

[0100]

[0101] wherein, denotes the second output component; , denote the output of the vertical and horizontal strip attention units, respectively; , denote the processing of the vertical and horizontal strip attention units, respectively; denotes the enhanced feature output by the spatial-channel strip attention module; , all denote learnable parameters.

[0102] It can be explained that each spatial strip attention unit uses a 1xK or Kx1 strip convolution kernel to perform content-aware processing on the input feature through dynamically generated filter weights; then, the boundary feature is processed using the reflection padding technique, which effectively avoids the boundary information loss problem caused by traditional zero padding; then, the processing results in the horizontal and vertical directions are fused through learnable weight coefficients, wherein, denotes that the spatial strip attention not only processes the horizontal and vertical directions respectively, but also needs to be fused through cross information to ensure that the attention features in each direction can influence each other, improve the expression ability of the features, and capture more rich spatial dependency relationships; the learnable parameters , are used to control the weighting to autonomously learn to retain information from the input and determine the retention strength; finally, the enhanced feature output by the spatial-channel strip attention module is integrated through a residual connection with the original feature , so that the module can capture long-distance spatial dependency relationships and local detail features at the same time, effectively improving the modeling ability of the model for complex spatial structures.

[0103] Step S23: fuse the first output component and the second output component to determine the first output result.

[0104] It is explained that the first output result is the output of the spatial-channel strip attention module and the parallel dynamic square attention path and the cubic strip attention path, that is, the first output component and the second output component are fused by adding the features to obtain the first output result. This setting shows that the design mode of multi-component cooperation can effectively improve the modeling ability of the network for multi-scale features while maintaining low computational complexity.

[0105] Understandably, traditional frequency domain processing methods typically rely on predefined fixed filters such as Hamming windows and Gaussian windows. These methods have two fundamental limitations: first, the filter parameters are not learnable and are difficult to adapt to the varying frequency domain feature distributions in different scenarios. For example, in noisy environments, fixed filters may not be able to effectively distinguish between signals and noise; second, the separation strategies for high-frequency and low-frequency components lack semantic awareness and cannot adjust the separation method according to specific application scenarios, leading to information loss or distortion. To address these issues, we propose the Adaptive Frequency Domain Fusion (ASF) module, which combines the physical priors of frequency domain decomposition with the adaptive capabilities of deep learning to achieve accurate capture of different frequency domain features.

[0106] Please see Figure 5 Furthermore, step S3 includes:

[0107] Step S31: Input the first output result into the adaptive frequency domain fusion module for channel compression.

[0108] Provide an explanation, through Convolutional channel compression can effectively reduce data redundancy, improve transmission speed, and reduce energy consumption; while the adaptive frequency domain fusion module, based on channel compression, can ensure optimal performance in different application scenarios.

[0109] Step S32: The compressed first output result is passed in parallel through the low-frequency path and the high-frequency path, and the corresponding low-frequency mask and high-frequency mask are output.

[0110] Understandably, in traditional methods, filters typically have fixed parameters, which makes them less adaptable to different features. For example, when processing signals of different frequencies, fixed filter parameters may not be able to effectively suppress noise or extract useful information. Furthermore, when faced with dynamically changing input signals, filters with fixed parameters are difficult to adjust in real time to optimize performance, resulting in poor signal processing effects and limiting the application of filters in complex environments. Therefore, a learnable convolution kernel is proposed, which transmits the first output result after processing to the low-frequency path and the high-frequency path for corresponding processing.

[0111] Further, step S32 includes:

[0112] Step S321: The low-frequency path design can learn the low-pass filter convolution kernel and extract global structure information to generate a low-frequency mask.

[0113] The adaptive frequency domain fusion module (ASF) uses a learnable two-dimensional convolutional kernel in the low-frequency path to achieve adaptive low-frequency feature extraction, efficiently capturing low-frequency components in the image while preserving important global structure and texture details to extract global structural information.

[0114] Further, a learnable low-pass filter kernel is designed, and the corresponding calculation formula is:

[0115]

[0116] wherein, represents the output of the learnable low-pass filter; represents the convolutional neural network operation; represents the input of the learnable low-pass filter; represents the learnable convolution kernel parameter.

[0117] It can be illustrated that the learnable convolution kernel parameter Through optimization by back propagation, the filter in the low-frequency channel can be dynamically adjusted according to the input features, which not only improves the accuracy of low-frequency information extraction, but also makes the extraction of frequency domain information more flexible and adaptive.

[0118] Specifically, the global structure information is obtained based on the learnable low-pass filter kernel, and a low-frequency mask is generated, and the corresponding calculation formula is:

[0119]

[0120] wherein, represents the low-frequency mask; represents a parameter, which is used to ensure that the spatial dimensions of the input and output feature maps are completely the same; represents the convolution kernel of the low-frequency channel.

[0121] It can be illustrated that when the convolution operation is performed, the convolution kernel slides along the input feature map; if no padding (padding = 0) is performed, the size of the output feature map will be slightly smaller than the input size, so in order to ensure that the input size = output size, a circle of pixels is padded around the input feature map based on the parameter padding.

[0122] Step S322: The high-frequency channel adopts residual high-pass filtering, and generates a high-frequency mask through residual calculation.

[0123] Specifically, the corresponding calculation formula is:

[0124]

[0125] wherein, represents the high-frequency mask.

[0126] It can be explained that the high-frequency channel is mainly responsible for extracting the detailed features in the image, such as edges, textures and local information, which is used to ensure the definition and texture of the image; while the low-frequency channel focuses on extracting the global structural information of the image, including the overall shape, color distribution and large-scale pattern, to understand the overall layout and background environment of the image, the two channels complement each other to ensure that both detailed and global information can be captured, providing more accurate feature representation for image semantic segmentation.

[0127] Step S33: Hierarchical fusion based on low-frequency mask and high-frequency mask, and corresponding content-aware up-sampling algorithm and residual compensation mechanism are adopted to optimize the features.

[0128] Specifically, in order to make full use of the low-frequency mask and the high-frequency mask, an adaptive feature fusion strategy based on dynamic weight is adopted to realize cross-band feature optimization through two-stage processing; in the low-frequency channel, a content-aware up-sampling algorithm is used to improve the resolution of global structural features, that is, a spatial attention weight is generated by the mask generator, and the corresponding calculation formula is:

[0129]

[0130]

[0131] Among them, represents the corresponding output of the low-frequency channel; represents the intermediate representation of the low-frequency channel; represents the activation function; represents the pixel reorganization process; represents the convolutional neural network operation; represents the compressed first output result; represents the residual compensation mechanism; represents the low-frequency mask; represents the convolution kernel of the low-frequency channel; represents the convolution group; represents the score of the low-frequency channel.

[0132] In the high-frequency channel, a residual compensation mechanism is designed to enhance the detailed features, and the corresponding calculation formula is:

[0133]

[0134]

[0135] Among them, represents the output of the high-frequency channel; represents the intermediate representation of the high-frequency channel; represents the high-frequency mask; represents the convolution kernel of the high-frequency channel; The score of the high-frequency path.

[0136] It can be explained that the score of the low-frequency path The score 1 of the high-frequency path is used to emphasize the detail information; the residual compensation mechanism The low-frequency information and the high-frequency information are compensated, and the high-frequency compensated result is added back to the mask through the residual connection.

[0137] Step S34: adjusting and fusing the optimized features to obtain a second output result.

[0138] It is explained that the optimized outputs of the high-frequency and low-frequency paths are dynamically adjusted and fused according to the characteristics of the input image to obtain the final feature representation, which ensures that the detail features and global information can be effectively combined and the adaptability to complex scenes is improved, especially in the detail capture of high-texture areas.

[0139] Preferably, the ASF module is designed to pay special attention to computational efficiency, and through adaptive adjustment and flexible fusion, the computational overhead caused by the introduction of a learnable low-frequency filter kernel and the increased complexity of the dual-path design is avoided, so that the entire module can maintain high computational efficiency while maintaining high accuracy.

[0140] Please refer to Figure 6 Further, in step S4, specifically:

[0141] The second output result is adjusted in channel and compressed in feature by 1x1 convolution based on the hierarchical feature fusion architecture to generate a pixel-level classification result.

[0142] It is explained that in this embodiment, a hierarchical architecture from bottom to top and progressive fusion is adopted to establish three stages of feature enhancement, multi-level fusion and output generation, and efficiently integrate multi-scale features from different stages of the encoder. Among them, the 1x1 convolution reduces the computational overhead while integrating cross-level features, ensuring the lightweight characteristics of the model.

[0143] It can be explained that in this application, the feature enhancement capability of the DFA module is combined with the frequency domain adaptive fusion mechanism of the ASF module, and a hierarchical feature fusion architecture is adopted to form an efficient processing path of "feature selection-frequency domain fusion-lightweight output", which not only effectively improves the representation ability of the overall constructed semantic segmentation network for multi-scale targets, especially in complex scenes and small target segmentation, while maintaining a low computational complexity, providing a reliable structural foundation for realizing real-time high-precision semantic segmentation.

[0144] It can be understood that in the traditional method, the semantic segmentation method is improved only for DFA module or ASF module alone, and there are some limitations accordingly.

[0145] The core advantage of DFA module is to model complex spatial dependencies such as long-range dependencies and multi-scale contexts in spatial domain, but there are some drawbacks: frequency domain information is not sensitive; convolution, attention and other operations in DFA are completely carried out in spatial domain, while details, edges and other high-frequency components and semantics, smooth areas and other low-frequency components in the image are coupled together in spatial domain, DFA has difficulty in explicitly separating and distinguishing these two types of information, which may lead to insufficient capture of key details or excessive processing of smooth areas; limited detail preservation; in the process of multiple spatial transformations and fusion, high-frequency detail information will inevitably be lost or smoothed; computational resources are concentrated in spatial domain; the computational overhead of DFA is mainly used to solve spatial relationship problems, and the computational budget is not optimally allocated to the equally important task of "separating" and "enhancing" different frequency domain components.

[0146] The core advantage of ASF module is to adaptively separate and fuse high-frequency and low-frequency information in frequency domain, and separate use also has some limitations: weak spatial context modeling capability; the frequency domain filtering operation of ASF learnable convolution kernel for low-pass filtering has globality and shift invariance, which leads to weak perception of spatial structure, object shape and position relationship and other context information, it can distinguish what texture details exist now, but cannot distinguish the position of texture; may introduce non-semantic noise; frequency domain separation is a signal processing-based method, which may not completely follow semantic boundaries, and ASF may mistakenly enhance irrelevant high-frequency noise inside uniform semantic areas (i.e. low frequency) or lose important semantic context in complex edge areas (i.e. high frequency), resulting in isolated noise points or holes in the segmentation result.

[0147] To this end, based on the built semantic segmentation network, the two modules are used collaboratively, DFA and ASF modules are combined in sequence in the decoder, forming DFA first and ASF later, producing a "complementary advantage" synergistic effect, which constitutes a more powerful processing flow.

[0148] Among them, DFA module acts as a "semantic relationship parser", which works in spatial domain and focuses on solving "what" and "where" problems, through powerful spatial attention mechanism, enhances the features of target objects, establishes semantic association between pixels, and suppresses irrelevant background information, providing a clear structure and semantic enhanced feature map for subsequent processing.

[0149] The ASF module acts as a "detail optimizer", working in the frequency domain, solving the problem of "how clear is the boundary", receiving the DFA-purified and smoothed features, and performing frequency domain decomposition and fusion on them. It can adaptively sharpen the object boundary of the DFA output to enhance the high-frequency components and maintain the consistency of its internal semantics to maintain the low-frequency components without destroying the spatial semantic relationship established by DFA.

[0150] It can be explained that the DFA module provides the ASF module with semantically enhanced and structurally clear feature basis, while the ASF module fills in the last piece of the puzzle of detail preservation and boundary sharpening for the output of the DFA module, ultimately achieving the best balance between computational efficiency and segmentation accuracy.

[0151] In addition, the effectiveness of DFA and ASF modules is comprehensively evaluated through ablation experiments of the system. The comparison indicators include parameter quantity, floating point operation quantity (FLOPs), and single-scale mIoU value. The average intersection over union (mIoU) is used as the main evaluation indicator, and Table 1 shows the effectiveness of each component in DFFSeg.

[0152] Table 1 Effectiveness of each component in DFFSeg

[0153]

[0154] It is explained that when using the DFA module alone, the model achieves an mIoU of 40.5% while maintaining a low computational overhead (3.8 GFLOPs). Although the baseline performance is slightly lower, the parameter quantity is only 2.9M, showing good computational efficiency advantage. The configuration using the ASF module alone achieves better balance, with an mIoU of 41.8% at a parameter quantity of 3.5M and a computational quantity of 4.6 GFLOPs, exceeding the baseline performance, which verifies the effectiveness of dynamic frequency domain decomposition in feature extraction. When both modules are integrated, the model exhibits a certain synergistic effect, achieving an mIoU of 43.2% at a parameter of 4.2M and a computational quantity of 5.3 GFLOPs, which is 2.1 percentage points higher than the baseline. This performance improvement is greater than the sum of the effects of the two modules used alone, fully proving the complementary advantages of the proposed hybrid attention mechanism and frequency domain analysis method. It is worth noting that in complex texture scenes and small target segmentation tasks, this synergistic effect is more pronounced, indicating that the module design can effectively enhance the model's ability to capture detailed features, indicating that the DFA module mainly improves the model's ability to model long-distance dependencies, while the ASF module effectively improves the preservation of high-frequency detailed features.

[0155] To better illustrate, to verify the effectiveness of the semantic segmentation method based on double focus attention and adaptive frequency domain fusion, experiments were evaluated based on the ADE20K and Cityscapes datasets, based on the mmsegmentation framework, using a double GTX 1080ti GPU for semantic segmentation network training, wherein the backbone network selects MSCAN (Main Subnet Communication Access Network), which realizes efficient feature extraction through the fusion of multi-scale depth separable convolution and spatial attention mechanism; MSCAN adopts a four-stage hierarchical structure, and the channel numbers of the four embedding layers are 64, 128, 256, and 512 respectively, and the MLP (Mass-Loading Parameter) expansion ratio is fixed at 4; the network depth can be configured as 3-4-6-3 or 3-4-18-3, corresponding to the DFFSeg-T (based on MSCAN-T) and DFFSeg-B (based on MSCAN-B) models, and the decoder part adopts a random initialization method.

[0156] The training process adopts a standard data augmentation strategy, including random horizontal flipping, 0.5-2.0 times random scaling, and random cropping; wherein the input size of the ADE20K dataset is fixed at 512x512, and the input size of the Cityscapes dataset is 1024x1024; the optimization process uses the AdamW optimizer, and iterates 160K times on the ADE20K and Cityscapes datasets, with batch sizes of 8 and 2 respectively, and the learning rate scheduling adopts a polynomial decay strategy, with an initial learning rate of , and a decay factor of 1.0. All experiments are completed under the same hardware environment and software configuration to ensure the comparability and reproducibility of the experimental results.

[0157] Specifically, based on the ADE20K dataset and the current mainstream lightweight segmentation method, Table 2 is obtained, and the comparison results of the DFFSeg model and the existing advanced method on the ADE20K dataset are obtained.

[0158] Table 2 Comparison results of DFFSeg model and existing advanced method on ADE20K dataset

[0159]

[0160] It can be explained that DFFSeg has certain advantages in the balance between accuracy and efficiency. In the lightweight model comparison, DFFSeg-T shows better performance. On the ADE20K dataset, DFFSeg-T achieves 43.2% mIoU with only 4.2M parameters and 5.3GFLOPs, compared with SegNeXt-T (4.3M parameters, 6.6 GFLOPs, 41.1% mIoU), the accuracy is improved by 2.1% while the calculation amount is reduced by 19.7%, which is due to the dynamic frequency domain decomposition mechanism of the ASF module, which effectively enhances the extraction ability of the semantic segmentation network to complex texture features; at the same time, the multi-branch design of DFA module effectively optimizes the calculation efficiency while maintaining the feature expression ability.

[0161] In terms of large models, DFFSeg-B performs outstandingly in terms of computational efficiency. On the ADE20K dataset, DFFSeg-B achieves 48.2% mIoU with 27.4M parameters and 30.6 GFLOPs of calculation amount, compared with SegNeXt-B (27.6M parameters, 34.9 GFLOPs, 48.5% mIoU), the accuracy only differs by 0.3 percentage points while the calculation amount is reduced by 12.3%, indicating that the DFA module realizes the optimal allocation of computing resources through the dynamic gating mechanism, which effectively reduces the computational overhead of the semantic segmentation network while maintaining the accuracy.

[0162] In particular, on the ADE20K dataset containing rich texture details, the accuracy advantage of DFFSeg series models is more obvious, which proves the effectiveness of the ASF module in frequency domain feature processing. Through the adaptive filtering mechanism, it realizes the accurate extraction of key frequency components, and DFFSeg realizes the best balance between accuracy and efficiency, verifying the effectiveness and generalization ability of the proposed method.

[0163] Next, to ensure the fairness of the comparison, the single-block RTX3090 GPU is used on the Cityscapes dataset, and the inference speed (FPS, Frames Per Second) of the whole image is tested under the input resolution of 2048x1024, as shown in Table 3. Compared with the current most advanced method on the Cityscapes dataset, the proposed DFFSeg-T shows certain advantages in inference efficiency while achieving higher mIoU.

[0164] Table 3 Comparison of FPS with current most advanced methods on Cityscapes dataset

[0165]

[0166] It can be illustrated that the FPS of DFFSeg-T reaches 20, which is 3.3 times of SegFormer-B0 (6 FPS) and 1.8 times of SegNeXt-T (11 FPS) respectively, which fully indicates that the method proposed in the application not only reduces the theoretical calculation amount by reducing FLOPs, but also effectively converts into a substantial improvement in the actual inference speed of GPU, which embodies excellent engineering deployment potential.

[0167] Please refer to Figure 7 and Figure 8 It can be illustrated that after introducing the DFA module and the ASF module, DFFSeg exhibits certain advantages in multiple aspects. Among them, in the segmentation results of slender objects such as wires, railings and branches, the DFA module effectively captures long-distance spatial dependency relationships through its unique hybrid attention mechanism, making the object edges more coherent and complete. The ASF module effectively improves the ability to retain high-frequency details through dynamic frequency domain decomposition, and performs particularly outstandingly in complex texture areas such as brick walls, leaves and ground paving. For large-scale continuous areas such as roads, water surfaces and walls, the synergistic effect of the two modules achieves smoother and more accurate segmentation results, avoiding the blocky artifacts commonly seen in traditional methods.

[0168] Particularly noteworthy is that in the identification of long-range small targets such as traffic signs, pedestrians and vehicles, the DFFSeg architecture design exhibits stronger robustness, capable of maintaining both local details and global consistency. The visualization results directly verify the complementary advantages of the DFA module in spatial feature modeling and the ASF module in frequency domain feature enhancement, and the organic combination of the two enables the entire semantic segmentation network to adaptively handle scenes of different scales and complexities.

[0169] It should be noted that the above-mentioned embodiments of the application are in the order of description only, and do not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or can be advantageous.

[0170] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A semantic segmentation method based on dual-focus attention and adaptive frequency domain fusion, characterized in that, The method comprises: An image data set is acquired, and a semantic segmentation network based on double-focus attention and adaptive frequency domain fusion is constructed, comprising a double-focus attention module, an adaptive frequency domain fusion module, and a hierarchical feature fusion architecture; The image data set is input into the double-focus attention module for preliminary enhancement and global context modeling, to obtain a first output result; The first output result is input into the adaptive frequency domain fusion module for alignment and complementation of high-level semantic features and low-level details, to obtain a second output result; The second output result is processed by the hierarchical feature fusion architecture to generate a pixel-level classification result.

2. The semantic segmentation method based on double focus attention and adaptive frequency domain fusion according to claim 1, characterized in that, The double-focus attention module comprises a space-channel strip attention module and parallel attention paths, which are respectively a dynamic square attention path and a cubic strip attention path.

3. The semantic segmentation method based on double focus attention and adaptive frequency domain fusion according to claim 2, characterized in that, The image data set is input into the double-focus attention module for preliminary enhancement and global context modeling, to obtain a first output result, comprising: The space-channel strip attention module fuses a space strip attention mechanism and a channel self-attention mechanism, and performs feature enhancement on spatial features and channel features respectively through double-stage parallel processing of the image data set; The enhanced features are input into the attention paths, the dynamic square attention path processes the enhanced features through parallel processing of a low-frequency channel and a high-frequency channel, and fuses the enhanced features to obtain a first output component, and the cubic strip attention path processes the enhanced features in parallel to integrate the spatial dependency in the horizontal direction and the vertical direction to obtain a second output component; The first output component and the second output component are fused to determine the first output result.

4. The semantic segmentation method based on double focus attention and adaptive frequency domain fusion according to claim 3, characterized in that, The double-stage parallel processing of the image data set to perform feature enhancement on spatial features and channel features comprises: Based on the space strip attention mechanism, the image data set is subjected to average pooling in the spatial dimension to obtain spatial features, including feature vectors in the height direction and the width direction, and is divided into several groups, and the feature vectors in the height direction and the width direction are processed by convolution kernels in turn, and are sequentially subjected to splicing, group normalization, and a gating function to generate a spatial attention map; Based on the channel self-attention mechanism, the image data set is processed by a downsampling strategy, and Query, Key, and Value vectors are generated by group normalization and convolution kernels to obtain a channel weight vector; The spatial attention map and the channel weight vector are fused to complete feature enhancement.

5. The semantic segmentation method based on double focus attention and adaptive frequency domain fusion according to claim 3, characterized in that, The cubic strip attention path comprises at least two space strip attention units.

6. The semantic segmentation method based on double focus attention and adaptive frequency domain fusion according to claim 5, characterized in that, Obtaining the first output component and the second output component comprises: The low-frequency features in the enhanced features are dynamically filtered by the low-frequency channel to obtain smooth features, the high-frequency features in the enhanced features are reserved by the high-frequency channel, and the smooth features and the high-frequency features are adaptively fused to obtain the first output component; Both of the space strip attention units use strip convolution kernels and reflection padding technology to process the spatial dependency in the horizontal direction and the vertical direction respectively to obtain the second output component.

7. The semantic segmentation method based on dual-focus attention and adaptive frequency domain fusion according to claim 1, characterized in that, The first output result is input into the adaptive frequency domain fusion module for alignment and complementation of high-level semantic features and low-level details, to obtain a second output result, comprising: The first output result is input into the adaptive frequency domain fusion module for channel compression; The compressed first output result is parallelly passed through a low-frequency channel and a high-frequency channel, and low-frequency masks and high-frequency masks are correspondingly output; Based on the low-frequency masks and the high-frequency masks, hierarchical fusion is performed, and content-aware up-sampling algorithms and residual compensation mechanisms are correspondingly adopted to optimize features; The adjusted and fused optimized features are obtained as a second output result.

8. The semantic segmentation method based on double focus attention and adaptive frequency domain fusion according to claim 7, characterized in that, The compressed first output result is parallelly passed through a low-frequency channel and a high-frequency channel, and low-frequency masks and high-frequency masks are correspondingly output, including: The low-frequency channel is designed to learn a low-pass filter convolution kernel, extract global structure information, and generate low-frequency masks; The high-frequency channel adopts residual high-pass filtering to generate high-frequency masks through residual calculation.

9. The semantic segmentation method based on double focus attention and adaptive frequency domain fusion according to claim 8, characterized in that, The low-pass filter convolution kernel is learned, and the corresponding calculation formula is: ； wherein, denotes the output of the learnable low-pass filter; denotes a convolutional neural network operation; denotes the input of the learnable low-pass filter; denotes the learnable convolutional kernel parameter.

10. The semantic segmentation method based on dual-focus attention and adaptive frequency domain fusion according to claim 1, characterized in that, The hierarchical feature fusion architecture processes the second output result to generate a pixel-level classification result, specifically: Based on the hierarchical feature fusion architecture, the second output result is processed through 1×1 convolution for channel adjustment and feature compression to generate a pixel-level classification result.

Images