A hybrid network framework fusing spatial and frequency domain information and application thereof

Abstract

This invention belongs to the field of image segmentation technology and relates to a hybrid network framework and its application that integrates spatial and frequency domain information. The framework includes four symmetrically arranged downsampling layers and four upsampling layers. Each downsampling layer is connected to a MAP module at its back end, and the first downsampling layer uses block embedding. The first three upsampling layers are each equipped with a MAP module at their front end, and the last three upsampling layers are connected to the first upsampling layer through an MFF module. The MAP module of each of the first three downsampling layers is added to the output of each of the last three upsampling layers through an FFE module, and the MAP module of the last downsampling layer is directly connected to the last upsampling layer through an FFE module. This architecture combines FFE and MAP modules to enhance spatial and frequency domain feature information, and designs TAA and MFF modules to effectively combine low-level detail information with high-level semantic information.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, specifically relating to a hybrid network framework that integrates spatial and frequency domain information and its applications. Background Technology

[0002] In the field of 3D medical image segmentation, most existing models primarily focus on learning spatial features, often neglecting the potential advantages of frequency domain information. These models typically rely on convolutional layers and attention mechanisms to capture local and global features in the spatial domain, which is crucial for accurately segmenting complex anatomical structures. However, this approach usually has certain limitations, especially when dealing with subtle boundaries or irregular shapes, often struggling to simultaneously preserve global context and detailed information.

[0003] Medical image segmentation, especially 3D Volume segmentation remains a significant challenge in medical imaging and is widely used in tasks such as tumor detection, organ localization, and pathological analysis. Traditionally, deep learning methods have employed similar techniques... U-Net An encoder-decoder architecture is used to solve this problem, where the encoder is responsible for learning. 3D The image is represented in low dimension, and the decoder maps these representations back to voxel-level segmentation. However, early methods based on convolutional neural networks... CNN Traditional methods, which use convolutional and deconvolutional layers in both the encoder and decoder, face challenges in achieving accurate segmentation due to their limited receptive field and dependence on local features. To overcome these limitations, recent research has focused on extending... CNN The receiver domain enhances the ability to model context. Nevertheless, CNN The method still faces challenges such as a limited receptive domain, fixed weights, and local connectivity, which may limit the model's ability to capture global dependencies. Furthermore, although based on... 3D CNN This method has advantages in parameter efficiency, but its computational cost is large, leading to... FLOPs This increases the speed of reasoning, thus slowing down the process.

[0004] In contrast, based on Transformer These methods have a clear advantage in global modeling, and although their complexity has increased in recent years, they still demonstrate competitive performance when handling complex tasks. These methods rely on global self-attention. Self-Attention However, its computational complexity increases quadratically with the input size, especially when dealing with inputs containing... 3D The computational cost is particularly prominent in medical image segmentation tasks involving volumetric data. To address these challenges, an increasing number of studies are exploring the combination of... CNNHybrid architectures combining the advantages of transformers. Recent research aims to leverage the strengths of both by combining local convolutional features with global attention mechanisms. For example, some methods employ transformer-based encoder-decoder structures, while others design hybrid blocks to process the encoder and decoder separately. However, most of these architectures focus on improving segmentation accuracy, but at the same time significantly increase the number of model parameters and... FLOPs This leads to insufficient robustness. We believe the main reason for this lack of robustness lies in the inefficient self-attention design, which is particularly prominent in volumetric medical image segmentation tasks. Furthermore, existing hybrid architectures fail to adequately capture the dependencies between spatial and channel features, which are crucial for improving segmentation accuracy. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a hybrid network framework that integrates spatial domain and frequency domain information and its applications.

[0006] To achieve the objectives of this invention, the following technical solutions are adopted.

[0007] A hybrid network framework integrating spatial and frequency domain information is proposed, employing a hierarchical encoder and decoder framework. The encoder and decoder include four symmetrically arranged downsampling layers and four upsampling layers, wherein the backend of each downsampling layer is connected to a... MAP The module uses block embedding in the first layer of downsampling; each of the first three upsampling layers is equipped with a front-end... MAP The module, with upsampling of the last three layers all passing through a... MFF The module is connected to the topmost upsampling layer; the first three downsampling layers each... MAP The modules are respectively passed through FFE The module is added to the output of each of the last three upsampled layers, and the output of the last downsampled layer is... MAP Module through FFE The module is directly connected to the last layer upsampling; where:

[0008] Embedded patch, used to downsample the first layer from the input 3D The feature map extracted from the image is segmented into non-overlapping patches, and the non-overlapping patches are projected onto different channel dimensions to generate the output feature map;

[0009] MAP This module enhances feature learning in image segmentation tasks. It includes: multi-scale convolutions capable of capturing features at different scales, and features that adaptively adjust their contribution based on the importance of features at different scales. MWF Modules, and position encoding that can perceive three-dimensional spatial coordinates at different scales;

[0010] FFE The module is used to enhance the low-frequency, mid-frequency and high-frequency information in the input features. It includes a dynamic frequency filter that can perform channel weighting on the input features to generate the frequency weight of each channel, and the frequency weight is divided into three frequency band weights: low frequency, mid frequency and high frequency; as well as low-frequency mask, mid-frequency mask and high-frequency mask that can obtain feature values ​​of different frequency bands from the frequency domain.

[0011] MFF The module is used to upsample the last three layers respectively. H axis, W axis, D Perform in three directions along the axis TAA The process involves aligning the channels spatially, then fusing the low-level features obtained through frequency and spatial domain fusion. This is followed by fusing the low-level features with the high-level semantic information obtained from the first-layer upsampling to produce an enhanced feature map. This enhanced feature map contains information that allows attention calculations to be performed using different axes at different decoding stages. TAA mechanism.

[0012] Furthermore, the process of generating the output feature map includes the following steps:

[0013] S21. The feature map extracted from the input 3D image by downsampling the first layer. The partition is divided into non-overlapping patches, which are expressed as:

[0014] ,

[0015] In the formula: P1, P2, and P3 represent the resolution of each patch, respectively;

[0016] S22. Project the patch obtained in step S21 onto C channel dimensions to generate the output feature map of the first-layer downsampling, wherein the output feature map is expressed as:

[0017] ,

[0018] In the formula: H, W, D, and C represent the height, width, depth, and number of channels of the feature, respectively.

[0019] S31. Enhance the input features by improving the projection layer. Preliminary feature mapping and normalization are performed; wherein: the enhanced projection layer is composed of a 1×1×1 convolution, GroupNorm and GELU Composed of activation functions;

[0020] S32. Using a constructed multi-scale convolution, features at three scales—local details, intermediate regions, and global context—are extracted from the input image processed in step S31. Wherein: the multi-scale convolution is a parallel group of depthwise convolutions, expressed as follows:

[0021] ;

[0022] In the formula: Conv k×k×k express kernal size for k Depth convolution, by setting the number of groups to the number of input channels. C This achieves a reduction in parameters;

[0023] S33. The features extracted in step S32 at the three scales are concatenated, and weights for each scale are generated through dynamic convolution. The features at the three scales are then weighted and fused using these weights. Wherein:

[0024] The splicing is expressed as follows:

[0025] ;

[0026] In the formula: concat Indicates feature splicing;

[0027] The weights are expressed as follows:

[0028] ;

[0029] In the formula: Softmax and GELU Representing different activation functions;

[0030] The weighted fusion is expressed as follows:

[0031] ;

[0032] In the formula: Represents matrix dot product;

[0033] S34. Add the location code to the weighted fusion feature obtained after weighted fusion in step S33 in the following manner:

[0034] ;

[0035] In the formula: Interpolate tri ( ) represents trilinear interpolation;

[0036] Basic Location Code PE base ∈R C×8×8×8 The system adapts to arbitrary input sizes using trilinear interpolation, and after interpolation, it undergoes layer normalization processing, injecting position information in the form of residuals.

[0037] ;

[0038] In the formula: LayerNorm( ) represents layer normalization. The scaling factor that can be learned;

[0039] S35. The features enhanced in step S34 are passed through a projection layer, subjected to a 1×1×1 convolution transformation, and then added... dropout The operation involves regularizing the features;

[0040] S36. The features projected in step S35 are fused with the original input features through residual connection to obtain the output feature map.

[0041] Furthermore, the specific process of enhancing low-frequency, mid-frequency, and high-frequency information is as follows:

[0042] Using a dynamic frequency filter, frequency weights for each channel are generated by weighting the input features by channel. W i ∈R B×3×C×1×1×1 The weight of each channel is divided into three frequency bands: low frequency, mid frequency, and high frequency; in the channel weighting part, the following is used: Sigmoid An activation function is used to maintain numerical stability; wherein the frequency weights are expressed as:

[0043] ;

[0044] In the formula: i ∈ {low, mid, high} , express Sigmoid function, F(X) These are channel weights generated through dynamic convolution;

[0045] Simultaneously, the input feature map is transformed from the spatial domain to the frequency domain through Fourier transform, and decomposed into low-frequency, mid-frequency, and high-frequency features:

[0046] ;

[0047] In the formula: F This is represented as a Fourier transform operation;

[0048] By defining a low-frequency mask M low , intermediate frequency M mid and high-frequency mask M high The frequency domain features are weighted for low frequency, mid frequency, and high frequency respectively to obtain the decomposed frequency domain features; wherein: the low frequency mask M low , intermediate frequency M mid and high-frequency mask M high They are expressed as follows:

[0049] ;

[0050] In the formula: for sigmoid function, D It is the calculated three-dimensional distance matrix, representing the distance from each location in the input image to the frequency center. and These are the thresholds for low frequency and high frequency, respectively. It is a very small constant of 1e-6, used to avoid numerical instability;

[0051] The inverse Fourier transform is used to restore the low-frequency, mid-frequency, and high-frequency features to the spatial domain:

[0052] ;

[0053] In the formula: F -1 This is represented as an inverse Fourier transform operation;

[0054] The features after separating frequency bands using Fourier transform are then convolved separately to enhance the information of each frequency band; each frequency band is enhanced through convolution operation. C low , C mid and C high The data is processed and fused with the original input features through residual connections. Then, weighted values ​​generated by a dynamic frequency filter are used to weight and fuse the enhanced low-frequency, mid-frequency, and high-frequency features. A spatial attention mechanism is applied to further enhance the spatial expressiveness of the features, ultimately yielding the enhanced feature map.

[0055] ;

[0056] ;

[0057] In the formula: SpatialAttention ( ) represents the spatial attention mechanism.

[0058] Furthermore, the aforementioned TAA The specific process of operation:

[0059] Through 1×1 3D Convolution pairs input features X ∈ R B×C×H×W×D Perform a query Q ,key K Sum V The projection is obtained Q, K, V ∈ R B×H×W×D×C ;

[0060] For each upsampling, attention is calculated using either the depth, width, or height axis:

[0061] ;

[0062] In the formula: d for Query and Key The dimension;

[0063] The outputs of each attention head are merged to generate a feature map. ;

[0064] The original input feature map is connected to the attention-processed feature map via residual connections. The feature maps are then fused, and the fused feature maps are processed through a linear transformation to obtain the output feature map:

[0065] ;

[0066] In the formula: γ It is a trainable parameter used to control the weighting between output and input features.

[0067] Furthermore, the process of obtaining the enhanced feature map:

[0068] The above TAA The depth, width, and height feature maps obtained from the operation are aligned in the channel space, and each input feature is transformed into the frequency domain to obtain the amplitude spectrum and phase spectrum.

[0069] The amplitude spectrum is enhanced by 1×1×1 convolution to strengthen low-frequency and high-frequency components;

[0070] pass 3D The inverse fast Fourier transform converts the enhanced frequency domain features back to the spatial domain:

[0071] ;

[0072] In the formula: i ={1,2,3}, F Indicates Fourier transform, F -1 Indicates the inverse Fourier transform. j It is the imaginary unit, representing , angle It involves extracting phase information;

[0073] After frequency domain processing, the output frequency domain features are fused with the original spatial domain features;

[0074] By using residual connections, spatial features and frequency domain features are added together to obtain a fused feature map;

[0075] The fused feature map from the three input feature maps is summed to obtain the enhanced feature map.

[0076] Furthermore, the enhanced feature map is passed to the first layer for upsampling and segmentation:

[0077] ;

[0078] In the formula: This indicates a summation.

[0079] Furthermore, the aforementioned FFE Input feature map of the module X The shape is B × C × H × W × D ,in: B For batch size, C For the number of channels, H, W, D These are the spatial dimensions of the image.

[0080] Furthermore, the loss function of the framework is:

[0081] ;

[0082] In the formula: For cross-entropy loss, These are the weighting coefficients; For the loss of soft dice, These are the weighting coefficients; C express the number of classes , V express the number of voxels , G c,v and P c,v They represent Class C exist voxel V place ground truth And output probability.

[0083] A hybrid network framework that integrates spatial and frequency domain information in segmentation 3D Applications in medical imaging.

[0084] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0085] The present invention proposes MedFreq-Net It is a hybrid that enhances frequency domain information. CNN and Transformer The model is designed to achieve accuracy. 3D Medical image segmentation;

[0086] MedFreq-Net Combined FFE and MAPModules were designed to enhance spatial and frequency domain feature information, and a design was developed. TAA and MFF The module effectively combines low-level detailed information with high-level semantic information;

[0087] With existing SOTA Model comparison, MedFreq-Net exist ACDC, Synapse and BraTS Excellent segmentation results were achieved on three datasets, while maintaining a high level of efficiency;

[0088] MedFreq-Net The potential benefits in clinical practice and research remain enormous, and further research and development will continue. MedFreq-Net It will play an even more important role in the field of medical image segmentation. Attached Figure Description

[0089] Figure 1 Hybrid networks that integrate spatial and frequency domain information MedFreq-Net The framework diagram; where: A is the multi-scale adaptive sensing module. MAP; B is the frequency function enhancement module. FFE C represents the multi-scale frequency fusion module. MFF D represents the multi-scale weight fusion module. MWF; E Fourier transform FFT; F Inverse Fourier Transform IFFT ;

[0090] Figure 2 Frequency function enhancement module FFE Framework diagram;

[0091] Figure 3 For multi-head attention mechanism TAA Framework diagram;

[0092] Figure 4 for ACDC Visual comparison chart;

[0093] Figure 5 for Synapse Visual comparison chart;

[0094] Figure 6 for BraTS Visual comparison chart;

[0095] Figure 7 This is a graph analyzing ablation experimental data on the ACDC dataset; where: A represents the multi-scale adaptive sensing module. MAP B is the frequency function enhancement module. FFE C represents the multi-scale frequency fusion module. MFF ;

[0096] Figure 8 A comparison chart of AvgDSC-FLOPS-Params for each model;

[0097] exist Figure 4 - Figure 6 and Figure 8 In the model, Model 1 is nnUNet; Model 2 is UNETR; Model 3 is Swin-UNETR; Model 4 is nnFormer; Model 5 is CoTr; Model 6 is MedNeXt-M-K3; Model 7 is MedNeXt-M-K5; Model 8 is MedFreq-Net; and Model 9 is Ground Truth. Detailed Implementation

[0098] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0099] Example 1, as Figure 1 to Figure 3 As shown, the structure of a hybrid network framework that integrates spatial and frequency domain information is as follows: Figure 1 As shown, the framework MedFreq-Net It combines a hierarchical encoder-decoder architecture, based on U-Net The design, and referenced UNETR The structure of the framework consists of four main parts: encoder, ... FFE decoder and TAA The encoder consists of four stages, first of which each... 3D image Divide into non-overlapping segments patch: (in: P 1 、P 2 、 P 3 Indicate each patch (resolution). Next, these patch Projected to C Generate feature maps using each channel dimension. and access the design MAP Block. We use with nnFormer The same patch Resolution (4,2,2). The subsequent three stages include a downsampling layer with a resolution reduced by a factor of 2, and are equipped with... MAP Downsampling leads to reduced image resolution and loss of spatial detail. To address this issue, we introduce a block between the encoder and decoder. FFEThe module transforms the spatial domain feature map into the frequency domain and extracts the low, mid, and high frequency components. It enhances information through features at different resolutions, thus transmitting it more effectively to the decoder. The decoder also consists of four stages, each including an upsampling layer that doubles the resolution, followed by a... MAP The blocks (except for the last decoder) are processed. Finally, low-level details and high-level semantic information are efficiently combined through multi-scale feature fusion. The output is then processed by convolutional layers to generate a voxel-by-voxel final segmentation mask.

[0100] FFE The module is a frequency function enhancement module, such as Figure 2 As shown. Traditional CNN and Transformer Convolutional operations rely on local receptive fields and cannot effectively capture global feature information. Furthermore, ordinary skip connections merely connect low-level and high-level features through simple concatenation and summation; while they can pass some boundary information, their effectiveness is limited when dealing with complex boundaries. Fourier transform can convert feature maps from the spatial domain to the frequency domain, enabling the model to capture global information in the image, especially in the high-frequency components. Through Fourier transform, the image is converted from the spatial domain to the frequency domain, allowing the model to separate the low-frequency, mid-frequency, and high-frequency components and perform feature enhancement at different frequencies. Low-frequency information typically contains the overall structure and general shape of the image, helping to distinguish between background and foreground and reducing the impact of background noise; mid-frequency information contains finer structures in the image, providing detailed information about complex structures, such as boundaries between organs and subtle tumors, which is crucial for enhancing structural contrast in the image; high-frequency information helps to accurately capture details, edges, and texture information in the image, making it an indispensable part of achieving fine segmentation. Therefore, we propose... FFE The module uses Fourier transform to divide the image into multiple frequency bands, and processes and enhances each frequency band separately, thereby helping the network to effectively restore image details at different frequency levels and improve segmentation accuracy.

[0101] Specifically, we first use a dynamic frequency filter to generate the frequency weight of each channel by weighting the input features by channel. W i ∈R B×3×C×1×1×1 The weight of each channel is divided into three frequency bands (low frequency, mid frequency, and high frequency). In the channel weighting part, we use... Sigmoid An activation function is used to maintain numerical stability. The formula is as follows:

[0102] ;

[0103] In the formula: i ∈ {low, mid, high} , express Sigmoid function, F(X) These are channel weights generated through dynamic convolution.

[0104] Next, the shape of the input feature map x is B × C × H × W × D ,in: B C is the batch size, and C is the number of channels. H, W, D These represent the spatial dimensions of the image. Through Fourier transform, we convert the input feature map from the spatial domain to the frequency domain, represented as:

[0105] ;

[0106] In the formula: F This is represented by a Fourier transform operation. Through this transform, the image is decomposed into low-frequency, mid-frequency, and high-frequency components. To obtain feature values ​​of different frequency bands from the frequency domain, we define a low-frequency mask in the frequency domain. M low , intermediate frequency M mid and high-frequency mask M high :

[0107] ;

[0108] In the formula: for sigmoid function, D It is the calculated three-dimensional distance matrix, representing the distance from each location in the input image to the frequency center. and These are the thresholds for low frequency and high frequency, respectively. A very small constant is 1e-6 This is used to avoid numerical instability.

[0109] Next, the frequency domain features are weighted for low, mid, and high frequencies using a mask to obtain the decomposed frequency domain features. Then, the inverse Fourier transform is used to restore the low, mid, and high frequency features to the spatial domain.

[0110] ;

[0111] In the formula: F -1 This is represented by the inverse Fourier transform operation. After restoring the features to the spatial domain, the features separated by the Fourier transform are then subjected to convolution processing to enhance the information of each frequency band. Each frequency band undergoes an enhancement convolution operation. C low , C mid and C highThe data is processed and fused with the original input features through residual connections. Finally, the enhanced low-frequency, mid-frequency, and high-frequency features are weighted and fused using the weighted values ​​generated by the dynamic frequency filter, and a spatial attention mechanism is applied to further enhance the spatial expressiveness of the features, ultimately yielding the enhanced feature map.

[0112] ;

[0113] ;

[0114] Multi-scale adaptive sensing:

[0115] In traditional U In this architecture, the encoder extracts high-level semantic features of the image through progressive downsampling. As the image resolution gradually decreases, spatial details in the feature maps may not be fully preserved; similarly, some detailed information is also lost during the decoder's upsampling process. To address this issue, we propose... MAP The module, a novel mechanism applied after each encoder and before each decoder, is designed to enhance feature learning in 3D medical image segmentation tasks. This module combines multi-scale convolution, dynamic weighted fusion, and positional encoding to effectively capture and utilize spatial and frequency domain information in medical images.

[0116] Multi-scale convolution enables the model to capture features at different scales, including local details, intermediate regions, and global contextual features, thereby enhancing its ability to perceive complex structures in medical images. By dynamically weighting and fusing features at different scales, the model can adaptively adjust the contribution of features according to their importance, reducing the interference of unimportant features on the model. This is crucial for features such as organ boundaries or minor lesions in medical images. The introduction of positional encoding allows the model to perceive three-dimensional spatial coordinates during training, helping it to better model anatomical location priors in segmentation tasks and further enhancing the model's spatial perception capabilities.

[0117] First, input 3D Medical Images ,in: B For batch size, C For the number of channels, H, W, D These represent the image's height, width, and depth, respectively. Then, it is processed by a 1×1×1 convolution... GroupNorm and GELU An enhanced projection layer composed of activation functions performs preliminary feature mapping and normalization on the input features. Next, we propose... Multi- scale Weight Fusion ( MWF To capture spatial features at different scales simultaneously, we constructed parallel deep convolutional groups:

[0118] ;

[0119] In the formula: Conv k×k×k express kernal size for k Depth convolution, by setting the number of groups to the number of input channels. C This reduces the number of parameters. Through these three different depthwise separable convolutions, the model can extract local details, intermediate regions, and global contextual features of the image, respectively.

[0120] Because different anatomical structures exhibit varying scale sensitivities, we introduce a dynamic weighted fusion mechanism to fuse features at different scales. First, features from all scales are concatenated using a 1×1×1 convolution:

[0121] ;

[0122] Next, weights for each scale are generated through dynamic convolution:

[0123] ;

[0124] In the formula: W The weights are generated through convolution operations, representing the importance of features at each scale. To ensure the stability of the weight values, we used... Softmax The function is then used to weight and fuse the features across the three scales.

[0125] ;

[0126] In the formula: ⊙ denotes element-wise multiplication. This mechanism enables the network to dynamically adjust the contribution of features at different scales, such as increasing the weight of local features at the edges of different organs in the abdomen, and increasing the weight of global features in the overall structure of the organs.

[0127] Next, we introduce positional encoding to enhance the model's perception of spatial relationships. Positional encoding is a trainable parameter. p ∈ R B×C×D×H×W This is used to represent the spatial information of each location in the image. To make the location encoding adaptable to input images of different sizes, we adjust it to the same size as the input image through interpolation.

[0128] Location coding p The features are added to the fused features in the following way:

[0129] ;

[0130] In the formula: Interpolatetri （） This is a trilinear interpolation.

[0131] Basic position encoding PE base ∈R C×8×8×8 The system adapts to arbitrary input sizes using trilinear interpolation. After interpolation, it undergoes layer normalization and finally injects position information in the form of residuals.

[0132] ;

[0133] In the formula: LayerNorm() For layer normalization; This is a learnable scaling factor. This design enables the network to perceive three-dimensional spatial coordinates, and... CT / MRI Prior modeling of the anatomical location of the organs is crucial.

[0134] Finally, the location-encoded enhanced features are passed through the final projection layer, undergoing a 1×1×1 convolution transformation, and then... dropout The model then performs further regularization on the features. Subsequently, it fuses the projected features with the original input features through residual connections, ultimately obtaining the output feature map.

[0135] Traditional convolutional neural networks CNN Feature extraction is typically performed within local regions, but global contextual information is often overlooked, especially in the depth direction. D Altitude direction H and width direction W The information above. To solve this problem, we propose a method TriAxial Attention mechanism( TAA This mechanism uses multi-head attention to calculate attention on each axis separately, enabling the network to capture global features of the space more accurately.

[0136] exist U In this architecture, the main function of the decoder is to gradually restore the spatial resolution of the image and fuse low-level and high-level features from the encoder.

[0137] In the initial stages of the decoder, it processes coarser feature maps, which typically contain less spatial information. (Depth axis) D The attention mechanism helps the decoder capture depth information at different levels, thus better recovering spatial features in the depth dimension. Therefore, the first decoder uses the depth axis. D Attention helps capture and fuse information at different levels in the depth direction of an image.

[0138] In the second decoder stage, the feature maps have a higher spatial resolution and the image width is... W This becomes particularly important. Using attention along the width axis helps the decoder better focus on important features of the image in the width direction, thereby enhancing width-related detail features in the spatial domain. Therefore, the second decoder uses the width axis. W Attention can more effectively restore details in the width direction of an image.

[0139] In the final decoder stage, the spatial resolution of the image has been restored to a higher level, at which point the image height... H Orientation becomes crucial. Use the height axis. H The attention mechanism helps the decoder focus on important features in the image along the height direction, thus better recovering height-related details. The third decoder uses the height axis. H Attention can effectively restore the detailed information of an image in the vertical direction.

[0140] exist Figure 3 In the middle we introduced TAA The framework diagram, firstly through 1×1 3D Convolution pairs input features X ∈ R B ×C×H×W×D Perform a query Q ,key K Sum V The projection is obtained Q, K, V ∈ R B×H×W×D×C .

[0141] exist TAA In this context, attention computation relies on slices along different spatial axes. Specifically, for each decoder, a different axis (depth) is used. D ,width W or height H Attention calculations are performed.

[0142] ;

[0143] In the formula: d for Query and Key Dimensions.

[0144] Next, we merge the outputs of each attention head to generate the final feature map. Finally, the original input and the attention-processed output are fused through residual connections. The fused output is then processed by a linear transformation to obtain the final output feature map.

[0145] ;

[0146] In the formula: γ It is a trainable parameter used to control the weighting between output and input features.

[0147] Multi-scale frequency fusion:

[0148] exist U In this architecture, the decoder is responsible for restoring high-level semantic information into spatial information of the image. However, shallow decoders often lose detailed features (such as edges, textures, and local structures) when capturing features. Top-level decoders can gather more contextual information, but if the model relies solely on the output of the top-level decoder, it will not be able to fully utilize the complementarity of low-level and high-level features, resulting in an inability to effectively handle targets of different sizes at multiple scales. To further enhance the model's performance, this invention proposes multi-scale frequency fusion (… MFF This method first processes the outputs of the three shallow decoders. TAA The model then performs operations, and by fusing these outputs in the frequency and spatial domains, it finally fuses them with the features of the top-level decoder, thereby enhancing the model's feature representation capabilities.

[0149] First, for the feature maps of different resolutions obtained from the three decoders, we use 1×1×1 convolutions to reduce the number of channels and standardize the feature map size. Next, trilinear interpolation is used to adjust the spatial dimensions of all feature maps to the same size for easier subsequent processing.

[0150] Each input feature is first transformed into the frequency domain, yielding amplitude and phase spectra. Subsequently, the amplitude spectrum is enhanced using a 1×1×1 convolution to amplify low- and high-frequency components. Finally, through... 3D Inverse Fast Fourier Transform IFFT The enhanced frequency domain features are then converted back to the spatial domain.

[0151] ;

[0152] In the formula: i={1,2,3} , F Indicates Fourier transform, F -1 Indicates the inverse Fourier transform. j It is the imaginary unit, representing , angle It extracts phase information.

[0153] ;

[0154] Verification experiment:

[0155] 1. Data:

[0156] In order to evaluate 3DMedical image segmentation model MedFreq-Net Performance, we ACDC, Synapse and BraTS Experiments were conducted on these three datasets.

[0157] ACDC (Automatic Cardiac Diagnosis Challenge) This dataset is a medical image dataset specifically designed for cardiac image segmentation and cardiac health assessment. It is primarily intended for the automated diagnosis of cardiac diseases, particularly those requiring magnetic resonance imaging (MRI). MRI Image analysis. The dataset contains 100 cases and focuses on the left ventricle. LV Right ventricle RV and myocardium Myo It has been annotated. (And) nnFormer Consistently, we divided the data into 70 sets for training, 20 sets for testing, and 10 sets for validation.

[0158] Synapse (Synapse Multi-Organ CT Dataset) It is a multi-organ CT This image segmentation dataset focuses on the segmentation of abdominal organs. The dataset contains 30 images of abdominal organs. CT The dataset consisted of scanned data, with 18 cases used for training and 12 cases used for testing. Each case included annotations for eight abdominal organs: spleen, right kidney, left kidney, gallbladder, liver, stomach, aorta, and pancreas. 3D NIFTI The image is stored in a format with a resolution ranging from (0.7, 0.7, 1.0) mm to (1.0, 1.0, 5.0) mm and a median image size of (512, 512, 50). Synapse The main task of the dataset is to segment multiple organs in the abdomen to evaluate the segmentation performance of the algorithm in complex anatomical structures.

[0159] BraTS(Brain Tumor Segmentation) yes Medical Segmentation Decathlon (MSD) This is a brain tumor segmentation task. The task consists of 484 images. MRI Image composition, each image contains four different MRI Scan type: T1W、T2W、FLAIR(Fluid Attenuated Inversion Recovery) and T1GD Each modality provides different information, which helps improve the precision and accuracy of tumor segmentation. In this experiment, our target categories included the entire tumor. WT Enhance tumor ET and tumor core TC These labels provide a standardized benchmark for the automatic segmentation and identification of tumors. Regarding data splitting, we follow... UNETR The division method is as follows, in which the proportions of training, validation, and testing are 80%, 5%, and 15%, respectively.

[0160] 2. Implementation details:

[0161] All experiments were conducted in Python 3.8 The environment in which the deep learning framework used is [insert framework here]. PyTorch 1.11.0 The operating system is Ubuntu 22.04.5 During training, all programs were equipped with 16GB of memory and NVIDIA 4080 SUPER GPU The training was performed on a machine with a total training cycle of 1000. epoch The learning rate employs a multinomial decay strategy, with the specific formula as follows:

[0162]

[0163] in: lr The learning rate; initial_lr Set the initial learning rate to 0.01. max_epochs The maximum training cycle is 1000. epoch .

[0164] During the optimization process, we used SGD The optimizer is set with momentum of 0.99 and weight decay coefficient of 3e-5 to improve model training stability and convergence speed. For different datasets ( ACDC , Synapse and BraTS We used different image resolutions for training: ACDC The dataset uses 160×160×16. Synapse The dataset uses 128×128×64, while BraTS The dataset uses a resolution of 128×128×128.

[0165] During the inference phase, we used a sliding window with 50% overlap and, instead of using ensemble techniques to improve prediction accuracy, directly reported the dice coefficients of the single model. Apart from the above settings, all other training hyperparameters and data augmentation methods were consistent with... nnFormer The model remained consistent. During training, we performed various augmentation operations on the data, including rotation, scaling, adding Gaussian noise, applying Gaussian blur, and adjusting image brightness and contrast. These augmentation operations were applied sequentially in a specified order to improve the model's robustness and generalization ability.

[0166] 3. Loss function:

[0167] The loss function used in this invention is soft dice loss (…). soft dice loss ) and cross-entropy loss ( cross entropy loss The weighted sum of the two losses is calculated as follows: Soft dice loss primarily focuses on the overlap of regions when classes are imbalanced, showing a strong advantage, especially in small target regions or backgrounds. Cross-entropy loss measures the difference between the predicted classification result and the true label for each pixel. By setting appropriate weight coefficients λ, the influence of these two losses can be adjusted according to the needs of the task. The combined loss function formula is as follows:

[0168] ;

[0169] In the formula: For cross-entropy loss, These are the weighting coefficients; For the loss of soft dice, These are the weighting coefficients; C express the number of classes , V express the number of voxels , G c,v and P c,v They represent Class C exist voxel V place ground truth And output probability.

[0170] By combining these two loss functions, the model can better handle small region segmentation and multi-class segmentation tasks, thereby improving segmentation performance.

[0171] 4. Evaluation Indicators:

[0172] according to nnFormer and UNETR The method we use Dice Similarity Coefficient(DSC) and 95% Hausdorff Distance(HD 95 ) The performance of all models was evaluated.

[0173] The Dice Similarity Coefficient (DSC) is a standard metric for evaluating the accuracy of binary segmentation, used to quantify the overlap between the predicted results and the ground truth annotations. Its value ranges from 0 to 1, where 1 represents perfect overlap, meaning the model's predicted segmentation result is highly similar to the ground truth annotation. ground truth A perfect match (0) indicates complete overlap, while 0 indicates no overlap, meaning the two segmentation results have no common pixels. Generally, the closer the dice coefficient is to 1, the better the segmentation effect, indicating that the model's prediction is consistent with the expected result. ground truth Consistent; conversely, a dice coefficient close to 0 indicates poor segmentation, and the model prediction is consistent with... ground truth The differences are quite significant. DSC The calculation is performed using the following formula:

[0174] ;

[0175] in: G and P Each represents all voxel of ground truth and output probability, Indicates the predicted segmentation result and ground truth The intersection of the segmentation results.

[0176] In medical image segmentation, Hausdorff Distance is commonly used to compare the difference between segmentation results (predicted contours) and ground truth annotations (ground truth). Its calculation helps capture the most severe biases that may exist in the segmentation results, especially when boundaries are imprecise. Specifically, Hausdorff Distance quantifies the distance between the farthest pairs of points in two sets and is often used to measure the similarity between two sets.

[0177] For two point sets A and B , Hausdorff distance H ( A,B ) is defined as a set A Each point in the set B The maximum distance between the nearest points in the set and the set of the nearest points in the set. A The larger of the maximum distances between the nearest points. HD 95 As Hausdorff The percentile variant of the distance is calculated as the distance between two contours within 95 degrees. th The percentile maximum distance is determined by ignoring the farthest 5% of the distances between all pairs of points and only considering the remaining 95%.

[0178] exist HD 95 In the assessment, the smaller HD 95 The value indicates that the model's predicted segmentation results have a better match with the true label boundaries, demonstrating better segmentation quality. The formula is defined as follows:

[0179] ;

[0180] in , It is represented as Euclidean distance. ground truth voxel and predict voxel The maximum between 95 th Percentile distance HD 95 (P,G) for predict voxel and ground truth voxel The maximum between 95 th Percentile distance.

[0181] State-of-the-art comparison:

[0182] We selected several state-of-the-art medical image segmentation networks for experimental comparison, covering those based on... CNN Networks, based on Transformer Networks and hybrid CNN and Transformer We constructed a network and conducted most of the experiments using publicly available source code. Among the networks compared, we selected [the network name is missing here]. UNet , Swin-UNet and nnUNet Three CNN-based network architectures, and those based on Transformer Architecture SegFormer and CoTr .at the same time, LeViT-UNet-384s、 Missformer、TransUNet、nnFormer、UNETR and Swin-UNETR Integrated CNN and Transformer A hybrid architecture. Furthermore, we selected one specifically designed for brain tumor segmentation. TransBTS The model, in BraTS Comparisons were performed on the dataset. Experiments validated that our network performs well. ACDC、Synapse and BraTS It has demonstrated leading performance on all datasets.

[0183] Table 1: Each model in ACDC Performance comparison table on the dataset

[0184]

[0185] ACDC Dataset: We ACDC The segmentation task on the dataset was quantitatively evaluated, mainly through... DSC To measure the right ventricle ( RV ), myocardium ( Myo ) and left ventricle ( LV The performance of three different segmentation results, and using AvgDSC and HD 95 To evaluate the overall segmentation performance, all segmentation performance is reported using single-model accuracy, without using any pre-training, model ensemble, or other external data. Based on the results in Table 1, combined with... CNN and Transformer Hybrid model UNETR and its variants Swin-UNETR The average performance in the segmentation task was 88.61% and 88.95%, respectively. DSC . MISSFormer It also achieved relatively high performance in the segmentation task. AvgDSC It was 87.90%, although it was lower than... Swin-UNET It achieves 90.0% accuracy, but still retains strong segmentation capabilities. Furthermore, LeViT-UNet-384s (90.32) demonstrates better segmentation accuracy. Compared to these models, nnUNet and SegFormer These two combinations U-Net and Transformer The model achieved even better results, with 90.92% and 90.96% respectively. AvgDSC . nnFormer Performance was further improved, reaching 92.06%. AvgDSC This surpasses most existing methods. However, our... MedFreq-Net exist LV、RV and Myo of DSC The scores were 96.34, 91.73, and 91.07 respectively, with an average DSC of 93.05, which is significantly better than... nnFormer And in HD 95 superior, MedFreq-Net The result was 1.05 mm, compared to nnFormer The 6.25% reduction demonstrates its advantage in boundary accuracy. In summary, MedFreq-Net It performs excellently in detail capture, boundary accuracy, and overall segmentation accuracy, enabling it to better handle segmentation tasks of various regions of the heart and significantly improve performance. ACDC Dataset segmentation performance.

[0186] Figure 4 Showing MedFreq-Net Compared with existing advanced models ACDC Qualitative analysis on the dataset. In the figure, we have marked the error areas with green dashed boxes. In the first row, we provide a simple sample from which we can observe... nnFormer and nnUNet Oversegmentation occurs in the segmentation of the left ventricle, and UNETR The junction between the left ventricle and myocardium is not clearly defined. In the second row, the segmented area of ​​the heart is small, and other models have failed to accurately determine the size of the left ventricle. In this region, MedFreq-Net It demonstrates strong segmentation capabilities, preserving the morphology of the myocardium well and accurately segmenting the left ventricular structure. In the third row, we present a more challenging segmentation task with boundary interference. As can be seen from the green box in the image, MedFreq-NetThe differences in right ventricular segmentation compared to other models are more pronounced. The remaining models failed to effectively capture the right ventricular boundary, especially in areas with significant boundary interference.

[0187] The three examples of qualitative analysis above demonstrate that, MedFreq-Net It can solve the problems of over-segmentation, blurred boundaries, and unclear segmentation of fine structures in existing models, further proving our... FFE and TAA The importance of modules in boundary awareness.

[0188] Table 2: Each model in Synapse Performance comparison table on the dataset

[0189]

[0190] Synapse Dataset: We Synapse The segmentation task on the dataset was quantitatively evaluated, mainly through... DSC To measure different organ regions (such as the spleen) Spl ), right kidney ( RKid ), left kidney ( LKid ),gallbladder( Gal ),liver( Liv ), aorta ( Aor ),pancreas( Pan The segmentation results, and used AvgDSC and HD 95 To evaluate the overall segmentation performance. All segmentation performance is reported through the accuracy of a single model, without using any pre-training, model ensemble, or external data. Based on the results in Table 2, traditional... UNet The model in Synapse The performance on the dataset is moderate. AvgDSC It is 76.86%. Based on this, the mixture... CNN and Transformer model TransUNet A slight improvement, reaching 77.49%. With further optimization of the architecture, UNETR and LeVit-UNet-384s Significant breakthroughs were achieved in segmentation accuracy, reaching 78.35% and 78.53% respectively, demonstrating the advantages of these variants in handling complex structures. More advanced models, such as... Swin-UNETR and CoTr They performed even better, achieving 83.48% and 85.05% respectively. AvgDSC ,show Transformer Advantages of the architecture in global feature capture. nnUNet and nnFormerAs models with strong segmentation capabilities, they achieved 86.44% and 86.57% respectively. AvgDSC This further demonstrates its effectiveness in medical image segmentation tasks. In contrast, MedFreq-Net It surpasses other models in both segmentation accuracy and boundary accuracy, especially in the right kidney (93.83), left kidney (93.15), and liver (97.04) segmentation accuracy, which is particularly outstanding. AvgDSC It reached 88.76%. HD 95 superior, MedFreq-Net It is 7.45 mm It significantly outperforms other models, demonstrating excellent performance. This indicates that... MedFreq-Net It can better capture the fine structure of each organ and significantly improve the overall segmentation accuracy.

[0191] Figure 5 Showing MedFreq-Net Compared with existing advanced models Synapse Qualitative analysis on the dataset. We have marked the segmentation error regions with red dashed boxes. In the first row, we notice that existing methods miss edge details when segmenting the pancreas, while... nnFormer and UNETR It was also impossible to accurately separate the right kidney, even UNETR Most details were lost during the dissection of the stomach. In contrast, ours... MedFreq-Net It can accurately segment the pancreas, right kidney, and stomach areas.

[0192] In the second row, we demonstrate a simpler segmentation task. As can be seen from the figure, nnFormer and... UNETR When dissecting the spleen, the left adrenal gland and stomach were confused, and nnUNet and UNETR In segmentation Portal vein and splenic vein At that time, it was incorrectly divided into two small pieces. Our model, however, accurately divided the spleen and... Portal vein and splenic vein The edge portion, which may benefit from edge details learned from frequency domain information.

[0193] Finally, in the third line, regarding the subtle segmentation task of the left adrenal gland, nnFormer and UNETR The task was lost entirely. (Regarding the dissection of the stomach and...) Portal vein and splenic vein When dealing with these two closely connected parts, existing models fail to capture the edge areas, leading to inaccurate segmentation. Our model, however, demonstrates strong segmentation capabilities and accurate boundary delineation when facing these challenges, without oversegmentation or undersegmentation.

[0194] Table 3: Each model in BRATS Performance comparison table on the dataset

[0195]

[0196] Based on the results in Table 3, MedFreq-Net exist BRATS The segmentation performance on the dataset is excellent. Specifically, MedFreq-Net Throughout the tumor area ( WT ), enhance tumor area ( ET ) and the core area of ​​the tumor ( TC On Dice Similarity coefficient ( DSC The percentages were 91.98%, 78.01%, and 83.34% respectively, and their overall percentages were... AvgDSC It reached 84.44%. Compared with the current state-of-the-art models, MedFreq-Net It exhibits excellent segmentation accuracy, especially in enhanced tumor segmentation (…). ET ) and total tumors ( TC In terms of segmentation, compared to nnUNet (82.41) and Swin-UNETR (82.39), representing improvements of 2.46% and 2.49% respectively. Regarding boundary accuracy, MedFreq-Net of HD 95 It is 4.98 mm It is significantly better than all other models, especially compared to TransUNet (12.98) mm )and CoTr (9.70) mm This demonstrates its advantages in detail capture and boundary accuracy. Overall, MedFreq-Net It not only surpasses existing methods in terms of segmentation accuracy SOTA The method also demonstrates greater refinement in capturing details and boundaries, significantly improving performance. BRATS Tumor segmentation accuracy of the dataset.

[0197] Figure 6 Showing our MedFreq-Net exist BRATS The visualization shows the segmentation results on the dataset, with yellow dashed boxes marking areas of incorrect segmentation. First, in the first row, we can see that in the segmentation... ET At that time, other models incorrectly... ET and NET Confusion. Secondly, in the second row, we selected smaller, more challenging segmentation targets. nnFormer , nnUNet and UNETR The data was mistakenly lost during the segmentation process.NET The boundary with ET resulted in the complete absence of [something] in the segmentation result. NET and ET Finally, in the third line, we can see... ET It is elongated, but the other models lack continuity during segmentation, which... ET The loss of details in the text led to ET It was divided into two segments. Our model, however, completely segmented the long strip. ET This is thanks to our MFF The module effectively combines detailed information and semantic information.

[0198] 6. Ablation test:

[0199] Table 4 Ablation Experiment

[0200]

[0201] Table 4 shows our... MedFreq-Net The above presents ablation experimental results for different modules. To ensure the fairness of the experiments, we used the same baseline model ( Baseline ),exist ACDC Experiments were conducted on the dataset, and with DSC (%)and HD 95 ( mm ) as an evaluation indicator. Baseline In the experiment, we ruled out MAP , FFE Modules and MFF Modules, while using traditional Transformer Block To replace FFE And use simple skip connections instead MAP .

[0202] Based on the experimental results, Baseline The model in ACDC On the dataset AvgDSC It was 89.12%. HD 95 It is 1.68 mm This is already better than UNETR Model AvgDSC (88.61) and HD 95 (1.70) is a slight improvement, proving our... Baseline It already possesses a certain degree of segmentation capability. Next, we will... FFE Add the module Baseline In this context, a variable is used to replace skip connections, aiming to enhance detailed features in images through frequency domain information, especially in complex boundary regions. FFE After that, the model AvgDSCIncreased to 91.65%, HD 95 Reduced to 1.26 mm , respectively Baseline Increased by 2.53% and 0.42%. mm Based on the results of the qualitative analysis, FFE It can effectively improve the ability to capture details, especially in the segmentation of small structures.

[0203] Next, we will use MAP replace Transformer Block This module primarily improves segmentation accuracy by enhancing the modeling of both global information and local details. MAP After that, the model DSC It reached 91.26%. HD 95 Further reduced to 1.21 mm In the fourth step of the experiment, we will MFF The module is added to the network. MFF This module helps fuse multi-layer features at different scales and enhances the model's attention to features at different scales. After adding this module, the model's... DSC It reached 89.70%. HD 95 1.55 mm Despite its AvgDSC Better than some baseline settings, but compared to FFE and MAP When used alone, the module failed to further improve segmentation accuracy. We speculate that this is because... MFF The complexity of the modules may lead to a decrease in the accuracy of the model in certain areas.

[0204] Finally, in the complete MedFreq-Net In the model, we will FFE , MAP and MFF When used in combination, this configuration achieves optimal performance. AvgDSC It is 92.87. HD 95 The value is 1.09, right ventricle RV Myocardium Myo and left ventricle LV of DSC The scores of 91.89, 90.69, and 96.02 respectively fully demonstrate the advantages of multi-module combination.

[0205] pass Figure 7 It can be seen that, MedFreq-Net Each module has made a positive contribution to improving model performance. In particular FFE and MAPThe module, by enhancing detailed and global feature modeling, significantly improves segmentation accuracy and boundary recovery capabilities. MFF The introduction of this module further optimizes the multi-scale feature fusion of the model. Overall, MedFreq-Net exist ACDC The dataset demonstrates powerful segmentation capabilities and boundary accuracy, providing an effective solution for medical image segmentation.

[0206] 7. Comparison of model complexity:

[0207] Each model in BraTS Statistical analysis and computational complexity of parameters on a dataset, using parameter count (Params ), number of floating-point operations ( FLOPs ) and average dice similarity coefficient ( AvgDSC Three indicators are used to comprehensively evaluate the complexity and accuracy of the model. Params This represents the total number of parameters used during model training, expressed in units of... million ; FLOPs This represents the total number of floating-point operations, typically used to measure the computational complexity of a model. It includes all multiplications, additions, and other possible operations (such as activation function calculations), and its unit is units of 1 / 2 volts. G To ensure a fair comparison, we used the same input size of 128×128×128 and the same preprocessing strategy, such as... Figure 8 As shown.

[0208] As can be seen from Table 5, our model used the least computational resources (70.1G), far lower than other models, and the lowest among all models. Despite nnUNet, MedNeXt-M-K3 and MedNeXt-M-K5 Due to the inherent design of the convolution kernel, the number of parameters is lower than... MedFreq-Net But our model in AvgDSC It outperforms these models, and in FLOPs The figures above are 82.91%, 71.73%, and 77.24% lower than theirs, respectively.

[0209] To more intuitively demonstrate the model's efficiency and segmentation accuracy, we designed... 3D Scatter plot. From Figure 6 It can be seen from this that MedFreq-Net exist BraTS Performance on the dataset indicates that fewer parameters and lower [performance / quality] are achieved. FLOPs It did not affect the segmentation accuracy; on the contrary, it achieved the best segmentation results while maintaining efficient computation.

[0210] Table 5. Comparison of Complexity

[0211]

[0212] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A hybrid network framework integrating spatial and frequency domain information, employing a hierarchical encoder and decoder, wherein the encoder and decoder comprise symmetrically arranged four downsampling layers and four upsampling layers, characterized in that: Each downsampling layer has a MAP module connected to its backend, with block embedding used in the first downsampling layer. The first three upsampling layers each have a MAP module at their frontend, while the last three upsampling layers are connected to the first upsampling layer via an MFF module. The MAP modules of each of the first three downsampling layers are added to the outputs of each of the last three upsampling layers via an FFE module, and the MAP module of the last downsampling layer is directly connected to the last upsampling layer via an FFE module. Wherein: Embedded patches are used to segment the feature map extracted from the input 3D image by the first-layer downsampling into non-overlapping patches, and project the non-overlapping patches onto different channel dimensions to generate the output feature map. The MAP module is used to enhance feature learning in image segmentation tasks. It includes: multi-scale convolution that can capture features at different scales, MWF module that can adaptively adjust the contribution of features at different scales according to their importance, and position encoding that can perceive the three-dimensional spatial coordinates of features at different scales. The FFE module is used to enhance the low-frequency, mid-frequency, and high-frequency information in the input features. It includes a dynamic frequency filter that can perform channel weighting on the input features to generate frequency weights for each channel, and the frequency weights are divided into three frequency band weights: low-frequency, mid-frequency, and high-frequency. It also includes low-frequency masks, mid-frequency masks, and high-frequency masks that can obtain feature values ​​of different frequency bands from the frequency domain. The MFF module performs TAA operations on the last three upsampled layers in the H, W, and D axes respectively. After channel spatial alignment, it fuses the low-level features obtained from the frequency and spatial domains with the high-level semantic information obtained from the first upsampled layer to obtain the output enhanced feature map; wherein: TAA The specific process of operation: Input features X∈R are processed by 1×1 3D convolution. B×C×H×W×D Perform a query Q ,key K Sum V The projection is obtained Q , K , V∈ R B×H×W×D×C ; For each upsampling, attention is calculated using either the depth, width, or height axis: ； In the formula: d for Query and Key The dimension; The outputs of each attention head are merged to generate a feature map. ; The original input feature map is connected to the attention-processed feature map via residual connections. The feature maps are then fused, and the fused feature maps are processed through a linear transformation to obtain the output feature map: ； In the formula: γ It is a trainable parameter used to control the weighting between output and input features.

2. The hybrid network framework for fusing spatial domain and frequency domain information according to claim 1, characterized in that: The process of generating the output feature map includes the following steps: S21. The feature map extracted from the input 3D image by downsampling the first layer. The partition is divided into non-overlapping patches, which are expressed as: ， In the formula: P1, P2, and P3 represent the resolution of each patch, respectively; S22. Project the patch obtained in step S21 onto C channel dimensions to generate the output feature map of the first-layer downsampling, wherein the output feature map is expressed as: ， In the formula: H, W, D, and C represent the height, width, depth, and number of channels of the feature, respectively.

3. The hybrid network framework for fusing spatial domain and frequency domain information according to claim 1, characterized in that: The feature learning process includes the following steps: S31. Enhance the input features by improving the projection layer. Preliminary feature mapping and normalization are performed; wherein: the enhanced projection layer consists of a 1×1×1 convolution, GroupNorm, and GELU activation function; S32. By constructing multi-scale convolutions, features at three scales—local details, intermediate regions, and global context—are extracted from the input image processed in step S31. Wherein: the multi-scale convolution is a parallel group of depthwise convolutions, expressed as follows: ； In the formula: Conv k×k×k express kernel size For a depthwise convolution of k, the number of parameters is reduced by setting the number of groups to the number of input channels C; S33. The features extracted in step S32 at the three scales are concatenated, and weights for each scale are generated through dynamic convolution. The features at the three scales are then weighted and fused using these weights. Wherein: The splicing is expressed as follows: ； In the formula: concat Indicates feature splicing; The weights are expressed as follows: ； In the formula: Softmax and GELU Representing different activation functions; The weighted fusion is expressed as follows: ； In the formula: Represents matrix dot product; S34. Add the location code to the weighted fusion feature obtained after weighted fusion in step S33 in the following manner: ； In the formula: Interpolate tri ( ) represents trilinear interpolation; Basic Location Code PE base ∈R C×8×8×8 The system adapts to arbitrary input sizes using trilinear interpolation, and after interpolation, it undergoes layer normalization processing, injecting position information in the form of residuals. ； In the formula: LayerNorm ( ) represents layer normalization. The scaling factor that can be learned; S35. The features enhanced in step S34 are passed through a projection layer, subjected to a 1×1×1 convolution transformation, and then added... dropout The operation involves regularizing the features; S36. The features projected in step S35 are fused with the original input features through residual connection to obtain the output feature map.

4. The hybrid network framework for fusing spatial domain and frequency domain information according to claim 1, characterized in that: The specific process of enhancing low-frequency, mid-frequency, and high-frequency information is as follows: Using a dynamic frequency filter, frequency weights for each channel are generated by weighting the input features by channel. W i ∈R B ×3×C×1×1×1 In this process, the weight of each channel is divided into three frequency bands: low frequency, mid frequency, and high frequency. In the channel weighting part, the following method is used: Sigmoid An activation function is used to maintain numerical stability; wherein the frequency weights are expressed as: ； In the formula: i ∈{ low, mid, high }, express Sigmoid function, F(X) These are channel weights generated through dynamic convolution; Simultaneously, the input feature map is transformed from the spatial domain to the frequency domain through Fourier transform, and decomposed into low-frequency, mid-frequency, and high-frequency features: ； In the formula: F This is represented as a Fourier transform operation; By defining a low-frequency mask M low , intermediate frequency M mid and high-frequency mask M high The frequency domain features are weighted for low frequency, mid frequency, and high frequency respectively to obtain the decomposed frequency domain features; wherein: the low frequency mask M low , intermediate frequency M mid and high-frequency mask M high They are expressed as follows: ； In the formula: for sigmoid function, D It is the calculated three-dimensional distance matrix, representing the distance from each location in the input image to the frequency center. f low and f high These are the thresholds for low and high frequencies, respectively. It is a very small constant of 1e-6, used to avoid numerical instability; The inverse Fourier transform is used to restore the low-frequency, mid-frequency, and high-frequency features to the spatial domain: ； In the formula: F -1 This is represented as an inverse Fourier transform operation; The features after separating frequency bands using Fourier transform are then convolved separately to enhance the information of each frequency band; each frequency band is enhanced through convolution operation. C low , C mid and C high The data is processed and fused with the original input features through residual connections. Then, weighted values ​​generated by a dynamic frequency filter are used to weight and fuse the enhanced low-frequency, mid-frequency, and high-frequency features. A spatial attention mechanism is applied to further enhance the spatial expressiveness of the features, ultimately yielding the enhanced feature map. ； ； In the formula: SpatialAttention ( ) represents the spatial attention mechanism.

5. A hybrid network framework integrating spatial domain and frequency domain information according to claim 1, characterized in that: The process of obtaining the enhanced feature map: The above TAA The depth, width, and height feature maps obtained from the operation are aligned in the channel space, and each input feature is transformed into the frequency domain to obtain the amplitude spectrum and phase spectrum. The amplitude spectrum is enhanced by 1×1×1 convolution to strengthen low-frequency and high-frequency components; pass 3D The inverse fast Fourier transform converts the enhanced frequency domain features back to the spatial domain: ； In the formula: i ={1,2,3}, F Indicates Fourier transform, F -1 Indicates the inverse Fourier transform. j It is the imaginary unit, representing , angle It involves extracting phase information; After frequency domain processing, the output frequency domain features are fused with the original spatial domain features; By using residual connections, spatial features and frequency domain features are added together to obtain a fused feature map; The fused feature map from the three input feature maps is summed to obtain the enhanced feature map.

6. A hybrid network framework for fusing spatial domain and frequency domain information according to claim 5, characterized in that: The enhanced feature map is passed to the first layer for upsampling and segmentation: ； In the formula: This indicates a summation.

7. A hybrid network framework integrating spatial domain and frequency domain information according to claim 1, characterized in that: The FFE Input feature map of the module X The shape is B×C×H×W×D ,in: B For batch size, C For the number of channels, H,W,D These are the spatial dimensions of the image.

8. A hybrid network framework for fusing spatial domain and frequency domain information according to claim 1, characterized in that: The loss function of the framework is: ； In the formula: For cross-entropy loss, These are the weighting coefficients; For the loss of soft dice, These are the weighting coefficients; C express the number of classes , V express the number of voxels , G c,v and P c,v They represent Class C exist voxel V place ground truth and output probability.

9. A hybrid network framework integrating spatial and frequency domain information in segmentation 3D Its application in medical imaging is characterized by: The hybrid network framework employs a hierarchical encoder and decoder structure. The encoder and decoder include four symmetrically arranged downsampling layers and four upsampling layers. Each downsampling layer has a MAP module connected to its backend. The first downsampling layer uses block embedding. The first three upsampling layers each have a MAP module at their frontend, and the last three upsampling layers are connected to the first upsampling layer via an MFF module. The MAP modules of each of the first three downsampling layers are added to the outputs of each of the last three upsampling layers via an FFE module. The MAP module of the last downsampling layer is directly connected to the last upsampling layer via an FFE module. Wherein: Embedded patches are used to segment the feature map extracted from the input 3D image by the first-layer downsampling into non-overlapping patches, and project the non-overlapping patches onto different channel dimensions to generate the output feature map. The MAP module is used to enhance feature learning in image segmentation tasks. It includes: multi-scale convolution that can capture features at different scales, MWF module that can adaptively adjust the contribution of features at different scales according to their importance, and position encoding that can perceive the three-dimensional spatial coordinates of features at different scales. The FFE module is used to enhance the low-frequency, mid-frequency, and high-frequency information in the input features. It includes a dynamic frequency filter that can perform channel weighting on the input features to generate frequency weights for each channel, and the frequency weights are divided into three frequency band weights: low-frequency, mid-frequency, and high-frequency. It also includes low-frequency masks, mid-frequency masks, and high-frequency masks that can obtain feature values ​​of different frequency bands from the frequency domain. The MFF module performs TAA operations on the last three upsampled layers in the H, W, and D axes respectively. After channel spatial alignment, it fuses the low-level features obtained from the frequency and spatial domains with the high-level semantic information obtained from the first upsampled layer to obtain the output enhanced feature map. This map includes a TAA mechanism that allows attention calculation to be performed on different axes at different decoding stages. TAA The specific process of operation: Input features X∈R are processed by 1×1 3D convolution. B×C×H×W×D Perform a query Q ,key K Sum V The projection is obtained Q , K , V∈ R B×H×W×D×C ; For each upsampling, attention is calculated using either the depth, width, or height axis: ； In the formula: d for Query and Key The dimension; The outputs of each attention head are merged to generate a feature map. ; The original input feature map is connected to the attention-processed feature map via residual connections. The feature maps are then fused, and the fused feature maps are processed through a linear transformation to obtain the output feature map: ； In the formula: γ It is a trainable parameter used to control the weighting between output and input features.

Images