Implementation method of visual temporal feature network based on multi-module feature fusion

Abstract

The present invention discloses a method for realizing a visual temporal feature network based on multi-module feature fusion. The present invention comprises a visual temporal feature encoder VTFE, a multi-scale self-attention fusion module MSAF, an efficient feature reconstruction engine EFRE and an adaptive feature refinement processor AFRP. The VTFE module integrates a convolutional neural network with an extended long short-term memory network to achieve high-precision capture of spatiotemporal features. The MSAF module processes feature frequency domain information through fast Fourier transform to enhance the recognition capability of complex boundaries and noise. The EFRE module optimizes feature reconstruction through deep separable convolution and jump connection to maintain global consistency. The AFRP module integrates multi-level features and improves the ability to express details through deep convolution and attention mechanism. The present invention performs well on large-scale OCT data sets, significantly improves the segmentation accuracy of retinal edema and the recognition capability of fine structures, and provides an efficient and non-invasive auxiliary tool for the early diagnosis of ophthalmic diseases.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and specifically to a method for implementing a visual temporal feature network based on multi-module feature fusion, namely a visual temporal feature network (VTF-Net). VTF-Net significantly improves the accuracy and robustness of OCT image segmentation through a multi-module structure, fusion of temporal feature encoding, frequency domain feature enhancement and adaptive refinement processing. The network is suitable for the detection and analysis of ophthalmic diseases such as diabetic retinopathy and age-related macular degeneration, and provides an efficient and accurate auxiliary tool for clinical diagnosis and treatment. Background Art

[0002] Retinal diseases, such as diabetic retinopathy (DR) and age-related macular degeneration (AMD), pose a major threat to global vision health. According to statistics, as of 2020, approximately 103 million people are affected by diabetic retinopathy and 196 million people are troubled by age-related macular degeneration, and this number is expected to increase to 288 million by 2040. DR and AMD are often accompanied by retinal edema, which leads to the accumulation of central retinal fluid, thereby interfering with visual signal transmission. Therefore, early diagnosis and intervention are key to avoiding vision damage.

[0003] OCT (Optical Coherence Tomography) is a non-invasive, high-resolution imaging technology that can provide a detailed layered structure of the retina. However, how to efficiently segment these images remains an urgent problem to be solved. Traditional manual segmentation methods are not only time-consuming and labor-intensive, but also highly subjective. Judgments between different clinicians are prone to differences, which limits the consistency and efficiency of manual segmentation in large-scale clinical applications. Therefore, automated segmentation technology has become an important development direction to improve diagnostic efficiency and reduce human errors.

[0004] In automatic segmentation technology, convolutional neural networks (CNNs) have been widely used in medical image segmentation tasks. CNN has improved segmentation accuracy to a certain extent through its hierarchical structural design and local feature extraction capabilities. However, CNN has limitations in capturing complex background information and long-distance dependencies, which makes it perform poorly when processing OCT images with more noise and blurred boundaries. To compensate for these shortcomings, some studies have introduced long short-term memory networks (LSTMs) to enhance the model's ability to model time series information. However, LSTM is still insufficient in capturing local detail features and high-frequency information, especially when dealing with dynamic changes in retinal structure, and its performance is not stable enough.

[0005] In recent years, Transformer and its variants have made some breakthroughs in the field of image analysis. These models capture global context information through the self-attention mechanism, which makes up for the shortcomings of CNN to a certain extent. For example, models such as Swin-Unet and UNETR integrate the Transformer structure to achieve better global feature extraction. However, such models have high computational overhead and have efficiency issues when processing high-resolution medical images. In addition, these methods have limited use of frequency domain information and fail to fully explore the multi-level features of the image.

[0006] To solve these problems, some studies have introduced the fast Fourier transform (FFT) technology to convert image data from the spatial domain to the frequency domain. FFT can separate high-frequency noise from key features and effectively accelerate data processing. This technology has achieved good results in the segmentation of medical images such as MRI and CT. However, how to more closely integrate FFT with deep learning models is still a hot topic. Figure 1 To maintain stable segmentation performance in noisy environments is still a challenge facing current research.

[0007] Although a lot of work has been devoted to improving medical image segmentation technology, existing methods still have obvious shortcomings in dealing with noise, artifacts and blurred boundaries in OCT images. The robustness of the model and the ability to recognize complex structures need to be improved, especially when segmenting subtle lesions such as retinal edema. The accuracy and consistency of existing technologies have not yet reached the ideal level. These technical defects urgently need further research and innovation to solve. Summary of the invention

[0008] In fundus optical coherence tomography (OCT) image processing, accurate segmentation of retinal edema areas is crucial for the diagnosis and treatment of ophthalmic diseases such as diabetic retinopathy and macular degeneration. However, existing image segmentation techniques face challenges such as inconsistent image quality, blurred boundaries, and noise interference, which limit the reliability and accuracy of segmentation results. In order to solve these problems, the present invention proposes a visual temporal feature network (VTF-Net) based on multi-module feature fusion. The network integrates a visual temporal feature encoder (VTFE), a multi-scale self-attention fusion module (MSAF), an adaptive feature refinement processor (AFRP), and an efficient feature reconstruction engine (EFRE), which comprehensively improves the segmentation accuracy and robustness of OCT images in complex noisy environments.

[0009] The specific implementation process of the present invention includes the following steps:

[0010] Step 1: Data preprocessing and feature extraction;

[0011] Step 2: Use the visual temporal feature encoder VTFE; VTFE targets the first four layers of feature maps Integrate multi-scale and spatiotemporal information; VTFE combines the local feature capture capability of CNN with the time series modeling capability of xLSTM, performs convolution and time series processing on multi-scale features, and maintains information integrity through residual connections;

[0012] Step 3: Combination of multi-scale self-attention fusion module MSAF; MSAF on the fifth layer feature map f 5 Perform FFT and convolution operations to extract high-frequency and low-frequency features in the frequency and spatial domains;

[0013] Step 4: Use the adaptive feature refinement processor AFRP; improve the spatiotemporal consistency and expression ability of features through the attention deep convolution unit ADU and the residual space refinement unit RSR;

[0014] Step 5: Fusion of the efficient feature reconstruction engine EFRE; EFRE reconstructs features at different levels into a complete segmented image, extracts spatial information through depthwise separable convolution, and fuses features through skip connection technology;

[0015] Step 6: Model training and reasoning;

[0016] Step 7: Model performance evaluation and result analysis.

[0017] Furthermore, step 2 is specifically implemented as follows:

[0018] 2-1. Linear mapping and feature flattening: The feature map {f 1 ,f 2 ,f 3 ,f 4 After the flattening operation, the feature map is converted from three-dimensional to two-dimensional matrix form and input into the linear layer for transformation. The linear transformation result is expressed as:

[0019] (1)

[0020] in, and are the weight matrix and bias vector of the linear layer, respectively. Represents a flattening operation; the result of a linear transformation Enter the structure containing two convolution blocks, each of which consists of a convolution layer, batch normalization (BatchNorm) and ReLU activation function to further extract and enhance local information; the convolution block operation is expressed as:

[0021] (2)

[0022] 2-2. Combination of convolution and xLSTM: The four feature maps processed by two convolution blocks are input into the improved xLSTM unit; the improved xLSTM unit combines the spatial feature extraction capability of CNN and the temporal modeling advantage of long short-term memory network; the specific structure of the xLSTM unit is as follows: the output of the convolution block is divided into two branches after the Flip layer operation, one of which passes through the VIL block layer and the Flip layer in turn; the other branch passes through the Linear layer, SiLU activation layer and Flip layer in turn; the output information of the two branches is added and then passed through convolution, BatchNorm and Leaky ReLU in turn to obtain the output information OUT, and finally the output information OUT is connected with the output information of the convolution block through residual connection to obtain the feature that retains the original information Figure I ;

[0023] The update process of the xLSTM unit is expressed as:

[0024] (3)

[0025] Among them, H t represents the context dependency captured at the tth moment, Represents the input feature map of the xLSTM unit, that is, the four feature maps {f 1 ,f 2 ,f 3 ,f 4};

[0026] 2-3. Residual connection and activation: The features obtained after VTFE processing Figure I With the input, that is, the four feature maps {f 1 ,f 2 ,f 3 ,f 4}Through residual connection addition, the output feature maps are obtained respectively , , and .

[0027] Furthermore, step 3 is implemented as follows:

[0028] 3-1.FFT conversion: The MSAF module is divided into two branches, one of which inputs the feature map The number of channels is divided into eight parts, each part Apply the spectral attention operation, where The value is 1-8; let the input image be After The output feature map after the layer convolution operation is , get the feature map set ; Introduce fast Fourier transform FFT to transform the feature map set in the spatial domain Convert to frequency domain features;

[0029] 3-2. Application of large selective convolution kernel attention: The frequency domain features are divided into real and imaginary parts, which are processed separately, stacked through the channel dimension, and then the processing results of the real and imaginary parts are superimposed to form a joint real tensor. The large selective kernel network is applied to the real tensor; it is expressed as follows:

[0030] (4)

[0031] Among them, F stacked The real part F in the frequency domain is real and the imaginary part F imag The tensor composed of 3 and W 4 is the learnable weight used to learn the frequency domain feature map; σ represents the activation function;

[0032] 3-3. Inverse FFT restoration: The output of the large selection kernel network is converted back to the spatial domain through inverse Fourier transform. The formula is as follows:

[0033] (5)

[0034] The final spatial domain feature is represented as F final =F ifft.real , F ifft.real F ifft The real part of

[0035] 3-4. Convolution and directional feature extraction: Through convolution and interpolation operations, the feature representation of directional enhancement is extracted to obtain the global feature description, and finally multi-scale feature fusion is performed, as follows:

[0036] The other branch of the MSAF module takes the feature map f as input 5 The global feature encoding is obtained by sequentially performing convolution, BatchNorm and ReLU, interpolate, and 1×1 convolution. The global feature encoding is subjected to global average pooling to obtain the global feature description dprior.

[0037] Directionally enhanced feature representation , which is expressed as follows:

[0038] (6)

[0039] By using the bilinear interpolation method Upsample to the original image scale, perform feature recoding, and then obtain the global feature description through global average pooling operation. ,Right now:

[0040] (7)

[0041] The generated global feature description then passes through the convolution layer, activation layer and convolution layer in sequence to obtain the final output feature of the branch ;

[0042] The output features of the two branches are subjected to multi-scale feature fusion and then passed through the convolution layer to obtain a feature map; the feature map is combined with the original input feature map f 5 After performing residual connection, the final fused feature map is output .

[0043] Furthermore, step 4 is implemented as follows: AFRP receives feature maps from multiple scales

[0044] , , , and , and restore the low-resolution feature map to high resolution through layer-by-layer upsampling operations; AFRP consists of 5 layers, of which 1-4 layers have the same structure, and the two adjacent layers fuse information at different levels through RSR; the specific structure is as follows: Feature map In the fifth layer, it passes through the GAP layer, ADU layer and GAP layer in turn; feature map In the fourth layer, the output r5 of the last GAP layer of the fifth layer is input into the ADU layer together. The output of the ADU layer passes through another GAP layer, and the output of the GAP layer is combined with the output processed by the RSR layer. Perform residual connection and output r4; similarly, the inputs of the ADU layers in the first, second, and third layers are feature maps , , And the outputs r2, r3 and r4 obtained in the next layer; finally, the feature results r1, r2, r3 and r4 of the information between the layers are obtained by combining the information output of each layer. The specific process of AFRP is simplified as follows:

[0045] (8)

[0046] in, Represents the feature map output after upsampling of the Jth layer, corresponding to r1, r2, r3, r4 and r5, Represents the feature map of the upsampled input of the Jth layer; Represents the feature map output after upsampling of the J-1th layer, where the value of J ranges from 1 to 5.

[0047] Step 4 is implemented as follows:

[0048] 4-1. Depth-separable convolution and attention mechanism adjustment of ADU module:

[0049] The expression formula of depth-wise separable convolution is as follows:

[0050] (9)

[0051] in, represents the depthwise convolution operation, represents the point convolution operation, Represents the input of ADU, that is , , , and ;

[0052] The specific calculation formula of the attention mechanism is:

[0053] 10)

[0054] in, represents the result graph generated by the attention mechanism, σ represents the activation function, and GAP represents the global average pooling;

[0055] Finally, the input of ADU is connected through residual connection to retain the original spatial information, which is expressed as:

[0056] (11)

[0057] in, represents the activation function, Represents the combination of Conv convolution, BatchNorm and ReLU operations;

[0058] 4-2. Residual fusion of RSR module: Through residual connection, the feature map output after the attention mechanism is Adding it to the initial feature map not only retains the original feature information, but also improves the expressiveness of the feature; its mathematical expression is:

[0059] (12)

[0060] Among them, F refined Represents the feature map after being processed by the RSR module, It represents the feature map output by the Jth layer after the SE attention mechanism. is the initial feature map of the Jth layer, , , , or .

[0061] Furthermore, the EFRE described in step 5 receives the feature results r1, r2, r3 and r4 generated by AFRP, which fuse the information between each layer, and the multi-scale feature map , , , and restore the low-resolution feature map to high resolution through layer-by-layer upsampling operations; the specific implementation is as follows:

[0062] 5-1. Use of depth-wise separable convolution: Perform depth-wise separable convolution on the feature map of each layer. Assume that the input feature map of the jth layer is , the result of the depth-wise separable convolution operation is , mathematically expressed as:

[0063] (13)

[0064] in, It is to perform independent convolution on each channel. It is a 1x1 convolution, used to mix the information of each channel, j=1, 2, ..., 4;

[0065] 5-2. Upsampling operation and attention mechanism adjustment: EFRE obtains the depth-separable convolution Upsampling is performed to gradually restore its spatial dimensions. The mathematical representation of this process is:

[0066] (14)

[0067] in, The upsampling operation uses bilinear interpolation to restore the spatial information of the low-resolution image.

[0068] Attention mechanism based on feature maps generated by upsampling After processing, the feature map obtained is recorded as , ,4;

[0069] 5-3. Skip connection fusion feature: The fusion of multi-level features is performed through residual connection and weighted average, which is expressed as:

[0070] = + (15)

[0071] in, Represents the j-th layer feature map obtained through the attention mechanism. Each layer combines the information of the previous layer. Represents the multi-scale feature map of the jth layer;

[0072] EFRE obtains the final output feature map by weighted averaging the feature maps of all layers, which is expressed as:

[0073] (16)

[0074] Where N is the total number of decoding layers.

[0075] Furthermore, the present invention also provides a visual temporal feature network based on multi-module feature fusion, which includes a visual temporal feature encoder (VTFE), a multi-scale self-attention fusion (MSAF) module, an efficient feature reconstruction engine (EFRE) and an adaptive feature refinement processor (AFRP). Among them, the VTFE module integrates a convolutional neural network and an extended long short-term memory network (xLSTM) to achieve high-precision capture of spatiotemporal features. The MSAF module processes feature frequency domain information through fast Fourier transform (FFT) to enhance the recognition of complex boundaries and noise. The EFRE module optimizes feature reconstruction through deep separable convolution and jump connection to maintain global consistency. The AFRP module integrates multi-level features and improves the ability to express details through deep convolution and attention mechanism.

[0076] Beneficial effects of the present invention:

[0077] The present invention achieves the following key technical advantages through the collaborative work of multiple modules: By capturing spatiotemporal information through VTFE, the model performs well in the segmentation of complex lesion areas and significantly improves the generalization ability in different patients and dynamic lesions. The MSAF module combines frequency domain and spatial domain information to enhance the model's recognition ability for fuzzy boundary areas and reduce the impact of noise on segmentation results. The AFRP module ensures the consistency of the model in information at different scales through multi-layer feature fusion and adaptive processing, and adapts to the needs of complex lesion structures. The EFRE module reduces computational overhead through deep separable convolution and jump connections, making the model suitable for real-time processing of high-resolution OCT images. In summary, the technical points of the present invention and its module design solve many challenges in OCT image segmentation, improve the accuracy, robustness and computational efficiency of segmentation, and are suitable for actual clinical scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is the overall framework diagram of VTF-Net.

[0079] Figure 2 This is the internal structure diagram of the visual temporal feature encoder module.

[0080] Figure 3 This is the internal structure diagram of the multi-scale self-attention fusion module. DETAILED DESCRIPTION

[0081] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0082] like Figure 1-Figure 3 As shown, the visual temporal feature network implementation method based on multi-module feature fusion specifically includes the following steps:

[0083] Step 1: Data preprocessing and feature extraction

[0084] The input of this invention is a normalized OCT image, which is resized to 256×256 pixels to ensure that it matches the input size of the model. The data preprocessing uses a three-channel RGB format. First, the input image The pre-trained ResNet34 network is used to perform preliminary feature extraction and obtain five levels of feature maps {f 1 ,f 2 ,f 3 ,f 4 ,f 5}, these feature maps correspond to different resolutions and semantic levels, laying the foundation for subsequent module processing.

[0085] Step 2: Application of Visual Temporal Feature Encoder VTFE

[0086] VTFE combines the local feature extraction capability of convolutional neural networks (CNNs) and the long-distance dependency modeling capability of extended long short-term memory networks (xLSTMs) to capture image information more carefully and comprehensively at different scales. First, we obtain five levels of feature maps {f 1 ,f 2 ,f 3 ,f 4 ,f 5}, the first four layers of feature maps {f 1 ,f 2 ,f 3 ,f 4}VTFE is introduced, combining the local feature extraction of CNN and the long-distance dependency modeling of xLSTM, so that the model can capture both local details and global structures.

[0087] Specifically, the workflow of VTFE is as follows:

[0088] First, the feature map {f 1 ,f 2 ,f 3 ,f 4 After the flattening operation, the feature map is converted from three-dimensional to two-dimensional matrix form and input into the linear layer for transformation. The linear transformation result is expressed as:

[0089] (1)

[0090] in, and are the weight matrix and bias vector of the linear layer, respectively. Indicates a flattening operation. Linear transformation result Enter the structure containing two convolution blocks, each of which consists of a convolution layer, batch normalization (BatchNorm) and ReLU activation function to further extract and enhance local information. The convolution block operation is expressed as:

[0091] (2)

[0092] This multi-layer convolution operation not only increases the complexity of feature representation, but also provides more semantic information input for subsequent time series modeling.

[0093] After being processed by two convolution blocks, the four feature maps are input into the improved xLSTM unit. The xLSTM unit combines the spatial feature extraction capability of CNN and the temporal modeling advantage of long short-term memory network (LSTM). The design of VisionLSTM (VIL block) is borrowed. After the output of the convolution block is operated by the Flip layer, it is divided into two branches. One branch passes through the VIL block layer and the Flip layer in sequence; the other branch passes through the Linear layer, SiLU activation layer and Flip layer in sequence; the output information of the two branches is added and then passed through convolution, BatchNorm and Leaky ReLU in sequence to obtain the output information OUT, which further improves the quality of feature expression. Finally, the output information OUT is connected with the output information of the convolution block through residual connection to obtain the feature map that retains the original information.

[0094] The xLSTM unit structure not only overcomes the limitation of the local receptive field of CNN, but also can capture the long-distance dependencies in the image and enhance the model's integration of global and local information. The update process of the xLSTM unit is expressed as:

[0095] (3)

[0096] Among them, H t represents the context dependency captured at the tth moment, Represents the input feature map of the xLSTM unit, that is, the four feature maps {f 1 ,f 2 ,f 3 ,f 4 The updating process of the xLSTM unit enables VTFE to capture local detail information while integrating global context information across different scales.

[0097] Finally, the feature map obtained after VTFE processing is consistent with the input feature map (i.e., the four feature maps {f 1 ,f 2 ,f 3 ,f 4}) Add through residual connection to get the output feature map , , and ; Ensure the integrity of information and improve the representation ability of features. The processed feature map is input into the subsequent decoding module to generate segmentation results.

[0098] Step 3: Combination of Multi-Scale Self-Attention Fusion Module MSAF

[0099] The MSAF module is divided into two branches, one of which takes the input feature map f 5 The number of channels is divided into eight parts, each part Apply the Spectral Attention operation, where L is 1-8. Suppose the input image is After the Lth convolution operation, the output feature map is , get the feature map set . Introducing the fast Fourier transform FFT, the feature map set in the spatial domain Converted into frequency domain features, so that high-frequency and low-frequency features can be extracted in the frequency domain, further solving the problem of blurred boundaries and complex structure recognition in OCT images.

[0100] The frequency domain features are divided into real and imaginary parts, which are processed separately, stacked through the channel dimension, and then the processing results of the real and imaginary parts are superimposed to form a joint real tensor. The large selective kernel network (LSK Network) is applied to the real tensor to further enhance the expression ability of the frequency domain features; this makes the model more flexible and accurate in the feature extraction process.

[0101] The output of the large selection kernel network is inverse Fourier transformed back to the spatial domain.

[0102] The other branch will input the feature map f 5 After convolution, BatchNorm and ReLU, interpolate, and 1×1 convolution, the global feature code is obtained, and the global feature code is subjected to the global average pooling (GAP) operation to obtain the global feature description dprior; the global feature description is then passed through the convolution layer, activation layer, and convolution layer in sequence to obtain the final output feature of the branch;

[0103] The output features of the two branches are subjected to multi-scale feature fusion and then passed through the convolution layer to obtain a feature map; the feature map is combined with the original input feature map f 5 After performing residual connection, the final fused feature map is output .

[0104] Furthermore, global feature encoding is added to the MSAF module, and a directionally enhanced feature representation is generated by performing channel mapping on the feature map. , this process is represented by the following expression:

[0105] (4)

[0106] Furthermore, the bilinear interpolation method is used to Upsample to the original image scale and re-encode the features. Then, perform a global average pooling (GAP) operation to obtain the global feature description. ,Right now:

[0107] (5)

[0108] Generated global features It plays a key role in the subsequent attention mechanism by re-encoding the global features , so that the global context information can be effectively integrated into the local feature processing process.

[0109] Traditional segmentation methods usually ignore the richness of frequency domain information and only perform feature weighting in the spatial domain, which has certain limitations when processing complex image structures. The MSAF module can capture high-frequency and low-frequency components by introducing fast Fourier transform (FFT) to enhance the ability to capture image details.

[0110] In order to further enhance the expressive power of frequency domain features, the real and imaginary parts are stacked through the channel dimension to form a joint real tensor, and a large selection kernel network is applied for processing, which is expressed as follows:

[0111] (6)

[0112] Among them, F stacked The real part F in the frequency domain is real and the imaginary part F imag The tensor composed of 3 and W 4is a learnable weight for learning frequency domain feature mapping. σ represents the activation function (Sigmoid or Softmax, selected according to the task requirements). By combining LSK with different convolution kernel operations, the key features in the frequency components, especially low-frequency information, are effectively captured while suppressing high-frequency noise. In the macular edema segmentation task, this method enhances the model's ability to detect lesion areas.

[0113] The features after LSK processing are re-divided into real and imaginary parts to form complex frequency domain features F freq-attended , and transform it back to the spatial domain via an inverse fast Fourier transform (IFFT), namely:

[0114] (7)

[0115] The final spatial domain feature is represented as F final =F ifft.real , F ifft.real F ifft The real part of ; it not only retains the time domain information in the original image, but also integrates the deep feature expression in the frequency domain.

[0116] Furthermore, in the final stage of the MSAF module, feature maps from different attention mechanisms are integrated through multi-scale feature fusion. First, the outputs of different attention mechanisms are aggregated by concatenating these feature maps in the channel dimension to form a comprehensive multi-channel feature tensor. This tensor initially consists of multiple high-dimensional features, which are then processed through a convolutional layer to reduce the dimensionality and combine these features into a more compact and expressive form.

[0117] This convolution operation not only reduces the number of channels, but also promotes the fusion of spatial and frequency domain information captured by various attention mechanisms. The mathematical expression of this operation is as follows:

[0118] (8)

[0119] Among them, X fused represents the final fused feature map, X 1 ,X 2 ,…,X 8 It represents the 8 feature maps of the SpectralAttention mechanism. Multi-scale feature fusion ensures that the model can effectively capture the local and global context information of the image and provide a more compact feature expression.

[0120] Step 4: Use of Adaptive Feature Refinement Processor (AFRP)

[0121] The Adaptive Feature Refinement Processor (AFRP) module is designed to improve the ability to refine and decode features in retinal OCT images. AFRP combines the Attention Depthwise Unit (ADU) and the Residual Spatial Refinement (RSR) to optimize the feature integration process and improve the perception of complex spatial structures through multi-scale feature fusion and detail enhancement.

[0122] AFRP receives feature maps from multiple scales , , , and And through layer-by-layer upsampling operations, the low-resolution feature map is restored to high resolution to improve the quality and accuracy of image reconstruction.

[0123] The AFRP processor consists of 5 layers, of which layers 1-4 have the same structure, and two adjacent layers fuse information at different levels through RSR. The specific structure is as follows: Feature map In the fifth layer, it passes through the GAP layer, ADU layer and GAP layer in turn; feature map In the fourth layer, the output r5 of the last GAP layer of the fifth layer is input into the ADU layer together. The output of the ADU layer passes through another GAP layer, and the output of the GAP layer is combined with the output processed by the RSR layer. Perform residual connection and output r4; similarly, the inputs of the ADU layers in the first, second, and third layers are feature maps , , And the outputs r2, r3 and r4 obtained in the next layer; finally, the information outputs of each layer are combined to obtain the feature results r1, r2, r3 and r4 that integrate the information between layers, providing input for the subsequent EFRE.

[0124] The specific process of AFRP is simplified as follows:

[0125] (9)

[0126] in, Represents the feature map output after upsampling at the Jth layer (i.e., r1, r2, r3, r4, or r5 ), Represents the feature map of the upsampled input of the Jth layer; represents the feature map output after upsampling of the J-1th layer, and J ranges from 1 to 5. In this way, AFRP not only optimizes the feature integration process but also enhances the perception of the retina and macula boundaries, especially when the boundaries are blurred. The generated high-resolution feature map is processed by a nonlinear activation function to effectively filter out noise and artifacts, ensuring the accuracy and robustness of the segmentation results.

[0127] Furthermore, the ADU module combines the depthwise separable convolution and the attention mechanism to improve the refined expression of feature extraction. The depthwise separable convolution decomposes the standard convolution into depthwise convolution and pointwise convolution, effectively reducing the amount of calculation while maintaining efficient feature extraction capabilities, which can be expressed as:

[0128] (10)

[0129] in, represents the depthwise convolution operation, represents the point convolution operation, Represents the input of ADU, that is ( , , , and , the processing object of depth-wise separable convolution). Represents the output of the depthwise separable convolution in ADU. In the feature extraction process, the ADU module further introduces the attention mechanism to adaptively adjust the weights in the feature map and enhance the focus on important areas. The specific calculation formula of the attention mechanism is:

[0130] (11)

[0131] in, represents the result graph generated by the attention mechanism, σ represents the activation function, and GAP represents the global average pooling. Through this design, the ADU module can effectively capture the local and global features of the image, thereby improving the segmentation accuracy.

[0132] The output of ADU combines the features of the depth-separable convolution and the features obtained using the attention mechanism, and finally the input of ADU is connected through a residual connection to retain the original spatial information, which is expressed as:

[0133] (12)

[0134] in, represents the activation function, Represents the combination of Conv convolution, BatchNorm and ReLU operations.

[0135] In order to further improve the expressiveness and robustness of features, AFRP also introduces the RSR module. The RSR module integrates multi-scale feature maps through residual connections to enhance the spatial detail expression of features. The main steps of the RSR module include:

[0136] Multi-scale feature fusion: Feature maps from different scales ( , , , and ) is processed through upsampling operations (F.interpolate()) and convolutions, and concatenated to form a multi-channel feature tensor. This operation can capture information from different scales, ensure the model's sensitivity to complex structures, and improve the ability to capture local details.

[0137] SE attention mechanism: In the process of multi-scale feature processing, the RSR module dynamically adjusts the channel weights by introducing the Squeeze-and-Excitation (SE) attention mechanism to the multi-channel feature tensor, so that the model can better focus on important feature information. The SE module can enhance the important signals in the feature map while suppressing irrelevant noise, thereby improving the accuracy of segmentation.

[0138] Residual connection: Through the residual connection, the feature map output after the SE attention mechanism Compared with the initial feature map ( , , , and ), which not only retains the information of the original features, but also improves the expressiveness of the features. Its mathematical expression is:

[0139] (13)

[0140] Among them, F refined Represents the feature map after being processed by the RSR module, It represents the feature map output by the Jth layer after the SE attention mechanism. is the initial feature map of the Jth layer. By introducing the RSR module, AFRP can effectively integrate multi-scale feature information and enhance the model's ability to handle details and noise.

[0141] Step 5: Fusion of the Efficient Feature Reconstruction Engine EFRE

[0142] The purpose of the Efficient Feature Reconstruction Engine (EFRE) is to use multi-level feature maps to perform refined feature fusion and reconstruction in image processing tasks to enhance the perception of image details. EFRE combines deep separable convolution, attention mechanism, and layer-by-layer upsampling to optimize the feature fusion process, making feature expression more accurate and detailed. The following is a detailed description of the EFRE workflow:

[0143] EFRE receives the feature results r1, r2, r3 and r4 generated by AFRP, which fuse the information between each layer, and the multi-scale feature map , , , and restore the low-resolution feature map to high resolution through layer-by-layer upsampling operations to improve the quality and accuracy of image reconstruction. Then, the feature map of each layer will undergo a depth-wise separable convolution operation to further extract features. Assume that the input feature map of the jth layer is , the result of the depth-wise separable convolution operation is , mathematically expressed as:

[0144] (14)

[0145] in, It is to perform independent convolution on each channel. It is a 1x1 convolution, which is used to mix the information of each channel, j = 1, 2, ..., 4. Through the combination of deep convolution and point convolution, EFRE can extract more detailed spatial features while ensuring computational efficiency.

[0146] Then EFRE obtains the depthwise separable convolution Upsampling is performed to gradually restore its spatial dimensions. The mathematical representation of this process is:

[0147] (15)

[0148] in, is the upsampled feature map, is the feature map of the original input. The upsampling operation uses bilinear interpolation to restore the spatial information of the low-resolution image.

[0149] In order to further enhance the model's attention to important areas, EFRE introduces an attention mechanism. This mechanism automatically adjusts the influence of each area by weighting different areas in the feature map, allowing the model to focus on more important local information. In EFRE, the attention mechanism is based on the feature map generated by upsampling. The attention weighting process is similar to formula (11). In this way, EFRE can dynamically adjust the weights of each region, thereby strengthening the key regions in the feature map and suppressing irrelevant noise, and still maintaining high accuracy in an environment with large noise interference. The feature map obtained by the attention mechanism is recorded as j=1,2,...,4.

[0150] Finally, the fusion of multi-level features is performed through residual connections and weighted averaging. EFRE processes feature maps layer by layer, and the output of each layer not only considers the information of the current layer, but also integrates the feature information of the previous layer. Through residual connections, the gradient vanishing problem can be effectively avoided and the original information of each layer can be retained. The mathematical expression is:

[0151] (16)

[0152] in, Represents the j-th layer feature map obtained through the attention mechanism. Each layer combines the information of the previous layer. Represents the multi-scale feature map of the jth layer. In this way, the output of each layer can retain the information of the previous layer and introduce the detailed information processed by the current layer, thereby improving the expressiveness of the features.

[0153] The final output of EFRE is the feature map after layer-by-layer fusion. In order to further improve the expressiveness of the model, EFRE obtains the final output feature map by weighted averaging the feature maps of all layers, which is mathematically expressed as:

[0154] (17)

[0155] Among them, N is the total number of decoding layers. Through the weighted average method, EFRE effectively fuses the feature information of each layer to generate a feature map with high resolution and rich details.

[0156] Step 6: Training and inference process

[0157] The present invention uses the CMED-18k dataset for training. During the training process, the AdamW optimizer is used, the initial learning rate is 0.0001, and the batch size is set to 8. The model training is completed in the NVIDIA RTX 3080 GPU environment, and the model performance is evaluated using a two-dimensional cross entropy loss function. The inference process of the model includes inputting new images into the pre-trained model and generating segmentation results.

[0158] Step 7: Model performance evaluation and result analysis

[0159] In order to comprehensively evaluate the segmentation performance of VTF-Net, the experiment used multiple indicators such as Dice similarity coefficient, 95% Hausdorff distance (HD95), normalized cross correlation (NCC), Kappa coefficient (Kappa) and mean intersection over union (mIoU). The definition and calculation formula of each indicator are as follows:

[0160] Dice Similarity Coefficient (DSC) Dice coefficient is used to measure the overlap between the predicted segmentation area and the actual segmentation area. The closer the value is to 1, the higher the segmentation accuracy of the model. Its calculation formula is:

[0161]

[0162] Among them, Vpred and Vtrue represent the predicted segmentation area and the true segmentation area respectively. The Dice coefficient ranges from 0 to 1, and the larger the value, the more accurate the segmentation result.

[0163] 95% Hausdorff distance (HD95) HD95 is used to measure the boundary similarity between the predicted segmentation and the true segmentation, calculate the maximum and minimum distance between the two point sets, and measure it with the 95th percentile to reduce the impact of outliers on the results. Its formula is:

[0164]

[0165] Where dist(a,b) represents the Euclidean distance between point a and point b. The lower the HD95 value, the closer the predicted segmentation boundary is to the true boundary.

[0166] Normalized Cross-Correlation (NCC) NCC is an indicator for evaluating the similarity between two images. It is usually used to eliminate image brightness and contrast differences and measure the correlation between the segmentation result and the true value. Its calculation formula is:

[0167]

[0168] Among them, Itrue and Ipred are the pixel values ​​of the true and predicted images respectively, and μtrue and μpred are their means. The closer the NCC value is to 1, the higher the similarity between the segmentation result and the true segmentation area.

[0169] Kappa coefficient (Kappa) Kappa coefficient is used to measure the consistency between the segmentation result and the random classification result. The value range is -1 to 1. The higher the value, the better the stability and accuracy of the classifier. Its calculation formula is:

[0170]

[0171] Among them, po is the observed consistency ratio, and pe is the random consistency ratio. The higher the Kappa coefficient, the stronger the stability of the model in the segmentation task.

[0172] Mean Intersection over Union (mIoU) mIoU is used to measure the overlap ratio between the predicted area and the true area, that is, the ratio of the area of ​​the intersection of the predicted area and the true area to the area of ​​the union. The calculation formula is:

[0173]

[0174] Where predj and truthj represent the predicted result and the true result respectively, and NNN is the number of categories. The higher the mIoU value, the higher the segmentation accuracy of the model.

[0175] Through the evaluation of the above indicators, VTF-Net performed well in key indicators such as Dice similarity coefficient and HD95, demonstrating its robustness and accuracy in dealing with complex boundaries and subtle lesion areas.

[0176] Table 1 Comparison of experimental results on the CMED-18K dataset

[0177] method Dice mIOU HD95 NCC Kappa UNet 71.46±24.09 60.03±24.08 5.87±2.20 72.22±23.29 71.14±24.09 FCN8s 69.36±21.88 56.65±21.70 6.06±2.12 69.86±21.18 69.00±21.83 SegNet 67.87±25.66 56.09±24.90 6.09±2.18 68.84±24.56 67.51±25.51 PSPNet 73.29±21.66 61.58±22.28 5.76±2.01 73.80±20.81 72.98±21.63 ENet 71.62±19.68 58.92±20.80 5.96±2.17 72.34±18.63 71.26±19.68 ICNet 63.44±26.87 51.29±25.02 6.44±2.27 64.50±25.99 63.08±26.78 UNet+AttGate 66.60±26.69 55.02±25.89 6.17±2.40 68.01±25.36 66.28±26.67 DANet 71.47±22.95 59.63±22.97 5.86±2.00 71.94±22.22 71.14±22.92 LEDNet 72.86±19.41 60.41±20.67 5.92±2.09 73.37±18.52 72.52±19.41 DUNet 71.04±24.26 59.51±23.97 5.90±2.14 71.64±23.54 70.71±24.24 CENet 74.36±20.87 62.73±21.77 5.67±2.04 74.94±19.99 74.04±20.87 CGNet 70.24±21.00 57.56±21.59 6.07±2.16 70.90±19.96 69.89±20.98 OCNet 62.92±28.12 51.29±26.34 6.39±2.30 63.97±27.20 62.58±28.06 GCN 73.32±19.64 61.04±20.78 5.81±2.08 74.00±18.56 72.97±19.65 Ours 76.91±24.09 64.89±20.72 5.57±2.09 77.38±18.21 76.63±19.25

[0178] Table 1 shows the experimental results of different segmentation methods on the CMED-18K dataset. As can be seen from the data in the table, the model of this study performs well in indicators such as Dice, mIoU, HD95, NCC and Kappa, especially in Dice and Kappa coefficients, reaching 76.91% and 76.63% respectively, which is significantly better than other methods. Compared with the second-best solution CENet, the model of this study improved by 2.55% and 2.59% in Dice and Kappa respectively, indicating a significant improvement in the segmentation accuracy and consistency of the model. At the same time, in terms of the mIoU indicator, the model of this study reached 64.89%, which is 2.16% higher than the second-best solution CENet, showing that the model has a stronger ability to recognize and segment the lesion area.

[0179] Table 2 Ablation experiment

[0180] VTFE MSAF EFRE AFRP Dice mIOU HD95 NCC Kappa √ √ √ √ 76.91±19.73 65.76±20.90 5.51±1.98 77.38±18.93 76.63±19.74 √ √ √ 76.01±18.98 64.39±20.55 5.61±2.05 76.59±18.01 75.72±18.99 √ √ √ 76.20±20.01 64.88±21.06 5.62±2.15 76.69±19.22 75.91±20.01 √ √ √ 75.64±20.66 64.32±21.53 5.60±1.97 76.13±19.84 75.34±20.65 √ √ 73.36±24.22 62.44±24.09 5.70±2.04 73.88±23.58 73.06±24.21 √ √ 74.98±21.59 63.73±22.20 5.64±2.05 75.52±20.76 74.68±21.59 √ √ 73.10±24.14 62.28±23.82 5.70±2.00 73.49±23.52 72.98±24.11 √ √ 74.21±22.26 62.91±22.58 5.70±2.06 74.65±21.61 73.91±22.24

[0181] In the ablation experiment in Table 2, after gradually removing the core modules of VTF-Net (VTFE, MSAF, AFRP, EFRE), the model performance decreased. First, after removing VTFE, the Dice coefficient decreased by 1.37 percentage points and the HD95 increased by 0.18, showing the key role of the spatiotemporal feature capture mechanism in dealing with dynamic changes and complex backgrounds. This module can effectively capture local and global information, and its removal leads to a significant decrease in the ability to recognize subtle lesion areas in complex scenes. After removing the MSAF module, the Dice coefficient decreased by 1.21% and the HD95 increased by 0.15, indicating that the MSAF module is particularly important in complex backgrounds. Its removal significantly weakens the model's ability to capture global context, which in turn affects the overall segmentation performance of the model. The core advantage of AFRP is that it can fuse multi-level features and retain important detail information. After removal, the HD95 value increased significantly by 0.21, and the model's performance in complex or irregular boundary areas deteriorated significantly, especially in subtle lesion areas. Finally, the removal of the EFRE module leads to a decrease in the Dice coefficient of 0.96% and an increase in HD95 of 0.12. EFRE reduces the computational effort of convolution, while ensuring efficient feature extraction and integrating global information. After removing this module, the image reconstruction capability of the model is reduced, especially in terms of detail preservation and global consistency, which in turn affects the segmentation accuracy.

[0182] In addition, joint ablation experiments of multiple modules were conducted to further verify the synergy between the modules. The experimental results show that the collaborative work between the modules can significantly improve the ability to capture complex features of OCT images and ensure the stable performance of the model in different segmentation tasks.

[0183] Table 3 Ablation experiment of multi-scale self-attention fusion module

[0184] Dice mIOU HD95 NCC Kappa CA EMA 74.43±22.73 63.33±22.90 5.65±2.01 74.88±22.10 74.14±22.70 CA EMA+CB EMA 75.61±21.26 64.46±21.96 5.55±2.00 76.10±20.45 75.32±21.27 FFT 76.91±19.73 65.76±20.90 5.51±1.98 77.38±18.93 76.63±19.74 FFT+CA EMA 75.54±20.83 64.24±21.65 5.59±2.02 76.05±20.02 75.24±20.82 FFT+CB EMA 76.13±20.28 64.85±21.19 5.58±2.00 76.65±19.42 75.83±20.27 FFT+LSK 75.55±19.94 64.03±21.16 5.60±2.06 76.17±18.95 75.26±19.95 FSA 73.88±22.86 62.68±23.08 5.69±2.07 74.45±22.07 73.58±22.84

[0185] In Table 3, the effects of different attention mechanisms in the MSAF module are analyzed to evaluate their impact on model performance. The experimental results show that FFT combined with large convolution kernel attention mechanism (LSKA) performs best in all key indicators, especially in the Dice coefficient, which reaches the highest value of 76.91%. This shows that by capturing high-frequency and low-frequency information in the image through FFT, noise can be effectively filtered and key features can be retained, while LSKA can focus on significant features at different spatial scales, thereby improving the effectiveness of feature extraction.

[0186] In contrast, the effect of FFT is reduced when used alone or in combination with more complex attention mechanisms. For example, the FFT combined with the time-domain CB EMA scheme, although achieving a Dice coefficient of 76.13%, is slightly lower than the FFT+LSKA combination. This shows that overly complex attention mechanism combinations (such as the series connection of CA EMA and CB EMA), although they can enhance feature interaction and fusion capabilities, may introduce unnecessary complexity, resulting in redundancy or loss of information, which has a negative impact on segmentation results.

[0187] In addition, we also tested the FSA (frequency segmentation attention) scheme, which processes the features in the frequency domain in blocks and then applies the attention mechanism to each block. The experimental results show that the FSA scheme fails to outperform the FFT combined with the time domain scheme. This may be because the frequency domain information cannot effectively integrate the global context after block processing, which in turn affects the segmentation accuracy.

[0188] Table 4 Ablation experiment of visual temporal feature encoder module

[0189] Layers Kernels Blocks Dice mIOU HD95 NCC Kappa 2 3*3 2 75.89±20.57 64.59±21.24 5.60±2.04 76.41±19.72 75.59±20.56 2 5*5 2 76.31±19.91 65.01±21.06 5.55±2.00 76.82±19.07 76.01±19.91 3 3*3 2 76.46±20.53 65.39±21.54 5.49±1.99 76.93±19.76 76.18±20.53 3 5*5 2 75.41±21.62 64.28±22.14 5.61±2.06 75.89±20.94 75.12±21.61 4 3*3 1 75.86±20.47 64.56±21.44 5.56±2.01 76.41±19.63 75.56±20.47 4 3*3 2 76.34±19.23 64.89±20.72 5.57±2.09 76.94±18.21 76.05±19.25 4 5*5 2 76.91±19.73 65.76±20.90 5.51±1.98 77.38±18.93 76.63±19.74 4 5*5 3 75.81±21.13 64.67±21.79 5.54±1.99 76.31±20.93 75.52±21.12 5 5*5 1 76.37±19.75 65.07±21.03 5.54±2.03 76.93±18.81 76.08±19.76

[0190] In Table 4, the effects of different VTFE configurations on model performance are analyzed, mainly including the changes in the number of layers, convolution kernel size, and number of residual blocks. Experimental results show that the configuration with 4 layers, a convolution kernel size of 5x5, and two residual blocks achieves the best performance, with a Dice coefficient of 76.91% and a HD95 value of 5.51. This configuration fully demonstrates the importance of deep convolution and a large receptive field in capturing complex features in OCT images, especially when dealing with subtle structures and boundaries.

[0191] On the contrary, reducing the number of layers or using smaller convolution kernels will lead to degraded performance. Shallower networks cannot capture enough depth information, while smaller convolution kernels limit the receptive field, making it difficult for the model to capture global features when processing complex OCT images. For residual blocks, reducing their number leads to a decrease in the Dice coefficient, indicating that residual connections are essential for maintaining feature integrity and improving the diversity of feature expression.

[0192] Through these ablation experiments, it can be seen that the depth, convolution kernel size and number of residual blocks in VTFE have an important impact on the segmentation performance of the model. Appropriate configuration enhances the model's ability to capture global and local features in complex OCT images, while improving the overall stability and accuracy of the segmentation task. The MSAF module enhances the integration of details and global information through a multi-scale self-attention mechanism, while the AFRP module performs well in processing complex boundaries. The EFRE module achieves a balance between segmentation accuracy and detail retention through efficient feature reconstruction. The synergy of each module enables VTF-Net to have good performance and robustness when processing complex medical images.

[0193] In summary, the VTF-Net proposed in this paper significantly improves the accuracy and robustness of medical image segmentation by integrating multi-module feature fusion, spatiotemporal feature encoding and multi-scale self-attention mechanism, and has broad application prospects.

[0194] The above contents are only preferred implementation modes of the present invention and do not limit the protection scope of the present invention. It should be emphasized that any minor adjustments and improvements made by technicians in this field without departing from the basic principles of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A method for implementing a visual temporal feature network based on multi-module feature fusion, characterized in that: The following steps are involved: Step 1: Data preprocessing and feature extraction; Step 2: Use the visual temporal feature encoder VTFE; VTFE integrates multi-scale and spatiotemporal information for the first four layers of feature maps {f1, f2, f3, f4}; VTFE combines the local feature capture capability of CNN with the time series modeling capability of xLSTM, performs convolution and temporal processing on multi-scale features, and maintains information integrity through residual connections; Step 3: Combination of multi-scale self-attention fusion module MSAF; MSAF performs FFT and convolution operations on the fifth-layer feature map f5, which can extract high-frequency and low-frequency features in the frequency domain and spatial domain; Step 4: Use the adaptive feature refinement processor AFRP; improve the spatiotemporal consistency and expression ability of features through the attention deep convolution unit ADU and the residual space refinement unit RSR; Step 5: Fusion of the efficient feature reconstruction engine EFRE; EFRE reconstructs features at different levels into a complete segmented image, extracts spatial information through deep separable convolution, and fuses features through skip connection technology; Step 6: Model training and reasoning; Step 7: Model performance evaluation and result analysis; Step 2 is implemented as follows: 2-1. Linear mapping and feature flattening: The feature map {f1, f2, f3, f4} extracted by ResNet34 is flattened, the feature map is converted from three-dimensional to two-dimensional matrix form, and input into the linear layer for transformation. The linear transformation result is expressed as: F linear =W linear ×F flat +b linear (1) Among them, W linear and b linear are the weight matrix and bias vector of the linear layer, respectively, flat Indicates the flattening operation; the linear transformation result F linear Enter the structure containing two convolution blocks, each of which consists of a convolution layer, batch normalization BatchNorm and ReLU activation function to further extract and enhance local information; the convolution block operation is expressed as: F Conv =ReLU(BatchNorm(Conv(F linear ))) (2) 2-2. Combination of convolution and xLSTM: The four feature maps processed by two convolution blocks are input into the improved xLSTM unit; the improved xLSTM unit combines the spatial feature extraction capability of CNN and the temporal modeling advantage of long short-term memory network; the specific structure of the xLSTM unit is as follows: the output of the convolution block is divided into two branches after the Flip layer operation, one of which passes through the VIL block layer and the Flip layer in turn; the other branch passes through the Linear layer, SiLU activation layer and Flip layer in turn; the output information of the two branches is added and then passed through convolution, BatchNorm and Leaky ReLU in turn to obtain the output information OUT, and finally the output information OUT is connected with the output information of the convolution block through residual connection to obtain the feature map I that retains the original information; The update process of the xLSTM unit is expressed as: H t =xLSTM(H t-1 ,F t ), (3) Among them, H t represents the context dependency captured at the tth moment, F t Represents the input feature map of the xLSTM unit, that is, four feature maps {f1,f2,f3,f4}; 2-3. Residual connection and activation: The feature map I obtained by VTFE processing is added to the input four feature maps {f1, f2, f3, f4} through residual connection to obtain the output feature maps and Step 3 is implemented as follows: 3-1.FFT conversion: The MSAF module is divided into two branches, one of which divides the number of channels of the input feature map f5 into eight parts, each part Apply the spectral attention operation, where L is 1-8; let the input image be After the Lth convolution operation, the output feature map is Get the feature mapping set F = {F1, F2, ..., F L }; Introduce fast Fourier transform FFT to convert the feature mapping set F in the spatial domain into frequency domain features; 3-2. Application of large selective convolution kernel attention: The frequency domain features are divided into real and imaginary parts, which are processed separately, stacked through the channel dimension, and then the processing results of the real and imaginary parts are superimposed to form a joint real tensor. The large selective kernel network is applied to the real tensor; it is expressed as follows: LSK(F stacked )=σ(W4×ReLU(W3×F stacked )) (4) Among them, F stacked The real part F in the frequency domain is real and the imaginary part F imag The tensor composed of W3 and W4 is the learnable weights used to learn the frequency domain feature map; σ represents the activation function; 3-3. Inverse FFT restoration: The output of the large selection kernel network is converted back to the spatial domain through inverse Fourier transform. The formula is as follows: F ifft =IFFT(F freq-attended ) (5) The final spatial domain feature is represented as F final =F ifft.real , F ifft.real F ifft The real part of 3-4. Convolution and directional feature extraction: Through convolution and interpolation operations, the feature representation of directional enhancement is extracted to obtain the global feature description, and finally multi-scale feature fusion is performed, as follows: The other branch of the MSAF module sequentially processes the input feature map f5 through convolution, BatchNorm and ReLU, interpolate, and 1×1 convolution to obtain a global feature encoding, and then performs a global average pooling operation on the global feature encoding to obtain a global feature description dprior. The directional enhanced feature representation directionalf5 is expressed as follows: directionalf5=ReLU(Conv1x1(Interpolate(Conv(f5)))) (6) The directionalf5 is upsampled to the scale of the original image through the bilinear interpolation method, and the features are re-encoded. Then, the global feature description d is obtained through the global average pooling operation. prior ,Right now: d prior =GAP(directionalf5) (7) The generated global feature description then passes through the convolution layer, activation layer and convolution layer in sequence to obtain the final output feature (X1, X2, ..., X8) of the branch; The output features of the two branches are subjected to multi-scale feature fusion and then passed through the convolution layer to obtain a feature map; the feature map is residually connected with the original input feature map f5 to output the final fused feature map Step 4 is implemented as follows: AFRP receives feature maps from multiple scales and And through layer-by-layer upsampling, the low-resolution feature map is restored to high resolution; AFRP consists of 5 layers, of which layers 1-4 have the same structure, and the two adjacent layers fuse information at different levels through RSR; the specific structure is as follows: Feature map In the fifth layer, it passes through the GAP layer, ADU layer and GAP layer in turn; feature map In the fourth layer, the output r5 of the last GAP layer of the fifth layer is input into the ADU layer together. The output of the ADU layer passes through another GAP layer, and the output of the GAP layer is combined with the output processed by the RSR layer. Perform residual connection and output r4; similarly, the inputs of the ADU layers in the first, second, and third layers are feature maps And the outputs r2, r3 and r4 obtained in the next layer; finally, the feature results r1, r2, r3 and r4 of the information between the layers are obtained by combining the information output of each layer. The specific process of AFRP is simplified as follows: in, represents the feature map output after upsampling at the Jth layer, corresponding to r1, r2, r3, r4 and r5, F AFRP-J Represents the feature map of the upsampled input of the Jth layer; It represents the feature map output after upsampling at the J-1th layer, where the value of J ranges from 1 to 5; Step 4 is implemented as follows: 4-1. Depth-separable convolution and attention mechanism adjustment of ADU module: The expression formula of depth-wise separable convolution is as follows: F sep =Conv depthwise (F' ADU )+Conv pointwise (F' ADU ) (9) Among them, Conv depthwise Represents the deep convolution operation, Conv pointwise represents the point convolution operation, F' ADU Represents the input of ADU, that is and The specific calculation formula of the attention mechanism is: F att =σ(Conv1×1(GAP(F sep ))) (10) Among them, F att represents the result graph generated by the attention mechanism, σ represents the activation function, and GAP represents the global average pooling; Finally, the input of ADU is connected through residual connection to retain the original spatial information, which is expressed as: Among them, Sigmod represents the activation function, ConBnReLU represents the combination of Conv convolution, BatchNorm and ReLU operations; 4-2. Residual fusion of RSR module: Through residual connection, the feature map F output after the attention mechanism is SE Adding it to the initial feature map not only retains the original feature information, but also improves the expressiveness of the feature; its mathematical expression is: Among them, F refined Represents the feature map after being processed by the RSR module, It represents the feature map output by the Jth layer after the SE attention mechanism. is the initial feature map of the Jth layer, or The EFRE described in step 5 receives the feature results r1, r2, r3 and r4 generated by AFRP, which fuse the information between each layer, and the multi-scale feature map and And through layer-by-layer upsampling operations, the low-resolution feature map is restored to high resolution; the specific implementation is as follows: 5-1. Use of depth-wise separable convolution: Perform depth-wise separable convolution on the feature map of each layer. Assume that the input feature map of the jth layer is r j , the result of the depthwise separable convolution operation is r' j , mathematically expressed as: r' j =Conv depthwise (r j )+Conv pointwise (r j ) (13) Among them, Conv depthwise (r j ) is to perform independent convolution on each channel, Conv pointwise (r j ) is a 1x1 convolution, used to mix the information of each channel, j = 1, 2, ..., 4; 5-2. Upsampling operation and attention mechanism adjustment: EFRE obtains r' from depthwise separable convolution j Upsampling is performed to gradually restore its spatial dimensions. The mathematical representation of this process is: r refined =Upsample(r' j ) (14) Among them, r refined The upsampling operation uses bilinear interpolation to restore the spatial information of the low-resolution image. The attention mechanism is based on the feature map r generated by upsampling refined After processing, the feature map obtained is recorded as 5-3. Skip connection fusion feature: The fusion of multi-level features is performed through residual connection and weighted average, which is expressed as: in, Represents the j-th layer feature map obtained through the attention mechanism. Each layer combines the information of the previous layer. Represents the multi-scale feature map of the jth layer; EFRE obtains the final output feature map by weighted averaging the feature maps of all layers, which is expressed as: Where N is the total number of decoding layers.

Images