Medical image segmentation method combining selective edge aggregation and deep neural network

Abstract

The application discloses a medical image segmentation method combining selective edge aggregation and deep neural networks, first constructs a Transformer-based encoder, and replaces MSA and MLP in a standard Transformer block with a selective edge aggregation module and a densely connected feedforward network to realize feature fusion and complementation; then constructs an encoder and a decoder based on densely connected CNN, connects the two encoders in parallel, enables the network to interact information at multiple levels, and fuses multi-scale features from the double encoders and the low-to-high up-sampling path based on the decoder of the densely connected CNN to restore the spatial resolution of the feature map in a fine-grained and deep-level manner; finally, a loss function combining target edges and regions is designed to simultaneously optimize the encoder and the decoder with a multi-level optimization strategy, so that the network further learns more semantic information and boundary details to refine the segmentation result. The application can solve the medical image segmentation problem in a real scene.

Description

Technical Field

[0001] This invention relates to medical image segmentation technology in the field of image processing, and particularly to a medical image segmentation method that combines selective edge aggregation and deep neural networks. Background Technology

[0002] Medical image segmentation is a widely studied and challenging topic, aiming to help clinicians focus more on pathological regions and extract detailed information from medical images for more accurate diagnosis and analysis. Common medical image segmentation tasks include skin lesion segmentation, glandular segmentation, and thyroid nodule segmentation. However, due to the large scale variations of the targets to be segmented in medical images, blurred target structure boundaries, and numerous modalities, coupled with the lack of high-quality labeled images for training in practice, obtaining accurate segmentation results is extremely difficult.

[0003] With the rapid development of deep learning technology, many end-to-end automatic segmentation methods have been proposed and applied to the field of medical image analysis. U-Net is one of the most widely used medical image segmentation models. It utilizes an encoder to learn high-level semantic representations, a decoder to recover lost spatial information, and applies skip connections to fuse features from different scales of the encoder and decoder to generate a more accurate segmentation mask. Many improved variants of U-Net have been proposed since then, but deep learning-based medical image segmentation methods based on U-Net and its variants do not explicitly consider that accurate boundary prediction can produce higher-quality segmentation masks. To address the problem of unclear structural boundaries, methods such as DeepLab and EANet have been reported, which recover boundary details by learning inter-pixel dependencies. However, they either require manual parameter tuning during post-processing or require carefully designed learnable modules to complete this labor-intensive task. Furthermore, most existing CNN methods cannot establish inter-pixel dependencies and global contextual relationships due to the limitation of the receptive field in convolutional operations. Repeated striding and pooling operations inevitably lose image resolution, making dense prediction tasks challenging. The emergence of Transformer greatly alleviated this problem. It was initially used for natural language processing tasks and is capable of encoding distance dependencies. While Transformer excels at global context modeling, it lacks spatial information about images, particularly limiting its ability to capture structural boundaries. These issues restrict the successful application of pure Transformer on relatively small medical image datasets.

[0004] In summary, efficient boundary prediction, fusion of context and spatial features, and good performance on smaller datasets are key issues that urgently need to be addressed in medical image segmentation. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to provide a medical image segmentation method that combines selective edge aggregation and deep neural networks. This method captures the global contextual features and shallow spatial features of the image, enabling the network to have multi-scale learning capabilities and to select and retain edge-related features without requiring additional learning. The technical solution is as follows:

[0006] A medical image segmentation method combining selective edge aggregation and deep neural networks includes the following steps:

[0007] Step 1: Select a publicly available medical image segmentation dataset and preprocess the training set in the dataset;

[0008] Step 2: Construct a selective edge aggregation module to enable the network to focus on the accuracy of edge segmentation;

[0009] Step 3: Construct a densely connected feedforward network to enable feature reuse and the network's multi-scale learning capabilities;

[0010] Step 4: Design a Transformer-based encoder structure that includes a selective edge aggregation module and a densely connected feedforward network, while preserving global contextual information of the image;

[0011] Step 5: Design an encoder and decoder structure based on densely connected CNNs to extract local image information and spatial texture information;

[0012] Step 6: Construct a multi-level optimization strategy to simultaneously optimize the encoder and decoder so that they learn boundary-related information and generate better feature representations;

[0013] Step 7: Design an image segmentation framework consisting of three parts: a Transformer encoder structure, a densely connected CNN-based encoder and decoder structure, and a multi-level optimization strategy, to complete the segmentation of medical images.

[0014] Furthermore, the medical image segmentation dataset in step 1 includes: ISIC2017, PH2, TN-SCUI 2020 challenge, GLAnd segmentation, and COVID-19 Infection segmentation. The training set in the dataset is preprocessed as follows: after normalizing the image colors of ISIC2017 and PH2, the resolution of all images is adjusted to 224x224 pixels; the resolution of all images in TN-SCUI 2020 challenge and GLAnd segmentation is adjusted to 224x224 pixels; and the resolution of all images in COVID-19 Infection segmentation is adjusted to 352x352 pixels.

[0015] Furthermore, the specific process of step 2 is as follows:

[0016] Step 2.1: Represent the input feature map after activation of any convolutional layer as follows:

[0017] X Sig ∈R H×W×C

[0018] Where H and W are the height and width of the image, respectively, and C represents the number of channels;

[0019] Step 2.2: Extract the input feature map X using edge extraction blocks through max pooling operation. Sig The boundary features are used to output the pooling operation result X. EEB It can be expressed by the following formula:

[0020] X EEB =Maxpooling(1-X Sig ,K)-(1-X Sig )

[0021] Where K represents the size of the sliding window;

[0022] Step 2.3: Use the salient feature selection block to set a threshold to select the input feature map mapping X. Sig The salient features are achieved through the following three steps:

[0023] ① Deep aggregation of input feature map mapping X Sig The channel information, i.e.

[0024] ② Calculate the aggregated channel information X agg Average value of all positions

[0025] ③ Average The aggregated channel information X is selected as a threshold. agg The saliency features, output results The formula is expressed as follows;

[0026]

[0027] Where the superscript (x,y) represents the coordinates of a specific location; x,y∈[0,1,…,H-1],[0,1,…,W-1], and X SFS ∈R H×W×1 ;

[0028] Step 2.4: Place X EEB and X SFSElement-wise multiplication yields a feature map M0∈R that simultaneously masks the background region and the target boundary. H×W×C ;

[0029] Step 2.5: Use the channel selection algorithm to preserve the feature map M0∈R H×W×C Select the channels that meet the expected results, block the channels that do not meet the expectations, and output the channel selection result X. out ∈R H×W×C ;

[0030] Step 2.6: Represent the input feature map after resolution downscaling as follows:

[0031] T in ∈R H×W×C

[0032] Where H and W are the height and width of the image, respectively, and C represents the number of channels;

[0033] Step 2.7: Aggregate the input feature map T using flat pooling. in Features, output result T avg It can be expressed by the following formula:

[0034] T avg =Avgpooling(T in ,K)-T in

[0035] Wherein, K represents the size of the sliding window, and in this invention, K is 3;

[0036] Step 2.8: Activate T using the Sigmoid activation function. avg With X out Feature concatenation is performed, and then the concatenated feature map is averaged and aggregated. The averaged and aggregated feature map is then activated to obtain the weight map T. out The formula is expressed as follows:

[0037]

[0038] Where f represents the Sigmoid activation function, C represents the number of channels, and c represents the index of the image channel. Indicates feature concatenation operation;

[0039] Step 2.9: For X EEB and T out Perform element-wise addition and then compare the result with T. out Perform element-wise multiplication to obtain the output of the Selective Edge Aggregation (SEA) module. out .

[0040] Furthermore, the specific process of constructing the densely connected feedforward network (Dense MLP) in step 3 is as follows:

[0041] Step 3.1: Represent the feature map output by the Selective Feature Aggregation (SEA) module as follows:

[0042]

[0043] Where H and W are the height and width of the image, respectively, and C represents the number of channels;

[0044] Step 3.2: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Reshape to Where S = H × W;

[0045] Step 3.3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] All of these are used as inputs to the next layer, expressed by the following formula:

[0046]

[0047]

[0048] .......,

[0049]

[0050] Wherein, MLP represents a layer in a densely connected feedforward network (Dense MLP), and M represents the channel growth rate, i.e., the output dimension of the MLP. In this invention, M is 16.

[0051] Furthermore, in step 4, the Transformer-based decoder consists of multiple interconnected Transformer blocks. Each Transformer block comprises a normalization layer, a Selective Edge Aggregation (SEA) module, and a Dense MLP (Dense MLP) network. A PatchEmbedding layer is added before each Transformer block to reduce the resolution of the input feature map. The processing procedure is as follows:

[0052] Step 4.1: Reduce the feature map resolution of the input Transformer block using the Patch Embedding layer. This is mainly accomplished in the following three steps:

[0053] ① Represent the input feature map of the Transformer block as X in ∈R H×W×C Where H and W are the height and width of the image, respectively, and C represents the number of channels;

[0054] ②For X inBy performing pixel-by-pixel sampling, the number of channels is increased to four times the original number, resulting in...

[0055] ③ Using a grouped convolution with a kernel of 1 and a group number of 4, The channel mapping is with X in With the same number of channels, we get X emb ∈R H / 2×W / 2×C ;

[0056] Step 4.2: Place X emb The input consists of a Transformer block comprising a normalization layer, an SEA module, and a Dense MLP, and is represented by the following formula:

[0057]

[0058]

[0059] in, These represent features from the Transformer branch after Patch Embedding and features from the CNN branch, respectively. Norm represents the normalization layer, SEA represents the Selective Edge Aggregation (SEA) module, and DenseMLP represents the Dense Connected Feedforward Network (Dense MLP).

[0060] Furthermore, step 5, which designs the encoder and decoder structure based on a densely connected CNN, includes the following specific steps:

[0061] Step 5.1: Starting from the first convolutional block of the encoder, the input feature map is downsampled by a factor of two, and the final resolution becomes (H / 16, W / 16);

[0062] Step 5.2: Construct the fusion features that enable skip connections between the encoder and decoder, expressed by the following formula:

[0063]

[0064] in, This indicates element-wise addition. and These represent the outputs of the (l-1)th block of the CNN encoder and the Transformer encoder, respectively, and DenseConv represents a densely connected convolutional block.

[0065] Step 5.3: Use a standard convolution to reduce the number of channels in the concatenated array to 1 / 4 of the original number, and then increase the number of channels to 1 / 2 of the original number through a series of densely connected convolutional blocks.

[0066] Furthermore, the specific steps for constructing the multi-level optimization strategy in step 6 are as follows:

[0067] Step 6.1: Calculate the overlap error between the predicted result and the true value using the IoU loss, i.e., the target region loss l. IoU This can be expressed as a formula:

[0068]

[0069] Where P represents the network's prediction result, G represents the true value, and the subscript i represents the pixel index;

[0070] Step 6.2: The boundary loss used to minimize the boundary error between P and G is calculated through the following steps:

[0071] ① Use max pooling to extract the boundary P between P and G. b With G b The formula is as follows:

[0072] G b =Maxpooling(1-G,K)-(1-G),

[0073] P b =Maxpooling(1-P,K)-(1-P)

[0074] Wherein, K represents the size of the sliding window, and in this invention, K is 3;

[0075] ②Use P b With G b Construction boundary loss l Edge The formula is as follows:

[0076]

[0077] in, and Let represent the true value and the predicted boundary probability value at the i-th position, respectively, and α be the weighting coefficient for balancing the number of pixels;

[0078] Step 6.3: Utilize target region loss IoU and boundary loss l Edge Calculate the loss function l Seg :

[0079] l Seg =λ1l IoU +λ2l Edge

[0080] Where λ1 and λ2 are the loss in the target region for balancing. IoU and boundary loss l Edge Weighting coefficients;

[0081] Step 6.4: Process the probability map P based on the Transformer encoder output. e The probability map P output by the decoder based on densely connected CNNs d Multi-level optimization is performed to obtain the overall loss during the training phase. Total The formula is as follows:

[0082]

[0083] Where N represents the number of Transformer blocks in the Transformer encoder, and n represents the index of the Transformer block.

[0084] Furthermore, step 7 designs an image segmentation framework consisting of three parts: a Transformer encoder structure, a densely connected CNN-based encoder and decoder structure, and a multi-level optimization strategy. The specific process for segmenting medical images is as follows:

[0085] Step 7.1: Input the original image into the Transformer encoder and the densely connected CNN encoder. Use the Transformer branch to capture and preserve global context information, and use the CNN branch to extract local information and spatial texture information.

[0086] Step 7.2: Fuse multi-scale features from the dual encoder and upsampling path from low to high using a densely connected CNN decoder;

[0087] Step 7.3: Directly extend the output of the Transformer encoder to the target size and calculate the loss with the ground truth value, calculate the loss with the output of the densely connected CNN decoder and the ground truth value, and simultaneously optimize the encoder and decoder in a multi-level optimization manner.

[0088] The beneficial effects of adopting the above technical solution are as follows:

[0089] 1) This invention proposes a novel and effective medical image segmentation framework that combines selective edge aggregation and deep neural networks to comprehensively solve the problem of medical image segmentation. This framework can handle the problems of scale diversity and structural boundary ambiguity in medical images of different modalities, and can still show excellent segmentation performance even on small medical image segmentation datasets.

[0090] 2) This invention designs a method that selectively aggregates edge information without requiring an additional supervised selective edge aggregation (SEA) module, allowing the network to focus more on the accuracy of edge segmentation. Furthermore, the consistently dense connection approach enables the codec to have a smaller number of parameters and multi-scale learning capabilities.

[0091] 3) This invention constructs a loss function that combines target edges and regions, and employs a multi-level optimization strategy to simultaneously optimize the encoder and decoder. This optimization approach encourages the encoder to learn more boundary-related information, resulting in better feature representations. Attached image description:

[0092] Figure 1 This is the selective edge aggregation module of the present invention.

[0093] Figure 2 This is the edge extraction block of the present invention.

[0094] Figure 3 This is a block for selecting significant features of the present invention.

[0095] Figure 4 This is the Transformer block of the present invention.

[0096] Figure 5 This is a flowchart of the medical image segmentation method combining selective edge aggregation and deep neural networks according to the present invention. Detailed Implementation

[0097] The technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings.

[0098] This invention presents a medical image segmentation method combining selective edge aggregation (SEA) and deep neural networks. First, an encoder is constructed by combining a densely connected CNN and a Transformer with a dense feedforward network (Dense MLP) in parallel, while the densely connected CNN serves as the decoder. This approach effectively captures shallow texture information and global contextual information in medical images at a deeper, multi-scale level. Second, a plug-and-play selective edge aggregation (SEA) module is proposed. This module removes noisy backgrounds in an unsupervised manner, selecting and preserving useful edge features, allowing the network to focus more on information related to the target boundary. Furthermore, a loss function combining target content and edges is designed, and a multi-level optimization strategy is employed to refine the blurred structure, helping the network learn better feature representations and produce more accurate segmentation results.

[0099] The proposed method was evaluated on several different challenging medical segmentation tasks and performed well compared to most state-of-the-art methods, with fewer parameters and fewer GFlops compared to other methods.

[0100] Step 1: Select a publicly available medical image segmentation dataset and preprocess the dataset.

[0101] The specific implementation of preprocessing the training set is as follows:

[0102] This invention performs segmentation training tasks on four publicly available medical image segmentation datasets. These datasets are: ISIC2017, PH2, TN-SCUI 2020 challenge, GLAnd segmentation, and COVID-19 Infectionsegmentation.

[0103] The ISIC2017 dataset, provided by the International Skin Imaging Collaboration, includes 2000 training images, 150 validation images, and 600 test images. The PH2 dataset consists of 200 skin mirror images with a resolution of 765×572 pixels. 140 images were randomly selected as the training set, 20 as the validation set, and the remaining 40 as the test set. First, the colors of the images on both datasets were normalized using a gray-world color consistency algorithm. Then, the resolution of all images was adjusted to 224×224 pixels for experimentation. Finally, training data augmentation was performed during training to improve the model's generalization ability.

[0104] The TN-SCUI 2020 challenge dataset provides 3644 thyroid nodule images of varying sizes, with annotations provided by experienced physicians. The training, validation, and test sets were initially divided into a 6:2:2 ratio. During training, data augmentation techniques such as random rotation, random horizontal and vertical shifts, and random flipping were applied to the training set to increase its diversity. All images were uniformly adjusted to a resolution of 224×224 pixels.

[0105] The GLAnd segmentation (GLAS) dataset contains microscopic images of hematoxylin and eosin stained slides, along with ground truth values ​​provided by expert pathologists. The dataset contains 165 images of varying resolutions, ranging from a minimum of 433×574 pixels to a maximum of 775×522 pixels. Eighty-five images were selected for training, and eighty for testing. In the experiments, all images were resized to 224×224 pixels.

[0106] The COVID-19 Infection segmentation dataset contains 100 axial CT images and corresponding labeled images from over 40 COVID-19 patients. Given the small size of this dataset, five-fold cross-validation was used in the experiments (i.e., 80 images were used for training and 20 images for validation each time). During training, data augmentation strategies were employed to increase the diversity of the training set, and the images were uniformly resized to 352×352 pixels.

[0107] Step 2: Construct a Selective Edge Aggregation (SEA) module to focus the network on the accuracy of edge segmentation. This module receives features from both the Transformer and CNN branches. Since CNN can capture the spatial information of the segmented target better, the CNN branch is used to supplement the Transformer branch, enabling feature fusion and complementarity between the two branches. (Refer to...) Figure 1 This is the selective edge aggregation module of the present invention. The specific construction steps are as follows:

[0108] 1) Represent the input feature map after activation of any convolutional layer as follows:

[0109] X Sig ∈R H×W×C

[0110] Where H and W are the height and width of the image, respectively, and C represents the number of channels.

[0111] 2) Extract X using edge extraction blocks (EEB) via max pooling in the CNN branch. Sig The boundary features, output result X EEB ,refer to Figure 2 The edge extraction block of this invention is represented by the following formula:

[0112] X EEB =Maxpooling(1-X Sig ,K)-(1-X Sig )

[0113] Where K represents the sliding window size of max pooling, and in this invention K is 3.

[0114] 3) In the CNN branch, use the Satisfactory Feature Selection Block (SFS) to set a threshold for selecting X. Sig The salient features, refer to Figure 3 The selection of salient features for this invention is accomplished through the following three steps:

[0115] ①Deep Aggregation X Sig The channel information, i.e.

[0116] ② Calculate X agg Average value of all positions

[0117] ③ X as a threshold selection agg The saliency features, output results The formula is expressed as follows;

[0118]

[0119] Where the superscript (x,y) represents the coordinates of a specific location; x,y∈[0,1,…,H-1],[0,1,…,W-1], and X SFS ∈R H×W×1 .

[0120] 4) X EEB and X SFS Element-wise multiplication yields a feature map M0∈R that simultaneously masks the background region and the target boundary. H×W×C .

[0121] 5) Use the channel selection algorithm to preserve the feature map M0∈R H×W×C Select the channels that meet the expected results, and block the channels that do not meet the expectations. Output the result X. out ∈R H×W×C .

[0122] 6) In the Transformer branch, the input feature map after resolution downscaling is represented as:

[0123] T in ∈R H×W×C

[0124] Where H and W are the height and width of the image, respectively, and C represents the number of channels.

[0125] 7) Aggregate the input feature map T using the flat pooling operation. in Features, output result T avg It can be expressed by the following formula:

[0126] T avg =Avgpooling(T in ,K)-T in

[0127] Where K represents the sliding window size of the average pooling, and in this invention K is 3.

[0128] 8) The T after activation using the Sigmoid activation function avg With X out Feature concatenation is performed, and then the concatenated feature map is averaged and aggregated. The averaged and aggregated feature map is then activated to obtain the weight map T. out The formula is expressed as follows:

[0129]

[0130] Where f represents the Sigmoid activation function, C represents the number of channels, and c represents the index of the image channel. This indicates a feature splicing operation.

[0131] 9) Regarding X EEB and T out Perform element-wise addition and then compare the result with T. out Perform element-wise multiplication to obtain the output of the Selective Edge Aggregation (SEA) module. out .

[0132] Step 3: Construct Transformer blocks. Each Transformer block contains a densely connected feedforward network (Dense MLP). This densely connected feedforward network is constructed using dense connections to apply linear layers along the channel direction, further improving information flow between channels. (See reference...) Figure 4 This is the Transformer block of the present invention. The specific construction process is as follows:

[0133] 1) The feature map output by the Selective Feature Aggregation (SEA) module is represented as:

[0134]

[0135] Where H and W are the height and width of the image, respectively, and C represents the number of channels.

[0136] 2) Reshape to Where N = H × W.

[0137] 3) All of these are used as inputs to the next layer, expressed by the following formula:

[0138]

[0139]

[0140] …….,

[0141]

[0142] Wherein, MLP represents a layer in a densely connected feedforward network (Dense MLP), and M represents the channel growth rate, i.e., the output dimension of the MLP. In this invention, M is 16.

[0143] Step 4: Construct a Transformer-based decoder. This decoder consists of multiple interconnected Transformer blocks. Each Transformer block comprises a normalization layer, a Selective Edge Aggregation (SEA) module, and a Dense MLP (Dense MLP) network. This allows it to adapt to high-resolution images and complement the spatial features captured by the CNN. Furthermore, a Patch Embedding layer is added before each Transformer block to reduce the resolution of the input feature map, enabling the Transformer to expand its receptive field layer by layer, similar to a CNN. (Refer to...) Figure 5 The flowchart of the medical image segmentation method combining selective edge aggregation and deep neural networks of the present invention is shown, wherein the Transformer-based encoder is... Figure 5 The "Transformer Encoder" branch is used. The specific implementation steps are as follows:

[0144] 1) Reduce the feature map resolution of the input Transformer block using the Patch Embedding layer, which mainly consists of the following three steps.

[0145] Finish:

[0146] ① Represent the input feature map of the Transformer block as X in ∈R H×W×C Where H and W are the height and width of the image, respectively, and C represents the number of channels;

[0147] ②For X in By performing pixel-by-pixel sampling, the number of channels is increased to four times the original number, resulting in...

[0148] ③ Using a grouped convolution with a kernel of 1 and a group number of 4, The channel mapping is with X in With the same number of channels, we get X emb ∈R Hl2×W / 2×C .

[0149] 2) X emb The input consists of a Transformer block comprising a normalization layer, an SEA module, and a Dense MLP, and is represented by the following formula:

[0150]

[0151]

[0152] in, These represent features from the Transformer branch after Patch Embedding and features from the CNN branch, respectively. Norm represents the normalization layer, SEA represents the Selective Edge Aggregation (SEA) module, and DenseMLP represents the Dense Connected Feedforward Network (Dense MLP).

[0153] Step 5: Construct a densely connected CNN-based encoder and decoder, which is a U-shaped network. The encoder extracts semantic information from the medical image from shallow to deep layers, and the decoder recovers the spatial resolution of the encoder's output features. Furthermore, skip connections are applied to obtain detailed information from the encoder and decoder to compensate for information loss due to downsampling and convolution operations. Reference Figure 5 The flowchart of the medical image segmentation method combining selective edge aggregation and deep neural networks of the present invention is shown, wherein the encoder and decoder are based on densely connected CNNs. Figure 5 The "CNNEncoder" and "CNN Decoder" branches in the code. The specific steps for designing an encoder and decoder architecture based on a densely connected CNN include:

[0154] 1) Starting from the first convolutional block of the encoder, the input feature map is downsampled by a factor of two, and the final resolution becomes (H / 16, W / 16).

[0155] 2) Construct the fusion features that make skip connections between the encoder and decoder, expressed by the following formula:

[0156]

[0157] in, This indicates element-wise addition. and represents the output of the (l-1)th block of the CNN encoder and the Transformer encoder, respectively, and DenseConv represents the densely connected convolutional block.

[0158] 3) Use a standard convolution to reduce the number of channels in the concatenation to 1 / 4 of the original number, and then increase the number of channels to 1 / 2 of the original number through a series of densely connected convolutional blocks.

[0159] Step 6: To reduce the discrepancy between the predicted and true values, this paper uses two loss functions to address two independent aspects: segmentation content and segmentation boundaries. The first is the IoU loss, used to minimize the overlap error between the predicted and true values; the second is the boundary loss, used to minimize the boundary error between the predicted and true values. Furthermore, a multi-level optimization strategy is introduced to simultaneously optimize the encoder and decoder, as described in [reference needed]. Figure 5The flowchart of the medical image segmentation method combining selective edge aggregation and deep neural networks of the present invention includes a multi-level optimization strategy and... Figure 5 The "MLOStrategy" branch in the code. The specific steps for designing the loss function and multi-level optimization strategy include:

[0160] 1) Calculate the overlap error between the predicted result and the true value using the IoU loss, i.e., the target area loss l. IoU This can be expressed as a formula:

[0161]

[0162] Where P represents the network's prediction result, G represents the true value, and i represents the index of all pixels in P and G.

[0163] 2) Used to minimize P i With G i The boundary loss of the boundary error between them is calculated through the following steps:

[0164] ① Use max pooling to extract the boundary P between P and G. b With G b The formula is as follows:

[0165] G b =Maxpooling(1-G,K)-(1-G),

[0166] P b =Maxpooling(1-P,K)-(1-P)

[0167] Where K represents the sliding window size of max pooling, and in this invention K is 3;

[0168] ②Use and Construction boundary loss l Edge The formula is as follows:

[0169]

[0170] in, and Let represent the true value and the predicted boundary probability value at the i-th position, respectively, and α be the weighting coefficient for balancing the number of pixels.

[0171] 3) Utilizing l IoU and l Edge Calculate the loss function l Seg ,Right now

[0172] l Seg =λ1l IoU +λ2l Edge

[0173] Where λ1 and λ2 are the loss in the target region for balancing. IoU and boundary loss l Edge Weighting coefficients;

[0174] 4) The probability map P based on the output of the Transformer encoder e The probability map P output by the decoder based on densely connected CNNs d Multi-level optimization is performed to obtain the overall loss during the training phase. Total The formula is as follows:

[0175]

[0176] Where N represents the number of Transformer blocks in the Transformer encoder, and n represents the index of the Transformer block.

[0177] Step 7: Design an image segmentation framework consisting of three parts: a Transformer encoder structure, a densely connected CNN-based encoder and decoder structure, and a multi-level optimization strategy. (Refer to...) Figure 4 This is a flowchart of the medical image segmentation method based on a full-resolution representation network according to the present invention. The specific process of completing the medical image segmentation is as follows:

[0178] 1) This framework consists of three modules:

[0179] This invention constructs a Transformer-based encoder consisting of multiple Transformer blocks to capture and retain important global contextual information. The Transformer blocks constructed in this invention differ from standard Transformer blocks. A Patch embedding layer is introduced before the Transformer blocks to adapt to high-resolution images and dense prediction tasks. Furthermore, the MSA and MLP in the standard Transformer blocks are replaced with the Selective Edge Aggregation (SEA) module and Dense Connection Feedforward Network (Dense MLP) constructed in this invention. This allows the network to accept features from both the Transformer-based encoder and the Dense Connection CNN-based encoder, achieving feature fusion and complementarity. Through the Dense Connection CNN-based encoder and decoder, the network possesses a natural multi-scale feature extraction capability. The parallel connection between the Transformer-based encoder and the Dense Connection CNN-based encoder enables information interaction at multiple levels, fully utilizing both local and global image information. The Dense Connection CNN-based decoder fuses multi-scale features from the dual encoders and upsampling paths from low to high scale, recovering the spatial resolution of the feature maps in a finer-grained and deeper manner. In addition, a loss function combining target edges and regions was designed to simultaneously optimize the encoder and decoder with a multi-level optimization strategy, enabling the network to learn more semantic information and boundary details, thus refining the segmentation results.

[0180] 2) Model architecture and hyperparameter settings:

[0181] This invention implements a Keras-based method through training on an NVIDIA RTX3090 GPU (24g). The Adam optimizer is used, with a fixed learning rate of 1e-4. The mini-batch size is set to 16, and training is stopped early when the validation loss stabilizes and shows no significant change over 30 epochs. Training data is augmented by applying random rotations (±25°), random horizontal and vertical shifts (15%), and random flips (horizontal and vertical). Furthermore, all comparative experiments use the same training and validation sets. After the second stage of the SEAformer CNN branch, the initial weights are derived from Blocks 2, 3, and 4 of a pre-trained DenseNet121 on ImageNet; the remaining layers are trained from scratch.

[0182] 3) Model Evaluation Methods

[0183] This invention uses five widely used metrics to evaluate model performance: accuracy (Acc), sensitivity (Sens), specificity (Spec), intersection over union (IoU), and dicesimilarity coefficient (Dice). The number of parameters, GFLOPs, and FPS of this invention are also reported.

[0184] 4) The specific implementation of the model is as follows:

[0185] The original image input is based on a Transformer encoder and a densely connected CNN encoder. The Transformer branch captures and preserves global contextual information, while the CNN branch extracts local and spatial texture information. Multi-scale features from the dual encoders and upsampling paths are fused through a densely connected CNN decoder. The output of the Transformer encoder is directly extended to the target size and the ground truth to calculate the loss. The output of the densely connected CNN decoder is also used to calculate the loss with the ground truth. The encoder and decoder are optimized simultaneously in a multi-level optimization manner.

Claims

1. A medical image segmentation method combining selective edge aggregation and deep neural networks, characterized in that, Includes the following steps: Step 1: Select a publicly available medical image segmentation dataset and preprocess the training set in the dataset; Step 2: Construct a selective edge aggregation module to enable the network to focus on the accuracy of edge segmentation; Step 3: Construct a densely connected feedforward network to enable feature reuse and the network's multi-scale learning capabilities; Step 4: Design a Transformer-based encoder structure that includes a selective edge aggregation module and a densely connected feedforward network, while preserving global contextual information of the image; The Transformer-based decoder consists of multiple Transformer blocks that are repeatedly connected. Each Transformer block includes three parts: a normalization layer, a selective edge aggregation module, and a densely connected feedforward network. Furthermore, a Patch Embedding layer is added before each Transformer block to reduce the resolution of the input feature map. Step 5: Design an encoder and decoder structure based on densely connected CNNs to extract local image information and spatial texture information; Step 6: Construct a multi-level optimization strategy to simultaneously optimize the encoder and decoder so that they learn boundary-related information and generate better feature representations; Step 7: Design an image segmentation framework consisting of three parts: a Transformer encoder structure, a densely connected CNN-based encoder and decoder structure, and a multi-level optimization strategy, to complete the segmentation of medical images; The specific process of step 2 is as follows: Step 2.1: Map the input feature map after activation of any convolutional layer Represented as: ； Where H and W are the height and width of the image, respectively, and C represents the number of channels; Step 2.2: Extract the input feature map mapping using edge extraction blocks through max pooling operation. The boundary features are used to output the pooling operation results. It can be expressed by the following formula: ； Where K represents the size of the sliding window; Step 2.3: Use salient feature selection blocks to set thresholds to select input feature map mappings. The salient features are achieved through the following three steps: ① Deep aggregation of input feature map mapping The channel information, i.e. ; ② Calculate the aggregated channel information Average value of all positions ; ③ Average Channel information selected as threshold The saliency features are identified, and the results of the saliency features are output. The formula is expressed as follows; ； Among them, superscript Coordinates representing a specific location; ,and ; Step 2.4: and Element-wise multiplication yields a feature map that simultaneously masks the background region and the target boundary. ; Step 2.5: Preserve the feature map using the channel selection algorithm. Select the channels that meet the expected results, block the channels that do not, and output the channel selection results. ; Step 2.6: Represent the input feature map after resolution downscaling as follows: ； Where H and W are the height and width of the image, respectively, and C represents the number of channels; Step 2.7: Aggregate the input feature maps using flat pooling. The characteristics of the output result of the flat pooling operation. It can be expressed by the following formula: ； Step 2.8: Will use After activation function and Feature concatenation is performed, and then the concatenated feature map is averaged and aggregated. The averaged and aggregated feature map is then activated to obtain the weight map. The formula is expressed as follows: ； in, express Activation function Indicates the number of channels. Indicates the index of the image channel. Indicates feature concatenation operation; Step 2.9: For and weighted graph Perform element-wise addition and compare the result with the weight graph. The output of the selective edge aggregation module is obtained by performing element-wise multiplication. ; The specific process of constructing the densely connected feedforward network in step 3 is as follows: Step 3.1: Represent the feature map output by the selective feature aggregation module as follows: ； Where H and W are the height and width of the image, respectively, and C represents the number of channels; Step 3.2: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Reshape to ,in ; Step 3.3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] All of these are used as inputs to the next layer, expressed by the following formula: ； ； ； ； in, This represents a layer in a densely connected feedforward network. This represents the growth rate of the channel, i.e. The output dimension.

2. The medical image segmentation method combining selective edge aggregation and deep neural networks according to claim 1, characterized in that, The medical image segmentation datasets in step 1 are: ISIC2017, PH2, TN-SCUI 2020 challenge, GLAnd segmentation, and COVID-19 Infection segmentation. The training set in the dataset is preprocessed as follows: after normalizing the image colors of ISIC2017 and PH2, the resolution of all images is adjusted to 224x224 pixels. The resolution of all images in TN-SCUI 2020 challenge and GLAnd segmentation is adjusted to 224x224 pixels. The resolution of all images in COVID-19 Infection segmentation is adjusted to 352x352 pixels.

3. The medical image segmentation method combining selective edge aggregation and deep neural networks according to claim 1, characterized in that, In step 4, the processing procedure is as follows: Step 4.1: Reduce the feature map resolution of the input Transformer block using the Patch Embedding layer, which is accomplished in the following three steps: ① Represent the input feature map of the Transformer block as follows: ; ② By performing pixel-by-pixel sampling, the number of channels is increased to four times the original number, resulting in... ; ③ Using a grouped convolution with a kernel of 1 and a group number of 4, The channel mapping is as follows: With the same number of channels, we get ; Step 4.2: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] The input consists of a Transformer block comprising a normalization layer, an SEA module, and a Dense MLP, and is represented by the following formula: ； ； in, , These represent features from the Transformer branch after Patch Embedding and features from the CNN branch, respectively. Represents the standardization layer. This indicates a selective edge aggregation module. This indicates a densely connected feedforward network.

4. The medical image segmentation method combining selective edge aggregation and deep neural networks according to claim 3, characterized in that, The specific steps in step 5, designing the encoder and decoder structure based on densely connected CNN, include: Step 5.1: Starting from the first convolutional block of the encoder, downsample the input feature map by a factor of two sequentially, resulting in a final resolution of [resolution value missing]. ; Step 5.2: Construct the fusion features that enable skip connections between the encoder and decoder, expressed by the following formula: ； in, This indicates element-wise addition. and These represent the first and second encoder steps of the CNN encoder and the Transformer encoder, respectively. Block output, Represents a densely connected convolutional block; Step 5.3: Use a standard convolution to reduce the number of channels in the concatenated array to the original number. Then, the number of channels is increased to that of the original channels through a series of densely connected convolutional blocks. .

5. The medical image segmentation method combining selective edge aggregation and deep neural networks according to claim 4, characterized in that, The specific steps for constructing the multi-level optimization strategy in step 6 are as follows: Step 6.1: Calculate the overlap error between the predicted result and the true value using the IoU loss, i.e., the target region loss. This can be expressed as a formula: ； in, This represents the network's prediction results. Indicates the actual value, subscript Indicates the index of a pixel; Step 6.2: For minimization and The boundary loss of the boundary error between them is calculated through the following steps: ① Extract using max pooling operation and boundary and The formula is as follows: ； ； Where K represents the size of the sliding window; ②Use and Construction boundary loss The formula is as follows: ； in, and They represent the first The true value and predicted boundary probability value at each location The weighting coefficients are used to balance the number of pixels; Step 6.3: Utilize target area loss and boundary loss Calculate the loss function : ； in, and To balance the loss in the target area and boundary loss Weighting coefficients; Step 6.4: Probability map based on Transformer encoder output and the probability graph output by the decoder based on densely connected CNN Multi-level optimization is performed to obtain the overall loss during the training phase. The formula is as follows: ； in, This indicates the number of Transformer blocks in the Transformer encoder. Indicates the index of the Transformer block.

6. The medical image segmentation method combining selective edge aggregation and deep neural networks according to claim 5, characterized in that, Step 7 designs an image segmentation framework consisting of three parts: a Transformer encoder structure, a densely connected CNN-based encoder and decoder structure, and a multi-level optimization strategy. The specific process for segmenting medical images is as follows: Step 7.1: Input the original image into the Transformer-based encoder and the densely connected CNN-based encoder. Use the Transformer branch to capture and preserve global contextual information, and use the CNN branch to extract local information and spatial texture information. Step 7.2: Fuse multi-scale features from the dual encoder and upsampling path from low to high using a densely connected CNN decoder; Step 7.3: Directly extend the output of the Transformer encoder to the target size and calculate the loss with the ground truth value, calculate the loss with the output of the densely connected CNN decoder and the ground truth value, and simultaneously optimize the encoder and decoder in a multi-level optimization manner.

Images