Pancreatic image segmentation method and system based on feature aggregation and modal adaptation

Abstract

The invention discloses a pancreatic image segmentation method and system based on feature aggregation and modal self-adaption, and the method comprises the steps: employing a plurality of stages of medical images as a main modal and an auxiliary modal, integrating the information of the main modal and the auxiliary modal in space, enhancing the feature expression of the images, building the remote dependence relation between the modals, and achieving the segmentation of the pancreatic image. And the automatic segmentation precision of the target tissue in the medical image is effectively improved. When the image segmentation method provided by the invention is applied to pancreas CT image segmentation, a high-precision segmentation result can be provided for label-free pancreas enhanced CT image data, and the time for manually sketching the image is saved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a pancreatic image segmentation method and system based on feature aggregation and modality adaptation. Background Art

[0002] Pancreatic cancer is a highly invasive and highly malignant tumor. In recent years, the incidence and mortality of pancreatic cancer have been on the rise worldwide. Early detection is difficult, resulting in a 5-year survival rate of only 11%. For the early detection and localization diagnosis of pancreatic diseases, the most commonly used imaging method is computed tomography (CT). Therefore, automatic and accurate pancreatic segmentation is crucial for locating pancreatic lesions and monitoring pathological growth.

[0003] Currently, accurate segmentation of the pancreas has the following challenges: first, the pancreas occupies less than 2% of the entire CT image area, which is relatively small; second, its shape and size show considerable individual differences; third, due to the similar intensity to the surrounding tissues, it is difficult to distinguish its boundaries from adjacent structures, resulting in blurred boundaries.

[0004] Traditionally applied methods for automatic pancreas segmentation, such as statistical shape modeling, level set techniques, discriminative learning, and dictionary learning, usually rely on predefined features or thresholds and may exhibit limited representation capacity when dealing with significant changes in pancreatic appearance and shape. In recent years, the development of deep learning has opened up new avenues for fast and accurate pancreas segmentation. Depending on the type of input data used, these techniques can be divided into unimodal and multimodal methods. For unimodal pancreas segmentation, techniques such as Convolutional Neural Networks (CNNs), fully convolutional networks, and U-Net have difficulty capturing long-range semantic relationships due to their reliance on standard convolutions and lack of awareness of global contextual information.

[0005] On the other hand, multimodal pancreas segmentation methods aim to utilize information from different imaging modalities. However, traditional methods usually use simple concatenation or element-by-element addition to combine multimodal information, ignoring the relationship between modalities, and have limited ability to exploit the complementary nature of multimodal images during feature extraction. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-modal pancreatic image segmentation method and system based on feature aggregation and modality adaptation, so as to achieve a more accurate segmentation effect of the pancreatic image.

[0007] In order to achieve the above-mentioned object of the invention, the present invention provides a pancreatic image segmentation method based on feature aggregation and modality adaptation, comprising:

[0008] Taking multi-phase medical images as input, including a main modality and an auxiliary modality, wherein the pancreatic enhanced CT venous phase image is the main modality, and the pancreatic enhanced CT plain scan phase image and the pancreatic enhanced CT arterial phase image are the auxiliary modalities;

[0009] Extracting features of the multiple medical images respectively, and integrating feature representations of the main modality and the auxiliary modality;

[0010] Establishing long-range spatial dependencies between different modalities;

[0011] Get medical segmentation images.

[0012] Optionally, the step of taking multi-phase medical images as input includes: pancreatic enhanced CT venous phase image as a main modality, and pancreatic enhanced CT plain scan phase image and pancreatic enhanced CT arterial phase image as auxiliary modalities.

[0013] Optionally, the step of respectively extracting the features of the multi-phase medical images includes: using multiple encoding paths to respectively calculate partition feature maps of the input modalities, and using the encoding paths to respectively obtain multi-scale representations of the input modalities to obtain original feature maps;

[0014] The step of integrating the feature representation of the main modality and the auxiliary modality comprises:

[0015] spatially integrating feature representations of the primary modality and the auxiliary modality at multiple levels of the encoding pathway;

[0016] The original feature maps of the main modality and the auxiliary modality are used as input feature maps, each pixel is weighted, and the information of the main modality and the auxiliary modality are spatially fused.

[0017] Optionally, it further includes: performing maximum pooling and average pooling on the input feature map in the channel dimension to obtain global feature maps of the main modality and the auxiliary modality, respectively, denoted as F′ Pri ∈R 2×H×W×D and F′ Aux ∈R 2 ×H×W×D .

[0018] Optionally, the method further includes: upsampling the features of the main modality and the auxiliary modality to the same size, connecting them in the channel dimension, and fusing the connected features to generate an initial weight map of the main modality and the auxiliary modality;

[0019] Normalize the initial weight map to obtain the pixel-by-pixel weight map W of the main mode Pri and the pixel-by-pixel weight map W of the auxiliary modality Aux ;

[0020] The fused feature map is expressed as:

[0021] in, represents element-by-element multiplication, W Pri ∈R C×H×W×D , W Aux ∈R C×H×W×D , and W Pri and W Aux Respectively represent the pixel-by-pixel weight maps of the main modality and the auxiliary modality; F Pri and F Aux Respectively represent the original feature map of the main modality and the original feature map of the auxiliary modality.

[0022] Optionally, it also includes: adding the fused feature map to the original feature maps of the main modality and the auxiliary modality respectively, and inputting them into the encoding paths of the corresponding modalities respectively to obtain a high-level feature map.

[0023] Optionally, the process of establishing long-range spatial dependencies between different modes includes:

[0024] Obtain N tokens from the high-level feature graph of each modality, where N represents the number of tokens of a single modality, and concatenate the tokens of each modality to form a long sequence T;

[0025] Will I V and T to interact with each other to obtain a multimodal fusion feature map, where I V is the high-level feature map of the main modality, I V ∈R C×H×W×D .

[0026] Optionally, the high-level feature map of the main modality is used as a query to interact with the long sequence T to capture the long-range dependencies between different modalities. After being processed by a multi-head self-attention mechanism, the features are further passed to the network layer for deep feature processing, and combined with residual connections to generate the multimodal fusion feature map.

[0027] Optionally, the process of acquiring the medical segmentation image includes:

[0028] The multimodal fusion feature map is gradually restored to the size of the original medical image through upsampling or deconvolution operations, and the segmented image is output in combination with the extracted features.

[0029] The present invention also provides a pancreatic image segmentation system based on feature aggregation and modality adaptation, which is used to implement the pancreatic image segmentation method based on feature aggregation and modality adaptation as described above, including an encoder, a decoder and a modality adaptation deep learning module, wherein the encoder includes a feature aggregation module, and the encoder is connected to the modality adaptation deep learning module;

[0030] The encoder is used to extract feature representations of different modalities, and the multiple feature aggregation modules at multiple levels of the encoder spatially integrate the feature representations of different modalities; the integrated feature representations are input into the modality adaptive deep learning module to establish long-range spatial dependencies between different modalities to obtain a multimodal fusion feature map; the multimodal fusion feature map is input into the decoder, and a segmented image is output.

[0031] Optionally, each layer of the encoder of the main modality is connected to each layer of the decoder of the main modality in a one-to-one correspondence, and the encoder includes a projection layer, an encoder block and a downsampling layer;

[0032] The projection layer is used to calculate the partition feature map of the input modality;

[0033] The encoder block includes a plurality of serially connected deep convolution blocks, and the encoder block is used to obtain a multi-scale representation of the input modality to obtain an original feature map;

[0034] The encoder blocks are alternately arranged with the downsampling layers;

[0035] The feature aggregation module includes a maximum pooling layer, a global average pooling layer, an additional layer, a convolutional layer and a Softmax layer;

[0036] The original feature map obtained by the encoder block is input into the maximum pooling layer and the average pooling layer for processing to obtain a global feature map;

[0037] The global feature maps of different modalities are connected and enter the additional layer and the convolution layer to generate initial weight maps of different modalities;

[0038] The initial weight map is input into the Softmax layer to obtain a pixel-by-pixel weight map.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The pancreatic image segmentation method and system based on feature aggregation and modality adaptation provided by the present invention extract feature representations of different modalities through the encoder, and multiple feature aggregation modules at multiple levels of the encoder spatially integrate the feature representations of different modalities; the integrated feature representations are input into the modality adaptation deep learning module to establish long-range spatial dependencies between different modalities to obtain a multimodal fusion feature map; the multimodal fusion feature map is input into the encoder, and a medical segmentation image is output. When the method and system provided by the present invention are applied to pancreatic CT image segmentation, high-precision segmentation results equivalent to manual segmentation results can be given for unlabeled medical pancreatic enhanced CT image data, saving time for manual image outlining, and facilitating the diagnosis and treatment of pancreatic diseases; further, morphological changes caused by lesions can be better processed, thereby enhancing accurate diagnosis and treatment effects downstream; multimodal information is reasonably integrated, and global context information is effectively integrated, thereby achieving more accurate pancreatic segmentation; in addition, the pancreatic image segmentation method based on feature aggregation and modality adaptation provided by the present invention has good consistency and high accuracy in calculating pancreatic volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flow chart of a pancreatic image segmentation method based on feature aggregation and modality adaptation provided in an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the structure of a pancreatic image segmentation system based on feature aggregation and modality adaptation provided in an embodiment of the present invention;

[0043] Figure 3 for Figure 2 The structural diagram of the deep convolutional block provided in;

[0044] Figure 4 A schematic diagram of the structure of a feature aggregation module provided in an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of the structure of a modality adaptive deep learning module provided in an embodiment of the present invention;

[0046] Figure 6 It is a three-phase abdominal enhanced CT image provided in an embodiment of the present invention;

[0047] Figure 7 A visual comparison of the pancreas segmentation methods provided in the embodiments of the present invention Figure 1 ;

[0048] Figure 8 A visual comparison of the pancreas segmentation methods provided in the embodiments of the present invention Figure 2 ;

[0049] Fig. 9This is a Bland-Altman analysis graph of the pancreatic volume provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] A pancreatic image segmentation method and system based on feature aggregation and modality adaptation of the present invention will be described in more detail below in conjunction with a schematic diagram, wherein a preferred embodiment of the present invention is shown, and it should be understood that those skilled in the art can modify the present invention described herein, while still achieving the advantageous effects of the present invention. Therefore, the following description should be understood as being widely known to those skilled in the art, and not as a limitation of the present invention.

[0051] For the sake of clarity, not all features of the actual embodiments are described. In the following description, well-known functions and structures are not described in detail because they would clutter the invention with unnecessary detail. It should be recognized that in the development of any actual embodiment, a large number of implementation details must be made to achieve the developer's specific goals, such as changing from one embodiment to another according to the limitations of the relevant system or the relevant business. In addition, it should be recognized that such development work may be complex and time-consuming, but it is just a routine task for those skilled in the art.

[0052] The present invention is described in more detail in the following paragraphs by way of example with reference to the accompanying drawings. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in illustrating the purpose of the embodiments of the present invention.

[0053] Embodiment 1

[0054] The present invention provides a pancreatic image segmentation method based on feature aggregation and modality adaptation. Figure 1 , including the steps of:

[0055] S1. Taking multi-phase medical images as input, including main modality and auxiliary modality, wherein pancreatic enhanced CT venous phase image is the main modality, and pancreatic enhanced CT plain scan phase image and pancreatic enhanced CT arterial phase image are auxiliary modalities;

[0056] S2, extracting features of the multiple medical images respectively, and integrating feature representations of the main modality and the auxiliary modality;

[0057] S3, establishing long-range spatial dependencies between different modalities;

[0058] S4. Obtain a medical segmentation image.

[0059] In step S2, the steps of respectively extracting features of the multi-phase medical images include: using multiple encoding paths, respectively calculating partition feature maps of the input modalities, and using the encoding paths to respectively obtain multi-scale representations of the input modalities to obtain original feature maps.

[0060] Furthermore, in step S2, the step of integrating the feature representation of the main modality and the auxiliary modality includes:

[0061] The feature representations of the primary modality and the auxiliary modality are spatially integrated at multiple levels of the encoding path; the original feature maps of the primary modality and the auxiliary modality are used as input feature maps, each pixel is weighted, and the information of the primary modality and the auxiliary modality is spatially fused.

[0062] Specifically, the step of integrating the feature representation of the main modality and the auxiliary modality specifically includes:

[0063] S21, performing maximum pooling and average pooling on the channel dimension on the input feature map to obtain global feature maps of the main modality and the auxiliary modality, respectively, denoted as F′ Pri ∈R 2×H×W×D and F′ Aux ∈R 2×H×W×D .

[0064] S22. Upsample the features of the main modality and the auxiliary modality to the same size, connect them in the channel dimension, and fuse the connected features to generate an initial weight map of the main modality and the auxiliary modality.

[0065] S23, normalizing the initial weight map to obtain a pixel-by-pixel weight map W of the main modality Pri and the pixel-by-pixel weight map W of the auxiliary modality Aux .

[0066] S24, the fused feature map is expressed as:

[0067] in, represents element-by-element multiplication, W Pri ∈R C×H×W×D , W Aux ∈R C×H×W×D , and W Pri and W Aux Respectively represent the pixel-by-pixel weight maps of the main modality and the auxiliary modality; F Pri and F Aux Respectively represent the original feature map of the main modality and the original feature map of the auxiliary modality.

[0068] S25. Add the fused feature map to the original feature maps of the main modality and the auxiliary modality respectively, and input them into the encoding paths of the corresponding modalities respectively to obtain a high-level feature map.

[0069] In this embodiment, the output high-level feature map includes features of a specific modality and fusion features, thereby achieving optimal utilization of multimodal information.

[0070] In step S3, the process of establishing long-range spatial dependencies between different modalities includes:

[0071] N tokens are obtained from the high-level feature map of each modality, where N represents the number of tokens of a single modality, and the tokens of each modality are concatenated to form a long sequence T.

[0072] Will I V and T to interact with each other to obtain a multimodal fusion feature map, where I V is the high-level feature map of the main modality, I V ∈R C×H×W×D .

[0073] Specifically, the high-level feature map of the main modality is used as a query to interact with the long sequence T to capture the long-range dependencies between different modalities. After being processed by the multi-head self-attention mechanism, the features are further passed to the network layer for deep feature processing, and combined with the residual connection to generate the multimodal fusion feature map.

[0074] The output multimodal fusion feature map contains information fused from all modalities and enhances the global feature expression capability.

[0075] Furthermore, in step S4, the process of obtaining the medical segmentation image specifically includes: gradually restoring the multimodal fusion feature map to the size of the original medical image through upsampling or deconvolution operations, and combining the extracted features to output the segmented image.

[0076] In this embodiment, a pancreatic image segmentation method based on feature aggregation and modality adaptation is provided, which uses the abdominal enhanced CT venous phase image as the main modality and the plain scan phase and arterial phase as auxiliary modalities, integrates the spatial information from the main modality and the auxiliary modality, enhances the feature expression of the image and establishes a long-range dependency relationship between the modalities, and effectively improves the automatic segmentation accuracy of the abdominal enhanced CT image; in addition, it fully utilizes the information of images of different modalities to improve the accuracy and reliability of medical image segmentation, optimizes the utilization of image information of different modalities, reduces the limitation of a single modality, enhances the understanding of image context information, improves boundary positioning, and ultimately achieves high-precision and high-detail medical image segmentation results.

[0077] Embodiment 2

[0078] The present invention provides a pancreatic image segmentation system based on feature aggregation and modality adaptation, which is used to implement the pancreatic image segmentation method based on feature aggregation and modality adaptation described in Example 1. Figure 2 - Figure 5 The system includes an encoder and a decoder. The encoder includes a feature aggregation module, and the encoder is connected to the modality adaptive deep learning module.

[0079] The encoder is used to extract feature representations of different modalities, and the multiple feature aggregation modules at multiple levels of the encoder spatially integrate the feature representations of different modalities; the integrated feature representations are input into the modality adaptive deep learning module to establish long-range spatial dependencies between different modalities to obtain a multimodal fusion feature map; the multimodal fusion feature map is input into the decoder, and a segmented image is output.

[0080] Wherein, each layer of the encoder of the main modality is connected to each layer of the decoder of the main modality in a one-to-one correspondence.

[0081] Furthermore, the encoder includes a projection layer, an encoder block and a downsampling layer; the projection layer is used to calculate the partition feature map of the input modality; the encoder block includes a plurality of series-connected deep convolution blocks, and the encoder block is used to obtain a multi-scale representation of the input modality to obtain the original feature map; the encoder block and the downsampling layer are alternately arranged.

[0082] Furthermore, the feature aggregation module includes a maximum pooling layer, a global average pooling layer, an additional layer, a convolutional layer and a Softmax layer.

[0083] The original feature map obtained by the encoder block is input into the maximum pooling layer and the average pooling layer for processing to obtain a global feature map.

[0084] The global feature maps of different modalities are connected and enter the additional layer and the convolution layer to generate initial weight maps of different modalities.

[0085] The initial weight map is input into the Softmax layer to obtain a pixel-by-pixel weight map.

[0086] In this embodiment, the main function of the encoder is to extract useful features from the input image. Through a series of convolution and downsampling (Pooling) operations, it gradually transforms the input image into smaller but deeper feature maps, which capture the high-level semantic information of the image. As the number of encoder layers increases, the spatial resolution of the image gradually decreases, but the number of channels (the depth of the feature map) gradually increases, and the representation of the feature map becomes more abstract.

[0087] The main function of the decoder is to gradually restore the deep features generated by the encoder to an output of the same size as the input image, while retaining high-level semantic information for generating segmentation results. The decoder gradually restores the low-resolution feature map to the size of the original image through up-sampling or deconvolution operations, and combines the features from the encoder to gradually improve the spatial details of the output.

[0088] The key relationship between the encoder and decoder is established through jump connections, which pass the feature maps of certain layers of the encoder directly to the corresponding layers of the decoder, thereby preserving the spatial details and low-level features lost during the downsampling process, helping the decoder to better recover the precise boundaries and structural details. In this embodiment, the jump connection combines the low-level details of the encoder and the high-level semantic information of the decoder, ensuring that the final output contains both high-level semantic features and the details in the original image. This is especially important for image segmentation tasks, which can improve the accuracy of segmentation boundaries.

[0089] The encoder and decoder complement each other through the process of feature extraction and restoration. The encoder converts the input image into high-level semantic features, and the decoder uses these features and jump connections to reconstruct the segmentation result of the same size as the input image.

[0090] The pancreatic image segmentation system based on feature aggregation and modality adaptation provided in this embodiment, the feature aggregation module can effectively integrate the feature representations of different modalities, improve the richness and representativeness of the features; the modality adaptation deep learning module can establish long-range spatial dependencies between different modalities, and enhance the segmentation algorithm's ability to understand multimodal data; the encoder block obtains multi-scale representations through multiple series of deep convolution blocks, which helps to capture different levels of detail in the image, thereby improving the accuracy of segmentation. The use of maximum pooling and global average pooling layers can extract global feature maps, while the pixel-by-pixel weight map retains local information. This combination helps to consider both global structure and local details during segmentation. The pixel-by-pixel weight map generated by the Softmax layer can dynamically adjust the importance of different modality features, improving the flexibility and efficiency of feature fusion. The system takes into account the complexity of multimodal data and improves the accuracy and robustness of medical image segmentation.

[0091] Embodiment 3

[0092] This embodiment adopts the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in Embodiment 1 and the pancreatic image segmentation system based on feature aggregation and modality adaptation provided in Embodiment 2, and specifically describes the process applied to pancreatic CT image segmentation.

[0093] In this embodiment, a main modality encoder and two auxiliary modality encoders are used to process the main modality image and the auxiliary modality image respectively. Feature Aggregation Module (FAM) is provided at multiple levels of the main modality encoder and the auxiliary modality encoder, and a Modality Adaptive Transformer (MAT), i.e., a modality adaptive module, is provided after the main modality encoder and the auxiliary modality encoder.

[0094] Specifically, each encoder uses depth-wise convolution blocks (DCBs) with large convolution kernels to extract hierarchical feature representations of different input modalities to capture a wide receptive field and global context features.

[0095] To process multimodal information, two key components are provided in this embodiment: Feature Aggregation Module (FAM) and Modality Adaptive Transformer (MAT). FAM spatially integrates the information of the primary and auxiliary modalities, enhances the effective information and provides a more comprehensive understanding of the pancreatic structure. It extracts local and global features of different modalities and logically fuses these features together through learned weight maps. MAT promotes long-range spatial dependencies between different modalities to enhance multimodal feature interactions. The MAT module uses the TokenLearner strategy to adaptively extract fine tokens, and then uses the transformer structure to guide the network to learn and integrate feature representations from different modalities.

[0096] In this embodiment, the encoder includes a projection layer, an encoder block, and a downsampling layer, and each encoder block consists of two deep convolution blocks DCB connected in series, because the deep convolution block DCB can effectively extract features of a large receptive field. At the beginning of the encoder, a projection layer with a kernel size of 7×7×7 is used to calculate the partition feature map, and then DCB is used to obtain the multi-scale representation of each modality. Specifically, DCB first extracts features using depth-wise convolution with a kernel size of 7×7×7. Subsequently, similar to a multi-layer perceptron (MLP), a DCS with a kernel size of 1×1×1 is used to expand the features of each channel to a wider hidden dimension (4 input channels), and then scaled back, enriching the feature representation while improving computational efficiency.

[0097] Therefore, the output Z of DCB is defined as follows:

[0098] Y = DWC (LN (X)) + X;

[0099] Z = DCS (LN (Y)) + Y;

[0100] Among them, LN represents layer normalization, DWC represents depth-wise convolution, and DCS represents depth-wise convolution scaling.

[0101] In addition, in order to capture broader contextual information, a downsampling layer with a kernel size of 2×2×2 and a stride of 2 is used after each encoder block. The downsampling layer halves the spatial resolution of the feature map output by the encoder block, allowing the system to focus more on general features.

[0102] Although the primary modality image can show good contrast between the target tissue and its surroundings, the auxiliary modality image can still provide auxiliary information and provide supplementary value for target tissue segmentation. In order to more fully understand the target tissue structure, FAM is also provided in this embodiment to integrate the features of the primary modality and the auxiliary modality.

[0103] Specifically, Figure 4 The detailed structure of FAM is presented, which spatially fuses the information of two modalities by reweighting each pixel.

[0104] Specifically, given the input feature map from the main modality and the input feature map from one of the auxiliary modalities: F Pri ∈R C×H×W×D and F Aux ∈R C×H×W×D , fused, the fused feature map F Fuse ∈R C×H×W×D It can be expressed as:

[0105] in, represents element-by-element multiplication, W Pri ∈R C×H×W×D and W Aux ∈R C×H×W×D Represent the pixel-by-pixel weight maps of the primary and auxiliary modalities, respectively.

[0106] Specifically, in order to calculate the pixel-by-pixel weight maps of the primary modality and the auxiliary modality, the input primary modality feature map and the auxiliary modality feature map are subjected to maximum pooling and average pooling in the channel dimension, respectively, to extract their modality-specific global information, denoted as F′ Pri ∈R 2×H×W×D (main modality global information) and F′ Aux ∈R 2×H×W×D (Auxiliary modal global information).

[0107] The modality feature maps after max pooling and average pooling are further concatenated and passed through additional layers for better feature interaction.

[0108] like Figure 4 As shown, the additional layer includes a global average pooling (GAP) layer to extract the global relationship of the input feature map, and the additional layer also includes two convolutional layers. The kernel sizes of the two convolutional layers are 3×3×3 and 5×5×5 respectively to capture the local details of the input feature map.

[0109] In this process, the feature maps of each modality are first upsampled to the same size and then concatenated in the channel dimension. These concatenated feature maps are fused through a convolutional layer with a kernel size of 3×3×3 to generate the initial weight maps of the two modalities. Subsequently, a Softmax layer is applied to normalize the initial weight maps to obtain the pixel-by-pixel weight map W of the main modality. Pri and the pixel-wise weight map W of the auxiliary modality Aux , ensure that W Pri +W Aux =1.

[0110] Therefore, the fusion feature map F Fuse ∈R C×H×W×D You can use the formula: The fused feature map is calculated and input into the encoder stream of the corresponding modality by adding it to the original feature maps of the main modality and the corresponding auxiliary modality.

[0111] In summary, in this way, the modal features processed in each encoder contain the features of the specific modality and the fusion features, achieving the best utilization of multimodal information.

[0112] In order to establish long-range spatial dependencies between multimodalities and enhance multimodal feature interactions, a MAT module is provided in the system provided in this embodiment. After the high-level feature maps from different encoders are input into the MAT module, the MAT module uses the TokenLearner strategy to obtain fine tokens instead of the traditional method of flattening the image.

[0113] like Figure 5 As shown in the figure, the convolutional layer is first used to calculate the spatial attention map to highlight the important areas. These important areas are used to weight the input and discard irrelevant areas. Finally, the weighted input is spatially pooled to generate the final learned tokens. TokenLearner can adaptively extract fine tokens based on the input while reducing the computational cost. Subsequently, the transformer (deep learning model) structure is used to guide the network to learn and integrate feature representations from different modalities.

[0114] Specifically, given a high-level feature map from a main modality and two auxiliary modalities, the high-level feature map of the main modality is denoted as I V , the MAT module obtains N tokens from each modality through the TokenLearner operation, where N (N=32) represents the number of fine tokens of a single modality, and concatenates these tokens to form a group of 3N tokens, denoted as T.

[0115] I V (Flattened into tokens and denoted as I′ V ) and T will be input into the subsequent transformer (deep learning model) structure for inter-modal information interaction.

[0116] It is worth noting that I′, which effectively integrates all modal information V As a query (Query). The output after MAT processing can be expressed as: O = MLP (LN (ε)) + ε, ε = MSA (I ' V ,T)+I′ V ;

[0117] Among them, MLP stands for Multi-Layer Perceptron and MSA stands for Multi-head Self Attention.

[0118] In summary, through the above methods, the MAT module can improve the overall feature representation and establish long-range spatial dependencies, thereby achieving more accurate pancreas segmentation.

[0119] Specifically, taking pancreas segmentation as an example, the specific implementation process of using the pancreas image segmentation method and system based on feature aggregation and modality adaptation provided in Embodiment 1 and Embodiment 2 to segment the pancreas is as follows:

[0120] First, a pancreatic enhanced CT dataset was collected from 264 participants, recorded as an internal dataset. According to the diagnostic report, 96 of them had normal pancreas and the remaining 168 were patients with chronic pancreatitis (CP) with varying degrees of pancreatic exocrine insufficiency (PEI). For all participants, each case contained pancreatic enhanced CT images in three phases: plain phase, venous phase, and arterial phase. The spatial resolution of all images was 1×1 mm. 2, with a size of 512×512 pixels, the number of slices ranging from 144 to 432, and a slice thickness of 1 mm. Following the commonly used data partitioning guidelines, the dataset was randomly divided into 186 cases for training, 34 cases for validation, and 44 cases for testing. All pancreatic regions in the dataset were carefully annotated by two junior radiologists, and these annotations were subsequently reviewed and corrected by other radiologists to ensure the accuracy of the segmentation.

[0121] In addition, an external dataset was collected to verify the downstream tasks of pancreas segmentation, such as computer-aided diagnosis of PEI in this embodiment. The external dataset includes 17 normal pancreases and 117 pancreases of CP patients with varying degrees of PEI. Similar to the internal dataset, each case in the external dataset also contains three-phase pancreatic enhanced CT images of the plain phase, venous phase, and arterial phase. Please refer to Figure 6 As shown, the spatial resolution of the image is 1×1 mm 2 , the size is 512×512 pixels, the number of slices ranges from 135 to 338, and the slice thickness is 1 mm.

[0122] Specifically, in order to reduce the impact of the patient's limb movement during the scan, the ANTs algorithm was used to register the arterial phase and plain scan phase images to the venous phase images. The image intensity was bounded by [-100, 240] Hounsfield units (HU) and further normalized to the range of [0, 1].

[0123] To avoid overfitting during the training phase, this example uses a series of data augmentation strategies, including random rotation (random angle range: -6° to 6°), intensity offset (offset value: 0.1), and scaling (scaling factor: 0.1). It should be noted that due to the GPU memory limitations of the computing device, large-scale 3D patches of size 96×96×96 are randomly cropped during the training process.

[0124] All experiments were implemented on the PyTorch platform. The training process was optimized using the AdamW optimizer, with a batch size of 2 and a total of 200 epochs. The initial learning rate was set to 0.0001 and halved every 50 training epochs. The best checkpoint model was selected on the validation set for testing.

[0125] The following are evaluation indicators for quantitatively evaluating the performance of the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment for pancreatic CT image segmentation:

[0126] In order to quantitatively evaluate the performance of the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment for pancreatic CT image segmentation, the experiment uses commonly used indicators: including Dice Similarity Coefficient (DSC) and Hausdorff Distance (HD). DSC measures the degree of overlap between the predicted region and the true value, and the higher the value, the better the segmentation performance. HD measures the distance between the predicted region and the true value, and the lower the value, the closer the predicted boundary is to the true value.

[0127] For the downstream task of pancreatic volume calculation based on the segmented pancreatic region, the Mean Absolute Error (MAE) and Intraclass Correlation Coefficient (ICC) were used as evaluation metrics. MAE calculates the average of the absolute differences between the predicted volume and the true volume, providing an intuitive measure of prediction error. ICC quantifies the consistency between the predicted volume and the true volume, with values ​​close to 1 indicating high agreement with the true value. In addition, Bland-Altman analysis was performed to further evaluate the consistency between the predicted and actual volumes.

[0128] To verify the diagnostic value of the downstream PEI classification task based on the segmented pancreatic region, the accuracy (ACC) and the area under the receiver operating characteristic curve (AUC) were used as evaluation indicators. ACC measures the proportion of samples that are correctly classified among all samples, while AUC quantifies the ability of the model to distinguish between positive and negative examples. Both indicators range from 0 to 1, and higher values ​​indicate better discrimination performance.

[0129] In addition, the p-value obtained using the paired t-test was used to verify the statistical significance of the method provided in this example compared with other methods.

[0130] The following are three groups of experiments and results for quantitatively evaluating the performance of the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment for pancreatic CT image segmentation according to the above evaluation indicators:

[0131] Experiment 1: Segmentation Performance Analysis

[0132] (1) Comparison with SOTA general segmentation methods and specific pancreas segmentation methods:

[0133] In this embodiment, the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment is compared with several SOTA (State of the Art) segmentation methods, including methods designed specifically for pancreatic segmentation. Specifically, the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment is evaluated with the performance of UNETR, PanKNet, nnFormer and SwinUNETR. The qualitative comparison results are shown in Table 1. Depending on the data modality used, the above SOTA segmentation methods can be divided into two categories: 1) single modality networks, using only the main modality (venous phase); 2) multimodal networks, using both the main modality and auxiliary modalities (arterial phase and plain scan phase), which are simply spliced ​​along the channel dimension as input.

[0134] It should be noted that the above-mentioned UNETR is a Transformer-based 3D medical image segmentation model, PanKNet is a 3D fully convolutional deep network specifically used for pancreatic CT image segmentation, and nnFormer is a 3D Transformer model that combines convolution and self-attention mechanisms for medical image segmentation, called the "nnFormer volumetric medical image segmentation model". SwinUNETR is a model that combines Swin Transformer with the U-Net architecture for semantic segmentation of brain tumors in MRI images.

[0135] The results show that multimodal networks are not always better than single-modal networks. As shown in Table 1, for PanKNet (Pseudo-siamese Attention Network with Kernels) and SwinUNETR, multimodal methods outperform single-modal methods on the entire test set, which is expected because multimodal data provides more comprehensive information. However, for UNETR and nnFormer, the segmentation results based on a single modality are slightly better than those based on multimodality. This suggests that simply splicing multimodal data at the input level may limit the potential for better feature interactions between different phases, resulting in performance degradation. Therefore, designing an effective multimodal feature fusion module may be crucial for pancreatic segmentation. The pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment outperforms other methods in both DSC and HD on the entire test set. Compared with UNETR, PanKNet, nnFormer and SwinUNETR, the method provided in this embodiment improves DSC by 6.70%, 6.02%, 4.17% and 1.79%, respectively. The DCB used in the embodiment of the present invention can effectively capture contextual features, while the FAM and MAT modules can better utilize multimodal information. More importantly, the method provided in the embodiment of the present invention also performs well in processing PEI pancreas, which usually presents with pancreatic parenchymal atrophy and stone conditions, with a DSC of 88.73% and an HD of 0.28 mm, which is better than the normal pancreas (DSC: 86.87%, HD: 0.40 mm). This shows that the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment can better handle the morphological changes caused by lesions, and may enhance the accurate diagnosis and treatment effects downstream.

[0136] It should be noted that the above-mentioned PanKNet is a network for pancreatic CT image segmentation, and SwinUNETR is a model that combines the Swin Transformer and U-Net architectures, which is specifically used for semantic segmentation of brain tumors in MRI images.

[0137] Table 1 Quantitative comparison with SOTA general segmentation methods and specific pancreas segmentation methods

[0138]

[0139] It should be noted that in Table 1, ± means (mean ± standard deviation), and * means p value < 0.05.

[0140] Figure 7 This is a visual comparison diagram of the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment, the SOTA general segmentation method, and the specific pancreatic image segmentation method. It should be noted that: Figure 7 In the table, each multimodal method is marked with a + sign after its name to mark and distinguish it.

[0141] Figure 7 Each row in corresponds to a test case in the venous phase. The green outline represents the true annotation, and the red outline represents the predicted results of each method. For cases 1, 3, 4, and 6, the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment can more accurately retain the boundary between the pancreas and its surrounding tissues, reducing the possibility of under-segmentation. For cases 2, 4, and 5, the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment can better distinguish the pancreas from surrounding tissues with similar intensity, thereby reducing over-segmentation. In addition, for abnormal pancreatic cases with PEI (cases 2, 4, 5, and 6), the method provided in this embodiment can better segment the pancreas and internal lesion areas as a whole volume, without producing ambiguous results like other methods. All these comparison results are consistent with the quantitative results, indicating the superiority of the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment.

[0142] (2) Comparison with other multimodal segmentation methods:

[0143] Figure 7 Methods with a + sign after their names are multimodal methods, which mainly focus on integrating multimodal images at the input level, that is, simply concatenating images as multi-channel inputs. In contrast, this experiment explores segmentation methods specifically designed for multimodal segmentation, which study the effectiveness of various fusion strategies at the feature level.

[0144] The method proposed in the embodiment of the present invention is compared with a variety of conventional multimodal segmentation methods, including HPN, mmFormer, NestedFormer and TransBTS. Table 2 shows the comparison results.

[0145] Table 2: Quantitative comparison of the proposed method with other multimodal segmentation methods

[0146]

[0147] It should be noted that the ± in Table 2 represents (mean ± standard deviation), and * represents p value < 0.05.

[0148] It can be seen from Table 2 that the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment performs best in terms of DSC and HD indicators, especially for PEI cases. The method provided in this embodiment improves the DSC indicator by 5.34%, 3.14%, 2.83% and 2.82% respectively compared with other multimodal segmentation methods. Figure 8 A visual comparison of different multimodal segmentation methods is further provided in . Each row corresponds to a test case in the venous phase. Figure 8 In the figure, Case 1 and Case 2 exhibit PEI, while Case 3 represents a normal pancreas. The comparison shows that the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment outperforms other conventional multimodal segmentation methods in all three cases and can more accurately capture the boundaries of pancreatic images. Comparative multimodal segmentation methods such as HPN and mmFormer lack consideration of pancreatic-specific prior knowledge and assume that each modality contributes equally in segmentation, so they perform poorly when the pancreatic boundary is blurred with other tissues. In contrast, the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment mainly focuses on venous phase images, can reasonably fuse multimodal information, and effectively integrate global contextual information through the MAT module, thereby achieving more accurate pancreatic image segmentation.

[0149] Experiment 2: Ablation study

[0150] Furthermore, ablation studies are conducted to verify the effectiveness of the proposed module.

[0151] Specifically, the effectiveness of three different encoder backbone networks is compared, and the effectiveness of the proposed fusion module is verified. As shown in Table 3, the encoder part compares three different encoder backbones: UNB, TRE and DCB. Among them, UNB represents the use of U-Net's encoder block for feature extraction, TRE represents the use of transformer encoder, and DCB represents the use of deep convolutional blocks; they all perform feature fusion by splicing. The results show that the DCB used in the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment can more effectively capture the multi-scale and contextual information in the image and achieve better segmentation performance, with a DSC value of 7.06% higher than UNB and 3.89% higher than TRE on the entire test set. The fusion part in Table 3 verifies the effectiveness of the FAM and MAT modules. In this embodiment, the FAM module effectively fuses multimodal features by learning the spatial correlation between modalities, which improves the DSC of the entire test set by 1.28%. The MAT module enables the system to adaptively adjust and focus on features at different phases, further improving the DSC of the entire test set by 1.53%, with significant improvements especially observed in the PEI case.

[0152] Table 3: Comparison of ablation studies

[0153]

[0154] It should be noted that the ± in Table 3 represents (mean ± standard deviation), and * represents p value < 0.05.

[0155] Experiment 3: Calculation of pancreatic volume

[0156] Calculating pancreatic volume is crucial for detecting abnormal volume changes and monitoring pathological growth. Pancreatic volume provides quantitative information about pancreatic size, which can be used to monitor the progression of pancreatic diseases and guide clinical decision-making. This experiment aims to better understand the potential clinical impact of the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment in calculating pancreatic volume, which is usually manually annotated by doctors in clinical practice.

[0157] This experiment calculates the MAE and ICC between the true volume based on the manually segmented pancreas mask and the predicted volume based on the automatic segmentation method. The results are summarized in Table 4. Compared with other methods, the pancreas image segmentation method based on feature aggregation and modality adaptation provided in this embodiment achieves the smallest MAE (10.5 cm 3 ) and the highest ICC (0.955), indicating that the pancreatic volume error calculated by the pancreatic image segmentation method based on feature aggregation and modality adaptation provided in this embodiment is small and highly consistent with the true value.

[0158] In order to more intuitively compare the consistency and deviation between the pancreatic volume obtained by the automatic segmentation method and the true volume, a Bland-Altman analysis was further performed. The analysis results can be found in Fig. 9 The Bland-Altman analysis results compare the true values ​​of different methods. The horizontal axis represents the average of the measured value and the true value, and the vertical axis represents the difference between the two. The deviation line represents the average difference, and the 95% agreement limits show the upper and lower limits of the difference. Fig. 9 As can be seen from the provided Bland-Altman graph, the data points of the method provided in this embodiment show small differences, low deviations, few outliers, and more points gathered in the area near the 0 line. In contrast, other methods show larger deviations and wider distribution of data points, indicating poor consistency and possible low accuracy in calculating pancreatic volume.

[0159] Table 4: MAE and ICC comparison results between the true volume obtained by manually segmenting the pancreas mask and the predicted volume based on the automatic segmentation method

[0160]

[0161] In summary, the pancreatic image segmentation method and system based on feature aggregation and modality adaptation provided by the embodiments of the present invention, when applied to pancreatic CT image segmentation, can provide high-precision segmentation results equivalent to manual segmentation results for unlabeled medical pancreatic enhanced CT image data, saving doctors' time in manually outlining images, and facilitating the diagnosis and treatment of pancreatic diseases; can better handle the morphological changes caused by lesions, thereby enhancing the accurate diagnosis and treatment effects downstream; can reasonably fuse multimodal information, and effectively integrate global context information through MAT, thereby achieving more accurate pancreatic segmentation; and has good consistency and high accuracy in calculating pancreatic volume.

[0162] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A pancreatic image segmentation method based on feature aggregation and modality adaptation, characterized in that: include: Taking multi-phase medical images as input, including a main modality and an auxiliary modality, wherein the pancreatic enhanced CT venous phase image is the main modality, and the pancreatic enhanced CT plain scan phase image and the pancreatic enhanced CT arterial phase image are the auxiliary modalities; Extracting features of the multiple medical images respectively, and integrating feature representations of the main modality and the auxiliary modality; Establishing long-range spatial dependencies between different modalities; Get medical segmentation images.

2. The pancreatic image segmentation method based on feature aggregation and modality adaptation according to claim 1, characterized in that: The step of respectively extracting the features of the multi-phase medical images comprises: using multiple encoding paths to respectively calculate the partition feature maps of the input modalities, and using the encoding paths to respectively obtain the multi-scale representation of the input modalities to obtain the original feature maps; The step of integrating the feature representation of the main modality and the auxiliary modality comprises: spatially integrating feature representations of the primary modality and the auxiliary modality at multiple levels of the encoding pathway; The original feature maps of the main modality and the auxiliary modality are used as input feature maps, each pixel is weighted, and the information of the main modality and the auxiliary modality are spatially fused.

3. The pancreatic image segmentation method based on feature aggregation and modality adaptation according to claim 2, characterized in that: Also includes: Perform maximum pooling and average pooling on the channel dimension on the input feature map to obtain the global feature maps of the main modality and the auxiliary modality, respectively, denoted as F′ Pri ∈R 2×H×W×D and F′ Aux ∈R 2×H×W×D .

4. The pancreatic image segmentation method based on feature aggregation and modality adaptation according to claim 3, characterized in that: Also includes: Upsampling the features of the main modality and the auxiliary modality to the same size, connecting them in the channel dimension, and fusing the connected features to generate an initial weight map of the main modality and the auxiliary modality; Normalize the initial weight map to obtain the pixel-by-pixel weight map W of the main mode Pri and the pixel-by-pixel weight map W of the auxiliary modality Aux ; The fused feature map is expressed as: in, represents element-by-element multiplication, W Pri ∈R C×H×W×D , W Aux ∈R C×H×W×D , and W Pri and W Aux Respectively represent the pixel-by-pixel weight maps of the main modality and the auxiliary modality; F Pri and F Aux Respectively represent the original feature map of the main modality and the original feature map of the auxiliary modality.

5. The pancreatic image segmentation method based on feature aggregation and modality adaptation according to claim 4, characterized in that: Also includes: The fused feature map is added to the original feature maps of the main modality and the auxiliary modality respectively, and is input into the encoding path of the corresponding modality respectively to obtain a high-level feature map.

6. The pancreatic image segmentation method based on feature aggregation and modality adaptation according to claim 5, characterized in that: The process of establishing long-range spatial dependencies between different modalities includes: Obtain N tokens from the high-level feature graph of each modality, where N represents the number of tokens of a single modality, and concatenate the tokens of each modality to form a long sequence T; Will I V and T to interact with each other to obtain a multimodal fusion feature map, where I V is the high-level feature map of the main modality, I V ∈R C×H×W×D .

7. The pancreatic image segmentation method based on feature aggregation and modality adaptation according to claim 6, characterized in that: The high-level feature map of the main modality is used as a query to interact with the long sequence T to capture the long-range dependencies between different modalities. After being processed by the multi-head self-attention mechanism, the features are further passed to the network layer for deep feature processing, and combined with the residual connection to generate the multimodal fusion feature map.

8. The pancreatic image segmentation method based on feature aggregation and modality adaptation according to claim 7, characterized in that: The process of obtaining the medical segmentation image comprises: The multimodal fusion feature map is gradually restored to the size of the original medical image through upsampling or deconvolution operations, and the segmented image is output in combination with the extracted features.

9. A pancreatic image segmentation system based on feature aggregation and modality adaptation, used to implement the pancreatic image segmentation method based on feature aggregation and modality adaptation as described in any one of claims 1 to 8, characterized in that: It includes an encoder, a decoder and a modality-adaptive deep learning module, wherein the encoder includes a feature aggregation module, and the encoder is connected to the modality-adaptive deep learning module; The encoder is used to extract feature representations of different modalities, and the multiple feature aggregation modules at multiple levels of the encoder spatially integrate the feature representations of different modalities; the integrated feature representations are input into the modality adaptive deep learning module to establish long-range spatial dependencies between different modalities to obtain a multimodal fusion feature map; the multimodal fusion feature map is input into the decoder, and a segmented image is output.

10. The pancreatic image segmentation system based on feature aggregation and modality adaptation according to claim 9, characterized in that: Each layer of the encoder of the main modality is connected to each layer of the decoder of the main modality in a one-to-one correspondence, and the encoder includes a projection layer, an encoder block and a downsampling layer; The projection layer is used to calculate the partition feature map of the input modality; The encoder block includes a plurality of serially connected deep convolution blocks, and the encoder block is used to obtain a multi-scale representation of the input modality to obtain an original feature map; The encoder blocks are alternately arranged with the downsampling layers; The feature aggregation module includes a maximum pooling layer, a global average pooling layer, an additional layer, a convolutional layer and a Softmax layer; The original feature map obtained by the encoder block is input into the maximum pooling layer and the average pooling layer for processing to obtain a global feature map; The global feature maps of different modalities are connected and enter the additional layer and the convolution layer to generate initial weight maps of different modalities; The initial weight map is input into the Softmax layer to obtain a pixel-by-pixel weight map.

Images