Medical image instance segmentation method based on cross-scale interactive fusion Transformer model

Abstract

The application discloses a medical image instance segmentation method based on a cross-scale interactive fusion Transformer model, and belongs to the technical field of image instance segmentation processing. The method comprises the following steps: inputting a pretreated PET / CT image into a backbone network; the backbone network comprises a plurality of basic bottleneck residual modules and a multi-head self-attention module connected in sequence; the basic bottleneck residual module is used for acquiring a local feature map; and the multi-head self-attention module is used for acquiring a non-local feature map; a cross-scale interactive feature enhancement module is used for aggregating the local feature map output by the adjacent basic bottleneck residual module, so as to acquire an enhanced feature map; a cross-scale interactive fusion FPN network is used for performing feature fusion on the enhanced feature map, so as to acquire a fusion feature map; a region establishment network is used for extracting a candidate frame for the fusion feature map; and ROIAlign is used for predicting the extracted candidate frame, so as to acquire a PET / CT image instance segmentation result. The above method solves the problem of insufficient feature extraction of different scales.

Description

Technical Field

[0001] This invention relates to the field of image instance segmentation technology, and more specifically to a medical image instance segmentation method based on a cross-scale interactive fusion Transformer model. Background Technology

[0002] Image segmentation divides a medical image into regions of interest (ROIs) with specific properties and unique meanings based on certain similarity features. Semantic segmentation predicts the category of each pixel in an input medical image. Unlike the former two, instance segmentation combines pixel-level semantic segmentation with object detection to perform instance-level segmentation and object-level recognition of organs or lesions within the detection box, achieving good segmentation accuracy and efficiency. However, traditional segmentation methods can only handle relatively simple scenarios. Due to the complexity of medical image features, blurred edges between different tissues or between tissues and lesions, irregular shapes of lesion regions, and data noise caused by irregular beatings of organs such as the heart, traditional methods are prone to missed and false detections, resulting in unsatisfactory segmentation results.

[0003] In recent years, deep learning methods based on convolutional neural networks have been widely used in the field of computer vision, especially in medical image processing such as CT images, X-ray images, ultrasound images, PET images and MRI images, where they have shown good performance and can achieve accurate localization and precise segmentation of lesion areas.

[0004] Currently, instance segmentation models such as Mask R-CNN, Cascade R-CNN, YOLACT, and SOLO have demonstrated that convolutional neural networks can not only achieve good results at the pixel level but also achieve instance-level learning, effectively providing object-specific labeled tumor lesion information. In 2017, He et al. proposed the Mask R-CNN model, which added a mask prediction branch and used ROI Align instead of ROI Pooling on the basis of Faster R-CNN, effectively solving the two-stage instance segmentation problem. In 2019, Huang et al. proposed the Mask Scoring R-CNN model, which used the MaskIoU head for mask scoring on the basis of Mask R-CNN, effectively improving the accuracy of instance localization or mask segmentation. Also in 2019, to address the issue that the Mask R-CNN model could not accurately segment the details and edges of the target object, H. Zhou et al. proposed the Corner Mask-R-CNN model, which added a corner prediction head to fit the edge parts, effectively obtaining instance detail information.

[0005] Although many researchers have conducted extensive research on instance segmentation, the methods mentioned above still suffer from problems in practical applications, such as insufficient ability to express lesion features in medical images and inadequate extraction of lesion features at different scales.

[0006] Therefore, how to fully extract lesion features from medical images is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a medical image instance segmentation method based on a cross-scale interactive fusion Transformer model, which is used to fully extract lesion features in medical images.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A medical image instance segmentation method based on a cross-scale interactive fusion Transformer model includes the following steps:

[0010] Feature extraction steps:

[0011] The preprocessed PET / CT image is input into the backbone network, which includes multiple basic bottleneck residual modules and a multi-head self-attention module connected in sequence. The basic bottleneck residual modules are used to acquire local feature maps, and the multi-head self-attention module is used to acquire non-local feature maps.

[0012] The cross-scale interactive feature enhancement module is used to aggregate local and non-local feature maps to obtain enhanced feature maps;

[0013] Feature fusion steps:

[0014] A cross-scale interactive fusion FPN network is used to perform feature fusion on the enhanced feature map to obtain a fused feature map.

[0015] Feature prediction steps:

[0016] For the fused feature map, a region-based network is used to extract candidate bounding boxes;

[0017] Based on ROI Align, the extracted candidate boxes are predicted to obtain the instance segmentation results of PET / CT images.

[0018] Preferably, before the feature extraction step, the following steps are also included:

[0019] Acquire PET / CT images of lung tumors and preprocess the PET / CT images.

[0020] Preferably, the backbone network includes four basic bottleneck residual modules and one multi-head self-attention module connected in sequence;

[0021] The basic bottleneck residual module includes a residual module and multiple bottleneck blocks stacked in series with the residual module.

[0022] Preferably, the multi-head self-attention module is used to acquire non-local feature maps, specifically including the following steps:

[0023] Input the lesion feature map into the multi-head self-attention module;

[0024] The parameter vector Rh represents the relative positional encoding of the lesion feature map height, and the parameter vector Rw represents the relative positional encoding of the lesion feature map width. The dimension of the parameter vectors Rh and Rw is d.

[0025] Add the parameter vectors Rh and Rw, then multiply them by the query matrix query to obtain the multiplication result a;

[0026] Calculate the product of the query matrix (query) and the key matrix (key), resulting in the value b.

[0027] The multiplication result a and the multiplication result b are added element-wise, and the result is processed by the softmax function.

[0028] The result of the softmax function is multiplied by the value matrix to obtain the nonlocal feature map.

[0029] Preferably, the local feature maps output by adjacent basic bottleneck residual modules are aggregated using a cross-scale interactive feature enhancement module to obtain enhanced feature maps, specifically including the following steps:

[0030] Use the current feature map x i With the adjacent upper layer feature map x i-1 Perform a 1x1 convolution operation and adjust the feature map x from the previous layer. i-1 The channel and size of the feature map, which includes local and non-local feature maps; the adjusted upper-layer feature map is compared with the current feature map x. i Perform addition and fusion to obtain the fusion result f. (i-1,i) ;

[0031] The result of the addition and fusion is f (i-1,i) and the previous layer feature map x i-1 CIFEM output results i-1 Element-wise addition and fusion are performed to obtain cross-scale input features f. (i-1,i-1) ;

[0032] Input features f across scales (i-1,i-1) The input soft attention unit processes the cross-scale input features f through channel attention and spatial attention branches, respectively.(i-1,i-1) Process it;

[0033] The results processed by the channel attention branch and the spatial attention branch are concatenated to obtain the intra-group fusion information feature map.

[0034] A random mixing layer is used to fuse cross-group feature information in the channel dimension to obtain an aggregated feature map, which is the enhanced feature map.

[0035] Preferably, the enhanced feature map is fused using a cross-scale interactive fusion FPN network, specifically including the following steps:

[0036] The low-level enhanced feature map x j-1 Enhanced feature map of adjacent high-rise buildings x j Perform element-wise addition to obtain the synthesized feature map f. (j-1,j) ;

[0037] The synthesized feature map f (j-1,j) Average pooling and max pooling are performed separately, and the results are merged to obtain the pooled features.

[0038] The pooled features are processed sequentially through a 1*1 convolutional layer, a ReLU activation layer, and another 1*1 convolutional layer, and then input into the Sigmoid activation function to obtain a weight attention map.

[0039] The acquired weighted attention map is compared with the adjacent high-level enhanced feature map x. j After multiplication, it is then compared with the adjacent high-rise enhanced feature map x. j The final fused feature map is obtained by adding them together.

[0040] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a medical image instance segmentation method based on a cross-scale interactive fusion Transformer model, which has the following advantages:

[0041] Beneficial effects:

[0042] This invention uses the ResNet-Transformer backbone network and combines global and local feature information from lung images to introduce a multi-head self-attention BoT module into the feature extraction process of lung tumors. By establishing long-distance dependencies between global pixels at the highest layer, it focuses on the lesion area from multiple dimensions to improve feature representation capabilities.

[0043] This invention utilizes a cross-scale interactive feature enhancement module (CIFEM) to interactively correlate feature maps of different scales extracted from the backbone network. By aggregating adjacent contextual information, it enhances the feature expression of the lesion region in the backbone and effectively supplements the lost detailed information. This invention also uses a cross-scale interactive fusion module (CIFM) to bidirectionally interact and fuse low-level detailed information with high-level semantic information. By aggregating feature information from adjacent stages, it further enhances the focus on tumor lesions. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1 A schematic diagram of the MCIF-Transformer Mask RCNN model structure provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the Bottleneck Transformer (BoT) module structure provided in an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the multi-head self-attention layer (MHSA) structure provided in an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of the cross-scale interactive feature enhancement module (CIFEM) structure provided in an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of the cross-scale interactive fusion module (CIFM) structure provided in an embodiment of the present invention;

[0050] Figure 6 Examples of PET / CT lung tumor images and annotations provided in embodiments of the present invention;

[0051] Figure 7 A radar chart comparison of instance segmentation results of different modules of the MCIF-Transformer Mask RCNN model provided in this embodiment of the invention;

[0052] Figure 8 This is a schematic diagram of the instance segmentation result of the MCIF-Transformer Mask RCNN model provided in an embodiment of the present invention, wherein... Figure 8 (a) and (d) represent PET / CT images; Figure 8(b)(e) represent the three-dimensional grayscale images of PET / CT images; Figure 8 (c)(f) represent the instance segmentation results. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] This invention discloses a medical image instance segmentation method based on a cross-scale interactive fusion Transformer model, comprising the following steps:

[0055] Feature extraction steps:

[0056] The preprocessed PET / CT image is input into the backbone network, which includes multiple basic bottleneck residual modules and a multi-head self-attention module connected in sequence. The basic bottleneck residual modules are used to acquire local feature maps, and the multi-head self-attention module is used to acquire non-local feature maps.

[0057] The local feature maps output by adjacent basic bottleneck residual modules are aggregated using a cross-scale interactive feature enhancement module to obtain enhanced feature maps;

[0058] Feature fusion steps:

[0059] The enhanced feature map is fused using a cross-scale interactive fusion FPN network to obtain a fused feature map.

[0060] Feature prediction steps:

[0061] For the fused feature map, a region-based network is used to extract candidate bounding boxes;

[0062] Based on ROI Align, the extracted candidate boxes are predicted to obtain the instance segmentation results of PET / CT images.

[0063] The overall structure of the cross-scale interactive fusion Transformer model MCIF-Transformer MaskRCNN (Multi-branch Cross-scale Interactive Fusion Transformer model) in this embodiment of the invention is as follows: Figure 1 As shown.

[0064] The backbone network ResNet-Transformer extracts tumor lesion feature maps at different scales from PET / CT lung images. It is decomposed into five stages. The first four stages extract features through basic residual blocks, and the fifth stage uses the multi-head self-attention BoT module to extract non-local feature information in the image.

[0065] The cross-scale interactive feature enhancement auxiliary network continuously supplements shallow information into deep features, and further enhances the fine-grained attention to lesion areas through the cross-scale interactive feature enhancement module (CIFEM);

[0066] A cross-scale interactive fusion FPN network is constructed to perform bidirectional interactive fusion of deep and shallow features. By enhancing low-level information such as details, location, and texture in deep features, more accurate and effective lesion features can be obtained.

[0067] In the Region Proposal Network (RPN), the feature map is traversed in a sliding window manner. The foreground and background are distinguished in the corresponding anchor boxes and the classification probability is calculated, as well as the coordinate point offset is regressed. The lesion category, bounding box, and mask information are regressed through three prediction branches.

[0068] For feature extraction, a master-slave network is constructed to enhance the perception of fine-grained lesion features. The ResNet-Transformer backbone network is responsible for feature extraction from preprocessed PET / CT lung tumor images, replacing the highest-level basic residual blocks with globally attenuated BoT modules to establish long-distance feature dependencies between pixels in non-local space. A cross-scale interactive feature enhancement auxiliary network is designed to aggregate adjacent feature maps of different resolutions and recalibrate the importance of spatial and channel features of lung tumors, thereby strengthening the feature representation of lesions in the backbone network. In the feature fusion part, a cross-scale interactive fusion FPN network is designed to achieve bidirectional interactive fusion of low-level feature information and high-level semantic information between features at different scales, further enhancing the model's feature representation capability.

[0069] The ResNet-Transformer backbone network uses the ResNet50 (Residual Neural Network, ResNet) residual neural network.

[14] The backbone network consists of the first four stages, which are composed of basic bottleneck residual blocks, and the Bottleneck Transformer (BoT).

[15] The fifth stage of formation consists of two parts, such as Figure 1 As shown, this network extracts non-local spatial information from the highest feature map by adding a multi-head self-attention module. By focusing on the global region, it improves the model's ability to extract lesion features from PET / CT lung images, thereby obtaining more accurate lesion features.

[0070] PET / CT lung tumor images are input into the backbone network and processed sequentially through five stages (Stage 0-Stage 4) from low to high layers. In the first four stages, four basic residual blocks extract lesion feature maps of different sizes. Within each residual block (C2-C4), 3, 4, and 6 bottleneck blocks are stacked in concatenation, respectively. In the fifth stage (Stage 4), the three bottleneck blocks in the highest layer are replaced with BoT modules, which consist of 1×1 convolutions, average pooling, multi-head self-attention (MHSA), batch normalization (BN), and ReLU activation. MHSA multi-head self-attention is used to replace the corresponding 3×3 spatial convolutions in the ResNet bottleneck blocks to learn global information from the high-level lung tumor feature maps. Figure 2 As shown in (b), the backbone network, based on the original capture of local lesion information, establishes global dependencies between pixels and learns long-distance spatial correlations between pixels in the same feature map.

[0071] The core module of BoT is Multi-Head Self-Attention SA (MHSA), and the self-attention layer structure is as follows: Figure 3 As shown, the input is a lesion feature map X of size H×W×d, where H, W, and d represent its height, width, and dimension, respectively; q, k, and r represent the query, key, and positional encoding, respectively. Two parameter vectors, Rh and Rw, represent the relative positional encodings of height and width, respectively. The two d-dimensional vectors are added together and multiplied by the query matrix. The result of this multiplication is then element-wise summed with the results of multiplying the query and key matrices. After processing with the softmax function, the sum is multiplied by the value matrix to obtain the final output. This approach learns rich lesion association features in lung images by modeling long-range semantic feature relationships.

[0072] Superficial lung tumor feature maps possess rich details of lesion contours, textures, and locations, but lack coarse-grained semantic information about the overall image; deep lung tumor feature maps possess abstract semantic features, but lack fine-grained spatial information about the lesion region. To extract richer lesion features and fully focus on subtle information, this invention designs an auxiliary network to achieve interactive fusion of cross-scale superficial feature information and deep semantic information. A cross-scale interactive feature enhancement module (CIFEM) aggregates feature maps of different resolutions from adjacent stages of the backbone network, enhancing the focus on lesion features between and within channels, and suppressing irrelevant noise other than lesions. Figure 1As shown, a master-slave network approach is used in the feature extraction part to select global and adjacent tumor features in lung images, making the model more focused on the lesion area.

[0073] The Cross-scale Interactive Feature Enhancement Network (CIFEN) consists of five stages: Stage 0 through Stage 4. The first stage involves adjusting the number of channels in C1 extracted from the backbone network using a 1×1 convolution, fusing it with the adjacent feature layer C2 via CIFEM1, and then element-wise adding it to C2 in the backbone network. The result of this first stage serves as the input to the first stage of the feature fusion network. The second stage involves adjusting the number of channels and size in C2 using a 1×1 convolution, fusing it with the adjacent feature layer C3 via cross-scale integration, combining the result with the output of CIFEM1, processing it through CIFEM2, and then element-wise adding it to C3 in the backbone network. The result of this second stage serves as the input to the second stage of the subsequent feature fusion network. Similarly, the third and fourth stages employ the same cross-scale feature interaction enhancement method as the second stage.

[0074] The Cross-scale Interactive Feature Enhancement Module (CIFEM) is a core component of the auxiliary network, such as... Figure 4 As shown, by considering adjacent features to enhance the connections between the five stages and between lesion features within each stage, an interactive fusion method is used to further refine the tumor instance segmentation task for cross-scale features, continuously enhancing the backbone network's ability to represent lesion features, thereby achieving fine-grained instance segmentation and coarse-grained semantic segmentation of lung tumors. When i-1 = 1, the input of CIFEM1 only has two neighboring feature layers C1 and C2; when i-1 ≥ 2, the input of CIFEM2 to CIFEM4 also includes the output of the previous CIFEM.

[0075] The pseudocode for this module (Algorithm 1) is represented as follows:

[0076]

[0077]

[0078] The input to the Cross-Scale Interactive Feature Enhancement Module (CIFEM) consists of the adjacent feature map from the previous layer, χ. i-1 Current feature map χ i And the previous CIFEM output CIFEM i-1 Composition, X∈R C×H×W First, let χ i-1The channels and dimensions are adjusted using a 1×1 convolution operation, then element-wise added to χi and merged, and finally combined with CIFEM. i-1 After element-wise addition and fusion, the cross-scale input features f are obtained. (i-1,i,i-1) Next, it goes through SA Unit

[16] The first step in processing is to input the feature f. (i-1,i,i-1) Divided into G groups along the channel dimension, f k Group =[f1,...,f G ],f k Group ∈R C / G×H×W Then, each set of features is segmented, i.e., along f k Group The channel direction is divided into two parts, namely the input channel attention branch and the spatial attention branch, respectively. in ,sa in ∈R C / 2G×H×W ;ca and sa represent channel attention and spatial attention, respectively; in CA, firstly, ca in Global average pooling (GAP) is used to integrate global spatial information, and then the sigmoid activation function is used to convert feature maps into probabilistic activation values ​​to recalibrate the importance of feature map channels. In SA, the first step is to... in Spatial lesion feature information was obtained by Group Normalization (GN) processing, and then Fc(·) enhancement was used to enhance sa in The feature representation; then the CA and SA outputs are concatenated according to the number of channels, f′ k =concat(CA,SA)∈R C / G×H×W This invention achieves feature fusion within a group; finally, a random mixing layer is used to fuse cross-group feature information across all aggregated sub-features in the channel dimension, yielding the final output image. This invention utilizes cross-scale proximity information to enhance the model's ability to perceive features in lesion areas. At different stages, it sequentially superimposes features to repeatedly focus on finer-scale and more discriminative fine-grained lesion features, thereby enhancing the attention given to tumor lesions in lung images.

[0079] The specific process of the cross-scale interactive feature enhancement module (CIFEM) is shown in formula (1):

[0080]

[0081] In the formula, χ i-1 and χ iCIFEM represents the adjacent previous layer feature and the current feature, respectively. i-1 This indicates the previous CIFEM output; cs(·) represents channel shuffle, chunk(·) represents chunking, reshape(·) represents grouping channels, GlobalAvgPool(·) represents global average pooling, and σ represents the sigmoid operation. Indicates multiplication. This represents element-wise addition; w1, w2 ∈ R C / 2G×1×1 b1, b2 ∈ R C / 2G×1×1

[0082] In lower-level FPN networks, the receptive field is smaller, enabling the capture of local lesion features in lung images and exhibiting strong detail perception capabilities. In higher-level FPN networks, the receptive field is larger, allowing the acquisition of overall global features of lung images and demonstrating strong semantic expression capabilities. Therefore, fusing local low-level information with global high-level information can effectively enhance the lesion region. To this end, this invention employs a bidirectional interactive fusion approach in the feature fusion part, moving from lower to higher levels and from higher to lower levels. It obtains cross-scale complementary information between adjacent feature layers through two paths: bottom-up and top-down. Figure 1 As shown, the bottom-up path effectively enhances low-level feature information in high-level feature maps, while the top-down path effectively strengthens high-level semantic information in low-level feature maps. The use of secondary fusion enhances the feature expression of tumor lesions, effectively improving the accuracy of lesion identification.

[0083] The Cross-scale Interactive Feature Fusion FPN Network (CIF-FPN) consists of four stages (Stage 1-Stage 4). Each stage inputs low-level features and neighboring high-level features into a cross-scale interactive feature fusion module (CIFM) for processing. Figure 5 As shown, interactive fusion is performed in a bottom-up manner, continuously enhancing low-level information such as lesion details, texture, and location in high-level feature maps. Then, the fused features obtained from layer-by-layer interaction are sequentially added element-by-element in a top-down manner to strengthen high-level semantic information in low-level feature maps. By aggregating feature information from each stage, the model's ability to represent and recognize lesion regions is enhanced.

[0084] The four-stage feature maps generated in the master and slave networks serve as inputs to the CIF-FPN, comprising both bottom-up and top-down approaches. The first part involves adjusting the channel count of the first-stage feature map C2 using a 1×1 convolution, and processing the second-stage feature map C3 through 1×1 convolution and deconvolution operations, then simultaneously inputting both into CIFM1 for interactive fusion. Next, the second-stage feature map C3 (after 1×1 convolution) and the third-stage feature map C4 (after 1×1 convolution and deconvolution operations) are input into CIFM2 for interactive fusion. Similarly, the processed third-stage feature map C4 and the fourth-stage feature map C5 are input into CIFM3 for interactive fusion. Here, the 1×1 convolutions adjust the channel count of the feature maps to 256. The second part first processes the output of CIFM3 in three ways: first, it is directly convolved into P5; second, it is max-pooled into P6; and third, after upsampling, it is added element-wise to the output of CIFM2. This result is then convolved into P4, upsampled, and added pixel-wise to the output of CIFM1. This result is then convolved into P3, added pixel-wise to C2, and then convolved into P2. The feature maps P2-P6 are used as inputs for the subsequent RPN.

[0085] The pseudocode for this module (Algorithm 2) is as follows:

[0086]

[0087] The input to the Cross-Scale Interactive Fusion Module (CIFM) is the feature map X from the neighboring scales. j-1 and X j First, the low-level feature map X j-1 and high-level feature map X j By adding elements one by one, the resulting characteristic f (j-1,j) After average pooling and max pooling, the weights are merged, followed by a 1×1 convolution, a ReLU activation layer, another 1×1 convolution, and a sigmoid activation function. This compresses the weights onto a value between 0 and 1, with larger coefficients indicating greater focus on the pixel's feature. Finally, the weight attention map is compared with the high-level feature map X. j The final fused feature map is obtained by multiplying and then adding the results. This invention utilizes a cross-scale interactive fusion module to further refine the representation of lesion features, highlighting tumor features through a secondary fusion method and suppressing irrelevant background noise.

[0088] The specific process of the cross-scale interactive fusion module (CIFM) is shown in formula (2):

[0089]

[0090] In the formula, AvgPool(·) represents average pooling, and MaxPool(·) represents max pooling.

[0091] Experimental Results and Analysis

[0092] Medical imaging plays a crucial role in clinical event tracking applications in diagnosis, planning, surgical procedures, and radiotherapy. PET images can reflect functional metabolic regions and are sensitive to abnormal areas. Due to the rapid metabolism of tumor lesions and their high uptake of fluoride (18F)-deoxyglucose (18F-FDG), the contrast between malignant tumors and normal tissues is significant; however, PET images suffer from low spatial resolution. CT images can reflect basic anatomical structures and have high spatial resolution, providing precise lesion location information; however, the intensity of tumors and normal soft tissues is similar, making them difficult to distinguish. PET / CT images effectively combine functional metabolic and anatomical information, fusing complementary information from both modalities to provide higher sensitivity and specificity. Therefore, the use of multimodal PET / CT images plays an important role in the early intervention and monitoring of lung cancer patients. This invention utilizes PET / CT medical images to better locate and segment tumor lesions in lung images.

[0093] Datasets and Data Preprocessing

[0094] In one specific application embodiment, the dataset used comes from original PET / CT scan images of lung cancer patients provided by the Department of Nuclear Medicine of a tertiary hospital in Ningxia, along with related clinical pathological diagnostic data. The clinical pathological diagnostic data includes textual information such as age, gender, tumor benign or malignant nature, medical orders, examination conclusions, and clinical diagnoses. The PET / CT modality dataset comprises 1052 sample images of patients aged 29–76 years from January 2018 to June 2020, of which 946 images were used as the training set and 106 images as the test set. First, the original PET / CT images in digital imaging and communication in medicine (DICOM) format are imported into the image processing software MicroDicomviewer and converted into JPG format data. Second, algorithms are used to crop the lung tumor images in both mediastinal and lung window formats to maximize the proportion of the entire lung in the image and reduce redundant background information (positions with a pixel value of 0). Then, under the guidance of a professional radiologist, Labelme software is used to complete the gold standard annotation of the lesion outline, and finally, the corresponding JSON annotation file and annotation map are generated. The annotation file includes information such as the lesion category label, annotation point coordinates, image width and height, and image path.

[0095] This invention utilizes a PyTorch-based deep learning framework to implement the proposed MCIF-TransformerMask RCNN model. The experimental environment was configured with an Intel Xeon Gold 6154 GPU, 256GB of RAM, an NVIDIA TITAN V graphics card, Python 3.7, PyTorch 1.7.0, and CUDA version 11.1.106. During network training, the training epochs were set to 300, the initial learning rate to 0.0001, and the batch size to 2. Stochastic gradient descent (SGD) was used as the optimizer to optimize the model, with a momentum parameter of 0.9 and a weight decay coefficient of 1×10⁻⁶. -4

[0096] To comprehensively and objectively evaluate the instance segmentation performance of the MCIF-Transformer Mask RCNN used in this invention, in a specific embodiment, Intersection over Union (IoU), Average Precision (AP), and Average Recall (AR) are used as evaluation criteria to quantitatively assess model performance. In the task of detecting and segmenting lung tumors, True Positives (TP) indicate that the model correctly identifies the lesion region; False Positives (FP) indicate that the model misdiagnoses normal tissue regions as lesion regions; False Negatives (FN) indicate that the model misses lesion regions as normal tissue regions; and True Negatives (TN) indicate that the model correctly identifies normal tissue regions. The definitions and formulas of the evaluation metrics are as follows:

[0097] Intersection over Union (IoU) is a metric for the accuracy of object segmentation detection in a specific dataset. It consists of two parts: the correct result of human labeling (ground truth, GT) and the prediction result of the model algorithm. IoU measures the correlation between the labeled boxes and the predicted boxes; the higher the value, the higher the correlation and the better the performance of the model. The specific calculation formula is shown in (3):

[0098] IOU=TP / (TP+FP+FN) (3)

[0099] Average precision (AP) is the percentage of correctly identified targets out of the total number of identified targets. It is used to measure the performance of a model detector on each category.

[22] Among them, AP 50 This represents the AP value when the IoU threshold is 0.5. The specific calculation formula is shown in (4):

[0100]

[0101] In the formula, c represents the detection category, which usually refers to a single category, th represents the threshold for each category, and t represents the number of detection samples.

[0102] Average Recall (AR) is the percentage of correctly identified targets out of the total number of targets identified in the test set. For the same model structure, the larger the test set, the better the AR performance.

[22] This paper calculates the AR value every 0.05 intervals within the IoU threshold range of [0.5, 0.95], and takes the average of all results as the final result. The specific calculation formula is shown in (5):

[0103]

[0104] To objectively evaluate the sophistication and effectiveness of the MCIF-Transformer Mask RCNN model, two sets of experiments were conducted on the same dataset: an ablation experiment and a comparative experiment. The first set of experiments explored the impact of each module on the model's instance segmentation performance; the second set of experiments compared the model with two-stage and single-stage instance segmentation networks. Both sets of experiments used PET / CT lung tumor images as input, and quantitative comparisons and qualitative analyses were performed using mean precision (AP) and mean recall (AR), evaluating the model's performance from both detection (det) and segmentation (segmentation) perspectives.

[0105] ablation experiment

[0106] To verify the effectiveness and feasibility of the proposed ResNet-Transformer backbone network, the cross-scale interactive feature enhancement auxiliary network CIFEN, and the cross-scale interactive fusion network CIF-FPN in improving model performance, four ablation experiments were conducted using Mask R-CNN with ResNet50 as the baseline model: Experiment 1, Mask R-CNN (resnet50), using ResNet50 as the backbone; Experiment 2, CIFEN-Mask R-CNN, adding the cross-scale interactive feature enhancement auxiliary network (CIFEN) to the ResNet50 backbone; Experiment 3, RT+CIFEN-Mask R-CNN, replacing the backbone of Experiment 2 with the ResNet-Transformer backbone network, which replaces the basic residual blocks with multi-head self-attention BoT modules at the highest layer; Experiment 4, MCIF-Transformer Mask R-CNN, adding the cross-scale interactive fusion FPN network (CIF-FPN) to the model of Experiment 3. Table 1 shows the quantitative comparison of the ablation experiment results of the proposed models.

[0107] Table 1 Ablation Experiment Results (%)

[0108]

[0109] The ablation experiment quantified the impact of the sub-module on the overall model's detection (det) and segmentation (seg) performance. The experimental results are shown in Table 1. The baseline model Mask R-CNN (resnet50) achieved APdet of 60.05%, APseg of 62.98%, ARdet of 41.89%, and ARseg of 42.83%. The baseline model, combined with the cross-scale interactive feature enhancement module CIFEM, showed relative improvements of 3.95%, 3.09%, 2.31%, and 6.23% in various metrics for CIFEN-Mask R-CNN. This indicates that constructing an auxiliary network based on CIFEM effectively alleviates the problem of missed detections caused by the loss of lesion information during feature extraction by aggregating features at adjacent scales and enhancing the focus on tumor lesions in lung images. The baseline model, combined with CIFEM and Bottleneck Transformer (BoT), showed further improvements in all metrics compared to the Mask R-CNN (resnet50) model, and also compared to CIFEN-Mask... RCNN achieves a 0.95% improvement in APdet, while APseg, ARdet, and ARseg show slight decreases. This is because, when trained on small to medium-sized datasets, the Transformer lacks the inductive biases inherent in CNNs, such as spatial locality and translation invariance, compared to convolutional neural networks.

[23] Therefore, when the dataset is insufficient, it is difficult to train the backbone network weights with the BoT module, resulting in weak generalization ability and poor performance. After combining the base model with CIFEM, BoT, and CIFM, all four metrics are improved: APdet, APseg, ARdet, and ARseg are 65.55%, 68.13%, 45.00%, and 49.62%, respectively. This indicates that the proposed backbone network, auxiliary network, and CIF-FPN network enhance the feature representation and recognition of tumor lesions in lung images by utilizing cross-scale adjacent high and low layer information, and address the problem of neglecting lesion features by fusing shallow feature information and deep semantic information. To more intuitively demonstrate the instance segmentation effect of this model, Figure 7 Radar charts were used to compare the model performance of different module combinations. It can be seen that the detection and segmentation performance of MCIF-Transformer Mask RCNN is better than that of other combination models. This indicates that introducing cross-scale feature information interaction in the feature extraction and feature fusion stages helps to preserve the detailed information of lesions and effectively alleviates the problem of insufficient feature extraction of lesion areas in lung image instance segmentation tasks. Figure 8 The paper presents 3D grayscale images of PET / CT images and instance segmentation results of the MCIF-TransformerMask RCNN model, demonstrating that the overall architecture and module design of the model are reasonable and can effectively achieve instance segmentation of tumor lesions in PET / CT lung images.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0111] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A medical image instance segmentation method based on a cross-scale interactive fusion Transformer model, characterized in that, The method The method comprises the following steps: The feature extraction step: The preprocessed PET / CT image is input into a backbone network, the backbone network comprises a plurality of basic bottleneck residual modules and a multi-head self-attention module connected in sequence, the basic bottleneck residual module is used to obtain a local feature map, and the multi-head self-attention module is used to obtain a non-local feature map; The local feature map and the non-local feature map are aggregated by using a cross-scale interactive feature enhancement module to obtain an enhanced feature map; specifically comprising the following steps: The current feature map x i The adjacent last layer feature map x i-1 is subjected to a 1*1 convolution operation, and the channel and size of the last layer feature map x i-1 including the local feature map and the non-local feature map are adjusted. The adjusted last layer feature map is added to the current feature map x i The addition fusion is performed to obtain a fusion result f (i-1,i) ; The output enhanced feature map CIFEM i-1 is obtained by performing element-wise addition fusion on the feature map x i-1 of the last layer and the feature map x (i-1,i) of the current layer. (i-1,i-1) ​ Input features f across scales (i-1,i-1) The input soft attention unit processes the cross-scale input features f through channel attention and spatial attention branches, respectively. (i-1,i-1) Processing is performed; specifically, the cross-scale input features f are first processed. (i-1,i-1) The features are divided into G groups along the channel dimension. Then, each group of features is segmented into two blocks along the channel direction and input into the channel attention branch and the spatial attention branch respectively. The results processed by the channel attention branch and the spatial attention branch are spliced to obtain intra-group fusion information feature maps; The random mixed layer is used to fuse cross-group feature information in the channel dimension to obtain an aggregated feature map, which is the enhanced feature map CIFEM i ; The feature fusion step: The enhanced feature map is fused by using a cross-scale interactive fusion FPN network to obtain a fusion feature map; The feature prediction step: The region proposal network is used to extract a bounding box for the fusion feature map; The extracted bounding box is predicted based on ROI Align to obtain a PET / CT image instance segmentation result.

2. The medical image instance segmentation method based on the cross-scale interactive fusion Transformer model according to claim 1, characterized in that, Before the feature extraction step, the following steps are further included: PET / CT images of lung tumors are obtained, and the PET / CT images are preprocessed.

3. The medical image instance segmentation method based on the cross-scale interactive fusion Transformer model according to claim 1, characterized in that, The backbone network comprises four basic bottleneck residual modules and a multi-head self-attention module connected in sequence; The basic bottleneck residual module comprises a residual module and a plurality of bottleneck blocks stacked in series with the residual module.

4. The medical image instance segmentation method based on the cross-scale interactive fusion Transformer model according to claim 1, characterized in that, The non-local feature map is obtained, specifically comprising: The lesion feature map is input into the multi-head self-attention module; The parameter vector Rh is used to represent the relative position encoding of the height of the lesion feature map, and the parameter vector Rw is used to represent the relative position encoding of the width of the lesion feature map, the dimensions of the parameter vectors Rh and Rw are d; The parameter vector Rh and the parameter vector Rw are added and multiplied by a query matrix query to obtain a multiplication result a; The multiplication result b of the query matrix query and the key matrix key is calculated; The multiplication result a and the multiplication result b are element-wise added, and the addition result is processed by a softmax function; The result of the softmax function processing is multiplied by a value matrix to obtain a non-local feature map.

5. The medical image instance segmentation method based on the cross-scale interactive fusion Transformer model according to claim 1, characterized in that, The enhanced feature map is fused by using a cross-scale interactive fusion FPN network, specifically comprising the following steps: performing element-wise addition to obtain a synthesized feature map f j-1 and the adjacent high-level enhanced feature map x j performing element-wise addition to obtain a synthesized feature map f (j-1,j) ; The synthetic feature map f (j-1,j) The average pooling processing and the maximum pooling processing are respectively performed, and the processing results are merged to obtain the pooled features. After the pooled features are processed by a 1*1 convolution layer, a ReLu activation layer and a 1*1 convolution layer in sequence, the processed features are input into a Sigmoid activation function to obtain a weight attention map; The obtained weight attention map is multiplied with the adjacent high-level enhanced feature map x j After multiplication, the adjacent high-level enhanced feature map x j is added to obtain the final fused feature map.

Images