Tumor segmentation method and system based on scale and view common perception

Abstract

The invention discloses a tumor segmentation method and system based on scale and view common perception, and the method comprises the steps: obtaining to-be-segmented medical image data, and carrying out the first preprocessing of the to-be-segmented medical image data, so as to obtain first image data; inputting the first image data after the second preprocessing into a trained patch-based segmentation model to obtain a complete tumor segmentation probability graph; inputting the first image data subjected to the third preprocessing into a trained global-based segmentation model, and extracting global context features to obtain a global context feature graph for extracting and representing the overall structure of the tumor; inputting the first image data into a trained scale sensing network to generate a multi-scale feature map; and performing dynamic weighted fusion based on the multi-scale feature map, the tumor segmentation probability map and the global context feature map to generate a final tumor segmentation mask. According to the tumor segmentation method and system based on scale and view common perception, the accuracy of tumor segmentation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular, to a tumor segmentation method and system based on scale and view co-awareness. Background Art

[0002] Tumor segmentation is a core task in medical image analysis, but the existing technologies have the following problems: multi-scale challenges, significant differences in tumor sizes, existing methods fuse multi-scale features through a multi-branch pyramid structure, but the computational complexity is too high to meet the clinical real-time requirements. Multi-view information islands, traditional methods independently train models for axial, sagittal, and coronal views, resulting in increased storage and computational burdens, and cross-view information cannot be effectively fused, affecting the segmentation accuracy. Local-global imbalance, the patch-based local model (PBSM) can capture details but lacks global structural information; the global-based model (GBSM) is prone to ignoring the boundaries of small tumors. Although existing technologies have proposed multi-view fusion, they rely on static weight allocation and cannot dynamically adapt to different tumor scales and data distributions. Therefore, there is an urgent need for an efficient, accurate segmentation method that can integrate multi-view and multi-scale information. Summary of the Invention

[0003] To solve the technical problems in the background art, the present invention proposes a tumor segmentation method and system based on scale and view co-awareness.

[0004] A tumor segmentation method based on scale and view co-awareness proposed by the present invention includes:

[0005] Obtain medical image data to be segmented, and perform first preprocessing on the medical image data to be segmented to obtain first image data;

[0006] Input the first image data after second preprocessing into a trained patch-based segmentation model to obtain a complete tumor segmentation probability map;

[0007] Input the first image data after third preprocessing into a trained global-based segmentation model to extract global context features for obtaining a global context feature map representing the overall structure of the tumor;

[0008] Input the first image data into a trained scale-aware network to generate a multi-scale feature map;

[0009] Perform dynamic weighted fusion based on the multi-scale feature map, the tumor segmentation probability map, and the global context feature map to generate a final tumor segmentation mask.

[0010] Preferably, the medical image data to be segmented is specifically a CT or MRI scan image with three orthogonal views; the three orthogonal views specifically include the axial plane, the sagittal plane, and the coronal plane; the first image data is specifically the medical image data to be segmented after the first preprocessing; the second preprocessing specifically includes:

[0011] Using a sliding window to decompose each of the three orthogonal views in the first image data into multiple local blocks one by one.

[0012] Preferably, inputting the first image data after the second preprocessing into the trained patch-based segmentation model to obtain a complete tumor segmentation probability map, specifically including:

[0013] Sequentially inputting the multiple local blocks corresponding to each of the three orthogonal views into the trained patch-based segmentation model to predict the tumor probability of each local block;

[0014] And combining the multi-view voting fusion strategy, finally fusing the tumor features corresponding to all local blocks to obtain a complete tumor segmentation probability map, where the tumor features include tumor negative and tumor positive.

[0015] Preferably, the multi-view voting fusion strategy specifically includes:

[0016] Obtaining the weights corresponding to the axial plane, the sagittal plane, and the coronal plane respectively, and performing weighted fusion on the probability values of the same local block corresponding to the three orthogonal views to obtain the probability value of this local block;

[0017] Comparing the probability value with a preset probability threshold, and when the probability value meets the preset probability threshold, marking the corresponding local block as tumor positive.

[0018] Preferably, the third preprocessing specifically includes: splicing the downsampled data of the three orthogonal views of the first image data into a multi-channel input.

[0019] Preferably, performing dynamic weighted fusion based on the multi-scale feature map, the tumor segmentation probability map, and the global context feature map to generate a final tumor segmentation mask, specifically including:

[0020] Obtaining the weights corresponding to the tumor segmentation probability map and the global context feature map, and performing weighted fusion on the tumor segmentation probability map and the global context feature map according to the weights corresponding to the tumor segmentation probability map and the global context feature map to obtain a fused probability map;

[0021] Compressing the multi-scale feature map to a single channel through convolution and adding it to the fused probability map to obtain a final tumor distribution probability map;

[0022] Generating a final tumor segmentation mask according to the tumor distribution probability map.

[0023] Preferably, the scale-aware network specifically includes an encoder, an HCA module, a decoder, a multi-scale awareness module MSAM, and an MV-SAM module; the step of inputting the first image data into the trained scale-aware network to generate a multi-scale feature map specifically includes:

[0024] The first image data is subjected to high-level multi-scale semantic feature extraction by the encoder to generate feature maps at four different resolution levels;

[0025] The four feature maps at different resolution levels are weighted and fused by the HCA module to obtain a fused feature map;

[0026] The decoder and the multi-scale awareness module MSAM are used to adjust the fused feature map to generate a channel-enhanced feature map;

[0027] The MV-SAM module is used to optimize the features of the channel-enhanced feature map to generate a multi-scale feature map.

[0028] Preferably, the first preprocessing specifically includes:

[0029] According to the type of the medical image data to be segmented, the medical image data to be segmented is subjected to data normalization and multi-view spatial registration to obtain the first image data.

[0030] A tumor segmentation system based on co-awareness of scale and view proposed by the present invention includes:

[0031] A first processing module, configured to obtain the medical image data to be segmented, and perform first preprocessing on the medical image data to be segmented to obtain the first image data;

[0032] A second processing module, configured to input the first image data after the second preprocessing into the trained patch-based segmentation model to obtain a complete tumor segmentation probability map;

[0033] A third processing module, configured to input the first image data after the third preprocessing into the trained global-based segmentation model to extract global context features, so as to obtain a global context feature map representing the overall structure of the tumor;

[0034] A fourth processing module, configured to input the first image data into the trained scale-aware network to generate a multi-scale feature map;

[0035] An output module, configured to perform dynamic weighted fusion based on the multi-scale feature map, the tumor segmentation probability map, and the global context feature map to generate a final tumor segmentation mask.

[0036] In the present invention, the proposed tumor segmentation method and system based on scale and view co-awareness acquire medical image data to be segmented, perform a first preprocessing on the medical image data to be segmented to obtain first image data; input the first image data after the second preprocessing into a trained patch-based segmentation model to obtain a complete tumor segmentation probability map; input the first image data after the third preprocessing into a trained global-based segmentation model to extract global context features to obtain a global context feature map representing the overall structure of the tumor; input the first image data into a trained scale-aware network to generate multi-scale feature maps; perform dynamic weighted fusion based on the multi-scale feature maps, the tumor segmentation probability map, and the global context feature map to generate a final tumor segmentation mask. Through multi-view preprocessing, local-global two-stream feature extraction, a scale-aware network (SA-Net), and dynamic weighted fusion, high-precision tumor segmentation is achieved, improving the accuracy of tumor segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 FIG. is a schematic diagram of the overall framework of a tumor segmentation method based on scale and view co-awareness proposed by the present invention;

[0038] Figure 2 FIG. is a schematic diagram of the SA-Net network structure of a tumor segmentation method based on scale and view co-awareness proposed by the present invention;

[0039] Figure 3 FIG. is a schematic diagram of the HCA module of a tumor segmentation method based on scale and view co-awareness proposed by the present invention;

[0040] Figure 4 FIG. is a schematic diagram of the MSAM module structure of a tumor segmentation method based on scale and view co-awareness proposed by the present invention;

[0041] Figure 5 FIG. is a schematic diagram of the workflow structure of a tumor segmentation method based on scale and view co-awareness proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] Referring to Figures 1-5 , a tumor segmentation method based on scale and view co-awareness proposed by the present invention includes:

[0043] Acquire medical image data to be segmented, and perform a first preprocessing on the medical image data to be segmented to obtain first image data.

[0044] In this embodiment, the first preprocessing specifically includes: performing data normalization and multi-view spatial registration on the medical image data to be segmented according to the type of the medical image data to be segmented to obtain first image data.

[0045] In this embodiment, the medical image data to be segmented is specifically a CT or MRI scan image with three orthogonal views; the three orthogonal views specifically include the axial plane, the sagittal plane, and the coronal plane; the first image data is specifically the medical image data to be segmented after the first preprocessing; the second preprocessing specifically includes:

[0046] Using a sliding window to decompose each of the three orthogonal views in the first image data into multiple local blocks one by one.

[0047] Input the first image data after the second preprocessing into the trained patch-based segmentation model to obtain a complete tumor segmentation probability map.

[0048] In this embodiment, inputting the first image data after the second preprocessing into the trained patch-based segmentation model to obtain a complete tumor segmentation probability map specifically includes:

[0049] Input the multiple local blocks corresponding to each of the three orthogonal views into the trained patch-based segmentation model in sequence to predict the tumor probability of each local block;

[0050] And combining the multi-view voting fusion strategy, finally fuse the tumor features corresponding to all local blocks to obtain a complete tumor segmentation probability map, where the tumor features include tumor negative and tumor positive.

[0051] In this embodiment, the multi-view voting fusion strategy specifically includes:

[0052] Obtain the weights corresponding to the axial plane, the sagittal plane, and the coronal plane respectively, and perform weighted fusion on the probability values of the same local block corresponding to the three orthogonal views to obtain the probability value of this local block;

[0053] Compare the probability value with a preset probability threshold, and when the probability value meets the preset probability threshold, mark the corresponding local block as tumor positive.

[0054] Specifically, for the patch-based segmentation model (PBSM), in the training stage, we directly fuse the medical images of different views, and randomly select view data for model optimization during the continuous iteration of the model to achieve efficient cross-view information fusion. Due to the limitation of computing resources, PBSM uses the sliding window method to decompose the original image data into overlapping blocks in sequence for model training. During the testing process, we use the voting method to fuse the results of the three views to obtain the segmentation result of PBSM.

[0055] For the global-based segmentation model (GBSM), in the training stage, we process the multi-view data by downsampling and then send it to the network for global feature extraction. In the testing stage, the prediction results of different views are upsampled and then fused by voting to obtain the segmentation result of GBSM. The final output result of scale-view co-awareness (SVCA) is the fusion of PBSM and GBSM. The overall architecture of the scale-view co-awareness (SVCA) segmentation method proposed in the present invention is as Figure 1 shown. SVCA mainly includes two parts: PBSM and GBSM, which enhance the feature perception of the segmentation framework at the macro level. At the same time, the multi-view fusion strategy is applied in both PBSM and GBSM to enhance the angle perception ability of the segmentation framework to obtain better segmentation results.

[0056] To achieve efficient multi-scale feature fusion, the present invention proposes a novel network (SA-Net) to enhance the multi-scale feature perception ability of the model at the micro level.

[0057] As Figure 2 shown, specifically, the input I ∈ R C×D×H×W is fed into the encoder for high-level multi-scale semantic feature extraction, generating four feature maps with different resolution levels, denoted as E i , where i represents the stage of the encoder. The resolutions of these feature maps are C i represents the number of channels in the i-th stage of the encoder, and D, H, and W represent the depth, height, and width of the input, respectively.

[0058] The encoder also integrates the HCA module to enhance the multi-scale feature extraction ability of the encoder.

[0059] In the decoder part, the feature maps are gradually restored to the same size as the input through upsampling operations with Resblocks. The resolutions of these feature maps are

[0060] At the end of the network, the feature maps {E4, D3, D2, D1} are input into the MSAM to enhance the multi-scale perception ability of the decoder. Finally, by introducing multi-layer auxiliary losses in the decoder and adopting a weighted depth supervision strategy, it helps the model learn higher-quality features and accelerate network convergence.

[0061] The HCA module aims to capture the synergy between different encoder features and adaptively aggregate multi-scale contexts through the channel attention mechanism, thereby further promoting the effective fusion of features. As Figure 3 shown, the feature maps at different encoder stages (i.e., E1, E2, E3) are first downsampled through the average pooling (Avgpool) operation corresponding to E4 to achieve semantic feature alignment. Subsequently, four 1×1×1 3D convolutional blocks are used to smooth the features to reduce the semantic differences between features at different scales. Next, the feature maps are integrated and optimized through the concatenation operation and the channel attention mechanism (CA) to achieve the adaptive fusion of multi-scale features. After that, the optimized feature maps are further compressed and smoothed through another 1×1×1 3D convolutional block. Finally, to avoid information loss, an identity connection of E4 is added to the output of HCA to generate the final output E of the HCA module h . The operation process of the HCA module can be expressed as:

[0062] E fuse = CB(CA(Concat(CB(Avgpool(E1)), CB(Avgpool(E2)), CB(Avgpool(E3)), CB(E4))))

[0063] E h = E fuse + E4;

[0064] where Concat refers to the concatenation operation, and CB represents the convolutional block (Conv + IN + ReLU).

[0065] The MSAM module adaptively adjusts the contributions of features from different decoder stages through the channel attention and spatial attention mechanisms. As Figure 4 shown, the outputs of different decoder stages (i.e., D1, D2, D3) and the last encoder stage (i.e., E4) are first compressed and upsampled through 1×1×1 3D convolutional blocks and upsampling layers to reach the same resolution as D1 and 16 channels respectively, obtaining D'1, D'2, D'3 and D'4. Next, after the element-wise addition operation, the global average pooling (GAP) layer is used to fuse the features and generate a new feature map D fuse , expressed as:

[0066]

[0067] D fuse = GAP(D′1 + D′2 + D′3 + D′4);

[0068] Then, D fuseAfter four 1×1×1 3D convolutional layers with Sigmoid activation, a channel attention map is generated to adaptively integrate D'1, D'2, D'3, and D'4. Then, a 1×1×1 3D convolutional block is used to further optimize the features, generating a channel-enhanced feature map D c . Its operation process is expressed as:

[0069]

[0070] Finally, D c is input into the multi-view strip attention module (MV-SAM) for spatial dependency modeling.

[0071] As Figure 4 shown, the MV-SAM module decomposes the input feature F ∈ R C×D×H×W along three orthogonal planes (axial, coronal, and sagittal) to generate three 2D planar feature maps: F axi ∈ R C×H×W , F cor ∈ R C×D×W and F sag ∈ R C×D×H . This decoupling of spatial features effectively reduces the computational burden of the network, converting 3D operations into more efficient 2D operations. Each planar feature map passes through a strip attention module (SAM) to generate a spatial attention map. Subsequently, matrix multiplication is performed on the spatial attention maps of different views and the input features, and the result is reshaped and transposed to match the dimensions of the input features. Next, the spatially enhanced features are fused and optimized through a 1×1×1 3D convolutional block. To avoid information loss, the D1 feature is fused with the optimized spatially enhanced feature through element-wise addition, and finally, another 1×1×1 3D convolutional block is used to smooth the features to generate the output of the MV-SAM.

[0072] As Figure 4 shown, the input feature F' ∈ R C×H×W first passes through three 1×1 convolutional layers to generate three new feature maps: Q, K, and V. Then, a strip operation is performed on the K feature map with a stride of H×1 to obtain the context representation in the vertical direction. Next, Q and K are reshaped into R C′×N and R C′×W , where N = H×W. Then, the affinity operation of Q T and K is performed to calculate the attention map A ∈ R N×W in the horizontal direction, and the calculation process is as follows:

[0073]

[0074] Similarly, the feature map V is reshaped to V ∈ R W×C after the strip operation. Then, the matrix multiplication of A and V T is performed, and the result is reshaped to R

[0075] . Finally, the input feature F' and the generated attention output F” ∈ R C′×H×W are obtained through element-wise summation operation: C′×H×W where F” is the feature vector at position j in the output feature map F”. j

[0076] The first image data after the third preprocessing is input into the trained global-based segmentation model to extract global context features, so as to obtain a global context feature map representing the overall structure of the tumor.

[0077] In this embodiment, the third preprocessing specifically includes: splicing the downsampled data of three orthogonal views of the first image data into a multi-channel input.

[0078] The first image data is input into the trained scale-aware network to generate a multi-scale feature map.

[0079] In this embodiment, the scale-aware network specifically includes an encoder, an HCA module, a decoder, a multi-scale awareness module MSAM, and an MV-SAM module; inputting the first image data into the trained scale-aware network to generate a multi-scale feature map specifically includes:

[0080] The first image data undergoes high-level multi-scale semantic feature extraction by the encoder to generate feature maps at four different resolution levels;

[0081] The four feature maps at different resolution levels are weighted and fused by the HCA module to obtain a fused feature map;

[0082] The decoder and the multi-scale awareness module MSAM are used to adjust the fused feature map to generate a channel-enhanced feature map;

[0083] The MV-SAM module is used to optimize the features of the channel-enhanced feature map to generate a multi-scale feature map. Based on the multi-scale feature map, the tumor segmentation probability map, and the global context feature map, dynamic weighted fusion is performed to generate the final tumor segmentation mask.

[0084] In this embodiment, based on the multi-scale feature map, the tumor segmentation probability map, and the global context feature map, dynamic weighted fusion is performed to generate the final tumor segmentation mask, specifically including:

[0085] ​Obtain the weights corresponding to the tumor segmentation probability map and the global context feature map, and perform weighted fusion on the tumor segmentation probability map and the global context feature map according to the corresponding weights to obtain a fused probability map;

[0086] Compress the multi-scale feature map to a single channel through convolution and add it to the fused probability map to obtain the final tumor distribution probability map;

[0087] Generate the final tumor segmentation mask according to the tumor distribution probability map.

[0088] The scale-aware network specifically includes an encoder, an HCA module, a decoder, a multi-scale awareness module MSAM, and an MV-SAM module; input the first image data into the trained scale-aware network to generate a multi-scale feature map, specifically including:

[0089] The first image data undergoes high-level multi-scale semantic feature extraction by the encoder to generate four feature maps at different resolution levels;

[0090] Perform weighted fusion on the four feature maps at different resolution levels through the HCA module to obtain a fused feature map;

[0091] Use the decoder and the multi-scale awareness module MSAM to adjust the fused feature map to generate a channel-enhanced feature map;

[0092] Use the MV-SAM module to optimize the features of the channel-enhanced feature map to generate a multi-scale feature map.

[0093] Refer to Figures 1-5 , a tumor segmentation system based on scale and view co-awareness proposed by the present invention includes:

[0094] The first processing module is used to obtain the medical image data to be segmented, perform first preprocessing on the medical image data to be segmented to obtain the first image data;

[0095] The second processing module is used to input the first image data after the second preprocessing into the trained patch-based segmentation model to obtain a complete tumor segmentation probability map;

[0096] The third processing module is used to input the first image data after the third preprocessing into the trained global-based segmentation model to extract global context features and obtain a global context feature map representing the overall structure of the tumor;

[0097] The fourth processing module is used to input the first image data into the trained scale-aware network to generate a multi-scale feature map;

[0098] An output module, configured to perform dynamic weighted fusion based on a multi-scale feature map, a tumor segmentation probability map, and a global context feature map to generate a final tumor segmentation mask.

[0099] As described above, the above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and all should be covered within the protection scope of the present invention.

Claims

1. A tumor segmentation method based on scale and view co-awareness, characterized in that, Including: Obtain medical image data to be segmented, and perform first preprocessing on the medical image data to be segmented to obtain first image data; Input the first image data after second preprocessing into a trained patch-based segmentation model to obtain a complete tumor segmentation probability map; Input the first image data after third preprocessing into a trained global-based segmentation model to extract global context features and obtain a global context feature map representing the overall structure of the tumor; Input the first image data into a trained scale-aware network to generate multi-scale feature maps; Perform dynamic weighted fusion based on the multi-scale feature maps, tumor segmentation probability map, and global context feature map to generate a final tumor segmentation mask.

2. The tumor segmentation method based on scale and view co-awareness according to claim 1, wherein The medical image data to be segmented is specifically a CT or MRI scan image with three orthogonal views; the three orthogonal views specifically include the axial plane, sagittal plane, and coronal plane; the first image data is specifically the medical image data to be segmented after first preprocessing; the second preprocessing specifically includes: Use a sliding window to decompose each of the three orthogonal views in the first image data into multiple local blocks one by one.

3. The tumor segmentation method based on scale and view co-awareness according to claim 2, characterized in that, The step of inputting the first image data after second preprocessing into a trained patch-based segmentation model to obtain a complete tumor segmentation probability map specifically includes: Input the multiple local blocks corresponding to each of the three orthogonal views into a trained patch-based segmentation model in sequence to predict the tumor probability of each local block; And combine a multi-view voting fusion strategy to finally fuse the tumor features corresponding to all local blocks to obtain a complete tumor segmentation probability map, where the tumor features include tumor negative and tumor positive.

4. The tumor segmentation method based on scale and view co-awareness according to claim 3, wherein The multi-view voting fusion strategy specifically includes: Obtain the weights corresponding to the axial plane, sagittal plane, and coronal plane respectively, and perform weighted fusion on the probability values of the same local block corresponding to the three orthogonal views to obtain the probability value of this local block; Compare the probability value with a preset probability threshold. When the probability value meets the preset probability threshold, mark the corresponding local block as tumor positive.

5. The tumor segmentation method based on scale and view co-awareness according to claim 1, wherein The third preprocessing specifically includes: splicing the downsampled data of the three orthogonal views of the first image data into a multi-channel input.

6. The tumor segmentation method based on scale and view co-awareness according to claim 1, characterized in that, The step of performing dynamic weighted fusion based on the multi-scale feature maps, tumor segmentation probability map, and global context feature map to generate a final tumor segmentation mask specifically includes: Obtain the weights corresponding to the tumor segmentation probability map and the global context feature map, and perform weighted fusion on the tumor segmentation probability map and the global context feature map according to the weights corresponding to the tumor segmentation probability map and the global context feature map to obtain a fused probability map; Compress the multi-scale feature maps to a single channel through convolution and add them to the fused probability map to obtain a final tumor distribution probability map; Generate a final tumor segmentation mask according to the tumor distribution probability map.

7. The tumor segmentation method based on scale and view co-awareness according to claim 1, wherein Also including: The scale-aware network specifically includes an encoder, an HCA module, a decoder, a multi-scale awareness module MSAM, and an MV-SAM module; The step of inputting the first image data into a trained scale-aware network to generate multi-scale feature maps specifically includes: The first image data undergoes high-level multi-scale semantic feature extraction by an encoder to generate feature maps at four different resolution levels; The HCA module is used to perform weighted fusion on the feature maps at four different resolution levels to obtain a fused feature map; The decoder and the multi-scale awareness module MSAM are used to adjust the fused feature map to generate a feature map with enhanced channels; The MV-SAM module is used to optimize the features of the feature map with enhanced channels to generate a multi-scale feature map.

8. The tumor segmentation method based on scale and view co-awareness according to claim 1, wherein, The first preprocessing specifically includes: According to the type of the medical image data to be segmented, data normalization and multi-view spatial registration are performed on the medical image data to be segmented to obtain the first image data.

9. A tumor segmentation system based on scale and view co-awareness, characterized in that, It includes: A first processing module, configured to obtain the medical image data to be segmented, and perform first preprocessing on the medical image data to be segmented to obtain the first image data; A second processing module, configured to input the first image data after the second preprocessing into a trained patch-based segmentation model to obtain a complete tumor segmentation probability map; A third processing module, configured to input the first image data after the third preprocessing into a trained global-based segmentation model to extract global context features for obtaining a global context feature map representing the overall structure of the tumor; A fourth processing module, configured to input the first image data into a trained scale-aware network to generate a multi-scale feature map; An output module, configured to perform dynamic weighted fusion based on the multi-scale feature map, the tumor segmentation probability map, and the global context feature map to generate a final tumor segmentation mask.

Images