Tumor image segmentation method

Abstract

The invention relates to a tumor image segmentation method. Performing multi-scale feature extraction on the image by using a convolutional neural network (CNN) to capture different-scale information of the tumor; introducing a self-attention mechanism to enhance the attention capability of the model on the tumor area and the accuracy of feature extraction; performing image segmentation through the improved model based on the U-Net architecture; according to the method, the advantages of a CNN and a VisionTransform are combined, local and global features are extracted by using a hybrid architecture, and position codes are added into the Transform so as to enhance the perception capability of spatial position information; adopting a comparative learning strategy to improve the discrimination capability of the model by comparing positive and negative sample pairs, and performing multi-task learning in combination with a tumor classification task and a segmentation task; designing a combination of a weighted cross entropy loss function and a Dice loss function, and improving the sensitivity to a small tumor region; and performing post-processing by using a conditional random field (CRF) to optimize a segmentation boundary, and eliminating small noise points and filling holes of a segmentation region by applying morphological operation to further optimize a segmentation result.

Description

Technical Field

[0001] The present invention relates to an image segmentation method, in particular to a tumor image segmentation method. Background Art

[0002] The current method for tumor image segmentation, Chinese invention patent CN118212407A, has made some progress in automation and accuracy, but there are still some shortcomings and drawbacks, which affect the performance and effect of the model in practical applications. First, these methods rely on a single CT scan image for segmentation, without fully utilizing the advantages of multimodal imaging data. In actual clinical applications, a single CT image cannot provide enough information to accurately distinguish between tumors and healthy tissues, especially when the tissue contrast is low or the tumor boundary is blurred. Integrating multimodal imaging data (such as MRI, PET, etc.) can significantly improve segmentation accuracy, but current methods do not fully consider this. Secondly, the tumor segmentation model in CN118212407A mainly includes an encoder, a decoder, and a multi-scale attention module. Although the multi-scale attention module helps to combine features of different scales for fusion processing, this structure is still not enough to capture all key features when dealing with complex tumor morphology and changing image backgrounds. Although the feature fusion of the multi-scale attention module at different scales helps to improve segmentation accuracy, if the small-scale features are not paid enough attention in the feature extraction stage, the model still cannot accurately identify small tumors or fine tumor boundaries.

[0003] In addition, the current method has the problem of insufficient sensitivity when dealing with small tumor areas. Although the multi-scale attention module can alleviate this problem to a certain extent, its main limitation is that it fails to fully optimize the response to small tumor areas. Small tumors are often ignored or misjudged during feature extraction and segmentation due to their small size. In order to improve the sensitivity to small tumor areas, more weight should be given to the loss function design, but the current model has not made sufficient optimization in this regard. Although the conditional random field CRF can optimize the segmentation boundary to a certain extent as a post-processing step, its computational complexity is high. When processing large-scale image data, computational efficiency will become a bottleneck. In addition, the effect of CRF is highly dependent on the quality of the initial segmentation result. If the initial segmentation result is not ideal, the optimization effect of CRF will also be limited. Therefore, relying solely on CRF for boundary optimization is not enough to cope with all complex image segmentation scenarios. When applying morphological operations such as opening and closing operations, the current method also has some shortcomings. The opening operation is used to eliminate small noise points, and the closing operation is used to fill the holes in the segmented area. These operations improve the smoothness and integrity of the segmentation results to a certain extent. However, the effect of morphological operations depends on the selection of structural elements and the parameter setting of the operation. In practical applications, different types of tumors and different imaging data require different structural elements and parameter settings, while current methods use fixed structural elements and parameters, which lack flexibility and adaptability. This results in unsatisfactory results of morphological operations when faced with diverse clinical data. Summary of the invention

[0004] The purpose of the present invention is to provide a tumor image segmentation method, thereby solving some of the drawbacks and shortcomings pointed out in the background technology.

[0005] The present invention solves the above-mentioned technical problems by adopting the following technical solutions:

[0006] S1. Use convolutional neural network (CNN) to extract multi-scale features from images to capture different scale information of tumors.

[0007] S2. Introducing the self-attention mechanism to enhance the model’s ability to focus on the tumor area and the accuracy of feature extraction; performing image segmentation through an improved model based on the U-Net architecture;

[0008] The improved model includes a dual-path network;

[0009] S2.1, one of the paths is used to capture global information;

[0010] S2.2, another path is used to capture local details, and the multi-level fusion module performs feature fusion between feature maps at different levels;

[0011] S3, combining the advantages of CNN and VisionTransformer, using a hybrid architecture to extract local and global features, and adding position encoding to Transformer to enhance the perception of spatial location information;

[0012] S4. Use contrastive learning strategy to improve the discrimination ability of the model by comparing positive and negative sample pairs, combine tumor classification task and segmentation task for multi-task learning, and share feature extraction network to improve the overall performance of the model;

[0013] S5. Design a combination of weighted cross entropy loss function and Dice loss function to improve sensitivity to small tumor areas; use conditional random field CRF for post-processing to optimize the segmentation boundaries, and apply morphological operations including opening and closing operations to eliminate small noise points and fill holes in the segmented area to further optimize the segmentation results.

[0014] Furthermore, the method for capturing information of different scales of a tumor includes:

[0015] First, use the target-specific multi-scale convolution kernel through the function:

[0016]

[0017] Where S is the set of convolution kernel sizes, x is the expected value of a specific tumor scale, and α is an adjustment parameter used to adapt to the scale characteristics of the tumor cell population;

[0018] Secondly, dilated convolution is applied to expand the receptive field of the model, using the function:

[0019]

[0020] Where d represents the expansion rate of the dilated convolution, y is the input feature map, and β is the receptive field expansion coefficient;

[0021] Finally, the pyramid network structure is integrated through the function:

[0022]

[0023] Where L represents the number of pyramid layers, z is the input of the feature map of each layer, and γ is the weight coefficient of inter-layer information fusion, which enhances the model's ability to process multi-level information.

[0024] Furthermore, the method for enhancing the ability to focus on the tumor area and the accuracy of feature extraction is as follows:

[0025] First, through the function:

[0026]

[0027] Where x and y represent the eigenvalues ​​of any two points on the feature map. wi j is the learned weight, which indicates the strength of association between point i and point j;

[0028] Then, the image is segmented by an improved model based on the U-Net architecture, using a multi-path feature fusion function:

[0029]

[0030] where z k represents the feature map of the kth path, θ k To fuse the weights, T represents the dimensional range of the entire feature map.

[0031] Furthermore, the dual-path network includes the following two parts:

[0032] P1. Global information path: The purpose is to capture the global features of the image, especially the approximate location and outline of the tumor in the whole image; a large convolution kernel and weighted integral are used to extract global information, and the formula is:

[0033]

[0034] Where x is the input image, k i Indicates that the large convolution kernel is used to extract global features, β i is a parameter controlling the weights of different features, w i is the weight coefficient of the convolution kernel output, and |·| represents the measurement of feature strength;

[0035] P2. Local detail path: aims to extract tumor edges and subtle texture features, using small convolution kernels and local feature enhancement functions:

[0036]

[0037] where h j is a small convolution kernel, δ j is the detail feature enhancement coefficient, η is the parameter that controls the local texture response, and the second-order derivative is used to enhance edge and detail features;

[0038] The multi-level fusion module described above performs feature fusion at different levels to enhance the accuracy of tumor segmentation. The formula is as follows:

[0039]

[0040] where y and z are the outputs of the global information path and the local detail path, respectively, l is the inter-layer fusion weight, and ψ(t) is the fusion ratio function used to dynamically adjust the global and local information.

[0041] Furthermore, the enhancement model's perception of the spatial location information of tumor features includes the following steps:

[0042] S1. Convolutional neural network (CNN): Multiple convolutional layers and pooling layers are used to extract the local features of the tumor and its surrounding areas in detail. This process is done through the function:

[0043]

[0044] Description, where x is the input image, k i represents the i-th convolution kernel, α is the coefficient for adjusting the local feature response intensity, and Ω represents the entire image domain;

[0045] S2. Vision Transformer, ViT part: Use the feature map output by CNN, convert it into a serialized feature vector, and process it through Vision Transformer to capture the global context of tumor features; through the self-attention mechanism, identify tumor-related features in the entire image;

[0046] The global feature extraction process is done by:

[0047]

[0048] Description, where p represents the eigenvector, β j Adjust the sensitivity of self-attention, Θ is the dimension of the feature vector;

[0049] S3. Introduction of position encoding: In order to overcome the problem that VisionTransformer is insensitive to position information when processing image data, position encoding is introduced:

[0050]

[0051] Where t represents the position index, γ k is the amplitude coefficient, and T is the maximum length of the eigenvector.

[0052] Furthermore, the method of improving the discrimination ability of the model includes constructing:

[0053] First, the feature space is optimized by generating positive and negative sample pairs and applying a specific contrast loss function; positive sample pairs: (x i ,x j ) are selected from the same category, negative samples for x i ,x k Select from different categories; the contrast loss function is defined as:

[0054]

[0055] Where f(x) represents the feature extraction function, and m is a predetermined boundary used to increase the boundary space between sample pairs;

[0056] Secondly, a shared feature extraction network f(x) is designed, which is subsequently divided into two task-specific branches: tumor classification and tumor segmentation. The shared feature network maximizes the efficiency of feature extraction, and the joint optimization of classification and segmentation tasks is performed jointly through the loss function:

[0057]

[0058] Where L class and L seg are the loss functions for classification and segmentation tasks respectively, and are the prediction results of classification and segmentation respectively, y and s are the corresponding true labels, and λ is a hyperparameter for adjusting the weight between the two losses.

[0059] Furthermore, the combined loss function L is used to improve the sensitivity to small tumor areas, where:

[0060]

[0061] L CE is a weighted cross entropy loss function that increases sensitivity by giving higher weights to small tumor regions. is the Dice coefficient, which is used to measure the predicted tumor area The overlap between the real tumor area S, λ 1 and λ 2 is the weight coefficient used to balance the two loss functions;

[0062] Secondly, the conditional random field CRF is used to optimize the segmentation boundary, which is defined as:

[0063]

[0064] Where Z is the label map of the image, x is the corresponding input image, ψ u is a unary potential function describing the label probability of a single pixel, ψ p is a pairwise potential function used to adjust the consistency of labels between adjacent pixels;

[0065] Finally, morphological operations are applied, including opening and closing operations, defined as:

[0066] M open (I)=dilation(erosion(I,B),B)

[0067] and

[0068] M close(I)=erosion(dilation(I,B),B)

[0069] Where I is the image to be processed, B is the structural element, which is used to define the size and shape of the operation, the opening operation is used to eliminate small noise points, and the closing operation is used to fill the holes in the segmented area.

[0070] Beneficial effects of the present invention:

[0071] 1. By using convolutional neural networks (CNNs) for multi-scale feature extraction, we can capture information on different scales of tumors. The multi-scale feature extraction method can handle tumors of different sizes and identify the overall layout and details of tumors at an early stage. This advantage ensures the robustness and accuracy of the model when dealing with diverse tumor morphologies.

[0072] 2. The self-attention mechanism enhances the model's ability to focus on the tumor area and the accuracy of feature extraction. Image segmentation is performed through an improved model based on the U-Net architecture. The dual-path network design enables the model to capture global information and local detail information at the same time, and perform feature fusion between feature maps at different levels through a multi-level fusion module. This design improves the segmentation accuracy of the model in complex image backgrounds.

[0073] 3. Combining the advantages of CNN and Vision Transformer (ViT), the hybrid architecture can extract local and global features simultaneously. The introduction of position encoding further enhances the model's perception of spatial position information, making the model perform well in processing tumor images with complex spatial relationships.

[0074] 4. By generating positive and negative sample pairs and applying a specific contrastive loss function, the model's discriminative ability has been significantly improved. Contrastive learning optimizes the feature space, making samples of the same category closer and samples of different categories farther apart, thereby improving segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a flow chart of the tumor image segmentation method of the present invention.

[0076] Figure 2 This is a module diagram of the method for capturing different scale information of tumors according to the present invention.

[0077] Figure 3 Module diagram of the method for enhancing the ability to focus on tumor areas and the accuracy of feature extraction in the present invention.

[0078] Figure 4 This is a dual-path network flow chart of the present invention. DETAILED DESCRIPTION

[0079] The specific implementation modes of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0080] In the tumor image segmentation method, a convolutional neural network (CNN) is first used to perform multi-scale feature extraction on the input image, and convolution kernels of different sizes are used to adapt to and capture the details of the tumor at various sizes, which enables the model to effectively identify tumors ranging from small to large. Subsequently, a self-attention mechanism is introduced to enhance the model's attention to specific areas of the tumor, calculate the interdependencies between different positions in the feature map, and help the model focus on important features more accurately. This mechanism is integrated in an improved model based on U-Net, which includes two main paths: one captures global information to understand the relationship between the tumor and the overall image; the other focuses on local details to ensure that subtle changes in the tumor boundary and interior are captured. These two paths interact through multi-level fusion modules that integrate information between feature maps of different resolutions to maintain segmentation quality.

[0081] In addition, this method combines the advantages of CNN and VisionTransformer to create a hybrid architecture to extract and process local and global features simultaneously. The position encoding added to VisionTransformer enhances the model's understanding of the spatial position information of the image, thereby improving the accuracy and consistency of segmentation. Through a contrastive learning strategy, this method enhances the model's ability to recognize subtle differences by comparing image pairs from the same class (positive sample pairs) and different classes (negative sample pairs). At the same time, multi-task learning is combined with tumor classification and segmentation tasks. Sharing the underlying feature extraction network not only optimizes resource usage, but also improves the overall performance of the model by learning from two related tasks. In terms of loss function design, the combination of weighted cross entropy loss and Dice loss is specifically adjusted to improve sensitivity to small tumor areas. Finally, conditional random field CRF is used as a post-processing step to optimize the boundaries of the segmentation results, making them smoother and more natural. In addition, morphological operations use opening and closing operations to eliminate noise and fill holes generated during the automatic segmentation process, further improving the visual quality and practicality of the segmentation results.

[0082] Embodiment 1:

[0083] In the tumor image segmentation method, a medical imaging dataset containing tumors of various types and sizes is processed. The image size in the dataset is 512x512 pixels, and three sizes of convolution kernels are designed: 3x3, 5x5, and 7x7 to capture small-scale details (edges of small tumors), medium-scale structures (overall shape of medium-sized tumors), and large-scale information (spatial relationships between large tumors or multiple tumors) in tumor images. For each size of convolution kernel, the following function is applied to extract features:

[0084]

[0085] Where S is the convolution kernel size set {3, 5, 7}, x is the expected size of the tumor (the expected value of the tumor size is the average size 4.5), and α is an adjustment parameter, assuming its value is 0.1, which is used to adjust the sensitivity of the size feature.

[0086] First calculate the integral When the convolution kernel size is 3:

[0087]

[0088] This integral can be calculated as:

[0089]

[0090] Then substitute this value into the function f:

[0091] f(3,4.5)=exp(-0.1×29.25)=exp(-2.925)≈0.0539

[0092] Similarly, the f-values ​​of convolution kernels of other sizes can be calculated. In this way, convolution kernels of different sizes can extract features at different levels, thereby effectively capturing the overall picture and detailed features of the tumor.

[0093] Next, the embodiment introduces the dilated convolution. By introducing the expansion rate parameter d in the standard convolution operation, the convolution kernel covers a larger input area without increasing the size of the convolution kernel. The processed feature map y is generated by the CNN network in the previous step, and its size is 256x256. In order to enhance the recognition ability of the tumor edge and its surrounding area, the expansion rate d=3 is selected, which means that when performing convolution, a distance of two pixels will be left between each point of the convolution kernel. In addition, the receptive field expansion coefficient β is introduced to adjust the effect of dilated convolution on expanding the receptive field. Definition of dilated convolution expansion function:

[0094]

[0095] Set β to 0.5 and calculate the integral value at the expansion rate d = 3:

[0096]

[0097] The solution to this integral can be computed directly:

[0098]

[0099] Since log(3)≈1.0986, we have:

[0100] 3×1.0986-2=3.2958-2=1.2958

[0101] Substitute into the original function:

[0102] g(3,y)=y·(1+0.5×1.2958)=y·1.6479

[0103] This means that after applying the dilated convolution to the feature map y, its eigenvalue will increase proportionally by about 64.79%, making the feature map show wider contextual information in space.

[0104] The last embodiment integrates the pyramid network structure and fuses features from different resolutions to enhance feature expression and improve segmentation effect. The number of layers L of the pyramid network is set to 3 to cover feature extraction from coarse to fine. The size of each layer of feature map z is gradually reduced from the original size. The first layer is 256x256, the second layer is 128x128, and the third layer is 64x64. The definition function of the pyramid network structure is:

[0105]

[0106] The value of γ is 0.9, which is the coefficient for adjusting the importance of inter-layer information fusion. A higher γ value indicates stronger inter-layer fusion.

[0107] Calculate the contribution of each layer:

[0108] First layer (n=1):

[0109]

[0110] Second layer (n=2):

[0111]

[0112] The third layer (n=3):

[0113]

[0114] Although the higher-level feature map z n The weights are gradually downgraded, but they still provide global context information to the model, while the bottom layer provides more detailed local information.

[0115] Embodiment 2:

[0116] In the tumor image segmentation method, the self-attention mechanism is used to enhance the model's focus on the tumor area by calculating the interdependencies between points in the feature map. A medical image dataset is processed, and the image resolution in the dataset is 512x512 pixels. The self-attention mechanism is used to enhance the model's ability to understand specific tumor features in these images. The self-attention function is defined as:

[0117]

[0118] Where x and y are feature vectors of two different pixels from the same feature map, and w ij is the weight between these two points, expressing the strength of the association between point i and point j.

[0119] Weight w ij The initial value obtained by model training is [-0.1, 0.1]. This range is small enough to avoid causing too strong weight influence in the early stages of training, allowing the model to gradually learn how to effectively adjust these weights to extract and emphasize key features.

[0120] Suppose there are two feature points with feature vectors x = [1.0, 2.0] and y = [2.0, 3.0], and weights w 11 =0.05,w 12 =0.03,w 21 =0.03,w 22 =0.02.

[0121] Calculate the self-attention score:

[0122]

[0123] Substitute into the self-attention function:

[0124]

[0125] Through this calculation method, the self-attention mechanism can evaluate and emphasize the interactions between specific pixels in the image, thereby enhancing the focus on the tumor area and the accuracy of feature extraction.

[0126] The embodiment further uses an improved model based on the U-Net architecture for image segmentation, designs K=3 paths, and each path extracts a different feature map z k , these feature maps reflect the features of the image at different scales and details. Path 1 extracts low-level features, path 2 extracts mid-level features, and path 3 extracts high-level features. Use feature fusion function:

[0127]

[0128] Among them, θ kis the fusion weight of path k, with a value range of (0.1, 0.5). These weights are used to adjust the influence of each path in the final fusion. T represents the dimensional range of the feature map.

[0129] Assume there are three paths of feature graph z 1 ,z 2 ,z 3 , each feature map is a 512x512 matrix, calculate the normalization factor of the feature map, for the feature map z of path 1 1 :

[0130]

[0131] Assume that feature map z 1 The value of is randomly generated, and the normalization factor is calculated to be approximately 1024.

[0132] For the feature map z 1 ,z 2 ,z 3 , assuming the calculated normalization factors are 1024, 950, and 870 respectively.

[0133] Set the weight θ 1 =0.3,θ 2 =0.4,θ 3 =0.3, calculate the fusion result:

[0134]

[0135] Through this calculation, the eigenvalues ​​of each feature map will be fused proportionally, combining the feature maps of each path, so that the final feature map contains comprehensive information at all levels, thereby effectively improving the segmentation accuracy and robustness of the model.

[0136] Embodiment 3:

[0137] The embodiment implements a dual-path network in a tumor image segmentation method. Assume that a data set containing a large number of tumor images is being processed, and the resolution of each image is 512x512 pixels. The global information path is used to effectively extract the overall position and contour information of the tumor in each image. The formula of the global information path is defined as:

[0138]

[0139] Where x is the input image, k i represents a large convolution kernel, β i is a parameter controlling the weights of different features, w i is the weight coefficient of the convolution kernel output, and |·| represents the measurement of feature strength.

[0140] Choose three large convolution kernels: k1 For 9x9, k 2 For 11x11, k 3 It is 13x13.

[0141] Parameter β i The value range is (0.01, 0.1) to balance the feature strength. 1 =0.05,β 2 =0.03,β 3 =0.07.

[0142] The weight coefficient w of the convolution kernel output i Set between (0.1, 0.3), select w 1 =0.2,w 2 =0.25,w 3 =0.15.

[0143] Assume that the input image x passes through the convolution kernel k 1 ,k 2 ,k 3 The feature intensities obtained after the convolution operation are a 1 =100,a 2 =120,a 3 =80.

[0144] Calculate the weighted output of each convolution kernel:

[0145]

[0146] Due to the characteristic intensity a 1 ,a 2 ,a 3 Too large, causing the output of the exp function to approach zero, indicating that β needs to be adjusted appropriately i to avoid over-suppressing features.

[0147] Reselect parameter β i is a smaller value to reduce the inhibitory effect, such as β 1 =0.001,β 2 =0.002,β 3 =0.0015.

[0148] Recalculate the feature strengths:

[0149]

[0150] The final global feature output:

[0151]

[0152] Through this calculation process, the global information path of the pyramid network structure can effectively extract the approximate position and contour features in the image, and combined with the results of other steps, further improve the model's recognition and segmentation accuracy of the tumor area.

[0153] The embodiment further uses a local detail path to extract tumor edges and subtle texture features. The local detail path uses a small convolution kernel and a local feature enhancement function to enhance edge and detail features through second-order derivatives and integral operations. The formula of the local detail path is defined as:

[0154]

[0155] where h j is a small convolution kernel, δ j is the detail feature enhancement coefficient, and η is the parameter that controls the local texture response.

[0156] Choose three small convolution kernels: h 1 is 3x3, h 2 For 5x5, h 3 is 7x7.

[0157] Detail feature enhancement coefficient δ j The value range is (0.1, 0.5), and the specific choice is δ 1 =0.3,δ 2 =0.4,δ 3 =0.5.

[0158] The parameter η that controls the local texture response has a value range of (0.01, 0.1), and specifically η=0.05.

[0159] Assume that the input image x passes through a small convolution kernel h 1 ,h 2 ,h 3 The feature maps obtained after the convolution operation are b 1 =50,b 2 =60,b 3 =70.

[0160] Calculate the second-order derivative enhancement term, for example, for feature map b 1 :

[0161]

[0162] Calculate the integral enhancement term, for example, for feature map b 1 :

[0163]

[0164] Substitute the formula to calculate the enhanced result:

[0165] L(x)=δ 1 1.31072×10 7 +η·13107200

[0166] L(x)=0.3·1.31072×10 7 +0.05 13107200 = 3.93216 × 10 6 +655360=4.58752×10 6

[0167] Through this calculation process, the local detail path can effectively extract and enhance the detail features in the image, especially the edge and texture information of the tumor.

[0168] The last embodiment uses a multi-level fusion module to perform feature fusion at different levels, and the formula is defined as:

[0169]

[0170] where y and z are the outputs of the global information path and the local detail path, respectively, l is the inter-layer fusion weight, with a value range of (0.1, 0.5), and ψ(t) is a function used to dynamically adjust the fusion ratio of global and local information.

[0171] Set the number of multi-level fusion layers L = 3;

[0172] Inter-layer fusion weight θ l The value range is (0.1, 0.5), θ 1 =0.2,θ 2 =0.3,θ 3 =0.5;

[0173] The dynamic adjustment function ψ(t) ranges from (0,1) and is a linear function ψ(t) = t, which means that the weight of global information is gradually increased during the fusion process;

[0174] Assume that the output of the global information path is y(t)=0.6, and the output of the local detail path is z(t)=0.8, and perform fusion calculations on the three layers.

[0175] First layer fusion (l=1):

[0176]

[0177] Calculate the integral part:

[0178]

[0179] F 1 (y,z)=0.2·0.7=0.14

[0180] Second layer fusion (l=2):

[0181]

[0182] Third layer fusion (l=3):

[0183]

[0184] The final fusion result:

[0185] F(y,z)=F 1 (y,z)+F 2 (y,z)+F 3 (y,z)=0.14+0.21+0.35=0.70

[0186] Through this calculation process, the multi-level fusion module effectively integrates global and local features and finally obtains a balanced feature output.

[0187] Embodiment 4:

[0188] In the tumor image segmentation method, a convolutional neural network (CNN) is used to extract local features. Assuming that a data set containing a large number of tumor images is processed, the resolution of each image is 512x512 pixels. A convolutional neural network containing multiple convolutional layers and pooling layers is designed to extract local features. The specific convolutional layers use convolution kernels of different sizes for feature extraction. The feature extraction function used is defined as:

[0189]

[0190] where x is the input image, k i represents the i-th convolution kernel, α is the coefficient for adjusting the strength of local feature response, and Ω represents the entire image domain.

[0191] Select three convolution kernels: k 1 is 3x3, k 2 For 5x5, k 3 is 7x7.

[0192] The coefficient α for adjusting the local feature response strength ranges from (0.01, 0.1), and specifically α=0.05 is selected.

[0193] Assume that the input image x is a 512x512 image.

[0194] Calculate the convolution result and apply the convolution kernel k 1 :

[0195]

[0196] Assume k1 The values ​​are:

[0197]

[0198] Take i=10, j=10, and assume that the value of x(i+m,j+n) is:

[0199]

[0200] Calculate the convolution result:

[0201]

[0202] Similarly, the results of other convolution kernels can be calculated.

[0203] Substitute the convolution result into the feature extraction function:

[0204]

[0205] Assume that the convolution result is a 1 =-60,a 2 =-40,a 3 =-20, then:

[0206]

[0207] Calculate the feature extraction function:

[0208] C(x)=∫ Ω exp(-0.05·5600)dω=∫ Ω exp(-280)dω≈0

[0209] Since the result of the feature extraction function is close to zero, it indicates that the value of α needs to be adjusted to avoid excessive feature attenuation.

[0210] Reselect α = 0.001:

[0211] C(x)=∫ Ω exp(-0.001·5600)dω=∫ Ω exp(-5.6)dω

[0212] ≈∫ Ω 0.0037dω

[0213] Assume that the integral in the image domain is 1:

[0214] C(x)=0.0037

[0215] Through this calculation process, the convolutional neural network can effectively extract local features in the image, especially the features of the tumor and its surrounding areas.

[0216] The embodiment further converts the extracted feature map into a serialized feature vector and processes it through VisionTransformer (ViT) to capture the global context information of the tumor features. ViT identifies tumor-related features in the entire image through a self-attention mechanism, thereby enhancing the overall recognition ability of the model.

[0217] Assume that we have obtained a feature map C(x) from the previous CNN part, and its size is 512x512.

[0218] This feature map is divided into several small blocks, each of which has a size of 16x16, so that 512 / 16×512 / 16=32×32=1024 small blocks can be obtained.

[0219] Each small block is flattened into a feature vector, resulting in 1024 feature vectors, each of which has a length of 16×16=256.

[0220] These feature vectors p are input into VisionTransformer.

[0221] ViT processes these feature vectors through a self-attention mechanism to identify important features related to the tumor.

[0222] The global feature extraction process is described using the following formula:

[0223]

[0224] Among them, p represents the eigenvector, β j It is a parameter for adjusting the sensitivity of self-attention, with a value range of (0.01, 0.1), and Θ is the dimension of the feature vector.

[0225] Assume that there is a feature vector p = [0.5, 0.6, 0.4, 0.7], whose dimension Θ = 4, and select parameter β j is 0.05.

[0226] Calculate the output of the self-attention mechanism:

[0227]

[0228] Assume m=4, that is, there are 4 attention heads, and the output of each head is calculated independently:

[0229] Output of the first attention head:

[0230]

[0231] Take p 1 =0.5:

[0232]

[0233] Assume the integral result is 0.2:

[0234] V 1 (p)=ln(1+0.2)=ln(1.2)≈0.182

[0235] Calculate the output of other attention heads in turn:

[0236] V 2 (p) = ln(1.1) ≈ 0.095

[0237] V 3 (p) = ln(1.3) ≈ 0.262

[0238] V 4 (p) = ln(1.2) ≈ 0.182

[0239] Overall global feature extraction results:

[0240] V(p)=0.182+0.095+0.262+0.182=0.721

[0241] Through this computational process, VisionTransformer can effectively extract the global features of the image and identify important tumor-related features through the self-attention mechanism.

[0242] The last example uses convolutional neural network CNN and VisionTransformer to process tumor images and identify important features through self-attention mechanism. In order to overcome the problem that ViT is insensitive to position information when processing image data, position encoding is introduced to enable the model to better understand the position of the feature vector in space. The function is defined as:

[0243]

[0244] Among them, t represents the position index, γ k is the amplitude coefficient, ranging from (0.01, 0.1), and T is the maximum length of the eigenvector.

[0245] The position index t is chosen to be in the range (0,1023), corresponding to the 1024 feature vectors extracted from the 512x512 image previously.

[0246] Set the amplitude coefficient γ k The value range is (0.01, 0.1), and the specific choice is γ 1 =0.05,γ 2 =0.07,γ 3 =0.1.

[0247] Set the maximum length of the feature vector T=1024.

[0248] Assume that position index t = 512, calculate the position code:

[0249]

[0250] For k=1:

[0251]

[0252] For k = 2:

[0253]

[0254] For k=3:

[0255]

[0256] Add these values ​​together to get the final positional encoding:

[0257] E(512)=-0.05+0.07-0.1=-0.08

[0258] Add the calculated position code E(t) to the feature vector p to form a new feature vector p with position code ′ =p+E(t).

[0259] Assume that the original feature vector p = [0.5, 0.6, 0.4, 0.7], the position index t = 512, and E(512) = -0.08, then:

[0260] p ′ =[0.5-0.08,0.6-0.08,0.4-0.08,0.7-0.08] =[0.42,0.52,0.32,0.62]

[0261] Through this calculation process, position encoding effectively enhances the spatial position information of the feature vector, allowing ViT to better understand the spatial relationship between different features in the image.

[0262] Embodiment 5:

[0263] The embodiment introduces a contrastive learning strategy in the tumor image segmentation method, generates positive and negative sample pairs and applies a specific contrast loss function to optimize the feature space. Assume that a data set containing multiple types of tumor images is processed, and the resolution of each image is 512x512 pixels. Sample pairs are selected from the data set, and the positive sample pair (x i ,x j ) are selected from the same category and are tumor images of the same type; negative sample pairs (x i,x k ) are selected from different categories and are different types of tumor images.

[0264] Assume the following samples:

[0265] Positive sample pair: (x 1 ,x 2 )(Same image of lung cancer)

[0266] Negative sample pair: (x 1 ,x 3 )(Lung cancer image and breast cancer image)

[0267] Feature extraction function f(x):

[0268] Use convolutional neural network CNN as the feature extraction function f(x) to extract the feature vector of each image.

[0269] Assume the eigenvector is:

[0270] f(x 1 )=[0.5,0.6,0.4,0.7]

[0271] f(x 2 )=[0.45,0.55,0.35,0.65]

[0272] f(x 3 )=[0.2,0.3,0.4,0.5]

[0273] The contrast loss function is defined as:

[0274]

[0275] Where m is a predetermined margin used to increase the margin space between sample pairs. Assume m = 0.1.

[0276] Compute the Euclidean distance between eigenvectors:

[0277] |f(x 1 )-f(x 2 )| 2 =(0.5-0.45) 2 +(0.6-0.55) 2 +(0.4-0.35) 2 +(0.7-0.65) 2

[0278] =0.0025+0.0025+0.0025+0.0025=0.01

[0279] |f(x 1 )-f(x 3)| 2 =(0.5-0.2) 2 +(0.6-0.3) 2 +(0.4-0.4) 2 +(0.7-0.5) 2

[0280] =0.09+0.09+0+0.04=0.22

[0281] Substitute into the contrast loss function to calculate:

[0282] L c =max(0,0.1+0.01-0.22)=max(0,0.1-0.21)=max(0,-0.11)=0

[0283] Since the result of the contrast loss function is zero, it indicates that the model is able to distinguish positive and negative sample pairs well.

[0284] The embodiment further optimizes the segmentation and classification of tumor images, designs a shared feature extraction network f(x), and then divides it into two task-specific branches: tumor classification and tumor segmentation. This method maximizes the efficiency of feature extraction through a shared feature network and performs joint optimization through a joint loss function.

[0285] Use convolutional neural network CNN as the shared feature extraction network f(x) to extract the basic features of each image. Assume that after the feature extraction network, the feature vector of image x is f(x) = [0.5, 0.6, 0.4, 0.7].

[0286] Tumor classification branch: Use a fully connected layer to map shared features to classification results The output is a probability distribution of tumor types.

[0287] Tumor segmentation branch: Use upsampling and convolutional layers to map shared features to segmentation results That is, a binary image of whether each pixel belongs to a tumor.

[0288] The joint loss function is defined as:

[0289]

[0290] The loss function of the cutting task (Dice loss), and are the prediction results of classification and segmentation respectively, y and s are the corresponding true labels, λ is a hyperparameter for adjusting the weight between the two losses, and its value range is (0.1, 1.0). Assume that λ=0.5 is selected.

[0291] Classification Task:

[0292] The true label y=[1,0,0] indicates that the image is lung cancer.

[0293] Prediction results It means that the model predicts a high probability of lung cancer.

[0294] Classification loss function (cross entropy loss):

[0295]

[0296] Segmentation Task:

[0297] The true label s is a 512x512 binary image, indicating whether each pixel belongs to the tumor area.

[0298] Prediction results It is a 512x512 binary image, representing the segmentation result predicted by the model.

[0299] Assume that the Dice loss is calculated as

[0300] Joint loss function:

[0301]

[0302] Through this calculation process, the joint optimization strategy can effectively combine the features of classification and segmentation tasks, so that the model can perform well on both tasks.

[0303] Embodiment 6:

[0304] In the embodiment, a combined loss function L is used in the tumor image segmentation method, and a weighted cross entropy loss function L is used. CE And Dice loss function to optimize model performance. Assuming that the medical imaging dataset contains tumors of various types and sizes, the resolution of each image is 512x512 pixels, the combined loss function is defined as:

[0305]

[0306] Where L CE is the weighted cross entropy loss function, is the Dice coefficient, and are the prediction results of classification and segmentation respectively, y and S are the corresponding true labels, λ 1 and λ 2 is the weight coefficient used to balance the two loss functions, with a value range of (0.1, 1.0). Assuming that λ is selected 1 = 0.7 and λ 2 =0.3.

[0307] In order to improve the sensitivity to small tumor areas, the cross entropy loss L CE Give small tumor areas higher weights. For pixel i, its weight w i The value is 2 if the pixel belongs to the small tumor area, while the weight of other areas is 1.

[0308] Dice coefficient is used to measure the predicted tumor area The overlap between the real tumor area S is defined as:

[0309]

[0310] Suppose there is a tumor image x, and the prediction result obtained by feature extraction and segmentation model is:

[0311] Classification results Indicates a higher probability of lung cancer;

[0312] Segmentation results It is a 512x512 binary image representing the tumor area predicted by the model.

[0313] The corresponding true label is:

[0314] The classification label y=[1,0,0] indicates that it is lung cancer;

[0315] The segmentation label S is a 512x512 binary image that represents the actual tumor region.

[0316] Weighted cross entropy loss L CE :

[0317]

[0318] Dice loss:

[0319] Calculate the true segmentation area S and the predicted segmentation area The overlap degree.

[0320] Assumptions |S|=2000,

[0321]

[0322] Combined loss function L:

[0323]

[0324] L=0.7·0.446+0.3·0.211=0.3122+0.0633=0.3755

[0325] Through this calculation process, the combined loss function can effectively improve the sensitivity to small tumor areas and optimize the overall segmentation accuracy.

[0326] The embodiment further uses the conditional random field CRF to refine the segmentation boundary, using the energy function:

[0327]

[0328] Where Z is the label map of the image, x is the corresponding input image, ψ u is a unary potential function describing the label probability of a single pixel, ψ p is a pairwise potential function used to adjust the consistency of labels between adjacent pixels.

[0329] Unary potential function ψ u (z i ,x i ) is usually based on pixel x i Assume that the model outputs the probability that each pixel is a tumor as P(z i =1|x i ).

[0330] For pixel x i , if P(z i =1|x i )=0.8, then:

[0331] ψ u (z i ,x i )=-log(P(z i =1|x i ))=-log(0.8)≈0.223

[0332] Pairwise potential function ψ p (z i ,z j ,x i ,x j ) is used to encourage neighboring pixels to have the same label, thereby smoothing the segmentation result. It is usually based on the color difference and spatial distance between pixels.

[0333] Assume that pixel x i and x j The color difference is d ij (Using Euclidean distance), the weight parameter is β = 0.5, then:

[0334]

[0335] For two pixels x i and x j , the color difference d ij=0.3, then:

[0336] ψ p (z i ,z j ,x i ,x j )=exp(-0.5·0.3 2 )=exp(-0.045)≈0.956

[0337] Assume that the image x has size 3x3 (to simplify calculations).

[0338] The input image and initial segmentation results are as follows:

[0339]

[0340] Compute the unary potential function:

[0341] ψ u (z i ,x i )=-log(P(z i =1|x i ))

[0342] For pixel x 11 , if P(z 11 =1|x 11 )=0.9, then:

[0343] ψ u (z 11 ,x 11 )=-log(0.9)≈0.105

[0344] Compute the pairwise potential function:

[0345]

[0346] For pixel x 11 and x 12 , the color difference d ij =|0.9-0.7|=0.2, then:

[0347] ψ p (z 11 ,z 12 ,x 11 ,x 12 )=exp(-0.5·0.2 2 )=exp(-0.02)≈0.980

[0348] By minimizing the energy function E(Z) of the conditional random field, the optimized label graph Z is obtained, and the label z of each pixel is adjusted using an iterative optimization algorithm.i , to minimize the overall energy.

[0349] The last embodiment applies morphological operations, including opening and closing operations, to optimize the segmentation results after CRF processing, making them smoother and more accurate.

[0350] Open operation M open (I) is used to eliminate small noise points by first performing an erosion operation and then a dilation operation, and is defined as:

[0351] M open (I)=dilation(erosion(I,B),B)

[0352] Where I is the image to be processed and B is the structure element that defines the size and shape of the operation.

[0353] Closing operation M close (I) is used to fill the holes in the segmented area by first performing a dilation operation and then an erosion operation, which is defined as:

[0354] M close (I)=erosion(dilation(I,B),B)

[0355] Where I is the image to be processed and B is the structure element that defines the size and shape of the operation.

[0356] The structural element B is selected as a 3x3 rectangular element, and the erosion and dilation operations are performed 3 times respectively to ensure sufficient noise removal and hole filling.

[0357] Assume that the segmentation result after CRF processing as follows:

[0358]

[0359] Open operation:

[0360] First perform the corrosion operation, assuming the result is

[0361]

[0362] Then perform the expansion operation to obtain

[0363]

[0364] Closing operation:

[0365] First perform the expansion operation, assuming the result is

[0366]

[0367] Then perform the corrosion operation to obtain

[0368]

[0369] Through this calculation process, morphological operations can effectively eliminate small noise points in the segmentation results and fill the holes in the segmentation area, making the final segmentation results smoother and more accurate.

[0370] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A tumor image segmentation method, characterized in that The steps include: S1. Use convolutional neural network (CNN) to extract multi-scale features from images to capture different scale information of tumors. S2. Introducing the self-attention mechanism to enhance the model’s ability to focus on the tumor area and the accuracy of feature extraction; performing image segmentation through an improved model based on the U-Net architecture; The improved model includes a dual-path network; S2.1, one of the paths is used to capture global information; S2.2, another path is used to capture local details, and the multi-level fusion module performs feature fusion between feature maps at different levels; S3, combining the advantages of CNN and VisionTransformer, using a hybrid architecture to extract local and global features, and adding position encoding to Transformer to enhance the perception of spatial location information; S4. Use contrastive learning strategy to improve the discrimination ability of the model by comparing positive and negative sample pairs, combine tumor classification task and segmentation task for multi-task learning, and share feature extraction network to improve the overall performance of the model; S5. Design a combination of weighted cross entropy loss function and Dice loss function to improve sensitivity to small tumor areas; use conditional random field CRF for post-processing to optimize the segmentation boundaries, and apply morphological operations including opening and closing operations to eliminate small noise points and fill holes in the segmented area to further optimize the segmentation results.

2. The tumor image segmentation method according to claim 1, characterized in that The method for capturing different scale information of tumors includes: First, use the target-specific multi-scale convolution kernel through the function: f(S,x)=exp(-α∫0 S |s-x| 2 ds) Where S is the set of convolution kernel sizes, x is the expected value of a specific tumor scale, and α is an adjustment parameter used to adapt to the scale characteristics of the tumor cell population; Secondly, dilated convolution is applied to expand the receptive field of the model, using the function: g(d,y)=y·(1+β∫1 d log(t)dt) Where d represents the expansion rate of the dilated convolution, y is the input feature map, and β is the receptive field expansion coefficient; Finally, the pyramid network structure is integrated through the function: Where L represents the number of pyramid layers, z is the input of the feature map of each layer, and γ is the weight coefficient of inter-layer information fusion, which enhances the model's ability to process multi-level information.

3. The tumor image segmentation method according to claim 1, characterized in that The method for enhancing the ability to focus on the tumor area and the accuracy of feature extraction: First, through the function: Where x and y represent the eigenvalues ​​of any two points on the feature map, and w ij is the learned weight, which represents the strength of association between point i and point j; Then, the image is segmented by an improved model based on the U-Net architecture, using a multi-path feature fusion function: where z k represents the feature map of the kth path, θ k To fuse the weights, T represents the dimensional range of the entire feature map.

4. The tumor image segmentation method according to claim 1, characterized in that The dual-path network includes the following two parts: P1. Global information path: The purpose is to capture the global features of the image, especially the approximate location and outline of the tumor in the whole image; a large convolution kernel and weighted integral are used to extract global information, and the formula is: Where x is the input image, k i Indicates that the large convolution kernel is used to extract global features, β i is a parameter controlling the weights of different features, w i is the weight coefficient of the convolution kernel output, and |·| represents the measurement of feature strength; P2. Local detail path: aims to extract tumor edges and subtle texture features, using small convolution kernels and local feature enhancement functions: where h j is a small convolution kernel, δ j is the detail feature enhancement coefficient, η is the parameter that controls the local texture response, and the second-order derivative is used to enhance edge and detail features; The multi-level fusion module described in P3. performs feature fusion at different levels to enhance the accuracy of tumor segmentation. The formula is as follows: where y and z are the outputs of the global information path and the local detail path, respectively, l is the inter-layer fusion weight, and ψ(t) is the fusion ratio function used to dynamically adjust the global and local information.

5. The tumor image segmentation method according to claim 1, characterized in that The enhancement model's perception of tumor feature spatial location information includes the following steps: S1. Convolutional neural network (CNN): Multiple convolutional layers and pooling layers are used to extract the local features of the tumor and its surrounding areas in detail. This process is done through the function: Description, where x is the input image, k i represents the i-th convolution kernel, α is the coefficient for adjusting the local feature response intensity, and Ω represents the entire image domain; S2. Vision Transformer, ViT part: Use the feature map output by CNN, convert it into a serialized feature vector, and process it through Vision Transformer to capture the global context of tumor features; through the self-attention mechanism, identify tumor-related features in the entire image; The global feature extraction process is done by: Description, where p represents the eigenvector, β j Adjust the sensitivity of self-attention, Θ is the dimension of the feature vector; S3. Introduction of position encoding: In order to overcome the problem that VisionTransformer is insensitive to position information when processing image data, position encoding is introduced: Where t represents the position index, γ k is the amplitude coefficient, and T is the maximum length of the eigenvector.

6. The tumor image segmentation method according to claim 1, characterized in that The method for improving the discriminative ability of the model includes: First, the feature space is optimized by generating positive and negative sample pairs and applying a specific contrast loss function; positive sample pairs: (x i ,x j ) are selected from the same category, negative samples for x i ,x k Select from different categories; the contrast loss function is defined as: Where f(x) represents the feature extraction function, and m is a predetermined boundary used to increase the boundary space between sample pairs; Secondly, we design a shared feature extraction network f(x), which is subsequently divided into two task-specific branches: tumor classification and tumor segmentation; The shared feature network maximizes the efficiency of feature extraction, and the joint optimization of classification and segmentation tasks is performed jointly through the loss function: Where L class and L seg are the loss functions for classification and segmentation tasks respectively, and are the prediction results of classification and segmentation respectively, y and s are the corresponding true labels, and λ is a hyperparameter for adjusting the weight between the two losses.

7. The tumor image segmentation method according to claim 1, characterized in that The combined loss function L is used to improve the sensitivity to small tumor areas, where: L CE is a weighted cross entropy loss function that increases sensitivity by giving higher weights to small tumor regions. is the Dice coefficient, which is used to measure the predicted tumor area The overlap between the real tumor region S, λ1 and λ2 are the weight coefficients used to balance the two loss functions; Secondly, the conditional random field CRF is used to optimize the segmentation boundary, which is defined as: Where Z is the label map of the image, x is the corresponding input image, ψ u is a unary potential function describing the label probability of a single pixel, ψ p is a pairwise potential function used to adjust the consistency of labels between adjacent pixels; Finally, morphological operations are applied, including opening and closing operations, defined as: M open (I)=dilation(erosion(I,B),B) and M close (I)=erosion(dilation(I,B),B) Where I is the image to be processed, B is the structural element, which is used to define the size and shape of the operation, the opening operation is used to eliminate small noise points, and the closing operation is used to fill the holes in the segmented area.

Images