Model training methods, medical image segmentation methods and devices

Abstract

To address the shortcomings of existing medical image segmentation methods, such as insufficient global feature capture, high computational complexity, and low generalization performance, this invention, belonging to the field of deep learning, provides a model training method, a medical image segmentation method, and an apparatus. It designs a Dual Feature Aggregation Transformer module (DFATB) based on a Feature Aggregation Feedforward Network (FAFN). By integrating spatial aggregation attention and channel aggregation attention mechanisms, it enhances the model's ability to capture richer contextual information and complex features. Furthermore, it designs the FAFN module, which introduces a gating mechanism to capture nonlinear spatial information and reduce channel redundancy, thereby achieving efficient feature extraction while reducing the model's computational complexity.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a model training method, a medical image segmentation method, and an apparatus. Background Technology

[0002] Medical images are visualized data of the human body's internal structure and function obtained using medical imaging technology, including images of multiple abdominal organs and skin lesions. Accurate and efficient medical image segmentation can identify lesion areas, playing a crucial role in disease prevention and diagnosis, disease analysis and assessment, and personalized medical planning.

[0003] In related technologies, deep learning methods are deeply integrated with medical image analysis for medical image segmentation, achieving good results. These methods can be broadly classified into three categories: First, Convolutional Neural Networks (CNNs). Due to their inherent inductive bias, CNNs have become the foundation for medical image segmentation and are widely used in processing medical image segmentation tasks, such as U-Net, U-Net++, and V-Net. However, due to the limitation of local receptive fields, traditional CNN models often face the challenge of insufficient spatial context modeling capabilities when dealing with medical image problems, especially when handling complex medical image analysis tasks such as abdominal multi-organ segmentation. Second, hybrid models combining CNN and Transformer networks. Transformers, with their unique self-attention mechanism, can effectively capture long-range dependencies in input data, thus achieving accurate modeling of the global context. This compensates for the insufficient spatial context modeling ability of traditional CNNs when processing medical images due to limitations in local receptive fields. Hybrid CNN and Transformer models can extract detailed local features and better capture global information, thereby enhancing the understanding, analysis, and processing of complex medical images. For example, the ContextNet architecture uses contextual information to supplement deficiencies in local feature representations, improving the model's accuracy in medical image segmentation tasks. However, these models still rely on the CNN backbone and still suffer from low generalization ability when processing unfamiliar medical data. Third, pure Transformer models. To address the insufficient generalization ability of CNN-Transformer models, purely Transformer-based models have emerged. For example, Swin-UNet, a new method using a pure Transformer architecture, adopts a structure similar to U-Net while incorporating skip connections to enhance feature representation. However, although Swin-UNet attempts to reduce computational complexity through sequential Transformer blocks and a sliding window setup, it still falls short in effectively handling spatial context, particularly in its ability to effectively focus on local details while preserving global information. It is evident that while Transformer-based models can effectively capture global features and improve generalization performance, they rely on relatively robust Transformer models, resulting in high computational complexity. This limits the development and widespread application of these models in practical medical image analysis problems.

[0004] In summary, while the three types of models mentioned above have achieved significant progress, especially the hybrid CNN-Transformer model and the pure Transformer model, both of which have demonstrated good performance, they still face challenges such as insufficient generalization performance and high computational complexity. These limitations are particularly pronounced when dealing with complex tasks like medical image segmentation, potentially leading to misdiagnosis by doctors based on the segmentation results and thus lowering the accuracy of the diagnostic methods. Summary of the Invention

[0005] In view of the above problems, embodiments of this application provide a model training method, a medical image segmentation method, and an apparatus to overcome or at least partially solve the above problems.

[0006] In a first aspect, embodiments of this application provide a model training method, the method comprising:

[0007] The patient's sample images were preprocessed to obtain the first medical image;

[0008] A medical image segmentation model was established based on the Transformer framework.

[0009] The first medical image is input into the medical image segmentation model, and the first segmentation mask and the first classification probability corresponding to the first segmentation mask are output.

[0010] Based on the first segmentation mask and the segmentation label of the first medical image, the final loss value of the medical image segmentation model is determined;

[0011] Based on the final loss value, the model parameters of the medical image segmentation model are adjusted to obtain the target medical image segmentation model; the target medical image segmentation model is used to identify the lesion area of ​​the patient to be detected based on the first detection medical image of the patient to be detected.

[0012] Optionally, the medical image segmentation model includes an overlapping patch embedding module, an encoder module, a decoder module, and a linear projection layer; wherein:

[0013] The encoder module consists of three dual feature aggregation blocks A distributed from top to bottom;

[0014] The decoder module consists of three dual-feature aggregation blocks B distributed from bottom to top;

[0015] A differential transformer module is set between each of two adjacent dual-feature aggregation blocks B;

[0016] The outputs of each differential transformer module are fused with the first two dual feature aggregation blocks A in the encoder module through skip connections.

[0017] Optionally, both the dual-feature aggregation block A and the dual-feature aggregation block B include:

[0018] The efficient feature aggregation block first extracts the spatial dependencies of input features by normalizing the key and query through a normalization function, then determines the spatial attention by multiplying the normalized key and value, and obtains the global context vector. Finally, the global context vector is combined with the normalized query to obtain the spatial features.

[0019] The channel attention feature aggregation block uses a cross-attention mechanism to obtain channel dependencies and determine channel attention, thus obtaining channel features;

[0020] The feature aggregation feedforward network performs nonlinear transformations on spatial features and channel features respectively to obtain enhanced spatial features and channel features.

[0021] Optionally, the feature aggregation feedforward network performs nonlinear transformations on the spatial features and channel features respectively to obtain enhanced spatial features and channel features, including:

[0022] The input feature vector is mapped through a linear projection layer to obtain the first linear projection feature, and then the first linear projection feature is mapped to the second linear projection feature through the GELU activation function.

[0023] The second linear projection feature is separated into two parts. One part is subjected to depthwise convolution, and the other part is processed by bypass. The result of depthwise convolution is multiplied element-wise with the result of bypass processing, and the result of element-wise multiplication is input into the linear projection layer to output the final feature.

[0024] Optionally, the first medical image is input into a medical image segmentation model, and a first segmentation mask and a first classification probability corresponding to the first segmentation mask are output, including an encoder processing stage and a decoder processing stage, wherein:

[0025] The encoder processing stage sequentially extracts features from the input first medical image through three dual feature aggregation blocks A, ultimately obtaining the aggregated feature vector of the third sample image;

[0026] In the decoder processing stage, the feature vector of the third sample image is extracted sequentially through three dual feature aggregation blocks B and two differential transformer modules to obtain the merged feature vector of the fifth sample image. Then, through the linear projection layer, the first segmentation mask and the first classification probability corresponding to each first segmentation mask are output.

[0027] Optionally, based on the first segmentation mask and the segmentation label of the first medical image, the final loss value of the medical image segmentation model is determined, including:

[0028] The Dice loss value of the medical image segmentation model is determined by measuring the overlap between the first segmentation mask and the segmentation label of the first medical image.

[0029] Based on the first classification probability and the segmentation label of the first medical image, the cross-entropy loss value of the medical image segmentation model is determined;

[0030] Based on the Dice loss value and the cross-entropy loss value, the final loss value of the medical image segmentation model is determined.

[0031] Secondly, embodiments of this application provide a medical image segmentation method, the method comprising:

[0032] Acquire the first medical images of the patient to be tested;

[0033] The first detection medical image is preprocessed to obtain the first detection image;

[0034] The preprocessed first detected image is input into the target medical image segmentation model, and the first classification category is output; wherein, the target medical image segmentation model is obtained based on the model training method described above;

[0035] Based on the first classification category, the lesion area of ​​the patient to be tested is identified.

[0036] Thirdly, embodiments of this application provide a medical image segmentation apparatus, the apparatus comprising:

[0037] The acquisition module is used to acquire the first medical images of the patient to be tested.

[0038] The preprocessing module is used to preprocess the first detection medical image to obtain the first detection image;

[0039] The segmentation module is used to input the preprocessed first detected image into the target medical image segmentation model and output the first classification category; wherein, the target medical image segmentation model is obtained based on the model training method described above;

[0040] The identification module is used to identify the lesion area of ​​the patient to be detected based on the first classification category.

[0041] The specific beneficial effects are as follows:

[0042] First, unlike traditional network structures that rely on specific datasets and have high computational complexity, this invention employs a more generalizable and lightweight pure transformer architecture based on differential transformers and featuring U-shaped dual feature aggregation, which can efficiently capture complex features. This provides a more general and universal processing method for medical image segmentation tasks, showing great potential and application prospects in practical medical problems.

[0043] Second, this invention proposes a Feature Aggregation Feedforward Neural Network (FAFN). FAFN is based on a nonlinear gating mechanism, which efficiently aggregates features and reduces information redundancy. This reduces computational complexity and provides a more lightweight option.

[0044] Third, this invention proposes a Dual Feature Aggregation Transformer Block (DFATB), which combines spatial aggregation and channel aggregation attention mechanisms to capture more complex and comprehensive features, thereby improving feature aggregation capabilities and thus enhancing the accuracy of medical image segmentation.

[0045] Fourth, this invention introduces a differential transformer module, which can reduce attention to unnecessary information and focus attention on key information. By introducing this module, regional features and edge information in medical images can be captured, improving segmentation accuracy and making the model widely applicable to diverse medical image datasets. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic flowchart of a model training method provided in an embodiment of this application;

[0048] Figure 2 This is a schematic flowchart of a medical image segmentation method provided in an embodiment of this application;

[0049] Figure 3 This is a diagram illustrating the architecture of the DFAFormer model provided in an embodiment of this application; wherein, Figure 3 (a) is a structural diagram of the feature aggregation feedforward neural network. Figure 3 (b) is a structural diagram of the dual-feature aggregation Transformer block. Figure 3 (c) is the structure diagram of the differential transformer;

[0050] Figure 4 A comparison diagram of standard attention and differential attention structures provided for embodiments of this application; wherein, Figure 4 (a) represents the standard attention mechanism. Figure 4 (b) shows the differential attention mechanism;

[0051] Figure 5 Visual comparison results of the DFAFormer model provided in this application embodiment with other advanced segmentation models on the Synapse dataset;

[0052] Figure 6 Visual comparison results of the DFAFormer model provided in this application embodiment with other advanced segmentation models on the ISIC 2018 dataset. Detailed Implementation

[0053] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0054] Reference Figure 1 , Figure 1 This is a flowchart illustrating a model training method provided in an embodiment of this application. The method may include:

[0055] Step 101: Preprocess the patient's sample image to obtain the first medical image.

[0056] Patient sample images typically refer to medical images obtained using advanced imaging techniques such as abdominal computed tomography (CT) and magnetic resonance imaging (MRI).

[0057] Next, all the obtained sample images are labeled with lesion region categories to obtain segmentation labels for the sample images. These segmentation labels may include liver, left kidney, right kidney, pancreas, spleen, stomach, aorta, and gallbladder. Then, the segmentation labels can be converted into a one-hot encoding format, which allows the segmentation label for each pixel to be represented by a vector, where each position corresponds to a category.

[0058] After the sample images are labeled, they are divided into training and test sets proportionally. N samples are allocated to the training set, and the remaining M samples to the test set. Random augmentation operations are performed on the sample images and segmentation labels in the training set. These operations include random flipping, rotation, and contrast adjustment to ensure image diversity and model robustness during training. Furthermore, these augmentation operations should be performed simultaneously on the sample images and their corresponding segmentation labels to avoid misalignment of segmentation labels. Meanwhile, the test set contains the original data without random augmentation, ensuring that the model sees more transformed images during training and maintains the original image format during validation.

[0059] Finally, all sample images and segmentation labels in both the training and test sets were uniformly resized to 224×224. During resizing, images were scaled using cubic interpolation, and labels were scaled using nearest-neighbor interpolation to avoid smoothing between labels.

[0060] After the above preprocessing, the first medical image obtained after preprocessing the sample image can be obtained.

[0061] Step 102: Based on the Transformer framework, establish a medical image segmentation model.

[0062] In the embodiments of this application, the medical image segmentation model is a U-shaped dual-feature aggregation pure Transformer model, abbreviated as DFAFormer model. The DFAFormer model employs a U-shaped encoder-decoder architecture, which is a multi-layered pure Transformer structure, such as... Figure 3 As shown.

[0063] Mainstream medical image segmentation methods often employ a U-shaped encoder-decoder structure. This invention improves upon the traditional method by designing three identical dual feature aggregation Transformer blocks (DFATB) in both the encoder and decoder modules. A differential transformer module is also introduced in the decoding stage, and the output of the differential transformer module is fused with the features from its parallel encoding stage through skip connection cross attention (SCCA) to construct the DFAFormer model.

[0064] Combination Figure 3 (a)-(c) For a given input image x∈R with spatial dimensions H×W and channels C H×W×CThe processing flow of the DFAFormer model includes: using the overlap patch embedding module to extract 4×4 patch tokens from the input image, and then processing the extracted patch tokens (denoted as x∈R)... n×d The features are passed to the encoder module, which consists of three stacked Dual Feature Aggregation Transformer Blocks (DFATBs). Each DFATB consists of two consecutive feature aggregation Transformer layers and a patch merging module. During patch merging, 2×2 patch tokens are aggregated, doubling the channel dimension while reducing the spatial dimension. This hierarchical merging enables the model to capture multi-scale representations while preserving fine-grained and global features.

[0065] During the decoding phase, the patch token is expanded by a factor of 2 in each block. A differential transformer module is then used to denoise the image. Following this, Skip Connection Cross-Attention (SCCA) fuses the output x1 of each differential transformer module with the output features x2 from the parallel encoding phase via skip connections. This model fuses two skip connections to ensure the fusion of feature streams across different scales; the fused features are then processed by a DFATB module. Finally, the model generates a segmentation mask through a linear projection layer.

[0066] The encoder module of the DFAFormer model includes three dual feature aggregation blocks A from top to bottom, and the decoder module includes three dual feature aggregation blocks B from bottom to top. Dual feature aggregation blocks A and B constitute the Dual Feature Aggregation Transformer Block (DFATB). The workflow of the DFAFormer model designed in this invention includes:

[0067] 1) Encoder stage:

[0068] The first input medical image is processed by the overlapping patch embedding module to obtain the patch token, which is the first sample segmentation image;

[0069] The first sample segmented image is input into the first dual feature aggregation block A to obtain the first sample image aggregation feature vector;

[0070] The aggregated feature vector of the first sample image is downsampled and input into the patch merging module to obtain the merged feature vector of the first sample image;

[0071] The first sample image merged feature vector is input into the second dual feature aggregation block A to obtain the second sample image aggregated feature vector;

[0072] The aggregated feature vector of the second sample image is downsampled and input into the patch merging module to obtain the merged feature vector of the second sample image;

[0073] The merged feature vector of the second sample image is input into the third dual feature aggregation block A to obtain the aggregated feature vector of the third sample image;

[0074] 2) Decoder stage:

[0075] The aggregated feature vector of the third sample image is input into the decoder module, and the aggregated feature vector of the fourth sample image is obtained after passing through the first dual feature aggregation block B.

[0076] The aggregated feature vector of the fourth sample image is upsampled and input into the patch merging module to obtain the merged feature vector of the third sample image;

[0077] The feature vector of the third sample image is merged, upsampled, and then input into the difference transformer module to obtain the first difference feature vector;

[0078] The first fused feature is obtained by skip-connecting the aggregated feature vector of the second sample image with the first difference feature vector;

[0079] The first fusion feature is input into the second dual feature aggregation block B to obtain the fifth sample image aggregation feature vector;

[0080] The aggregated feature vector of the fifth sample image is upsampled and input into the patch merging module to obtain the merged feature vector of the fourth sample image;

[0081] The merged feature vector of the fourth sample image is upsampled and then input into the difference transformer module to obtain the second difference feature vector;

[0082] The first sample image aggregate feature vector and the second difference feature vector are connected by a skip connection to obtain the second fused feature;

[0083] The second fusion feature is input into the third dual feature aggregation block B to obtain the aggregated feature vector of the sixth sample image;

[0084] The aggregated feature vector of the sixth sample image is upsampled and input into the patch merging module to obtain the merged feature vector of the fifth sample image;

[0085] The merged feature vector of the fifth sample image is input into the linear projection layer, and the first segmentation mask and the first classification probability corresponding to each first segmentation mask are output.

[0086] Optionally, the Dual Feature Aggregation Transformer module (DFATB) is implemented based on a Feature Aggregation Feedforward Network (FAFN). The Feature Aggregation Feedforward Network employs a nonlinear gating mechanism, and its architecture is as follows: Figure 3 As shown in (a). The feature aggregation-based feedforward network includes depthwise convolution and element-wise multiplication. The depthwise convolution operation can convolve the input image to extract image features; at the same time, the element-wise multiplication operation multiplies the input image element by element, and finally the linear projection layer is used to output the final features.

[0087] In the embodiments of this application, given input X∈R H×W×C The formula for calculating FAFN is as follows:

[0088] (Equation 1)

[0089] (Equation 2)

[0090] (Equation 3)

[0091] in, and Represents the linear projection weights. This represents the learnable convolution parameters along the depth direction. and Represents the separated images, all of which are in R. H×W×C’ In the set, C' represents the hidden dimension in FAFN. This represents the activation function. This represents the feature map after processing with the GELU activation function.

[0092] Combination Figure 3 In (a), the workflow of the feature aggregation-based feedforward network is as follows: the input feature vector is mapped to a higher-dimensional space through a linear projection layer to obtain the first linear projection feature. Then, the nonlinear representation capability of the model is increased by the GELU activation function, and the first linear projection feature is mapped back to the original dimension or the required output dimension to obtain the second linear projection feature. Then, the second linear projection feature is separated into two parts, one part is subjected to depthwise convolution, and the other part is processed by bypass. The result of depthwise convolution is multiplied element-wise with the result of bypass processing. Finally, the result of element-wise multiplication is passed through the linear projection layer to output the final feature.

[0093] Compared to traditional feedforward neural networks (FFNs), this invention, through the design of a feature aggregation feedforward network (FAFN), can capture nonlinear spatial information while reducing channel redundancy. Furthermore, it can reduce computational complexity and improve overall computational efficiency through deep convolution operations.

[0094] Optionally, such as Figure 3 As shown in Figure (b), the Dual Feature Aggregation Transformer Module (DFATB) based on the feature aggregation feedforward network comprises two key modules: the Efficient Feature Aggregation Transformer Block (EFATB) and the Channel Attention Feature Aggregation Transformer Block (CFATB). EFATB focuses on spatial aggregation through an efficient attention mechanism, while CFATB emphasizes channel information aggregation. Together, EFATB and CFATB promote inter-block feature aggregation across spatial and channel dimensions, enhancing the model's ability to capture complex features at multiple levels.

[0095] The Efficient Feature Aggregation Transformer Block (EFATB) is a lightweight spatial attention module used to capture spatial dependencies in input features while maintaining computational efficiency. Its structure is as follows: Figure 3 As shown in (b), the efficient feature aggregation Transformer block can solve the computational complexity (O(N)) of the traditional self-attention mechanism. 2 This addresses the issue of high computational burden by reducing the computational load.

[0096] Given an input feature x∈R H×W×C EFATB operates as follows: First, the Efficient Feature Aggregation Transformer block extracts the spatial dependencies of the input features through normalized keys and queries. Then, it computes attention by multiplying normalized keys and values ​​to obtain a global context vector, which is then multiplied by the normalized query to generate a new feature representation.

[0097] The calculation process of EFATB can be expressed by the following formula:

[0098] (Equation 4)

[0099] in, and These represent the normalization functions applied to the query and the key, respectively; Indicates a query. Indicates key, Represents value, T Indicates transpose; Represents the generated spatial features;

[0100] Attention output is passed through residual connections and layer normalization to ensure stable feature propagation. Subsequently, FAFN performs nonlinear transformations on the features using equations (1)-(3) to further enhance spatial representation.

[0101] Therefore, the final output of EFATB is:

[0102] (Equation 5)

[0103] (Equation 6)

[0104] in, Indicates input features, These represent the query, key, and value calculated from the input feature X, respectively. This represents the Feature Aggregation Feedforward Network (FAFN) in the EFATB module.

[0105] Furthermore, to capture channel dependencies and further improve the robustness and generalization of the model, this invention designs a Channel Feature Aggregation Transformer Block (CFATB). CFATB utilizes channel attention to capture channel dependencies. Specifically, CFATB first transposes the key and query, and then generates channel attention weights based on the cross-covariance matrix. The calculation process is shown below:

[0106] (Equation 7)

[0107] (Equation 8)

[0108] in, The context vector obtained by transposing the attention. These are balancing parameters to ensure stable training. These are the generated channel features.

[0109] After calculating channel attention, residual connections and layer normalization are applied. Finally, FAFN is used to perform nonlinear modeling of the channel features, reducing redundant information and enhancing the representation of meaningful features. The output of CFATB is represented as:

[0110] (Equation 9)

[0111] (Equation 10)

[0112] (Equation 11)

[0113] in, These represent the query, key, and value of the input channel feature aggregation Transformer block (CFATB), respectively. This represents the Feature Aggregation Feedforward Network (FAFN) in the CFATB module.

[0114] As shown above, the dual feature aggregation Transformer module in this invention performs spatial and channel aggregation through EFATB and CFATB respectively, thereby achieving the purpose of dual feature aggregation. In the DFATB design, both EFATB and CFATB achieve robust feature modeling in both spatial and channel dimensions. This dual attention mechanism improves the model's ability to represent complex feature relationships and reduces computational complexity to some extent.

[0115] In summary, the dual-feature aggregation Transformer module enhances the model's ability to capture richer contextual information and complex features by integrating spatial aggregation attention and channel aggregation attention mechanisms.

[0116] Optionally, the DFAFormer model incorporates a differential transformer from the field of natural language processing, namely a differential attention mechanism, such as... Figure 4 As shown, where Figure 4 (a) is the standard attention mechanism, and (b) is the differential attention mechanism.

[0117] Compared to the traditional Transformer, the Differential Transformer focuses more on the similarity during the attention computation process and emphasizes the differences between features through difference operations. This approach effectively removes unnecessary noise and improves the model's ability to capture relevant features. The key to the Differential Transformer is splitting the attention heads into two groups and then processing their outputs using difference operations.

[0118] The specific calculation process of the differential transformer can be expressed by the following formula:

[0119] (Equation 12)

[0120] (Equation 13)

[0121] (Equation 14)

[0122] (Equation 15)

[0123] (Equation 16)

[0124] (Equation 17)

[0125] in, These are learnable scalar parameters; , , These are the weight matrices for the query, key, and value, respectively. Input features; , These are two attention outputs calculated using the flash_attention function; This represents a segmentation operation, which divides the vector resulting from multiplying the input feature X by the corresponding weight matrix into two parts, which are used to calculate two different attention outputs A1 and A2, respectively.

[0126] Because the Differential Transformer highlights feature differences through differential operations, it enhances the model's ability to capture context and key information. Simultaneously, combined with FlashAttention technology, the Differential Transformer significantly reduces computational complexity while maintaining performance. This invention uses the Differential Transformer in the decoder stage, further improving the model's robustness and generalization performance.

[0127] Step 103: Input the first medical image into the medical image segmentation model and output the first segmentation mask and the first classification probability corresponding to the first segmentation mask.

[0128] The preprocessed first medical image is input into a medical image segmentation model, thereby obtaining the first segmentation mask and its corresponding first classification probability output by the medical image segmentation model. The first segmentation mask represents the predicted mask of the lesion region. The first classification probability represents the confidence level of the corresponding first segmentation mask.

[0129] Step 104: Based on the first segmentation mask and the segmentation label of the first medical image, determine the final loss value of the medical image segmentation model.

[0130] In embodiments of this application, the loss value of the medical image segmentation model can be calculated based on the first segmentation mask and the segmentation label of the first medical image. The loss value may include, but is not limited to, Dice loss and cross-entropy loss (CE), or a combination of these loss values.

[0131] Optionally, step 104 may include:

[0132] Sub-step 1041: Determine the Dice loss value of the medical image segmentation model by measuring the overlap between the first segmentation mask and the segmentation label of the first medical image.

[0133] In the embodiments of this application, Dice loss can be used to quantify the overlapping area between the prediction results and the actual annotations, and is particularly suitable for multi-organ segmentation with significant size differences and shape changes, thus solving the problem of data imbalance.

[0134] The method for calculating the Dice loss value is shown in the following formula:

[0135] (Equation 18)

[0136] in, This represents the Dice loss value. To represent an extremely small number, This represents the pixel value of the first segmentation mask. The pixel value representing the segmentation label of the first medical image. This indicates intelligent multiplication of elements.

[0137] Sub-step 1042: Based on the first classification probability and the segmentation label of the first medical image, determine the cross-entropy loss value of the medical image segmentation model.

[0138] In the embodiments of this application, the cross-entropy loss value is calculated using the following formula:

[0139] (Equation 19)

[0140] in, This represents the cross-entropy loss value. The weights of positive samples are represented. Indicates the first i The first classification probability corresponding to the first segmentation mask. Indicates the first i The segmentation labels of the first medical image corresponding to each of the first segmentation masks, where N represents the total number of first segmentation masks, and log() represents the logarithmic function. The weights can be adjusted... To balance the importance of different categories.

[0141] Dice loss is better suited for directly optimizing the similarity between the predicted segmented image and the ground truth label, especially for tasks such as abdominal multi-organ segmentation where targets vary in size, shape, and distribution. On the other hand, cross-entropy loss focuses more on improving the model's performance in multi-class classification. Therefore, this invention combines the two to determine the final loss value of the medical image segmentation model.

[0142] Sub-step 1043: Based on the Dice loss value and the cross-entropy loss value, determine the final loss value of the medical image segmentation model.

[0143] In the embodiments of this application, the weighted sum of the Dice loss and cross-entropy loss is used as the final loss value. The final loss value is calculated as follows:

[0144] (Equation 20)

[0145] in, This represents the final loss value of the medical image segmentation model. The weights for the Dice loss values, This represents the Dice loss value. The weights are the cross-entropy loss values. Let be the cross-entropy loss value, and . ;

[0146] In the embodiments of this application, by using a combined loss function that includes Dice loss and cross-entropy loss, the segmentation accuracy and training efficiency are effectively balanced. While optimizing the training process, the model is ensured to meet the requirements of the task, resulting in a better performance and more accurate segmentation results for the final medical image segmentation model.

[0147] Step 105: Based on the final loss value, adjust the model parameters of the medical image segmentation model to obtain the target medical image segmentation model.

[0148] In the embodiments of this application, the model parameters of the medical image segmentation model can be adjusted according to the model's loss value to obtain the target medical image segmentation model. The model parameters can be adjusted multiple times until the loss value meets the convergence condition or the number of model parameter adjustments reaches a preset number. The medical image segmentation model obtained after the last model parameter adjustment is then determined as the target medical image segmentation model.

[0149] In embodiments of this application, model parameters may include the parameters of the Dual Feature Aggregation Transformer Block (DFATB) and the Differential Transformer Module of the medical image segmentation model, as well as the connection relationships between them. When adjusting model parameters, the parameters of the Dual Feature Aggregation Transformer Block (DFATB) and the Differential Transformer Module, as well as the connection relationships between the Dual Feature Aggregation Transformer Block (DFATB) and the Differential Transformer Module, may be adjusted, or both parameters and connection relationships may be adjusted simultaneously.

[0150] Optionally, step 105 may include the following sub-steps:

[0151] Sub-step 1051: Based on the final loss value, adjust the model parameters of the medical image segmentation model to obtain the updated medical image segmentation model.

[0152] In the embodiments of this application, the model parameters of the medical image segmentation model can be adjusted once according to the final loss value of the model, thereby obtaining an updated medical image segmentation model.

[0153] Sub-step 1052: Repeat step 101 to obtain a new first medical image.

[0154] Sub-step 1053: If the number of times the model parameters are adjusted is greater than or equal to the first threshold, or if the model loss value meets the convergence condition, stop adjusting the model parameters to obtain the medical image segmentation model.

[0155] In the embodiments of this application, the maximum number of training iterations of the model can be set as a first threshold. If the number of model parameter adjustments is greater than or equal to the first threshold, the model training is considered complete. At this point, adjusting the model parameters can be stopped, and the medical image segmentation model obtained after the last parameter adjustment is used as the target medical image segmentation model. Alternatively, the convergence condition of the medical image segmentation model can be set to a preset number of consecutive occurrences of the final loss value being less than or equal to a second threshold. That is, counting begins when the final loss value of the model is less than or equal to the second threshold. Each time the final loss value is less than or equal to the second threshold, the count is incremented by 1, and each time the final loss value is greater than the second threshold, the count is reset to zero. Thus, when the count reaches the preset number, the model is considered to have converged, model parameter adjustment can be stopped, and the medical image segmentation model obtained after the last parameter adjustment is used as the target medical image segmentation model.

[0156] In the embodiments of this application, the Dual Feature Aggregation Transformer (DFATB) module enhances the model's ability to capture richer contextual information and complex features. By introducing a Differential Transformer, it focuses on key information and suppresses unnecessary noise, thereby improving the model's robustness and generalization ability. Furthermore, the DFATB module is implemented based on a Feature Aggregation Feedforward Network (FAFN). FAFN captures nonlinear spatial information and reduces channel redundancy, thus achieving efficient feature extraction while reducing the model's computational complexity. Therefore, the medical image segmentation model obtained in this invention maintains high accuracy in medical image segmentation while saving computational resources and possessing strong generalization ability, demonstrating its great potential and promising application prospects in the field of medical image segmentation.

[0157] Reference Figure 2 , Figure 2 A flowchart illustrating a medical image segmentation method provided in this application embodiment is shown. The method may include:

[0158] Step 201: Obtain the first medical image of the patient to be tested.

[0159] In the embodiments of this application, the first medical images of the patient to be examined can be obtained by medical imaging examination methods such as abdominal computed tomography (CT) and magnetic resonance imaging (MRI).

[0160] Step 202: Preprocess the first detection medical image to obtain the first detection image.

[0161] In the embodiments of this application, the first detection medical image can be preprocessed. The relevant implementation of the preprocessing is similar to the preprocessing process of the sample images in the training set. Please refer to the content of step 101, which will not be repeated here.

[0162] Step 203: Input the preprocessed first detected image into the target medical image segmentation model and output the first classification category; wherein, the target medical image segmentation model is obtained based on any of the model training methods described above.

[0163] In the embodiments of this application, the preprocessed first detection image is input into the target medical image segmentation model, thereby obtaining the first classification category output by the target medical image segmentation model. The target medical image segmentation model is obtained based on the model training method described above.

[0164] Step 204: Based on the first classification category, identify the lesion area of ​​the patient to be tested.

[0165] In the embodiments of this application, the first classification category can be displayed in the output image of the target medical image segmentation model, and the lesion location can be identified using bright color areas. Based on this, the lesion category of the patient to be detected can be determined according to the first classification category.

[0166] Simulation Case:

[0167] To demonstrate the effectiveness and feasibility of the proposed medical image segmentation method, training and testing were conducted on the Synapse abdominal multi-organ dataset and the ISIC 2018 skin lesion dataset. Comparisons with other methods were performed, and extensive ablation experiments were conducted. The Synapse dataset contains 3779 axial abdominal clinical CT scan images from 30 cases. All images in the dataset are in PNG format with a resolution of 512×512 pixels. Eighteen samples were assigned to the training set, and the remaining 12 samples to the test set. The preprocessing steps in step 101 were used to preprocess all images in the training set, while the images in the test set were the unenhanced raw data. This ensures that the model sees more transformed images during training while maintaining the original image format during validation. Furthermore, the ISIC 2018 skin lesion dataset was used to further evaluate the performance of this method in skin lesion segmentation.

[0168] During the simulation experiments, the medical segmentation model was trained on the training set using the SGD optimizer with a learning rate of 0.05, momentum of 0.9, weight decay of 1e-4, and a batch size of 20. The model was trained for 580 epochs. Following the methods described in steps 301-304, the medical segmentation model was used to conduct medical image segmentation experiments, comparative experiments, and extensive ablation experiments on sample images from the Synapse dataset and the ISIC2018 skin lesion dataset.

[0169] After the experiment, the Dice coefficient and Hausdorff distance were used as evaluation metrics. Specifically:

[0170] The Dice coefficient measures the degree of overlap between segmentation prediction results and actual values, and it is defined as follows:

[0171] ;

[0172] Where X and Y represent the sets of predicted results and true labels, respectively. This represents the intersection operation. This represents the Dice coefficient, which is typically used to quantify the degree of overlap, and its value ranges from 0 to 1.

[0173] Hausdorff distance, which calculates the maximum distance between the predicted result and the true value, is particularly useful for evaluating the quality and accuracy of segmented contours or boundaries. Its definition is as follows:

[0174] ;

[0175] in, A and B For two finite points, This represents the Euclidean distance between two points, with `sup()` and `inf()` representing the minimum upper bound and maximum lower bound, respectively. The Hausdorff distance focuses on the part of the boundary where the predicted boundary is least matched to the actual boundary. This is the Hausdorff distance.

[0176] When evaluating the results, a higher Dice coefficient or a smaller Hausdorff distance indicates a more accurate segmentation.

[0177] The results of the medical image segmentation experiments using the methods described above are shown in Table 1. It demonstrates the performance comparison of the DFAFormer model with 10 other state-of-the-art segmentation models on the Synapse dataset.

[0178] As can be observed from Table 1, compared with previous state-of-the-art methods, DFAFormer achieved the highest Dice score on the spleen, improving the Dice score by 0.43% compared to the previous best method. Furthermore, our method achieved results second only to the best on the gallbladder, left kidney, liver, pancreas, and stomach. It can be seen that the DFAFormer method exhibits good segmentation performance on almost all organs. Overall, compared with previous models, this invention achieves the highest average Dice score of 83.6%. Specifically, compared with the previous best model DAEFormer, this invention improves the Dice score by 0.97% while maintaining approximately the same number of parameters, indicating that our model can obtain more accurate segmentation results.

[0179] Table 1. Comparison of the DFAFomrer model with other segmentation models on the Synapse dataset:

[0180]

[0181] Table 2 presents the comparison results of our method with other state-of-the-art methods on the ISIC 2018 dataset. Compared with previous methods, our model achieves the best results in Dice score, sensitivity, and accuracy. Compared with the previous state-of-the-art model DAEFormer, our model improves Dice score, sensitivity, and accuracy by 0.8%, 1.4%, and 0.55%, respectively. Overall, our DFAFormer achieves superior performance compared to the baseline model while maintaining low computational complexity, demonstrating the good generalization performance and widespread applicability of this invention.

[0182] Table 2. Comparison results of the DFAFomrer model with other segmentation models on the ISIC 2018 dataset:

[0183]

[0184] The model's segmentation results on the Synapse dataset are as follows: Figure 5As shown, the leftmost column is the original input image, the second column is the actual segmentation result, and the rightmost column is the segmentation visualization result of our method. Specifically, in the first example image (row 1), we achieved results closest to the ground truth segmentation image for the gallbladder (green), liver (pink), and spleen (dark blue). Observing the second example image (row 2), we show even more accurate results for the pancreas (yellow) and right kidney (light blue). Overall, our model can achieve smoother results that are closer to the ground truth. Our quantitative evaluation and qualitative visualization results demonstrate that our model can achieve more accurate segmentation, reduces computational complexity to some extent, and has stronger competitiveness and application prospects.

[0185] The visualization results of this model on the ISIC 2018 dataset are as follows: Figure 6 As shown, the leftmost column is the original input image, the second column is the actual segmentation result, and the rightmost column is the segmentation visualization result of our method. From the segmentation results of multiple skin lesions, it can be seen that our model can obtain smoother, more detailed, and more accurate segmentation results. In summary, the segmentation results of this invention have the highest similarity to the actual ground truth, indicating that our proposed DFAFormer model has stronger accuracy, robustness, and generalization ability.

[0186] Ablation experiment:

[0187] To verify the impact of various components in the proposed DFAFormer model on model performance, ablation experiments were conducted on the Synapse dataset. The ablation experiments included: different up / down iterations, performance of the FAFN module and the differential transformer module, and the optimal placement of the differential transformer module in the model.

[0188] 1) Different Up / Down Sampling Orders: Ablation experiments were conducted to evaluate the impact of different up / down sampling orders (or the number of skip connections) on model performance. Three different architectures were tested: 2 up / down sampling orders (1 skip connection), 3 up / down sampling orders (2 skip connections, our model's configuration), and 4 up / down sampling orders (3 skip connections). As shown in Table 3, the results reported in the table are the best results from the current model. The results indicate that more up / down sampling operations do not necessarily lead to better results; appropriate fine-grained information plays a crucial role in feature fusion. The ablation experiment results show that our designed 3-layer encoder-decoder structure has the best performance.

[0189] Table 3. Performance comparison results for different upsampling / downsampling times:

[0190]

[0191] 2) Effectiveness of the FAFN and Differential Transformer Modules: To evaluate the impact of the Differential Transformer module on model performance, we conducted ablation experiments with and without the Differential Transformer. As shown in Table 4, the Differential Transformer effectively denoised the images and improved model performance. Similarly, to evaluate the effectiveness of our designed FAFN module, we also conducted ablation experiments. As shown in Table 4, the newly designed FAFN module significantly improved model performance, and our architecture achieved optimal segmentation results.

[0192] Table 4 Ablation study results of FAFN module and differential transformer module:

[0193]

[0194] 3) Ablation experiments were conducted to compare the impact of different locations of the differential transformer on model performance. These experiments did not involve the FAFN module. The results are shown in Table 5, where the reported results represent the best performance of the current model. As expected, applying the differential transformer before the skip connection cross-attention (SCCA) stage in the decoding phase effectively removes unwanted noise and enhances model performance.

[0195] Table 5. Ablation results at different locations of the differential transformer module:

[0196]

[0197] In summary, through comparative experiments on multiple datasets and ablation experiments on the Synapse dataset, the results show that the proposed DFAFormer model exhibits high generalization and high accuracy. The DFAFormer architecture, while ensuring high accuracy in medical image segmentation, saves computational resources and possesses strong generalization capabilities, demonstrating its great potential and promising application prospects in the field of medical image segmentation.

[0198] Next, this application embodiment also provides a medical image segmentation apparatus, the apparatus may include:

[0199] The acquisition module is used to acquire the first medical images of the patient to be tested.

[0200] The preprocessing module is used to preprocess the first detection medical image to obtain the first detection image;

[0201] The segmentation module is used to input the preprocessed first detected image into the target medical image segmentation model and output the first classification category; wherein, the target medical image segmentation model is obtained based on the above-described model training method;

[0202] The identification module is used to identify the lesion area of ​​the patient to be detected based on the first classification category.

[0203] The medical image segmentation device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a GPU box, mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0204] The medical image segmentation device in this application embodiment can be a device with an operating system. This operating system can be Android, Linux, Windows, or other possible operating systems; this application embodiment does not specifically limit it.

[0205] This application provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the model training method or the medical image segmentation method described in the foregoing embodiments.

[0206] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps in the model training method or the medical image segmentation method disclosed in this application.

[0207] This application also provides a computer program product that, when run on an electronic device, enables a processor to execute steps in the model training method or medical image segmentation method disclosed in this application.

[0208] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0209] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0210] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0211] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0212] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0213] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0214] The above provides a detailed description of the model training method, medical image segmentation method, and apparatus provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model training method, characterized in that, The method includes: The patient's sample images were preprocessed to obtain the first medical image; A medical image segmentation model was established based on the Transformer framework. The first medical image is input into the medical image segmentation model, and the first segmentation mask and the first classification probability corresponding to the first segmentation mask are output. Based on the first segmentation mask and the segmentation label of the first medical image, the final loss value of the medical image segmentation model is determined; Based on the final loss value, the model parameters of the medical image segmentation model are adjusted to obtain the target medical image segmentation model; the target medical image segmentation model is used to identify the lesion area of ​​the patient to be detected based on the first detection medical image of the patient to be detected. The medical image segmentation model includes an overlapping patch embedding module, an encoder module, a decoder module, and a linear projection layer; wherein: The encoder module consists of three dual feature aggregation blocks A distributed from top to bottom; The decoder module consists of three dual-feature aggregation blocks B distributed from bottom to top; A differential transformer module is set between each of two adjacent dual-feature aggregation blocks B; The outputs of each differential transformer module are fused with the first two dual feature aggregation blocks A in the encoder module through skip connections; Both the dual-feature aggregation block A and the dual-feature aggregation block B include: The efficient feature aggregation block first extracts the spatial dependencies of input features by normalizing the key and query through a normalization function, then determines the spatial attention by multiplying the normalized key and value, and obtains the global context vector. Finally, the global context vector is combined with the normalized query to obtain the spatial features. The channel attention feature aggregation block uses a cross-attention mechanism to obtain channel dependencies and determine channel attention, thus obtaining channel features; A feature aggregation feedforward network performs nonlinear transformations on spatial features and channel features respectively to obtain enhanced spatial features and channel features. The feature aggregation feedforward network performs nonlinear transformations on spatial features and channel features respectively to obtain enhanced spatial features and channel features, including: The input feature vector is mapped through a linear projection layer to obtain the first linear projection feature, and then the first linear projection feature is mapped to the second linear projection feature through the GELU activation function. The second linear projection feature is separated into two parts. One part is subjected to depthwise convolution, and the other part is processed by bypass. The output of the depthwise convolution is multiplied element-wise with the result of the bypass processing, and the result of the element-wise multiplication is input into the linear projection layer to output the final feature. The first medical image is input into the medical image segmentation model, which outputs a first segmentation mask and the corresponding first classification probability. The model includes an encoder processing stage and a decoder processing stage, wherein: The encoder processing stage sequentially extracts features from the input first medical image through three dual feature aggregation blocks A, ultimately obtaining the aggregated feature vector of the third sample image; In the decoder processing stage, the feature vector of the third sample image is extracted sequentially through three dual feature aggregation blocks B and two differential transformer modules to obtain the merged feature vector of the fifth sample image. Then, through the linear projection layer, the first segmentation mask and the first classification probability corresponding to each first segmentation mask are output.

2. The method according to claim 1, characterized in that, Based on the first segmentation mask and the segmentation label of the first medical image, the final loss value of the medical image segmentation model is determined, including: The Dice loss value of the medical image segmentation model is determined by measuring the overlap between the first segmentation mask and the segmentation label of the first medical image. Based on the first classification probability and the segmentation label of the first medical image, the cross-entropy loss value of the medical image segmentation model is determined; Based on the Dice loss value and the cross-entropy loss value, the final loss value of the medical image segmentation model is determined.

3. A medical image segmentation method, characterized in that, The method includes: Acquire the first medical images of the patient to be tested; The first detection medical image is preprocessed to obtain the first detection image; The preprocessed first detected image is input into the target medical image segmentation model, and the first classification category is output; wherein, the target medical image segmentation model is obtained based on the model training method of claim 1 or 2; Based on the first classification category, the lesion area of ​​the patient to be tested is identified.

4. A medical image segmentation device, characterized in that, The device includes: The acquisition module is used to acquire the first medical images of the patient to be tested. The preprocessing module is used to preprocess the first detection medical image to obtain the first detection image; The segmentation module is used to input the preprocessed first detected image into the target medical image segmentation model and output a first classification category; wherein the target medical image segmentation model is obtained based on the model training method described in claim 1 or 2. The identification module is used to identify the lesion area of ​​the patient to be detected based on the first classification category.

Images