Model training method, medical image segmentation method and device

Abstract

The method aims at solving the problems that an existing medical image segmentation method is insufficient in global feature capturing, high in calculation complexity and low in generalization performance. The invention belongs to the field of deep learning, provides a model training method, a medical image segmentation method and a medical image segmentation device, and designs a dual feature aggregation Transform module (DFATB) based on a feature aggregation feed-forward network (FAFN). The capability of capturing richer context information and complex features by the model is enhanced; and an FAFN module is designed, and a gating mechanism is introduced to capture nonlinear spatial information and reduce channel redundancy, so that the calculation complexity of the model is reduced while efficient feature extraction is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and in particular to a model training method and a medical image segmentation method and device. BACKGROUND

[0002] A medical image is visualized data of internal structure and function information of a human body obtained by using a medical imaging technique, and contains images of multiple organs in an abdomen, images of skin lesions, and the like. Accurate and efficient medical image segmentation can identify a lesion region, and plays a vital role in disease prevention and diagnosis, disease analysis and evaluation, and individualized medical scheme making.

[0003] In the related art, deep learning methods are deeply integrated with medical image analysis for medical image segmentation and have achieved good results. These methods can be roughly divided into three categories: first, convolutional neural networks (CNNs). CNNs have become the basis for medical image segmentation due to their inherent bias and are widely used in processing medical image segmentation tasks, such as U-Net, U-Net++, V-Net, etc. However, due to the limitations of local receptive fields, traditional CNN models often face the challenge of insufficient spatial context modeling when processing medical image problems, especially when dealing with complex medical image analysis tasks such as abdominal multi-organ segmentation. Second, hybrid models of CNN and Transformer networks. Transformer can effectively capture long-range dependencies in input data due to its unique self-attention mechanism, thus achieving accurate modeling of global context. This makes up for the lack of spatial context modeling ability of traditional CNNs when processing medical images due to the limitations of local receptive fields. Hybrid CNN and Transformer models can extract detailed local features and better capture global information, thus enhancing the understanding, analysis, and processing of complex medical images. For example, the ContextNet architecture uses context information to supplement the deficiencies in local feature representation, improving the model's accuracy in medical image segmentation tasks. However, this type of model still relies on CNN backbone networks, and when dealing with unfamiliar medical data, it still has the problem of low generalization ability. Third, pure Transformer models. To make up for the lack of generalization ability of CNN-Transformer models, pure Transformer-based models have emerged. For example, Swin-UNet is a new method using a pure Transformer architecture, which adopts a similar structure to U-Net and combines skip connections to enhance feature representation. However, despite Swin-UNet's attempt to reduce computational complexity through consecutive Transformer blocks and sliding window settings, it still has limitations in effectively handling spatial context, especially in retaining global information while effectively focusing on local details. Therefore, Transformer-based models can capture global features and improve model generalization performance, but they rely on relatively heavy Transformer models, which have high computational complexity, limiting the model's development and widespread application in actual medical image analysis problems.

[0004] In summary, although the three types of models mentioned above have made breakthrough progress, especially the mixed CNN-Transformer model and the pure Transformer model have shown good performance, the models still face the problems of insufficient generalization performance and high computational complexity. Especially when dealing with medical image segmentation, which is a relatively complex task, these limitations are particularly evident, leading to the possibility of misdiagnosis when doctors diagnose based on medical image segmentation results, resulting in low accuracy of the examination method. SUMMARY

[0005] In view of the above problems, the embodiments of the present application provide a model training method and a medical image segmentation method and device to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, the embodiments of the present application provide a model training method, which comprises: preprocessing a sample image of a patient to obtain a first medical image; establishing a medical image segmentation model based on a Transformer framework; inputting the first medical image into the medical image segmentation model and outputting a first segmentation mask and a first classification probability corresponding to the first segmentation mask; determining a final loss value of the medical image segmentation model based on the first segmentation mask and a segmentation label of the first medical image; adjusting model parameters of the medical image segmentation model based on the final loss value to obtain a target medical image segmentation model; the target medical image segmentation model is used to identify a lesion area of a to-be-detected patient based on a first detection medical image of the to-be-detected patient.

[0007] Optionally, the medical image segmentation model comprises an overlapping patch embedding module, an encoder module, a decoder module and a linear projection layer; wherein: the encoder module comprises three double feature aggregation blocks A distributed from top to bottom; the decoder module comprises three double feature aggregation blocks B distributed from bottom to top; one difference transformer module is arranged between each two adjacent double feature aggregation blocks B; the outputs of each difference transformer module are respectively fused with the features of the first two double feature aggregation blocks A in the encoder module through a skip connection.

[0008] Optionally, the double feature aggregation block A and the double feature aggregation block B each comprise: The high-efficiency feature aggregation block first extracts the spatial dependency of the input features through a normalization function, then determines the spatial attention by multiplying the normalized key and value, and obtains a global context vector, and finally obtains the spatial features by multiplying the global context vector with the normalized query; The channel attention feature aggregation block adopts a cross-attention mechanism to obtain channel dependency and determine channel attention, and obtains channel features. The feature aggregation feedforward network respectively performs nonlinear transformation on the spatial features and the channel features to obtain enhanced spatial features and channel features.

[0009] Optionally, the feature aggregation feedforward network respectively performs nonlinear transformation on the spatial features and the channel features to obtain enhanced spatial features and channel features, including: The input feature vector is mapped through a linear projection layer to obtain first linear projection features, and then the first linear projection features are mapped to second linear projection features through a GELU activation function. The second linear projection features are separated into two parts, one part is subjected to deep convolution, and the other part is subjected to bypass processing, and the results of deep convolution and bypass processing are multiplied element by element, and the element multiplication result is input to a linear projection layer to output the final features.

[0010] Optionally, the first medical image is input into the medical image segmentation model, and the first segmentation mask and the first classification probability corresponding to the first segmentation mask are output, including an encoder processing stage and a decoder processing stage, wherein: The encoder processing stage sequentially extracts features from the input first medical image through three double feature aggregation blocks A, and finally obtains a third sample image aggregation feature vector; The decoder processing stage sequentially extracts features from the third sample image aggregation feature vector through three double feature aggregation blocks B and two difference transformer modules to obtain a fifth sample image merging feature vector, and outputs the first segmentation mask and the first classification probability corresponding to each first segmentation mask through a linear projection layer.

[0011] Optionally, based on the first segmentation mask and the segmentation label of the first medical image, the final loss value of the medical image segmentation model is determined, including: The Dice loss value of the medical image segmentation model is determined by measuring the overlap between the first segmentation mask and the segmentation label of the first medical image; The cross-entropy loss value of the medical image segmentation model is determined based on the first classification probability and the segmentation label of the first medical image; The final loss value of the medical image segmentation model is determined based on the Dice loss value and the cross-entropy loss value.

[0012] In a second aspect, the embodiments of the present application provide a medical image segmentation method, the method comprising: obtaining a first detection medical image of a patient to be detected; preprocessing the first detection medical image to obtain a first detection image; inputting the preprocessed first detection medical image into a target medical image segmentation model, and outputting a first classification category; wherein the target medical image segmentation model is obtained based on the model training method described above; based on the first classification category, identifying a lesion area of the patient to be detected.

[0013] In a third aspect, the embodiments of the present application provide a medical image segmentation device, the device comprising: an acquisition module configured to acquire a first detection medical image of a patient to be detected; a preprocessing module configured to preprocess the first detection medical image to obtain a first detection image; a segmentation module configured to input the preprocessed first detection medical image into a target medical image segmentation model, and output a first classification category; wherein the target medical image segmentation model is obtained based on the model training method described above; an identification module configured to identify a lesion area of the patient to be detected based on the first classification category.

[0014] Specific beneficial effects are as follows: First, the present application is different from the traditional network structure which relies on a specific data set and has high computational complexity, but adopts a more general and lighter U-shaped dual feature aggregation pure transformer architecture based on differential transformer, which can efficiently capture complex features. It provides a more general and universal processing method for medical image segmentation tasks, and has good potential and application prospect in actual medical problems.

[0015] Second, the present application proposes a feature aggregation feedforward neural network (FAFN), which is based on a nonlinear gating mechanism, efficiently aggregates features and reduces information redundancy. Thus, the computational complexity is reduced, and a more lightweight option is provided.

[0016] Third, the present application proposes a dual feature aggregation transformer block (DFATB), which can capture more complex and comprehensive features by combining spatial aggregation and channel aggregation attention mechanisms, thereby improving the feature aggregation capability and improving the medical image segmentation accuracy.

[0017] Fourthly, the differential transformer module is introduced, which can reduce the attention to unnecessary information and focus on key information. By introducing the module, the regional features and edge information in the medical image can be captured, the segmentation accuracy is improved, and the model can be widely applied to diversified medical image datasets. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is a flow diagram of a model training method provided by an embodiment of the present application; Figure 2 is a flow diagram of a medical image segmentation method provided by an embodiment of the present application; Figure 3 is a DFAFormer model architecture diagram provided by an embodiment of the present application; wherein, Figure 3 (a) in the above figure is a structure diagram of a feature aggregation feedforward neural network, Figure 3 (b) in the above figure is a structure diagram of a double feature aggregation transformer block, Figure 3 (c) in the above figure is a structure diagram of a differential transformer; Figure 4 is a standard attention and differential attention structure comparison diagram provided by an embodiment of the present application; wherein, Figure 4 (a) in the above figure is a standard attention mechanism, Figure 4 (b) in the above figure is a differential attention mechanism; Figure 5 is a visual comparison result of the DFAFormer model and other advanced segmentation models on the Synapse dataset provided by an embodiment of the present application; Figure 6 is a visual comparison result of the DFAFormer model and other advanced segmentation models on the ISIC 2018 dataset provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood, and so that the scope of the present application can be conveyed to those skilled in the art.

[0021] Referring to Figure 1 , Figure 1 A flowchart of a model training method provided by an embodiment of the present application is shown in FIG. 1. The method can include the following steps. Step 101, pre-processing a sample image of a patient to obtain a first medical image.

[0022] The sample image of the patient generally refers to a medical image obtained by using advanced imaging technologies such as abdominal computed tomography (CT) and magnetic resonance imaging (MRI).

[0023] Next, all the obtained sample images are labeled in the lesion region category to obtain segmentation labels of the sample images, which can include liver, left kidney, right kidney, pancreas, spleen, stomach, aorta, and gallbladder. Then, the segmentation labels can be converted into One-hot encoding format, which allows the segmentation label of each pixel to be represented by a vector, where each position corresponds to a class.

[0024] After the sample image is labeled, it is divided into a training set and a test set in proportion, N samples in the training set, and the remaining M samples in the test set. Random augmentation operations are performed on the sample images and segmentation labels in the training set, including random flipping, rotation, and contrast adjustment, etc., to ensure the diversity of the images and the robustness of the model during the training process. Moreover, the enhancement operation should be performed synchronously on the sample image and its corresponding segmentation label to avoid misplacement of the segmentation label. Meanwhile, the test set is the original data without random augmentation operation, so as to ensure that the model can see more transformed images during the training process, while maintaining the original form of the images during the verification process.

[0025] Finally, all the sample images and segmentation labels in the training set and the test set are uniformly adjusted to a size of 224x224. When adjusting, the image is scaled using cubic interpolation, and the label is scaled using nearest neighbor interpolation to avoid smoothing between labels.

[0026] After the above pre-processing, the first medical image obtained after pre-processing of the sample image can be obtained.

[0027] Step 102, establishing a medical image segmentation model based on a Transformer framework.

[0028] In the embodiment of the present application, the medical image segmentation model is a U-shaped dual feature aggregation pure Transformer model, referred to as the DFAFormer model. The DFAFormer model adopts a U-shaped encoder-decoder architecture and is a multi-layer pure Transformer structure, such as Figure 3 shown.

[0029] Mainstream medical image segmentation methods often adopt a U-shaped encoder-decoder structure. The present invention improves on this traditional method by designing three identical dual feature aggregation Transformer blocks (DFATBs) in the encoder module and decoder module, respectively. At the same time, a differential transformer module is introduced in the decoding stage. The output of the differential transformer module is fused with the features of its parallel encoding stage through skip connection cross attention (SCCA) to construct the DFAFormer model.

[0030] Combine Figure 3 (a)-(c), for an input image x∈R with a given spatial dimension of H×W and channels of C H×W×C The processing flow of the DFAFormer model includes: using the overlap patch embedding module to extract 4×4 patch tokens from the input image, and then embedding the extracted patch tokens (denoted as x∈R n×d ) is passed to the encoder module, which consists of three stacked dual feature aggregation Transformer blocks (DFATBs). Each DFATB block consists of two consecutive feature aggregation Transformer layers and a patch merging module. During patch merging, 2×2 patch tokens are aggregated, reducing the spatial dimension while doubling the channel dimension. This layered merging enables the model to capture multi-scale representations while preserving fine-grained and global features.

[0031] During the decoding phase, the patch tokens are expanded by a factor of 2 in each block. The image is then denoised using a differential transformer module. Following this, skip connection cross attention (SCCA) fuses the output x1 of each differential transformer module with the parallel output features x2 from the encoding phase via skip connections. The model incorporates two skip connections to ensure the fusion of feature streams across different scales. The fused features are then processed by the DFATB module. Finally, the model generates a segmentation mask through a linear projection layer.

[0032] The encoder module in the DFAFormer model includes three dual feature aggregation blocks A from top to bottom, and the decoder module includes three dual feature aggregation blocks B from bottom to top. The dual feature aggregation blocks A and B are the dual feature aggregation Transformer blocks (DFATB). The workflow of the DFAFormer model designed by the present invention includes: 1) Encoder stage: First, the input first medical image is processed by the overlapping patch embedding module to obtain patch tokens, i.e., the first sample segmentation image; Input the first sample segmentation image into the first dual feature aggregation block A to obtain the first sample image aggregation feature vector; Downsampling the first sample image aggregated feature vector and inputting the result into a patch merging module to obtain a first sample image merged feature vector; Inputting the merged feature vector of the first sample image into the second dual feature aggregation block A to obtain the aggregated feature vector of the second sample image; Downsampling the aggregated feature vector of the second sample image and inputting the result into a patch merging module to obtain a merged feature vector of the second sample image; Inputting the merged feature vector of the second sample image into the third dual feature aggregation block A to obtain a third sample image aggregated feature vector; 2) Decoder stage: The aggregated feature vector of the third sample image is input into the decoder module, and the aggregated feature vector of the fourth sample image is obtained after passing through the first dual feature aggregation block B; Upsampling the aggregated feature vector of the fourth sample image and inputting it into the patch merging module to obtain a merged feature vector of the third sample image; The third sample image is merged with the feature vector and upsampled, and then input into the difference transformer module to obtain the first difference feature vector; Performing a jump connection between the aggregated feature vector of the second sample image and the first differential feature vector to obtain a first fusion feature; Input the first fused feature into the second dual feature aggregation block B to obtain a fifth sample image aggregated feature vector; Up-sampling the aggregated feature vector of the fifth sample image and inputting it into the patch merging module to obtain a merged feature vector of the fourth sample image; The fourth sample image is merged with the feature vector and up-sampled, and then input into the difference transformer module to obtain a second difference feature vector; Performing a jump connection between the aggregated feature vector of the first sample image and the second differential feature vector to obtain a second fusion feature; Input the second fused feature into the third dual feature aggregation block B to obtain the sixth sample image aggregated feature vector; Upsampling the aggregated feature vector of the sixth sample image and inputting it into the patch merging module to obtain a merged feature vector of the fifth sample image; Inputting the fifth sample image merged feature vector into the linear projection layer, outputting the first segmentation mask and the first classification probability corresponding to each first segmentation mask; Optionally, the Dual Feature Aggregation Transformer Module (DFATB) is implemented based on the Feature Aggregation Feedforward Network (FAFN). The feature aggregation feedforward network is implemented using a nonlinear gating mechanism, and its architecture is as follows: Figure 3 As shown in (a), the feature aggregation-based feedforward network includes deep convolution operations and element-wise multiplication. The deep convolution operation can convolve the input image to extract image features. At the same time, the input image is multiplied element by element through element-wise multiplication, and finally the linear projection layer is used to output the final features. In the embodiment of the present application, given input X∈R H×W×C , the calculation formula of FAFN is as follows: (Formula 1) (Equation 2) (Equation 3) in, and represents the linear projection weight, represents the convolution parameters that can be learned along the depth direction, and Represents the separated images, and both are in R H×W×C’ In the set, C' represents the hidden dimension in FAFN, represents the activation function, Represents the feature map after being processed by the GELU activation function.

[0033] Combine Figure 3 In (a), the workflow of the feature aggregation feedforward network is as follows: the input feature vector is mapped to a higher-dimensional space through a linear projection layer to obtain a first linear projection feature, and then the nonlinear representation capability of the model is increased through the GELU activation function, and the first linear projection feature is mapped back to the original dimension or the required output dimension to obtain a second linear projection feature, and then the second linear projection feature is separated into two parts, one part of which is subjected to depth convolution and the other part is subjected to bypass processing, and the result of the depth convolution output is multiplied by the result of the bypass processing element by element, and finally, the result of the element multiplication is output as the final feature through the linear projection layer.

[0034] Compared with traditional feedforward neural network (FFN), the application can capture nonlinear spatial information while reducing channel redundancy by designing feature aggregation feedforward network (FAFN). And it can reduce the computational complexity and improve the overall computational efficiency through deep convolution operation.

[0035] Optionally, as shown in (b) of the figure, Figure 3 The double feature aggregation Transformer module (DFATB) based on the feature aggregation feedforward network includes two key modules: an efficient feature aggregation Transformer block (EFATB) and a channel attention feature aggregation Transformer block (CFATB). The EFATB focuses on spatial aggregation through an efficient attention mechanism, while the CFATB focuses on channel information aggregation. The EFATB and CFATB together promote inter-block feature aggregation across spatial and channel dimensions, enhancing the model's ability to capture complex features at multiple levels.

[0036] The efficient feature aggregation Transformer block (EFATB) is a lightweight spatial attention module that captures spatial dependencies in input features while maintaining computational efficiency. Its structure is shown in (b) of the figure. Figure 3 The efficient feature aggregation Transformer block can solve the problem of high computational complexity (O(N 2 ) ) of traditional self-attention mechanisms, reducing the computational burden.

[0037] Given an input feature x∈R H×W×C , the operation of the EFATB is as follows: First, the efficient feature aggregation Transformer block extracts the spatial dependency of the input feature by normalizing the key and query. Then, it calculates the attention by multiplying the normalized key and value, and obtains the global context vector. Finally, it multiplies the global context vector with the normalized query to generate a new feature representation.

[0038] The calculation process of the EFATB can be represented by the following formula: (Formula 4) Where, and represent the normalization function applied to the query and key, respectively; represents the query, represents the key, represents the value, T represents the transpose; represents the generated spatial feature; The attention output is passed through residual connections and layer normalization to ensure stable feature propagation. Subsequently, FAFN performs nonlinear transformation on the features through formulas (1)-(3) to further enhance the spatial representation.

[0039] Therefore, the final output of EFATB is: (Formula 5) (Equation 6) in, represents the input features, Represent the query, key, and value calculated from the input feature X respectively; Represents the feature aggregation feedforward network (FAFN) in the EFATB module; In addition, to capture channel dependencies and further improve the robustness and generalization of the model, we designed a Channel Feature Aggregation Transformer Block (CFATB). CFATB uses channel attention to capture channel dependencies. Specifically, CFATB first transposes the key and query, and then generates channel attention weights based on the cross-covariance matrix. The calculation process is as follows: (Equation 7) (Equation 8) in, is the context vector obtained by transposing the attention, This is to ensure stable training parameters. is the generated channel feature.

[0040] After calculating the channel attention, residual connections and layer normalization are applied. Finally, FAFN is used to perform nonlinear modeling on the channel features to reduce redundant information and enhance the representation of meaningful features. The output of CFATB is represented as: (Equation 9) (Equation 10) (Equation 11) in, Represent the query, key, and value that will be input into the channel feature aggregation transformer block (CFATB) respectively; Represents the feature aggregation feedforward network (FAFN) in the CFATB module; As can be seen from the above, the dual feature aggregation Transformer module in the present invention performs spatial and channel aggregation through EFATB and CFATB respectively, thereby achieving the purpose of dual feature aggregation. In the DFATB design, EFATB and CFATB both achieve robust feature modeling in both spatial and channel dimensions. This dual attention mechanism improves the model's ability to represent complex feature relationships and reduces computational complexity to a certain extent.

[0041] In summary, the dual feature aggregation Transformer module enhances the model's ability to capture richer contextual information and complex features by integrating spatial aggregation attention and channel aggregation attention mechanisms.

[0042] Optionally, the differential transformer from the field of natural language processing, namely the differential attention mechanism, is introduced into the DFAFormer model, such as Figure 4 As shown, Figure 4 (a) is the standard attention mechanism, and (b) is the differential attention mechanism.

[0043] Compared to traditional Transformers, the Differential Transformer focuses more on similarities during attention calculations and emphasizes differences between features through differential operations. This approach effectively removes unnecessary noise and improves the model's ability to capture relevant features. The key to the Differential Transformer lies in splitting the attention heads into two groups and then processing their outputs using differential operations.

[0044] The specific calculation process of the differential transformer can be expressed by the following formula: (Equation 12) (Equation 13) (Equation 14) (Equation 15) (Equation 16) (Equation 17) in, is a learnable scalar parameter; 、 、 are the weight matrices for query, key, and value respectively; is the input feature; 、 These are the two attention outputs calculated by the flash_attention function; Represents a split operation, which divides the vector obtained by multiplying the input feature X by the corresponding weight matrix into two parts, which are used to calculate two different attention outputs A1 and A2 respectively.

[0045] Because the differential transformer emphasizes feature differences through differential operations, it enhances the model's ability to capture context and key information. Furthermore, combined with FlashAttention technology, the differential transformer significantly reduces computational complexity while maintaining performance. This paper uses the differential transformer in the decoder stage, further improving the model's robustness and generalization performance.

[0046] Step 103: Input the first medical image into a medical image segmentation model, and output a first segmentation mask and a first classification probability corresponding to the first segmentation mask.

[0047] The first medical image obtained after preprocessing is input into a medical image segmentation model, thereby obtaining a first segmentation mask and a first classification probability corresponding to the first segmentation mask output by the medical image segmentation model. The first segmentation mask represents a predicted mask of the lesion region. The first classification probability represents a confidence level of the first segmentation mask.

[0048] Step 104 : determining a final loss value of the medical image segmentation model based on the first segmentation mask and the segmentation label of the first medical image.

[0049] In an embodiment of the present application, a loss value of the medical image segmentation model may be calculated based on the first segmentation mask and the segmentation label of the first medical image. The loss value may include, but is not limited to, a Dice loss value and a cross entropy loss (CE), or a combination of these loss values.

[0050] Optionally, step 104 may include: Sub-step 1041 , determining a Dice loss value of the medical image segmentation model by measuring the overlap between the first segmentation mask and the segmentation label of the first medical image.

[0051] In an embodiment of the present application, Dice loss can be used to quantify the overlapping area between the predicted results and the actual annotations, which is particularly suitable for multi-organ segmentation with significant size differences and shape changes, solving the problem of data imbalance.

[0052] The calculation method of Dice loss value is as follows: (Equation 18) in, represents the Dice loss value, Represents a very small number, represents the pixel value of the first segmentation mask, denoting the pixel values ​​of the segmentation labels of the first medical image, Represents element-wise multiplication operation.

[0053] Sub-step 1042 : determining a cross entropy loss value of the medical image segmentation model based on the first classification probability and the segmentation label of the first medical image.

[0054] In the embodiment of the present application, the cross entropy loss value is calculated as follows: (Equation 19) in, represents the cross entropy loss value, represents the weight of the positive sample, Indicates the i The first classification probability corresponding to the first segmentation mask, Indicates the i The segmentation labels of the first medical image corresponding to the first segmentation masks, N represents the total number of first segmentation masks, and log() represents the logarithmic function. to balance the importance of different categories.

[0055] The Dice loss is more suitable for directly optimizing the similarity between the predicted segmented image and the true label. It is particularly suitable for tasks such as abdominal multi-organ segmentation, where objects vary in size, have complex shapes, and are unevenly distributed. On the other hand, the cross-entropy loss focuses more on improving the model's performance in multi-class classification. Therefore, this paper combines the two to determine the final loss value of the medical image segmentation model.

[0056] Sub-step 1043 , determining a final loss value of the medical image segmentation model based on the Dice loss value and the cross entropy loss value.

[0057] In the embodiment of the present application, the weighted sum of the Dice loss value and the cross entropy loss value is used as the final loss value. The calculation method of the final loss value is shown in the following formula: (Equation 20) in, is the final loss value of the medical image segmentation model, is the weight of the Dice loss value, is the Dice loss value, is the weight of the cross entropy loss value, is the cross entropy loss value, and ; In an embodiment of the present application, by using a combined loss function including Dice loss and cross entropy loss, the segmentation accuracy and training efficiency are effectively balanced, and the model is ensured to meet the requirements of the task while optimizing the training process, so that the final medical image segmentation model has better performance and more accurate segmentation results.

[0058] Step 105: Based on the final loss value, adjust the model parameters of the medical image segmentation model to obtain a target medical image segmentation model.

[0059] In an embodiment of the present application, model parameters of a medical image segmentation model can be adjusted based on the model loss value, thereby obtaining a target medical image segmentation model. The model parameters can be adjusted multiple times until the loss value meets a convergence condition or the number of model parameter adjustments reaches a preset number. The medical image segmentation model obtained after the last model parameter adjustment is determined as the target medical image segmentation model.

[0060] In an embodiment of the present application, model parameters may include parameters of a dual feature aggregation transformer block (DFATB) and a differential transformer module of a medical image segmentation model, as well as the connection between the two. When adjusting the model parameters, the parameters of the dual feature aggregation transformer block (DFATB) and the differential transformer module may be adjusted, the connection between the dual feature aggregation transformer block (DFATB) and the differential transformer module may be adjusted, or both the parameters and the connection may be adjusted simultaneously.

[0061] Optionally, step 105 may include the following sub-steps: Sub-step 1051: Based on the final loss value, adjust the model parameters of the medical image segmentation model to obtain an updated medical image segmentation model.

[0062] In an embodiment of the present application, the model parameters of the medical image segmentation model can be adjusted once according to the final loss value of the model, so that an updated medical image segmentation model can be obtained.

[0063] Sub-step 1052 , re-execute step 101 to obtain a new first medical image.

[0064] Sub-step 1053, when the number of adjustments of the model parameters is greater than or equal to a first threshold, or the model loss value satisfies a convergence condition, stop adjusting the model parameters to obtain a medical image segmentation model.

[0065] In an embodiment of the present application, the maximum number of training times of the model can be set to a first threshold value. If the number of times the model parameters are adjusted is greater than or equal to the first threshold value, the model training can be considered complete. At this time, the adjustment of the model parameters can be stopped, and the medical image segmentation model obtained after the last parameter adjustment is used as the target medical image segmentation model. The convergence condition of the medical image segmentation model can also be set as the number of times the final loss value is less than or equal to the second threshold value and reaches a preset number of consecutive occurrences, that is, when the final loss value of the model is less than or equal to the second threshold value, the counting starts, and each time the final loss value is less than or equal to the second threshold value, the count value is increased by 1, and each time the final loss value is greater than the second threshold value, the count value is reset. Therefore, when the count value reaches the preset number of times, the model is considered to have converged, the adjustment of the model parameters can be stopped, and the medical image segmentation model obtained after the last parameter adjustment is used as the target medical image segmentation model.

[0066] In the embodiments of the present application, due to the design of the dual feature aggregation Transformer module (DFATB), the model's ability to capture richer contextual information and complex features can be enhanced, and by introducing the differential Transformer, it focuses on key information and suppresses unnecessary noise, thereby improving the robustness and generalization ability of the model. In addition, when implementing the dual feature aggregation Transformer module (DFATB), it is based on the feature aggregation feedforward network (FAFN), which can capture nonlinear spatial information and reduce channel redundancy, thereby achieving efficient feature extraction while reducing the computational complexity of the model. Therefore, the medical image segmentation model obtained by the present invention saves computing resources while ensuring high-accuracy medical image segmentation capabilities, and has strong generalization capabilities, showing its great potential and good application prospects in the field of medical image segmentation.

[0067] Reference Figure 2 , Figure 2 A schematic flow chart of a medical image segmentation method provided in an embodiment of the present application, the method may include: Step 201: Acquire a first medical image of a patient to be detected.

[0068] In an embodiment of the present application, a first detection medical image of the patient to be detected may be acquired through medical imaging examination methods such as abdominal computed tomography (CT) and magnetic resonance imaging (MRI).

[0069] Step 202: pre-process the first detection medical image to obtain a first detection image.

[0070] In the embodiments of the present application, the first detection medical image can be preprocessed, and the related implementation content of the preprocessing is similar to the preprocessing process of the sample images in the training set. For details, please refer to the content of step 101, which will not be repeated here.

[0071] Step 203, input the preprocessed first detection image into the target medical image segmentation model, and output the first classification category; wherein the target medical image segmentation model is obtained based on the model training method as described above.

[0072] In the embodiments of the present application, the preprocessed first detection image is input into the target medical image segmentation model, so that the first classification category output by the target medical image segmentation model can be obtained. The target medical image segmentation model is obtained based on the model training method described above.

[0073] Step 204, identifying the lesion area of the patient to be detected based on the first classification category.

[0074] In the embodiments of the present application, the first classification category can be displayed in the output image of the target medical image segmentation model, and the lesion position can be identified by using a bright color area. On this basis, the lesion category of the patient to be detected can be determined according to the first classification category.

[0075] Simulation case: In order to prove the effectiveness and feasibility of the medical image segmentation method proposed in the present application, training and testing are carried out on the Synapse abdominal multi-organ data set and the ISIC 2018 skin lesion data set, and comparison is made with other methods, and extensive ablation experiments are carried out. Among them, the Synapse data set contains 30 cases of 3779 axial abdominal clinical CT scan images, and all the images in the data set are in PNG format with a pixel of 512x512. Among them, 18 samples are divided into a training set, and the other 12 samples are divided into a test set. The pre-processing step in step 101 is used to preprocess all the images in the training set, and the images in the test set are original data without enhancement. This ensures that the model can see more transformed images during the training process, while maintaining the original form of the images during the validation process. In addition, the ISIC 2018 skin lesion data set is also used to further evaluate the performance of the method in skin lesion segmentation.

[0076] In the process of simulation experiment, when training the medical segmentation model, it is performed on the training set, using the SGD optimizer, the learning rate is 0.05, the momentum is 0.9, the weight decay is 1e-4, the batch size is 20, and the training number of the model reaches 580 rounds. According to the method of step 301-step 304, the medical segmentation model is used to perform medical image segmentation experiment, comparison experiment and extensive ablation experiment on sample images in Synapse dataset and ISIC2018 skin lesion dataset respectively.

[0077] After the experiment, Dice coefficient and Hausdorff distance are used as evaluation indexes. Among them: The Dice coefficient measures the overlap between the segmentation prediction result and the true value, which is defined as follows: ; Where X and Y represent the set of prediction results and true labels respectively, represents the intersection operation, represents the Dice coefficient, which is usually used to quantify the overlap, and its value ranges from 0 to 1.

[0078] Hausdorff distance calculates the maximum distance between the prediction result and the true value, which is particularly suitable for evaluating the quality and accuracy of segmentation contours or boundaries, and is defined as follows: ; Where, A and B are two finite points, represents the Euclidean distance between two points, sup() and inf() represent the minimum upper bound and maximum lower bound respectively. Hausdorff distance focuses on the least matching part between the predicted boundary and the actual boundary, is the Hausdorff distance.

[0079] When evaluating, the higher the Dice coefficient or the smaller the Hausdorff distance, the more accurate the segmentation result.

[0080] The results of the medical image segmentation experiment by the above method are shown in Table 1. It shows the performance comparison of DFAFormer model and other 10 advanced segmentation models on Synapse dataset.

[0081] From Table 1, it can be observed that DFAFormer achieves the highest Dice score on Spleen compared to previous advanced methods, with an improvement of 0.43% over the previous best method. In addition, our method achieves results second only to the best on Gallbladder, Left Kidney, Liver, Pancreas, and Stomach. It can be seen that the DFAFormer method performs well on almost all organs. Overall, compared to previous models, the present application achieves the highest average Dice score of 83.6%. Specifically, compared to the previous optimal model DAEFormer, the present application improves the Dice score by 0.97% while maintaining approximately the same number of parameters, indicating that our model can achieve more accurate segmentation results.

[0082] Table 1 Comparison results of DFAFomrer model with other segmentation models on Synapse dataset

[0083] Table 2 shows the comparison results of the segmentation performance of our method and other advanced methods on the ISIC 2018 dataset. Compared with previous methods, our model achieves the best results in Dice score, sensitivity, and accuracy. Compared with the previous optimal model DAEFormer, our model improves the Dice score, sensitivity, and accuracy by 0.8%, 1.4%, and 0.55%, respectively. Overall, our DFAFormer achieves better performance than the baseline model while maintaining low computational complexity, indicating that the present application has good generalization performance and universal applicability.

[0084] Table 2 Comparison results of DFAFomrer model with other segmentation models on ISIC 2018 dataset

[0085] The visual segmentation results of the model on the Synapse dataset are as follows Figure 5As shown in the figure, the leftmost first column is the original input image, the second column is the true segmentation result, and the rightmost column is the segmentation visualization result of the method. Specifically, in the first example image (row 1), we achieve the closest result to the true segmentation image on the gallbladder (green), liver (pink) and spleen (dark blue). Observe the second example image (row 2), we show more accurate results on the pancreas (yellow) and right kidney (light blue). Overall, the model can achieve smoother and more ground truth results. The quantitative evaluation and qualitative visualization results of the model show that the model can achieve more accurate segmentation, reduce the computational complexity to a certain extent, and have stronger competitiveness and application prospect.

[0086] The visualization segmentation results of the model on the ISIC 2018 dataset are as shown in the figure Figure 6 As shown in the figure, the leftmost first column is the original input image, the second column is the true segmentation result, and the rightmost column is the segmentation visualization result of the method. Specifically, in the first example image (row 1), we achieve the closest result to the true segmentation image on the gallbladder (green), liver (pink) and spleen (dark blue). Observe the second example image (row 2), we show more accurate results on the pancreas (yellow) and right kidney (light blue). Overall, the model can achieve smoother and more ground truth results. The quantitative evaluation and qualitative visualization results of the model show that the model can achieve more accurate segmentation, reduce the computational complexity to a certain extent, and have stronger competitiveness and application prospect.

[0087] Ablation experiment: In order to verify the influence of various components in the DFAFormer model proposed by the application on the performance of the model, an ablation experiment is carried out on the Synapse dataset. The ablation experiment includes: the performance of different upsampling / downsampling times, FAFN module and differential transformer module, and the best position of the differential transformer module in the model.

[0088] 1) Different upsampling / downsampling times: The influence of different upsampling / downsampling times (or the number of skip connections) on the performance of the model is evaluated through the ablation experiment. The application experiments three different architectures: 2 times of upsampling / downsampling (1 time of skip connection), 3 times of upsampling / downsampling (2 times of skip connection, the configuration of our model) and 4 times of upsampling / downsampling (3 times of skip connection). As shown in Table 3, the results reported in the table are taken from the best results of the current model. The research results show that more upsampling / downsampling operations do not necessarily bring better results, and appropriate fine-grained information plays a crucial role in feature fusion. The results of the ablation experiment show that the 3-layer encoder-decoder structure designed by us has the best performance.

[0089] Table 3 Performance comparison results of different upsampling / downsampling times:

[0090] 2) Effectiveness of the FAFN and Differential Transformer Modules: To evaluate the impact of the Differential Transformer module on model performance, we conducted ablation experiments with and without the Differential Transformer. As shown in Table 4, the Differential Transformer effectively denoised the image and improved model performance. Similarly, to evaluate the effectiveness of our designed FAFN module, we also conducted ablation experiments. Table 4 shows that the newly designed FAFN module significantly improved model performance, and our architecture also achieved optimal segmentation results.

[0091] Table 4 Ablation study results of FAFN module and differential transformer module:

[0092] 3) Ablation experiments were conducted to compare the impact of different placements of the differential transformer on model performance. These experiments did not involve the FAFN module. The results are shown in Table 5, where the reported results are the best results of the current model. As expected, applying the differential transformer before the skip connection cross attention (SCCA) in the decoding stage effectively removes unnecessary noise and enhances model performance.

[0093] Table 5 Ablation study results of different positions of the differential transformer module:

[0094] In summary, comparative experiments on multiple datasets and ablation experiments on the Synapse dataset demonstrate that the proposed DFAFormer model demonstrates high generalization and accuracy. While maintaining high-accuracy medical image segmentation, the DFAFormer architecture conserves computational resources and exhibits strong generalization capabilities, demonstrating its significant potential and promising application prospects in the field of medical image segmentation.

[0095] Next, an embodiment of the present application further provides a medical image segmentation device, which may include: An acquisition module, configured to acquire a first detection medical image of a patient to be detected; a preprocessing module, configured to preprocess the first detected medical image to obtain a first detected image; a segmentation module, configured to input the preprocessed first detection image into a target medical image segmentation model and output a first classification category; wherein the target medical image segmentation model is obtained based on the above-mentioned model training method; The identification module is configured to identify a lesion region of the patient to be detected based on the first classification category.

[0096] The medical image segmentation apparatus in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other device than a terminal. For example, the electronic device can be a GPU BOX, a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and can also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like. The embodiments of the present application are not limited in this regard.

[0097] The medical image segmentation apparatus in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, a Linux operating system, a Windows operating system, or other possible operating systems, and the embodiments of the present application are not limited in this regard.

[0098] The embodiments of the present application provide an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the medical image segmentation method or the model training method when executing the program.

[0099] The embodiments of the present application further provide a computer readable storage medium having a computer program / instruction stored thereon, and the computer program / instruction implements the steps of the medical image segmentation method or the model training method when executed by a processor.

[0100] The embodiments of the present application further provide a computer program product, which, when executed on an electronic device, causes a processor to implement the steps of the medical image segmentation method or the model training method.

[0101] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0102] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0105] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0106] Finally, it is to be understood that the phraseology or terminology such as "first" and "second" etc. used herein is merely intended to differentiate one entity or operation from another entity or operation, without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0107] The above describes in detail a model training method, a medical image segmentation method and device provided by the present application. The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges can be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Preprocessing the sample image of the patient to obtain a first medical image; Establish a medical image segmentation model based on the Transformer framework; Inputting the first medical image into a medical image segmentation model, and outputting a first segmentation mask and a first classification probability corresponding to the first segmentation mask; determining a final loss value of the medical image segmentation model based on the first segmentation mask and the segmentation label of the first medical image; Based on the final loss value, the model parameters of the medical image segmentation model are adjusted to obtain a target medical image segmentation model; the target medical image segmentation model is used to identify the lesion area of ​​the patient to be detected based on the first detection medical image of the patient to be detected.

2. The method according to claim 1, characterized in that The medical image segmentation model includes an overlapping patch embedding module, an encoder module, a decoder module and a linear projection layer; wherein: The encoder module consists of three dual feature aggregation blocks A distributed from top to bottom; The decoder module consists of three dual feature aggregation blocks B distributed from bottom to top; A differential transformer module is set between two adjacent dual feature aggregation blocks B; The outputs of each differential transformer module are respectively fused with the first two dual feature aggregation blocks A in the encoder module through skip connections.

3. The method according to claim 2, characterized in that The dual feature aggregation block A and the dual feature aggregation block B both include: The efficient feature aggregation block first extracts the spatial dependencies of the input features by normalizing the key and query through a normalization function, then determines the spatial attention by multiplying the normalized key and value, and obtains the global context vector. Finally, the global context vector is combined with the normalized query to obtain the spatial features. The channel attention feature aggregation block uses the cross-attention mechanism to obtain channel dependencies and determine channel attention to obtain channel features; The feature aggregation feedforward network performs nonlinear transformation on spatial features and channel features respectively to obtain enhanced spatial features and channel features.

4. The method according to claim 3, characterized in that The feature aggregation feedforward network performs nonlinear transformation on spatial features and channel features respectively to obtain enhanced spatial features and channel features, including: The input feature vector is mapped through the linear projection layer to obtain the first linear projection feature, and then the first linear projection feature is mapped into the second linear projection feature through the GELU activation function; The second linear projection feature is separated into two parts, one of which is subjected to depth convolution and the other is subjected to bypass processing. The result of the depth convolution output is multiplied by the result of the bypass processing element by element, and the result of the element multiplication is input into the linear projection layer to output the final feature.

5. The method according to claim 2, characterized in that Inputting the first medical image into a medical image segmentation model and outputting a first segmentation mask and a first classification probability corresponding to the first segmentation mask includes an encoder processing stage and a decoder processing stage, wherein: The encoder processing stage sequentially extracts features from the input first medical image through three dual feature aggregation blocks A, and finally obtains the aggregated feature vector of the third sample image; In the decoder processing stage, the third sample image aggregated feature vector is extracted through three dual feature aggregation blocks B and two differential transformer modules in sequence to obtain the fifth sample image merged feature vector. After passing through the linear projection layer, the first segmentation mask and the first classification probability corresponding to each first segmentation mask are output.

6. The method according to claim 1, characterized in that Determining a final loss value of the medical image segmentation model based on the first segmentation mask and the segmentation label of the first medical image includes: determining a Dice loss value of the medical image segmentation model by measuring an overlap between the first segmentation mask and the segmentation label of the first medical image; Determining a cross entropy loss value of the medical image segmentation model based on the first classification probability and the segmentation label of the first medical image; Based on the Dice loss value and the cross entropy loss value, a final loss value of the medical image segmentation model is determined.

7. A medical image segmentation method, characterized in that: The method comprises: Acquire a first medical image of the patient to be tested; Preprocessing the first detection medical image to obtain a first detection image; Inputting the preprocessed first detection image into a target medical image segmentation model and outputting a first classification category; wherein the target medical image segmentation model is obtained based on the model training method according to any one of claims 1 to 6; Based on the first classification category, the lesion area of ​​the patient to be detected is identified.

8. A medical image segmentation device, characterized in that: The device comprises: An acquisition module, configured to acquire a first detection medical image of a patient to be detected; a preprocessing module, configured to preprocess the first detected medical image to obtain a first detected image; a segmentation module, configured to input the preprocessed first detection image into a target medical image segmentation model and output a first classification category; wherein the target medical image segmentation model is obtained based on the model training method according to any one of claims 1 to 6; An identification module is used to identify the lesion area of ​​the patient to be detected based on the first classification category.

Images