Tumor segmentation model training method, tumor segmentation method and medium

Abstract

The invention discloses a tumor segmentation model training method, a tumor segmentation method and a medium. The training method comprises the following steps: providing a training data set; pre-training the first model by using the labeled image and the truth value label to obtain a pre-trained model; segmenting the first phase image and the second phase image by using a second model to respectively obtain a first segmentation result and a second segmentation result; generating a prompt from the second segmentation result, and inputting the prompt and the first phase image into a pre-training model to obtain a third segmentation result; and iteratively updating the pre-training model to obtain a tumor segmentation model based on the contrast of the first segmentation result and the third segmentation result. According to the method, the segmentation result of the multi-phase CT image can be optimized, the tumor is positioned by utilizing the phase image with a better display result, and a better segmentation effect can be obtained at a lower training cost without fine adjustment again in the phase with a poorer display result by virtue of the pre-training model subjected to fine adjustment according to a specific data set.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a training method for a tumor segmentation model, a tumor segmentation method and a medium. Background Art

[0002] Tumor segmentation is an important branch of medical image processing. With the development of computer technology, a variety of methods and frameworks have emerged that can be applied to this branch.

[0003] Specifically, early applications include the UNet framework: UNet is a deep learning architecture for image segmentation, consisting of a contracting path and an expansive path. The contracting path gradually extracts image features through convolution and pooling, while the expansive path maps the features back to the original resolution through upsampling and convolution. Skip connections help retain more detailed information. UNet is suitable for fields such as medical imaging, with excellent performance and a wide range of applications.

[0004] Subsequent improvements such as the nnUNet (no-new-UNet) framework: nn-UNet is a neural network structure that is an improvement and optimization of the classic UNet. Compared with the original UNet, nnUNet uses some new technologies and strategies to improve performance and efficiency, but does not introduce a completely new architecture. It still retains the encoder-decoder structure of UNet, but improves performance by introducing deeper layers, using more powerful feature extractors, and applying more advanced loss functions and optimization algorithms. nnUNet has been widely used in fields such as medical image segmentation, and can effectively extract precise features from complex images and perform accurate segmentation. It provides better performance and robustness while keeping UNet simple and easy to understand.

[0005] In addition to the UNet framework-based approach, there is also the SAM (Segment anything) method, which aims to address the challenges of multiple different types of image segmentation tasks. This method proposes a new framework that uses adaptive clustering and instance segmentation to represent all instances in an image using a unified embedding space. With this approach, the model can learn to handle various types of instances within a single training phase without building a separate model for each type. This general approach has shown good performance and flexibility in handling different types of instance segmentation tasks.

[0006] Based on this framework, the Slide-SAM method was born: Slide-SAM only needs hints from the central slice to infer multiple adjacent slices at the same time, and the generated predictions can be used to generate hints for the next set of adjacent slices. This is achieved through a sliding window approach, which ultimately enables one hint to segment the entire volume. In addition, the architecture and tasks of Slide-SAM are similar to the original SAM, making it easier to utilize SAM's pre-trained weights. Slide-SAM achieves excellent 3D image reasoning performance on medical image segmentation tasks at a lower training cost.

[0007] Although nnUNet can achieve good segmentation results in a single phase, its segmentation results are very unstable, especially in the first phase. In some cases, it is even impossible to segment the tumor. At the same time, a separate segmentation model must be trained for each phase, which has a high training cost.

[0008] Moreover, in the field of medical imaging, SAM provides new possibilities for accelerating data annotation by using non-fixed points, bounding boxes, and coarse mask cues to define segmentation region categories. However, the huge distribution gap between natural images and medical images prevents SAM from being directly applied to medical images.

[0009] How to achieve better segmentation results with lower training cost in phase images with relatively ambiguous segmentation (usually low phases, such as phase 1) is a practical problem that needs to be solved in this field. Summary of the invention

[0010] In view of the deficiencies in the prior art, the object of the present invention is to provide a tumor segmentation model training method, a tumor segmentation method and a medium.

[0011] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention includes:

[0012] In a first aspect, the present invention provides a method for training a tumor segmentation model, characterized by comprising:

[0013] Providing a training data set, the training data set comprising multiple phase images, labeled images and true value labels corresponding to the labeled images, the multiple phase images comprising a first phase image and a second phase image of a target organ, the first phase image having a lower enhancement degree than the second phase image;

[0014] Pre-training a first model using the labeled images and true value labels to obtain a pre-trained model, wherein the first model is a Slide-SAM network structure;

[0015] Segment the first phase image and the second phase image using a second model to obtain a first segmentation result and a second segmentation result respectively, wherein the second model is a nn-UNet network structure;

[0016] generating a hint from the second segmentation result, and inputting the hint and the first phase image into the pre-trained model to obtain a third segmentation result, wherein the hint is used to indicate the location of the tumor figure in the second segmentation result;

[0017] Based on the comparison between the first segmentation result and the third segmentation result, the parameters of the pre-trained model are iteratively updated to obtain the tumor segmentation model.

[0018] In a second aspect, the present invention also provides a tumor segmentation model trained by the above training method.

[0019] In a third aspect, the present invention further provides a tumor segmentation method for organ images, comprising:

[0020] Providing the above tumor segmentation model;

[0021] The image to be segmented in the same phase as the first phase image is input into the tumor segmentation model to obtain a tumor segmentation result.

[0022] In a fourth aspect, the present invention further provides a readable storage medium in which the above-mentioned tumor segmentation model is stored, or a computer program is stored, and when the computer program is run, the above-mentioned training method or the steps of the above-mentioned tumor segmentation method are executed.

[0023] Based on the above technical solution, compared with the prior art, the beneficial effects of the present invention at least include:

[0024] The multi-phase organ tumor segmentation optimization method based on the Slide-SAM architecture proposed in the present invention can optimize the segmentation results of multi-phase CT images, locate the tumor using the segmentation results of the phase images with better display results, and then rely on the pre-trained Slide-SAM model that has been fine-tuned according to the specific data set to achieve better segmentation effects in the phases with poor display results without further fine-tuning and with a lower training cost.

[0025] The above description is only an overview of the technical solution of the present invention. In order to enable those skilled in the art to more clearly understand the technical means of the present application and implement them according to the contents of the specification, the following is a description of the preferred embodiments of the present invention in conjunction with detailed drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a process diagram of a training method and a segmentation method provided by a typical implementation case of the present invention;

[0027] Figure 2a It is a second phase image example diagram provided by a typical implementation case of the present invention;

[0028] Figure 2b It is a first phase image example diagram provided by a typical implementation case of the present invention;

[0029] Figure 3a It is a segmentation result of the control group of the first phase image provided by a typical implementation case of the present invention;

[0030] Figure 3b It is the segmentation result of the experimental group of the first phase image provided by a typical implementation case of the present invention;

[0031] Figure 3c is a true value segmentation result of the first phase image provided by a typical implementation case of the present invention;

[0032] Figure 4 It is a schematic diagram of the training process of the Slide-SAM network structure in the training method provided by a typical implementation case of the present invention. DETAILED DESCRIPTION

[0033] In view of the deficiencies in the prior art, the inventor of this case has proposed the technical solution of the present invention after long-term research and extensive practice. The technical solution, its implementation process and principle will be further explained as follows.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0035] Moreover, relational terms such as “first” and “second” are merely used to distinguish one component or method step from another with the same name, but do not necessarily require or imply any such actual relationship or order between these components or method steps.

[0036] Clinicians use multi-phase CT scans to improve the accuracy of kidney lesion diagnosis through contrast enhancement of different phases. At the same time, although there are differences between CT images of different phases due to the different contrast enhancement effects of contrast agents, the location of the tumor remains roughly unchanged. Therefore, inspired by doctors, we propose that the tumor location in the phase with relatively clear organ edges is helpful for tumor location and segmentation in the phase with relatively blurred edges, and the tumor location here can be used as a prompt by Slide-SAM.

[0037] Therefore, using the Slide-SAM method to optimize the segmentation results of nnUNet for multi-phase tumors can achieve better segmentation results at a lower training cost. The main purpose of this method is to significantly improve the segmentation results of phase one, which will be useful in some scenarios, such as when there is already high-phase data, but it is still hoped to obtain the segmentation results of the corresponding low-phase data as pseudo-labels to help train the low-phase segmentation model. In the case of extremely high and inaccurate manual labeling costs, the pseudo-labels generated by this method are better than other mainstream methods.

[0038] Based on the above invention concept, see Figure 1 As shown, an embodiment of the present invention provides a method for training a tumor segmentation model, which includes the following steps:

[0039] Providing a training data set, the training data set comprising multiple phase images, labeled images and true value labels corresponding to the labeled images, the multiple phase images comprising a first phase image and a second phase image of a target organ, the first phase image having a lower enhancement degree than the second phase image;

[0040] Pre-training a first model using the labeled images and true value labels to obtain a pre-trained model, wherein the first model is a Slide-SAM network structure;

[0041] Segment the first phase image and the second phase image using a second model to obtain a first segmentation result and a second segmentation result respectively, wherein the second model is a nn-UNet network structure;

[0042] generating a hint from the second segmentation result, and inputting the hint and the first phase image into the pre-trained model to obtain a third segmentation result, wherein the hint is used to indicate the location of the tumor figure in the second segmentation result;

[0043] Based on the comparison between the first segmentation result and the third segmentation result, the parameters of the pre-trained model are iteratively updated to obtain the tumor segmentation model.

[0044] In the above technical scheme, the target organ refers to a variety of organs that may produce tumors. The following implementation case takes the kidney as a representative case, but it does not mean that the implementation method provided by the present invention is limited to this organ; and, the "first phase" and "second phase" defined in the present invention are intended to refer to the distinguishing names of different phases, rather than the serial numbers of specific phases. For example, in the typical case below, the first phase refers to phase one, and the second phase refers to phase three, but it is not limited to this. The first and second phases may also refer to other phases. For example, for the kidney, phase three is usually the clearest phase, but for different organs, the clearest phase may not be the same. It is sufficient that the second phase is more enhanced (with clear boundaries) than the first phase.

[0045] Regarding some of the specific implementation details, in some implementation schemes, the updating process of the pre-trained model includes:

[0046] Inputting a plurality of adjacent slices in the first phase image into a backbone network, generating a plurality of image features through a feature encoder; encoding the prompts using a prompt encoder to obtain prompt features;

[0047] Using a mask decoder to receive the image features and the prompt features, and generate output masks corresponding to a plurality of slices;

[0048] For the output mask, a total loss function is calculated.

[0049] In some embodiments, the loss function of the pre-trained model is expressed as:

[0050]

[0051] in, represents the segmentation loss function; and Represent the cross entropy loss function and Dice loss function respectively; and Represent the predicted mask and the true value label respectively, H represents the pixel width of the image, and W represents the pixel height of the image; λ1 and λ2 represent the weights of the cross entropy loss function and the Dice loss function respectively; represents the IoU (Intersection over Union) loss function; U represents the predicted value of IoU; It represents the average of the squares of the differences between the predicted value and the actual value; IoU(·) represents the operation that measures the overlap between the predicted value and the true value, and its formula is:

[0052]

[0053] Among them, Intersection Area is the area of ​​the overlapping area between the predicted box and the true box. Union Area is the area of ​​the union area of ​​the predicted box and the true box.

[0054] Represents the total loss function; U represents IoU prediction; argmin represents returning the value of the independent variable that minimizes the function value or the index of the minimum value in the array; j represents the sequence number of the Slide-SAM input slice and its upper and lower adjacent slices, j = {1, 2, 3}.

[0055] In the above loss function, λ1 and λ2 are weights of different loss terms, which are used to balance the influence between the two loss terms. In practice, λ1 and λ2 are set to 20 and 1 respectively, and are not limited to this. The value can be adjusted based on existing experience.

[0056] As some typical examples of the above schemes, Figure 4 As shown in the figure, during training, three adjacent slices can first be input into the backbone network. Then the Prompt encoder is used to encode the points or boxes. The Mask decoder receives the features generated in the previous step and generates a mask for each slice using a different head. The mixed loss is calculated only for the labeled layers.

[0057] Specifically, given three slices of images, the Mask decoder can embed the corresponding images and map the output tokens to feature maps For the feature map F input into the Mask decoder o , Slide-SAM has three different MLP blocks to transform the feature map F o Converted into three feature maps and Each feature map represents a slice. H is divided into three heads, and Used to segment different semantic areas. Used for IoU prediction. Similar to SAM, Mask prediction can be obtained And IoU prediction

[0058] M ij =F i ⊙H sj , j = {1, 2, 3}

[0059] U j =MLP(H u ),

[0060] where ⊙ represents point-wise multiplication.

[0061] In some embodiments, the process of generating the prompt specifically includes the following process:

[0062] Acquire a circumscribed rectangle of the tumor figure from the third segmentation result;

[0063] According to the number of slices of the second phase image, the outer rectangle is divided into a corresponding number of multiple rectangular frames as the prompt.

[0064] In some embodiments, the process of generating the prompt can be expressed as:

[0065] Prompt Pb =BB(M Pb )

[0066] Among them, Prompt Pb Indicates the prompt; M Pb represents the second segmentation result; BB(·) represents the calculation of M Pb Operation on the bounding rectangle of .

[0067] In some embodiments, the generation process of the third segmentation result is expressed as:

[0068] M SAM-Pa =SAM(Volume Pa , Prompt Pb )

[0069] Among them, M SAM-Pa represents the third segmentation result; SAM(·) represents the pre-trained model; Volume Pa Represents the first phase image.

[0070] In some embodiments, the training process of the pre-trained model may specifically include the following process:

[0071] In combination with the third segmentation result, updating the mask of the first segmentation result to obtain a final segmentation result;

[0072] Based on the evaluation of the final segmentation result, the parameters of the pre-trained model are updated to obtain the tumor segmentation model.

[0073] In some embodiments, the process of obtaining the final segmentation result can be expressed as:

[0074] M=SelectMask(M SAM-Pa , M Pa )

[0075] Wherein, M represents the final segmentation result; SelectMask(·) represents the operation of updating the segmentation result with poor effect by checking the connected domain, checking the pixel value, etc.; M Pa represents the first segmentation result.

[0076] Correspondingly, the second aspect of the embodiment of the present invention further provides a tumor segmentation model trained by the training method provided in any of the above-mentioned implementation modes.

[0077] As an application of the above technical solution, the present invention also provides a tumor segmentation method for organ images, which comprises:

[0078] Providing the above tumor segmentation model;

[0079] The image to be segmented in the same phase as the first phase image is input into the tumor segmentation model to obtain a tumor segmentation result.

[0080] Furthermore, an embodiment of the present invention also provides a readable storage medium, in which the above-mentioned tumor segmentation model is stored, or a computer program is stored, and when the computer program is run, the above-mentioned training method or the steps of the above-mentioned tumor segmentation method are executed.

[0081] The technical solution of the present invention is further described in detail below through several embodiments and in conjunction with the accompanying drawings. However, the selected embodiments are only used to illustrate the present invention, and do not limit the scope of the present invention.

[0082] Example 1

[0083] This embodiment illustrates a specific kidney image segmentation method, which is as follows:

[0084] As some typical examples of the above implementation scheme, in this implementation case, the model with pre-trained weights of Slide-SAM-B is first used, and the fully annotated renal tumor segmentation dataset provided by medical workers is used to fine-tune the model.

[0085] Then, nnUnet training is used on the dataset to select the phase 3 image P3 with clear organ boundaries and the phase 1 image P1 with blurred boundaries to obtain the corresponding prediction masks M. P3 and M P1 Then get M P3 The circumscribed rectangle of , and divide the rectangle into n rectangular boxes according to the number of slices n of each CT as prompts P3 :

[0086] Prompt P3 =BB(M P3 )

[0087] After that, put Prompt P3 Combined with the fine-tuned Slide-SAM model of the phase 1 image input, the prediction mask M of the phase 1 is obtained. SAM-P1 :

[0088] M SAM-P1 =SAM(Volume P1 , Prompt P3 )

[0089] Finally, by checking the connected domain and pixel values, M P1 The masks with poor performance in are updated to obtain the final segmentation result M.

[0090] M=SelectMask(M SAM-P1 , M P1 ).

[0091] Example 2

[0092] This embodiment illustrates a specific application test of the segmentation method shown in Example 1, and the test process and results are as follows:

[0093] 2-1 Dataset

[0094] The dataset is a multi-phase renal tumor dataset provided by Guangdong Provincial People's Hospital. It is divided into five phases: P1, P2, P3, P4, and P5 (Phase 1, Phase 2, Phase 3, Phase 4, and Phase 5). Different phases refer to CT scan images captured at different time intervals after the injection of contrast agent. Generally, clinicians can use CT scans of different phases to improve the accuracy of lesion diagnosis through contrast enhancement of different phases.

[0095] The lesions include five types, namely ccRCC (clear cell renal cell carcinoma), chRCC (chromophobe renal cell carcinoma), pRCC (chromophobe renal cell carcinoma), fpAML (fat-poor renal angiomyolipoma), and RO (renal oncocytoma).

[0096] There are obvious differences in the quality of multi-phase CT images. For example, the images of phase 3 and phase 1 are as follows: Figure 2a and Figure 2b As shown, organs in the same location vary greatly in different phases. Phase three is obviously clearer and can better distinguish tumors with the naked eye.

[0097] 2-2 Evaluation Indicators

[0098] Dice coefficient: Also called overlap coefficient or The coefficient is an indicator that measures the degree of overlap between the segmentation result and the true label. Its calculation formula is:

[0099]

[0100] Where TP is the true positive (the number of pixels classified as positive by both the model and the true label), FP is the false positive (the number of negative pixels incorrectly classified as positive by the model), and FN is the false negative (the number of positive pixels incorrectly classified as negative by the model).

[0101] 2-3 Experimental Details

[0102] nnUNet uses the V2 version, trained for 1000 Epochs on 111 samples using the 3d_fullres configuration, and verified on 28 samples.

[0103] Slide-SAM uses the publicly available Slide-SAM-B weights, fine-tuned on the same 111 training samples, and validated on 28 samples.

[0104] 2-4 Test Results

[0105] As shown in Table 1, 28 severe examples were selected in the table where nnUNet had significantly poor performance and were replaced by Slide-SAM. It can be seen that there are many examples that nnUNet cannot segment at all, but after using the auxiliary box formed by the mask obtained by nnUNet based on the phase 3 image as a prompt for training, Slide-SAM can achieve better segmentation results for these examples. In terms of the average performance of the 28 examples (all verification examples), compared with directly using nnUNet to segment the phase 1 image, the average Dice can be improved by 0.1254.

[0106] The confidence in the table is a hyperparameter used to constrain the segmentation quality of Slide-SAM. It calculates the IoU (Intersection over Union) between the bounding rectangle of each segmented result and the original prompt, retains the part where the IoU exceeds the confidence value, and increases the accuracy of the segmentation.

[0107] Table 1: Performance comparison between Slide-SAM and nnUNet used in this method

[0108]

[0109] like Figure 3a , Figure 3b and Figure 3c As shown, from the segmentation results after nnUNet training, Figure 3a The nnUNet shown in Figure 1 is completely unable to segment the tumor in the same image. Figure 3b The Slide-SAM method, which uses the auxiliary frame formed by the three-phase image mask as a prompt, can segment the image close to Figure 3c The result of the true value in .

[0110] Based on the above implementation cases, it can be clearly seen that the multi-phase renal tumor segmentation optimization method based on the Slide-SAM architecture proposed in the embodiment of the present invention can optimize the segmentation results of multi-phase CT images, and locate the tumor using the segmentation results of the phase images with better display results. Then, relying on the pre-trained Slide-SAM model that has been fine-tuned according to the specific data set, better segmentation effects can be achieved in the phases with poor display results without further fine-tuning at a lower training cost.

[0111] It should be understood that the above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A training method for a tumor segmentation model, It is characterized in that include: Providing a training data set, the training data set comprising multiple phase images, labeled images and true value labels corresponding to the labeled images, the multiple phase images comprising a first phase image and a second phase image of a target organ, the first phase image having a lower enhancement degree than the second phase image; Pre-training a first model using the labeled images and true value labels to obtain a pre-trained model, wherein the first model is a Slide-SAM network structure; Segment the first phase image and the second phase image using a second model to obtain a first segmentation result and a second segmentation result respectively, wherein the second model is a nn-UNet network structure; generating a hint from the second segmentation result, and inputting the hint and the first phase image into the pre-trained model to obtain a third segmentation result, wherein the hint is used to indicate the location of the tumor figure in the second segmentation result; Based on the comparison between the first segmentation result and the third segmentation result, the parameters of the pre-trained model are iteratively updated to obtain the tumor segmentation model.

2. The training method according to claim 1, It is characterized in that The updating process of the pre-trained model includes: Inputting a plurality of adjacent slices in the first phase image into a backbone network, generating a plurality of image features through a feature encoder; encoding the prompts using a prompt encoder to obtain prompt features; Using a mask decoder to receive the image features and the prompt features, and generate output masks corresponding to a plurality of slices; For the output mask, calculating a total loss function; The loss function of the pre-trained model is expressed as: in, represents the segmentation loss function; and Represent the cross entropy loss function and Dice loss function respectively; and Represent the predicted mask and the true value label respectively, H represents the pixel width of the image, and W represents the pixel height of the image; λ 1 and λ 2 Represent the weights of the cross entropy loss function and the Dice loss function respectively; represents the IoU loss function; U represents the predicted value of IoU; It represents the average of the squares of the differences between the predicted value and the actual value; IoU(·) represents the operation that measures the overlap between the predicted value and the true value, and its formula is: Among them, Intersection Area is the area of ​​the overlapping area between the predicted box and the true box, and Union Area is the area of ​​the union area of ​​the predicted box and the true box; represents the total loss function; argmin represents returning the value of the independent variable that minimizes the function value or the index of the minimum value in the array; j represents the sequence number of the Slide-SAM input slice and its upper and lower adjacent slices, j = {1, 2, 3}.

3. The training method according to claim 1, It is characterized in that The process of generating the prompt specifically includes: Acquire a circumscribed rectangle of the tumor figure from the third segmentation result; According to the number of slices of the second phase image, the outer rectangle is divided into a corresponding number of multiple rectangular frames as the prompt.

4. The training method according to claim 3, It is characterized in that The process of generating the prompt is expressed as: Prompt Pb =BB(M Pb ) Among them, Prompt Pb Indicates the prompt; M Pb represents the second segmentation result; BB(·) represents the calculation of M Pb Operation on the bounding rectangle of .

5. The training method according to claim 4, It is characterized in that The generation process of the third segmentation result is expressed as: M SAM-Pa =SAM(Volume Pa ,Prompt Pb ) Among them, M SAM-Pa represents the third segmentation result; SAM(·) represents the pre-trained model; Volume Pa Represents the first phase image.

6. The training method according to claim 1, It is characterized in that The training process of the pre-trained model includes: In combination with the third segmentation result, updating the mask of the first segmentation result to obtain a final segmentation result; Based on the evaluation of the final segmentation result, the parameters of the pre-trained model are updated to obtain the tumor segmentation model.

7. The training method according to claim 6, It is characterized in that The process of obtaining the final segmentation result is expressed as: M=SelectMask(M SAM-Pa ,M Pa ) Wherein, M represents the final segmentation result; SelectMask(·) represents the operation of updating the segmentation result by checking the connected domain and viewing the pixel value; M Pa represents the first segmentation result.

8. A tumor segmentation model trained by the training method according to any one of claims 1 to 7.

9. A tumor segmentation method for organ imaging, It is characterized in that include: Providing the tumor segmentation model according to claim 8; The image to be segmented in the same phase as the first phase image is input into the tumor segmentation model to obtain a tumor segmentation result.

10. A readable storage medium, It is characterized in that The readable storage medium stores the tumor segmentation model according to claim 8, or stores a computer program, and when the computer program is run, it executes the training method according to any one of claims 1 to 7, or the steps of the tumor segmentation method according to claim 9.

Images