Multimodal tumor image segmentation method and device based on fully supervised contrastive learning

Abstract

The present invention provides a multimodal tumor image segmentation method and device based on fully supervised contrast learning, comprising: inputting the acquired multimodal tumor image to be segmented into an image segmentation model to obtain the classification results of each voxel to be segmented; determining the tumor area in the multimodal tumor image based on the classification results of each voxel to be segmented; the image segmentation model is obtained by training based on the target contrast learning loss after determining the target contrast learning loss based on at least one anchor point in the sample multimodal tumor image, the positive sample voxels and the negative sample voxels corresponding to each anchor point; wherein the anchor point is determined based on the segmentation window confidence corresponding to each erroneous voxel with type prediction error in the sample multimodal tumor image; the positive sample voxel is determined based on the confidence of at least one target correct voxel, and the negative sample voxel is determined based on the confidence of at least one target erroneous voxel. The present invention can improve the accuracy of tumor area segmentation in multimodal tumor images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multimodal tumor image segmentation method and device based on fully supervised contrast learning. Background Art

[0002] In today's society, malignant tumors are seriously endangering people's lives and safety. Prevention, diagnosis and treatment of malignant tumors have become an important research topic. With the development of medical imaging technology, magnetic resonance imaging technology has become an important means for doctors to observe, analyze and diagnose tumor morphology because it can provide information on the shape, size, location and metabolism of tumors.

[0003] In the actual diagnosis process, doctors often manually segment the tumor area by integrating the information of magnetic resonance images of multiple modalities. This work often takes a lot of time and effort, so it is very valuable to develop an automatic tumor segmentation technology. However, due to the characteristics of multimodal MRI images such as modality diversity, unclear and discontinuous anatomical boundaries, and scarcity of training data, the automatic segmentation of multimodal MRI tumor images is very challenging.

[0004] Therefore, how to accurately and automatically segment multimodal magnetic resonance tumor images is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a multimodal tumor image segmentation method and device based on fully supervised contrast learning, which is used to solve the defect that the prior art cannot accurately perform automatic segmentation of tumor regions in multimodal tumor images, and improve the accuracy of tumor region segmentation in multimodal tumor images.

[0006] The present invention provides a multimodal tumor image segmentation method based on fully supervised contrast learning, comprising:

[0007] Acquire a multimodal tumor image to be segmented;

[0008] Inputting the multimodal tumor image into an image segmentation model to obtain a classification result of each voxel to be segmented output by the image segmentation model;

[0009] Determining a tumor region in the multimodal tumor image to be segmented based on the classification results of each of the voxels to be segmented;

[0010] The image segmentation model is obtained by determining a target contrast learning loss based on at least one anchor point in the sample multimodal tumor image, positive sample voxels and negative sample voxels corresponding to each anchor point, and adjusting model parameters based on the target contrast learning loss;

[0011] Among them, the anchor point is determined based on the segmentation window confidence corresponding to each erroneous voxel with incorrect type prediction in the sample multimodal tumor image, and the segmentation window confidence is determined based on the confidence of each target voxel in the segmentation window; the positive sample voxel is determined based on the confidence of at least one target correct voxel, and the target correct voxel is the voxel closest to the anchor point among the correct voxels with correct type prediction in the sample multimodal tumor image, and the negative sample voxel is determined based on the confidence of at least one target erroneous voxel.

[0012] According to a multimodal tumor image segmentation method based on fully supervised contrast learning provided by the present invention, the image segmentation model is trained based on the following method:

[0013] Determining the segmentation window confidence corresponding to each of the erroneous voxels in the sample multimodal tumor image;

[0014] According to the order of the confidence of the segmentation window corresponding to each of the erroneous voxels from high to low, determining the first preset number of erroneous voxels as the anchor points;

[0015] Determine, from the correct voxels included in the first neighborhood corresponding to the anchor point, at least one of the target correct voxels that is closest to the anchor point, and determine the target correct voxel with the highest confidence among the at least one target correct voxel as the positive sample voxel;

[0016] Sort the erroneous voxels included in the second neighborhood corresponding to the anchor point from high to low according to the confidence, and determine the first second preset number of target erroneous voxels as the negative sample voxels;

[0017] Determine a target contrastive learning loss based on each of the anchor points, the positive sample voxels and the negative sample voxels corresponding to each of the anchor points;

[0018] Based on the target contrast learning loss, the model parameters of the initial image segmentation model are adjusted to obtain the image segmentation model.

[0019] According to a multimodal tumor image segmentation method based on fully supervised contrast learning provided by the present invention, the method further includes:

[0020] Inputting the sample multimodal tumor image into an initial image segmentation model to obtain a predicted classification result of each voxel type output by the initial image segmentation model;

[0021] Based on the predicted classification results of each of the voxel types and the actual classification results of each of the voxel types, a plurality of the erroneous voxels and a plurality of the correct voxels included in the sample multimodal tumor image are determined.

[0022] According to a multimodal tumor image segmentation method based on fully supervised contrast learning provided by the present invention, the decoder of the initial image segmentation model includes at least one PCM module;

[0023] The determining the target contrast learning loss based on each anchor point, the positive sample voxel and the negative sample voxel corresponding to each anchor point includes:

[0024] For each of the PCM modules, determining feature information of each of the anchor points, feature information of each positive sample voxel, and feature information of the negative sample voxel through a mapping header in the PCM module, and determining a contrastive learning loss corresponding to the PCM module based on the feature information of each of the anchor points, the feature information of each positive sample voxel, and the feature information of the negative sample voxel;

[0025] Based on the contrastive learning losses corresponding to the PCM modules, the target contrastive learning loss is determined.

[0026] According to a multimodal tumor image segmentation method based on fully supervised contrast learning provided by the present invention, the model parameters of the initial image segmentation model are adjusted based on the target contrast learning loss to obtain the image segmentation model, including:

[0027] For each of the PCM modules, the cross entropy loss and the dice loss are determined by the predicted classification results of each voxel type output by the segmentation head in the PCM module; based on the target contrast learning loss, the cross entropy loss and the dice loss, the model parameters of the initial image segmentation model are adjusted to obtain the image segmentation model.

[0028] According to a multimodal tumor image segmentation method based on fully supervised contrastive learning provided by the present invention, the contrastive learning loss corresponding to the PCM module is determined based on the feature information of each anchor point, the feature information of each positive sample voxel and the feature information of the negative sample voxel, including:

[0029] The contrastive learning loss corresponding to the PCM module is determined based on formula (1):

[0030]

[0031] Among them, A is the anchor point set, is the set of negative sample voxels, f i Represents the feature of the anchor point in the feature space, f i+ represents the feature of the i-th positive sample voxel in the feature space, f i- represents the feature of the i-th negative sample voxel in the feature space, cos(·) is the cosine similarity function, and f i ∈R D , f i-∈R D , R D represents d-dimensional features, and τ is the temperature hyperparameter.

[0032] According to a multimodal tumor image segmentation method based on fully supervised contrast learning provided by the present invention, the method further includes:

[0033] The confidence of the segmentation window corresponding to each erroneous voxel is determined based on formula (2):

[0034]

[0035] Among them, M wconf (x, y, z) represents the confidence of the segmentation window corresponding to the erroneous voxel, (x, y, z) represents the coordinates of the erroneous voxel, N represents the window length of the segmentation window, i, j and k represent the index values ​​of each voxel in the segmentation window, M conf represents the segmentation confidence of each voxel within the segmentation window.

[0036] The present invention also provides a multimodal tumor image segmentation device based on fully supervised contrast learning, comprising:

[0037] An acquisition module, used for acquiring a multimodal tumor image to be segmented;

[0038] An input module, used for inputting the multimodal tumor image into an image segmentation model to obtain a classification result of each voxel to be segmented output by the image segmentation model;

[0039] A determination module, configured to determine a tumor region in the multimodal tumor image to be segmented based on the classification results of each of the voxels to be segmented;

[0040] The image segmentation model is obtained by determining a target contrast learning loss based on at least one anchor point in the sample multimodal tumor image, positive sample voxels and negative sample voxels corresponding to each anchor point, and adjusting model parameters based on the target contrast learning loss;

[0041] Among them, the anchor point is determined based on the segmentation window confidence corresponding to each erroneous voxel with incorrect type prediction in the sample multimodal tumor image, and the segmentation window confidence is determined based on the confidence of each target voxel in the segmentation window; the positive sample voxel is determined based on the confidence of at least one target correct voxel, and the target correct voxel is the voxel closest to the anchor point among the correct voxels with correct type prediction in the sample multimodal tumor image, and the negative sample voxel is determined based on the confidence of at least one target erroneous voxel.

[0042] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal tumor image segmentation method based on fully supervised contrast learning as described in any one of the above-mentioned methods.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multimodal tumor image segmentation methods based on fully supervised contrast learning as described above.

[0044] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned multimodal tumor image segmentation methods based on fully supervised contrast learning.

[0045] The present invention provides a method and device for segmenting a multimodal tumor image based on fully supervised contrast learning. The method and device input the acquired multimodal tumor image to be segmented into an image segmentation model to obtain the classification results of each voxel to be segmented output by the image segmentation model, and determine the tumor region in the multimodal tumor image to be segmented based on the classification results of each voxel to be segmented. The image segmentation model is obtained by adjusting the model parameters based on the target contrast learning loss after determining the target contrast learning loss based on at least one anchor point in the sample multimodal tumor image, the positive sample voxels and the negative sample voxels corresponding to each anchor point, the anchor point is determined based on the segmentation window confidence corresponding to each erroneous voxel with a type prediction error in the sample multimodal tumor image, and the segmentation window confidence is determined based on the confidence of each target voxel in the segmentation window; the positive sample voxel is determined based on the confidence of at least one target correct voxel, the target correct voxel is the voxel closest to the anchor point among the correct voxels with a correct type prediction in the sample multimodal tumor image, and the negative sample voxel is determined based on the confidence of at least one target erroneous voxel. Since the segmentation window confidence corresponding to each erroneous voxel of the determined type prediction error can be used to determine the anchor point from these erroneous voxels based on the segmentation window confidence, the determined anchor point is a point in the erroneous voxel that has great help and potential for improving the performance of the image segmentation model. Therefore, the image segmentation model obtained after comparative learning based on the anchor point, positive sample voxel and negative sample voxel has better performance, and thus the tumor area in the multimodal tumor image can be accurately segmented based on the image segmentation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of a flow chart of a multimodal tumor image segmentation method based on fully supervised contrast learning provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a segmentation window corresponding to an erroneous voxel provided in an embodiment of the present invention;

[0049] Figure 3 A 3D schematic diagram of an erroneous voxel provided by an embodiment of the present invention;

[0050] Figure 4 The embodiment of the present invention provides Figure 3 3D schematic diagram of the anchor points obtained by sampling in the erroneous voxels shown;

[0051] Figure 5 A schematic diagram of anchor point selection provided by an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of positive sample selection provided by an embodiment of the present invention;

[0053] Figure 7 A schematic diagram of the structure of an initial image segmentation model provided by an embodiment of the present invention;

[0054] Figure 8 A schematic diagram of the structure of a PCM module provided in an embodiment of the present invention;

[0055] Fig. 9 A grayscale schematic diagram of the verification result provided by an embodiment of the present invention;

[0056] Fig.10 A color schematic diagram of the verification result provided by an embodiment of the present invention;

[0057] Fig.11 A schematic diagram of the structure of a multimodal tumor image segmentation device based on fully supervised contrast learning provided by an embodiment of the present invention;

[0058] Fig.12 An example physical structure diagram of an electronic device is provided. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] At present, many studies have been conducted on the segmentation methods of multimodal tumor images, and most of the existing high-performance methods are based on deep learning technology. Since the Unet structure has the characteristics of multi-scale information extraction and utilization, strong generalization ability and less training data requirements, most of the current multimodal tumor image segmentation methods are based on this structure. In view of the problems that the Unet structure has poor segmentation effect on small lesions, unclear segmentation of complex and fuzzy tumor edges, and is not robust enough for multimodal image feature extraction, many studies have improved the segmentation performance of the model by inserting attention mechanisms, optimizing network structures, and introducing multi-network cascade strategies.

[0061] Although the above methods have been applied to the segmentation of multimodal tumor images, most existing methods only improve the model architecture and extract more robust features through more complex networks to improve the segmentation performance of the network. These methods often ignore the study of the model feature space, resulting in insufficient mining of model performance.

[0062] Fully supervised contrastive learning applies the idea of ​​contrastive learning to the fully supervised paradigm. It uses the idea of ​​data comparison and can effectively use label information to improve the quality and discrimination of features. In the field of natural image semantic segmentation, current research has used fully supervised contrastive learning methods to structure the feature space of the model, making the feature space have intra-class aggregation and inter-class dispersion, thereby significantly improving the segmentation accuracy of the model.

[0063] However, most of the current research on fully supervised contrastive learning is based on the two-dimensional natural image segmentation scenario. Compared with this scenario, the multimodal tumor image segmentation scenario has the following differences: (1) Multimodal tumor images are three-dimensional multimodal images, which are more complex and need to consider information from multiple modalities; (2) The boundaries of each class in a two-dimensional natural image are coherent and clear, while multimodal tumor images have the characteristics of unclear and discontinuous anatomical boundaries; (3) The instances of each class in a two-dimensional natural image are usually regular in shape, while tumor images have irregular shapes due to the characteristics of tumor invasive growth.

[0064] Due to the differences between two-dimensional natural images and multimodal tumor images, the existing fully supervised contrastive learning method cannot be applied to the scene of tumor region segmentation in multimodal tumor images. Therefore, how to accurately automatically segment multimodal magnetic resonance tumor images is a technical problem that needs to be solved urgently.

[0065] In view of the above problems, the embodiment of the present invention proposes a multimodal tumor image segmentation method based on fully supervised contrast learning. In this method, when training an image segmentation model for tumor region segmentation of a multimodal tumor image, the anchor point can be first determined based on the segmentation window confidence corresponding to each erroneous voxel with type prediction error in the sample multimodal tumor image, and the positive sample voxel can be determined based on the confidence of at least one target correct voxel, and the negative sample voxel can be determined based on the confidence of at least one target erroneous voxel, so as to determine the target contrast learning loss based on the anchor point, the positive sample voxel corresponding to each anchor point, and the negative sample voxel, and then adjust the model parameters based on the target contrast learning loss to obtain the image segmentation model. By calling the trained image segmentation model, the tumor region in the multimodal tumor image to be segmented can be segmented. Since the segmentation window confidence corresponding to each erroneous voxel of type prediction error is determined, the anchor point is determined from these erroneous voxels based on the segmentation window confidence. Therefore, the determined anchor point is a point in the erroneous voxel that has great help and potential for improving the performance of the image segmentation model. Therefore, the image segmentation model obtained after comparative learning based on the anchor point, positive sample voxel and negative sample voxel has better performance, and thus the tumor area in the multimodal tumor image can be accurately segmented based on the image segmentation model.

[0066] Combine the following Figures 1 to 10 The multimodal tumor image segmentation method based on fully supervised contrast learning provided in an embodiment of the present invention is described. The embodiment of the present invention can be applicable to the scenario of segmenting the tumor area in the multimodal tumor image, and in particular can be applied to the segmentation scenario of the three-dimensional multimodal tumor image. The execution subject of the present method can be a computer, a server, a server cluster or an electronic device such as a specially designed multimodal tumor image segmentation device based on fully supervised contrast learning, or it can be a multimodal tumor image segmentation device based on fully supervised contrast learning arranged in the electronic device. The multimodal tumor image segmentation device based on fully supervised contrast learning can be implemented by software, hardware or a combination of both.

[0067] Figure 1 A schematic diagram of a flow chart of a multimodal tumor image segmentation method based on fully supervised contrast learning provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0068] Step 101: Acquire a multimodal tumor image to be segmented.

[0069] In this step, the multimodal tumor image to be segmented may be a three-dimensional image, which may be obtained by CT or MRI. The multimodal tumor image includes information of multiple modalities, and the multimodal tumor image includes a tumor region, which has the characteristics of unclear anatomical boundaries and discontinuity. In addition, the tumor region has an irregular shape due to the invasive growth characteristics of the tumor.

[0070] Step 102: Input the multimodal tumor image into the image segmentation model to obtain the classification results of each voxel to be segmented output by the image segmentation model.

[0071] The image segmentation model is obtained by determining the target contrast learning loss based on at least one anchor point in the sample multimodal tumor image, the positive sample voxels and the negative sample voxels corresponding to each anchor point, and adjusting the model parameters based on the target contrast learning loss. The anchor point is determined based on the segmentation window confidence corresponding to each erroneous voxel with incorrect type prediction in the sample multimodal tumor image, and the segmentation window confidence is determined based on the confidence of each target voxel in the segmentation window; the positive sample voxel is determined based on the confidence of at least one target correct voxel, and the target correct voxel is the voxel closest to the anchor point among the correct voxels with correct type prediction in the sample multimodal tumor image, and the negative sample voxel is determined based on the confidence of at least one target erroneous voxel.

[0072] In this step, the image segmentation model can be pre-trained or trained when performing image segmentation on multimodal tumor images. The image segmentation model is obtained by contrastive learning training using anchor points, positive sample voxels, and negative sample voxels. The anchor points with the greatest potential and influence on improving the performance of the image segmentation model are selected by sampling, and positive sample voxels and negative sample voxels are selected based on the anchor point sampling, so that the performance of the image segmentation model can be significantly improved through a small amount of contrastive learning. In addition, since fully supervised contrastive learning is an efficient semantic segmentation paradigm, it can directly improve the feature space of the model, and therefore, it can effectively improve the accuracy of image segmentation.

[0073] After obtaining the trained image segmentation model, the multimodal tumor image to be segmented can be input into the image segmentation model, and the classification results of each voxel to be segmented of the multimodal tumor image can be determined by the image segmentation model. Among them, the classification results include whether it is a foreground voxel or a background voxel. If a voxel to be segmented is a foreground voxel, it means that the voxel to be segmented belongs to a voxel in the tumor area. If a voxel to be segmented is a background voxel, it means that the voxel to be segmented belongs to a voxel in the non-tumor area.

[0074] Step 103: Based on the classification results of each voxel to be segmented, determine the tumor region in the multimodal tumor image to be segmented.

[0075] In this step, after determining the classification result of each voxel to be segmented in the multimodal tumor image, the voxels belonging to the foreground can be divided out based on the classification result, that is, the tumor area in the multimodal tumor image can be obtained, thereby completing the automatic segmentation of the multimodal tumor image.

[0076] The multimodal tumor image segmentation method based on fully supervised contrast learning provided by the embodiment of the present invention inputs the acquired multimodal tumor image to be segmented into the image segmentation model, obtains the classification results of each voxel to be segmented output by the image segmentation model, and determines the tumor area in the multimodal tumor image to be segmented based on the classification results of each voxel to be segmented. Wherein, the image segmentation model is obtained after the target contrast learning loss is determined based on at least one anchor point in the sample multimodal tumor image, the positive sample voxels and the negative sample voxels corresponding to each anchor point, and the model parameters are adjusted based on the target contrast learning loss. The anchor point is determined based on the segmentation window confidence corresponding to each erroneous voxel with type prediction error in the sample multimodal tumor image, and the segmentation window confidence is determined based on the confidence of each target voxel in the segmentation window; the positive sample voxel is determined based on the confidence of at least one target correct voxel, and the target correct voxel is the voxel closest to the anchor point among the correct voxels with correct type prediction in the sample multimodal tumor image, and the negative sample voxel is determined based on the confidence of at least one target erroneous voxel. Since the segmentation window confidence corresponding to each erroneous voxel of the determined type prediction error can be used to determine the anchor point from these erroneous voxels based on the segmentation window confidence, the determined anchor point is a point in the erroneous voxel that has great help and potential for improving the performance of the image segmentation model. Therefore, the image segmentation model obtained after comparative learning based on the anchor point, positive sample voxel and negative sample voxel has better performance, and thus the tumor area in the multimodal tumor image can be accurately segmented based on the image segmentation model.

[0077] Exemplarily, based on the above embodiment, the above image segmentation model can be trained in the following way:

[0078] Determine the segmentation window confidence corresponding to each erroneous voxel in the sample multimodal tumor image; determine the first first preset number of erroneous voxels as anchor points in descending order of the segmentation window confidence corresponding to each erroneous voxel; determine at least one target correct voxel closest to the anchor point from each correct voxel included in the first neighborhood corresponding to the anchor point, and determine the target correct voxel with the highest confidence among the at least one target correct voxel as the positive sample voxel; sort the erroneous voxels included in the second neighborhood corresponding to the anchor point according to the confidence from high to low, and determine the first second preset number of target erroneous voxels as negative sample voxels; determine the target contrast learning loss based on each anchor point, the positive sample voxels corresponding to each anchor point, and the negative sample voxels; based on the target contrast learning loss, adjust the model parameters of the initial image segmentation model to obtain the image segmentation model.

[0079] Specifically, multiple sample multimodal tumor images can be obtained by CT or MRI, and each sample multimodal tumor image includes a tumor area. After each sample multimodal tumor image is input into the initial image segmentation model, the initial image segmentation model predicts the type of each voxel in the sample multimodal tumor image to obtain the predicted type of each voxel. In addition, each sample multimodal tumor image also includes label information, which can characterize the true type of each voxel in the sample multimodal tumor image. By comparing the predicted type with the true type, the correct voxels with correct identification and the wrong voxels with wrong identification in each voxel in the sample multimodal tumor image can be determined.

[0080] Due to the complexity of tumor structure, automatic segmentation of tumors faces many problems such as large changes in tumor scale, irregular tumor shape and boundaries, and uneven grayscale inside the tumor. These factors make it difficult for models trained by cross entropy loss and dice loss to accurately segment tumor boundaries or micro-tumor areas. These difficult-to-distinguish erroneous voxels are usually located in very small areas and only occupy a small part of the tumor area. However, the cross entropy loss is evenly transmitted to all voxels in the image, which causes the cross entropy loss to relatively ignore the information from these erroneous voxels and focus more on easily distinguishable voxels, that is, correct voxels; dice loss is designed to deal with scenarios with strong imbalance of positive and negative samples in semantic segmentation, and also ignores these erroneous voxels. Therefore, the features learned by the model do not have sufficient discriminative power for these relatively difficult-to-distinguish erroneous voxels and their surrounding local areas.

[0081] It should be understood that in the corresponding local area around the erroneous voxel, there are usually some voxels that are very similar to the erroneous voxel but are correctly segmented by the model. These correct voxels can be used as high-quality positive sample voxels to provide valuable clues for optimizing the segmentation of erroneous voxels. Based on this consideration, in an embodiment of the present invention, a local area contrast learning strategy is provided, including a sampler for selecting the erroneous voxels with the greatest potential to improve the performance of the model from the erroneous voxels as anchor points and a local contrast sample selector for selecting corresponding high-quality contrast samples. Then, using the clues provided by these high-quality samples, better features of the hard-to-separate voxels are learned through contrast learning loss.

[0082] Compared with two-dimensional medical images, three-dimensional multimodal tumor images can provide additional information about the three-dimensional structure, but the amount of calculation required is significantly increased. Although the model and its improved model have achieved excellent results in three-dimensional image segmentation, due to the large volume of three-dimensional images, there will still be a large number of erroneous voxels that are misidentified when it comes to voxel segmentation. Performing contrast learning on all erroneous voxels will result in a huge computational overhead. Therefore, in an embodiment of the present invention, a window confidence-based anchor sampling strategy is provided, which only samples a small number of anchor points with the greatest potential to improve model performance for contrast learning each time.

[0083] Specifically, the confidence of the segmentation window corresponding to each erroneous voxel can be determined. In one implementation, Figure 2 A schematic diagram of a segmentation window corresponding to an erroneous voxel provided in an embodiment of the present invention is shown in FIG. Figure 2 As shown, the split window can be set based on actual conditions or experience, such as Figure 2 The window size of the segmentation window in is 3. The segmentation window confidence can be obtained by calculating the average segmentation confidence in the segmentation window. The segmentation window confidence can be used to represent the classification confidence of the image segmentation model in the segmentation window centered on each erroneous voxel. Specifically, the segmentation window confidence corresponding to each erroneous voxel can be determined based on formula (2):

[0084]

[0085] Among them, M wconf (x, y, z) represents the confidence of the segmentation window corresponding to the erroneous voxel, (x, y, z) represents the coordinates of the erroneous voxel, N represents the window length of the segmentation window, i, j and k represent the index values ​​of each voxel in the segmentation window, M conf represents the segmentation confidence of each voxel in the segmentation window, which can be understood as the classification confidence. conf ∈R H ×W×D . M conf The value range of M is [0,1]. confThe smaller the value of M, the lower the confidence. conf It can be determined based on formula (3):

[0086]

[0087] Among them, D c ∈R H×W×D is the segmentation result of the initial image segmentation model. Since one-hot labels are often used in semantic segmentation, D c represents the probability that each voxel p belongs to category c (lesion foreground or background), D max (p) = max[D1(p), D2(p)] represents the maximum score of each voxel in the sample multimodal tumor image, D min (p)=min[D1(p),D2(p)] represents the minimum score for each voxel.

[0088] In this embodiment, by determining the confidence of the segmentation window corresponding to each erroneous voxel, a basis is provided for selecting anchor points for subsequent sampling.

[0089] Further, after determining the segmentation window confidence corresponding to each erroneous voxel, the first preset number of erroneous voxels can be selected as anchor points in descending order of the segmentation window confidence corresponding to each erroneous voxel. The first preset number can be set according to actual conditions or experience, and the embodiment of the present invention does not limit the specific value of the first preset number.

[0090] Figure 3 A 3D schematic diagram of an erroneous voxel provided by an embodiment of the present invention, Figure 4 The embodiment of the present invention provides Figure 3 A 3D schematic diagram of the anchor points obtained by sampling in the erroneous voxels shown in FIG. Figure 3 and Figure 4 As shown, in the embodiment of the present invention, not all erroneous voxels are used for comparative learning, but a part of erroneous voxels with the greatest potential for improving the performance of the image segmentation model are selected as anchor points, so as to perform comparative learning based on the selected anchor points.

[0091] Figure 5 A schematic diagram of anchor point selection provided by an embodiment of the present invention is shown in FIG. Figure 5As shown, after all erroneous voxels are determined based on the label information of the sample multimodal tumor image, anchor points can be sampled based on the segmentation window confidence. By selecting the first preset number of erroneous voxels with the highest segmentation window confidence as anchor points from all erroneous voxels, on the one hand, the image segmentation model can automatically focus on the most difficult area to distinguish, where the most difficult area can be understood as an area with high segmentation window confidence but erroneous voxels. Usually, the image segmentation model has high confidence in erroneous voxels. On the other hand, compared with the method of randomly sampling anchor points, the anchor points obtained by sampling based on the high and low confidence of the segmentation window have continuity and can automatically focus on a specific difficult area.

[0092] After the anchor points are determined, positive sample voxels and negative sample voxels need to be selected. When determining the positive sample voxels and negative sample voxels, the embodiment of the present invention still adopts a local positive and negative sample selection strategy based on confidence. Figure 6 A schematic diagram of positive sample selection provided by an embodiment of the present invention, such as Figure 6 As shown in , when selecting positive sample voxels, it is considered that there are usually correctly segmented voxels around the anchor point. The closer to the anchor point, the higher the appearance similarity between the surrounding correctly segmented voxels and the anchor point. Therefore, the voxel closest to the anchor point and with the highest segmentation confidence can be selected as the positive sample voxel among the correctly segmented correct voxels. Specifically, for each anchor point, the first neighborhood corresponding to the anchor point can be determined with the anchor point as the center point. The first neighborhood can be, for example, a 3*3*3 neighborhood, such as Figure 6 As shown in the box in the right figure, first determine the correct voxels with correct segmentation in the 3*3*3 neighborhood, and determine at least one target correct voxel that is closest to the anchor point among these correct voxels. If there is only one target correct voxel that is closest, then directly determine the target correct voxel that is closest as the positive sample voxel. If there are multiple target correct voxels that are closest, it is also necessary to determine the confidence of each target correct voxel according to formula (3) in the above embodiment, and determine the target correct voxel with the highest confidence as the positive sample voxel. Therefore, each anchor point will only correspond to one positive sample voxel.

[0093] It should be noted that if the positive sample voxel corresponding to the anchor point is not determined in the 3*3*3 neighborhood, the size of the first neighborhood can be expanded, such as re-searching in a 4*4*4 neighborhood, and so on, until the positive sample voxel corresponding to the anchor point is found, or the first area exceeds the regional boundary of the sample multimodal tumor image.

[0094] When selecting negative sample voxels, the second neighborhood corresponding to the anchor point can be determined with the anchor point as the center, wherein the second area and the first area can be the same or different. After determining the second neighborhood, the erroneous voxels of the segmentation error can be determined in the second neighborhood, and the confidence of each erroneous voxel can be determined according to formula (3) in the aforementioned embodiment, and after sorting in order from high to low according to the confidence, the first second preset number of target erroneous voxels are determined as negative sample voxels. Among them, the second preset number can be set according to actual conditions or experience, and the embodiment of the present invention does not limit the specific value of the second preset number. Based on the above method of determining negative sample voxels, it can be known that each anchor point can correspond to a second preset number of negative sample voxels.

[0095] Furthermore, after determining the anchor points, the positive sample voxels corresponding to each anchor point, and the negative sample voxels corresponding to each anchor point, the InfoNCE loss can be used to increase the similarity between the anchor point and the positive sample voxel and reduce its similarity with the negative sample voxel, thereby completing the comparative supervision of the anchor point features to determine the target comparative learning loss. Based on the determined target comparative learning loss, the model parameters of the initial image segmentation model are adjusted, and the above process is continuously iterated until the obtained model converges or the number of iterations reaches a preset number, and the final model is determined as the image segmentation model.

[0096] In this embodiment, by determining the segmentation window confidence corresponding to each erroneous voxel in the sample multimodal tumor image, and selecting the first preset number of erroneous voxels with the highest segmentation window confidence from the erroneous voxels as anchor points according to the segmentation window confidence, the image segmentation model obtained by anchor point training can pay more attention to the most difficult to classify area. In addition, since some anchor points are sampled from the erroneous voxels, the amount of calculation can be significantly reduced, and the efficiency of image segmentation model training is improved. In addition, the target correct voxel with the highest confidence can be determined as the positive sample voxel from the first neighborhood, that is, at least one target correct voxel closest to the anchor point in the local area, so that the selected positive sample voxel is closest to the feature of the anchor point, and the selected positive sample voxel and negative sample voxel are both voxels with the highest confidence. In the process of comparative learning, the anchor point can be closer to the positive sample voxel and away from the negative sample voxel, thereby significantly improving the performance of the trained image segmentation model.

[0097] Exemplarily, based on the above embodiments, when determining a plurality of erroneous voxels and a plurality of correct voxels included in a sample multimodal tumor image, it may be performed in the following manner:

[0098] The sample multimodal tumor image is input into the initial image segmentation model to obtain the predicted classification results of each voxel type output by the initial image segmentation model; based on the predicted classification results of each voxel type and the true classification results of each voxel type, multiple erroneous voxels and multiple correct voxels included in the sample multimodal tumor image are determined.

[0099] Specifically, the initial image segmentation model can be a neural network model or a Unet model. The embodiment of the present invention does not limit the specific structure of the initial image segmentation model. After obtaining the sample multimodal tumor image, it is necessary to first label each sample multimodal tumor image, that is, determine the label of each sample multimodal tumor image. The label is used to characterize the type of each voxel in each sample multimodal tumor image, such as whether it belongs to the foreground or background of the lesion. If a voxel belongs to the tumor area, the label of the voxel can be that it belongs to the foreground. If a voxel does not belong to the tumor area, the label of the voxel can be that it belongs to the background.

[0100] After the sample multimodal tumor image is input into the initial image segmentation model, the initial image segmentation model will predict the type of each voxel in the sample multimodal tumor image, thereby obtaining the predicted classification results of each voxel type. For each voxel, the predicted classification result of the voxel is compared with the annotated true classification result. If they are consistent, the voxel is determined to be a correct voxel. If they are inconsistent, the voxel is determined to be an erroneous voxel, thereby obtaining multiple erroneous voxels and multiple correct voxels included in the multimodal tumor image.

[0101] In this embodiment, by comparing the predicted classification results of each voxel type with the pre-labeled true classification results, a plurality of erroneous voxels and a plurality of correct voxels are obtained, which not only makes the method of determining the erroneous voxels and the correct voxels simpler, but also improves the accuracy of the determined erroneous voxels and the correct voxels.

[0102] Exemplarily, based on the above embodiments, the decoder of the above initial image segmentation model includes at least one voxel wise contrastive learning module (PCM) module, and when determining the target contrastive learning loss based on each anchor point, the positive sample voxel corresponding to each anchor point, and the negative sample voxel, it can be performed in the following manner:

[0103] For each PCM module, the feature information of each anchor point, the feature information of each positive sample voxel, and the feature information of each negative sample voxel are determined through the mapping head in the PCM module, and the contrastive learning loss corresponding to the PCM module is determined based on the feature information of each anchor point, the feature information of each positive sample voxel, and the feature information of the negative sample voxel; based on the contrastive learning loss corresponding to each PCM module, the target contrastive learning loss is determined.

[0104] Specifically, Figure 7 A schematic diagram of the structure of the initial image segmentation model provided by an embodiment of the present invention, such as Figure 7 As shown, since dResUNet has a good image segmentation effect, in this embodiment, on the basis of dResUNet, a PCM module can be introduced in the final stage of the decoder, and the PCM module can guide the network to learn. Figure 7 The initial image segmentation model shown is a multimodal semantic segmentation network based on fully supervised contrastive learning-assisted segmentation. The network of this initial image segmentation model follows the U-shaped network architecture, using dResU-Net as the basis, and introducing PCM modules in the last few stages of its decoding end to perform deep supervision and contrastive learning, thereby normalizing the feature space at all scales of the network. Specifically, cross entropy loss and dice loss are calculated in each PCM module for semantic segmentation, and InfoNCE loss is calculated for contrastive learning.

[0105] Figure 8 A schematic diagram of the structure of a PCM module provided in an embodiment of the present invention is shown in FIG. Figure 8 As shown, each PCM module includes a segmentation head (seg head) and a mapping head (project head). After the output results of the last several stages of the decoder of the initial image segmentation model are input to the segmentation head (seg head), the segmentation head (seg head) uses convolution to align the number of feature maps with the number of labels, and then uses interpolation to align the feature map size of the current scale with the label size. The segmentation result can be generated by processing the convolution layer (conv) and the up sampling layer (Up sample) in the segmentation head (seghead). In addition, in this embodiment, the mapping head (project head) is constructed by means of a convolution layer (conv), a normalization layer (InstanceNorm), a nonlinear activation function layer (Leaky ReLU) and a convolution layer (conv). The above output results also need to be input into the mapping head (project head). Through the processing of the convolution layer (conv), the normalization layer (InstanceNorm), the nonlinear activation function layer (Leaky ReLU) and the convolution layer (conv) in the mapping head (project head), the output results of this stage can be mapped to the feature space of the current scale. The number of feature maps in the mapping head of each stage is consistent with the number of feature maps output by that stage.

[0106] In each PCM module, after the feature information of each anchor point, the feature information of each positive sample voxel, and the feature information of each negative sample voxel can be determined through the mapping head in the PCM module, in the back propagation stage of the initial image segmentation model, the contrastive learning loss corresponding to each PCM module is determined through the feature information of each anchor point, the feature information of each positive sample voxel, and the feature information of the negative sample voxel.

[0107] After determining the contrastive learning loss corresponding to each PCM module, the average value of the contrastive learning losses corresponding to all PCM modules can be calculated, and the average value can be determined as the target contrastive learning loss, or the maximum or minimum value of the contrastive learning losses corresponding to all PCM modules can be determined as the target contrastive learning loss.

[0108] In this embodiment, at least one PCM module is set in the decoder of the initial image segmentation model, and the contrastive learning loss is determined by the PCM module, thereby providing a network architecture basis for contrastive learning of the image segmentation model.

[0109] Exemplarily, based on the above embodiments, when determining the contrastive learning loss corresponding to the PCM module based on the feature information of each anchor point, the feature information of each positive sample voxel, and the feature information of the negative sample voxel, the contrastive learning loss corresponding to the PCM module can be determined based on formula (1):

[0110]

[0111] Among them, A is the anchor point set, is the set of negative sample voxels, f i Represents the feature of the anchor point in the feature space, f i+ represents the feature of the i-th positive sample voxel in the feature space, f i- represents the feature of the i-th negative sample voxel in the feature space, cos(·) is the cosine similarity function, and f i ∈R D , f i- ∈R D , R D represents d-dimensional features, and τ is the temperature hyperparameter.

[0112] Exemplarily, on the basis of the above embodiments, when adjusting the model parameters of the initial image segmentation model based on the target contrast learning loss to determine the image segmentation model, the cross entropy loss and dice loss can be determined for each PCM module through the predicted classification results of each voxel type output by the segmentation head in the PCM module, and the model parameters of the initial image segmentation model are adjusted based on the target contrast learning loss, cross entropy loss and dice loss to obtain the image segmentation model.

[0113] Specifically, for each PCM module, the predicted classification results of each voxel output by the segmentation head in the PCM module can be compared with the true label to calculate the cross entropy loss and dice loss, and the predicted classification results of each voxel output by the mapping head of the PCM module can be compared with the true label to calculate the contrast (InfoNCE) loss. The initial image segmentation model learns high-quality features in feature spaces of various scales.

[0114] After determining the cross entropy loss, dice loss, and target contrastive learning loss, the overall loss function can be determined based on formula (4):

[0115]

[0116] in, represents dice loss, represents the cross entropy loss, and α is a contrastive learning hyperparameter, which is used to balance the target contrastive learning loss with the dice loss and cross entropy loss. Since the main task is to learn the segmentation model, this value can be set to a small value, for example, 0.005.

[0117] It should be understood that since the initial image segmentation model cannot segment the tumor area well in the early stage, a warmup mechanism can be set to not introduce contrast loss in the first 10% epochs of training, and add contrast learning loss in all subsequent epochs.

[0118] In this embodiment, the image segmentation model is determined by determining the target contrast learning loss, cross entropy loss and dice loss, and adjusting the model parameters of the initial image segmentation model. Introducing the fully supervised contrast learning method into the three-dimensional multimodal image segmentation task can better explore the performance of the image segmentation model, so that the obtained image segmentation model has a higher accuracy when segmenting three-dimensional multimodal tumor images.

[0119] Fig. 9 A grayscale schematic diagram of the verification result provided by an embodiment of the present invention, Fig.10 A color schematic diagram of the verification result provided by the embodiment of the present invention, such as Fig. 9 and Fig.10 As shown in FIG. 1 , the image segmentation model provided by the embodiment of the present invention is compared with the most advanced semantic segmentation model of brain tumors. The research results show that compared with the existing segmentation methods, the image segmentation model provided by the embodiment of the present invention can more accurately segment these tumor regions (WT, TC and ET). Fig.10In the brain tumor segmentation task, enhanced tumor and its diffuse necrosis are the most difficult areas to segment. The existing segmentation model performs poorly in the segmentation of enhanced tumor ET and tumor core TC. The image segmentation model provided by the embodiment of the present invention can successfully segment this part of the area, and the dice index on WT, TC and ET has reached the most advanced level.

[0120] like Fig. 9 and Fig.10 As shown, compared with the baseline method, the image segmentation model provided by the embodiment of the present invention significantly improved the segmentation results of ET and TC. The data results on the 332nd case showed that the dice score of the image segmentation model provided by the embodiment of the present invention on WT was 0.967, slightly exceeding the baseline model, but the dice scores on ET and TC were 0.776 and 0.905, respectively, significantly exceeding the 0.350 and 0.895 of the baseline method. In the 232nd case, the dice scores of the image segmentation model provided by the embodiment of the present invention on ET and TC were 0.764 and 0.789, significantly exceeding the 0.721 and 0.708 of the baseline method.

[0121] The multimodal tumor image segmentation device based on fully supervised contrastive learning provided by the present invention is described below. The multimodal tumor image segmentation device based on fully supervised contrastive learning described below and the multimodal tumor image segmentation method based on fully supervised contrastive learning described above can be referred to each other.

[0122] Fig.11 A schematic diagram of a multimodal tumor image segmentation device based on fully supervised contrast learning provided by an embodiment of the present invention, referring to Fig.11 As shown, a multimodal tumor image segmentation device 1100 based on fully supervised contrast learning includes:

[0123] An acquisition module 1101 is used to acquire a multimodal tumor image to be segmented;

[0124] An input module 1102 is used to input the multimodal tumor image into an image segmentation model to obtain a classification result of each voxel to be segmented output by the image segmentation model;

[0125] A determination module 1103 is used to determine the tumor region in the multimodal tumor image to be segmented based on the classification results of each voxel to be segmented;

[0126] The image segmentation model is obtained by determining a target contrast learning loss based on at least one anchor point in the sample multimodal tumor image, positive sample voxels and negative sample voxels corresponding to each anchor point, and adjusting model parameters based on the target contrast learning loss;

[0127] Among them, the anchor point is determined based on the segmentation window confidence corresponding to each erroneous voxel with incorrect type prediction in the sample multimodal tumor image, and the segmentation window confidence is determined based on the confidence of each target voxel in the segmentation window; the positive sample voxel is determined based on the confidence of at least one target correct voxel, and the target correct voxel is the voxel closest to the anchor point among the correct voxels with correct type prediction in the sample multimodal tumor image, and the negative sample voxel is determined based on the confidence of at least one target erroneous voxel.

[0128] In an exemplary embodiment, the apparatus further comprises: an adjustment module, wherein:

[0129] The determination module 1103 is further used to determine the segmentation window confidence corresponding to each of the erroneous voxels in the sample multimodal tumor image;

[0130] The determination module 1103 is further configured to determine the first preset number of erroneous voxels as the anchor points in descending order of the confidence of the segmentation windows corresponding to the erroneous voxels;

[0131] The determination module 1103 is further configured to determine, from the correct voxels included in the first neighborhood corresponding to the anchor point, at least one of the target correct voxels that is closest to the anchor point, and determine the target correct voxel with the highest confidence among the at least one target correct voxel as the positive sample voxel;

[0132] The determination module 1103 is further configured to sort the erroneous voxels included in the second neighborhood corresponding to the anchor point from high to low according to confidence, and determine the first second preset number of target erroneous voxels as the negative sample voxels;

[0133] The determination module 1103 is further configured to determine a target contrastive learning loss based on each of the anchor points, and the positive sample voxels and negative sample voxels corresponding to each of the anchor points;

[0134] An adjustment module is used to adjust the model parameters of the initial image segmentation model based on the target contrast learning loss to obtain the image segmentation model.

[0135] In an exemplary embodiment, the input module 1102 is further used to input the sample multimodal tumor image into an initial image segmentation model to obtain a predicted classification result of each voxel type output by the initial image segmentation model;

[0136] The determination module 1103 is further configured to determine a plurality of erroneous voxels and a plurality of correct voxels included in the sample multimodal tumor image based on the predicted classification results of each voxel type and the actual classification results of each voxel type.

[0137] In an example embodiment, the decoder of the initial image segmentation model includes at least one PCM module;

[0138] The determining module 1103 is specifically used for:

[0139] For each of the PCM modules, determining feature information of each of the anchor points, feature information of each positive sample voxel, and feature information of the negative sample voxel through a mapping header in the PCM module, and determining a contrastive learning loss corresponding to the PCM module based on the feature information of each of the anchor points, the feature information of each positive sample voxel, and the feature information of the negative sample voxel;

[0140] Based on the contrastive learning losses corresponding to the PCM modules, the target contrastive learning loss is determined.

[0141] In an exemplary embodiment, the adjustment module is specifically configured to:

[0142] For each of the PCM modules, the cross entropy loss and the dice loss are determined by the predicted classification results of each voxel type output by the segmentation head in the PCM module; based on the target contrast learning loss, the cross entropy loss and the dice loss, the model parameters of the initial image segmentation model are adjusted to obtain the image segmentation model.

[0143] In an exemplary embodiment, the determination module 1103 is specifically configured to:

[0144] The contrastive learning loss corresponding to the PCM module is determined based on formula (1):

[0145]

[0146] Among them, A is the anchor point set, is the set of negative sample voxels, f i Represents the feature of the anchor point in the feature space, f i+ represents the feature of the i-th positive sample voxel in the feature space, f i- represents the feature of the i-th negative sample voxel in the feature space, cos(·) is the cosine similarity function, and f i ∈R D , f i- ∈R D , R D represents d-dimensional features, and τ is the temperature hyperparameter.

[0147] In an exemplary embodiment, the determining module 1103 is further configured to:

[0148] The confidence of the segmentation window corresponding to each erroneous voxel is determined based on formula (2):

[0149]

[0150] Among them, M wconf (x, y, z) represents the confidence of the segmentation window corresponding to the erroneous voxel, (x, y, z) represents the coordinates of the erroneous voxel, N represents the window length of the segmentation window, i, j and k represent the index values ​​of each voxel in the segmentation window, M conf represents the segmentation confidence of each voxel within the segmentation window.

[0151] The device of this embodiment can be used to execute the method of any embodiment in the embodiment of the multimodal tumor image segmentation method based on fully supervised contrast learning. Its specific implementation process and technical effects are similar to those in the embodiment of the multimodal tumor image segmentation method based on fully supervised contrast learning. For details, please refer to the detailed introduction in the embodiment of the multimodal tumor image segmentation method based on fully supervised contrast learning, which will not be repeated here.

[0152] Fig.12 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.12 As shown, the electronic device may include: a processor 1210, a communication interface 1220, a memory 1230 and a communication bus 1240, wherein the processor 1210, the communication interface 1220 and the memory 1230 communicate with each other through the communication bus 1240. The processor 1210 may call the logic instructions in the memory 1230 to execute a multimodal tumor image segmentation method based on fully supervised contrast learning, the method comprising: obtaining a multimodal tumor image to be segmented; inputting the multimodal tumor image into an image segmentation model to obtain a classification result of each voxel to be segmented output by the image segmentation model; determining a tumor region in the multimodal tumor image to be segmented based on the classification result of each voxel to be segmented; the image segmentation model is a method for determining a target contrast learning loss function based on at least one anchor point in a sample multimodal tumor image, positive sample voxels corresponding to each anchor point and negative sample voxels. After loss, the model parameters are adjusted based on the target contrast learning loss; wherein the anchor point is determined based on the segmentation window confidence corresponding to each erroneous voxel with incorrect type prediction in the sample multimodal tumor image, and the segmentation window confidence is determined based on the confidence of each target voxel in the segmentation window; the positive sample voxel is determined based on the confidence of at least one target correct voxel, and the target correct voxel is the voxel closest to the anchor point among the correct voxels with correct type prediction in the sample multimodal tumor image, and the negative sample voxel is determined based on the confidence of at least one target erroneous voxel.

[0153] In addition, the logic instructions in the above-mentioned memory 1230 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0154] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal tumor image segmentation method based on fully supervised contrast learning provided by the above methods, the method comprising: obtaining a multimodal tumor image to be segmented; inputting the multimodal tumor image into an image segmentation model to obtain a classification result of each voxel to be segmented output by the image segmentation model; determining a tumor area in the multimodal tumor image to be segmented based on the classification result of each voxel to be segmented; the image segmentation model is based on at least one of the sample multimodal tumor images. The method comprises the following steps: the first step is to determine the target contrast learning loss based on the anchor points, the positive sample voxels and the negative sample voxels corresponding to the anchor points, and the model parameters are adjusted based on the target contrast learning loss; wherein the anchor points are determined based on the segmentation window confidence corresponding to each erroneous voxel with incorrect type prediction in the sample multimodal tumor image, and the segmentation window confidence is determined based on the confidence of each target voxel in the segmentation window; the positive sample voxel is determined based on the confidence of at least one target correct voxel, and the target correct voxel is the voxel closest to the anchor point among the correct voxels with correct type prediction in the sample multimodal tumor image, and the negative sample voxel is determined based on the confidence of at least one target erroneous voxel.

[0155] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal tumor image segmentation method based on fully supervised contrast learning provided by the above methods, the method comprising: obtaining a multimodal tumor image to be segmented; inputting the multimodal tumor image into an image segmentation model to obtain a classification result of each voxel to be segmented output by the image segmentation model; determining a tumor area in the multimodal tumor image to be segmented based on the classification result of each voxel to be segmented; the image segmentation model is based on at least one anchor point in a sample multimodal tumor image, a positive sample corresponding to each anchor point, and a positive sample corresponding to each anchor point. The target contrast learning loss is determined by the main voxels and the negative sample voxels, and the model parameters are adjusted based on the target contrast learning loss; wherein the anchor point is determined based on the segmentation window confidence corresponding to each erroneous voxel with incorrect type prediction in the sample multimodal tumor image, and the segmentation window confidence is determined based on the confidence of each target voxel in the segmentation window; the positive sample voxel is determined based on the confidence of at least one target correct voxel, and the target correct voxel is the voxel closest to the anchor point among the correct voxels with correct type prediction in the sample multimodal tumor image, and the negative sample voxel is determined based on the confidence of at least one target erroneous voxel.

[0156] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0157] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal tumor image segmentation method based on fully supervised contrastive learning, characterized in that: include: Acquire a multimodal tumor image to be segmented; Inputting the multimodal tumor image into an image segmentation model to obtain a classification result of each voxel to be segmented output by the image segmentation model; Determining a tumor region in the multimodal tumor image to be segmented based on the classification results of each of the voxels to be segmented; The image segmentation model is obtained by determining a target contrast learning loss based on at least one anchor point in the sample multimodal tumor image, positive sample voxels and negative sample voxels corresponding to each anchor point, and adjusting model parameters based on the target contrast learning loss; Wherein, the anchor point is determined based on the segmentation window confidence corresponding to each erroneous voxel with incorrect type prediction in the sample multimodal tumor image, and the segmentation window confidence is determined based on the confidence of each target voxel in the segmentation window; the positive sample voxel is determined based on the confidence of at least one target correct voxel, and the target correct voxel is the voxel closest to the anchor point among the correct voxels with correct type prediction in the sample multimodal tumor image, and the negative sample voxel is determined based on the confidence of at least one target erroneous voxel.

2. The multimodal tumor image segmentation method based on fully supervised contrastive learning according to claim 1, characterized in that: The image segmentation model is trained based on the following method: Determining the segmentation window confidence corresponding to each of the erroneous voxels in the sample multimodal tumor image; According to the order of the confidence of the segmentation window corresponding to each of the erroneous voxels from high to low, determining the first preset number of erroneous voxels as the anchor points; Determine, from the correct voxels included in the first neighborhood corresponding to the anchor point, at least one of the target correct voxels that is closest to the anchor point, and determine the target correct voxel with the highest confidence among the at least one target correct voxel as the positive sample voxel; Sort the erroneous voxels included in the second neighborhood corresponding to the anchor point from high to low according to the confidence, and determine the first second preset number of target erroneous voxels as the negative sample voxels; Determine a target contrastive learning loss based on each of the anchor points, the positive sample voxels and the negative sample voxels corresponding to each of the anchor points; Based on the target contrast learning loss, the model parameters of the initial image segmentation model are adjusted to obtain the image segmentation model.

3. The multimodal tumor image segmentation method based on fully supervised contrastive learning according to claim 2, characterized in that: The method further comprises: Inputting the sample multimodal tumor image into an initial image segmentation model to obtain a predicted classification result of each voxel type output by the initial image segmentation model; Based on the predicted classification results of each of the voxel types and the actual classification results of each of the voxel types, a plurality of the erroneous voxels and a plurality of the correct voxels included in the sample multimodal tumor image are determined.

4. The multimodal tumor image segmentation method based on fully supervised contrast learning according to claim 2 or 3, characterized in that: The decoder of the initial image segmentation model includes at least one PCM module; each of the PCM modules includes a segmentation header and a mapping header; The determining the target contrast learning loss based on each anchor point, the positive sample voxel and the negative sample voxel corresponding to each anchor point includes: For each of the PCM modules, determining feature information of each of the anchor points, feature information of each positive sample voxel, and feature information of the negative sample voxel through a mapping header in the PCM module, and determining a contrastive learning loss corresponding to the PCM module based on the feature information of each of the anchor points, the feature information of each positive sample voxel, and the feature information of the negative sample voxel; Based on the contrastive learning losses corresponding to the PCM modules, the target contrastive learning loss is determined.

5. The multimodal tumor image segmentation method based on fully supervised contrastive learning according to claim 4, characterized in that: The adjusting the model parameters of the initial image segmentation model based on the target contrast learning loss to obtain the image segmentation model includes: For each of the PCM modules, the cross entropy loss and the dice loss are determined by the predicted classification results of each voxel type output by the segmentation head in the PCM module; based on the target contrast learning loss, the cross entropy loss and the dice loss, the model parameters of the initial image segmentation model are adjusted to obtain the image segmentation model.

6. The multimodal tumor image segmentation method based on fully supervised contrastive learning according to claim 4, characterized in that: The determining the contrastive learning loss corresponding to the PCM module based on the feature information of each anchor point, the feature information of each positive sample voxel, and the feature information of the negative sample voxel includes: Based on formula (1), the contrastive learning loss corresponding to the PCM module is determined Among them, A is the anchor point set, is the set of negative sample voxels, f i Represents the feature of the anchor point in the feature space, f i+ represents the feature of the i-th positive sample voxel in the feature space, f i- represents the feature of the i-th negative sample voxel in the feature space, cos(·) is the cosine similarity function, and f i ∈R D , f i- ∈R D , R D represents d-dimensional features, and τ is the temperature hyperparameter.

7. The multimodal tumor image segmentation method based on fully supervised contrastive learning according to claim 2, characterized in that: The method further comprises: The confidence of the segmentation window corresponding to each erroneous voxel is determined based on formula (2): Among them, M wconf (x, y, z) represents the confidence of the segmentation window corresponding to the erroneous voxel, (x, y, z) represents the coordinates of the erroneous voxel, N represents the window length of the segmentation window, i, j and k represent the index values ​​of each voxel in the segmentation window, M conf represents the segmentation confidence of each voxel within the segmentation window.

8. A multimodal tumor image segmentation device based on fully supervised contrastive learning, characterized in that: include: An acquisition module, used for acquiring a multimodal tumor image to be segmented; An input module, used for inputting the multimodal tumor image into an image segmentation model to obtain a classification result of each voxel to be segmented output by the image segmentation model; A determination module, configured to determine a tumor region in the multimodal tumor image to be segmented based on the classification results of each of the voxels to be segmented; The image segmentation model is obtained by determining a target contrast learning loss based on at least one anchor point in the sample multimodal tumor image, positive sample voxels and negative sample voxels corresponding to each anchor point, and adjusting model parameters based on the target contrast learning loss; Among them, the anchor point is determined based on the segmentation window confidence corresponding to each erroneous voxel with incorrect type prediction in the sample multimodal tumor image, and the segmentation window confidence is determined based on the confidence of each target voxel in the segmentation window; the positive sample voxel is determined based on the confidence of at least one target correct voxel, and the target correct voxel is the voxel closest to the anchor point among the correct voxels with correct type prediction in the sample multimodal tumor image, and the negative sample voxel is determined based on the confidence of at least one target erroneous voxel.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multimodal tumor image segmentation method based on fully supervised contrast learning as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal tumor image segmentation method based on fully supervised contrast learning as described in any one of claims 1 to 7 is implemented.

Images