Method and apparatus for visualization of tumor segmentation

Abstract

A computer-implemented method for tumor segmentation is provided, the method including acquiring image data of a region of interest in a subject's body, the region of interest corresponding to a tumor in the subject's body, generating two or more tumor segmentation prediction results based on the image data, calculating a divergence between the two or more tumor segmentation prediction results, and generating a visualization of tumor segmentation uncertainty based on the calculated divergence between the two or more tumor segmentation prediction results.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 155,564, filed 2 March 2021, U.S. Provisional Patent Application No. 63 / 155,626, filed 2 March 2021, and U.S. Non-Provisional Patent Application No. 17 / 683,643, filed 1 March 2022, which are incorporated herein by reference. [Background technology]

[0002] A tumor treatment electric field (TT field) is a low-intensity alternating current electric field in the intermediate frequency range that can be used to treat tumors, as described in U.S. Patent No. 7,565,205. The TT field is non-invasively induced within the region of interest by transducers placed directly on the patient's body and between them, applying an AC voltage. An AC voltage is applied between a first pair of transducers over a first time interval to generate an electric field whose lines of force generally extend in the anterior-posterior direction. Then, an AC voltage is applied between a second pair of transducers over a second time interval at the same frequency to generate an electric field whose lines of force generally extend in the lateral direction. The system then repeats this two-step sequence throughout the treatment.

[0003] TT field treatment planning may include segmenting tissue on medical images (e.g., MR images) to evaluate the distribution of TT fields and quantitative treatment effectiveness. Manual segmentation is time-consuming, often requiring 20–50 minutes even for an annotator with high skill and experience. Furthermore, large amounts of data and data annotation can result in noisy labeling and intra- and inter-observer variability. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] U.S. Patent No. 7565205 [Overview of the project] [Means for solving the problem]

[0005] One aspect of the present invention relates to a computer implementation method for tumor segmentation, the method comprising: acquiring image data of a region of interest of a subject's body, wherein the region of interest corresponds to a tumor in the subject's body; generating two or more tumor segmentation prediction results based on the image data; calculating a divergence between the two or more tumor segmentation prediction results; and generating a visualization of tumor segmentation uncertainty based on the calculated divergence between the two or more tumor segmentation prediction results. [Brief explanation of the drawing]

[0006] [Figure 1] This figure shows an example of a computer implementation method for tumor segmentation. [Figure 2] This figure shows an example of a computer-based architecture for tumor segmentation and uncertainty visualization. [Figure 3A] This figure shows an exemplary result of one method of tumor segmentation. [Figure 3B] This figure shows an exemplary result of one method of tumor segmentation. [Figure 3C] This figure shows an exemplary result of one method of tumor segmentation. [Figure 3D] This figure shows an exemplary result of one method of tumor segmentation. [Figure 3E] This figure shows an exemplary result of one method of tumor segmentation. [Figure 3F] This figure shows an exemplary result of one method of tumor segmentation. [Figure 3G] This figure shows an exemplary result of one method of tumor segmentation. [Figure 3H] A diagram showing exemplary results of one aspect of tumor segmentation. [Figure 3I] A diagram showing exemplary results of one aspect of tumor segmentation. [Figure 3J] A diagram showing exemplary results of one aspect of tumor segmentation. [Figure 3K] A diagram showing exemplary results of one aspect of tumor segmentation. [Figure 3L] A diagram showing exemplary results of one aspect of tumor segmentation. [Figure 3M] A diagram showing exemplary results of one aspect of tumor segmentation. [Figure 3N] A diagram showing exemplary results of one aspect of tumor segmentation. [Figure 3O] A diagram showing exemplary results of one aspect of tumor segmentation. [Figure 3P] A diagram showing exemplary results of one aspect of tumor segmentation. [Figure 4] A diagram showing an example of an apparatus for applying a TT field having a modulated electric field to a subject's body. [Figure 5] A diagram showing an exemplary computer device.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Methods and apparatuses for tumor segmentation and uncertainty visualization are disclosed. When an image of a subject's body with a tumor is provided, the computer-based techniques of the present invention segment the tumor in the image, yield a degree of uncertainty of the segmentation, and provide a visualization of the tumor segmentation and a visualization of the degree of uncertainty. The segmentation of the image is determined by a trained segmentation network. To improve the segmentation of the image, surgical data and / or anatomical data can be included.

[0008] The disclosed techniques can be used to overcome inter-observer variability, and users can edit the segmentation according to the uncertainty level. In some embodiments, the disclosed techniques use a segmentation network that is co-trained together to handle noisy labels and inter-observer variability. In one example, the Kullback-Leibler (KL) divergence can be used to visualize segmentation uncertainty.

[0009] Since different human annotators may segment the same object differently, the inventors have realized that there is a need for a technique to edit the segmentation results internally. Using the computer-based segmentation of the present invention disclosed herein, a visualization of segmentation uncertainty is provided, which can be used to improve the credibility of human annotators.

[0010] The systems and methods for segmentation disclosed herein can be used to plan TT field therapy and other clinical applications. In non-limiting embodiments, the systems and methods for segmentation of tumors (e.g., glioblastoma multiforme ("GBM")) disclosed herein can be used for postoperative patients. In one example, the surgical type and non-tumorous tissue delineation can be integrated to automatically segment the tumor. The disclosed segmentation techniques can be extended to facilitate accurate segmentation of postoperative GBM using surgical type, anatomical information, and uncertainty visualization.

[0011] Disclosure techniques can outperform other segmentation methods for postoperative data. For example, disclosure techniques can reduce segmentation errors between resection cavities and necrotic cores. Since manual annotation of images can be subject to expert disagreement, disclosure techniques can provide uncertainty maps combined with segmentation results. This can enable tissue visualization and rapid editing to improve results that depend on user preferences.

[0012] Figure 1 shows an exemplary computer implementation method 100 for tumor segmentation and uncertainty visualization, and Figure 2 shows an example of a computer-based architecture 200 for tumor segmentation and uncertainty visualization. For non-limiting illustrative purposes, each step of method 100 in Figure 1 will be discussed in relation to the exemplary architecture 200 in Figure 2.

[0013] In step 102, a training dataset consisting of images of other subjects can be obtained from a computer-readable medium. The training dataset consisting of images can include any medical images. For example, the training dataset can include images from X-ray, computed tomography (CT) scans, magnetic resonance imaging (MRI), ultrasound, nuclear medicine imaging, positron emission tomography (PET), arthrography, myelography, or a combination thereof. When developing the training set, a trained human annotator can segment tumors according to predetermined labels (e.g., resection site, necrotic core, and enhancing tumor). The resulting training set can be stored on a computer-readable medium.

[0014] In step 104, an augmentation can optionally be applied to the training dataset. The augmentation may include at least one of the following: intensity normalization, random shifting, random scaling up of intensity values, random inversion, or random scaling of image patches. For example, medical images (e.g., 1 mm 3 Resampling (down to voxel resolution) is possible. In non-limiting embodiments, bias electric field correction and / or skin and skull removal can be applied as preprocessing. In one example, a patch size of 128 × 128 × 128 voxels can be used, and to increase diversity and improve the robustness of the model, the following extensions can be applied to the data samples, namely intensity normalization of the image patch, random shift up to 0.1, random scale-up of intensity values ​​up to 10%, random inversion, and random scaling of ±10%.

[0015] In step 106, two or more segmentation networks 201a, 201b can be trained using the training dataset and the integrated training loss. Each of the segmentation networks 201a, 201b may include, for example, a variational autoencoder that reconstructs image data from shared encoder parameters. In a non-limiting embodiment, the method of disclosure may include obtaining two or more segmentation networks 201a, 201b that have been trained using a common training set and multiple training losses.

[0016] In some embodiments, the segmentation networks 201a, 201b output prediction results 202a, 202b, and the segmentation networks 201a, 201b can also be trained in parallel using the prediction results 202a, 202b and the combined loss. The combined loss may include at least one of the segmentation losses 203a, 203b, the reconstruction loss, or the divergence loss 204. In non-limiting embodiments, the combined loss may include parameters to balance the segmentation losses 203a, 203b and the divergence loss 204. For example, the segmentation losses 203a, 203b may include Dice coefficients, cross-entropy loss, or a combination thereof. For example, the divergence loss 204 may include the Kullback-Leibler (KL) divergence loss. In some embodiments, the segmentation networks 201a, 201b can be trained using an augmented training dataset. For example, segmentation networks 201a and 201b can be trained using PyTorch and MONAI, as well as the Adam optimizer and epochs (e.g., 200 epochs). The learning rate is initially set to 1e-4 and can be reduced whenever the metric plateaus over 10 epochs. Two or more segmentation networks 201a and 201b can be trained simultaneously. These segmentation networks 201a and 201b may have the same architecture but different parameters. Each segmentation network 201a and 201b may have its own segmentation loss relative to ground truth labels. A divergence loss 204 can be calculated between the prediction results 202a and 202b of segmentation networks 201a and 201b so that the entire system is trained and updated jointly.

[0017] As an example, given sample x iThe integrated training loss (or total training loss) for can be calculated as follows:

[0018]

number

[0019] In the above equation, λ is the individual segmentation loss (203a, 203b), i.e., L seg And the divergence loss (204), i.e., L div It is a parameter that balances the relationship between the two. In one example, λ can be set to 0.1. In a non-limiting embodiment, the reconstruction loss of the autoencoder, i.e., L rec For each segmentation network 201a and 201b, L Total It can be added to the calculation of L. rec This is the L2 loss between the input image and the reconstructed image. The weight β can be set to 0.1.

[0020] For example, the segmentation losses 203a and 203b can be a combination of the Dice coefficient and cross-entropy, as follows.

[0021]

number

[0022] In the above equation, K is the number of segmentation networks 201 (for example, set to 2). k is the prediction output of each network. α is used to balance the dice and cross-entropy loss (e.g., set to 0.3). In a non-restrictive embodiment, two different segmentation networks 201a and 201b can agree on prediction results for correct labels but not for incorrect labels. Thus, the coregularization term can lead segmentation networks 201a and 201b to a more stable model for clean labels.

[0023] In one example, the divergence loss 204 measures the agreement between the segmentation prediction results 202a and 202b of segmentation networks 201a and 201b, and coregularizes both segmentation networks 201a and 201b. The symmetric Kullback-Leibler (KL) divergence loss 204 can be calculated as follows:

[0024]

number

[0025] To properly handle noisy labels, a small-loss criterion technique can be employed. This criterion is based on the idea that small-loss samples are more likely to be correctly labeled. Therefore, in each mini-batch, voxels can be sorted according to their congruent losses given in equation (1) and averaged over a subset P of voxels with the minimum value. In this way, the impact of noisy-labeled voxels on the total loss and backpropagation updates is reduced. P can be the percentage of correctly labeled samples estimated in the dataset. The small-loss criterion can be used for divergence loss and cross-entropy loss calculated per voxel.

[0026] In step 108, image data 205 of the region of interest of the body of a subject with a tumor can be obtained. For example, the image data of the region of interest may include data from X-ray, computed tomography (CT) scans, magnetic resonance imaging (MRI), ultrasound, nuclear medicine imaging, positron emission tomography (PET), arthrography, myelography, or a combination thereof. Inter-observer variability / differences may be observed from the image data. For example, differences in training and image quality may increase inter-observer variability. Since the dice score between the model's predictions and ground truth annotations may be limited by the inter-observer score, the techniques of disclosure provide a segmentation method that takes into account the presence of noisy labels and does not require a clean annotated dataset. To overcome noisy labels in the dataset and inter-observer variability in the dataset, and to facilitate interactive visual review of the segmentation results, uncertainty can be computed by agreement maximization.

[0027] In step 110, two or more tumor segmentation prediction results 202a and 202b can be generated based on the image data 205 and two or more segmentation networks 201a and 201b.

[0028] In step 112, the divergence loss 204 between two or more tumor segmentation prediction results 202a, 202b generated by two or more segmentation networks 201a, 201b can be calculated.

[0029] In step 114, a visualization of tumor segmentation uncertainty can be generated based on the calculated divergence loss 204 between two or more tumor segmentation prediction results 202a, 202b. In one example, the visualization may include an image of a subject with segmentation prediction results 206 and / or uncertainty map 207. The uncertainty map 207 can be based on the calculated divergence loss 204 between two or more tumor segmentation prediction results 202a, 202b. The KL divergence, which is the relative entropy between two probability distributions, can be calculated as the uncertainty between segmentation prediction results 202a, 202b. To visualize the uncertainty, the KL divergence between the prediction results 202a, 202b of the baseline network can be calculated using equation (3). The uncertainty values ​​can be normalized across the dataset such that the maximum uncertainty is set to 1 and the minimum uncertainty is set to 0. The uncertainty map 207 can then be generated as a heatmap over the segmentation.

[0030] In a non-limiting embodiment, uncertainty map 207 can be used to remove voxels associated with high uncertainty from segmentation map 206 using a user-defined threshold. In a non-limiting embodiment, the visualization of the subject's body region of interest includes uncertainty estimation regarding the generated postoperative tumor segmentation of the subject's body.

[0031] In step 116, anatomical and surgical information of the subject's body can be obtained. In non-limiting embodiments, tumor segmentation prediction results 202a and 202b can be generated based on image data 205, anatomical information, and / or surgical information. In one example, image data 205, anatomical information, and surgical information of the subject's body's region of interest can be obtained, where the region of interest may correspond to the postoperative region of the tumor on the subject's body. Postoperative tumor segmentation and / or visualizations of the subject's body can be generated based on the image data, anatomical information, and surgical information.

[0032] In step 118, anatomical structures can be added to or removed from the visualization based on the subject's anatomical and surgical information. By adding or removing anatomical structures, the disclosure technique can improve segmentation performance. For example, several issues affecting segmentation performance for postoperative GBM can be addressed (e.g., misidentification of necrotic core and / or CSF and excision site).

[0033] Errors resulting from misidentifying the necrotic core with the resected area may be related to the fact that the necrotic core and the resected area look similar in some images. The disclosure technique can reduce this error by incorporating the surgical type into the predictive results as a post-process. If the patient underwent a biopsy, there is no resected area in the image. If the patient underwent a grand-total resection (GTR), the necrotic core is completely removed. In post-processing of the segmentation results, if the surgical type is a biopsy, the voxels labeled as the resected area can be changed to necrotic core voxels. In the case of a GTR surgery, the voxels of the necrotic core can be changed to resected area voxels.

[0034] Errors resulting from misidentification of the CSF (Chronic Surgical Fibrocartilage) and the resection site can be observed when the resection cavity is adjacent to the CSF. To address this issue, the technique disclosed involves segmenting the CSF in addition to the tumor tissue. The dataset disclosed may include validated CSF labels that can be used to train segmentation networks 201a and 201b. Co-segmentation of the CSF using the technique disclosed may reduce the mislabeling of voxels in the resection cavity.

[0035] In step 120, voxels with an uncertainty greater than a threshold can be removed from the segmentation prediction results. The threshold can be a user-defined threshold.

[0036] In step 122, the location on the subject's body where the transducer for applying the TT field should be placed can be determined based on a visualization of tumor segmentation uncertainty or at least one of two or more tumor segmentation prediction results 202a, 202b. In one example, an uncertainty map 207 can be generated based on the visualization of tumor segmentation uncertainty and / or two or more tumor segmentation prediction results 202a, 202b. The uncertainty map 207 can highlight areas that require careful examination. For example, the mean KL divergence can be overlaid as a heatmap on a medical image of the target tissue. The uncertainty map 207 can show the most uncertain areas.

[0037] In some embodiments, the TT field may include certain parameters. For example, the TT field may include an intensity within an intensity range of about 1 V / cm to about 20 V / cm. For example, the TT field may include a frequency within a frequency range of about 50 kHz to about 1 MHz. Other possible exemplary parameters for the TT field may include, among other parameters, the active time, dimming time, and duty cycle (all of which can be measured, for example, in ms).

[0038] In some embodiments, an apparatus for tumor segmentation and uncertainty visualization can be provided. In one example, the apparatus may include one or more processors and a memory storing processor-executable instructions, and when the processor-executable instructions are executed by one or more processors, the apparatus causes the apparatus to perform the disclosed method for tumor segmentation and / or uncertainty visualization.

[0039] In some embodiments, a transducer can be provided for applying an alternating current electric field having predetermined parameters to a subject's body. The alternating current electric field can be applied based on at least one of a visualization of tumor segmentation uncertainty or a segmentation prediction result.

[0040] In some embodiments, the disclosed system can include a computer-based architecture for tumor segmentation and uncertainty visualization. As an example, FIG. 2 shows an example of a computer-based architecture for tumor segmentation and uncertainty visualization. The computer-based architecture can incorporate a segmentation consensus loss for regularizing two or more baseline models that are trained simultaneously. This consensus loss can be utilized to visualize segmentation uncertainty. In non-limiting embodiments, the computer-based architecture can include a derivative of Unet. For example, the computer-based architecture can incorporate a variational autoencoder that reconstructs MRI images from shared encoder parameters for regularization. In non-limiting embodiments, the computer-based architecture can be designed according to segmentation (e.g., postoperative GBM segmentation) and can incorporate multiple modalities. To improve the accuracy of segmentation, the computer-based architecture can incorporate anatomical information and / or surgical information. For example, this data can include medical images (e.g., MRI scans) of various types of surgeries (e.g., total resection of a tumor, partial resection, and biopsy of a tumor).

[0041] In some embodiments, two or more segmentation networks can be jointly trained using an integrated loss consisting of a segmentation loss and a divergence loss. In different tumor tissues, other anatomical structures can be added to overcome confusion with healthy tissues.

[0042] In some embodiments, the image data can be resampled. For example, medical images can be resampled to a resolution of 1mm 3 voxels. Bias field correction and skin / skull removal can be applied as preprocessing steps.

[0043] In some embodiments, various augmentations can be applied to the data samples to increase diversity and improve the robustness of the model. For example, during training, voxel patches (e.g., 128×128×128) can be used, and to increase diversity and improve the robustness of the model, the following augmentations can be applied to the data samples: intensity normalization of image patches, random shift (e.g., up to 0.1), random scale-up (e.g., up to 10% of the intensity value), random inversion, and / or random scaling by ±10%. This ensures a balance between patches containing tumors and patches without tumors.

[0044] In some embodiments, improvements in the Dice coefficient and false detection rate (FDR) can be achieved. For example, a network (e.g., a Resnet-VAE network) can be trained on a training dataset using a combination of Dice and cross-entropy loss, as well as several labels (e.g., resection site, necrotic core, and augmented tumor). In non-limiting embodiments, surgical information can be added as a post-process of Resnet-VAE to reduce segmentation errors.

[0045] Experimental results Using several embodiments disclosed herein, a set of 340 labeled T1-weighted MRI brain scans from postoperative GBM patients was divided into 270 training images and 70 test images. All of these images were 1 mm 3The images were resampled to voxel resolution. Bias field correction and skin and skull removal were applied as preprocessing steps. During training, a 128×128×128 voxel patch size was used, and the following augments were applied to the data samples to increase diversity and improve model robustness: intensity normalization of image patches, random shift up to 0.1, random scale-up up to 10% of intensity values, random inversion, and ±10% random scaling. A balance was maintained between patches containing tumors and patches without tumors. Two segmentation networks were trained using PyTorch and MONAI, along with the Adam optimizer, for 200 epochs. The learning rate was initially set to 1e-4 and reduced whenever the metric plateaued over 10 epochs.

[0046] Next, the Resnet-VAE network was trained using a postoperative GBM tumor training dataset, as well as a combination of DICE and cross-entropy loss, and three labels: (1) resection site, (2) necrotic core, and (3) enhanced tumor. Then, the Resnet-VAE was evaluated against the test dataset. Next, surgical information was incorporated as a post-hoc process of the Resnet-VAE to reduce segmentation errors, and it was evaluated against the test dataset.

[0047] Table 1 presents quantitative DICE data, and Table 2 presents false detection rate (FDR) data. In Table 1, the "res" column is quantitative DICE data for the resection label, the "nec" column is quantitative DICE data for the necrotic core label, the "enh" column is quantitative DICE data for the enhanced tumor label, and the "wt" column is quantitative DICE data for the whole tumor label. In Table 2, the "csf as res" column is FDR data when the CSF is segmented and labeled with the resection site, the "nc as res" column is FDR data when the necrotic core is segmented and labeled with the resection site, and the "res as nc" column is FDR data when the resection site is segmented and labeled with the necrotic core.

[0048] Furthermore, in Tables 1 and 2, the "BL" row represents data for a single (i.e., baseline) segmentation network. The "SD" row represents data for post-processing using surgical data. The "AD" row represents data from training with CSF labels. The "UR" row represents data from joint training of two segmentation networks using both anatomical and surgical data and uncertainty regularization. The "UR*" row is the same as the "UR" row, but with uncertain voxels removed according to user preference.

[0049] As shown in Table 1, the Daice score improves, particularly in cases of confusion between the necrotic core and the excised tissue (higher values ​​indicate better results). Furthermore, as shown in Table 2, the FDR (Fat Recovery Rate) is reduced in the tissues of the necrotic core and the excised tissue (lower values ​​indicate better results).

[0050] [Table 1]

[0051] [Table 2]

[0052] In some cases, automated segmentation of the resected area "extended" into the ventricles. To address this issue, the segmentation network was trained using additional labels (i.e., CSFs). Results for the test dataset are shown in the third "AD+SD" row in Tables 1 and 2. When this method was applied with the aforementioned surgical data, both DICE and FDR improved dramatically. These results demonstrate the importance of adding surgical and anatomical data to the training and post-training processes.

[0053] Finally, the segmentation network was trained using uncertainty regularization with equation (1). Then, a user-defined threshold was simulated to demonstrate that the uncertainty map facilitates segmentation revision. The results of the final segmentation without removing uncertain voxels, and the results of the final segmentation with the user-defined threshold simulated, are shown in rows 4 "UR" and 5 "UR*" in Tables 1 and 2, respectively. Using the technique of the present invention, the proportion of mislabeled voxels is the lowest compared to all other methods, and DICE is also improved for augmented tumors and necrotic cores. The uncertainty map highlights areas requiring careful examination (e.g., Figure 3K). Furthermore, the simulation of the user-defined threshold demonstrates that the uncertainty map can be used to revise the segmentation and reduce mislabeled voxels.

[0054] It should be noted that when images in a dataset are segmented by a particular human annotator, another human annotator may consider these mislabeled voxels to be correct. Therefore, satisfactory results can be produced by using user-defined thresholds, as disclosed herein. Thus, human annotators can easily edit computer-generated segmentation based on computer-generated uncertainty maps. Providing human annotators with some control over computer-generated segmentation results can facilitate increased confidence in computer-generated segmentation results and make it easier to express personal preferences within the inter-observer variability space.

[0055] Figures 3A to 3P (collectively referred to as Figure 3) show exemplary experimental results for various aspects of tumor segmentation according to different embodiments. The images in Figure 3 show color-coded segmentation results, where red identifies the "resected area," green identifies "necrotizing," blue identifies "enhancement," and yellow identifies "cerebrospinal fluid (CSF)."

[0056] The first row of Figure 3 (Figures 3A to 3D) shows images comparing the baseline model with the post-operative process using surgical data (SD). The first row shows a comparison of baseline segmentation with the post-operative process using surgical data. Figure 3A is the original (org) image. Figure 3B is a baseline image obtained using several embodiments, which includes an error where the necrotic core segment (green) was incorrectly labeled as the excised portion (red). In Figure 3C, this label has been corrected to the necrotic core after incorporating surgical data ("SD") using several embodiments. Figure 3D is an image in the case of manual segmentation. When comparing the image in Figure 3C obtained using several embodiments with the image in Figure 3D obtained using manual segmentation, the two images are nearly identical.

[0057] The second row of Figure 3 (Figures 3E-3H) shows images demonstrating the addition of anatomical structures (i.e., CSFs) to segmentation. The second row demonstrates the results of adding (yellow) CSF labels to the training of the segmentation network. Figure 3E is the original image. Figure 3F is an image obtained by segmenting and adding surgical type information using several embodiments. However, adding surgical type information alone is insufficient because some CSFs are incorrectly labeled as excision sites (red). This is corrected when segmentation is performed on CSFs in addition to tumors, as shown in Figure 3G. Figure 3H is an image in the case of manual segmentation. When comparing the image in Figure 3G obtained using several embodiments with the image in Figure 3H obtained using manual segmentation, the two images are nearly identical.

[0058] The third row of Figure 3 (Figures 3I-3L) shows images illustrating the results of uncertainty regularization using KL divergence as the uncertainty. The third row of Figure 3 demonstrates exemplary results for the entire algorithm. Figure 3I is the original image. Figure 3J shows the segmentation results of the co-trained network. Figure 3K shows the average KL divergence overlaid on the image as a heatmap. The yellow areas indicate the most uncertain areas, which lie on the boundary between the excised region and the CSF. Figure 3L is an image from the case of manual segmentation, showing that the human annotator was also uncertain about this area.

[0059] The last row of Figure 3 (Figures 3M to 3P) shows images illustrating different uncertainty thresholds for excision labeling. The bottom row shows excision tissue segmentation obtained using different levels of uncertainty thresholds according to several embodiments. Figure 3M has the most uncertain voxels, Figure 3O has the fewest uncertain voxels, and Figure 3N has some uncertain voxels between those in Figure 3M and Figure 3O. Figure 3P is an image of manual segmentation. As the uncertainty of segmentation decreases from Figure 3M to Figure 3N and then to Figure 3O, the results begin to agree well with the manual annotation in Figure 3P.

[0060] In some embodiments, computer-generated segmentation maps can be provided jointly with computer-generated uncertainty maps, which can further assist in enabling users to quickly and interactively select personal preferences. The quantitative results in Tables 1 and 2 of the disclosure, as well as the quantitative results in Figure 3, demonstrate that the techniques of the present invention can overcome inter-observer variability and that users can edit segmentation according to uncertainty levels.

[0061] Exemplary device Figure 4 shows an example of a device for applying a TT field having a modulated electric field to a subject's body. The first transducer 401 includes 13 electrode elements 403 arranged on a substrate 404, the electrode elements 403 being electrically and mechanically connected to each other through conductive wiring 409. The second transducer 402 includes 20 electrode elements 405 arranged on a substrate 406, the electrode elements 405 being electrically and mechanically connected to each other through conductive wiring 410. The first transducer 401 and the second transducer 402 are connected to an AC voltage generator 407 and a controller 408. The controller 408 may include one or more processors and a memory accessible by one or more processors. The memory can store instructions, which, when executed by one or more processors, control the AC voltage generator 407 to implement one or more embodiments of the present invention. In some implementations, the AC voltage generator 407 and the controller 408 can be integrated into the first transducer 401 and the second transducer 402 to form the first electric field generator and the second electric field generator.

[0062] Figure 5 shows an exemplary computer device for use in embodiments of this specification. For example, device 500 may be a computer for implementing some of the techniques of the present invention disclosed herein. For example, device 500 may be a controller device for applying a TT field having a modulated electric field for embodiments of this specification. Controller device 500 can be used as controller 408 in Figure 4. Device 500 may include one or more processors 502, one or more output devices 505, and memory 503.

[0063] In one example, based on input 501, one or more processors generate control signals to control a voltage generator to implement one embodiment of the present invention. In one example, input 501 is a user input. In another example, input 501 may be from another computer communicating with controller device 500. Output device 505 can provide the operating state of the present invention, such as transducer selection, the voltage being generated, and other operational information. Output device 505 can provide visualization data according to some embodiments of the present invention.

[0064] Memory 503 is accessible via link 504 by one or more processors 502, and therefore one or more processors 502 can read information from memory 503 and write information to memory 503. Memory 503 can store instructions, and when these instructions are executed by one or more processors 502, one or more embodiments of the present invention are implemented.

[0065] Exemplary Embodiments The present invention includes other exemplary embodiments, such as the following:

[0066] Exemplary Embodiment 1. A computer implementation method for tumor segmentation, comprising: acquiring image data of a region of interest of a subject's body, wherein the region of interest corresponds to a tumor in the subject's body; generating two or more tumor segmentation prediction results based on the image data; calculating a divergence between the two or more tumor segmentation prediction results; and generating a visualization of tumor segmentation uncertainty based on the calculated divergence between the two or more tumor segmentation prediction results.

[0067] Exemplary Embodiment 2: The method of Exemplary Embodiment 1, further comprising resampling image data to a predetermined resolution.

[0068] Exemplary Embodiment 3. The method of Exemplary Embodiment 1, wherein the image data includes magnetic resonance imaging (MRI) data.

[0069] Exemplary Embodiment 4. The method of Exemplary Embodiment 1, further comprising adding or removing anatomical structures from a visualization based on anatomical and surgical information of the subject's body.

[0070] Exemplary Embodiment 5. The method of Exemplary Embodiment 1, wherein the divergence is calculated through a symmetric Kullback-Leibler (KL) divergence loss.

[0071] Exemplary Embodiment 6. The method of Exemplary Embodiment 1, wherein two or more tumor segmentation prediction results are generated by a segmentation network, each of which comprises a variational autoencoder that reconstructs image data from shared encoder parameters.

[0072] Exemplary Embodiment 7. The method of Exemplary Embodiment 1, further comprising removing voxels with an uncertainty greater than a threshold from the segmentation prediction results.

[0073] Exemplary Embodiment 8. An apparatus comprising one or more processors and a memory storing processor-executable instructions, wherein, when executed by one or more processors, the apparatus causes the apparatus to: acquire a training dataset consisting of images of other subjects; train two or more segmentation networks using the training set and integrated loss based on a comparison between the training outputs of the two or more segmentation networks; acquire image data of a region of interest of a subject's body, wherein the region of interest corresponds to a tumor in the subject's body; generate two or more segmentation prediction results based on the image data and the two or more trained segmentation networks; calculate the divergence between the two or more segmentation prediction results; and generate a visualization of tumor segmentation uncertainty based on the segmentation prediction results, wherein the visualization of segmentation uncertainty is generated based on the calculated divergence between the two or more segmentation prediction results.

[0074] Exemplary Embodiment 9. The apparatus of Exemplary Embodiment 8, wherein memory stores processor-executable instructions, and when executed by one or more processors, the apparatus further causes the apparatus to determine the location on the body of a subject where transducers for applying a tumor-therapeutic electric field should be positioned, based on at least one of a visualization of tumor segmentation uncertainty or a segmentation prediction result.

[0075] Exemplary Embodiment 10: The apparatus of Exemplary Embodiment 8, further comprising a transducer for applying an alternating current electric field having predetermined parameters to the body of a subject, wherein the alternating current electric field is applied based on at least one of a visualization of tumor segmentation uncertainty or a segmentation prediction result.

[0076] Exemplary Embodiment 11. A computer implementation method for generating tumor segmentation of a subject's body, comprising: acquiring image data, anatomical information, and surgical information of a region of interest of the subject's body, wherein the region of interest corresponds to the postoperative region of the subject's tumor; generating postoperative tumor segmentation of the subject's body based on the image data, anatomical information, and surgical information; and generating a visualization of the region of interest of the subject's body based on the generated postoperative tumor segmentation of the subject's body.

[0077] Exemplary Embodiment 12. The method of Exemplary Embodiment 11, wherein the visualization of the region of interest of the subject's body includes an uncertainty estimation regarding the generated postoperative tumor segmentation of the subject's body.

[0078] Exemplary Embodiment 13. A computer implementation method for tumor segmentation, comprising: acquiring two or more segmentation networks trained using a common training set and multiple training losses, wherein the training set includes images of other subjects; acquiring image data of a region of interest of a subject's body, wherein the region of interest corresponds to a tumor in the subject's body; generating two or more tumor segmentation prediction results based on the image data and the two or more segmentation networks; calculating divergences between the two or more tumor segmentation prediction results; and generating a visualization of tumor segmentation uncertainty based on the calculated divergences between the two or more tumor segmentation prediction results.

[0079] Exemplary Embodiment 14. The method of Exemplary Embodiment 13, further comprising applying an augmentation to a training dataset to obtain an augmented training set, wherein the augmentation includes at least one of intensity normalization, random shifting, random scaling up of intensity values, random inversion, or random scaling of image patches, and training a segmentation network using the augmented training dataset.

[0080] The embodiments shown under any heading or within any portion of this disclosure may be combined with the embodiments shown under the same or any other heading or other portion of this disclosure, unless otherwise indicated herein or unless otherwise clearly inconsistent with the context.

[0081] Numerous modifications, alterations, and changes are possible to the described embodiments without departing from the scope of the invention as defined in the claims. It is intended that the invention is not limited to the described embodiments, but rather encompasses the entire scope defined by the wording of the appended claims and their equivalents. [Explanation of symbols]

[0082] 100 Computer Implementation Methods 200 Computer-Based Architectures 201a Segmentation Network 201b Segmentation Network 202a Tumor segmentation prediction results 202b Tumor segmentation prediction results 203a Segmentation Loss 203b Segmentation Loss 204 Symmetric Kullback-Leibler (KL) Divergence Loss 205 Image data 206 Segmentation prediction results, segmentation map 207 Uncertainty Map 401 First Transducer 402 Second transducer 403 Electrode elements 404 circuit board 405 Electrode element 406 circuit board 407 AC Voltage Generator 408 Controller 409 Conductive wiring 410 Conductive wiring 500 Controller Device 501 Input 502 Processors 503 memory 504 Link 505 Output Device

Claims

1. A computer implementation method for tumor segmentation, A step of acquiring image data of a region of interest on the subject's body, wherein the region of interest corresponds to a tumor on the subject's body. The steps include obtaining anatomical and surgical information of the subject, The steps include generating two or more tumor segmentation prediction results based on the image data, anatomical information, and surgical information, The steps include calculating the divergence between the two or more tumor segmentation prediction results, A step of generating a visualization of tumor segmentation uncertainty based on the calculated divergence between the two or more tumor segmentation prediction results. Methods that include...

2. The method according to claim 1, wherein the visualization includes an image of the subject with segmentation prediction results and an uncertainty map, the uncertainty map being based on the calculated divergence between the two or more tumor segmentation prediction results.

3. A computer implementation method for tumor segmentation, A step of obtaining two or more segmentation networks trained using a common training set and multiple training losses, wherein the training set includes images of other subjects. A step of acquiring image data of a region of interest on the subject's body, wherein the region of interest corresponds to a tumor on the subject's body. The steps include generating two or more tumor segmentation prediction results based on the image data and the two or more segmentation networks, The steps include calculating the divergence between the two or more tumor segmentation prediction results, A step of generating a visualization of tumor segmentation uncertainty based on the calculated divergence between the two or more tumor segmentation prediction results. Methods that include...

4. The method according to claim 3, wherein the segmentation network is trained in parallel using a combined loss, the combined loss comprising at least one of a segmentation loss or a divergence loss.

5. A step of obtaining anatomical information and surgical information of the subject, wherein the two or more tumor segmentation prediction results are generated based on the image data, the two or more segmentation networks, the anatomical information, and the surgical information. The method according to claim 3, further comprising:

6. The steps of adding anatomical structures to the visualization or removing anatomical structures from the visualization, based on the anatomical information and surgical information of the subject's body. The method according to claim 5, further comprising:

7. The step of determining the location on the subject's body where a transducer for applying a tumor therapeutic electric field should be placed, based on the visualization of tumor segmentation uncertainty or at least one of the two or more tumor segmentation prediction results. The method according to claim 3, further comprising:

8. A device comprising one or more processors and a memory storing processor-executable instructions, wherein when the processor-executable instructions are executed by the one or more processors, the device provides: Obtaining a training dataset consisting of images from other subjects, Using the aforementioned training dataset and integrated loss, two or more segmentation networks are trained based on a comparison of the training outputs of the two or more segmentation networks. Acquiring image data of a region of interest on the subject's body, wherein the region of interest corresponds to a tumor on the subject's body. Based on the aforementioned image data and the two or more trained segmentation networks, two or more segmentation prediction results are generated. Calculating the divergence between the two or more segmentation prediction results, To generate a visualization of tumor segmentation uncertainty based on the segmentation prediction results, wherein the visualization of tumor segmentation uncertainty is generated based on the calculated divergence between the two or more segmentation prediction results. A device that performs an action.

9. The apparatus according to claim 8, wherein the integrated loss includes segmentation loss, divergence loss, and reconstruction loss.

10. The apparatus according to claim 9, wherein the segmentation loss includes a die coefficient and a cross-entropy loss.

11. The apparatus according to claim 9, wherein the divergence loss includes a Kullback-Leibler divergence loss.

12. The apparatus according to claim 9, wherein each of the segmentation networks comprises a variational autoencoder, and the reconstruction loss includes the reconstruction loss of the variational autoencoder for each segmentation network.

13. The apparatus according to claim 9, wherein the combined loss includes a parameter for balancing the segmentation loss and the divergence loss.

14. When the aforementioned processor-executable instruction is executed by one or more processors, the device will: The acquisition of anatomical and surgical information of the subject, wherein the acquisition of two or more segmentation prediction results is generated based on the image data, the two or more trained segmentation networks, the anatomical information, and the surgical information. The apparatus according to claim 8, which further performs the following.

Images