User-guided domain adaptation method and related apparatus

Abstract

The present application provides a user-guided domain adaptation (UGDA) method and related devices. The method includes training a combined model by minimizing a supervised loss of the combined model using source image datasets to obtain first shared weights of a first FCN and second shared weights of a second FCN; training a discriminator by inputting extreme point-mask prediction pairs for each of the source image datasets and a target image dataset and by minimizing a discriminator loss to obtain discriminator weights; and fine-tuning the combined model by fooling the discriminator by predicting extreme point-mask prediction pairs for the target image dataset and by matching a distribution of the extreme point-mask prediction pairs for the target image dataset with a distribution of the extreme point-mask prediction pairs for the source image datasets.

Description

[0001] Cross-reference to related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 061555, filed August 5, 2020, and to U.S. Patent Application No. 17 / 138251, filed December 30, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application generally relates to the field of domain adaptation technology, and more specifically, to a user-guided domain adaptation method and related apparatus. Background Technology

[0003] Reliable computer-aided segmentation of anatomical structures in medical images can provide quantitative biomarkers for disease diagnosis, prognosis, and progression. Given the extreme workload of fully labeled data (especially 3D volumes), much work has focused on weakly supervised segmentation solutions. Solutions that leverage user interactions (UIs), including extreme points, doodles, and boundary annotations, constitute an important category in this field.

[0004] The main challenge is effectively leveraging UI constraints or guiding mask generation. Classical methods, such as the Random Walk (RW) algorithm, can perform this by propagating seed regions using intensity similarity. With the advent of deep learning, reconciling mask prediction with UIs remains a challenge. Deep Extrema Points (DEXTR) is a popular and effective method that requires the user to click on the extreme boundary points of an object. However, DEXTR only adds extreme point annotations as additional channels when training the segmenter, meaning the predicted mask may be inconsistent with the UI. Therefore, it is necessary to develop a fast annotation method that uses minimal user interaction to enhance mask prediction or generation on target datasets or deployment scenarios. Summary of the Invention

[0005] One aspect or embodiment of this application provides a User-Guided Domain Adaptation (UGDA) method. The method includes: training an ensemble model using a source image dataset of labeled medical images by minimizing a supervised loss of the ensemble model, the ensemble model including a first fully convolutional network (FCN) and a second FCN linked to the first FCN to obtain one or more first shared weights of the first FCN and one or more second shared weights of the second FCN; generating extreme point predictions based on the one or more first shared weights and generating mask predictions based on the one or more second shared weights using the ensemble model for each source image dataset and target image dataset; training a discriminator by inputting extreme point-mask prediction pairs to each source image dataset and target image dataset and obtaining one or more discriminator weights by minimizing a discriminator loss; and fine-tuning the ensemble model by predicting extreme point-mask prediction pairs in the target image dataset to deceive the discriminator having the one or more discriminator weights, and by matching the distribution of extreme point-mask prediction pairs in the target image dataset with the distribution of extreme point-mask prediction pairs in the source image dataset.

[0006] Optionally, predicting the extreme point-mask prediction pair of the target image dataset includes: obtaining the extreme point prediction of the target image dataset; and using the obtained extreme point prediction as an anchor point to obtain the mask prediction of the target image dataset, wherein adversarial loss is used to guide the mask prediction to be correctly aligned with the obtained extreme point prediction.

[0007] Optionally, the adversarial loss is calculated according to the following formula: in, This represents the adversarial loss. Represents cross-entropy loss; This represents the cardinality of the target image dataset; This represents the volume of the target image dataset; This indicates the mask prediction. This indicates the prediction of the extreme point; The label indicates the extreme point-mask prediction pair from the source image dataset.

[0008] Optionally, the first FCN is configured to predict the extreme points of the input second FCN; and the second FCN is configured to predict the mask based on the extreme point prediction of the first FCN.

[0009] Optionally, the supervision loss is calculated based on the extreme point loss and the segmentation loss.

[0010] Optionally, the discriminator loss is calculated according to the following formula: in, This indicates the loss of the discriminator; Represents cross-entropy loss; This represents the cardinality of the source image dataset; This represents the cardinality of the target image dataset; This represents the volume of the source image dataset; This represents the volume of the target image dataset; This indicates the mask prediction. 0 represents the extreme point prediction; 1 and 0 represent labels, indicating extreme point-mask prediction pairs from the target image dataset and the source image dataset, respectively.

[0011] Optionally, the overall training of the UGDA method is to minimize the total loss calculated based on the supervised loss and the adversarial loss.

[0012] Optionally, the source image dataset includes the volumes of pathological organs and healthy organs; the target image dataset includes the volumes of pathological organs, the etiology / features of which are not visible in the source image dataset.

[0013] Optionally, the first FCN outputs multiple Gaussian heatmaps to the second FCN to predict a mask, each Gaussian heatmap corresponding to an extreme point.

[0014] Optionally, the target image dataset includes user-interaction (UI) labeled and unlabeled volumes; the source image dataset includes fully supervised volumes with masks.

[0015] Another aspect or embodiment of this application provides a user-guided domain adaptation (UGDA) device. The device includes a memory storing a computer program; and a processor coupled to the memory and configured to perform the following methods when the computer program is executed: using a source image dataset of labeled medical images, training the combined model by minimizing a supervised loss of the combined model, the combined model including a first fully convolutional network (FCN) and a second FCN linked to the first FCN to obtain one or more first shared weights of the first FCN and one or more second shared weights of the second FCN; for each source image dataset and target image dataset, generating extreme point predictions based on the one or more first shared weights and generating mask predictions based on the one or more second shared weights using the combined model; training a discriminator by inputting extreme point-mask prediction pairs to each source image dataset and target image dataset and obtaining one or more discriminator weights by minimizing a discriminator loss; and fine-tuning the combined model by predicting extreme point-mask prediction pairs of the target image dataset to deceive a discriminator having the one or more discriminator weights, and by matching the distribution of extreme point-mask prediction pairs of the target image dataset with the distribution of extreme point-mask prediction pairs of the source image dataset.

[0016] Optionally, predicting the extreme point-mask prediction pair of the target image dataset includes: obtaining the extreme point prediction of the target image dataset; and using the obtained extreme point prediction as an anchor point to obtain the mask prediction of the target image dataset, wherein adversarial loss is used to guide the mask prediction to be correctly aligned with the obtained extreme point prediction.

[0017] Optionally, the first FCN is configured to predict the extreme points of the input second FCN; and the second FCN is configured to predict the mask based on the extreme point prediction of the first FCN.

[0018] Optionally, the first FCN outputs multiple Gaussian heatmaps to the second FCN to predict a mask, with each Gaussian heatmap corresponding to an extreme point.

[0019] Another aspect or embodiment of this application provides a computer program product including a non-transitory computer-readable storage medium storing program instructions configured to be executed by a computer to enable the computer to implement: using a source image dataset of annotated medical images, training the combined model by minimizing the supervised loss of the combined model, the combined model including a first fully convolutional network (FCN) and a second FCN linked to the first FCN, to obtain one or more first shared weights of the first FCN and one or more second shared weights of the second FCN; using the combined model for each of the source image dataset and the target image dataset. The combined model generates extreme point predictions based on one or more first shared weights and generates mask predictions based on one or more second shared weights; a discriminator is trained by inputting extreme point-mask prediction pairs into each of the source image dataset and the target image dataset and obtaining one or more discriminator weights by minimizing the discriminator loss; and the combined model is fine-tuned by predicting extreme point-mask prediction pairs of the target image dataset to deceive the discriminator with the one or more discriminator weights, and by matching the distribution of extreme point-mask prediction pairs of the target image dataset with the distribution of extreme point-mask prediction pairs of the source image dataset.

[0020] Optionally, predicting the extreme point-mask prediction pair of the target image dataset includes: obtaining the extreme point prediction of the target image dataset; and using the obtained extreme point prediction as an anchor point to obtain the mask prediction of the target image dataset, wherein adversarial loss is used to guide the mask prediction to be correctly aligned with the obtained extreme point prediction.

[0021] Optionally, the first FCN is configured to predict the extreme points of the input second FCN; and the second FCN is configured to predict the mask based on the extreme point prediction of the first FCN.

[0022] Optionally, the first FCN outputs multiple Gaussian heatmaps to the second FCN to predict a mask, with each Gaussian heatmap corresponding to an extreme point.

[0023] Other aspects or embodiments of this application may be understood by those skilled in the art based on the description, claims and drawings of this application. Attached Figure Description

[0024] The accompanying drawings are merely illustrative examples based on various disclosed embodiments and are not intended to limit the scope of this application.

[0025] Figure 1 Exemplary block diagrams of user-guided domain adaptation (UGDA) methods according to various disclosed embodiments of this application are shown; Figure 2 A flowchart illustrating an exemplary training process for a user-guided domain adaptation (UGDA) method according to various disclosed embodiments of this application is shown; Figure 3 Exemplary images of real liver masks according to various disclosed embodiments of this application and exemplary images of liver masks predicted by different methods are shown; and Figure 4 Box plots of Dice-Sørensen coefficient (DSC) scores for pathological liver segmentation according to various disclosed embodiments of this application are shown. Detailed Implementation

[0026] For detailed reference, please refer to the exemplary embodiments disclosed herein, which are illustrated in the accompanying drawings. Where possible, the same reference numerals may be used in the drawings to denote the same or similar parts.

[0027] While examples and features of the disclosed principles have been described herein, modifications, adaptations, and other implementations may be made without departing from the spirit and scope of the disclosed embodiments. Furthermore, words such as “compose,” “own,” “comprise,” and “including,” and other similar forms, have equivalent meanings and are interpreted as open-ended; therefore, one or more items following any of these words do not imply an exhaustive list of one or more items, or are limited to only the listed one or more items. The singular form, however, includes the plural meaning unless the context clearly specifies otherwise.

[0028] Annotation of mask-based medical images, especially for 3D data, is a bottleneck in developing reliable machine learning models. Using user interactions (UIs) with minimal labor to guide annotation is promising, but challenges remain in optimally reconciling mask predictions with UIs. UIs may include extrema, doodles, boundary labels, etc. To address these technical issues, this application provides a User-Guided Domain Adaptation (UGDA) method that models the combined distribution of UIs and mask predictions using prediction-based adversarial domain adaptation (PADA). The UIs are then used as anchors to guide and align the mask predictions. Importantly, the UGDA method can learn from unlabeled data and model the high-level semantics behind different UIs. Furthermore, it should be noted that the UGDA method maintains state-of-the-art performance even when only a small subset of available UIs is seen, demonstrating the ability to achieve robust and reliable UI-guided segmentation with extremely low labor requirements.

[0029] The UGDA method described herein can guide mask prediction of UIs using prediction-based adversarial domain adaptation (PADA). An advantage of the UGDA method may be its ability to model the high-level meanings behind different types of UIs and how these UIs influence the final mask prediction. Importantly, UIs are used as anchor points when adjusting the mask. Another advantage of the UGDA method may be that, like PADA, it can learn and utilize completely unlabeled data in addition to the data accompanying the UIs. Without loss of generality, due to the intuitiveness and effectiveness of the embodiments in this application, it focuses on using extremum point UIs of the DEXTR type. However, other types of UIs (e.g., boundary corrections) may also be used as supplementary or alternative extremum points. One constraint is that, in order to model the interaction between the mask and the UIs, a fully supervised dataset is assumed to be available. However, such data can come from sources other than the target dataset, such as public data.

[0030] Various embodiments provide a medical image annotation method, apparatus, and computer program product for pathological organ segmentation using User-Guided Domain Adaptation (UGDA). Any organ of interest can be applied to the disclosed method, apparatus, and computer program product, although the liver is used as an example organ for illustrative purposes in this application.

[0031] Figure 1 Exemplary diagrams of user-guided domain adaptation (UGDA) methods according to various disclosed embodiments of this application are shown. References Figure 1 By providing only a minimal number of UIs, reliable mask predictions can be generated on the target image dataset or deployment scenario. More formally, assume that the target image dataset consists of volumes labeled with UIs and completely unlabeled volumes; ,in and Let UI represent the image and the extreme points, respectively. Furthermore, it is assumed that a fully supervised source image dataset with masks is also available. As long as the mask and extreme points describe the same anatomical structure, It could come from a completely different source, such as public data. Figure 1 The UGDA method (e.g., algorithm or framework) shown can be configured to efficiently and more effectively utilize extreme point UIs.

[0032] refer to Figure 1 The UGDA method can link a first (or initial) FCN (fully convolutional network) that predicts the extrema of an object (e.g., an image) with a second FCN that accepts extrema predictions from the first FCN to predict a mask. For the source image dataset containing mask labels... Fully supervised loss can be calculated for extreme point prediction and mask prediction. For the target image dataset... When UIs are available, the fully supervised loss can be calculated. For all The volume, whether UI-annotated or completely unannotated, can be used to guide mask prediction based on extreme point anchors.

[0033] Supervised workflow (or training) The backbone of the UGDA method comprises two linked 3D FCNs, where the first FCN (i.e., the extremum model) predicts extrema, and the second FCN (i.e., the segmentation model or segmenter) predicts the complete mask. Working backward, the second FCN may be similar to DEXTR, where the second FCN predicts a mask for a given input image and extremum UIs, as described below: Where s(.) is used to denote the second (i.e., the splitter) FCN.

[0034] Each of the multiple (e.g., 6) extreme points can be represented by a 3D Gaussian heatmap centered on the user click and presented to an additional input channel. It can be observed that the UGDA method may be insensitive to the size of the Gaussian heatmap, and a kernel with a standard deviation of 5 pixels can be used according to embodiments of this application. However, unlike DEXTR, it does not assume that all training volumes have extreme points, because Only specific target volumes have UIs that include extreme points.

[0035] To address the aforementioned issues, a first FCN can be used to predict the extreme point heatmap for each volume. Following heatmap regression conventions, the first FCN (i.e., the extreme point model) h(.) can output six 3D Gaussian heatmaps, each corresponding to an extreme point. These six 3D Gaussian heatmaps can be merged into a single channel before being input into the second FCN (i.e., the segmentation model or segmenter), as shown below: For simplicity, six heatmaps can be skipped to a single channel sum. Based on the above predictions, this method is feasible even for unlabeled image data.

[0036] Regarding losses, if Used to represent any input quantity related to extreme points in UIs, whether from... still Then the monitoring loss can be expressed as: in This represents the monitoring loss; This represents the segmentation loss of the second FCN; This represents the extreme point loss of the first FCN; express The cardinality; express The cardinality; Indicates mean square error; This represents the sum of cross-entropy and Dice-Sørensen coefficient (DSC) loss.

[0037] While focusing on the extreme points of this application, other types of UIs, such as boundary correction, can be easily incorporated into the above method according to the embodiments of this application.

[0038] User-guided domain adaptation Similar to DEXTR, Equation (6) indirectly guides mask prediction by using an extreme point heatmap as an additional input channel for the second FCN. Furthermore, for The volume of UI annotations in Equation (5) can encourage extreme point predictions to actually match UIs (e.g., annotated or actual extreme points). However, mask predictions may conflict with UIs because there is no penalty for inconsistencies between mask predictions and UIs. Therefore, an additional mechanism may be needed to align the mask with UIs. An adversarial domain adaptation method can be chosen to penalize inconsistent mask predictions. While image translation-based adversarial domain adaptation methods show ideal results, this approach may not be suitable for the current task because it involves adjusting the prediction space to produce a mask that is well aligned with UIs. Therefore, prediction-based adversarial domain adaptation (PADA) can be used in this application.

[0039] In some embodiments of this application, the discriminator d(.) can be used to learn the distribution and interaction between mask predictions and extreme point predictions. (The remaining text appears to be a fragment and doesn't translate directly.) The samples are considered to be of the "correct" distribution, and the discriminator loss can be expressed as: in This indicates the loss of the discriminator; Represents cross-entropy loss; This represents the cardinality of the source image dataset; This represents the cardinality of the target image dataset; This represents the volume of the source image dataset; This represents the volume of the target image dataset; This indicates the mask prediction. The extreme point prediction is represented by 0 and 1, which represent labels indicating extreme point-mask prediction pairs from the target image dataset and the source image dataset, respectively. The discriminator loss can be set to incentivize the discriminator to correctly distinguish whether the prediction comes from the source image dataset or the target image dataset.

[0040] Importantly, in order to model the combined distribution, the discriminator can accept both the UIs and the mask predictions simultaneously. After standard adversarial training, the gradient can flow only through the discriminator at this point. Then, UGDA can be used to predict... Extreme point / mask pairs are used to attempt to deceive the discriminator. These extreme point / mask pairs are... Distribution matching. More formally, for The volume in the image is configured with an adversarial loss, as shown below: in, This represents the adversarial loss; Represents cross-entropy loss; This represents the cardinality of the target image dataset; This represents the volume of the target image dataset; This indicates the mask prediction. The value represents the extreme point prediction; and 1 represents the label, indicating an (erroneous) extreme point / mask pair from the source distribution (e.g., the source image dataset). The adversarial loss can incentivize the combined model to generate extreme point / mask predictions for the target image dataset, which can deceive the discriminator into believing that such extreme point / mask predictions originate from the source image dataset.

[0041] It is worth noting that, compared with equation (7), The “label” has been switched from 0 to 1. Similar to the standard PADA setup, the gradient may not flow through the discriminator weights in Equation (8). Importantly, the gradient may also not flow through the extreme point prediction when the UIs are present. Therefore, when the UIs are available, the extreme point prediction may only be affected by the supervised loss in Equation (5) to match the UIs. Thus, the extreme point prediction can serve as an anchor point, and the adversarial loss in Equation (8) can guide the mask prediction to align correctly with the UIs. This alignment may not simply be about making the mask prediction consistent with the UIs, as PADA can also guide the mask region away from the UIs by modeling interactions. Finally, using PADA can provide another important benefit because The completely unlabeled volumes can seamlessly facilitate the learning process in equations (7) and (8). In fact, UGDA can be viewed as integrating a domain-adaptive learning process in addition to the DEXTR-type guidance from UIs. Therefore, the overall training objective of the UGDA method may be to minimize the following total loss: in, These are the weights used to balance the supervised loss and the adversarial loss; and the loss weights are retained only on the adversarial component to reduce hyperparameter fine-tuning.

[0042] In some embodiments of this application, a target image dataset of 927 venous phase computed tomography (CT) scans from the Picture Archiving and Communication System (PACS) of Chang Gung Memorial Hospital (CGMH) can be used. The UGDA method was tested on segmented pathological organs of interest (e.g., the liver). Selection criteria might include biomarking or resection of liver lesions in patients, with CT scans performed within one month prior to surgery. Patients directly reflect clinical distribution, presenting with hepatocellular carcinoma (HCC), intrahepatic cholangiocarcinoma (ICC), benign or metastatic lesions, accompanied by conditions such as liver fibrosis, splenomegaly, or embolic lesions. From the aforementioned image dataset, 47 and 100 studies were selected as validation and test sets, respectively, to delineate patient livers. The aforementioned 147 CTs can be named evaluation volumes. The remainder of the aforementioned image dataset can only be annotated using extreme points (UIs). Source image dataset This can include 235 fully annotated venous phase CT studies collected from public datasets, and... The difference is that this dataset includes both healthy and pathological livers, and represents only HCC and metastatic tumors. The corresponding extreme points "UIs" can be generated from the complete mask. For internal validation, the source dataset can be... The weights are divided into 70%, 20%, and 10%, which are used for training, testing, and validation, respectively.

[0043] In some embodiments of this application, a 3D depth-supervised progressive global nested network (PHNN) can be configured for the two FCN architectures of the UGDA method. The PHNN provides an efficient pipeline without a decoder. For example, equation (4) can be used first in The fully supervised baseline is trained and then fine-tuned after convergence using Equation (9). The dual PHNN baseline can be very powerful on the public data used. A DSC score of approximately 96.9% is achieved on the test set. For the discriminator, a popular 3D architecture with dilated convolutions can be configured, which has proven to be a useful liver mask discriminator.

[0044] In some embodiments of this application, the UGDA method can be used to evaluate UI annotations using only extreme points. How effective is it? Evaluating the volume and its extrema (UIs) can be included during training, but the corresponding masks may be hidden. For evaluation, the DSC score and Mask Extrema Consistency (MXA) can be measured. MXA measures the average distance between all six predicted mask extrema and the true extrema. In this way, it directly measures the effectiveness of the method (e.g., a combined model) in producing mask predictions that match the extrema UIs.

[0045] Table 1 summarizes all variables in the annotation. Time performance. It can be seen that, compared to... Compared to the performance of the public liver dataset, the performance of the fully supervised dual PHNN decreased from approximately 96.9% to approximately 93.0% due to the major differences between the public liver dataset and the PACS-based clinical target dataset used.

[0046] Table 1: Mean and standard deviation scores of DSC and MXA, and UI annotations used for training Fraction of volume Referring to Table 1, DEXTR significantly improves MXA by using UIs, but its less-than-ideal DSC score may indicate that while the resulting mask aligns better with extrema, it may still fail to accurately capture the liver extent. On the other hand, mask-based PADA variants outperform both PHNN and DEXTR, suggesting that modeling mask distributions on top of UI guidance of the DEXTR type can provide more reliable annotation. Finally, the UGDA method exhibits the highest performance among all the aforementioned models (or methods), suggesting that modeling the interaction between UIs and mask predictions may further improve performance. Importantly, the MXA of the UGDA method is highly desirable (e.g., 1.1 mm), which likely indicates that the mask predictions match the UIs very well.

[0047] Figure 2 An exemplary training process flowchart of a user-guided domain adaptation (UGDA) method according to various embodiments of this application is shown.

[0048] In S202, a combined model is trained using medical images from a labeled source image dataset (Ds). One or more first shared weights of a first FCN and one or more second shared weights of a second FCN are obtained by minimizing the supervised loss of the combined model. The combined model may include the first FCN (e.g., an extremum model) and the second FCN (e.g., a segmentation model or segmenter). The extremum loss can be based on... ) and the segmentation loss ( ) Calculate the supervision loss ( In the exemplary UGDA method of this application, the first FCN can be linked with the second FCN as the backbone of the UGDA method, where h(.) represents the first FCN, which outputs six Gaussian heatmaps, each corresponding to an extremum point; and s(.) represents the second FCN, which predicts a mask for a given input image and the extremum points. The first FCN can be configured to predict the extremum points input to the second FCN; and the second FCN can be configured to predict the mask based on the extremum point predictions from the first FCN. The supervision loss of the combined model can be calculated and minimized. The first FCN can obtain one or more first shared weights, and the second FCN can obtain one or more second shared weights, by training the combined model.

[0049] In one embodiment of this application, the combined model may have two FCNs, including a first FCN and a second FCN configured in two stages. The contribution of the first stage is to generate extreme point predictions (e.g., UI-based predictions) given the input image. The combined model may also allow for other minimal-labor UIs, including graffiti or boundary correction. The contribution of the second stage is to generate mask predictions given the same input image and the extreme point predictions of the first stage. Obtaining UI-based supervision (such as extreme point predictions) is very easy, so the first stage can be trained from the target image dataset and / or the source image dataset. However, mask predictions are more difficult to obtain, so the second FCN can be trained using only the real masks from the source image dataset.

[0050] After training the combined model in S202, S204 can be performed. In S204, the combined model can be used to generate extreme point predictions based on one or more first shared weights for each of the source image datasets and the target image datasets, and to generate mask predictions based on one or more second shared weights.

[0051] According to the combined model of this application, the extreme point-mask prediction pairs (or prediction pairs) of the source image dataset and the target image dataset, including the extreme point prediction and the mask prediction, can be input into the discriminator. The discriminator can be trained using equation (7) to obtain the minimum discriminator loss. This allows for the optimization of one or more discriminator weights.

[0052] In S206, the combined model is fine-tuned by predicting the extreme point-mask prediction pairs of the target image dataset to deceive the discriminator having the weights of the one or more discriminators, and by matching the distribution of the extreme point-mask prediction pairs of the target image dataset with the distribution of the extreme point-mask prediction pairs of the source image dataset. In some embodiments of this application, the extreme point predictions of the target image dataset are used as anchors, and the adversarial loss shown in Equation (8) can be used to guide the mask predictions to be correctly aligned with the extreme point predictions, thereby labeling the target image dataset.

[0053] In one embodiment, to model the interaction between the extreme point prediction and the corresponding mask, both the extreme point prediction and the corresponding mask can be input into an adversarial learning process in pairs, and this process can be used to generate masks that are better aligned with the extreme point prediction. Therefore, technically, the extreme point prediction can be input into a discriminator and then used as an anchor. However, when labeled (e.g., actual) extreme points (uis) are available, the extreme point prediction itself may be directly supervised by the labeled extreme points, thus the labeled extreme points will indirectly serve as anchors for mask prediction.

[0054] In various embodiments of this application, the main reasons for inputting extreme point predictions instead of the labeled extreme points themselves may include the following. It is not strictly necessary that all data samples in the target image dataset have associated, labeled extreme points. To make the UGDA framework more flexible, it is desirable for the UGDA framework to function correctly when only a portion of the target image dataset has extreme point supervision. Therefore, to allow the UGDA framework to still learn effectively from completely unsupervised target image data, extreme point predictions can be input into the discriminator. Such extreme point predictions are not directly supervised by labeled (e.g., actual) extreme points, and therefore can produce ideal extreme points relying solely on the accuracy of the first-stage FCN. These unsupervised data may only have an adversarial effect on them. The number of extreme points (e.g., extreme point UI percentage) in the target image dataset required for the UGDA framework to function can be evaluated according to various embodiments of this application.

[0055] Figure 3 Exemplary images illustrating the realism and prediction of liver masks using different methods according to various disclosed embodiments of this application are shown. References Figure 3 Contour 302 presents the realistic liver mask, and mask 304 presents the predicted liver mask. A Gaussian heatmap centered on the extreme points UIs is shown as a solid black circle. It can be seen that the UGDA method can significantly improve the alignment between the mask prediction and the extreme points UIs. Arrows highlight the selected baseline prediction error corrected by the UGDA method. Figure 3Qualitative examples can enhance quantitative improvements. In particular, the UGDA method ensures that mask predictions are consistent with extrema and provide robust predictions of UIs far from extrema.

[0056] Figure 4 Box plots of Dice-Sørensen coefficient (DSC) scores for pathological liver segmentation according to various disclosed embodiments of this application are shown. Figure 4 The box plot supports the average score of MXA, indicating that the UGDA method can provide a significant improvement in reliability, achieving a worst-case performance of 94.9% DSC compared to 93.2% for the mask-based PADA (with limit point UIs) variant.

[0057] In addition, when When only a portion of the UI is annotated, the UGDA method performs almost equally well, outperforming both DEXTR and the mask-based PADA variant, both of which see 100% of the extreme UIs. These results likely indicate that the UGDA method works well even in scenarios with very few UI annotations, providing further evidence of its significant high generality.

[0058] Using only the extreme point UIs, approximately 96.1% (94.5%) of the state-of-the-art (worst-case) Dice-Sørensen coefficient (DSC) scores are achieved on the aforementioned image dataset, compared to approximately 93.0% (79.0%) for the strongly fully supervised baseline and approximately 93.1% (87.0%) for DEXTR. It can be demonstrated that the UGDA method improves the DSC by approximately 1.3% compared to PADA, and remains robust even when only an incomplete set of extreme point UIs is displayed. Finally, it can also be demonstrated that the predicted mask aligns very well with the extreme point UIs, allowing users to interact with high confidence and minimal frustration.

[0059] According to various embodiments of this application, the UGDA method aligns mask predictions with extreme point UIs (used as anchor points). Importantly, the UGDA method can learn and utilize completely unlabeled data and model the high-level semantics behind different UIs. Furthermore, the UGDA method maintains high performance even when only a small subset of available UIs are seen, demonstrating its ability to achieve robust and reliable UI-guided segmentation with extremely low labor requirements.

[0060] This application also provides a device for user-guided domain adaptation (UGDA). The device includes a memory storing a computer program; and a processor coupled to the memory and configured to perform a method, when the computer program is executed, comprising: training an ensemble model using a source image dataset of labeled medical images by minimizing a supervised loss of the ensemble model, the ensemble model including a first fully convolutional network (FCN) and a second FCN linked to the first FCN to obtain one or more first shared weights of the first FCN and one or more second shared weights of the second FCN; generating extreme point predictions based on the one or more first shared weights and generating mask predictions based on the one or more second shared weights using the ensemble model for each source image dataset and target image dataset; training a discriminator by inputting extreme point-mask prediction pairs for each source image dataset and target image dataset and obtaining one or more discriminator weights by minimizing a discriminator loss; and fine-tuning the ensemble model by predicting extreme point-mask prediction pairs of the target image dataset to deceive a discriminator having the one or more discriminator weights and by matching the distribution of extreme point-mask prediction pairs of the target image dataset with the distribution of extreme point-mask prediction pairs of the source image dataset.

[0061] This application also provides a computer program product including a non-transitory program instruction stored therein, the program instruction being configured to be executed by a computer to enable the computer to perform the following operations: using a source image dataset of labeled medical images, training the combined model by minimizing a supervised loss of the combined model, the combined model including a first fully convolutional network (FCN) and a second FCN linked to the first FCN to obtain one or more first shared weights of the first FCN and one or more second shared weights of the second FCN; for each source image dataset and target image dataset, using the combined model to generate extreme point predictions based on the one or more first shared weights and to generate mask predictions based on the one or more second shared weights; training a discriminator by inputting extreme point-mask prediction pairs to each source image dataset and the target image dataset and by minimizing a discriminator loss to obtain one or more discriminator weights; and fine-tuning the combined model by predicting extreme point-mask prediction pairs of the target image dataset to deceive a discriminator having the one or more discriminator weights, and by matching the distribution of extreme point-mask prediction pairs of the target image dataset with the distribution of extreme point-mask prediction pairs of the source image dataset.

[0062] While this disclosure has been described with respect to one or more embodiments, changes and / or modifications may be made to the illustrated examples without departing from the spirit and scope of the appended claims. Furthermore, while specific features of the invention may be disclosed only with respect to one of several embodiments, such features may be desired and advantageous in combination with one or more other features of other implementations for any given or particular function. Additionally, within the scope of the terms “comprising,” “including,” “having,” “possessing,” “with,” or variations thereof used in the detailed description and claims, these terms are intended to encompass in a manner similar to the term “comprising.” The term “at least one” is used to indicate that one or more of the listed items may be selected.

[0063] While the numerical ranges and parameters described in this application are approximate, the values ​​illustrated in the specific examples should be reported as precisely as possible. However, any numerical value inherently contains a certain degree of error, which is necessarily due to the standard deviation found in the respective test measurements. Furthermore, all ranges disclosed herein should be understood to include any and all subranges contained herein. In some cases, the numerical values ​​specified for parameters may be negative.

[0064] In light of the specification and practice disclosed herein, other embodiments of this disclosure will be apparent to those skilled in the art. This specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

Claims

1. A user-guided domain-adaptive UGDA method, comprising: Using a source image dataset of labeled medical images, the combined model is trained by minimizing the supervised loss of the combined model, which includes a first fully convolutional network FCN and a second FCN linked to the first FCN, to obtain one or more first shared weights of the first FCN and one or more second shared weights of the second FCN. For each of the source image dataset and the target image dataset, an extremum prediction is generated based on one or more first shared weights using a combined model, and a mask prediction is generated based on one or more second shared weights; wherein, the target image dataset includes volumes labeled and unlabeled by the user interaction UI; The extreme point-mask predictions for each of the source image datasets and the target image datasets are input to the discriminator, and the discriminator is trained by minimizing the discriminator loss to obtain one or more discriminator weights; and The combined model is fine-tuned by predicting extreme point-mask prediction pairs of the target image dataset to deceive a discriminator with the weights of the one or more discriminators, and by matching the distribution of extreme point-mask prediction pairs of the target image dataset with the distribution of extreme point-mask prediction pairs of the source image dataset.

2. The method of claim 1, wherein, Predicting the extreme point-mask prediction pairs of the target image dataset includes: Obtain the extreme point predictions of the target image dataset; and The obtained extreme point predictions are used as anchor points to obtain mask predictions for the target image dataset, wherein adversarial loss is used to guide the mask predictions to be correctly aligned with the obtained extreme point predictions.

3. The method as described in claim 2, wherein, The adversarial loss is calculated according to the following formula: in, This refers to the discriminator. This represents the adversarial loss. Represents cross-entropy loss; This represents the cardinality of the target image dataset; This represents the volume of the target image dataset; This indicates the mask prediction. This indicates the prediction of the extreme point; The label indicates the extreme point-mask prediction pair from the source image dataset.

4. The method of claim 1, wherein, The first FCN is configured to predict the extreme points of the input to the second FCN; and The second FCN is configured to predict the mask based on the extreme point prediction of the first FCN.

5. The method of claim 1, wherein, The supervised loss represents the sum of the extreme point loss and the segmentation loss.

6. The method of claim 1, wherein, The discriminant loss is calculated according to the following formula: in, This refers to the discriminator. This indicates the loss of the discriminator; Represents cross-entropy loss; This represents the cardinality of the source image dataset; This represents the cardinality of the target image dataset; This represents the volume of the source image dataset; This represents the volume of the target image dataset; This indicates the mask prediction. The extreme point prediction is represented by 0 and 1, which represent labels indicating extreme point-mask prediction pairs from the target image dataset and the source image dataset, respectively.

7. The method of claim 3, wherein, The overall training of the UGDA method is to minimize the total loss calculated based on the supervised loss and the adversarial loss.

8. The method of claim 1, wherein, The source image dataset includes the volumes of pathological and healthy organs; the target image dataset includes the volumes of pathological organs, the causes / features of which do not exist in the source image dataset.

9. The method of claim 1, wherein, The first FCN outputs multiple Gaussian heatmaps to the second FCN to predict a mask, and each Gaussian heatmap corresponds to an extreme point.

10. The method of claim 1, wherein, The source image dataset comprises a fully supervised volume with masks.

11. A user-guided domain-adaptive UGDA device, comprising: A memory, wherein a computer program is stored; and A processor, coupled to and configured to implement the following methods when the computer program is executed: Using a source image dataset of labeled medical images, the combined model is trained by minimizing the supervised loss of the combined model, which includes a first fully convolutional network FCN and a second FCN linked to the first FCN, to obtain one or more first shared weights of the first FCN and one or more second shared weights of the second FCN. For each of the source image dataset and the target image dataset, an extremum prediction is generated based on one or more first shared weights using a combined model, and a mask prediction is generated based on one or more second shared weights; wherein, the target image dataset includes volumes labeled and unlabeled by the user interaction UI; The extreme point-mask predictions for each of the source image datasets and the target image datasets are input to the discriminator, and the discriminator is trained by minimizing the discriminator loss to obtain one or more discriminator weights; and The combined model is fine-tuned by predicting extreme point-mask prediction pairs of the target image dataset to deceive a discriminator with the weights of the one or more discriminators, and by matching the distribution of extreme point-mask prediction pairs of the target image dataset with the distribution of extreme point-mask prediction pairs of the source image dataset.

12. The device of claim 11, wherein, Predicting the extreme point-mask prediction pairs of the target image dataset includes: Obtain the extreme point predictions of the target image dataset; and The obtained extreme point predictions are used as anchor points to obtain mask predictions for the target image dataset, wherein adversarial loss is used to guide the mask predictions to be correctly aligned with the obtained extreme point predictions.

13. The device as claimed in claim 11, wherein, The first FCN is configured to predict the extreme points of the input second FCN; and the second FCN is configured to predict the mask based on the extreme point prediction of the first FCN.

14. The device as claimed in claim 11, wherein, The first FCN outputs multiple Gaussian heatmaps to the second FCN to predict a mask, and each Gaussian heatmap corresponds to an extreme point.

15. A computer program product comprising a non-volatile computer-readable storage medium storing program instructions configured to be executed by a computer to cause the computer to perform the following operations: Using a source image dataset of labeled medical images, the combined model is trained by minimizing the supervised loss of the combined model, which includes a first fully convolutional network FCN and a second FCN linked to the first FCN, to obtain one or more first shared weights of the first FCN and one or more second shared weights of the second FCN. For each source image dataset and target image dataset, a combined model is used to generate extreme point predictions based on one or more first shared weights, and to generate mask predictions based on one or more second shared weights; wherein, The target image dataset includes both labeled and unlabeled volumes from the user interaction UI; The extreme point-mask predictions for each of the source image datasets and the target image datasets are input to the discriminator, and the discriminator is trained by minimizing the discriminator loss to obtain one or more discriminator weights; and The combined model is fine-tuned by predicting extreme point-mask prediction pairs of the target image dataset to deceive a discriminator with the weights of the one or more discriminators, and by matching the distribution of extreme point-mask prediction pairs of the target image dataset with the distribution of extreme point-mask prediction pairs of the source image dataset.

16. The product of claim 15, wherein, Predicting the extreme point-mask prediction pairs of the target image dataset includes: Obtain the extreme point predictions of the target image dataset; and The obtained extreme point predictions are used as anchor points to obtain mask predictions for the target image dataset, wherein adversarial loss is used to guide the mask predictions to be correctly aligned with the obtained extreme point predictions.

17. The product as claimed in claim 15, wherein, The first FCN is configured to predict the extreme points of the input second FCN; and the second FCN is configured to predict the mask based on the extreme point prediction of the first FCN.

18. The product as claimed in claim 15, wherein, The first FCN outputs multiple Gaussian heatmaps to the second FCN to predict a mask, and each Gaussian heatmap corresponds to an extreme point.

Images