Method and system for efficient learning of annotations for medical image analysis

Abstract

Embodiments of the present disclosure provide a method and system for efficient learning of annotations for medical image analysis. The system receives a medical image acquired by an image acquisition device. The system may also additionally include at least one processor configured to apply the learning model to perform an image analysis task on the medical image. The learning model and an error estimator are jointly trained using training images including a first set of labeled images and a second set of unlabeled images. The error estimator is configured to estimate an error of the learning model associated with performing the image analysis task.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 161,781, filed on March 16, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to systems and methods for analyzing medical images, and more particularly, to systems and methods for training image analysis learning models using an error estimator to improve the performance of the learning models due to a lack of labels in the training images. Background Art

[0004] Machine learning techniques have demonstrated promising performance in medical image analysis. For example, machine learning models are used to segment or classify medical images or detect objects, such as tumors, within medical images. However, to obtain accurate machine learning models—those with low prediction errors—the training process typically requires a large amount of annotated data, such as labeled images.

[0005] Obtaining annotations for training is time-consuming and laborious, especially for medical images. For example, in the three-dimensional (3D) medical image segmentation problem, voxel-level annotations need to be obtained, which is very time-consuming, especially for high-dimensional and high-resolution volumetric medical images such as thin-slice CT. In addition, the boundaries of the segmented targets are often irregular and fuzzy, which makes detailed voxel-level delineation challenging even for experienced radiologists. For example, lesion regions such as pneumonia lesions in the lungs have irregular and fuzzy boundaries. Therefore, there is an unmet need for a learning framework for medical image analysis with low annotation cost.

[0006] Embodiments of the present disclosure address the aforementioned problems by providing methods and systems for training image analysis learning models using error estimators that augment labeled training images, thereby improving the performance of the learning models. Summary of the Invention

[0007] New systems and methods are disclosed for training a learning model for analyzing medical images using an error estimator and applying the trained model to image analysis.

[0008] In one aspect, an embodiment of the present disclosure provides a system for analyzing medical images using a learning model. The system may include a communication interface configured to receive a medical image acquired by an image acquisition device. The system may additionally include at least one processor configured to apply the learning model to perform an image analysis task on the medical image. The learning model and an error estimator are jointly trained using training images including a first set of labeled images and a second set of unlabeled images. The error estimator is configured to estimate an error of the learning model associated with performing the image analysis task.

[0009] In another aspect, embodiments of the present disclosure further provide a computer-implemented method for analyzing medical images using a learning model. The method may include receiving, via a communication interface, a medical image acquired by an image acquisition device. The method may also include applying, via at least one processor, the learning model to perform an image analysis task on the medical image. The learning model is trained jointly with an error estimator using training images comprising a first set of labeled images and a second set of unlabeled images. The error estimator is configured to estimate an error of the learning model associated with performing the image analysis task.

[0010] In yet another aspect, embodiments of the present disclosure further provide a non-transitory computer-readable medium having a computer program stored thereon. When executed by at least one processor, the computer program performs a method for analyzing medical images using a learning model. The method may include receiving a medical image acquired by an image acquisition device. The method may also include applying the learning model to perform an image analysis task on the medical image. The learning model and an error estimator are jointly trained using training images comprising a first set of labeled images and a second set of unlabeled images. The error estimator is configured to estimate an error of the learning model associated with performing the image analysis task.

[0011] In some embodiments, the learning model and the error estimator can be trained by: training an initial version of the learning model and the error estimator using the first set of labeled images; applying the error estimator to the second set of unlabeled images to determine corresponding errors associated with the unlabeled images; determining a third set of labeled images from the second set of unlabeled images based on the corresponding errors; and training an updated version of the learning model using a combination of the first set of labeled images and the third set of labeled images.

[0012] In some embodiments, the image analysis task is an image segmentation task, and the learning model is configured to predict a segmentation mask. The error estimator is correspondingly configured to estimate an error map of the segmentation mask.

[0013] In some embodiments, the image analysis task is an image classification task, the learning model is configured to predict a classification label, and the error estimator is correspondingly configured to estimate a classification error between the classification label predicted by the learning model and a ground truth label included in the labeled image.

[0014] In some embodiments, the image analysis task is an object detection task, and the learning model is configured to detect an object from the medical image, for example, by predicting a bounding box surrounding the object and a classification label for the object. The error estimator is correspondingly configured to estimate a localization error between the predicted bounding box and a ground truth bounding box included in the labeled image, or a classification error between the classification label predicted by the learning model and a ground truth label included in the labeled image.

[0015] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Three exemplary segmented images of the lung region are illustrated.

[0017] Figure 2 A schematic diagram of an exemplary image analysis system according to certain embodiments of the present disclosure is illustrated.

[0018] Figure 3 A schematic diagram of a model training apparatus according to certain embodiments of the present disclosure is illustrated.

[0019] Figure 4A A schematic overview of a workflow performed by a model training apparatus to train a main model and an error estimator using labeled images according to certain embodiments of the present disclosure is illustrated.

[0020] Figure 4B A schematic overview of another workflow performed by a model training apparatus to enhance training data by deploying a main model and an error estimator on unlabeled images according to certain embodiments of the present disclosure is illustrated.

[0021] Figure 5 A schematic overview of a training workflow performed by a model training apparatus according to certain embodiments of the present disclosure is illustrated.

[0022] Figure 6 is a flowchart of an example method for training a main model and an error estimator for performing an image analysis task using labeled and unlabeled training data according to certain embodiments of the present disclosure.

[0023] Figure 7A A schematic overview of a workflow performed by a model training apparatus to train an image classification model and an error estimator using labeled images according to certain embodiments of the present disclosure is illustrated.

[0024] Figure 7B A schematic overview of another workflow performed by a model training apparatus to augment training data by deploying an image classification model and an error estimator on unlabeled images according to certain embodiments of the present disclosure is illustrated.

[0025] Figure 8 is a flowchart of an example method for training an image classification model and an error estimator for performing an image classification task using labeled and unlabeled training data, according to certain embodiments of the present disclosure.

[0026] Figure 9A A schematic overview of a workflow performed by a model training apparatus to train an object detection model and an error estimator using labeled images according to certain embodiments of the present disclosure is illustrated.

[0027] Figure 9B Illustrated is a schematic overview of another workflow performed by a model training apparatus to augment training data by deploying an object detection model and an error estimator on unlabeled images, according to certain embodiments of the present disclosure.

[0028] Figure 10 is a flowchart of an example method for training an object detection model and an error estimator for performing an object detection task using labeled and unlabeled training data according to certain embodiments of the present disclosure.

[0029] Figure 11A A schematic overview of a workflow performed by a model training apparatus to train an image segmentation model and an error estimator using labeled images according to certain embodiments of the present disclosure is illustrated.

[0030] Figure 11B A schematic diagram illustrating another workflow performed by a model training apparatus to augment training data by deploying an image segmentation model and an error estimator on unlabeled images according to certain embodiments of the present disclosure is illustrated.

[0031] Figure 12 is a flowchart of an example method for training an image segmentation model and an error estimator for performing an image segmentation task using labeled and unlabeled training data, according to certain embodiments of the present disclosure.

[0032] Figure 13is a flowchart of an example method for performing an image task on a medical image using a learning model trained with an error estimator, according to certain embodiments of the present disclosure. DETAILED DESCRIPTION

[0033] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings.

[0034] The present disclosure provides an image analysis system and method for analyzing medical images acquired by an image acquisition device. The image analysis system and method utilize a novel error estimation model to improve learning model training at a low annotation cost. The error estimation model automatically predicts the error in the output of the current learning model on unlabeled samples and guides the learning model to improve training by adding unlabeled samples with low prediction errors to the training dataset and requesting annotation of unlabeled samples with high prediction errors.

[0035] In some embodiments, the training images used to train the learning model include a first set of labeled images and a second set of unlabeled images. The systems and methods first train the learning model and error estimator using the first set of labeled images. The learning model is trained to perform an image analysis task, and the error estimator is trained to estimate the error of the learning model associated with performing the image analysis task. The error estimator is then applied to the second set of unlabeled images to determine corresponding errors associated with the unlabeled images, and a third set of labeled images is determined from the second set of unlabeled images based on the corresponding errors. The first set of labeled images is then combined with the third set of labeled images to train an updated learning model.

[0036] The error estimation model of the present disclosure is intended to predict the difference between the output of the main model and the basic ground truth, that is, the error predicted by the main model. It learns the error pattern of the main model and predicts the possible error of even unseen unlabeled data. Using the error estimation model, the disclosed system and method are therefore able to select unlabeled samples with possible low prediction errors from the main learning model to add to the training data set and expand the training data, thereby improving training and leading to improved performance and generalization ability of the learning model. In some embodiments, they are also able to select the unlabeled samples with possible high prediction errors to request manual annotation, thereby providing the most informative annotations to the main learning model. This results in maximizing the use of limited manual annotation resources. When the annotation task is intensive (such as voxel annotation of a segmentation model), the image can be segmented into smaller blocks or regions of interest (ROIs) for sparse labeling.

[0037] Furthermore, the disclosed scheme allows a separate error estimator to be trained to learn the complex error patterns of any master model. Compared to the limited built-in error estimation functions of certain master models (which only capture specific types of errors under strict assumptions), the disclosed scheme allows for greater flexibility and more thorough error estimation.

[0038] The disclosed system and method can be applied to any medical image analysis task (e.g., classification, detection, segmentation, etc.) on any image modality (e.g., CT, X-ray, MRI, PET, ultrasound, etc.). Taking the segmentation task as an example, obtaining voxel-level annotations for training purposes is very time-consuming. For example, Figure 1 Three exemplary images of the lung region extracted from a 3D chest CT image are shown. Figure 1 Each 2D image shown in contains an annotated region of interest (ROI) for the lung region. The lung regions shown in these images were confirmed to be infected with COVID-19 by a positive RT-PCR test. As can be seen, the boundaries of the pneumonia regions are irregular and fuzzy, making detailed voxel-level delineation challenging even for experienced radiologists. Therefore, there is a need for an improved training system and method for training learning models for medical image analysis with low annotation cost.

[0039] Although Figure 1 Medical images from 3D chest CT scans are shown, but in some embodiments, the disclosed image analysis system can also perform image analysis on images acquired using other suitable imaging modalities, including, for example, magnetic resonance imaging (MRI), functional MRI (e.g., fMRI, DCE-MRI, and diffusion MRI), positron emission tomography (PET), single photon emission computed tomography (SPECT), X-ray, optical coherence tomography (OCT), fluorescence imaging, ultrasound imaging, radiotherapy portal imaging, etc. The present disclosure is not limited to any particular type of image.

[0040] Figure 2 FIG. 2 shows an exemplary image analysis system 200 according to some embodiments of the present disclosure. Figure 2 As shown, the image analysis system 200 may include components for performing two phases (a training phase and a prediction phase). The prediction phase may also be referred to as an inference phase. To perform the training phase, the image analysis system 200 may include a training database 201 and a model training device 202. To perform the prediction phase, the image analysis system 200 may include an image analysis device 203 and a medical image database 204. In some embodiments, as Figure 2 As shown, image analysis system 200 may include more or fewer components.

[0041] Consistent with the present disclosure, the image analysis system 200 can be configured to analyze biomedical images acquired by the image acquisition device 205 and perform diagnostic prognostics based on the image analysis. In some embodiments, the image acquisition device 205 can be a CT scanner that acquires 2D or 3D CT images. For example, the image acquisition device 205 can be a 3D cone CT scanner for volumetric CT scanning. In some embodiments, the image acquisition device 205 can utilize one or more other imaging modalities, including, for example, magnetic resonance imaging (MRI), functional MRI (e.g., fMRI, DCE-MRI, and diffusion MRI), positron emission tomography (PET), single photon emission computed tomography (SPECT), X-ray, optical coherence tomography (OCT), fluorescence imaging, ultrasound imaging, radiotherapy portal imaging, and the like.

[0042] In some embodiments, the image acquisition device 205 can capture medical images containing at least one anatomical structure or organ, such as the lungs or chest cavity. For example, each volumetric CT examination can contain 51 to 1094 CT slices, with slice thickness ranging from 0.5 mm to 3 mm. The reconstruction matrix can have 512×512 pixels, with an in-plane pixel spatial resolution of 0.29×0.29 mm. 2 to 0.98×0.98mm 2 .

[0043] In some embodiments, the acquired image can be sent to a labeling station 301 for labeling at least a subset of the image. In some embodiments, the labeling station 301 can be operated by a user to provide manual labeling. For example, the user can use the keyboard, mouse, or other input interface of the labeling station 301 to label the image, such as drawing the boundary line of an object in the image, or identifying what anatomical structure the object is. In some embodiments, the labeling station 301 can execute an automatic or semi-automatic labeling program to label the image. The labeled image can be included as part of the training data provided to the model training device 202.

[0044] The image analysis system 200 may optionally include a network 206 to facilitate communication between various components of the image analysis system 200 (e.g., databases 201 and 204, devices 202, 203, and 205). For example, the network 206 may be a local area network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service), a client server, a wide area network (WAN), etc. In some embodiments, the network 206 may be replaced by a wired data communication system or device.

[0045] In some embodiments, the various components of the image analysis system 200 may be remote from one another or located in different locations. Figure 2 206 . In some alternative embodiments, certain components of image analysis system 200 may be located at the same site or within a single device. For example, training database 201 may be located at or as part of model training device 202 . As another example, model training device 202 and image analysis device 203 may be located within the same computer or processing device.

[0046] The model training device 202 can use the training data received from the training database 201 to train a learning model (also referred to as a master learning model) for performing image analysis tasks on medical images received from, for example, the medical image database 204. Figure 2 As shown, the model training device 202 can communicate with the training database 201 to receive one or more sets of training data. In some embodiments, the training data may include a first subset of labeled data, such as labeled images, and a second subset of unlabeled data, such as unlabeled images. "Labeled data" is training data that includes ground truth results obtained by manual annotation and / or automatic annotation procedures. For example, for an image segmentation task, the labeled data includes pairs of original images and corresponding ground truth segmentation masks for these images. As another example, for an image classification task, the labeled data includes pairs of original images and corresponding ground truth category labels for these images. On the other hand, "unlabeled data" is training data that does not include ground truth results. Throughout the disclosure, labeled data / images may also be referred to as labeled data / images, and unlabeled data / images may also be referred to as unlabeled data / images.

[0047] Consistent with the present disclosure, an error estimation model (also referred to as an error estimator) is trained with labeled data together with the main learning model to learn the error pattern of the main model. The trained error estimation model is then deployed to predict the possible errors of unlabeled data. Based on this error prediction, unlabeled data with possible low prediction errors can be labeled using the main learning model and then added to the labeled data to expand the training data. On the other hand, unlabeled data with possible high prediction errors can be sent for manual labeling, and the manually labeled data can also be added to the training data. The main learning model can then be trained using the expanded training data, thereby improving the performance and generalization ability of the learning model.

[0048] In some embodiments, the training phase can be performed "online" or "offline." "Online" training refers to performing the training phase and the prediction phase simultaneously, for example, learning the model in real time before analyzing the medical image. "Online" training can have the benefit of obtaining an up-to-date learned model based on the training data available at the time. However, if the training data is large and / or the model is complex, "online" training can be computationally expensive to perform and may not always be feasible. Consistent with the present disclosure, "offline" training is used, in which the training phase is performed separately from the prediction phase. The learned model after offline training is saved and reused for analyzing images.

[0049] The model training device 202 can be implemented with hardware specifically programmed by software that performs a training process. For example, the model training device 202 can include a processor and a non-transitory computer-readable medium. The processor can perform training by executing instructions for the training process stored in the computer-readable medium. The model training device 202 can further include input and output interfaces to communicate with the training database 201, the network 206, and / or a user interface (not shown). The user interface can be used to select multiple sets of training data, adjust one or more parameters of the training process, select or modify the framework of the learning model, and / or manually or semi-automatically provide prediction results associated with images used for training.

[0050] The image analysis device 203 can communicate with the medical image database 204 to receive medical images. The medical images can be acquired by the image acquisition device 205. The image analysis device 203 can automatically perform image analysis tasks (such as segmentation, classification, object detection, etc.) on the medical images using the trained main learning model from the model training device 202. The image analysis device 203 may include a processor and a non-transitory computer-readable medium. The processor can execute instructions of a medical image diagnostic analysis program stored in the medium. The image analysis device 203 may further include input and output interfaces to communicate with the medical image database 204, the network 206 and / or a user interface (not shown). The user interface can be used to select a medical image for analysis, start the analysis process, and display the diagnosis results.

[0051] The systems and methods mentioned in this disclosure may be implemented using computer systems, such as Figure 3 Although Figure 3 Detailed components within the model training apparatus 202 are shown, but it is anticipated that the image analysis apparatus 203 may include similar components, and the following description of the components of the model training apparatus 203 also applies to the components of the image analysis apparatus 203, whether adapted or not.

[0052] In some embodiments, the model training device 202 can be a dedicated device or a general-purpose device. For example, the model training device 202 can be a computer customized for a hospital to train a learning model for processing image data. The model training device 202 can include one or more processors 308 and one or more storage devices 304. The processors 308 and the storage devices 304 can be configured in a centralized or distributed manner. The model training device 202 can also include a medical image database (optionally stored in the storage device 304 or remote storage), input / output devices (not shown, but may include a touch screen, keyboard, mouse, speaker / microphone, etc.), a network interface such as the communication interface 302, a display (not shown, but may be a cathode ray tube (CRT) or liquid crystal display (LCD), etc.), and other accessories or peripheral devices. The various components of the model training device 202 can be connected via a bus 310, which can be a physical and / or logical bus within or between computing devices.

[0053] The processor 308 may be a processing device including one or more general-purpose processing devices (e.g., a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc.). More specifically, the processor 308 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor that executes other instruction sets, or a processor that executes a combination of instruction sets. The processor 308 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), etc.

[0054] The processor 308 may be communicatively coupled to the memory device 304 and configured to execute computer-executable instructions stored therein. Figure 3 As shown, bus 310 may be used, although a logical or physical star or ring topology may be examples of other acceptable communication topologies. Storage device 304 may include read-only memory (ROM), flash memory, random access memory (RAM), static memory, volatile or non-volatile, magnetic semiconductor, tape, optical, removable, non-removable, or other types of storage devices or tangible (e.g., non-transitory) computer-readable media. In some embodiments, storage device 304 may store computer-executable instructions for one or more processing programs and data generated when executing the computer programs. The processor may execute the processing programs to implement each step of the method described below. The processor may also send and receive image data to and from the storage device.

[0055] The model training device 202 may also include Figure 3 One or more digital and / or analog communication (input / output) devices not shown in the figure. For example, the input / output device may include a keyboard and a mouse or a trackball that allow the user to provide input. The model training device 202 may further include a network interface (illustrated as a communication interface 302), such as a network adapter, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transmission adapter such as an optical fiber, USB 3.0, lightning, a wireless network adapter such as a WiFi adapter, or a telecommunications (3G, 4G / LTE, etc.) adapter. The model training device 202 can also be connected to a network via a network interface. As described above, the model training device 202 may further include a display. In some embodiments, the display may be any display device suitable for displaying medical images and their segmentation results. For example, the image display may be an LCD, CRT, or LED display.

[0056] As above Figure 2 As described above, the model training device 202 can be connected to the image analysis device 203 and the image acquisition device 205. In some embodiments, the model training device 202 can implement various workflows to train the learning model for use by the image analysis device 203 to perform a predetermined image analysis task, such as Figure 4A-4B 、 Figure 5 、 Figure 7A-7B 、 Figure 9A-9B and Figures 11A-11B .

[0057] Figure 4AThe figure illustrates a schematic overview of a workflow 400 performed by the model training device to train a main model and an error estimator using labeled images according to certain embodiments of the present disclosure. In the workflow 400, the labeled images are used as training samples to train the main model 404 and the separate error estimator 406. Each labeled image may include an original image 402 and a corresponding ground truth result 410. The original image 402 may be a medical image acquired using any imaging modality such as CT, X-ray, MRI, ultrasound, or PET. For example, the original image 402 may be a medical image acquired by the image acquisition device 205. In some embodiments, the original image 402 may be preprocessed after being acquired by the image acquisition device 205 to improve image quality (e.g., reduce noise, etc.). Depending on the image analysis task, the ground truth result 410 may be a label of the original image 402. For example, for a classification task, the ground truth result 410 may be a binary or multi-category label indicating to which category the input image belongs. As another example, for an object detection task, the ground truth result 410 may include the coordinates of the bounding box of the detected object, as well as the category label of each object. As another example, for a segmentation task, the ground truth result 410 can be an image segmentation mask of the same size as the input image, indicating the category of each pixel in the input image. The annotation can be performed by a human (e.g., a physician or image analysis operator) or by an automated process.

[0058] The original image 402 is input to a main model 404. Main model 404 is a learning model configured to perform the primary medical image analysis task (e.g., classification, object detection, or segmentation). Main model 404 outputs a main model result 408, and the type of output depends on the image analysis task, similar to that described above for ground truth result 410. For example, for a classification task, main model result 408 can be a class label; for an object detection task, main model result 408 can be the bounding box coordinates of the detected objects and the class label for each object; for a segmentation task, main model result 408 can be an image segmentation mask. In some embodiments, the main model can be implemented using ResNet, U-Net, V-Net, or other suitable learning models.

[0059] The error estimator can be another learning model that is configured to predict the error of the main model output based on the input image and the intermediate results of the main model (e.g., the extracted feature map). In some embodiments, the error estimator 406 can receive the original image 402 as input. In some embodiments, the error estimator 406 can also or alternatively receive certain intermediate results, such as feature maps, from the main model 404. The error estimator outputs an estimated error of the main model 412. During training, the error estimator 406 is trained by the error of the main model 404, which is the difference between the main model result 408 and the ground truth result 410 of the labeled data.

[0060] In some embodiments, the training and inference of the error estimator are embedded as part of the training of the main model. For example, in the workflow 400, the training of the main model 404 and the error estimator 406 can be performed sequentially or simultaneously. For example, each training sample can be used to train the main model 404, while the difference between the main model results 408 predicted by the main model 404 and the ground truth results 410 in the training sample is used to train and update the error estimator. As another example, the main model 404 can be first trained using all training samples in the training data, and the difference between the main model results 408 and the ground truth results 410 in the training samples can be collected and used to train the error estimator 406.

[0061] Figure 4B A schematic overview of another workflow 450 performed by the model training apparatus to augment training data by deploying the master model and error estimator on unlabeled images, according to certain embodiments of the present disclosure, is illustrated. In workflow 450, error estimator 406, trained using workflow 400, is applied to unlabeled training data, such as unlabeled images 414, to predict the error produced by master model 404. As shown, unlabeled images 414 and, optionally, certain intermediate results (e.g., feature maps) from master model 404 when master model 404 is applied to the same unlabeled images 414 can be input to error estimator 406. The error estimator uses these inputs to predict the error of master model 404. If the predicted error is low, e.g., less than a predetermined threshold, unlabeled image 414 is added to training data 416 along with the master model results produced by master model 404. Otherwise, if the predicted error is high, e.g., above a predetermined threshold, manual annotation 418 can be requested, and the annotated image can be added to training data 416.

[0062] In some embodiments, to ensure that the error estimator 406 is performing well and contributing to the training of the master model 404, an optional independent labeled validation set can be used to validate the performance of the error estimator 406. In some embodiments, an independent labeled validation set can be selected from the labeled training data and set aside for validation purposes. To maintain its "independence," the validation set will not be used as part of the labeled data to train the master model 404 and the error estimator 406. In one embodiment, the performance of the error estimator can be evaluated via workflow 400 by directly comparing the ground truth error (e.g., the difference between the ground truth result 410 and the master model result 408) obtained by the master model 404 on this validation set with the error estimate output by the error estimator 406. In another embodiment, the performance of the error estimator can be evaluated by evaluating the performance of the updated master model on this validation set using the low-error and high-error data identified by the error estimator 406, via workflow 450, and comparing it to the performance of the initial master model using only labeled data on the validation set. These validations provide additional assurance that the error estimator is well-behaved and provide benefits for training the main model.

[0063] Figure 5 A schematic overview of a training workflow 500 performed by a model training apparatus according to certain embodiments of the present disclosure is illustrated. Figure 6 6 is a flowchart of an example method 600 for training a main model and an error estimator for performing an image analysis task using labeled and unlabeled training data according to certain embodiments of the present disclosure. The method 600 may be performed by the model training device 202 and may include steps S602-S620. It is contemplated that some steps may be optional and may be performed in different ways than Figure 6 Perform certain steps in the order shown. Figure 5-Figure 6 Will be described together.

[0064] Method 600 begins when the model training device 202 receives the training data (step S602). For example, the training data can be received from the training database 201. In some embodiments, the training data includes a first labeled data subset (e.g., labeled data 502 in the workflow 500) and a second unlabeled data subset (e.g., unlabeled data 508 in the workflow 500). For example, the training data may include labeled images and unlabeled images. In some embodiments, the training images can be acquired using the same imaging mode as those that will be analyzed later by the main model to improve training accuracy. The imaging mode can be any suitable mode, including, for example, MRI, fMRI, DCE-MRI, diffusion MRI, PET, SPECT, X-ray, OCT, fluorescence imaging, ultrasound imaging, radiotherapy portal imaging, etc.

[0065] Then, the model training device 202 trains the initial master model and the error estimator using the labeled data (step S604). The master model is trained to acquire an input image and predict the output of a specified image analysis task (segmentation / classification / detection, etc.). The error estimator can take the original input image or the intermediate result or feature map of the master model as input. For example, as shown in the workflow 500, the initial master model training 504 and the error estimator training 506 are performed using the labeled data 502. In some embodiments, the initial master model training 504 uses the ground truth results included in the labeled data 502, while the error estimator training 506 relies on the difference between the ground truth results and the predicted results using the initial master model.

[0066] Then, the model training device 202 applies the error estimator trained in step S604 to estimate the prediction error of the master model (step S606). For example, as shown in the workflow 500, the error estimator deployment 510 is performed by applying the error estimator provided by the error estimator training 506 to the unlabeled data 508 to estimate the prediction error of the master model provided by the initial master model training 504.

[0067] The model training device 202 determines whether the estimated error exceeds a predetermined first threshold (step S608). In some embodiments, the first threshold may be a relatively low value, such as 0.1. If the error does not exceed the first threshold (S608: No), the error is considered low, and the model training device uses the initial master model to obtain predicted annotations for the unlabeled data (step S610) to form labeled data samples, and adds the labeled data samples to the training data (step S616). For example, in the workflow 500, when the error may be "low", the unlabeled data 508 and the prediction results ("pseudo-annotations") of the trained initial master model are added to the training data 512. These samples can expand the training data and improve the performance and generalization ability of the master model.

[0068] Otherwise, if the error exceeds the first threshold (S608: Yes), the model training device 202 further determines whether the estimated error exceeds a predetermined second threshold (step S612). In some embodiments, the second threshold can be a relatively high value, higher than the first threshold, such as 0.9. If the error exceeds the second threshold (S612: Yes), the error is considered high, and the model training device 202 requests manual annotation of the unlabeled data (step S614) to form labeled data samples and adds the manually labeled data samples to the training data (step S616). For example, in workflow 500, when the error is likely "high," manual annotation 514 is requested, and the unlabeled data 508 is added to the training data 512 along with the manual annotations 514. These manually annotated samples provide the richest information for improving the master model, as the initial master model is expected to perform poorly on these samples according to the error estimator. Thus, limited annotation resources are utilized to achieve optimal performance in an annotation-efficient learning scenario. Therefore, the training data is augmented by including data that is labeled automatically (via the main model) or manually (via human annotation).

[0069] Using the augmented training data, model training device 202 trains an updated master model (step S618) to replace the initial master model trained only using the labeled data included in the initial training data. For example, in workflow 500, three labeled data sources are used to train updated master model 516: the original labeled data 502, the low-error portion of the unlabeled data 508 output by the initial master model as pseudo-labels, and the high-error portion of the unlabeled data 508 with the newly requested manual labels.

[0070] In some embodiments, due to limited manual labeling resources, not all high-error unlabeled data can be manually labeled in step S614. In this case, a high second threshold can be selected so that in step S614, the model training device 202 can request to first label the data with the highest prediction error according to the error estimator. In some embodiments, some data can remain unlabeled, neither pseudo-labeled by the main model nor manually labeled by request. For example, if the error exceeds the first threshold (S608: Yes) but does not exceed the second threshold (S612: No), the data sample can remain unlabeled during this update iteration. Figure 5 The illustrated workflow 500 can be repeated one or more times to use the updated master model (trained in step S618) as the initial master model and update it again. As the master model becomes stronger, more data may be able to be pseudo-labeled by the master model, and the unlabeled portion of the data will be further reduced.

[0071] The model training device 202 then provides the updated main model as a learning model for analyzing the new medical image (step S620). The training method 600 then ends. The image analysis device 203 can deploy the updated main model to complete the specified medical image analysis task for the new medical image. In some embodiments, if the error estimation of the main model is not needed in the application, the error estimator can be disabled. In some alternative embodiments, the error estimator can be kept turned on to provide an estimate of the potential error in the output of the main model. For example, the error estimator can be used to generate the error of the main model in parallel with the main model performing the image analysis task, and provide the error to the user for visual inspection, for example, through the display of the image analysis device 203, so that the user understands the performance of the main model. The following will be combined with Figure 13 Provides more details about applying the trained model and error estimator.

[0072] By identifying unlabeled data that will result in high prediction errors when applying the master model and only requesting manual annotation for such unlabeled data, method 600 can allocate limited manual annotation resources to analyze only those images that cannot be accurately analyzed by the master model. By augmenting the training data with both automatically and manually annotated data (e.g., pseudo-annotations and manual annotations), method 600 also helps master model training fully utilize existing unlabeled data.

[0073] The master model can be trained to perform any predetermined image analysis task, such as image segmentation, image classification, and object detection from images. Based on the specific image analysis task, the features extracted by the master model during prediction, the prediction results, the ground truth results included in the labeled data, the error estimated by the error estimator, the configuration of the learning model, and the configuration of the error estimator can all be designed accordingly.

[0074] For example, when the image analysis task is image classification, the primary model can be an image classification model configured to predict the class label of an image. In this case, the output of the primary model is a binary or multi-class classification label. The output of the error estimator is the classification error, for example, the cross entropy loss between the predicted and ground truth labels. Figure 7A A schematic overview of a workflow 700 performed by the model training apparatus 202 to train a main classification model 704 and an error estimator 706 using labeled images is illustrated in accordance with certain embodiments of the present disclosure. Figure 7B Illustrated is a schematic overview of another workflow 750 performed by a model training apparatus to augment training data by deploying a main classification model 704 and an error estimator 706 on unlabeled images, according to certain embodiments of the present disclosure. Figure 8 800 is a flowchart of an example method 800 for training an image classification model and an error estimator for performing an image classification task using labeled and unlabeled training data according to certain embodiments of the present disclosure. The method 800 may be performed by the model training device 202 and may include steps S802-S820. It is contemplated that some steps may be optional and may be performed in different ways than Figure 8 Perform certain steps in the order shown. Figure 7A-7B and Figure 8 Will be described together.

[0075] The method 800 begins when the model training device 202 receives training data (step S802) similar to step S602 described above. The model training device 202 then trains the main classification model and the error estimator using the labeled data (step S804). As shown in the workflow 700, the main classification model 704 is trained to take the original image 702 as input and predict the classification label as output. The error estimator 706 can take the original image 702 or the intermediate result or feature map of the main model as input. Figure 7AAs shown, the main classification model 704 and the error estimator 706 are initially trained using labeled data including pairs of original images 702 and their corresponding ground truth classification labels 710. In some embodiments, the main classification model 704 is trained to minimize the difference between the predicted classification labels 708 when the main classification model 704 is applied to the original images 702 and the ground truth classification labels 710 corresponding to the original images 702. In some embodiments, the main classification model 704 can be implemented by any classification network, including ResNet, EfficientNet, NAS, etc.

[0076] On the other hand, the error estimator 706 is trained using a "ground truth error," which is determined by the ground truth class labels 710 and the predicted class labels 708. In one example, the error can be the cross entropy loss between the ground truth class labels 710 and the predicted class labels 708. The error estimator 706 is trained to minimize the difference between the estimated classification error 712 estimated by the error estimator 706 and the "ground truth error," which is determined using the ground truth class labels 710 and the predicted class labels 708. In some embodiments, the error estimator 706 can be implemented by a multilayer perceptron or other network.

[0077] Then, the model training device 202 applies the error estimator trained in step S804 to estimate the classification error of the main classification model (step S806). For example, as shown in the workflow 750, the error estimator 706 is applied to the unlabeled image 714 to estimate the classification error of the main classification model 704.

[0078] The model training device 202 determines whether the estimated classification error exceeds a predetermined first threshold (step S808). In some embodiments, the first threshold may be a low value, such as 0.1. If the classification error does not exceed the threshold (S808: No), the model training device 202 applies the main classification model 704 to obtain a predicted classification label for the unlabeled data (step S810) to form a pseudo-labeled data sample and add the pseudo-labeled data sample to the training data (step S816). For example, in the workflow 700, when the classification error may be "low", the unlabeled image 714 is added to the training data 716 along with the classification label predicted by the main classification model 704.

[0079] Otherwise, if the classification error exceeds the first threshold (S808: Yes), the model training device 202 determines whether the estimated classification error exceeds a predetermined second threshold (step S812). In some embodiments, the second threshold may be a higher value than the first threshold, such as 0.9. If the classification error exceeds the second threshold (S812: Yes), the model training device 202 requests manual annotation of the unlabeled image (step S814) to form a manually labeled data sample, which is then added to the training data (step S816). For example, in workflow 750, when the classification error is likely to be "high", manual annotation 718 is requested, and the unlabeled image 714 is added to the training data 716 along with the manual annotation 718. If the error exceeds the first threshold (S808: Yes) but does not exceed the second threshold (S812: No), the data sample may remain unlabeled.

[0080] Using the expanded training data, the model training device 202 trains an updated main classification model (step S818) to replace the initial main classification model trained using only the labeled images, and provides the updated main classification model as a learning model for analyzing new medical images (step S820), similar to the above combination Figure 6 The updated main classification model can be deployed to predict binary or multi-class labels for new medical images.

[0081] As another example, when the image analysis task is object detection, the primary model can be an object detection model (also called a detector model) configured to detect objects. In this case, the output of the primary model includes the coordinates of the bounding box around the object and the class label of the object. The output of the error estimator includes the localization error (e.g., the mean squared error difference between the predicted bounding box coordinates and the ground truth bounding box coordinates) and / or the classification error (e.g., the cross entropy loss between the predicted object class label and the ground truth object class label).

[0082] Figure 9A A schematic overview of a workflow 900 performed by the model training apparatus 202 to train an object detection model 904 and an error estimator 906 using labeled images is illustrated in accordance with certain embodiments of the present disclosure. Figure 9B Illustrated is a schematic overview of another workflow 950 performed by a model training apparatus to augment training data by deploying an object detection model 904 and an error estimator 906 on unlabeled images, in accordance with certain embodiments of the present disclosure. Figure 101 is a flow chart of an example method 1000 for training an object detection model and an error estimator for performing an object detection task using labeled and unlabeled training data according to certain embodiments of the present disclosure. Method 1000 may be performed by model training apparatus 202 and may include steps S1002-S1020. It is contemplated that some steps may be optional and may be performed in different ways than Figure 10 Perform certain steps in the order shown. Figure 9A-9B It will be described together with FIG10.

[0083] Method 1000 begins when model training device 202 receives training data (step S1002) similar to step S802 described above. Model training device 202 then trains the main object detection model and error estimator using the labeled data (step S1004). As shown in workflow 900, main object detection model 904 is trained to take the original image 902 as input and predict the coordinates of the object bounding box and the class label of the object as output. Error estimator 906 can take the original image 902 or the intermediate result or feature map of the main model as input. Figure 9A As shown, the main object detection model 904 and the error estimator 906 are initially trained using labeled data including pairs of original images 902 and their corresponding ground truth bounding boxes and class labels 910. In some embodiments, the main object detection model 904 is trained to minimize the difference between the predicted bounding boxes and classes and the ground truth bounding boxes and classes. In some embodiments, the main object detection model 904 can be implemented by any object detection network, including R-CNN, YOLO, SSD, CenterNet, CornerNet, etc.

[0084] On the other hand, error estimator 906 is trained using a "ground truth error," which is determined by ground truth bounding box and class label 910 and predicted bounding box and class label 908. In one example, the error may be the cross-entropy loss between ground truth class label 910 and predicted class label 908. Error estimator 906 is trained to minimize the difference between the estimated localization and / or classification error 912 estimated by error estimator 906 and the "ground truth error." In some embodiments, error estimator 906 may be implemented by two multilayer perceptrons, or other types of networks, for respectively estimating localization and classification errors.

[0085] Then, the model training device 202 applies the error estimator trained in step S1004 to estimate the localization error and / or classification error of the primary object detection model (step S1006). For example, as shown in the workflow 950, the error estimator 906 is applied to the unlabeled image 914 to estimate the localization error and / or classification error of the primary object detection model 904. In some embodiments, the error estimator 906 can further determine a combined error reflecting the localization and classification errors, for example, as a weighted sum of the two errors, or otherwise summarize the two errors.

[0086] Steps S1008-S1020 are the same as above Figure 8 Steps S808-S820 are similarly performed, except that the annotations in this scene include the bounding boxes and category labels of the detected objects. Detailed descriptions are not repeated here.

[0087] As another example, when the image analysis task is image segmentation, the primary model can be a segmentation model configured to segment the image. In this case, the output of the primary model is a segmentation mask. The output of the error estimator is an error map of the segmentation mask. If the image to be segmented is a 3D image, the segmentation mask is correspondingly a voxel-based segmentation mask, and the error map is a voxel-based map, such as a voxel-based cross entropy loss map.

[0088] Figure 11A A schematic overview of a workflow 1100 performed by the model training apparatus 202 to train a primary segmentation model 1104 and an error estimator 1106 using labeled images is illustrated in accordance with certain embodiments of the present disclosure. Figure 11B A schematic overview of another workflow 1150 performed by a model training apparatus to augment training data by deploying a primary segmentation model 1104 and an error estimator 1106 on unlabeled images is shown, in accordance with certain embodiments of the present disclosure.

[0089] ]Workflow 1100 / 1150 is similar to the above combination Figure 7A-7B and Figure 9A-9B1100 / 1150 is similar to workflows 700 / 750 and 900 / 950, except that when applied to the original image 1102, the prediction of the primary segmentation model 1104 is a segmentation mask 1108, and the error estimated by the error estimator 1106 is a segmentation error map 1112. The ground truth segmentation mask 1110 corresponding to the original image 1102 included in the labeled image is used to train the primary segmentation model 1104 and to determine the "ground truth" segmentation error map for training the error estimator 1106. In some embodiments, the segmentation error map can be a voxel-based cross entropy loss map. The detailed description of workflows 1100 / 1150 can be found in and adapted from those of workflows 700 / 750 and 900 / 950 described above and is therefore not repeated here.

[0090] Figure 12 1 is a flow chart of an example method 1200 for training a segmentation model and an error estimator for performing an image segmentation task using labeled and unlabeled training data according to certain embodiments of the present disclosure. The method 1200 may be performed by the model training device 202 and may include steps S1202-S1220. It is contemplated that some steps may be optional and some steps may be performed in accordance with the Figure 12 The different orders shown are executed.

[0091] Method 1200 begins when the model training device 202 receives training data (step S1202) similar to steps S802 and S1002 described above. The model training device 202 then trains the primary segmentation model and the error estimator using the labeled data (step S1204). As shown in the workflow 1100, the primary segmentation model 1104 is trained to take the original image 1102 as input and predict a segmentation mask as output. The error estimator 1106 can take the original image 1102 or an intermediate result or feature map of the primary model as input. Figure 11A As shown, the main segmentation model 1104 and the error estimator 1106 are initially trained using labeled data including pairs of original images 1102 and their corresponding ground truth segmentation masks 1110. In some embodiments, the main segmentation model 1104 is trained to minimize the difference between the predicted bounding boxes and categories and the ground truth bounding boxes and categories. In some embodiments, the main segmentation model 1104 can be implemented by any segmentation network, including U-Net, V-Net, DeepLab, Feature Pyramid Network, etc.

[0092] On the other hand, the error estimator 1106 is trained using a "ground truth error," which is determined by the ground truth segmentation mask 1110 and the predicted segmentation mask 1108. In one example, the error can be a cross-entropy loss map determined based on the ground truth segmentation mask 1110 and the predicted segmentation mask 1108. The error estimator 1106 is trained to minimize the difference between the estimated segmentation error map 1112 estimated by the error estimator 1106 and the "ground truth error." The error estimator 1106 can be implemented by a decoder network in a U-Net or other types of segmentation networks.

[0093] Then, the model training device 202 applies the error estimator trained in step S1204 to estimate the segmentation error map of the primary segmentation model (step S1206). For example, as shown in the workflow 1150, the error estimator 1106 is applied to the unlabeled image 1114 to estimate the segmentation error map of the primary segmentation model 1104.

[0094] Steps S1208-S1220 are combined with the above Figure 8 Steps S808-S820 and combined Figure 10 Steps S1008-S1020 are similarly performed, except that the annotation in this scenario is a segmentation mask. Detailed description will not be repeated.

[0095] Due to the intensive nature of image segmentation tasks, annotating the entire image can be expensive. The primary segmentation model may only make errors in certain areas of the image. In some embodiments, to further improve annotation efficiency, the image can be broken down into small patches or ROIs (regions of interest) after receiving the image in step S1202 and before performing training in step S1204. Thus, steps S1206-S1218 can be performed on a patch / ROI basis. For example, the primary segmentation model can predict a segmentation mask for each patch or ROI, and the error estimator can evaluate the error for each patch or ROI rather than for the entire image, providing more refined guidance. In another example, the primary segmentation model and error estimator can predict a segmentation mask and error estimate for the entire image, but only those patches or ROIs indicated by the error estimator as containing large amounts of error are provided to the annotator for further annotation. In such an embodiment, the annotator can be instructed to annotate only the smaller areas where the primary model may have errors in step S1214, significantly reducing the annotation burden. Annotations can be obtained manually, semi-manually, or fully automatically. For example, more expensive models / methods can be used to automatically generate annotations. Annotations can also be obtained semi-automatically or automatically with the help of other imaging modalities.

[0096] Figure 131 is a flow chart of an example method 1300 for performing an image task on a medical image using a learning model trained by an error estimator according to certain embodiments of the present disclosure. The method 1300 may be performed by the image analysis apparatus 203 and may include steps S1302-S1314. It is contemplated that some steps may be optional and may be performed in a manner different from the method described above. Figure 13 Perform certain steps in the order shown.

[0097] Method 1300 begins when image analysis device 203 receives a medical image acquired by an image acquisition device (step S1302). In some embodiments, image analysis device 203 may receive the medical image directly from image acquisition device 205 or from a medical image database 204 storing acquired images. Similarly, any imaging modality may be used to acquire the medical image, including, for example, CT, cone-beam CT, MRI, fMRI, DCE-MRI, diffusion MRI, PET, SPECT, X-ray, OCT, fluorescence imaging, ultrasound imaging, and radiotherapy portal imaging.

[0098] Then, the image analysis device 203 applies the trained learning model to the medical image to perform the image analysis task (step S1304). In some embodiments, the learning model can be trained together with a separate error estimator on partially labeled training images. For example, the learning model can be trained using Figure 5 Workflow 500 or Figure 6 The method 600 trains the updated main model 516.

[0099] In steps S1304 and S1306, the image analysis task may be any predetermined task for analyzing or otherwise processing a medical image. In some embodiments, the image analysis task is an image segmentation task, and the learning model is designed to predict a segmentation mask of a medical image, for example, a segmentation mask for a lesion in a lung region. The segmentation mask can be a probability map. For example, it can be used Figure 11A - Figure 11B Workflow 1100 / 1150 and Figure 12 Method 1200 is used to train a segmentation learning model and an error estimator. In some embodiments, the image analysis task is an image classification task, and the learning model is designed to predict a classification label of a medical image. For example, the classification label is a binary label indicating whether the medical image contains a tumor, or a multi-class label indicating what type of tumor the medical image contains. For example, the classification learning model and the error estimator can be used Figure 7A-7B Workflow 700 / 750 and Figure 8In some embodiments, the image analysis task is an object detection task, and the learning model is designed to detect objects from medical images, for example, by predicting a bounding box surrounding the object and a classification label for the object. For example, the coordinates of a lung nodule bounding box can be predicted, and a class label can be predicted to indicate that it is a lung nodule. For example, Figure 9A-9B Workflow 900 / 950 and Figure 10 Method 1000 to train an object detection learning model and error estimator.

[0100] The image analysis device 203 can also apply the trained error estimator to the medical image to estimate the error of the learning model when performing the image analysis task on the medical image (step S1306). In some embodiments, the error estimator can be applied to generate errors in parallel with the main model performing the image analysis task in step S1304. The type of error estimated by the error estimator depends on the image analysis task. For example, when the image analysis task is image segmentation, the error estimator can be designed to estimate the error map or error estimate of the segmentation mask. When the image analysis task is image classification, the error estimator is correspondingly designed to estimate the classification error between the classification label predicted by the learning model and the ground truth label contained in the labeled image, such as the cross entropy loss. When the image analysis task is object detection, the error estimator is correspondingly configured to estimate the positioning error between the predicted bounding box and the ground truth bounding box contained in the labeled image, or the classification error between the classification label predicted by the learning model and the ground truth label contained in the labeled image, or a combination of the two.

[0101] The image analysis device 203 may provide the error estimated in step S1306 to the user for visual inspection (step S1308). For example, the error may be an error map provided as an image through the display of the image analysis device 203 so that the user can understand the performance of the master model.

[0102] In step S1310, it is determined whether the error is too high. In some embodiments, the user can make the determination by visual inspection. In some alternative embodiments, the determination can be made automatically by the image analysis device 203, for example, by comparing the error with a threshold. If the error is too large (S1310: yes), the image analysis device 203 can request user interaction to improve the learning model or request the learning model to be retrained by the model training device 202 (step S1314). The image analysis device 203 repeats steps S1306-S1310 using the new learning model improved or retained by the user. For example, Figure 5In the workflow 500 of FIG. 5 , the current learning model is used as the initial master model to update the learning model. Otherwise ( S1310 : No), the image analysis device 203 may provide an image analysis result (step S1312 ), such as a classification label, a segmentation mask, or a bounding box.

[0103] According to certain embodiments, a non-transitory computer-readable medium may store a computer program. When executed by at least one processor, the computer program may perform a method for biomedical image analysis. For example, any of the above-described methods may be performed in this manner.

[0104] In some embodiments, the computer-readable medium may include volatile or non-volatile, magnetic semiconductor, magnetic tape, optical, removable, non-removable, or other types of computer-readable media or computer-readable storage devices. For example, as disclosed, the computer-readable medium may be a storage device or memory module having computer instructions stored thereon. In some embodiments, the computer-readable medium may be a disk or flash drive having computer instructions stored thereon.

[0105] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed systems and related methods.Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed systems and related methods.

[0106] It is intended that the specification and examples be considered as exemplary only, with a true scope being indicated by the following claims and their equivalents.

Claims

1. A system for analyzing medical images using a learning model, comprising: a communication interface configured to receive a medical image acquired by an image acquisition device; as well as at least one processor configured to apply the learned model to perform an image analysis task on the medical images, wherein the learned model is jointly trained with an error estimator using training images comprising a first set of labeled images and a second set of unlabeled images, wherein the error estimator is configured to estimate an error of the learned model associated with performing the image analysis task; Wherein, the at least one processor is further configured to: applying the error estimator to the medical image to estimate the error when the learning model performs the image analysis task on the medical image; To train the learning model and the error estimator, the at least one processor is configured to: training an initial version of the learning model and an error estimator using the first labeled image set; applying the error estimator to the second set of unlabeled images to determine corresponding errors associated with the unlabeled images; determining a third set of labeled images from the second set of unlabeled images based on the corresponding errors; and training an updated version of the learning model using the first labeled image set and the third labeled image set in combination; and The updated version of the learned model is provided to perform the image analysis task on the medical image.

2. The system according to claim 1, wherein: Also included is a display configured to provide the error to a user for visual inspection.

3. The system according to claim 1, wherein: To determine the third set of labeled images from the second set of unlabeled images, the at least one processor is further configured to: identifying at least one unlabeled image from the second set of unlabeled images that is associated with an error below a predetermined first threshold; Applying the learning model to the identified unlabeled image to generate a corresponding pseudo-labeled image; and The pseudo-labeled image is included in the third set of labeled images.

4. The system according to claim 1, wherein: To determine the third set of labeled images from the second set of unlabeled images, the at least one processor is further configured to: identifying at least one unlabeled image from the second set of unlabeled images that is associated with an error above a predetermined second predetermined threshold; Acquire annotations on the identified unlabeled image to form a corresponding new labeled image; and The new labeled image is included in the third set of labeled images.

5. The system according to claim 1, wherein: The first labeled image set includes original images and corresponding ground truth results, Wherein, the error estimator is trained based on the difference between the ground truth results in the first labeled image set and the image analysis results obtained by applying the learning model to the original images in the first labeled image set.

6. The system according to claim 1, wherein: The image analysis task is an image segmentation task, and the learning model is configured to predict a segmentation mask, wherein the error estimator is configured to estimate an error map of the segmentation mask.

7. The system according to claim 1, wherein: The image analysis task is an image classification task, and the learning model is configured to predict classification labels. Wherein, the error estimator is configured to estimate the classification error between the classification label predicted by the learning model and the ground truth label included in the labeled image.

8. The system according to claim 1, wherein: The image analysis task is an object detection task, and the learning model is configured to predict a bounding box surrounding an object and a classification label of the object.

9. The system according to claim 8, wherein: The error estimator is configured to estimate a localization error between the predicted bounding box and a ground truth bounding box included in the labeled image, or a classification error between the classification label predicted by the learning model and a ground truth label included in the labeled image.

10. A computer-implemented method for analyzing medical images using a learning model, comprising: receiving a medical image acquired by an image acquisition device through a communication interface; as well as applying, by at least one processor, the learned model to perform an image analysis task on the medical image, wherein the learning model and an error estimator are jointly trained using training images comprising a first set of labeled images and a second set of unlabeled images, wherein the error estimator is configured to estimate an error of the learning model associated with performing the image analysis task; further comprising: applying the error estimator to the medical image to estimate the error of the learning model when performing the image analysis task on the medical image; The learning model and the error estimator are trained in the following manner: training an initial version of the learning model and an error estimator using the first labeled image set; applying the error estimator to the second set of unlabeled images to determine corresponding errors associated with the unlabeled images; determining the third set of labeled images from the second set of unlabeled images based on the corresponding errors; training an updated version of the learning model using a combination of the first set of labeled images and the third set of labeled images; and The updated version of the learned model is provided to perform the image analysis task on the medical image.

11. The computer-implemented method of claim 10, wherein: Further including: The error is provided to the user via a display for visual inspection.

12. The computer-implemented method of claim 10, wherein: Determining the third set of labeled images from the second set of unlabeled images further comprises: identifying at least one unlabeled image from the second set of unlabeled images that is associated with an error below a predetermined first threshold; applying the learned model to the identified unlabeled image to generate a corresponding pseudo-labeled image; and The pseudo-labeled image is included in the third set of labeled images.

13. The computer-implemented method of claim 10, wherein: Determining the third set of labeled images from the second set of unlabeled images further comprises: identifying at least one unlabeled image from the second set of unlabeled images that is associated with an error above a predetermined second threshold; Obtaining manual annotations on the identified unlabeled image to form a corresponding new labeled image; and The new labeled image is included in the third set of labeled images.

14. The computer-implemented method of claim 10, wherein: The image analysis task is an image segmentation task, and the learning model is configured to predict a segmentation mask, Wherein, the error estimator is configured to estimate an error map of the segmentation mask.

15. The computer-implemented method of claim 10, wherein: The image analysis task is an image classification task, and the learning model is configured to predict classification labels. Wherein, the error estimator is configured to estimate the classification error between the classification label predicted by the learning model and the ground truth label included in the labeled image.

16. The computer-implemented method of claim 10, wherein: The image analysis task is an object detection task, and the learning model is configured to predict a bounding box surrounding an object and a classification label of the object, The error estimator is configured to estimate a localization error between the predicted bounding box and a ground truth bounding box included in the labeled image, or a classification error between the classification label predicted by the learning model and a ground truth label included in the labeled image.

17. A non-transitory computer-readable medium storing a computer program, wherein: When the computer program is executed by at least one processor, a method for analyzing a medical image using a learning model is performed, the method comprising: receiving a medical image acquired by an image acquisition device; and applying the learned model to perform an image analysis task on the medical image, wherein the learning model and an error estimator are jointly trained using training images comprising a first set of labeled images and a second set of unlabeled images, wherein the error estimator is configured to estimate an error of the learning model associated with performing the image analysis task; further comprising: applying the error estimator to the medical image to estimate the error of the learning model when performing the image analysis task on the medical image; The learning model and the error estimator are trained in the following manner: training an initial version of the learning model and an error estimator using the first labeled image set; applying the error estimator to the second set of unlabeled images to determine corresponding errors associated with the unlabeled images; determining the third set of labeled images from the second set of unlabeled images based on the corresponding errors; training an updated version of the learning model using a combination of the first set of labeled images and the third set of labeled images; and The updated version of the learned model is provided to perform the image analysis task on the medical image.

Images