System and method for memory augmentation domain adaptation
Through memory amplification domain adaptive system and meta-learning technology, the machine learning model of the target domain features and support image amplification is solved, and the problems of domain adaptive performance degradation and sample diversity in the existing technology are realized. efficient adaptation and continuous learning.
Patent Information
- Application Number
- CN201980060218.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-10-16
- Filing Date
- 2019-10-09
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2039-10-09
AI Technical Summary
Existing machine learning models experience performance degradation problems in domain adaptation, especially in environments with high sample diversity, and current solutions require large samples or rely on meta-learning, resulting in complex deployment and training.
A memory amplification domain adaptive system and method are proposed to calculate the features of the target domain image, identify the supporting images, use the supporting images to amplify the machine learning model, make it adapt to the target domain, and through meta-learning Technology achieves continuous learning.
A machine learning model that uses a small sample set to adapt to the new domain is implemented, which avoids dependence on a large amount of training data, improves the performance and adaptability of the model in the new domain, and simplifies the clinical workflow.
Smart Images

Figure CN112714914B_ABST
Abstract
Description
Background Art
[0001] Embodiments of the present specification relate generally to machine learning, and more particularly to systems and methods for memory-augmented continuous learning to adapt to domains.
[0002] It should be understood that traditional machine learning techniques require large data sets to "learn" via extensive training. In addition, machine learning techniques are typically trained using data sets corresponding to the source domain. However, most statistical methods, including machine learning techniques, are known to perform quite poorly in scenarios / domains that are rarely encountered. In particular, many machine learning techniques, including neural networks, encounter the problem of domain adaptation and therefore suffer from degradation of the performance of the model for samples associated with different but related domains. Problems in domain adaptation are generally attributed to the diversity in the samples. This diversity in samples is encountered even in controlled environments such as medical imaging, where training samples differ due to differences in equipment, demographics, pathological conditions, protocol / operator variability, patient / subjects, etc. In addition, the problem of domain adaptation is further complicated by the difficulty in obtaining large amounts of data in health care and other regulated domains to train or retrain models.
[0003] It is desirable that an algorithm / technique trained using data corresponding to a source domain adapt to a new target domain using as few samples as possible. Some currently available solutions to the domain adaptation problem require enabling the system to learn from errors during deployment. One exemplary approach requires correcting observed errors by retraining the existing technology with new samples. However, such approaches disadvantageously have drawbacks. In one example, adapting an algorithm to a new target domain requires a large number of samples corresponding to the target domain. Additionally, in another example, neural networks suffer from a phenomenon known as "catastrophic forgetting".
[0004] In addition, some currently available techniques use memory augmentation neural networks (MANNs) to remember rare events. Certain other techniques use few-shot learning methods to quickly adapt to changes in either the domain or the task through a meta-learning paradigm. However, most of these methods disadvantageously rely on meta-learning from many classes. Summary of the invention
[0005] According to various aspects of the present specification, a system is proposed. The system includes an acquisition subsystem configured to obtain an image corresponding to a target domain. In addition, the system includes a processing subsystem that is operably associated with the acquisition subsystem and includes a memory augmentation domain adaptation platform, wherein the memory augmentation domain adaptation platform is configured to calculate one or more features corresponding to an input image, wherein the input image corresponds to a target domain; identify a set of support images based on one or more features corresponding to the input image, wherein the set of support images corresponds to the target domain; augment the input to the machine learning model with a set of features, a set of masks, or both a set of features and a set of masks corresponding to the set of support images to adapt the machine learning model to the target domain; and generate output based on at least the set of features, the set of masks, or both the set of features and the set of masks corresponding to the set of support images. In addition, the system includes an interface unit configured to present the output for analysis.
[0006] According to another aspect of the present specification, a processing system for adapting a machine learning model is provided. The processing system includes a memory augmentation domain adaptation platform, which is configured to calculate one or more features corresponding to an input image, wherein the input image corresponds to a target domain; identify a set of support images based on the one or more features corresponding to the input image, wherein the set of support images corresponds to the target domain; augment the input to the machine learning model with a set of features, a set of masks, or both a set of features and a set of masks corresponding to the set of support images to adapt the machine learning model to the target domain; generate output based on at least the set of features, the set of masks, or both the set of features and the set of masks corresponding to the set of support images; and provide the output to facilitate analysis.
[0007] According to another aspect of the present specification, a method for adapting a machine learning model is proposed. The method includes receiving an input image, wherein the input image corresponds to a target domain. In addition, the method includes calculating one or more features corresponding to the input image. In addition, the method includes identifying a set of support images based on one or more features corresponding to the input image, wherein the set of support images corresponds to the target domain. In addition, the method includes augmenting the input to the machine learning model with a set of features, a set of masks, or both a set of features and a set of masks corresponding to the set of support images to adapt the machine learning model to the target domain. The method also includes generating an output based at least on the set of features, the set of masks, or both the set of features and the set of masks corresponding to the set of support images. In addition, the method includes outputting the output to facilitate analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] These and other features, aspects and advantages of the present disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings, in which like characters refer to like parts throughout, and in which:
[0009] Figure 1 is a schematic diagram of an exemplary system for memory augmentation domain adaptation according to aspects of the present specification;
[0010] Figure 2 is a flow chart of an exemplary method for memory augmentation domain adaptation according to aspects of the present specification;
[0011] Figure 3 is a schematic diagram illustrating an exemplary method for memory augmentation domain adaptation according to aspects of the present specification;
[0012] Figure 4(a)-Figure 4(c) is a diagrammatic representation of different data sets corresponding to different domains in a system and method for memory augmentation domain adaptation according to aspects of the present specification; and
[0013] Figure 5(a)-Figure 5(e) is a graphical illustration of a comparison of different approaches to performing domain adaptation according to aspects of the present specification. DETAILED DESCRIPTION
[0014] The following description presents exemplary systems and methods for memory augmentation domain adaptation. Specifically, the embodiments described below present exemplary systems and methods that facilitate enhanced memory augmentation continuous learning to adapt machine learning models to new domains, thereby delivering better performance with a relatively small set of samples. For example, when a machine learning model is deployed in a target domain using a small set of samples, the systems and methods facilitate enhanced execution of tasks such as classification and segmentation. In addition, the systems and methods presented below provide a high-quality solution that avoids the shortcomings associated with currently available methods. Specifically, the systems and methods for memory augmentation domain adaptation present learning techniques that adapt machine learning to newer domains with as few samples as possible. Newer samples associated with the deployed target domain are "remembered" so that the output generated by the present system and method continues to evolve while avoiding any modifications to the basis.
[0015] The system and method entail the use of "meta-learning" techniques designed to improve the performance of machine learning models when deployed in new domains. Specifically, the system includes a memory unit that acts like a programmable memory and is used to continuously learn and facilitate adaptation to a target domain using a small set of samples corresponding to a target domain. Thus, when the machine learning model subsequently encounters a similar situation, the memory unit is queried for a match. More specifically, the additional memory unit facilitates the retrieval of samples and corresponding annotations that are similar to a specific use case. In addition, the annotations corresponding to the retrieved samples stored in the memory unit can be revised, thereby providing enhanced control over subsequent predictions of the machine learning model. It may be noted that the terms "domain" and "site" are used interchangeably.
[0016] For the sake of clarity, exemplary embodiments of the systems and methods of the present invention are described in the context of medical imaging systems. It may be noted that, although the exemplary embodiments shown below are described in the context of medical imaging systems, other imaging systems and applications are also contemplated, such as industrial imaging systems and non-destructive evaluation and inspection systems, such as pipeline inspection systems, liquid reactor inspection systems. Some examples of medical imaging systems may include computed tomography (CT) systems, single photon emission computed tomography systems (SPECT) systems, X-ray imaging systems, magnetic resonance imaging (MRI) systems, optical imaging systems, and / or ultrasound imaging systems. In addition, the exemplary embodiments shown and described below may be applied to multimodal imaging systems that employ X-ray imaging systems in combination with other imaging modalities, position tracking systems, or other sensor systems. In one example, a multimodal imaging system may include a positron emission tomography (PET) imaging system-X-ray imaging system. In addition, according to aspects of the present specification, in other non-limiting examples of multi-modality imaging systems, the X-ray imaging system can be used in combination with other imaging systems, such as but not limited to computed tomography (CT) imaging systems, contrast-enhanced ultrasound imaging systems, ultrasound imaging systems, optical imaging systems, magnetic resonance (MR) imaging systems, and other imaging systems. Figure 1 Exemplary environments suitable for practicing various implementations of the systems and methods of the present invention are discussed.
[0017] Figure 1An exemplary imaging system 100 is shown, which is configured to receive and process an input image corresponding to a target domain to generate an output, the target domain corresponding to a target volume in a subject 102 such as a patient or a non-biological object, wherein the output is used for further analysis. Specifically, the system 100 is configured to adapt the machine learning model 106 to the target domain using an exemplary memory augmentation domain adaptation technique. It can be noted that the machine learning model 106 is typically trained using a data set corresponding to a source domain. The system 100 is configured to adapt the machine learning model 106 to the target domain using a relatively small set of samples corresponding to the target domain. In one embodiment, the imaging system 100 may, for example, include an X-ray imaging system, a PET system, a SPECT system, a CT imaging system, an MRI system, a hybrid imaging system, and / or a multimodal imaging system.
[0018] In one embodiment, the patient 102 may be suitably positioned, for example, on a table, to allow the system 100 to image a target volume of the patient 102. During imaging, an image acquisition device 104 operably coupled to a medical imaging system 108 may be used to acquire image data corresponding to a target volume / region of interest in the subject or patient 102. However, in certain other embodiments, the input image may be retrieved from a data storage device.
[0019] In addition, the medical imaging system 108 is configured to receive an input image or image data corresponding to the patient 102, and process the image data to generate an output corresponding to the patient 102. In the currently contemplated configuration, the system 100 may be configured to acquire image data representing the patient 102. As noted above, in one embodiment, the system 100 may acquire image data corresponding to the patient 102 via the image acquisition device 104. Moreover, in one embodiment, the image acquisition device 104 may include a probe, wherein the probe may include an invasive probe or a non-invasive or external probe, such as an external ultrasound probe, and the probe is configured to assist in acquiring image data. In addition, in certain other embodiments, image data may be acquired via one or more sensors (not shown) that may be disposed on the patient 102 or via other devices that use image data acquired corresponding to the patient 102. By way of example, these sensors may include physiological sensors (not shown), such as position sensors. In certain embodiments, the position sensors may include electromagnetic field sensors or inertial sensors. For example, these sensors may be operably coupled to a data acquisition device, such as an imaging system, via leads (not shown). Other methods of acquiring image data corresponding to the patient 102 are also contemplated.
[0020] Furthermore, in one embodiment, the medical imaging system 108 may include an acquisition subsystem 110 and a processing subsystem 112. Furthermore, in one embodiment, the acquisition subsystem 110 of the medical imaging system 108 is configured to acquire image data or input images representing the patient 102 via the image acquisition device 104. It may be noted that the terms "image", "image frame", and "input image" may be used interchangeably.
[0021] In addition, the acquisition subsystem 110 may also be configured to acquire images stored in an optical data storage item (not shown). It may be noted that the optical data storage item may be an optical storage medium, such as a compact disc (CD), a digital versatile disc (DVD), a multi-layer structure (such as a DVD-5 or DVD-9), a multi-sided structure (such as a DVD-10 or DVD-18), a high-definition digital versatile disc (HD-DVD), a Blu-ray disc, a near-field optical storage disc, a holographic storage medium, or other similar volumetric optical storage medium (such as, for example, a two-photon or multi-photon absorption storage format). In addition, for example, the 2D images so acquired by the acquisition subsystem 110 may be stored locally on the medical imaging system 108 in the data repository 116.
[0022] Additionally, the processing subsystem 112 may then process the image data acquired from the patient 102. For example, the processing subsystem 112 may include one or more dedicated processors, graphics processing units, digital signal processors, microcomputers, microcontrollers, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and / or other suitable processing devices. Alternatively, the processing subsystem 112 may be configured to store the acquired image data and / or user input in a data repository 116 and / or in a memory unit 118 for later use. In one embodiment, for example, the data repository 116 may include a hard drive, a floppy disk drive, a compact disk read / write (CD-R / W) drive, a digital versatile disk (DVD) drive, a flash drive, and / or a solid-state storage device.
[0023] It may be noted that examples, demonstrations, and processing steps that may be performed by certain components of the present system (e.g., by the processing subsystem 112) may be implemented by appropriate code on a processor-based system. To this end, the processor-based system may include, for example, a general-purpose computer or a special-purpose computer. It may also be noted that different embodiments of the present specification may perform some or all of the steps described herein in a different order or substantially simultaneously.
[0024] According to various aspects of the present specification, image data collected and / or processed by the medical imaging system 108 can be used to perform one or more tasks. In one example, the processing subsystem 112 may include a machine learning model 106, such as a neural network configured to perform a task. Specifically, the machine learning model 106 can be trained to perform a task using a data set corresponding to a source domain. As a non-limiting example, the machine learning model 106 can be trained to classify an input image and / or segment one or more regions in an input image to assist a clinician in providing a diagnosis. In certain embodiments, the processing subsystem 112 may also be coupled to a storage system, such as a data repository 116, wherein the data repository 116 is configured to store the collected image data. In certain embodiments, the data repository 116 may include a local database (not shown).
[0025] In addition, according to aspects of the present specification, the imaging system 100 may further include a memory unit 118. Figure 1 The configuration depicts the data repository 116 as including the memory unit 118, but in other embodiments, the memory unit 118 may be a separate unit external to the data repository 116 and / or the imaging system 100. The memory unit 118 is configured to store input images and outputs generated by the system 100.
[0026] As previously described, when a machine learning model is deployed in a new target domain, currently available techniques suffer from performance degradation of the machine learning model. According to various aspects of the present specification, the imaging system 100 is designed to avoid the shortcomings of currently available techniques. More specifically, the imaging system 100 includes a memory augmentation domain adaptation platform 114, which is configured to help the machine learning model automatically adapt to the new target domain. The exemplary system 100 including the memory augmentation domain adaptation platform 114 provides a framework for deploying the machine learning model 106 in the target domain by enabling the machine learning model 106 to adapt to the target domain using a relatively small set of images corresponding to the target domain, which in turn will simplify the clinical workflow. Specifically, the memory augmentation domain adaptation platform 114 works in conjunction with the machine learning model 106 to enhance the adaptability of the machine learning model 106 to the target domain, thereby improving the performance of the imaging system 100. It can be noted that the terms "new domain", "target site" and "target domain" are used interchangeably.
[0027] Moreover, in Figure 1In the presently contemplated configuration shown, the processing subsystem 112 is shown to include a memory augmentation domain adaptation platform 114. However, in certain embodiments, the memory augmentation domain adaptation platform 114 may also be used as a standalone unit that is physically separate from the processing subsystem 112 and the medical imaging system 108. By way of example, the memory augmentation domain adaptation platform 114 may be located external to the medical imaging system 108 and operably coupled to the medical imaging system.
[0028] The exemplary memory augmentation domain adaptation platform 114 is configured to circumvent the shortcomings of currently available threshold adaptation techniques. More specifically, the memory augmentation domain adaptation platform 114 is configured to facilitate adaptation of a machine learning model to a target domain using a small set of sample images corresponding to the target domain, thereby leading to consistent results.
[0029] As previously mentioned, a given model is typically trained using a data set corresponding to a source domain. It is desirable that the machine learning model 106 adapts to the target domain when deployed in the target domain while maintaining the performance of the system 100. Therefore, when the imaging system 100 is deployed in the target domain, the memory augmentation domain adaptation platform 114 is configured to adapt the machine learning model 106 to the target domain to perform a given task. By way of example, the memory augmentation domain adaptation platform 114 may be configured to help the machine learning model process the acquired input image to classify the input image and / or segment one or more regions of interest in the input image.
[0030] Therefore, in operation, the machine learning model 106 and / or the memory augmentation domain adaptation platform 114 are configured to receive an input image, wherein the input image corresponds to the target domain. In addition, the memory augmentation domain adaptation platform 114 is configured to calculate one or more features corresponding to the input image. Features include shape features, texture features, etc. In certain other embodiments, other features may be used to identify a set of support images. Some non-limiting examples of other features include the age of the patient 102, the gender of the patient 102, electronic medical record (EMR) information corresponding to the patient 102, demographics, etc.
[0031] It may be noted that the memory unit 118 is configured to store one or more images corresponding to the target domain. In addition, the memory unit 118 may also be configured to store one or more features, such as texture features and / or shape features, and masks corresponding to the images of the target domain. In addition, other features corresponding to the images of the target domain may be stored, such as age, gender, EMR information, demographics, etc.
[0032] In addition, after receiving the input image, the memory augmentation domain adaptation platform 114 is configured to query the data repository 116, and in particular, the memory unit 118, to identify a set of matching images based on the features of the input image. In one example, following the query, the memory augmentation domain adaptation platform 114 may identify a set of supporting images in the memory unit 118 based on the image features corresponding to the input image. It can be noted that according to aspects of the present specification, in some embodiments, the system 100 can be configured to allow a clinician to identify the set of supporting images.
[0033] It may be noted that the set of support images corresponds to the target domain and may be a subset of images corresponding to the target domain. In one non-limiting example, the set of support images includes images ranging from about three images to about five images. However, other numbers of images are also contemplated for use in the set of support images. More specifically, a small subset of target domain images may be used as the set of support images. Using a small subset of target domain images as the set of support images helps avoid the need for currently available techniques to use a large pool / set of training data.
[0034] Conventionally, currently available techniques only provide input images to a machine learning model to perform a given task. According to aspects of the present specification, the memory augmentation domain adaptation platform 114 is configured to augment the input to the machine learning model 106 with the set of support images. Specifically, the memory augmentation domain adaptation platform 114 is configured to compute a set of features and / or a set of masks corresponding to the set of support images, and provide the set of features and / or the set of masks as additional input to the machine learning model 106 to adapt the machine learning model 106 to the target domain. Thus, the machine learning model 106 "adapts" or "learns" the target domain using a small set of support images.
[0035] In addition, the memory augmentation domain adaptation platform 114 facilitates the generation of outputs by the machine learning model 106 based at least on the set of features and / or the set of masks corresponding to the set of support images. In addition, the memory augmentation domain adaptation platform 114 is configured to provide outputs to facilitate analysis. In addition, the generated outputs may be based on the tasks performed by the machine learning model 106. For example, if the machine learning model 106 is configured to classify the input image, the output may be a binary value. However, if the machine learning model 106 is configured to segment the input image, the output may be an image corresponding to the segmented region of interest. In addition, in one example, the output may be visualized on an interface unit such as a display 120.
[0036] In addition, if Figure 1As shown, the medical imaging system 108 may include a display 120 and a user interface 122. In certain embodiments, such as in a touch screen, the display 120 and the user interface 122 may overlap. Moreover, in some embodiments, the display 120 and the user interface 122 may include a common area. According to various aspects of the present specification, the display 120 of the medical imaging system 108 may be configured to display or present the output generated by the machine learning model 106. In addition, any quality metrics / indicators generated by the memory augmentation domain adaptation platform 114 may also be visualized on the display 120.
[0037] In addition, the user interface 122 of the medical imaging system 108 may include a human interface device (not shown) that is configured to help the clinician manipulate the image data displayed on the display 120. The human interface device may include a mouse-type device, a trackball, a joystick, a stylus, or a touch screen that is configured to facilitate the clinician to identify one or more regions of interest in the image. However, it should be understood that other human interface devices, such as but not limited to touch screens, may also be used. In addition, according to various aspects of the present specification, the user interface 122 may be configured to help the clinician navigate through the collected images and / or outputs generated by the medical imaging system 108. In addition, the user interface 122 may also be configured to help manipulate and / or organize the display images and / or generated indicators displayed on the display 120.
[0038] Implementing the imaging system 100 including the memory augmentation domain adaptation platform 114 as described above helps to enhance the performance of the machine learning model 106 when the model 106 is deployed in a new target domain. Specifically, the memory augmentation domain adaptation platform 114 helps to facilitate the adaptation of the machine learning model 106 to the target domain via the use of a small set of support images. In addition, the memory augmentation domain adaptation platform 114 provides continuous learning to the machine learning model 106 via the use of the set of support images, thereby improving the performance of the machine learning model 106 when the model 106 is deployed in the new target domain.
[0039] In this specification, the instructions may be described in the general context of computer-executable instructions on a computing system or processor. Figure 2-Figure 3 Generally speaking, computer-executable instructions may include routines, programs, objects, components, data structures, procedures, modules, functions, etc. that perform specific functions or implement specific abstract data types.
[0040] In addition, it can also be practiced in a distributed computing environment Figure 2-Figure 3Embodiments of the exemplary method of the invention, wherein the optimization function is performed by a remote processing device linked by a wired and / or wireless communication network. In a distributed computing environment, computer executable instructions may be located in local and remote computer storage media (including memory storage devices).
[0041] In addition, Figure 2-Figure 3 In the present invention, the exemplary method is shown as a collection of blocks in a logical flow diagram, which represents operations that can be implemented in hardware, software, or a combination thereof. Various operations are depicted in blocks to illustrate the functions performed. In the context of software, the blocks represent computer instructions that, when executed by one or more processing subsystems, perform the recited operations.
[0042] describe Figure 2-Figure 3 The order of the exemplary methods is not intended to be construed as limiting, and any number of the described blocks may be combined in any order to implement the exemplary methods disclosed herein or an equivalent alternative method. Additionally, certain blocks may be deleted from the exemplary methods or enhanced by additional blocks with added functionality without departing from the spirit and scope of the subject matter described herein. Although the exemplary embodiments shown below are described in the context of a medical imaging system, it should be understood that use of these systems and methods in industrial applications is also contemplated in conjunction with this specification.
[0043] See now Figure 2 , presents a flowchart 200 depicting an exemplary method for adapting a machine learning model to a target domain. Figure 1 to describe the components Figure 2 In one embodiment, the method 200 may be performed by the memory augmentation domain adaptation platform 114 in conjunction with the machine learning model 106.
[0044] The method includes receiving an input image when the imaging system 100 (particularly the machine learning model 106) is deployed in the target domain, as indicated in step 202. The input image corresponds to the target domain. In addition, the input image can be received by the machine learning model 106 and the memory augmentation domain adaptation platform 114.
[0045] In addition, at step 204, one or more features corresponding to the input image are calculated by the memory augmentation domain adaptation platform 114. These features may include texture features, shape features, or a combination thereof. Some non-limiting examples of texture features include wavelet features, machine learning features, etc. In addition, some non-limiting examples of shape features include contour-based features, features derived from dictionary-based methods, moments, shape representations such as area, cut angles, contour curvature, shape transform domain features such as Fourier transform, etc. In addition, as previously mentioned, some examples of other features include the patient's age, gender, EMR information, etc.
[0046] It may be noted that the images corresponding to the target domain may be stored in the memory unit 118. After calculating the image features corresponding to the input image, a set of supporting images may be identified by the memory augmentation domain adaptation platform 114 based on the image features of the input image, as indicated by step 206. Specifically, the memory augmentation domain adaptation platform 114 is configured to query the memory unit 118 using the image features corresponding to the input image to identify the set of supporting images from the images stored in the memory unit 118. It may be noted that the set of supporting images is a subset of the images corresponding to the target domain. In a non-limiting example, the set of supporting images includes images in the range of about three images to about five images. In addition, in a scenario where the query to the memory unit 118 fails to identify one or more supporting images, one or more images corresponding to the source domain may be used as supporting images.
[0047] Conventional machine learning techniques generate outputs based solely on received input images, resulting in performance degradation of machine learning models in new domains. According to aspects of the present specification, the shortcomings of currently available techniques are avoided by using a retrieved set of support images. The machine learning model 106 uses the set of support images to provide context for predictions, rather than using only the input image.
[0048] More specifically, the method includes augmenting the input to the machine learning model with a set of features and / or a set of masks corresponding to the set of support images to adapt the machine learning model to the target domain, as indicated in step 208. Thus, the memory augmentation domain adaptation platform 114 is configured to compute one or more features and / or masks corresponding to the retrieved set of support images. In one example, features corresponding to the set of support images can be computed using tunable feature extraction suitable for segmentation. In another example, support context vector augmentation can be used to compute features corresponding to the set of support images. In yet another example, features corresponding to the set of support images can be computed by imitating a setting corresponding to the target domain with data corresponding to the source domain during a training phase of the machine learning model 106.
[0049] As previously described, these features may include texture features and / or shape features corresponding to the set of support images. Subsequently, the memory augmentation domain adaptation platform 114 provides the features and / or masks corresponding to the set of support images as additional inputs to the predictor of the machine learning model 106, thereby enhancing the input to the machine learning model 106. In addition, other features may also be provided as additional inputs to the predictor of the machine learning model 106, as previously described.
[0050] In addition, at step 210, the machine learning model 106 is configured to generate an output based on at least a set of features and / or a set of masks corresponding to the set of support images provided by the memory augmentation domain adaptation platform 114 and the input image. The output generated by the machine learning model 106 may vary based on the task performed by the machine learning model 106. By way of example, if the machine learning model 106 is used to perform a classification task, the output generated by the machine learning model 106 may be a binary value. In a similar manner, if the machine learning model 106 is used to perform a segmentation task, the output generated by the machine learning model 106 may be a mask or a segmented image.
[0051] In addition, the output can be used to facilitate analysis, as indicated in step 212. By way of example, the memory augmentation domain adaptation platform 114 can be configured to visualize the mask or segmentation image and / or binary value generated by the machine learning model 106 on the display 120. In some embodiments, a visual comparison of the performance of the system 100 with and without domain adaptation can be visualized on the display 120 to assist the clinician in any diagnosis or analysis. In addition, any metrics associated with the output can also be visualized on the display 120. In some embodiments, the metrics can be superimposed on the corresponding output on the display 120.
[0052] In another example, the memory augmentation domain adaptation platform 114 may also be configured to transmit the generated output to a user, such as a clinician or another system. The clinician and / or another system may use the output to facilitate diagnosis and / or analysis. Figure 3 The method 200 is described in more detail.
[0053] Now go to Figure 3 , depicts a diagram showing the Figure 2 Schematic diagram 300 of an exemplary method for memory augmentation domain adaptation. Figure 1-Figure 2 to describe the components Figure 3 .
[0054] As previously noted, the machine learning model 106 is typically trained using a data set corresponding to a source domain. Once the machine learning model 106 is deployed in a new target domain, the exemplary memory augmentation domain adaptation platform 114 is configured to adapt the machine learning model 106 to the target domain using a relatively small set of support images. In a non-limiting example, the set of support images may include images ranging from about three images to about five images.
[0055] In addition, if Figure 3As depicted, an input image 302 corresponding to a target domain is received by the machine learning model 106 and the memory augmentation domain adaptation platform 114. Subsequently, the memory augmentation domain adaptation platform 114 computes one or more features 304 corresponding to the input image 302. These features include texture features, shape features, and / or other features.
[0056] In addition, the memory augmentation domain adaptation platform 114 is configured to identify a set of support images corresponding to the target domain using the computed features associated with the input image 302. Therefore, the memory augmentation domain adaptation platform 114 is configured to query the memory unit 118 based on the features corresponding to the input image 302 to identify the presence of a match in the memory unit 118. Specifically, the memory augmentation domain adaptation platform 114 is configured to identify a set of support images based on one or more image features corresponding to the input image 302. The set of support images corresponds to the target domain and includes features that match the features of the input image 302.
[0057] It can be pointed out that in the traditional U-Net, given a training pair of image and segmentation mask {I k , S k}(k=1, 2, ..., N), the framework learns a predictor defined by the parameters w that minimizes the training loss One such example is presented in equation (1).
[0058]
[0059] in is a learned predictor and is a composite of an encoder and a decoder
[0060] According to aspects of the present specification, a memory cell M is provided, such as the memory cell 118. The memory cell M118 is composed of a matrix T N×Ft (where F t is the characteristic length of the texture feature) and the matrix G N×Fs (where F s In one example, the memory cell M 118 is defined as:
[0061] M=(T N×Ft , G N×Fs ) (2)
[0062] In response to the query, for each input image I K , the memory unit M118 is configured to return a set of supporting images S(I K )306. The group supports images S(I K)306 is presented in formula (3).
[0063] S(I K )=p t , {t=1,2,...,T} (3)
[0064] In one example, in formula (1), T is a constant and represents the set of support images S(I K ) 306. As previously noted, the set of support images S(I k ) 306 corresponds to the target domain and is a subset of images corresponding to the target domain. In a non-limiting example, the set of support images includes images ranging from about three images to about five images. However, it is also contemplated to use other numbers of images in the set of support images.
[0065] For a given image I K , the group supports image S(I k )306 is calculated as:
[0066] (s1,s2,...,s T )=NN T (q(I k ), M) (4)
[0067] Where q corresponds to the input image I K The features of NN T is a given input q(I k )’s T nearest neighbors.
[0068] The nearest neighbor operator NN is defined as:
[0069] A=NN(I k , M) = argmax i q(I k )·q(M i ) (5)
[0070] The memory unit 118 functions similarly to a programmable memory and facilitates adaptation and continuous learning of the machine learning model 106 to the target domain using a small set of support images corresponding to the target domain. Thus, when the machine learning model 106 subsequently encounters a similar situation, the memory unit 106 is queried for a match. More specifically, the additional memory unit 118 facilitates retrieval of support images / samples and corresponding annotations that are similar to a particular use case.
[0071] In addition, the memory augmentation domain adaptation platform 114 is configured to support the image S (I k ) 306 augments the input to the machine learning model 106. The machine learning model 106 uses the set of support images S(Ik ) 306 to provide context for prediction, rather than using only the input image. More specifically, the memory augmentation domain adaptation platform 114 is configured to compute contexts related to the set of support images S(I k ) 306 corresponding to a set of features and / or a set of masks 308. As previously referenced Figure 2 As noted, different methods can be used to calculate the image corresponding to the set of support images S(I k )308 features / mask 308.
[0072] In addition, the memory augmentation domain adaptation platform 114 is configured to associate the image with the set of support images S(I k ) 306 and / or the set of masks 308 are provided as additional inputs to the machine learning model 106 to adapt the machine learning model 106 to the target domain. More specifically, in the machine learning model 106 (e.g., supporting augmented neural networks), the predictor of the machine learning model 106 is The input is used with the set of support images S(I k ) 306 and a set of texture and / or shape features and a set of masks 308 to augment. In addition, the machine learning model 106 uses the set of support images S(I k ) 306 to provide context for prediction, rather than using only the input image 302. Texture features and shape features are defined as follows:
[0073] Texture features: T:I K →T Ft×1 (6)
[0074] Shape features: G:S k →T Fs×1 (7)
[0075] More specifically, to the predictor The decoder input is changed to the learned encoded input E(I K ) and the group of support images S(I k ) 306 corresponding to the shape and / or texture features and / or masks 308. By way of example, the memory augmentation domain adaptation platform 114 is configured to provide the machine learning model 106 with one or more machine learning features, one or more hard-coded features, and a combination of shape and / or texture features and / or masks 308 corresponding to the set of support images S(I k )306 mask 308 or a combination thereof as input.
[0076] In addition, the machine learning model 106 is configured to perform desired tasks, such as but not limited to segmentation tasks and classification tasks. Therefore, the machine learning model 106 is configured to perform at least one segmentation task based on the image corresponding to the set of support images S(I k) 306 and / or a set of masks 308 to generate a modified output 310. In one example, the output 310 includes a segmented image / mask, a binary value, or a combination thereof 312. Additionally, in one example, the modified output 310 generated by the machine learning model 106 is represented as:
[0077]
[0078] in is an operator such as, but not limited to, concatenation, average, sum, etc., and is used to combine different features and / or masks.
[0079] The mask / image and / or binary value 312 generated as an output by the machine learning model 106 is transmitted to the memory augmentation domain adaptation platform 114. In addition, the memory augmentation domain adaptation platform 114 is configured to facilitate continuous learning of the machine learning model 106. Therefore, the memory augmentation domain adaptation platform 114 is configured to verify the validity of the output 310 (step 314). At step 314, if the output 310 is verified to be valid, the memory unit 118 is updated to store the output 310 (step 316). Specifically, at step 316, the memory unit 118 corresponding to the set of support images S (I k ) 306 and / or one or more masks 320 are stored in the memory unit 118. It can be noted that the features and / or masks 320 are tuned for performing tasks such as, but not limited to, classification and segmentation.
[0080] However, at step 314, if the validity of output 310 is not verified, the image data corresponding to the set of supporting images S(I) may be revised. k ) 306 to generate a revised set of support images (step 318). Subsequently, the memory unit 118 is updated with the revised set of support images and the features and / or masks corresponding to the revised set of support images. This verification and updating of the memory unit 118 helps to promote the continuous learning of the machine learning model 106, thereby enhancing the domain adaptation capability of the machine learning model 106.
[0081] In some embodiments, the memory augmentation domain adaptation platform 114 may also be configured to update the memory unit 118 to optimize the memory unit M 118. By way of example, the memory augmentation domain adaptation platform 114 may update the memory unit 118 based on the set of support images S(I k )306 to delete the group of support images S(I k) 306 to optimize the memory unit 118 while achieving enhanced performance of the imaging system 100. By way of example, to track the relevance of the supporting images 306, the memory augmentation domain adaptation platform 114 may be configured to monitor the number of times the supporting images 306 are used for prediction, check the age of the supporting images 306 in the memory unit 118, determine the similarity of the supporting images 306 to other images in the memory unit 118, etc.
[0082] Additionally, the memory augmentation domain adaptation platform 114 is configured to provide or transmit output 310 to facilitate analysis (step 322). In one non-limiting example, the output 310 may be visualized on an interface unit such as a display 120. The output 310 may be used to provide a diagnosis or for further analysis.
[0083] Implementing the memory augmentation domain adaptation platform 114 as described above facilitates adapting the machine learning model 106 to the target domain while avoiding the need to retrain the machine learning model 106. Furthermore, since the machine learning model 106 is adapted to the target domain using a small set of support images 306, the need for a large data set corresponding to the target domain to retrain the machine learning model 106 is avoided. Furthermore, the machine learning model 106 is continuously trained using a set of support images 306 stored in the memory unit 118 and provided by the memory augmentation domain adaptation platform 114, thereby further enhancing the adaptation of the machine learning model 106 to the target domain.
[0084] See now Figure 4(a)-Figure 4(c) , presents a schematic diagram of different datasets 402, 404, 406 corresponding to different domains. Specifically, these datasets 402, 404, 406 correspond to different domains with subjects suffering from different diseases to simulate the deployment scenario of the machine learning model. In addition, reference Figure 1-Figure 3 to describe the components Figure 4(a)-Figure 4(c) It can be noted that in this example, the data sets 402, 404, 406 include X-ray images corresponding to three different domains. In addition, the samples such as X-ray images corresponding to the three data sets 402, 404, 406 have variations in texture, disease conditions, and gender. Figure 4(a)-Figure 4(c) In the depicted example, the use of a machine learning model to perform a lung segmentation task based on an X-ray image is shown. It should be understood that lung segmentation is considered a challenging task due to variations in the lungs due to variations in anatomy, disease, etc. in different patients. Three datasets 402, 404, 406 are used to understand the impact of variations in the domain on the performance of the machine learning model. U-Net is used as the base learning model.
[0085] FIG4( a) depicts a first data set 402. In this example, the first data set 402 is the Montgomery TB data set. The Montgomery TB data set 402 is an open source NHS data set and includes 138 posterior-anterior X-ray images. Of these images, 80 X-ray images represent normal data and 58 X-ray images represent abnormal data with tuberculosis manifestations. Furthermore, in this example, the Montgomery TB data set 402 is a source data set for training the machine learning model 106.
[0086] In addition, FIG4( b) depicts a second dataset 404. The second dataset 404 is a GE pneumoconiosis dataset. The GE pneumoconiosis dataset 404 includes 330 images with lung mask annotations. It can be noted that pneumoconiosis is an occupational lung disease that commonly affects factory workers, and the early stages of the disease can be detected by dust and other metal precipitation in the lungs. In this example, the GE pneumoconiosis dataset 404 corresponds to the first target domain.
[0087] In addition, FIG4(c) shows a third data set 406. The third data set 406 is a Japanese Society of Radiological Technology (JSRT) data set. The JSRT data set 406 includes chest X-rays with lung nodules. In addition, the JSRT data set 406 includes 247 images. In this example, the JSRT data set 406 corresponds to the second target domain.
[0088] Furthermore, to demonstrate the effectiveness of using the memory augmentation domain adaptation platform 114, the machine learning model 106 is trained using samples obtained only from the MontgomeryTB dataset 402. Once the machine learning model 106 is trained using samples from the MontgomeryTB dataset 402, the machine learning model 106 is deployed in a new target domain to test the adaptability of the machine learning model 106 to the target domain using samples from the corresponding target domain. In one example, the trained machine learning model 106 is deployed in the new target domain, and the domain adaptation of the machine learning model 106 is tested using samples corresponding to the GE pneumoconiosis dataset 404 and the JSRT dataset 406. The results are verified using the Dice score.
[0089] Table 1 presents a comparison of domain adaptation execution of various machine learning models using the datasets 402, 404, 406 of Figures (4a)-4(c). The first column of Table 1 lists the techniques used in the comparison study. In addition, the second column of Table 1 corresponds to the source dataset 402 (Montgomery TB dataset). In addition, the third column of Table 1 corresponds to the first target dataset 404 (GE pneumoconiosis dataset). The fourth column of Table 1 corresponds to the second target dataset 406 (JSRT dataset). In addition, the first row of Table 1 corresponds to the execution of the base technique or learning model (U-Net) in the three domains 402, 404, 406. Similarly, the second row of Table 1 corresponds to the execution of the method for adapting the machine learning model 106 (SupportNet) described above in the three domains 402, 404, 406. It can be noted that for the results presented in the second row of Table 1, the augmented input provided to the machine learning model 106 by the memory augmentation domain adaptation platform 114 only includes a set of features corresponding to a set of support images. In addition, row 3 of Table 1 corresponds to the execution of the present method (SupportNet) in three domains 402, 404, 406. It can be noted that in row 3 of Table 1, the augmented input provided by the memory augmentation domain adaptation platform 114 to the machine learning model 106 only includes a set of features and / or a set of masks corresponding to a set of support images.
[0090] Table 1
[0091]
[0092] Furthermore, a Dice score of 1 is used as the metric to obtain the results presented in Table 1. Based on the results presented in Table 1, it can be inferred that SupportNet (see rows 2 and 3) performs better than U-Net (see row 1) on images obtained from different cohorts / domains using augmented inputs of support features and support masks. Therefore, SupportNet enjoys better generalization than U-Net.
[0093] Now go to Figure 5(a)-Figure 5(e) , depicts a graphical illustration of the comparison of the results of domain adaptation performance of different machine learning models presented in Table 1 to perform the desired task. Figure 5(a)-Figure 5(e) In the example presented in , it is desired to use the machine learning model 106 to segment the lung region in the input image 502. Figure 1 - Figure 4 components are described Figure 5(a)-Figure 5(e) .
[0094] FIG5(a) shows an input image 502, such as Figure 35( b ) shows a ground truth mask 504. In one example, the image 504 may include a clinician's ground truth annotations.
[0095] In addition, FIG5(c) shows a segmentation image 506 generated using a baseline technique such as U-net without domain adaptation. Similarly, FIG5(d) shows a segmentation image 508 generated by SupportNet using only features corresponding to a set of support images. In addition, FIG5(e) shows a segmentation image 510 generated by SupportNet using features corresponding to a set of support images and masks.
[0096] It can be noted that using only features from the set of support images corresponding to the nearest neighbors helps the SupportNet to have suitable feature priors. Furthermore, when the features from the set of support images are augmented with masks from the set of support images, the machine learning model achieves the dual purpose of balancing the shape and providing high fidelity to the image.
[0097] Embodiments of the present system and method for memory augmentation domain adaptation advantageously present techniques based on continuous learning to adapt machine learning models to a target domain, thereby delivering better performance with a small set of samples. In addition, the output generated by the present system and method promotes enhanced understanding of predictions by comparing a set of similar examples used to reach a decision. In addition, the system and method achieves domain adaptation and continuous learning of machine learning models without the constraint of requiring a large data set of target domain samples during the development phase of the technology. Specifically, the system and method enable domain adaptation with a very small set of samples.
[0098] In addition, the system and method entail the use of "meta-learning" techniques designed to improve the performance of machine learning models when deployed in new domains. Specifically, the memory unit is used to facilitate the adaptation and continuous learning of the machine learning model to the target domain using a small set of samples corresponding to the target domain. Thus, when the machine learning model subsequently encounters a similar situation, the memory unit is queried for a match. More specifically, the additional memory unit facilitates the retrieval of samples and corresponding annotations that are similar to a specific use case. In addition, the annotations corresponding to the retrieved samples stored in the memory unit can be revised, thereby providing enhanced control over subsequent predictions of the machine learning model.
[0099] It may be noted that the above-described examples, demonstrations, and processing steps that may be performed by certain components of the present system (e.g., by the processing subsystem 112 and, in particular, the memory augmentation domain adaptation platform 114) may be implemented by appropriate code on a processor-based system. For example, the processor-based system may include a general-purpose computer or a special-purpose computer. It may also be noted that different implementations of the present specification may perform some or all of the steps described herein in a different order or substantially simultaneously.
[0100] Additionally, these functions may be implemented in a variety of programming languages, including but not limited to Ruby, Hypertext Preprocessor (PHP), Perl, Delphi, Python, C, C++, or Java. Such code may be stored or adapted to be stored on one or more tangible machine-readable media, such as a data repository chip, a local or remote hard disk, an optical disk (i.e., a CD or DVD), a solid-state drive, or other media accessible by a processor-based system to execute the stored code.
[0101] Although specific features of the embodiments of the present specification may be shown in some drawings and not in others and / or described with reference to some drawings and not in others, this is for convenience only. It should be understood that in various embodiments, the described features, structures and / or characteristics may be combined and / or used interchangeably in any suitable manner.
[0102] Although only certain features of the present disclosure have been shown and described herein, many modifications and changes will occur to those skilled in the art. It should be understood, therefore, that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the invention.
Claims
1. A system for memory augmentation domain adaptation, comprising: an acquisition subsystem configured to obtain an image corresponding to a target domain; a processing subsystem operably associated with the acquisition subsystem and comprising a memory augmentation domain adaptation platform, wherein the memory augmentation domain adaptation platform is configured to: computing one or more features corresponding to an input image, wherein the input image corresponds to a target domain; identifying a set of support images based on the one or more features corresponding to the input image, wherein the set of support images corresponds to the target domain; augmenting an input to a machine learning model with a set of features, a set of masks, or both, corresponding to the set of support images to adapt the machine learning model to the target domain, wherein the features corresponding to the set of support images are computed by imitating a setting corresponding to the target domain with data corresponding to a source domain during a training phase of the machine learning model; generating an output based at least on the set of features, the set of masks, or both the set of features and the set of masks corresponding to the set of supporting images; and An interface unit is configured to present the output for analysis.
2. The system of claim 1, wherein the machine learning model is configured to perform at least classification, segmentation, or both classification and segmentation of the input image.
3. The system of claim 1, wherein the one or more features corresponding to the input image and the set of features corresponding to the set of support images include shape features, texture features, or both shape features and texture features.
4. The system of claim 1 , wherein the memory augmentation domain adaptation platform is configured to store one or more features, one or more masks, or both the one or more features and the one or more masks corresponding to the set of supporting images, and wherein the one or more features, the one or more masks, or both the one or more features and the one or more masks are tuned for performing classification, segmentation, or both classification and segmentation in a memory unit. 5 . The system of claim 1 , wherein to identify the set of supporting images, the memory augmentation domain adaptation platform is configured to query a memory unit to determine the set of supporting images based on the one or more features corresponding to the input image.
6. The system of claim 1 , wherein to augment the input to the machine learning model, the memory augmentation domain adaptation platform is configured to provide one or more machine learning features, one or more hard-coded features, one or more masks corresponding to the set of support images, or a combination thereof as input to the machine learning model. 7 . The system of claim 1 , wherein to generate the output, the memory augmentation domain adaptation platform is configured to generate a mask, generate a binary value, or a combination thereof.
8. The system of claim 1, wherein the memory augmentation domain adaptive platform is further configured to: verifying the validity of said output; revising one or more annotations corresponding to the set of supporting images based on the outputted validity to generate a revised set of supporting images; and The memory unit is updated by storing the revised set of supporting images and features corresponding to the revised set of images.
9. The system of claim 1, wherein the memory augmentation domain adaptation platform is further configured to adapt the machine learning model to the target domain without retraining the machine learning model.
10. A processing system for adapting a machine learning model, comprising: A memory expansion domain adaptive platform, wherein the memory expansion domain adaptive platform is configured as follows: computing one or more features corresponding to an input image, wherein the input image corresponds to a target domain; identifying a set of support images based on the one or more features corresponding to the input image, wherein the set of support images corresponds to the target domain; augmenting an input to the machine learning model with a set of features, a set of masks, or both, corresponding to the set of support images to adapt the machine learning model to the target domain, wherein the features corresponding to the set of support images are computed by imitating a setting corresponding to the target domain with data corresponding to a source domain during a training phase of the machine learning model; generating an output based at least on the set of features, the set of masks, or both the set of features and the set of masks corresponding to the set of supporting images; and The output is provided to facilitate analysis.
11. A method for adapting a machine learning model, the method comprising: receiving an input image, wherein the input image corresponds to a target domain; calculating one or more features corresponding to the input image; identifying a set of support images based on the one or more features corresponding to the input image, wherein the set of support images corresponds to the target domain; augmenting an input to the machine learning model with a set of features, a set of masks, or both, corresponding to the set of support images to adapt the machine learning model to the target domain, wherein the features corresponding to the set of support images are computed by imitating a setting corresponding to the target domain with data corresponding to a source domain during a training phase of the machine learning model; generating an output based at least on the set of features, the set of masks, or both the set of features and the set of masks corresponding to the set of supporting images; and The output is provided to facilitate analysis.
12. The method of claim 11, further comprising storing one or more features and one or more masks corresponding to the group of supporting images, the output, or a combination thereof, wherein the one or more features, the one or more masks, or both the one or more features and the one or more masks are tuned for performing classification, segmentation, or both classification and segmentation in a memory unit.
13. The method of claim 11, wherein identifying the set of supporting images comprises querying a memory unit to determine the set of supporting images based at least on the one or more features corresponding to the input image.
14. The method of claim 11, wherein augmenting the input to the machine learning model comprises: computing the one or more machine-learned features, the one or more hard-coded features, or both the one or more machine-learned features and the one or more hard-coded features corresponding to the set of support images; and The machine learning model is provided with one or more machine learning features corresponding to the set of support images, one or more hard-coded features corresponding to the set of support images, one or more masks corresponding to the set of support images, or a combination thereof as input.
15. The method of claim 11, wherein generating the output comprises generating a mask, generating a binary value, or a combination thereof.
16. The method according to claim 11, further comprising: verifying the validity of said output; revising one or more annotations corresponding to the set of supporting images based on the outputted validity to generate a revised set of supporting images; as well as The memory unit is updated by storing the revised set of supporting images, features and masks corresponding to the revised set of images, or a combination thereof.
17. The method of claim 16, further comprising updating the memory unit by deleting one or more of the set of supporting images based on relevance of the set of supporting images.
18. The method of claim 11, wherein the set of support images includes a subset of images corresponding to the target domain.
19. The method of claim 11 further comprising adapting the machine learning model to the target domain without retraining the machine learning model.
20. The method of claim 11, wherein providing the output comprises presenting the output on an interface unit.
