Iterative framework for learning multi-modal mapping customized for medical image inference tasks

Abstract

The invention relates to an iterative framework for learning multi-modal mappings customized for medical image inference tasks. An iterative framework for learning multi-modal mappings tailored for medical image inference tasks is provided. In an example, a computer-implemented method may include receiving multi-modal annotation data for a medical image, the multi-modal annotation data including non-image annotation data and image annotation data; and employing one or more machine learning (ML) processes to learn a bidirectional mapping between non-image features included in the non-image annotation data and image features associated with the medical image and the image annotation data. The method further includes generating, as a result of the one or more ML processes, a model configured to: infer, given a new medical image, one or more of the non-image features associated with the new medical image, and / or inferring one or more of the image features associated with the new medical image given the new medical image and the non-image input corresponding to at least some of the non-image annotation data.

Description

Technical Field

[0001] The present application relates to artificial intelligence (AI) in the field of medical imaging, and more specifically to an iterative framework for learning multimodal mappings tailored for medical image inference tasks. Background Art

[0002] Deep learning (DL) models have demonstrated state-of-the-art performance in various medical image processing tasks such as organ segmentation, anomaly detection, diagnostic classification, risk prediction, temporal analysis, image reconstruction, etc. However, they suffer from three major problems: 1) the need for large-scale annotated data, 2) asynchronous model development between different development stages (e.g., data curation and annotation, model building, and report generation), and 3) lack of interpretability. Summary of the invention

[0003] The following presents a summary of the invention to provide a basic understanding of one or more embodiments of the invention. The summary of the invention is not intended to identify key or important elements, nor is it intended to delineate any scope of the different embodiments or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a prelude to a more detailed description presented later. In one or more embodiments, a system, computer-implemented method, apparatus, and / or computer program product that provides an iterative framework for learning multimodal mappings tailored for medical image inference tasks is described.

[0004] According to one embodiment, a system is provided, the system comprising a memory and a processor, the memory storing a computer executable component, the processor executing the computer executable component stored in the memory. The computer executable component may include a receiving component, the receiving component receiving multimodal annotation data for a medical image included in a medical image dataset, the multimodal annotation data including non-image annotation data (e.g., text data and / or other forms of non-image input) and image annotation data (e.g., label data applied to the medical image). The computer executable component also includes a machine learning component, the machine learning component using one or more machine learning processes to learn a bidirectional mapping between non-image features included in the non-image annotation data and image features associated with the medical image and the image annotation data.

[0005] In some embodiments, as a result of one or more machine learning processes, the machine learning component generates a model that is configured to infer one or more of the non-image features associated with the new medical image given the new medical image.

[0006] Additionally or alternatively, as a result of one or more machine learning processes, the machine learning component generates a model configured to infer one or more of the image features associated with the new medical image given a new medical image and non-image inputs that identify or indicate one or more of the non-image features respectively associated with the new medical image. By these embodiments, in combination with the inferred one or more of the image features, the machine learning component can configure the model to generate new image annotation data for the new medical image that defines one or more of the image features. For example, the image annotation data and the new image annotation data can include labeling data applied to the medical image that labels one or more anatomical features, and the non-image annotations and non-image inputs can include text data, respectively. The computer executable component can also include an annotation component that employs the model to generate new image annotation data for the new medical image given the new medical image and non-image inputs.

[0007] In various embodiments, the non-image annotation data follows one or more standards (e.g., expert-defined clinical standards customized for a specific medical image inference task and / or the medical image at hand). The system can also facilitate iteratively updating the standards to improve bidirectional mapping and model performance, thereby coupling data annotation and model development in the same framework. For example, the receiving component can receive updated non-image annotation data for medical images that follows one or more updated (e.g., expert-defined) standards based on one or more errors associated with the model's performance of the new medical image. The machine learning component can also employ one or more second machine learning processes to learn an updated bidirectional mapping between updated non-image features included in the updated non-image annotation data and image features associated with the medical image, and wherein as a result of the one or more second machine learning processes, the machine learning component generates an updated version of the model, the updated version of the model being configured to more accurately infer one or more of the image features associated with the new medical image given the new medical image and non-image input.

[0008] In some embodiments, elements described in conjunction with the disclosed systems can be embodied in different forms, such as a computer-implemented method, a computer program product, or another form. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An iterative process for learning a multimodal mapping tailored for a medical image inference task in accordance with one or more embodiments of the disclosed subject matter is presented.

[0010] Figure 2Examples of multimodal annotation data suitable for medical images in accordance with one or more embodiments of the disclosed subject matter are presented.

[0011] Figure 3 Exemplary multimodal lesion segmentation models according to one or more embodiments of the disclosed subject matter are presented.

[0012] Figure 4 An exemplary training process for generating one or more multimodal AI models tailored to specific concepts in medical imaging in accordance with one or more embodiments of the disclosed subject matter is presented.

[0013] Figure 5 Another exemplary training process for generating one or more multimodal AI models tailored to specific concepts in medical imaging in accordance with one or more embodiments of the disclosed subject matter is presented.

[0014] Figure 6 A block diagram of an exemplary non-limiting computer system facilitating an iterative framework for learning multimodal mappings tailored for medical image inference tasks is shown in accordance with one or more embodiments of the disclosed subject matter.

[0015] Figure 7 A block diagram of an exemplary non-limiting computer-implemented method for generating one or more multimodal AI models customized for specific concepts in medical imaging in accordance with one or more embodiments of the disclosed subject matter is shown.

[0016] Figure 8 A block diagram of another exemplary non-limiting computer-implemented method for generating one or more multimodal AI models customized for specific concepts in medical imaging in accordance with one or more embodiments of the disclosed subject matter is shown.

[0017] Fig. 9 A block diagram is shown of an exemplary non-limiting operating environment in which one or more embodiments described herein may be facilitated.

[0018] Fig.10 A block diagram is shown of another exemplary non-limiting operating environment in which one or more embodiments described herein may be facilitated. DETAILED DESCRIPTION

[0019] The following specific embodiments are merely exemplary and are not intended to limit the application or use of the embodiments and / or the embodiments. In addition, it is not intended to be bound by any express or implied information set forth in the aforementioned "background technology" section or "content of the invention" section or "specific embodiments" section.

[0020] The present disclosure provides systems, computer-implemented methods, apparatuses, and / or computer program products that facilitate an iterative framework for learning multimodal mappings tailored for medical image inference tasks. As mentioned in the background section, there are three main problems with DL models in the field of medical imaging: 1) the need for large-scale annotated data, 2) asynchronous model development between different development stages (e.g., data curation and annotation, model building, and report generation), and 3) and lack of interpretability.

[0021] In this regard, most DL models for medical image inference tasks require a large number of annotated medical images for model training. Annotation data typically includes a label or tag associated with the medical image that defines or indicates the ground truth (GT) information for the inference task of the model to be trained. For example, when applied to lesion detection, the annotation data may include a label applied to the medical image indicating whether a lesion is present in the medical image, and / or label data in the form of a bounding box or another type of visual marker marking the location and size of the lesion present in the medical image.

[0022] Current data curation pipelines for obtaining annotated medical images typically involve the use of clinical expert annotators to review annotations and manually apply annotations to medical images, which is a time-consuming and expensive process. In addition, existing frameworks for model training and development are slightly out of touch with respect to the data curation phase and the model development phase. In this regard, expert annotators typically involve applying annotations to a medical image dataset customized for a specific target inference task (e.g., diagnosis, segmentation, disease staging, etc.). Once the medical image dataset has been annotated, it is sent to a data science team that uses the annotated data to build (e.g., train) a model. Once developed, the model is deployed to be applied to new medical images in a runtime environment, wherein typically only the task-specific output data that the model is trained to generate is provided to the end user, which typically lacks interpretability of the relevant features or factors on which the output results are based. For example, to facilitate a lesion detection example, the output data may indicate whether a lesion is present in an input image and / or include graphical marker data applied to a medical image indicating the location of the lesion in the input image. However, additional clinical review is required to generate a radiology report describing the lesion and other relevant information that provides a detailed clinical interpretation of the input images and the basis for the model output.

[0023] With these issues in mind, the disclosed technology provides a new framework for data annotation, model development, and deployment in the field of medical imaging, which is implemented through multimodal mapping. Specifically, the disclosed technology is based on a multimodal DL learning model that can learn the relationship between input image features and additional input data associated with different modalities or types of input images (such as text information describing relevant attributes associated with the input image) (e.g., multimodal mapping). Based on the learned relationship, the multimodal DL model can be configured to perform a target inference task that can relate the information included in the non-image input data associated with the input image (such as text describing the input image) to the corresponding image characteristics, and vice versa. An example of such a model developed outside the field of medical imaging has been used to identify the outlines of objects in real-world images (e.g., depicting real-world objects, places, things, animals, etc.) given an input image and object description text.

[0024] However, existing multimodal models of this type developed for real-world images and real-world objects cannot generate any useful output applied to medical images. This is because medical images are much more complex than real-world images, and the specific tasks or problems that the models are trained to perform on medical images require the models to have the intelligence of an expert radiologist adapted to understand features related to the task or problem, the type of medical image, the type of anatomical structure, the specialty, etc. This is also because the medical terminology used to describe the relevant attributes of medical images not only covers a complex ontology that is unique to the medical field, but also varies across different medical sub-fields, contexts, and medical providers, as well as clinical problems. Therefore, multimodal models developed based on learned embeddings between descriptions of real-world objects and images depicting the real-world objects cannot be reasonably domain-adapted to medical imaging tasks without facing many challenges. As discussed above, these challenges are exacerbated by the existing disjointed data curation and model development frameworks in the field of medical imaging.

[0025] In one or more embodiments, the disclosed subject matter provides a framework for establishing (e.g., training) a multimodal model that is configured to perform a specific medical image inference task. As used herein, a multimodal model refers to a model that can handle two or more different types of inputs. For example, two or more different forms of input may include image data (e.g., a medical image) and text data (e.g., text describing a medical image and / or a clinical interpretation of a medical image associated with a specific task at hand such as lesion detection, segmentation, classification, etc.). To this end, when applied to medical image inference, two or more different types of input data will generally always include at least image data input, i.e., an input medical image.

[0026] The input data of one or more additional types can change, but in most embodiments, will include text data of at least some form.For example, text data can include the text data describing the relevant characteristics of the input image for the task at hand, such as the detailed description of the characteristics of the focus depicted in the input image applied to the focus detection / classification example.For this reason, auxiliary text data input can include the text data input of different types corresponding to the different input types or modalities of multimodal input data respectively.For example, the text data manually received by the expert annotator who reviews medical images and provides text input (for example, in the form of natural language, etc.) can be considered as additional input data of a type of multimodal input data (for example, except inputting medical images themselves).Other examples of text input data corresponding to the different modalities or types of multimodal input data can include, but are not limited to, the text data extracted from the metadata associated with the medical image (for example, describing image type, image acquisition parameters, resolution, capture position / orientation, etc.), the text data extracted from the electronic medical record (EMR) data associated with the patient depicted in the medical image, the text data extracted from the associated laboratory report, the text data extracted from the associated clinical report, etc. In some embodiments, additional forms of multimodal input may include related medical images (e.g., past scans of the same patient, similar scans of similar lesions, etc.) Additional forms of multimodal input may also include sensory data or signal data captured via one or more medical imaging devices, such as signal data measuring various relevant physiological parameters of the patient depicted in the input medical image.

[0027] In various embodiments, the multimodal model can be trained to perform bidirectional inference. This means that the multimodal model can be trained to learn the relationship or mapping between the input features associated with different types of multimodal input data and the task at hand during training, such as the relationship between the image features of the lesions in the medical image and the text descriptors describing the lesions of the medical image when applied to, for example, lesion detection tasks. The multimodal model can also be trained to adopt these learned relationships to generate a target inference output, which identifies the corresponding features of another type or modality given the corresponding input features of one type or modality (such as medical image features associated with the corresponding text input and / or text features associated with the corresponding medical image data input). In various embodiments, these multimodal models can be used to perform intelligent annotations and intelligent reports, as well as other applications.

[0028] For example, in some embodiments, a multimodal model may include a model trained to identify specific anatomical image features depicted in a medical image based on text input describing the characteristics of the specific anatomical features. In some specific implementations of these embodiments, in conjunction with identifying specific anatomical features, the model may also be trained to identify and define the location of specific anatomical features in a medical image, the contours of the features, and additional detailed characteristics associated with the anatomical features given only limited text input. These types of multimodal models may be used to accelerate the annotation process by automatically generating annotation data (e.g., bounding box data, mask data, etc.) for training medical images, which defines the location, size, count, etc. of one or more relevant anatomical structures for the task at hand given the medical image and only a text description of the one or more anatomical structures. The annotation data may also include additional, automatically generated text data and / or other types of multimodal data features that map to text descriptions (and / or portions thereof) and / or corresponding anatomical structures for the task at hand.

[0029] In another example embodiment, the multimodal model may include a model configured to perform bidirectional queries, such as finding medical images that satisfy a text query (e.g., "get medical images containing lesions of type "xyz""). To this end, unlike traditional query techniques, the trained model does not use labels or annotations associated with the medical image, but instead, as a result of learning a mapping during training, can inherently understand the relationship between image features extracted from the medical image itself and corresponding text descriptors associated with the extracted features. In another example, the multimodal model may include a model configured to generate text information describing relevant image features depicted in an input medical image based on a learned mapping between the relevant image features and text features associated with the relevant features. According to this example, the multimodal model can be used to perform intelligent reporting by generating a detailed text description that provides a clinical interpretation of the medical image and related related auxiliary data (e.g., EMR data, laboratory data, similar scans, etc.).

[0030] In order to generate a robust multimodal model with such capabilities in medical imaging, the learned mapping between image features of the corresponding medical images and additional multimodal data features (e.g., text data and / or other forms of auxiliary data) takes into account highly specific clinical concepts, features, and ontologies that are appropriate for the type of medical images and the specific context of clinical interpretation (i.e., a specific medical image inference task). To facilitate this purpose, the disclosed framework leverages the clinical expertise of expert annotators and allows expert annotators to define and generate additional multimodal data features relevant to the target inference task and the medical image dataset based on clinical concepts, features, and ontologies that they believe are relevant and useful for the target inference task, and / or provide robust detailed clinical interpretations of the medical images in the dataset.

[0031] In addition, the disclosed framework combines the data annotation and model development phases in an iterative framework that leverages the expertise of clinical annotators to refine and update relevant additional multimodal data features that control the learned mapping between medical image features and additional multimodal data features when the model is built (e.g., based on the performance of the model during training). For example, in some embodiments, the annotation pipeline may involve having an expert annotator (e.g., a radiologist or another person who is able to read and interpret medical images in a clinical context with appropriate expertise for the target inference task at hand) perform a "strong" annotation of a subset of the medical images included in the training dataset. In this context, "strong" annotation refers to manually reviewing medical images and providing not only conventional GT annotation data for a target inference task (e.g., segmentation, diagnosis, classification, etc.), such as (when applied to segmentation and / or detection tasks) labeled data (e.g., bounding box data, mask data, etc.) that labels anatomical features of interest, text data that defines the classification of anatomical features of interest (e.g., when applied to classification tasks), but also additional annotation data in the form of text (or speech-to-text, gesture-to-text, etc.) that describes relevant attributes of each medical image according to specific clinical concepts, features, and ontologies that are appropriate for the type of medical image and the specific context of clinical interpretation (e.g., one or more target medical image inference tasks). To this end, "strong" annotation also refers to generating multimodal annotation data for medical images. Importantly, in conjunction with applying multimodal annotation data to a subset of medical images, expert annotators can constrain additional annotation data (e.g., in addition to GT data) according to defined criteria (e.g., clinical concepts, clinical ontologies, clinical terms, etc.) determined by the expert to be relevant to the medical image and the target inference task (or, in some specific implementations, multiple target inference tasks).

[0032] Once an initial subset of medical images has been strongly annotated using a combination of conventional GT annotation data and additional descriptive text data (and / or other forms of non-image annotation data), the annotated medical images are used to learn bidirectional relationships between relevant image features in the medical images and additional input data features for the task at hand (e.g., text terms and phrases, etc.), which in turn are used to train one or more multimodal models to perform tasks associated with the medical images and / or additional bidirectional inference tasks. For example, in various embodiments, a learned mapping can be used to generate a model that is configured to generate a mask (bounding box, etc.) that defines the location and contour of a lesion in the input medical image given a medical image as input and a text description of the lesion. According to this example (and additional examples described below), a bidirectional model trained on strongly annotated samples can then be used to more effectively annotate the remaining medical images included in the training dataset using only limited text input through mask annotation data and / or additional relevant multimodal input data.

[0033] In addition, the iterative framework allows expert annotators to review preliminary results of bidirectional model versions trained on strongly annotated data samples. For example, to facilitate lesion / mask examples, expert annotators can review masks generated by the model on additional medical images included in the training data set (e.g., or a portion thereof) based on the provided descriptive text input. Based on errors in the model output observed on new images using text input, the annotation pipeline allows expert annotators to refine or update clinical standards for controlling additional strongly annotated data (e.g., text input), and modify the multimodal data applied to a subset of medical images according to the updated standards. In other words, the expert annotator can add additional descriptors, change the terms used, change the phrases used, etc. Then use a subset of medical images with updated multimodal annotation data to refine or update the learned mapping and model, and the process can be iteratively repeated until convergence is achieved. In this regard, through the disclosed technology, expert annotators can control annotation data before deploying the model in a runtime environment, and the annotation data controls the performance of the model.

[0034] In this regard, the proposed disclosed subject matter provides tools that handle both intelligent annotation and intelligent reporting for AI-based medical image analysis. These tools can provide several benefits, including (but not limited to): 1) utilizing less expert time to train data annotations, 2) faster and more robust AI model building for complex clinical tasks, 3) immersing expert-driven concepts into DL models, and 4) intelligent reporting in expert-defined clinical language. As a result, the disclosed tools enable faster time-to-market for AI products, increase the chances of adoption due to explainability, and ultimately improve patient outcomes.

[0035] The term "image data" is used herein to refer to one or more images, image data structures (e.g., image files in any format), and / or video data consisting of picture elements (e.g., pixels) that can be rendered as visual data objects via a display. For example, the term image data can refer to one or more digital images, native images, synthesized images, etc. The term image data can also refer to a portion of an image, labeled data (e.g., visual markers, bounding boxes, annotations, lines, masks, segmentation masks, etc.), an enhanced image, a graphical image, a sinusoidal image, or another type of image construction that includes visual elements that can be rendered via a display.

[0036] The term "medical image data" is used to refer to image data depicting one or more anatomical regions of a patient. References herein to medical images or medical image data may include any type of medical image associated with various types of medical image acquisition / capture modalities. For example, medical images may include (but are not limited to): radiation therapy (RT) images, X-ray (XR) images, digital radiography (DX) X-ray images, X-ray angiography (XA) images, panoramic X-ray (PX) images, computed tomography (CT) images, mammography (MG) images (including tomosynthesis equipment), magnetic resonance imaging (MRI) images, ultrasound (US) images, color flow Doppler (CD) images, positron emission tomography (PET) images, single photon emission computed tomography (SPECT) images, nuclear medicine (NM) images, etc. Medical images may also include synthetic versions of native medical images, such as enhanced, modified or improved versions of native medical images, enhanced versions of native medical images generated using one or more image processing techniques, etc. In some embodiments, the term "image data" may include raw measurement data (or simulated measurement data) used to generate a medical image (eg, raw measurement data captured via a medical image acquisition process).

[0037] The term "non-image data" is used herein to refer to any type of data that does not include image data. In this regard, the term "non-image data" may include text data (e.g., a text data file in any text data type format), audio data (e.g., an audio file in any type of audio file format), signal data generated via one or more machines or sensors, or another type of data format other than image data.

[0038] Unless the context warrants a specific distinction between the terms, the terms "algorithm" and "model" are used interchangeably herein. Unless the context warrants a specific distinction between the terms, the terms "artificial intelligence (AI) model" and "machine learning (ML) model" are used interchangeably herein. References to AI models or ML models herein may include any type of AI model or ML model, including (but not limited to): deep learning (DL) models, neural network models, deep neural network models (DNN), convolutional neural network models (CNN), generative adversarial neural network models (GAN), transformer models, etc. AI models or ML models may include supervised learning models, unsupervised learning models, semi-supervised learning models, combinations thereof, and models that employ other types of ML learning techniques. AI models or ML models may include a single model or a group of two or more models (e.g., a set of models, a chain model, etc.).

[0039] The term "image inference model" is used herein to refer to an AI / ML model suitable for performing image processing or analysis tasks on image data. The image processing or analysis tasks may vary. In various embodiments, the image processing or analysis tasks may include (but are not limited to): segmentation tasks, image reconstruction tasks, object recognition tasks, motion detection tasks, video tracking tasks, optical flow tasks, etc. The image inference model described herein may include a two-dimensional (2D) image processing model and a three-dimensional (3D) image processing model. The image processing model may adopt various types of AI / ML models (e.g., deep learning models, neural network models, deep neural network models, DNN, CNN, GAN, etc.). Unless the context warrants a specific distinction between the terms, the terms "image inference model", "image processing model", "image analysis model", etc. are used interchangeably herein.

[0040] The term "image-based inference output" is used herein to refer to a determination or prediction that an image processing model is configured to produce. For example, an image-based inference output may include a segmentation mask, a reconstructed image, an adapted image, an annotated image, a classification, a value, etc. The image-based inference output will vary based on the type of model and the specific task that the model is configured to perform. The image-based inference output may include a data object that can be presented (e.g., a visual data object), stored, used as an input for another processing task, etc. The output may be in different formats, such as: a digital imaging and communications in medicine (DICOM) structured report (SR), a DICOM secondary capture, a DICOM parameter map, an image, a text, and / or a JavaScript object representation (JSON). The terms "image-based inference output", "inference output", "inference result", "inference", "output", "result", "prediction", etc. are used interchangeably herein unless the context determines that a special distinction is made between these terms.

[0041] As used herein, a "medical imaging inference model" refers to an image inference model that is customized to perform image processing / analysis tasks on medical image data. For example, medical imaging processing / analysis tasks may include (but are not limited to): disease / condition classification, disease region segmentation, organ segmentation, disease quantification, disease / condition staging, risk prediction, temporal analysis, anomaly detection, anatomical feature characterization, medical image reconstruction, etc. Unless the context warrants a specific distinction between the terms, the terms "medical image inference model", "medical image processing model", "medical image analysis model", etc. may be used interchangeably herein.

[0042] Embodiments of the systems and devices described herein may include one or more machine-executable (i.e., computer-executable) components or instructions embodied in one or more machines (e.g., embodied in one or more computer-readable storage media associated with one or more machines). Such components, when executed by one or more machines (e.g., processors, computers, computing devices, virtual machines, etc.), may cause one or more machines to perform the described operations. These computer / machine-executable components or instructions (and other components or instructions described herein) may be stored in a memory associated with one or more machines. The memory may also be operably connected to at least one processor so that the components may be executed by the at least one processor to perform the described operations. In some embodiments, the memory may include a non-transitory machine-readable medium that includes executable components or instructions that facilitate the execution of the operations described for the corresponding executable components when executed by the processor. Examples of the described and memory and processors and other suitable computer or computing-based elements may refer to Fig. 9 find (e.g., processing unit 904 and system memory 906, respectively), and can be combined to achieve the combination Figure 1 Or one or more of the systems or components shown and described in other figures disclosed herein.

[0043] One or more embodiments are now described with reference to the accompanying drawings, wherein the same reference numerals are used to represent the same elements throughout. In the following description, for the purpose of illustration, many specific details are set forth in order to provide a more thorough understanding of one or more embodiments. However, it is apparent that in various cases, one or more embodiments may be practiced without these specific details.

[0044] Turning now to the accompanying drawings, Figure 1An iterative process 100 for learning a multimodal mapping customized for a medical image inference task according to one or more embodiments of the disclosed subject matter is presented. The process 100 leverages the expertise of one or more expert annotators to combine the training data curation phase and the model building (e.g., model training) phase into the same framework. In various embodiments, the disclosed technology can be performed in conjunction with the use of a single expert annotator so that the learned embeddings used to generate one or more bidirectional ML models are customized for the specific natural language and clinical expertise of a specific expert annotator. Through these embodiments, the process 100 (and the additional processes and systems described herein) can be used to generate a "personal" AI model that is suitable for processing input data and generating corresponding output data that reflects the natural language, dialect, and clinical expertise of a specific medical professional. Additionally or alternatively, the disclosed technology can be performed in conjunction with the use of multiple different expert annotators that combine their expertise and clinical knowledge. In this regard, elements of the process 100 that involve obtaining manual input from one or more annotators are indicated by (cropped) human icons. It is assumed that references to an "annotator," "expert annotator," "annotation entity," etc. are understood to correspond to a single person, but it is understood that an annotator may include two or more different persons.

[0045] The process 100 is described in conjunction with generating multimodal annotation data for a medical image dataset 102. In various embodiments, the medical image dataset 102 corresponds to a training dataset comprising a plurality of medical images that need to be annotated with GT information for a specific target medical image inference task. The types of medical images included in the training dataset and the specific medical image inference task may vary. However, when applied to a medical image inference model, the medical images that make up the training dataset typically include different variants of the same type of medical images (e.g., the same modality, the same or similar anatomical region, the same or similar lesions, etc.), where the different variants are typically based on different individuals or patients from which the medical images were captured.

[0046] By way of example, process 100 is described in the context of annotating an ultrasound image of a chest in conjunction with generating a lesion segmentation model that is configured to detect and segment lesions depicted in the image. According to this example, all medical images included in the medical image dataset 102 will include different ultrasound images of the chest, such as different images captured from different individuals providing different examples of lesions. However, it should be understood that this particular use case is merely exemplary, and process 100 and the techniques described herein can be applied to any type of medical image and any type of medical image inference task. In addition, the target medical image inference task can include or correspond to the "main" task of the AI ​​model to be trained for the annotated training images. As described in more detail below, the annotated training images can also be used to train the AI ​​model and / or one or more independent AI models to perform additional bidirectional inference tasks based on the learned embedding between the image features extracted from the medical image and the additional features included in the multimodal annotation data.

[0047] According to the process 100, at 104, an expert annotator may define multimodal annotation criteria that govern multimodal annotation data to be applied to at least some of the medical images included in the medical image dataset 102. For example, according to the process 100, the annotator may perform strong annotation on a small subset 102a of the medical images included in the medical image dataset 102. In some embodiments, the strongly annotated data samples may then be used to generate one or more bidirectional models (e.g., one or more models 110) that are configured to automatically infer annotation data and apply the annotation data to additional medical images 102b included in the medical image dataset 102 in a more efficient and automated manner, referred to herein as "smart annotation," as explained in more detail below.

[0048] The multimodal annotation criteria defined at 104 may control and / or guide the type and content of multimodal annotation data to be applied to the corresponding medical images included in the subset 102a. The term "multimodal annotation data" is used herein to refer to two or more different types of annotation data modalities. In various embodiments, the type of multimodal annotation data may include a first type corresponding to conventional GT annotation data for a particular target medical image inference task. For example, when applied to lesion detection, the first type of multimodal data may include labeling data applied to the corresponding medical images in the subset 102a that marks any lesions observed in the medical images, such as a graphical bounding box drawn around the lesion that reflects the size and geometry or outline of the lesion, such as Figure 2 shown.

[0049] In this regard, a brief reference Figure 2 , Figure 2 Examples of multimodal annotation data suitable for medical images in accordance with one or more embodiments of the disclosed subject matter are presented. Figure 2 Two different exemplary ultrasound images of a chest (e.g., image 202a and image 202b), each including a lesion, are presented. In this example, the lesions are marked on the respective images using graphical bounding boxes 204a and 204b (it should be noted that the term "bounding box" herein can encompass any geometric shape (i.e., a circle) and is not limited to a rectangular box). These bounding boxes 204a and 204b correspond to conventional GT image annotation data used to annotate medical images for ML tasks involving automatically detecting target anatomical features of interest (in this case, lesions), performing segmentation (e.g., generating a segmentation mask that defines the location, size, and / or contour of the anatomical feature of interest), and various other medical image inference tasks. Other forms of visual or graphical GT annotation data that can be applied to medical images for various types of inference tasks include calipers or markers (e.g., symbols, lines, etc.) that mark anatomical features of interest and / or the relative positions of two or more anatomical features of interest. In some implementations, the target anatomical features of interest may not be "anatomical" features, but rather artifacts to be distinguished from true anatomical features and / or removed by the model (e.g., when applied to an artifact removal model), and / or regions / areas of an image associated with quality issues (e.g., streaks, poor resolution, etc.) to be enhanced by the model (e.g., when applied to a medical image quality enhancement model). In this regard, any type of visual markup annotation data applied to medical images is considered "image annotation data" for the purposes of the disclosed subject matter.

[0050] Reference again Figure 1 as well as Figure 2 , the first type of annotation data is not limited to graphical label data (i.e., image annotation data) such as described above. For example, in some specific implementations, depending on the target medical inference task, the GT annotation information may include text data (i.e., non-image annotation data) that only indicates whether a specific medical condition is present in the image. In this regard, the conventional GT annotation data corresponding to the first type of annotation data may vary based on the target medical image inference task.

[0051] In some embodiments, the first type of annotation data may be predefined for a given medical image dataset 102 and a target inference task. In other embodiments, an expert annotator may define the target inference task and / or the criteria for the first type of annotation data at 104. Generally speaking, because the first type of annotation data corresponds to GT information, which directly corresponds to specific output information that the ML model will be trained to predict, the first type of annotation data may be considered as structured data that follows relatively structured or strict criteria.

[0052] Additionally or alternatively, the multimodal annotation data to be applied to subset 102a may include one or more additional types of annotation data. In various embodiments, the one or more additional types of annotation data may include a second type of annotation data corresponding to manually applied text data (i.e., non-image annotation data) that describes relevant clinical and / or non-clinical features associated with the medical image and the target inference task. For example, the text data may include unstructured text input provided by an annotator that describes an expert clinical interpretation of the medical image included in subset 102a using a natural language description.

[0053] In this regard, at 104, the annotator can use unstructured text input to define the relevant clinical concepts or standards to be described by the annotator, and the unstructured text input can be received via any suitable input mechanism, such as via keyboard, touch screen, voice to text, gesture to text, etc. For this reason, in addition to GT annotation data, process 100 also allows experts to determine and define the type of text information that is or may be relevant to the main inference task. For example, the standard defined by the expert that can control such text input can include clinical and / or non-clinical information of defined categories. Some example categories related to the lesion detection task can include but are not limited to the description of the region of interest, lesion type, lesion characteristics, anatomical structure, and image artifacts. For this reason, as used herein, the term "clinical standard" refers to the standard directly related to the anatomy and physiology of a living body (e.g., human body, animal body or another organism), including all its parts, processes, conditions affecting it, and the procedures performed on it. The term "non-clinical" standard used herein refers to any standard except the clinical standard. Examples of non-clinical criteria may include, but are not limited to, information describing or about the medical image acquisition process (e.g., acquisition parameters), individual demographics (e.g., age, gender, location, socioeconomic status), information describing or about image quality (e.g., colorization, artifacts, resolution, blur, etc.), information describing or about individual behavior, etc.

[0054] It should be understood that the specific clinical and non-clinical concepts that are or may be relevant to the primary inference task may vary. In addition, the specific clinical and non-clinical concepts used to define and control the natural language text description annotation data are not limited to those concepts related to the primary inference task. In this regard, the second type of multimodal annotation data may include essentially unlimited descriptive text information for the subset 102a of medical images, the descriptive text information providing a comprehensive description of clinical and non-clinical image features depicted in the medical images, which descriptive text information can be used for other tasks in addition to or as an alternative to the primary task.

[0055] In some embodiments, one or more additional types of annotation data may include other forms of text input data corresponding to different modalities or types of multimodal input data. Some examples may include (but are not limited to) text data extracted from metadata associated with the medical image (e.g., describing image type, image acquisition parameters, resolution, capture position / orientation, etc.), text data extracted from electronic medical record (EMR) data associated with the patient depicted in the medical image, text data extracted from associated laboratory reports, text data extracted from associated clinical reports, etc. For example, EMR text data may include patient information about the corresponding patient represented in the medical image included in one or more EMR data files. In some embodiments, additional forms of multimodal input may include related medical images (e.g., past scans for the same patient, similar scans for similar lesions, etc.). Additional forms of multimodal input may also include sensory data or signal data captured via one or more medical imaging devices, such as signal data measuring various relevant physiological parameters of the patient depicted in the input medical image. To this end, the amount, type, and content of the multimodal annotation data applied to subset 102a may vary and substantially cover any type of information that may be included in a comprehensive radiology report for a medical image.

[0056] Continuing with process 100, at 106, the expert annotator may generate multimodal annotation data for the corresponding medical images included in the subset 102a according to the multimodal annotation criteria defined at 104. To facilitate this, the annotator may employ a suitable medical imaging application that allows the annotator to view the medical images included in the subset displayed via a graphical user interface (GUI), apply label annotation data to the medical images (e.g., using existing medical image label annotation software), provide unstructured text input describing relevant image features (e.g., via free-form data entry, via speech that is subsequently converted to text, or via another suitable input mechanism), and optionally generate and / or provide (e.g., import, upload, provide links thereto, etc.) additional input in the form of relevant medical records, laboratory reports, metadata, relevant medical images, etc.

[0057] For example, when applied to a lesion segmentation task as the primary inference task, assume that the multimodal annotation standard includes a first standard defining a first type of annotation data to be applied to a medical image, the first type of annotation data corresponding to providing GT image annotation data to the medical image with bounding boxes marking any observed lesions. Further assume that the multimodal annotation standard includes a second standard describing a second type of annotation data to be applied to the medical image, the second type of annotation data comprising unstructured text data provided in the annotator's natural language, and the second standard comprising descriptive information following the following two categories: 1) a description of the type of lesion and 2) a description of the characteristics of the lesion. Figure 2 An example of such multimodal annotation data applied to exemplary images 202a and 202b according to a first criterion and a second criterion is provided. For example, as described above, circular bounding boxes 204a and 204b correspond to a first type of annotation data (i.e., image annotation data), and text descriptions 206a and 206b correspond to a second type of annotation data, respectively. In this example, the text description applied by the expert annotator describes the lesion observed in image 202a as "an inhomogeneous mixed echogenic fibroma with shading," and the text description applied to the lesion observed in image 202b describes the lesion as "an inhomogeneous mixed echogenic fibroma with shading." It should be understood that references to "applying" text data to medical images in accordance with the disclosed subject matter include any mechanism for associating text data with corresponding images. For example, text data can be "applied" as a separate text file, metadata, etc.

[0058] In some embodiments, in conjunction with providing text input according to defined standards or categories of text input types defined by the annotator category, the annotator may indicate the specific category being described. For example, an expert annotator may indicate "the depicted region of interest includes...", "the type of lesion is...", "the characteristics of the lesion include...", etc. These category hints can add context to the medical terms associated with them, which can facilitate learning contextual embeddings of the medical terms used by the annotators relative to the specific contexts in which they are used. Contextual embeddings assign representations or meanings to each word based on the context of each word, thereby capturing the use of words in various contexts. To this end, the specific context in which the annotator uses the term reflects the specific type of medical image being evaluated and the content of the image (i.e., image features), as well as the specific target medical image inference task. Therefore, the contextual embeddings of the term can facilitate learning the mapping between the term to the corresponding medical image and / or specific image features (e.g., anatomical features) included in the medical image.

[0059] although Figure 2 The exemplary text descriptions shown are relatively concise, but it should be understood that the amount of descriptive annotation text data applied by an expert annotator may vary, and Figure 2 The text description provided in is merely exemplary. In addition, although generating multimodal annotation data at 106 is a manual task that requires expert annotation to carefully review the strong annotation data and apply the strong annotation data to individual medical images, the process 100 limits the strong annotation task to only a subset 102a of the medical images included in the medical image dataset 102, and enables the use of an intelligent annotation model trained on the strong annotation data applied to the subset 102a to annotate the remaining majority of medical images (e.g., additional medical images 102b), as described in more detail below.

[0060] Continuing with process 100, a subset 102a of medical images having strong multimodal annotation data applied thereto is represented in process 100 as annotated subset 102a'. At 108, the annotated subset 102a' of medical images is used to build (e.g., train) one or more AI models 110. Specifically, at 108, the annotated subset 102a of medical images can be provided (e.g., sent, transmitted, accessed, etc.) to a machine learning system (i.e., computing system 600 or similar system) that is configured to employ one or more machine learning processes to train one or more multimodal models (e.g., model 110) to perform a primary inference task and / or one or more additional bidirectional inference tasks. For example, in various embodiments, one or more machine learning processes may involve employing / training one or more transformer models and / or one or more task-specific decoder models to learn a bidirectional mapping using the annotated subset 102a', and training a task-specific decoder 414 in conjunction with training the one or more transformer models and / or one or more task-specific decoder models to perform a specific bidirectional inference task on a medical image using the learned mapping. For example, the one or more transformer / task-specific decoder models may include or correspond to one or more contrastive language image pre-training (CLIP) models and / or CLIP-SEG (CLIP segmentation model) trained to perform segmentation of one or more specific anatomical features of interest associated with a specific type of input medical image.

[0061] In this regard, at 108, one or more multimodal models 110 can be trained to learn the relationship or mapping between input features associated with different types of multimodal input data and image features extracted from input medical images, such as the relationship between image features of lesions in exemplary lesion medical images and text descriptors describing lesions of medical images when applied to, for example, lesion segmentation tasks. To this end, the relationship or mapping can be customized for a specific target inference task or a primary inference task because the multimodal data is customized for the primary inference task. One or more multimodal models 110 can also be trained to employ these learned relationships to generate a target inference output that identifies the corresponding features of another type or modality given the corresponding input features of one type or modality (such as medical image features associated with the corresponding text input and / or text features associated with the corresponding medical image data input). In various embodiments, these multimodal models can be used to perform intelligent annotation and intelligent reporting, as well as other applications.

[0062] For example, in some embodiments, one or more multimodal models 110 may include a model trained to identify specific anatomical image features depicted in a medical image based on text input describing characteristics of the specific anatomical features. The model can be customized for the specific type of medical image included in the subset 102a and the related features described, which can be customized for a specific medical image inference task (e.g., lesion segmentation). These techniques can be used to generate various different versions of the model customized for different types of medical images (e.g., modality, one or more anatomical regions depicted, lesions depicted, etc.) and the related clinical and non-clinical features of the described images.

[0063] In some specific implementations of these embodiments, these types of multimodal models can be used to perform bidirectional queries, such as finding medical images that satisfy a text query (e.g., "get medical images containing lesions of type "xyz""). To this end, unlike traditional query techniques, the trained model does not use labels or annotations associated with the medical image, but instead, as a result of learning a mapping during training, can inherit the relationship between image features extracted from the medical image itself and the corresponding text descriptors associated with the extracted features. In another example, these types of multimodal models may include a model configured to generate text information describing relevant image features depicted in an input medical image based on a learned mapping between the relevant image features and text features associated with the relevant features. According to this example, the multimodal model can be used to perform intelligent reporting by generating a detailed text description that provides a clinical interpretation of the medical image and related related auxiliary data (e.g., EMR data, laboratory data, similar scans, etc.). For example, the model may be configured to extract one or more image feature vectors from an input medical image and generate text output data describing the content of the medical image based on a learned mapping between the image feature vectors and text data associated with the corresponding image feature vectors.

[0064] Still in other specific implementations, these types of multimodal models can be used to perform intelligent annotation. For example, in some embodiments, in combination with identifying specific anatomical features based on additional non-image data (e.g., textual data), the model can also be trained to identify and define the location of specific anatomical features in medical images, the contours of the features, and additional detailed characteristics associated with the anatomical features given only limited textual input. These types of multimodal models can be used to accelerate the annotation process by automatically generating annotation data (e.g., bounding box data, mask data, etc.) for training medical images, which defines the location, size, count, etc. of one or more relevant anatomical structures for the task at hand given the medical image and only a textual description of one or more anatomical structures. The annotation data can also include additional, automatically generated textual data and / or other types of multimodal data features that map to textual descriptions (and / or portions thereof) and / or corresponding anatomical structures for the task at hand.

[0065] For example, when applied to an exemplary lesion segmentation task as a primary inference task, at 108, the machine learning system can train a lesion segmentation model to detect and segment in an input medical image using an annotated subset 102a' of the training image (e.g., via generating a bounding box or mask around the lesion). Typically, such a model is trained in a supervised manner by training a deep learning model to identify the lesion and its contour as depicted in the training image using the GT tag annotation data associated therewith that defines the boundary or contour of the lesion. Once trained, the model should be configured to receive a new input medical image corresponding to the training image (e.g., the same modality and anatomical region depicted) as input, and generate segmentation data (e.g., bounding box, mask, etc.) defining the contour of the lesion depicted in the new medical image.

[0066] However, in addition to training the model to detect and segment the lesion based on such GT labeled annotation data associated with the lesion, the disclosed technology also trains the model to learn the mapping between non-image input features (i.e., text terms and phrases and / or other forms of multimodal annotation data) and corresponding image features. For example, when applied to the lesion segmentation model example, the mapping may include a mapping between image features of lesions and their contours within the training images, as well as textual descriptors of the lesions, such as Figure 2 206a and 206b. These mappings can also be used to train one or more AI models 110 to perform various bidirectional inference tasks.

[0067] For example, in some embodiments, when applied to the lesion segmentation example, these mappings can be used to train a lesion segmentation model to use text descriptions of lesions observed in a new input image as additional input (e.g., in addition to the new input image) to facilitate detecting and segmenting any lesions depicted in the new image. To this end, the additional text input describing the lesions can improve the accuracy of the segmentation model. For example, the lesion segmentation model can be configured to receive an input medical image and a text description of a lesion observed in the input image, and generate a bounding box or mask around the lesion, such as Figure 3 shown.

[0068] Brief reference Figure 3 , Figure 3 An exemplary multimodal lesion segmentation model 306 corresponding to the lesion detection model described above is shown. For example, the multimodal lesion segmentation model 306 corresponds to a trained version of a lesion segmentation model trained on training data samples corresponding to Figure 2 A medical image with paired GT label data and additional text description of the lesion as depicted in . Figure 2 As shown, the multimodal lesion segmentation model 306 is configured to receive as input a new input medical image 302 corresponding to a training image (e.g., an ultrasound image of a chest) and a text description 304 of a lesion observed in the input image. The output of the multimodal lesion segmentation model 306 includes a bounding box 304 overlaid on the medical image 302, which encloses the boundaries of the lesion detected in the medical image 302.

[0069] refer to Figures 1 to 3 According to some embodiments of the disclosed technology, additional text input describing the lesions can be used to train a preliminary version of the multimodal lesion segmentation model 306, which can detect and segment lesions using only a (small) annotated subset 102a of the training images. In this regard, the additional text input can enable a preliminary version of the lesion detection model to detect and segment lesions using significantly less training time than training the segmentation model for the entire training data set. In some specific implementations of these embodiments, the preliminary version of the multimodal lesion segmentation model 306 can be used by an expert annotator to generate GT segmentation data (e.g., bounding boxes, masks, etc.) for additional medical images 102b of the medical image dataset 102 using text prompts as additional input, a process referred to herein as "smart annotation".

[0070] In this regard, continuing with process 100, at 112, the expert annotator may employ one or more of the AI ​​models 110 to generate inferences about the additional medical image based on the limited input. For example, according to an embodiment in which the one or more AI models 110 correspond to an intelligent annotation model configured to conveniently generate GT annotation data for a medical image based on a textual prompt (and / or other types of additional multimodal input data), at 112, the expert annotator may employ the intelligent annotation model to generate GT annotation data for the additional medical image 102b (or portion thereof) based on the textual prompt provided by the annotator. For example, assume that the one or more models 110 include a multimodal lesion segmentation model 306 that has been trained on the annotated subset 102a, where the corresponding data samples included in the annotated subset 102a correspond to Figure 2 202a and 202b with paired GT label data and additional text descriptions of lesions. According to this example, at 112, the expert annotator can review / view the additional medical image (or portion thereof), provide only the text description of the observed lesions, and employ the multimodal lesion segmentation model 306 to generate GT bounding box annotation data for the additional medical image, such as Figure 3 As shown in Figure 1. To this end, the expert annotator does not need to perform a "strong" annotation process on most of the training images, and thus can use the textual prompts as input to more efficiently generate GT labeled data for most of the training images. According to this example, the rest of the training images (or parts thereof) with model-generated GT labeled data are Figure 1 The additional medical image 102b' is represented as annotated.

[0071] It should be understood that the specific implementation of the lesion segmentation model is only exemplary. In this regard, process 100 can be used to generate a preliminary version of other types of models 110, which are configured to generate other types of GT annotation data for other types of medical images based on limited input (e.g., text prompts and / or other types of multimodal inputs). To this end, model 110 may include other types of intelligent annotation models configured to generate other types of GT annotation data and / or for other types of medical images and inference tasks. To this end, the annotated additional image 102b' is not limited to the lesion image with the GT bounding box data applied thereto, and may correspond to any type of intermediate image with the GT annotation data applied thereto via the intelligent annotation model generated according to the disclosed technology. In addition, as described in more detail below, one or more models 110 can perform other bidirectional inference tasks except intelligent annotation.

[0072] Continuing with process 100, at 114, an expert annotator may review the inferred output generated by the model on the additional medical image. For example, when applied to smart annotation, the expert annotator may review the annotated additional medical image 102b' having GT annotation data applied thereto by the smart annotation model (one or more of the models 110). For example, the expert annotator may review the results of the smart annotation model, such as the automatically generated GT lesion bounding box according to the multimodal lesion segmentation model example. To this end, the expert annotator may evaluate the performance of the smart annotation model and determine at 116 whether the model output data is acceptable. For example, continuing with the multimodal lesion segmentation model example, the expert annotator may evaluate whether the bounding box generated by the model is sufficiently accurate as generated based on the annotator's textual prompt. It should be understood that this may involve using the model to generate GT label data for a portion of the additional medical image 102b, and thereafter evaluating the performance of the model for that portion of the additional medical image at time 102b. In conjunction with evaluating the performance of the model at 116 , the expert annotator may determine whether the performance of the model is unacceptable or acceptable based on the results generated for the portion of the additional medical image 102 b or the entirety of the additional medical image 102 b .

[0073] At 116, if the expert annotator determines that the performance of the model is acceptable, then in some embodiments, the version of the model 110 trained for the initial strong annotation subset 102a can be considered ready for deployment and thus finalized. For these embodiments, process 100 can continue to 120, where the final version of one or more models 110 can be used for intelligent annotation, intelligent reporting, and other applications. In this regard, the term "intelligent annotation" is used herein to refer to the process of generating annotation data for one or more medical images using a trained version of one or more ML or AI models (e.g., one or more models 110), which is configured to infer and generate at least some of the annotation data. Similarly, the term "intelligent reporting" is used herein to refer to the process of generating a medical or clinical report using a trained version of one or more ML or AI models (e.g., one or more models 110), which is configured to infer and generate at least some of the information included in the medical or clinical report.

[0074] However, at 116, if the expert annotator determines that the performance of the model is unacceptable, in some embodiments, the expert annotator can facilitate updating / retraining the model to achieve better performance by adjusting or updating the multimodal annotation criteria at 118 and repeating process 100 following the dashed arrow flow. In this regard, the expert annotator can update the multimodal annotation criteria so that the type and / or content of the multimodal annotation data applied to subset 102a (or new subset) is different from the type and / or content initially applied. For example, the annotator can adjust the clinical concepts and / or categories that control additional multimodal data (e.g., in addition to the GT annotation data). For example, continuing with the lesion segmentation example, the expert annotator can determine that the performance of the model can be improved using additional descriptive text describing the relative position of the lesions observed in the image or a description of the size of the lesions, etc. Therefore, at 118, the expert annotator can update or adjust the multimodal annotation criteria defined at 104 accordingly. At 106, the expert annotator may further update the text annotation data applied to subset 102a, and generate updated multimodal annotation data for subset 102a (and / or a new subset) according to the updated criteria. At 108, the (updated) annotation subset 102a' may then be used to retrain or update one or more AI models. To this end, the machine learning system may employ one or more machine learning processes to learn updated mappings between updated non-image (e.g., text data) annotation data and image features, and generate an updated version of the model 110 based on the updated mapping. The process 100 may also continue according to steps 112 and 114 as described above using the updated version of the model. The process 100 may also be iteratively executed according to the dashed arrow flow until the performance of the model is acceptable at 116. In this regard, through the disclosed technology, the expert annotator may control and provide annotation data that controls the performance of the model before deploying the model in the runtime environment.

[0075] Figure 4 An exemplary training process 400 for generating one or more multimodal AI models tailored to specific concepts in medical imaging is presented in accordance with one or more embodiments of the disclosed subject matter. Figures 1 to 4In various embodiments, the training process 400 can be used to establish one or more multimodal models in the multimodal model 110 described in step 108 of the reference process 100. According to the process 400, the multimodal model includes three components or sub-models, including an image transformer 404, a text transformer 406, and a task-specific decoder 414. The image transformer 404, the text transformer 406, and the task-specific decoder 414 can correspond to deep learning neural network models, respectively. In this example, the multimodal model is trained to process multimodal input data 402 consisting of medical images paired with textual prompt data. The textual prompt data can correspond to text provided by an annotator, which describes relevant features of the medical image, such as relevant features of the medical image for a target task (such as lesion segmentation). In some embodiments, the medical image can include labeled annotation data (e.g., bounding boxes, masks, etc.) applied thereto, which labels the relevant anatomical features of interest described in the textual prompt (e.g., a description of a labeled lesion).

[0076] In various embodiments, the image transformer 404 may transform each input medical image into an image vector representation 408 of the image. Meanwhile, the text transformer 406 may transform the corresponding text prompt into a text representation vector 410 of the text prompt using natural language processing (NLP) processing techniques. The image transformer 404 and the text transformer 406 may be further configured to project the corresponding vector representations onto a common dimensional space 412. The task-specific decoder 414 may also use one or more unsupervised machine learning processes to learn mappings between text prompt features as represented in the text representation vector 410 and image features as represented in the image representation vector 408. In various embodiments, as a result of learning these mappings for all training samples 402 (i.e., included in the subset 102a), the task-specific decoder 414 may be further trained to perform one or more bidirectional inference tasks for medical images. For example, the task-specific decoder 414 may be trained to detect and generate segmentation data (e.g., bounding box data, masks, etc.) for a target anatomical feature (e.g., a lesion), given an input medical image and a corresponding text prompt describing the target anatomical feature.

[0077] Figure 5 Another exemplary training process 500 for generating one or more multimodal AI models tailored to specific concepts in medical imaging according to one or more embodiments of the disclosed subject matter is presented. Figures 1 to 5 The training process 500 is similar to the training process 400, wherein additional multimodal input data associated with the medical image is added. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0078] In various embodiments, the training process 500 can be used to build one or more of the multimodal models 110 described with reference to step 108 of process 100. According to process 500, the multimodal model includes the three components or sub-models described with reference to process 400. In addition, the multimodal model can include additional transformers, one for each of the additional forms of multimodal input (e.g., acquisition transformer 502, EMR transformer 504, and / or one or more other transformers 506).

[0079] As described above, textual prompt data corresponding to unstructured text input provided by an expert annotator can correspond to a form of multimodal data that can be paired with a corresponding medical image. Various other forms of multimodal data can also be paired with the input image, including acquisition (ACQ) parameter data describing acquisition parameters (e.g., metadata) of the medical image, EMR data providing various relevant information about the patient and the medical image, and various other forms of clinical and non-clinical data. Through these embodiments, a multimodal model is trained to process multimodal input data 502, which consists of medical images paired with textual prompt data, acquisition data, EMR data, and potentially various other forms. To this end, the multimodal model can include independent transformers for each of different types of multimodal input data. Each of the transformers (e.g., image transformer 404, text transformer 406, acquisition transformer 504, and / or one or more other transformers 506) can generate a representation vector of the corresponding input data type (e.g., image representation vector 408, text representation vector 410, ACQ representation vector 506, EMR representation vector 508, and / or other representation vector 510), respectively. The corresponding transformer can be further configured to project the corresponding vector representation onto a common multidimensional space 512.

[0080] The task-specific decoder 414 can also learn mappings between different image and non-image features associated with different multimodal inputs based on projection vector representations of the different multimodal inputs and one or more unsupervised machine learning processes. In various embodiments, as a result of learning these mappings for training samples 502 (i.e., included in subset 102a), the task-specific decoder 414 can be further trained to perform one or more bidirectional inference tasks for medical images. For example, the task-specific decoder 414 can be trained to detect and generate segmentation data (e.g., bounding box data, masks, etc.) for target anatomical features (e.g., lesions) given an input medical image and one or more additional forms of input (e.g., textual prompt data, acquisition data, EMR data, etc.).

[0081] Figure 6A block diagram of an exemplary non-limiting computer system 600 that facilitates an iterative framework for learning multimodal mappings tailored for medical image inference tasks in accordance with one or more embodiments of the disclosed subject matter is shown. Figures 1 to 6 , computing system 600 may include or correspond to one or more computing systems that may perform or facilitate the execution of the operations described with reference to process 100 and other processes described herein (e.g., process 400, process 500, etc.). In this regard, system 600 may include or correspond to one or more computing devices, machines, virtual machines, computer executable components, data repositories, etc. that may be communicatively coupled to each other directly or via one or more wired or wireless communication frameworks.

[0082] The computing system 600 may include machine-executable (i.e., computer-executable) components or instructions embodied in one or more machines (e.g., embodied in one or more computer-readable storage media associated with the one or more machines) that may perform one or more of the operations described with reference to process 100 and the additional processes described herein. For example, the computing system 600 may include (or be operatively coupled to) at least one memory 612 storing computer-executable components and at least one processor (e.g., processing unit 614) executing the computer-executable components stored in the at least one memory 612. These computer-executable components may include (but are not limited to) a receiving component 602, an annotation component 604, a machine learning component 606, and a bidirectional inference component 608. In some embodiments, the memory 612 may be configured to include at least one processor (e.g., processing unit 614) storing computer-executable components. Figures 1 to 5 Some or all of the data described (e.g., medical image datasets, multimodal annotation data, model output data, etc.) and / or the model 110 (e.g., including pre-trained versions, partially trained versions, intermediate versions, and final versions). Examples of the memory and processor and other suitable computer or computing-based elements described can be referenced in Fig. 9 find (e.g., processing unit 904 and system memory 906, respectively), and can be combined to achieve the combination Figure 6 Or one or more of the systems or components shown and described in other figures disclosed herein.

[0083] Computing system 600 may also include one or more input / output devices 616 to facilitate receiving user input and presenting data to a user in conjunction with performing the various operations described with respect to machine-executable components and / or process 100 and the additional processes described herein. Fig. 9Suitable examples of input / output devices 616 are depicted (eg, input device 928 and output device 936). Computing system 100 may also include a system bus 610 that couples memory 612, processing unit 614, and input / output devices 616 to one another.

[0084] refer to Figure 1 and Figure 6 In one or more embodiments, the receiving component 602 can receive multimodal annotation data for medical images included in the medical image dataset 102. For example, the receiving component 602 can receive the annotated subset 102a' in response to the generation of the annotated subset by an expert annotator. In some embodiments, the annotation component 604 can include or correspond to a medical imaging application that allows an annotator to view the medical images included in the subset 102a as displayed via a GUI (and a display device), apply the tag annotation data to the medical image (e.g., using existing medical image tagging annotation software), provide unstructured text input describing relevant image features (e.g., via free-form data input, via voice that is subsequently converted to text, or via another suitable input mechanism), and optionally generate and / or provide (e.g., import, upload, provide a link to it, etc.) additional input in the form of relevant medical records, laboratory reports, metadata, relevant medical images, etc. With these embodiments, the receiving component 602 can receive the annotated subset 102a in response to the expert annotator generating the annotated subset using the annotation component 604. In some embodiments, the annotation component 604 may also facilitate receiving input from an expert annotator defining multimodal annotation criteria (according to stop 104 of process 100). With these embodiments, in conjunction with receiving the annotated subset 102a, the receiving component may also receive the multimodal annotation criteria applied by the annotator.

[0085] The machine learning component 606 can perform the multimodal model building (or training) operation described in step 108 of the reference process 100. To this end, the machine learning component 606 can use the annotated subset 102a' and one or more machine learning processes to train and generate one or more multimodal AI models 110. For example, in some embodiments, the machine learning component 606 can perform the multimodal model building (or training) operation described in step 108 of the reference process 100. To this end, the machine learning component 606 can use the annotated subset 102a' and one or more machine learning processes to train and generate one or more multimodal AI models 110. Figure 4 and Figure 5The training processes 400 and 500 described herein are used to generate various bidirectional models that can relate medical images and / or image features included in the medical images to corresponding non-image features (e.g., text features, acquisition parameter features, patient EMR features, etc.). In this regard, the machine learning component 606 may employ one or more machine learning processes to learn a bidirectional mapping between non-image features included in non-image annotation data applied to the medical images (e.g., multimodal annotation data in the form of text data and / or other forms in addition to the medical images themselves) and image features associated with the medical images. In embodiments where the multimodal annotation data includes image annotation data (such as labeled data (e.g., bounding boxes, masks, etc.) applied to the medical images in conjunction with learning a bidirectional mapping), the machine learning component 606 may use such GT image annotation data to facilitate learning a bidirectional mapping with labeled image features. In this regard, the machine learning component 606 may employ one or more machine learning processes to learn a bidirectional mapping between non-image features included in non-image annotation data applied to the medical images and image features associated with the medical images and the image annotation data (e.g., labeled data).

[0086] As a result of the one or more machine learning processes, the machine learning component 606 may also generate one or more models (e.g., model 110) configured to infer one or more of the non-image features associated with the new medical image given a new medical image (e.g., included in the additional medical image 102b or the corresponding runtime environment medical image). Additionally or alternatively, as a result of the one or more machine learning processes, the machine learning component 606 may generate one or more models (e.g., model 110) configured to infer one or more of the image features associated with the new medical image given the new medical image and a non-image input that identifies or indicates one or more of the non-image features associated with the new medical image, respectively.

[0087] In various embodiments, the bidirectional inference component 608 can execute one or more of the trained versions of the model on new medical images corresponding to the training images in conjunction with the iterative annotation and model building process (e.g., the dashed arrow line flow of process 100). Additionally or alternatively, the bidirectional inference component 608 can execute one or more of the trained version and the final version of the model 110 on new medical images corresponding to the training images in a runtime environment. For example, the bidirectional inference component 608 can execute a model suitable for performing a query task to facilitate finding relevant medical images that satisfy a text prompt query. In another example, the bidirectional inference component 608 can execute one or more of the models 110 in conjunction with performing an intelligent medical report.

[0088] Additionally or alternatively, when applied to a model suitable for performing intelligent annotation on medical images corresponding to training images, the annotation component 604 can be configured to perform the intelligent annotation model in conjunction with receiving corresponding textual prompts from an expert annotator (e.g., according to step 112). By these embodiments, according to steps 114, 116, and 118 of process 100, the annotation component 604 can also enable reviewing the results of the intelligent annotation model and updating the multimodal criteria.

[0089] Figure 7 A block diagram of an exemplary non-limiting computer-implemented method 700 for generating one or more multimodal AI models customized for specific concepts in medical imaging according to one or more embodiments of the disclosed subject matter is shown. At 702, the method 700 may include receiving, by a system (e.g., computing system 600) including a processor, multimodal annotation data for a medical image included in a medical image dataset, the multimodal annotation data including non-image annotation data (e.g., natural language text data provided by an expert annotator, EMR data, acquisition parameter data, laboratory data, etc.) and image annotation data (e.g., labeling data that marks and / or defines one or more anatomical features, artifacts, regions of interest, etc.). At 704, the method 700 also includes employing, by the system, one or more machine learning processes to learn a bidirectional mapping between non-image features included in the non-image annotation data and image features associated with the medical image and the image annotation data (e.g., via machine learning component 606). At 706, method 700 also includes generating, by the system as a result of one or more machine learning processes (e.g., via machine learning component 606), a model (e.g., of model 110) configured to infer one or more of the non-image features associated with a new medical image (e.g., the additional medical image 102b and / or the corresponding runtime environment medical image) given a new medical image. For example, in various embodiments, the one or more machine learning processes may involve employing / training one or more transformer models and / or one or more task-specific decoder models (e.g., CLIP model, CLIP-SEG model, etc.) to learn a bidirectional mapping using the annotated subset 102a', and training a task-specific decoder 414 in conjunction with training the one or more transformer models and / or the one or more task-specific decoder models to perform a specific bidirectional inference task on the medical image using the learned mapping.

[0090] Figure 8A block diagram of another exemplary non-limiting computer-implemented method 800 for generating one or more multimodal AI models customized for specific concepts in medical imaging according to one or more embodiments of the disclosed subject matter is shown. At 802, the method 800 may include receiving, by a system (e.g., computing system 600) including a processor, multimodal annotation data for a medical image included in a medical image dataset, the multimodal annotation data including non-image annotation data (e.g., natural language text data provided by an expert annotator, EMR data, acquisition parameter data, laboratory data, etc.) and image annotation data (e.g., labeling data that marks and / or defines one or more anatomical features, artifacts, regions of interest, etc.). At 804, the method 800 also includes employing, by the system, one or more machine learning processes to learn a bidirectional mapping between non-image features included in the non-image annotation data and image features associated with the medical image and the image annotation data (e.g., via machine learning component 606). At 806, method 800 also includes generating, by the system as a result of one or more machine learning processes (e.g., via machine learning component 606), a model (e.g., of model 110) configured to infer one or more of the image features associated with the new medical image given a new medical image (e.g., additional medical image 102b and / or runtime environment medical image) and non-image input (e.g., unstructured textual hint data), the non-image input identifying or indicating one or more of the non-image features associated with the new medical image, respectively. For example, in various embodiments, the one or more machine learning processes may involve employing / training one or more transformer models and / or one or more task-specific decoder models (e.g., CLIP model, CLIP-SEG model, etc.) to learn a bidirectional mapping using annotated subset 102a', and training the task-specific decoder 414 in conjunction with training the one or more transformer models and / or the one or more task-specific decoder models to generate segmentation annotation data corresponding to image annotation data (e.g., bounding box data, mask data, etc.) for the new medical image defining one or more target anatomical features of interest.

[0091] Exemplary Operating Environment

[0092] One or more embodiments may be systems, methods and / or computer program products at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0093] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following items: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or a raised structure in a groove on which instructions are recorded) and any suitable combination of the foregoing items. As used herein, a computer readable storage medium should not be understood as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire. To this end, computer-readable storage media, machine-readable storage media, and the like used herein may include non-transitory computer-readable storage media, non-transitory machine-readable storage media, and the like.

[0094] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the corresponding computing / processing device.

[0095] The computer-readable program instructions for implementing the operation of the present invention can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and process programming languages ​​(such as "C" programming language or similar programming languages). Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In some embodiments, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions, so as to perform aspects of the present invention.

[0096] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It is understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0097] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to function in a specific manner, so that the computer-readable storage medium with instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0098] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0099] The flow chart and block diagram in the accompanying drawings illustrate the architecture, function and operation of the possible implementation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each frame in the flow chart or block diagram can represent a module, a fragment or a part of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the function indicated in the frame may not occur in the order indicated in the figure. For example, depending on the function involved, two frames of continuous display can actually be performed substantially simultaneously, or sometimes the frame can be performed in reverse order. It will also be noted that each frame of the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be implemented by a system based on dedicated hardware that performs a specified function or action or implements a combination of dedicated hardware and computer instructions.

[0100] Combination Fig. 9 , the systems and processes described below may be embodied in hardware, such as a single integrated circuit (IC) chip, multiple ICs, application specific integrated circuits (ASICs), etc. In addition, the order in which some or all of the program blocks appear in each program should not be considered limiting. On the contrary, it should be understood that some program blocks can be executed in various orders, not all of which can be explicitly shown in this article.

[0101] refer to Fig. 9 , an illustrative example environment 900 for implementing various aspects of the claimed subject matter includes a computer 902. The computer 902 includes a processing unit 904, a system memory 906, a codec 935, and a system bus 908. The system bus 908 couples system components including, but not limited to, the system memory 906 to the processing unit 904. The processing unit 904 can be any of a variety of available processors. Dual microprocessors and other multi-processor architectures can also be used as the processing unit 904.

[0102] The system bus 908 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, or a native bus using any of a variety of available bus architectures including, but not limited to, Industry Standard Architecture (ISA), Micro Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Native Bus (VLB), Peripheral Component Interconnect (PCI), card bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association (PCMCIA), FireWire (IEEE 1394), and Small Computer System Interface (SCSI).

[0103] In various embodiments, the system memory 906 includes a volatile memory 910 and a non-volatile memory 912, which may adopt one or more memory architectures in the disclosed memory architecture. The basic input / output system (BIOS) (including basic routines such as transferring information between elements in the computer 902 during startup) is stored in the non-volatile memory 912. In addition, according to the present invention, the codec 935 may include at least one of an encoder or a decoder, wherein at least one of the encoder or the decoder may be composed of hardware, software, or a combination of hardware and software. Although the codec 935 is depicted as a separate component, the codec 935 may be included in the non-volatile memory 912. By way of illustration and not limitation, the non-volatile memory 912 may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a 3D flash memory, or a resistive memory, such as a resistive random access memory (RRAM). In at least some embodiments, the non-volatile memory 912 may adopt one or more memory devices in the disclosed memory devices. In addition, the non-volatile memory 912 can be a computer memory (e.g., physically integrated with the computer 902 or its motherboard) or a removable memory. Examples of suitable removable memory that can be used to implement the disclosed embodiments can include a secure digital (SD) card, a compact flash (CF) card, a universal serial bus (USB) memory stick, etc. The volatile memory 910 includes a random access memory (RAM) that acts as an external cache memory, and one or more of the disclosed memory devices can also be used in various embodiments. By way of illustration and not limitation, RAM can be provided in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and enhanced SDRAM (ESDRAM), etc.

[0104] Computer 902 may also include removable / non-removable, volatile / non-volatile computer storage media. Fig. 9For example, a disk storage device 914 is shown. The disk storage device 914 includes, but is not limited to, devices such as disk drives, solid-state disks (SSDs), flash memory cards, or memory sticks. In addition, the disk storage device 914 may include a separate storage medium or a storage medium combined with other storage media, including but not limited to an optical drive, such as a CD-ROM device (CD-ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), or a digital universal magneto-optical disk ROM drive (DVD-ROM). In order to facilitate the connection of the disk storage device 914 to the system bus 908, a removable or non-removable interface, such as an interface 916, is generally used. It should be understood that the disk storage device 914 can store information related to the user. Such information can be stored at a server or provided to an application running on a server or user device. In one embodiment, the user can be notified (for example, by an output device 936) of the type of information stored in the disk storage device 914 or transmitted to a server or application. The user can be provided with the opportunity to select to join or opt out of collecting or sharing such information through a server or application (for example, by input from an input device 928).

[0105] It should be understood that Fig. 9 Software is described that acts as an intermediary between a user and the basic computer resources described in the appropriate operating environment 900. Such software includes an operating system 910. The operating system 910, which may be stored on a disk storage device 914, is used to control and allocate the resources of the computer 902. Application programs 920 utilize the management of resources by the operating system 910 through program modules 924, and program data 926 stored in the system memory 906 or on the disk storage device 914, such as a startup / shutdown transaction table. It should be appreciated that the claimed subject matter may be implemented with various operating systems or combinations of operating systems.

[0106] The user enters commands or information into the computer 902 through input devices 928. Input devices 928 include, but are not limited to, pointing devices such as mice, trackballs, styluses, touchpads, keyboards, microphones, joysticks, gamepads, satellite dishes, scanners, TV tuner cards, digital cameras, digital video cameras, webcams, and the like. These and other input devices are connected to the processing unit 904 through the system bus 908 via interface ports 930. Interface ports 930 include, for example, serial ports, parallel ports, game ports, and universal serial buses (USB). Output devices 936 use some of the same types of ports as input devices 928. Thus, for example, a USB port can be used to provide input to the computer 902 and output information from the computer 902 to output devices 936. Output adapters 934 are provided to illustrate that there are some output devices 936 such as monitors, speakers, and printers, as well as other output devices 936 that require special adapters. By way of illustration and not limitation, output adapters 934 include video cards and sound cards that provide connection means between output devices 936 and the system bus 908. It should be noted that other devices or systems of devices provide input and output capabilities, such as remote computer 938 .

[0107] The computer 902 can work in a networked environment using a logical connection to one or more remote computers, such as a remote computer 938. The remote computer 938 can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based device, a peer device, a smart phone, a tablet computer, or other network node, and typically includes many of the elements described relative to the computer 902. For the purpose of simplicity, only a memory storage device 940 is shown for the remote computer 938. The remote computer 938 is logically connected to the computer 902 through a network interface 942, and then connected via a communication connection 944. The network interface 942 covers a wired communication network or a wireless communication network, such as a local area network (LAN) and a wide area network (WAN) and a cellular network. LAN technologies include fiber distributed data interface (FDDI), copper distributed data interface (CDDI), Ethernet, token ring, etc. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks such as integrated services digital networks (ISDN) and variations thereon, packet switching networks, and digital subscriber lines (DSL).

[0108] The communication connection 944 refers to the hardware / software used to connect the network interface 942 to the bus 908. Although the communication connection 944 is shown internal to the computer 902 for clarity of illustration, the communication connection can also be external to the computer 902. For exemplary purposes only, the hardware / software required to connect to the network interface 942 includes internal technology and external technology, such as modems, including conventional telephone grade modems, cable modems and DSL modems, ISDN adapters, as well as wired and wireless Ethernet cards, hubs and routers.

[0109] Fig.10 1 is a schematic block diagram of a sample computing environment 1000 with which the subject matter of the present disclosure can interact. System 1000 includes one or more clients 1002. Client 1002 (e.g., corresponding to client system 700 in some embodiments) can be hardware and / or software (e.g., threads, processes, computing devices). System 1000 also includes one or more servers 1004 (e.g., corresponding to computing system 600 in some embodiments). Therefore, in addition to other models, system 1000 can correspond to a two-tier client server model or a multi-tier model (e.g., client, middle-tier server, data server). Server 1004 can also be hardware and / or software (e.g., threads, processes, computing devices). For example, server 1004 can accommodate threads to perform conversions by adopting the present disclosure. One possible communication between client 1002 and server 1004 can be in the form of data packets transmitted between two or more computer processes (e.g., including feedback information 132).

[0110] The system 1000 includes a communication framework 1006 that can be used to facilitate communication between the client 1002 and the server 1004. The client 1002 is operatively connected to one or more client data repositories 1008, which can be used to store information local to the client 1002. Similarly, the server 1004 is operatively connected to one or more server data repositories 1012, which can be used to store information local to the server 1004.

[0111] It should be noted that various aspects or features of the present disclosure can be utilized in substantially any wireless telecommunication or radio technology, for example, Wi-Fi; Bluetooth; Worldwide Interoperability for Microwave Access (WiMAX); Enhanced General Packet Radio Service (Enhanced GPRS); 3rd Generation Partnership Project (3GPP) Long Term Evolution (LTE); 3rd Generation Partnership Project 2 (3GPP2) Ultra Mobile Broadband (UMB); 3GPP Universal Mobile Telecommunications System (UMTS); High Speed ​​Packet Access (HSPA); High Speed ​​Downlink Packet Access (HSDPA); High Speed ​​Uplink Packet Access (HSUPA); GSM (Global System for Mobile Communications) EDGE (Enhanced Data Rates for GSM Evolution) Radio Access Network (GERAN); UMTS Terrestrial Radio Access Network (UTRAN); LTE Advanced (LTE-A); etc. In addition, some or all aspects described herein can be utilized in traditional telecommunication technologies (e.g., GSM). In addition, mobile as well as non-mobile networks (e.g., the Internet, data service networks such as Internet Protocol Television (IPTV), etc.) can utilize aspects or features described herein.

[0112] Although the subject matter has been described above in the general context of computer executable instructions of a computer program running on one and / or multiple computers, it will be appreciated by those skilled in the art that the present disclosure may also or may be implemented in combination with other program modules. In general, program modules include routines, programs, components, data structures, etc. that perform specific tasks and / or implement specific abstract data types. In addition, it will be appreciated by those skilled in the art that the method of the present invention may be practiced with other computer system configurations, including single-processor or multi-processor computer systems, small computing devices, large computers, and personal computers, handheld computing devices (e.g., PDAs, phones), microprocessor-based or programmable consumer or industrial electronics, etc. The aspects shown may also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices linked by a communication network. However, some (if not all) aspects of the present disclosure may be practiced on stand-alone computers. In a distributed computing environment, program modules may be located in local and remote memory storage devices.

[0113] As used in this application, the terms "component", "system", "platform", "interface", etc. may refer to and / or may include computer-related entities or entities associated with an operating machine having one or more specific functionalities. The entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a server and a server may be a component. One or more components may reside within a process and / or a thread of execution, and a component may be located on one computer and / or distributed between two or more computers.

[0114] As another example, the corresponding components can be executed according to various computer-readable media having various data structures stored thereon. The components can communicate via local and / or remote processes, such as according to signals having one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, and / or interacts with other systems across a network (such as the Internet) via a signal). As another example, a component can be a device having a specific function provided by a mechanical part operated by an electrical or electronic circuit, which is operated by a software or firmware application executed by a processor. In such cases, the processor can be inside or outside the device and can execute at least a portion of the software or firmware application. As another example, a component can be a device that provides specific functionality through an electronic component rather than a mechanical part, wherein the electronic component may include a processor or other devices for executing software or firmware that at least partially imparts functionality to the electronic component. On the one hand, the component can simulate the electronic component via a virtual machine, for example, in a cloud computing system.

[0115] Additionally, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise or clear from the context, "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing instances. Furthermore, the articles "a" and "an" used in the specification and the drawings should generally be construed to mean "one or more" unless specified otherwise or clear from the context to be directed to a singular form.

[0116] As used herein, the terms "example" and / or "exemplary" are used to mean serving as an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited to such examples. In addition, any aspect or design described herein as "example" and / or "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs, nor is it intended to exclude equivalent exemplary structures and techniques known to those of ordinary skill in the art.

[0117] The various aspects or features described herein can be implemented as methods, devices, systems, or products using standard programming or engineering techniques. In addition, the various aspects or features disclosed in the present disclosure can be implemented by implementing at least one or more of the program modules of the methods disclosed herein, and the program modules are stored in a memory and executed at least by a processor. Other combinations of hardware and software or hardware and firmware can implement or implement the aspects described herein, including the disclosed methods. As used herein, the term "product" can cover a computer program that can be accessed from any computer-readable device, carrier, or storage medium. For example, a computer-readable storage medium can include, but is not limited to, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe ...), an optical disk (e.g., a compact disk (CD), a digital versatile disk (DVD), a blue optical disk (BD) ...), a smart card, and a flash memory device (e.g., a card, a stick, a key drive ...), etc.

[0118] As used in this specification, the term "processor" may refer to substantially any computational processing unit or device, including but not limited to a single-core processor; a single processor with software multithreaded execution capability; a multi-core processor; a multi-core processor with software multithreaded execution capability; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. In addition, a processor may refer to an integrated circuit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. In addition, the processor may utilize nanoscale architectures (such as, but not limited to, transistors, switches, and gates based on molecules and quantum dots) in order to optimize space usage or enhance the performance of user equipment. The processor may also be implemented as a combination of computational processing units.

[0119] In this disclosure, terms such as "repository," "storage," "data repository," "data storage," "data storage," "database," and substantially any other information storage component related to the operation and functionality of the component are used to refer to a "memory component," an entity embodied in a "memory," or a component that includes a memory. It should be understood that the memory and / or memory components described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory.

[0120] By way of illustration and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). For example, volatile memory may include RAM that may act as external cache memory. By way of illustration and not limitation, RAM may be provided in a variety of forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). In addition, the disclosed memory components of the systems or methods herein are intended to include, but are not limited to, including these and any other suitable types of memory.

[0121] It should be recognized and appreciated that components described with respect to a particular system or method may include the same or similar functionality as corresponding components described with respect to other systems or methods disclosed herein (eg, separately named components or similarly named components).

[0122] What has been described above includes examples of systems and methods that provide the advantages of the present disclosure. Of course, it is not possible to describe every conceivable combination of components or methods for the purpose of describing the present disclosure, but one of ordinary skill in the art will recognize that many additional combinations and permutations of the present disclosure are possible. In addition, to the extent that the terms "including," "having," "having," and the like are used in the detailed description, claims, appendices, and drawings, such terms are intended to be inclusive in a manner similar to the term "comprising," as interpreted when "including" is used as a transitional word in a claim.

Claims

1. A system, comprising: a memory storing computer executable components; and a processor that executes the computer executable components stored in the memory, wherein the computer executable components include: a receiving component, the receiving component receiving multimodal annotation data for a medical image comprising a medical image dataset, the multimodal annotation data comprising non-image annotation data and image annotation data; and A machine learning component employs one or more machine learning processes to learn a bidirectional mapping between non-image features included in the non-image annotation data and image features associated with the medical image and the image annotation data.

2. A system according to claim 1, wherein as a result of the one or more machine learning processes, the machine learning component generates a model, the model being configured to infer one or more of the non-image features associated with the new medical image given a new medical image.

3. The system of claim 2 , wherein the non-image annotation data complies with one or more standards, wherein the receiving component receives updated non-image annotation data for the medical image that complies with one or more updated standards based on one or more errors associated with the performance of the model on the new medical image, wherein the machine learning component employs one or more second machine learning processes to learn an updated bidirectional mapping between updated non-image features included in the updated non-image annotation data and the image features associated with the medical image and the image annotation data, and wherein as a result of the one or more second machine learning processes, the machine learning component generates an updated version of the model, the updated version of the model being configured to more accurately infer the one or more of the non-image features associated with the new medical image given the new medical image.

4. The system of claim 1 , wherein as a result of the one or more machine learning processes, the machine learning component generates a model configured to infer one or more of the image features associated with the new medical image given a new medical image and a non-image input, wherein the non-image input identifies or indicates one or more of the non-image features associated with the new medical image, respectively.

5. The system of claim 4 , wherein the non-image annotation data complies with one or more standards, wherein the receiving component receives updated non-image annotation data for the medical image that complies with one or more updated standards based on one or more errors associated with the performance of the model on the new medical image, wherein the machine learning component employs one or more second machine learning processes to learn an updated bidirectional mapping between updated non-image features included in the updated non-image annotation data and the image features associated with the medical image and the image annotation data, and wherein as a result of the one or more second machine learning processes, the machine learning component generates an updated version of the model, the updated version of the model being configured to more accurately infer the one or more of the image features associated with the new medical image given the new medical image and the non-image input.

6. The system of claim 4, wherein in conjunction with inferring the one or more of the image features, the machine learning component configures the model to generate new image annotation data defining the one or more of the image features for the new medical image.

7. The system of claim 6, wherein the image annotation data and the new image annotation data comprise labeling data applied to the medical image that labels one or more anatomical features, and the non-image annotation data and the non-image input each comprise textual data.

8. The system of claim 6, wherein the computer executable components further comprise: An annotation component employing the model to generate the new image annotation data for the new medical image given the new medical image and the non-image input, wherein the new medical image comprises an additional medical image included in the medical image dataset other than the medical image.

9. The system of claim 1, wherein the non-image annotation data follows defined clinical criteria.

10. The system of claim 1, wherein the non-image annotation data comprises unstructured text data.

11. The system of claim 10, wherein the unstructured text data comprises natural language text data describing a clinical interpretation of the medical image.

12. The system of claim 11, wherein the receiving component receives the natural language text data from a single entity, and wherein the bidirectional mapping is customized for a natural language employed by the single entity.

13. The system of claim 1, wherein non-image annotation data and the bidirectional mapping are customized based on a target medical image inference task and a type of the medical image.

14. The system of claim 1, wherein the non-image annotation data comprises patient information about a corresponding patient represented in the medical image included in one or more electronic medical record data files.

15. A method comprising: receiving, by a system including a processor, multimodal annotation data for a medical image comprising a medical image dataset, the multimodal annotation data comprising non-image annotation data and image annotation data; as well as One or more machine learning processes are employed by the system to learn a bidirectional mapping between non-image features included in the non-image annotation data and image features associated with the medical image and the image annotation data.

16. The method according to claim 15, further comprising: As a result of the one or more machine learning processes, a model is generated by the system, the model being configured to, given a new medical image, infer one or more of the non-image features associated with the new medical image.

17. The method according to claim 15, further comprising: As a result of the one or more machine learning processes, a model is generated by the system, the model being configured to infer one or more of the image features associated with the new medical image given a new medical image and a non-image input, the non-image input identifying or indicating one or more of the non-image features associated with the new medical image, respectively.

18. The method of claim 17, wherein generating the model comprises configuring the model to generate new image annotation data defining the one or more of the image features for the medical image, and wherein the method further comprises: The model is employed by the system to generate the new image annotation data for the new medical image given the new medical image and the non-image input, wherein the new medical image includes an additional medical image included in the medical image dataset other than the medical image.

19. The method of claim 15, wherein the non-image annotation data comprises natural language text data describing a clinical interpretation of the medical image according to one or more defined criteria.

20. A non-transitory machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations comprising: receiving multimodal annotation data for a medical image comprising a medical image dataset, the multimodal annotation data comprising non-image annotation data and image annotation data; employing one or more machine learning processes to learn a bidirectional mapping between non-image features included in the non-image annotation data and image features associated with the medical image and the image annotation data; and As a result of the one or more machine learning processes, a model is generated, the model being configured to: inferring one or more of the non-image features associated with the new medical image given the new medical image, and inferring one or more of the image features associated with the new medical image given the new medical image and non-image input identifying or indicating one or more of the non-image features respectively associated with the new medical image; and The model is employed to infer the one or more non-image features or the one or more image features associated with the new medical image.

Images