Generation of composite images from non-image feedback for continuous model refinement

Abstract

This disclosure describes techniques for refining or updating a deployed medical image inference model using a composite image generated from non-image data feedback. In an example, a system may include a memory storing computer executable components and a processor executing the computer executable components stored in the memory. The computer executable component may include an image generation component that generates a synthetic medical image based on feedback information associated with a performance of a medical image inference model that is received in association with application of the medical image inference model to the medical image in a deployment environment, wherein the feedback information does not include image data. The computer executable component may also include a refinement component that updates the medical image inference model using the composite image and a model update process.

Description

Technical Field

[0001] The present application relates to techniques for refining or updating a deployed medical image inference model using synthetic images generated from non-image data feedback. Background Art

[0002] Machine learning (ML) models are used in many medical image processing and analysis tasks, such as organ segmentation, anomaly detection, diagnostic classification, risk prediction, temporal analysis, image reconstruction, etc. However, one of the fundamental problems with data-driven machine learning approaches is that the final model inference ability is limited by the scope of the training data used to develop the model. In order to continuously improve these models over time, it is necessary to retrain and update the models to improve their performance based on feedback about model errors or failure modes observed in actual deployment environments (e.g., after initial training and development). Typically, failure modes cannot be understood without access to the input patient imaging data processed and / or generated by the model in the deployment environment. However, obtaining deployment site patient imaging data can be difficult because there are often contractual restrictions and privacy issues that prohibit sharing medical image data between model development sites (i.e., vendor sites) and model deployment sites (e.g., customer sites). Summary of the invention

[0003] The following presents a summary of the invention to provide a basic understanding of one or more embodiments of the invention. The summary of the invention is not intended to identify key or important elements, nor is it intended to delineate any scope of different embodiments or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a prelude to a more detailed description presented later. In one or more embodiments, systems, computer-implemented methods, devices, and / or computer program products are described that facilitate the use of synthetic images generated based on non-image data feedback to refine or update deployed medical image inference models.

[0004] According to one embodiment, a system is provided, the system comprising a memory and a processor, the memory storing a computer executable component, the processor executing the computer executable component stored in the memory. The computer executable component may include an image generation component, the image generation component generating a synthetic medical image based on feedback information associated with the performance of a medical image inference model, the feedback information being received in association with the application of the medical image inference model to a medical image in a deployment environment, wherein the feedback information does not include image data. The computer executable component may also include a refinement component, the refinement component updating the medical image inference model using the synthetic image and a model update process (e.g., wherein the model update process includes a machine learning process).

[0005] In one or more embodiments, the feedback information includes non-image data regarding properties of at least some of the medical images, and wherein the feedback information does not include the medical images. For example, at least some of the medical images may correspond to a subset of medical images for which performance of the medical imaging model is associated with one or more negative performance criteria. Additionally or alternatively, the feedback information may include non-image data related to results of the medical image inference model.

[0006] In various embodiments, the feedback information includes text data, and the image generation component generates a synthetic medical image using a synthetic image generation model machine learning model that generates the synthetic image based on the text data and without input data including image data or additional image data. In some embodiments, the medical image inference model includes a first machine learning model trained on a training data set including a first medical image, and the synthetic image generation model includes a second machine learning model trained on training text description data associated with the first medical image. In some embodiments, the computer executable component also includes a training component that trains the first machine learning model and the second machine learning model.

[0007] In one or more embodiments, the feedback information may include automatic assessment data regarding an assessment of at least one of: performance, at least some of the medical images, or results of a medical image inference model, the automatic assessment data being generated using one or more artificial intelligence (AI) assessment models. Additionally or alternatively, the feedback information may include metadata associated with at least some of the medical images. Additionally or alternatively, the feedback information may include clinical report data associated with at least some of the medical images. Additionally or alternatively, the feedback information may include user-provided feedback regarding an assessment of at least one of: performance, at least some of the medical images, or results of a medical image inference model. In some specific implementations of these embodiments, the computer executable component may also include a feedback component that facilitates receiving user-provided feedback using one or more feedback tools associated with the medical imaging application in association with reviewing the results using the medical imaging application.

[0008] In some embodiments, elements described in conjunction with the disclosed systems can be embodied in different forms, such as a computer-implemented method, a computer program product, or another form. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1A flowchart is presented of an example high-level process for refining or updating a deployed medical image inference model using synthetic images generated from non-image data feedback, in accordance with one or more embodiments of the disclosed subject matter.

[0010] Figure 2 A table summarizing some examples of non-image feedback available for a medical image inference model in a deployment environment according to one or more embodiments is presented.

[0011] FIG. 3A to FIG. 3D Examples of automated evaluation data regarding evaluations of input images and / or results of medical image inference models that may be obtained via various types of AI evaluation models are presented.

[0012] Figure 4 An example feedback tool associated with a medical imaging application is presented that facilitates receiving user feedback regarding failure information associated with a medical image inference model in accordance with one or more embodiments of the disclosed subject matter.

[0013] Figure 5 An example process for generating a synthetic medical image using textual feedback in accordance with one or more embodiments of the disclosed subject matter is illustrated.

[0014] Figure 6 An example non-limiting computing system that can be associated with a vendor site that facilitates using synthetic images generated from non-image data feedback to refine or update a deployed medical image inference model in accordance with one or more embodiments of the disclosed subject matter is illustrated.

[0015] Figure 7 An example non-limiting computing system that can be associated with a client site that facilitates using synthetic images generated based on non-image data feedback to refine or update a deployed medical image inference model in accordance with one or more embodiments of the disclosed subject matter is illustrated.

[0016] Figure 8 A block diagram illustrating an example non-limiting computer-implemented method for refining or updating a deployed medical image inference model using synthetic images generated from non-image data feedback in accordance with one or more embodiments of the disclosed subject matter.

[0017] Fig. 9 A block diagram illustrating another example, non-limiting computer-implemented method for refining or updating a deployed medical image inference model using synthetic images generated from non-image data feedback, in accordance with one or more embodiments of the disclosed subject matter.

[0018] Fig.10A block diagram illustrating an example non-limiting operating environment in which one or more embodiments described herein may be facilitated.

[0019] Fig.11 A block diagram is shown of another exemplary non-limiting operating environment in which one or more embodiments described herein may be facilitated. DETAILED DESCRIPTION

[0020] The following specific embodiments are merely exemplary and are not intended to limit the application or use of the embodiments and / or the embodiments. In addition, it is not intended to be bound by any express or implied information set forth in the aforementioned "background technology" section or "content of the invention" section or "specific embodiments" section.

[0021] The present disclosure provides systems, computer-implemented methods, apparatus, and / or computer program products that facilitate the use of synthetic images generated based on non-image data feedback to refine or update deployed image inference models. Various embodiments of the disclosed techniques may be employed in association with a medical image inference model configured to perform inference tasks on one or more medical images. However, the disclosed techniques may also be applied to other types of image inference models suitable for performing various inference tasks on other types of images besides medical images.

[0022] As described in the background section, the data-driven nature of ML models brings challenges in the inference stage. Specifically, one of the fundamental problems of data-driven machine learning methods is that the final model inference ability is limited by the scope of the training data used to develop the model. For example, in various fields, AI models are usually trained and developed by specific model development entities, such as suppliers that develop ML models for use by multiple client systems. For example, in the field of healthcare, suppliers usually include software development systems that specialize in developing ML tools for healthcare systems (such as hospitals, medical imaging centers, etc.). According to this example, healthcare systems (e.g., hospitals, medical imaging centers, etc.) correspond to client systems. The terms "supplier", "supplier site", "supplier system" or their variations are used to refer to a supplier entity or system that generates, updates ML models and provides the ML models to one or more client systems for use. The terms "client", "client site", "client system" or their variations are generally used herein to refer to client systems that receive and adopt ML models on site.

[0023] According to the above vendor-client system architecture for ML model development and deployment, the scope of the models created by the vendor is limited by the training data used by the vendor. Depending on the nature of the model and the data being processed, it can be very difficult, if not impossible, for the vendor training data to fully understand all of the variations that the model will encounter when deployed in the field. For example, an ML model trained on clinical data from one hospital may not perform as well on the same type of data from another hospital. In this regard, the vendor / model developer does not have access to every possible medical image that has ever been taken in the world, and as patients and acquisition technologies evolve over time, new medical images and features will continue to evolve. Therefore, techniques for regularly monitoring the performance of models deployed at client sites and updating the models over time based on the specific nuances and variations of the data at the client site are critical to the successful integration of ML models for real-world applications.

[0024] In order to continuously improve these models over time, it is necessary to retrain and update the models to improve their performance based on feedback about model errors or failure modes observed in actual deployment environments (e.g., after initial training and development). Typically, failure modes cannot be understood without access to the input patient imaging data processed and / or generated by the model in the deployment environment. However, obtaining deployment site patient imaging data can be difficult because there are often contractual restrictions and privacy issues that prohibit sharing medical image data between vendor sites and client sites.

[0025] The disclosed technology solves this problem by using techniques to obtain non-image feedback from a deployment site about performance issues associated with an image inference model at the deployment site without obtaining input patient imaging data processed and / or generated by the model at the deployment site. For example, the non-image feedback may include textual data describing an input image or an image processed by a model in which the output is associated with a performance issue (e.g., an accuracy issue, a specificity issue, a failure mode, or another performance measure indicating poor model performance). The non-image feedback may also include textual data describing the results or model output for one or more performance issues observed therefor. The non-image feedback may also include textual data describing or indicating the one or more performance issues (e.g., a failure mode).

[0026] The non-image feedback will vary depending on the type and / or task of the image inference model. For example, assume that the deployed model includes a medical image optimization model that is configured to process an input medical image to generate an output image with improved image quality (e.g., resolution, alignment, or another quality-related characteristic). According to this example, assume that the quality of the output image generated by the model is not sufficiently improved or as expected by the model (e.g., as determined via one or more quality assessment metrics). In such a scenario, the non-image feedback may include text information identifying or indicating how the quality of the output image is insufficient, text information describing the input image, and / or text information describing quality defects associated with the output image.

[0027] The disclosed technology also employs one or more text-to-synthetic image generation techniques to generate a synthetic image representing a deployment site input image associated with a model performance problem using non-image feedback. The disclosed technology also employs the synthetic image to continuously or regularly improve or update the model over time.

[0028] The term "image data" is used herein to refer to one or more images, image data structures (e.g., image files in any format), and / or video data consisting of picture elements (e.g., pixels) that can be rendered as visual data objects via a display. For example, the term image data may refer to one or more digital images, native images, synthesized images, etc. The term image data may also refer to a portion of an image, a segmentation mask, an enhanced image, a graphical image, a sinusoidal image, or another type of image construction that includes picture elements that can be rendered via a display.

[0029] The term "medical image data" is used to refer to image data depicting one or more anatomical regions of a patient. References herein to medical images or medical image data may include any type of medical image associated with various types of medical image acquisition / capture modalities. For example, medical images may include (but are not limited to): radiation therapy (RT) images, X-ray (XR) images, digital radiography (DX) X-ray images, X-ray angiography (XA) images, panoramic X-ray (PX) images, computed tomography (CT) images, mammography (MG) images (including tomosynthesis equipment), magnetic resonance imaging (MRI) images, ultrasound (US) images, color flow Doppler (CD) images, positron emission tomography (PET) images, single photon emission computed tomography (SPECT) images, nuclear medicine (NM) images, etc. Medical images may also include synthetic versions of native medical images, such as enhanced, modified or improved versions of native medical images, enhanced versions of native medical images generated using one or more image processing techniques, etc. In this regard, the term "native" image or "real" image is used herein to refer to an image in its originally captured form and / or its received form prior to processing via one or more medical image inference models. The term "synthetic" image is used herein to distinguish between a native image or a real image, and refers to an image generated or derived from a native or real image using one or more synthetic processing techniques (e.g., generating a synthetic image from text data). In some embodiments, the term "image data" may include raw measurement data (or simulated measurement data) used to generate a medical image (e.g., raw measurement data captured via a medical image acquisition process).

[0030] The term "non-image data" is used herein to refer to any type of data that does not include image data. In this regard, the term "non-image data" may include text data (e.g., a text data file in any text data type format), audio data (e.g., an audio file in any type of audio file format), or another type of data format other than image data.

[0031] Unless the context warrants a specific distinction between the terms, the terms "algorithm" and "model" are used interchangeably herein. Unless the context warrants a specific distinction between the terms, the terms "artificial intelligence (AI) model" and "machine learning (ML) model" are used interchangeably herein. References to AI models or ML models herein may include any type of AI model or ML model, including (but not limited to): deep learning models, neural network models, deep neural network models (DNNs), convolutional neural network models, and generative adversarial neural network models, etc. AI models or ML models may include supervised learning models, unsupervised learning models, semi-supervised learning models, combinations thereof, and models that employ other types of ML learning techniques. AI models or ML models may include a single model or a group of two or more models (e.g., enabling models, etc.).

[0032] The term "image inference model" is used herein to refer to an AI / ML model suitable for performing image processing or analysis tasks on image data. The image processing or analysis tasks may vary. In various embodiments, the image processing or analysis tasks may include (but are not limited to): segmentation tasks, image reconstruction tasks, object recognition tasks, motion detection tasks, video tracking tasks, optical flow tasks, etc. The image inference model described herein may include a two-dimensional (2D) image processing model and a three-dimensional (3D) image processing model. The image processing model may adopt various types of AI / ML models (e.g., deep learning models, neural network models, deep neural network models, DNN, CNN, GAN, etc.). Unless the context warrants a specific distinction between the terms, the terms "image inference model", "image processing model", "image analysis model", etc. are used interchangeably herein.

[0033] The term "inference output based on an image" is used herein to refer to a determination or prediction that an image processing model is configured to generate. For example, an inference output based on an image may include a segmentation mask, a reconstructed image, an adapted image, an annotated image, a classification, a value, etc. The inference output based on an image will vary based on the type of model and the specific task that the model is configured to perform. The inference output based on an image may include a data object that can be presented (e.g., a visual data object), stored, used as an input for another processing task, etc. The output may be in different formats, such as: Digital Imaging and Communication in Medicine (DICOM) Structured Report (SR), DICOM Secondary Capture, DICOM Parameter Mapping, Image, Text, and / or JavaScript Object Designation (JSON). The terms "inference output based on an image", "inference output", "inference result", "inference", "output", "result", "prediction", etc. are used interchangeably herein unless the context determines that a special distinction is made between these terms.

[0034] As used herein, a "medical imaging inference model" refers to an image inference model that is customized to perform image processing / analysis tasks on medical image data. For example, medical imaging processing / analysis tasks may include (but are not limited to): disease / condition classification, disease region segmentation, organ segmentation, disease quantification, disease / condition staging, risk prediction, temporal analysis, anomaly detection, anatomical feature characterization, medical image reconstruction, etc. Unless the context warrants a specific distinction between the terms, the terms "medical image inference model", "medical image processing model", "medical image analysis model", etc. may be used interchangeably herein.

[0035] Embodiments of the systems and devices described herein may include one or more machine-executable (i.e., computer-executable) components or instructions embodied in one or more machines (e.g., embodied in one or more computer-readable storage media associated with one or more machines). Such components, when executed by one or more machines (e.g., processors, computers, computing devices, virtual machines, etc.), may cause one or more machines to perform the described operations. These computer / machine-executable components or instructions (and other components or instructions described herein) may be stored in a memory associated with one or more machines. The memory may also be operably connected to at least one processor so that the components may be executed by the at least one processor to perform the described operations. In some embodiments, the memory may include a non-transitory machine-readable medium that includes executable components or instructions that facilitate the execution of the operations described for the corresponding executable components when executed by the processor. Examples of the described and memory and processors and other suitable computer or computing-based elements may refer to Fig. 9 (e.g., processing unit 904 and system memory 906, respectively), and can be combined to achieve the combination Figure 1 Or one or more of the systems or components shown and described in other figures disclosed herein.

[0036] One or more embodiments are now described with reference to the accompanying drawings, wherein the same reference numerals are used to represent the same elements throughout. In the following description, for the purpose of illustration, many specific details are set forth in order to provide a more thorough understanding of one or more embodiments. However, it is apparent that in various cases, one or more embodiments may be practiced without these specific details.

[0037] Turning now to the accompanying drawings, Figure 1 A flowchart of an example high-level process 100 for refining or updating a deployed medical image processing model using a synthetic image generated based on non-image data feedback is presented in accordance with one or more embodiments of the disclosed subject matter. In one or more embodiments, process 100 can be divided into sub-processes 101, 103, and 105.

[0038] Subprocess 101 corresponds to a model development process, in which one or more ML / AI models are initially trained and developed. The one or more ML / AI models may include a task-specific model 118, which is trained to perform a specific image inference task on a set of training images (e.g., included in training data 102). The type of the specific inference task and the training image may vary. In various embodiments, the training images may include medical images, and the specific inference task may include a medical image inference task (e.g., organ segmentation, disease classification, image reconstruction / enhancement, etc.). In other embodiments, the training images may include any other type of image (e.g., in addition to medical images), and the inference task may include any type of image processing or analysis task (e.g., object recognition, object classification, image optimization, etc.).

[0039] The one or more ML / AI models may also include a synthetic image generation model 112 that is trained to generate synthetic images based on input data that includes only text input. Text to synthetic image generation is the creation of artificially generated images based only on text input that look as realistic as real images. These images can be created by various types of neural network models trained on text-image pairs, where the text associated with the corresponding image includes a text descriptor of the corresponding image. For example, in some specific implementations, the synthetic image generation model 112 can be a GAN that uses a generator-discriminator architecture to train, generate synthetic images, and rate synthetic images using a creation feedback loop that runs multiple times until the generated synthetic image can deceive the discriminator enough to be considered a real image. For another example, the synthetic image generation model 112 may include a variational autoencoder (VAE) or a vector quantized variational autoencoder (VQ-VAE).

[0040] As another example, the synthetic image generation model 112 may include a contrastive language-image pre-training (CLIP) model. CLIP is a type of neural network that can be trained in natural language on various (image, text) pairs to make predictions based on images described with respect to the most relevant text. During training, the model learns the relationship between the entire sentence and the image described by the sentence; in a sense, when the model is trained, given an input sentence, it will be able to generate the most accurate image associated with the sentence. The CLIP architecture is based on zero-shot learning, where the model attempts to predict categories that appear zero times in the training data (e.g., a model trained specifically for cats and dogs can be used to detect rabbits). In this regard, the CLIP model is well suited for application in the disclosed context, where the model is used to generate synthetic images that correspond to image variants observed in a deployment environment that are not included in or well represented in the training distribution used to train the task-specific model 118.

[0041] Subprocess 103 corresponds to a model deployment process, in which the task-specific model 118 is deployed in a deployment environment after initial training and / or updating (per subprocess 105). For example, when applied to an embodiment in which the task-specific model 118 corresponds to a medical image inference model, the deployment environment may be associated with a clinical environment such as a hospital, a medical imaging system, a medical imaging facility, etc., in which the deployment site employs the medical image inference model to generate inferences about its patient images.

[0042] Subprocess 105 corresponds to a model refinement or update process in which the mission-specific model 118 is updated based on feedback information 132 received from the deployment site. According to the disclosed technology, the model update process involves updating the mission-specific model 118 using a synthetic image (e.g., synthetic image 114) generated solely based on text input (e.g., text input 136) included in the feedback information 132 and / or generated based on the feedback information and using the synthetic image generation model 112.

[0043] In this regard, referring to subprocess 101, at 120, subprocess 101 may include training or obtaining a task-specific model 118. In some embodiments, the task-specific model 118 may include a previously trained / developed model. In other embodiments, subprocess 101 may involve initial training and development of the task-specific model 118. In any of these embodiments, the training process may involve any existing or future developed ML training process (e.g., supervised training, unsupervised training, semi-supervised training, etc.), which may vary depending on the type of task-specific model 118. In any of these embodiments, the training process involves training the task-specific model to generate image-based inference outputs based on a training data set including a set of training images (e.g., training images 104, included in training data 102). In some embodiments, (e.g., as applied to supervised and semi-supervised ML techniques) the training image 104 may include or be associated with a label providing a ground truth annotation. For example, when applied to training an organ segmentation model to segment a region of interest (ROI) in a medical image, the label may include information identifying or defining the ROI.

[0044] Additionally or alternatively, the training image 104 may be associated with labels and / or image metadata describing relevant attributes associated with the training image. Image metadata (or simply metadata) is used herein to refer to text information belonging to an image file that is associated with an image file (e.g., embedded in the image file or saved in a separate file). Image metadata may include information related to the image itself and its production. For example, when applied to medical images or medical image data, medical image metadata may be text information describing the type of medical image, acquisition process, acquisition parameters, acquisition background, captured anatomical region, content of the medical image, etc. The acquisition parameter / protocol information may vary depending on the modality of the medical imaging study performed. Some example acquisition parameter / protocol information may include (but is not limited to): contrast phase, imaging frequency, reconstruction kernel size, slice thickness, radiation dose, view of XR images, MR sequence, capture resolution, voxel size, scan prescription plane, scan area, bounding box, and scan time. Image metadata may also include textual information describing known visual properties of medical images, such as image quality, image resolution, signal noise, pixel spacing, imaging depth, image capture position / orientation (e.g., including relative scan position to other scan images generated for the medical image series), relative position / size of anatomical feature points, etc. Medical image metadata may also include information describing known properties of the patient / subject represented in the medical image (e.g., patient identifier, patient demographic information, body mass index (BMI), medical history, lesions, etc.).

[0045] At 106, the process 101 may involve creating textual descriptors for the training images 104, which may be used as textual inputs 108 associated with training the synthetic image generation model 112 at 110. In this regard, the textual descriptors may include or correspond to textual information describing relevant attributes of the corresponding training images 104, which may be used to train the synthetic image generation model 112 to generate synthetic versions of the training images 104. In some embodiments where the synthetic image generation model 112 corresponds to a CLIP neural network, the textual inputs 108 may include an aggregation of any natural and structured textual information associated with the corresponding training images (e.g., as extracted from image metadata and / or any other text files associated with the corresponding training images). Additionally or alternatively, various natural language processing (NLP) techniques may be employed at 106 to generate structured textual descriptors for the training images 104. In general, the training process for training the synthetic image generation model 112 may involve training the model to generate synthetic versions of the relevant training images 104, given corresponding textual descriptions of the training images as inputs and the training images themselves as ground truth examples.

[0046] According to the above-described embodiments of subprocess 101, the training images 104 used to train the synthetic image generation model 112 at 110 correspond to the same training images used to initially train the task-specific model 118. In other embodiments, the training images 104 used to train the synthetic image generation model 112 at 110 may include additional or alternative images. When applied to the field of medical imaging, the training images used to train the task-specific model 118 and the synthetic image generation model 112 correspond to medical images.

[0047] Referring to the subprocess 103, at 122, the task-specific model 118 (e.g., included in the runtime data 124) is executed on the runtime data. In this regard, at 122, the task-specific model 118 may be applied to the runtime image 126 in the deployment environment to generate the corresponding inference output (i.e., result 128) that the task-specific model 118 has been trained to generate. For example, when applied to a medical image inference model, the runtime image 126 may include an actual patient image (e.g., a real or native patient image) associated with a clinical deployment site (e.g., a hospital, a medical imaging system, a medical imaging facility, etc.). The result 128 will vary depending on the type of medical image inference model and the specific task that the model is trained to perform (e.g., organ segmentation, disease classification, disease quantification, disease localization, lesion detection, image quality optimization, etc.). In this regard, in some embodiments, the result 128 may include image data, such as an organ segmentation mask, an enhanced version of the input image, or another type of image data output related to the input image. Additionally or alternatively, the results 128 may include textual data, such as textual information describing or relating to disease classification, disease quantification, diagnosis, quality metrics, confidence scores associated with the model's confidence in the output, textual information describing the size, shape, location and / or another inferred characteristic of anatomical feature points of interest, and the like.

[0048] At 130, non-image feedback information (i.e., feedback information 132) associated with the execution of the task-specific model 118 is obtained. In some embodiments, the feedback information 132 may include aggregated feedback about all (or some) results 128 and corresponding input images, regardless of whether the results are attributed to positive or negative performance by the task-specific model 118. In other embodiments, the feedback information 132 may specifically relate to the portions of the results 128 that are attributed to negative performance of the task-specific model 118 (e.g., one or more errors or failure modes, low confidence scores, etc.) and the corresponding runtime input images that generated those negative results. For example, the feedback information 132 may include information that identifies or indicates any of the results 128 that are associated with an error or otherwise indicates poor performance of the task-specific model 118 (e.g., as determined via one or more objective and / or subjective performance evaluation metrics). In this regard, in various embodiments, the disclosed technology specifically relates to generating a synthetic version of the underperforming portions of the task-specific model 118 in the runtime image 126 in order to update / optimize the model using examples of these images in a deployment environment. For these embodiments, the feedback information 132 may include information identifying or indicating why and / or how such results are attributable to poor model performance (e.g., information identifying or indicating how the results are inaccurate and / or inadequate) and information describing relevant properties of the runtime images associated with the poor results, respectively. For example, when applied to medical images, the feedback information 132 may include textual information of those runtime images for which the task-specific model 118 generated poor results, such as, but not limited to: information describing relevant properties of the runtime images (e.g., image content, scanning parameters used, observed artifacts, observed lesions, presence / absence of anatomical structures), and information describing potential sources of error (e.g., failure modes) for the task-specific model 118.

[0049] The feedback information 132 can be obtained in various ways. For example, the feedback information 132 can be obtained in an automatic manner via the extraction of relevant image metadata and / or clinical report data associated with the runtime image (e.g., radiology reports on findings, lesion characterization, measurements of pathological areas, etc.). The metadata and / or clinical report data can be associated with the runtime image in the runtime data 124 or in another accessible storage structure (e.g., a network-accessible database or data repository, a local memory associated with a computing device that provides the runtime image, etc.). The feedback information 132 can also be obtained in an automatic manner via one or more AI models (e.g., confidence scoring models, image quality assessment models, result accuracy / dedicated assessment models, image feature extraction models, etc.) via automatic evaluation of the results 128 and / or runtime input images 126. The feedback information 132 may also include user-provided feedback received in association with the review results 128 and / or the corresponding runtime input images 126. In some implementations, user-provided feedback can be obtained in a semi-automatic manner via utilization of one or more feedback reporting tools that facilitate collecting user input in association with reviewing results 128 and / or runtime images 126. For example, in some embodiments applied to medical imaging, subprocess 103 can be facilitated in association with utilization of a medical imaging application at a deployment site that facilitates reviewing runtime input images 126 and results 128 and collecting relevant user feedback (e.g., in a structured format and / or in a natural language format).

[0050] Regardless of the manner in which feedback is obtained at 130, the feedback information 132 does not include the runtime image 126. In embodiments where the result 128 includes image data (e.g., an enhanced version of an input image, an enhanced version of an input image, a portion of an input image, a segmentation mask applied to an input image, an object or feature extracted from an input image, etc.), the feedback information 132 may further not include the result. In this regard, in various embodiments, the feedback information 132 may include only non-image data (e.g., text data, audio data, or another data format other than image data).

[0051] Figure 2 A table (table 200) is presented that summarizes some examples of non-image feedback (e.g., feedback 132) that may be obtained according to subprocess 103, wherein task-specific model 118 includes a medical image inference model and wherein runtime image 126 includes a medical image, according to one or more embodiments. According to these embodiments, non-image feedback may include (but is not limited to) AI-generated feedback, metadata, clinical report data, and user-provided feedback.

[0052] refer to Figure 1 and Figure 2 , the AI-generated feedback may include automatic assessment data regarding an assessment of the result 128 and / or the corresponding runtime image 126, the automatic assessment data being generated using one or more AI assessment models. For example, the one or more AI assessment models may include one or more AI models (e.g., medical image inference models) configured to generate one or more image quality parameters (e.g., in the form of textual data) regarding the quality of an input image and / or an output image (in an embodiment in which the result 128 includes an output image). The one or more AI models may also include one or more AI models (e.g., medical image inference models) configured to generate textual description data regarding the location of anatomical features, detected lesions, and / or detected artifacts in the input image and / or the output image. The one or more AI assessment models may also include one or more AI models (e.g., medical image inference models) configured to generate textual description data regarding the content (e.g., anatomical feature points or features) and / or style (e.g., contrast versus non-contrast) of the input and / or output images. Various other types of AI-generated feedback are contemplated.

[0053] In some embodiments, the task-specific model 118 may include a multi-task model configured to generate multiple different inference outputs, including a primary inference output corresponding to the AI-generated feedback and one or more secondary inference outputs. Using these embodiments, the one or more AI evaluation models and the task-specific model 118 may be merged into a single unified multi-task model (e.g., an integrated model, a stacked model, etc.). In other embodiments, the one or more AI evaluation models may include one or more separate models that can be applied separately to the runtime image 126 and / or the result 128 to generate corresponding feedback.

[0054] In this regard, in some embodiments, the one or more AI assessment models may be applied to the input and / or output images to generate corresponding AI-generated feedback information associated with executing the task-specific model 118 at 122. Additionally or alternatively, the one or more AI assessment models may be applied to the input image and / or the result in response to identifying that the input image is associated with the output (i.e., the result 128) of the task-specific model 118 having an error or defect. For example, in a specific implementation in which the one or more AI assessment models include one or more separate models from the task-specific model 118, the one or more AI assessment models may be selectively applied to generate corresponding feedback information only for those cases where the result is poor, thereby optimizing the utilization of computing resources (e.g., processing and computing resources associated with running and executing the AI ​​assessment model). For example, in some embodiments, the task-specific model 118 and / or a separate confidence assessment model (e.g., included in the AI ​​assessment model) may be configured to generate a confidence score or another evaluation metric that reflects the level of accuracy or confidence in the result 128 generated by the task-specific model 118, respectively. In some specific implementations of these embodiments, one or more additional AI evaluation models may be applied to corresponding runtime images and / or result data to generate corresponding AI-generated feedback in response to a confidence score or other evaluation metric failing to meet defined confidence score and / or evaluation metric criteria (e.g., a confidence score is below a defined threshold, etc.).

[0055] FIG. 3A to FIG. 3D Additional examples of automated AI evaluation data regarding evaluations of input images and / or results of medical image inference models that may be obtained via various types of AI evaluation models are presented. Figures 1 to 3D ,according to FIG. 3A to FIG. 3D In the example illustrated in , the task-specific model 118 corresponds to an organ segmentation model configured to segment a cardiac cavity in an input medical image (e.g., runtime image 126) corresponding to an ultrasound image (or image set / sequence) of a thoracic cavity. In this regard, the organ segmentation model is suitable for performing a primary inference task, wherein the output of the primary inference task includes segmentation information indicating a relative geometry of the cardiac cavity (e.g., relative to the input image and / or one or more other anatomical feature points depicted in the input image). The segmentation information may include a segmentation mask that can be overlaid on the input image and / or embedded within the input image. The segmentation information may additionally or alternatively include text data defining the relative geometry of the cardiac cavity.

[0056] In some implementations, the organ segmentation model may also be adapted to generate one or more secondary outputs that provide feedback (i.e., AI-generated feedback) on an evaluation of the input image and / or an evaluation of the performance of the primary inference task (e.g., confidence evaluation, evaluation of the segmentation results, etc.). Additionally or alternatively, the one or more secondary outputs (i.e., AI-generated feedback) may be generated by a separate AI evaluation model that may be applied to the input image and / or results. FIG. 3A to FIG. 3D Example types of AI-generated feedback AGs are provided, which may be generated as a secondary task (e.g., a secondary output) of a segmentation model or generated by one or more separate AI evaluation models.

[0057] refer to Figure 3A , in this example, the AI-generated feedback A provides information describing or indicating the relative size of the thorax in the input image. For example, the AI-generated feedback A may be generated by a thorax locator model (e.g., integrated with an organ segmentation model or a separate model) that generates information indicating the size of the thorax relative to a field of view (FOV) and / or information indicating whether the magnification level of the thorax meets defined acceptable criteria (e.g., 60%-80% of the FOV). The AI-generated feedback A corresponds to a type of quality metric that evaluates whether the magnification level of the thorax in the input image meets acceptable criteria for processing by the organ segmentation model in association with generating accurate results.

[0058] exist Figure 3B In the example, AI-generated feedback B corresponds to a confidence score that represents a measure of confidence in the organ segmentation model associated with generating an accurate output on the input image. The example AI-generated feedback C provides an automatic evaluation of one or more anatomical feature points depicted in the input image. These two types of secondary outputs may also correspond to types of quality metrics that reflect a measure of the quality of the input image and can be used to evaluate whether the input image meets one or more defined quality metrics (or another criterion) for generating a quality output (e.g., regarding accuracy, a specific or another quality assessment) by the organ segmentation model (e.g., whether the input image has defined anatomical features, whether the input image depicts a specific view, whether the model has sufficient confidence in the input image, etc.).

[0059] exist Figure 3CExamples of automatic feedback types (e.g., example AI feedback D, E, and F) that may be generated by a quality regressor model for automatic assessment of an input image are presented in . In this example, the quality regressor model corresponds to an AI model suitable for generating information about general image quality, such information describing artifacts (e.g., their presence and / or absence, artifact type, artifact size / location, etc.), image content, image FOV, and an overall quality score (e.g., on a scale of 0-1). The quality regression model may correspond to a model separate from the segmentation model or integrated with the segmentation model (e.g., as a secondary task of the segmentation model).

[0060] Figure 3D An example of the type of automatic feedback that may be generated by another type of quality regressor model is presented (e.g., example feedback G). In this example, the quality regressor model corresponds to an AI model suitable for generating information about task-specific image quality. In this regard, task-specific image quality refers to a customized quality assessment of the input image data that is particularly relevant to the task-specific model 118 being assessed. In this example, the task-specific image quality information includes a quality score that measures the quality of the cardiac axis view and provides information about specific relevant anatomical features of the input image data (e.g., proximity and median values ​​for the ventricles and atria).

[0061] Return to reference Figure 2 (exist Figure 1 The metadata may include textual descriptors associated with the input image data (e.g., runtime image 126) that describe acquisition parameters (e.g., frequency, depth, gain, etc.) and other relevant information about the type of image data, the content of the image data, the patient, the acquisition process, and other clinical and non-clinical information (e.g., pediatric, scan depth, depth of the object, etc.). Non-image feedback may also include textual information extracted from a clinical report associated with the input image data. Some examples of non-image data types that may be extracted from a clinical report associated with the corresponding runtime image 126 may include lesion information, measurements of pathological areas, characterization of nodules (e.g., echogenicity), and descriptions of other adjacent anatomical structures.

[0062] Non-image feedback may also include user-provided feedback. Some examples of user-provided feedback may include providing feedback on the accuracy of the results of the task-specific model 118 (e.g., a binary value and / or another non-binary measure). In some specific implementations, the user-provided feedback may also relate to the accuracy of any secondary outputs generated by the task-specific model 118 and / or a separate AI assessment model (e.g., the accuracy of the information described in the example AI-generated feedback AG). Another example of user-provided feedback may include providing information indicating one or more predefined model failure modes, which may be selected via a drop-down menu provided by a feedback tool associated with a medical imaging application that facilitates reviewing task-specific model results in association with receiving input image data, such as Figure 4 shown.

[0063] In this regard, Figure 4 An example feedback tool associated with a medical imaging application is presented in accordance with one or more embodiments of the disclosed subject matter, the example feedback tool facilitating receiving user feedback regarding failure information associated with a medical image inference model. In this example, the feedback tool provides an interactive feedback menu 404 that is accessible via the medical imaging application in association with reviewing medical image data processed and / or generated by a task-specific model 118 in a main viewing area 404. The interactive feedback menu 404 provides a selectable list of predefined failure modes (e.g., four predefined failure modes, with the "not visible at far field" option being selected in this example) that may be associated with the task-specific model 118 and selected by the reviewer to provide feedback indicating potential failure modes. The interactive feedback menu 404 also provides the option of providing natural language feedback (e.g., free text input) that indicates any other reasons that the user may consider as a source of error in the task-specific model output. It should be understood that the interactive feedback menu 404 is not intended to be construed as a "failure mode" or "failure mode" that may be associated with the task-specific model 118 and selected by the reviewer to provide feedback indicating potential failure modes. Figure 4 The example user feedback for the failure modes illustrated in is only one example of the type of user feedback that may be included in the feedback information 132, and the user feedback may include various other types of text descriptions (or speech-to-text input) describing properties of input images for which the output of the task-specific model 118 is inaccurate.

[0064] Return to reference Figure 1and subprocess 105, at 134, a textual descriptor (e.g., textual input 136) for one or more of the runtime images 126 (e.g., one or more of the runtime images associated with an error / failure in a task-specific model in a deployment environment) may be generated based on the feedback information 132 and the textual descriptor (e.g., textual input data 136). For example, in some embodiments, the process of creating the textual descriptor may involve converting the feedback information 132 (e.g., non-image feedback) into a structured format for processing by the synthetic image generation model 112. For these embodiments, the process of creating the textual descriptor at 134 may correspond to the same or similar process for creating the textual descriptor at 106. In other embodiments, the feedback information 132 can be received in an appropriate format suitable for direct input into the synthetic image generation model 112. For these embodiments, step 134 may be omitted, and the textual input 136 may correspond to the feedback information 132. At 138, the synthetic image generation model is applied to the textual input 136 (or feedback information 132) to generate a corresponding synthetic image 114. In various embodiments, the synthetic image 114 generated by the synthetic image generation model 112 based on the textual input 136 (or feedback information 132) corresponds to a synthetic version of a corresponding image of the runtime image 126 attributable to an error or failure in the task-specific model 118 (or another assessment of poor model performance) observed in the runtime environment.

[0065] Figure 5 An example process 500 for generating a synthetic medical image using textual feedback is illustrated in accordance with one or more embodiments of the disclosed subject matter. Figure 1 and Figure 5 , illustrating a process 500 associated with viewing the results of a task-specific model using a medical imaging application via an interactive visual user interface 501. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted. In this example, UI 501 provides buttons corresponding to "Accept Results" and "Reject Results", which can be selected by an entity reviewing the results and / or the input image data processed by the task-specific model 118 to generate the results, respectively. The choice of accepting the results should be applied based on the reviewer's belief that the results are accurate and / or sufficient, and the choice of rejecting the results should be applied based on the reviewer's belief that the results are inaccurate and / or insufficient (e.g., based on defined and / or subjective / objective criteria).

[0066] According to process 500, at 502, a user may review the task-specific model results and provide input to select or accept the results via UI 501. At 504, feedback information 132 may be generated and / or aggregated based on the rejection (e.g., extracted from metadata, medical reports, additional user feedback received via a medical imaging application, AI-generated feedback, etc.). At 506, a text description of one or more native input images (e.g., text input 508) on which the undesirable results are based may be generated (or in other specific implementations, the feedback information 132 may be directly processed by the synthetic image generation model 112). The text description of one or more native input images (e.g., text input 508) is applied as input to the synthetic image generation model 112 to generate one or more synthetic images 510 as output, the one or more synthetic images corresponding to the synthesized versions of the one or more native input images.

[0067] The techniques used to implement process 500 may vary. For example, in some embodiments, all feedback information 132 generated and / or received for a particular input image or set of input images and the corresponding results of the medical image inference model may be aggregated and stored in a log file (e.g., a text file), which may be associated with a storage system associated with a deployment site and / or an entity performing the synthetic image generation. In other specific implementations, the deployment site may be configured to send a log file to the entity performing the synthetic image generation in response to the creation of the synthetic image generation. Various specific implementation scenarios are contemplated.

[0068] Reference again Figure 1 In subprocess 105, at 116, the task-specific model 118 is refined (e.g., updated / retrained) using the synthetic image 114 (corresponding to the synthetic image 510, or vice versa) to generate an updated version of the task-specific model 118 that has improved performance on the image type corresponding to the synthetic image 114 relative to the previous version of the model. The process of refining / updating the task-specific model may include one or more conventional machine learning model update processes. Typically, the model update involves retraining the task-specific model 114 and adjusting one or more parameters and / or weights of the model based on processing new training data (i.e., the synthetic image 114).

[0069] In various embodiments, sub-process 103 and sub-process 105 correspond to continuous processes that are executed continuously (or regularly) over time, and sub-process 101 corresponds to a process that may be initially executed once to train and / or obtain the task-specific model 118 and the synthetic image generation model 112. In this regard, the deployment site may regularly and / or continuously provide feedback information 132 that may be used to update the task-specific model 118 based on how the task-specific model 118 underperforms in the deployment environment, and the supplier site may regularly and / or continuously employ the feedback information 132 to update and refine the task-specific model 118. The supplier site may also provide updated versions of the task-specific model 118 to the client site (i.e., the deployment site) over time. In this regard, it should be understood that over time and after the deployment of the initially trained version of the task-specific model 118, the version of the task-specific model 118 applied at the client site according to sub-process 103 may include the latest updated version generated by the supplier site.

[0070] In this regard, references herein to a vendor site generally refer to an entity that performs model development and update processes (e.g., subprocess 101 and subprocess 105), while references to a client site or deployment site refer to an entity that deploys (e.g., utilizes / executes) task-specific models 118 after training and / or updating. In various implementations, the vendor site entity and the client site entity may be different entities. Furthermore, the client site entity may correspond to multiple different entities (e.g., different clients, such as different clinical systems applied in the healthcare field). In other implementations, the vendor site entity and the client site entity may correspond to the same entity.

[0071] The operating environment of process 100 and / or subprocesses 101, 103, and 105 may vary. In this regard, it should be understood that process 100 corresponds to a computer-implemented process that can be executed by one or more computing systems, including one or more computing devices, machines, virtual machines, computer-executable components, data repositories, etc. that can be communicatively coupled to each other directly or via one or more wired or wireless communication frameworks. In various embodiments, the one or more computing systems may include a vendor computing system that can execute subprocess 101 and subprocess 105 and one or more client computing systems that execute subprocess 103. For these embodiments, the vendor computing system and the client computing system may adopt a cloud-based computing architecture, a server-client type architecture, etc., and their examples are referenced in detail below. Fig.10 to describe.

[0072] Figure 6An example non-limiting vendor system 600 is illustrated that can be associated with a vendor site that facilitates using synthetic images generated from non-image data feedback to refine or update a deployed medical image inference model in accordance with one or more embodiments of the disclosed subject matter. Figure 7 An example non-limiting client system 700 that can be associated with a client site that facilitates using synthetic images generated based on non-image data feedback to refine or update a deployed medical image inference model in accordance with one or more embodiments of the disclosed subject matter is illustrated. In one or more example embodiments, vendor system 600 corresponds to a computing system that executes subprocess 101 and / or subprocess 103, and client system 700 corresponds to another computing system that executes subprocess 105. Vendor system 600 and client system 700 can each include or correspond to one or more computing devices that include appropriate hardware and software that facilitates execution of the respective operations of processes 101, 103, and 105.

[0073] In this regard, reference Figure 1 and Figure 6 In one or more embodiments, the supplier system 600 may include (or be operatively coupled to) at least one memory 622 storing computer executable components and at least one processor (e.g., processing unit 624) executing the computer executable components stored in the at least one memory 622. Examples of the memory 622 and the processing unit 624 and other suitable computer or computing-based elements can be found in Fig. 9 Find and combine to achieve combination Figure 6 The computer executable components may include, but are not limited to, a model development component 602, a training component 604, a refinement component 606, a synthetic image generation component 608, a text descriptor component 612, and a feedback aggregation component 612.

[0074] The memory 622 may also include (e.g., store) a model repository 614, training data 616, and a feedback repository 618. Additionally or alternatively, the model repository 614, the training data 616, and / or the feedback repository 618 may be associated with one or more additional information storage structures (e.g., transient memory devices, non-transient memory devices, etc.), which may be coupled to the vendor system 600 directly or via one or more wired or wireless communication networks. The model repository may include various models (e.g., ML / AI models and / or other types of models or algorithms) employed by the vendor system in association with executing the process subprocess 101 and the subprocess 105. For example, the model repository 614 may include pre-trained and trained versions of the synthetic image generation model 112 and the task-specific model 118. In embodiments where the task-specific model 118 may be associated with one or more additional AI evaluation models suitable for generating secondary outputs that provide AI evaluation feedback, the model repository 614 may also include the one or more additional AI evaluation models. In some embodiments, the training data 616 may correspond to the training data 102. Additionally or alternatively, the training data 616 may be synthetic images 114 generated by the synthetic image generation model 112 using the text input 136, which corresponds to additional training images used by the refinement component 606 in association with updating / refining the task-specific model 118 at 116 of the subprocess 105. The feedback repository 618 may include feedback information 132 and / or text input 136 generated from the feedback information 132 (e.g., by the text descriptor component 610). In this regard, the feedback repository 618 may correspond to a database that aggregates and stores non-image feedback information 132 and / or text input 136 generated therefrom, which is aggregated over time from one or more deployment environments for one or more task-specific models.

[0075] The vendor system 600 may also include one or more input / output devices 626 to facilitate receiving user input in connection with training the synthetic image generation model 112 and / or the task-specific model 118 and refining or updating the task-specific model 118. In this regard, any information received, generated, and / or accessible by the vendor system 600 (e.g., training data 102, textual input 108, feedback information 132, textual input 136, synthetic images 114, etc.) may be presented or rendered to a user via an appropriate output device such as a display, speaker, etc. Fig. 9 Suitable examples of input / output devices 626 are depicted (eg, input device 928 and output device 936). Vendor system 600 may also include a system bus 620 that couples memory 622, processing unit 624, and input / output devices 626 to one another.

[0076] Model development component 102 can facilitate training, developing, and updating synthetic image generation model 112 and task-specific model 118 according to subprocess 101 and subprocess 105. For example, model development component 602 may include training component 604 and refinement component 606. In some embodiments, training component 604 can perform initial model training and development of synthetic image generation model 112 and / or task-specific model 118 (e.g., step 110 of subprocess 101 and / or step 120 of subprocess 101 in a specific implementation in which a task-specific model has not been previously trained / developed). As described above with reference to Figure 1 As described, the training process may vary depending on the type of corresponding model and include supervised, semi-supervised and / or unsupervised machine learning training processes.

[0077] The refinement component 606 can use the synthetic image 114 and the model update process to refine or update the task-specific model (e.g., step 116 of the subprocess 105). The model update process can correspond to a training process that still uses the synthetic image 114 as a new training data input. For example, the model update process may include one or more traditional machine learning model update processes, which involve adjusting one or more parameters and / or weights of the task-specific model 118 based on the processing of the synthetic image 114. In some embodiments, the refinement 606 can selectively control the specific synthetic image generated and used to update and refine the task-specific model 112. In this regard, in some embodiments, the feedback information 132 can cover all feedback information for the task-specific model 118 received / aggregated from the deployment site over time, including information about both positive results and negative results. In some specific implementations of these embodiments, the refinement component 606 can evaluate the feedback information (e.g., as included in the feedback repository 618) and filter the feedback information to identify and select a subset of the feedback information associated with the result and the corresponding input image of the runtime image 126 associated with one or more negative performance criteria. For example, negative performance criteria may be predefined for the task-specific model and relate to one or more attributes of the input image (e.g., content, quality, image acquisition characteristics, model confidence score, etc.). The negative performance criteria may additionally or alternatively relate to the output of the task-specific model 118, such as, but not limited to, information indicating or identifying one or more errors in the output, one or more indicated failure modes, etc. For these embodiments, refinement component 606 606 directs synthetic image generation component 608 to generate a synthetic image 114 corresponding to a subset of the runtime image for which the performance of the task-specific model 118 is associated with one or more defined negative performance criteria.

[0078] Composite image generation component 608 may perform the composite image generation process (step 138) of subprocess 105. In this regard, composite image generation component 608 may apply composite image generation model 112 to text input 136 (or in some embodiments directly to feedback information 132) to generate a corresponding composite image 114. In some embodiments, text descriptor component 610 may generate text input (e.g., text input 108 and / or text input 136) according to steps 106 and / or 134 of processes 101 and 105, respectively.

[0079] Feedback aggregation component 612 can facilitate collecting, extracting, and / or receiving non-feedback information 132 from the deployment site and aggregating the feedback in feedback repository 618. For example, in some embodiments, feedback information 132 generated at the deployment site can be stored in log files in one or more local data storage structures (e.g., local feedback repository 124) associated with the deployment site, and feedback aggregation component 612 can regularly or continuously access and retrieve the log files and aggregate them in the feedback repository.

[0080] refer to Figure 1 and Figure 7 In one or more embodiments, the client system 700 may include (or be operatively coupled to) at least one memory 716 storing computer executable components and at least one processor (e.g., processing unit 718) executing the computer executable components stored in the at least one memory 716. Examples of the memory 716 and the processing unit 718 and other suitable computer or computing-based elements can be found in Fig. 9 Find and combine to achieve combination Figure 7 The computer executable components may include, but are not limited to, a model execution component 702 , a medical imaging application component 704 , and a feedback component 706 .

[0081] The memory 716 may also include (e.g., store) a model repository 708, runtime data 710, and a local feedback repository 712. Additionally or alternatively, the model repository 708, runtime data 710, and / or local feedback repository 712 may be associated with one or more additional information storage structures (e.g., transient memory devices, non-volatile memory devices, etc.) that may be coupled to the client system 700 directly or via one or more wired or wireless communication networks. The model repository 708 may include a task-specific model 118 (or multiple models) employed by the client system 700 in association with executing the subprocess 103. For example, the model repository 708 may be one or more trained and / or updated versions of the task-specific model 118. In which the task-specific model 118 may be associated with one or more additional AI evaluation models (e.g., reference FIG. 3A to FIG. 3D In a specific implementation associated with the type of AI evaluation feedback described in the examples above, the model repository 708 may also include one or more additional AI evaluation models. The runtime data 710 may correspond to the runtime data 124 and / or the runtime image 126. The local feedback repository 712 may correspond to a local version of the feedback repository 712 associated with the client system 700. For example, the local feedback repository may store feedback information 132 generated / received by the client device in association with the task-specific model deployed according to the sub-process 103 (e.g., a local log file).

[0082] The client system 700 may also include one or more input / output devices 720 to render information (e.g., runtime images 126 and results 128) to a user and facilitate receiving user input in association with executing the task-specific model 118 and generating / obtaining feedback information 132 from the sub-process 132. Fig. 9 Suitable examples of input / output devices 720 are depicted (eg, input device 928 and output device 936). Client system 700 may also include a system bus 714 that couples memory 716, processing unit 718, and input / output devices 720 to one another.

[0083] The model execution component 702 may execute the task-specific model and / or the one or more additional AI evaluation models (e.g., where applicable) on the runtime image 126 according to step 122 of the subprocess 103. The feedback component 706 may be useful for obtaining non-image feedback information (e.g., feedback information 132) associated with the execution of the task-specific model according to step 130 of the subprocess 103. For example, in some embodiments, for each execution instance of the task-specific model 118, the feedback component 706 may extract metadata and / or clinical report data associated with the runtime image and any automatic AI evaluation information generated in association with applying the task-specific model to the runtime image, and store the aggregated text information in a log file associated with the execution instance. The feedback component 706 may also collect and associate any received user-provided feedback for the execution instance in the log file. In some embodiments, the feedback component 706 may be configured to generate only log files for execution instances associated with one or more predefined negative performance criteria (e.g., it may cover criteria related to the results of the input image and / or model).

[0084] When applied to the field of medical imaging, in one or more embodiments, the client system 700 may include or otherwise access (e.g., as a software as a service (SaaS) model, etc.) a medical imaging application component 704 that facilitates reviewing medical images, applying task-specific models (and optionally one or more additional AI evaluation models) to runtime images, reviewing results, and receiving user feedback (e.g., such as references). Figure 2 , Figure 4 Utilizing these embodiments, feedback component 706 can facilitate receiving user-provided feedback using one or more feedback tools associated with medical imaging application component 704 in association with reviewing results using a medical imaging application supported by medical imaging application component 704.

[0085] Figure 8 A block diagram of an example non-limiting computer-implemented method 800 for refining or updating a deployed medical image inference model using a synthetic image generated from non-image data feedback according to one or more embodiments of the disclosed subject matter is illustrated. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0086] At 802, method 800 includes generating, by a system including a processor (e.g., vendor system 600, etc.), a synthetic medical image based on feedback information about performance of a medical image inference model, the feedback information received in association with application of the medical image inference model to a medical image in a deployment environment, wherein the feedback information does not include image data (e.g., using the synthetic image generation model 112 via the synthetic image generation component 608). At 804, method 800 also includes updating, by the system, the medical image inference model using the synthetic image and a model update process (e.g., via the refinement component 606). According to one or more embodiments of method 800, the feedback information includes non-image data about properties of at least some of the medical images, and wherein the feedback information does not include the medical image. The feedback information may also include non-image data related to the results of the medical image inference model. In this regard, the feedback information includes text data, and wherein generating includes employing a synthetic image generation model that generates the synthetic image based on the text data and without input data including image data or additional image data. In some embodiments of method 800, the medical image inference model includes a first machine learning model trained on a training dataset including a first medical image, and wherein the synthetic image generation model includes a second machine learning model trained on training text description data associated with the first medical image.

[0087] Fig. 9 A block diagram of an example non-limiting computer-implemented method 900 for refining or updating a deployed medical image inference model using a synthetic image generated from non-image data feedback according to one or more embodiments of the disclosed subject matter is illustrated. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted.

[0088] At 902, method 900 includes receiving, by a system including a processor (e.g., vendor system 600, etc.), non-image feedback information (e.g., feedback information 132) describing properties of a medical image processed by a medical image inference model (e.g., task-specific model 118) in a deployment environment. At 904, method 900 also includes generating, by the system, a synthesized version of the medical image (e.g., synthesized image 114, synthesized image 510, etc.) using a synthetic image generation model (e.g., synthetic image generation model 112) and the non-image feedback information (and / or text input 136 generated based on the non-image feedback information) as input to the synthetic image generation model (e.g., via synthetic image generation component 608). At 906, method 900 also includes updating, by the system, the medical image inference model using the synthesized version and a model update process (e.g., via refinement component 606). In one or more embodiments, the medical image inference model includes a first machine learning model trained on a training dataset including a first medical image, and the synthetic image generation model includes a second machine learning model trained on training text description data associated with the first medical image, wherein the synthetic image generation model generates a synthetic image based solely on the text input.

[0089] Exemplary Operating Environment

[0090] One or more embodiments may be systems, methods and / or computer program products at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0091] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following items: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or a raised structure in a groove on which instructions are recorded) and any suitable combination of the foregoing items. As used herein, a computer readable storage medium should not be understood to be a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0092] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the corresponding computing / processing device.

[0093] The computer-readable program instructions for performing the operation of the present invention can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and process programming languages ​​(such as "C" programming language or similar programming languages). Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In some embodiments, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions, so as to perform aspects of the present invention.

[0094] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It is understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0095] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to function in a specific manner, so that the computer-readable storage medium with instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0096] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0097] The flow chart and block diagram in the accompanying drawings illustrate the possible concrete implementation architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in the flow chart or block diagram can represent a module, a fragment or a part of an instruction, which includes one or more executable instructions for implementing a specified logical function.In some alternative implementations, the function indicated in the frame may not occur in the order indicated in the figure.For example, depending on the function involved, two frames shown in succession can actually be performed substantially simultaneously, or these frames may be performed in reverse order sometimes.It will also be noted that each frame of the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be implemented by a system based on dedicated hardware that performs a specified function or action or implements a combination of dedicated hardware and computer instructions.

[0098] Combination Fig. 9 , the systems and processes described below may be embodied in hardware, such as a single integrated circuit (IC) chip, multiple ICs, application specific integrated circuits (ASICs), etc. In addition, the order in which some or all of the program blocks appear in each program should not be considered limiting. On the contrary, it should be understood that some program blocks can be executed in various orders, not all of which can be explicitly shown in this document.

[0099] refer to Fig.10 , an example environment 1000 for implementing various aspects of the claimed subject matter includes a computer 1002. The computer 1002 includes a processing unit 1004, a system memory 1006, a codec 1035, and a system bus 1008. The system bus 1008 couples system components including, but not limited to, the system memory 1006 to the processing unit 1004. The processing unit 1004 can be any of a variety of available processors. Dual microprocessors and other multi-processor architectures can also be used as the processing unit 1004.

[0100] The system bus 1008 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, or a native bus using any of a variety of available bus architectures including, but not limited to, Industry Standard Architecture (ISA), Micro Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Native Bus (VLB), Peripheral Component Interconnect (PCI), card bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association (PCMCIA), FireWire (IEEE 13104), and Small Computer System Interface (SCSI).

[0101] In various embodiments, the system memory 1006 includes a volatile memory 1010 and a non-volatile memory 1012, which may adopt one or more memory architectures in the disclosed memory architecture. A basic input / output system (BIOS) (including basic routines such as transferring information between elements within the computer 1002 during startup) is stored in the non-volatile memory 1012. In addition, according to the present invention, the codec 1035 may include at least one of an encoder or a decoder, wherein at least one of the encoder or the decoder may be composed of hardware, software, or a combination of hardware and software. Although the codec 1035 is depicted as a separate component, the codec 1035 may be included in the non-volatile memory 1012. By way of illustration and not limitation, the non-volatile memory 1012 may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a 3D flash memory, or a resistive memory, such as a resistive random access memory (RRAM). In at least some embodiments, the non-volatile memory 1012 may employ one or more of the disclosed memory devices. In addition, the non-volatile memory 1012 may be a computer memory (e.g., physically integrated with the computer 1002 or its motherboard) or a removable memory. Examples of suitable removable memories that may be used to implement the disclosed embodiments may include a secure digital (SD) card, a compact flash (CF) card, a universal serial bus (USB) memory stick, and the like. The volatile memory 1010 includes a random access memory (RAM) that acts as an external cache memory, and one or more disclosed memory devices may also be employed in various embodiments. By way of illustration and not limitation, RAM may be provided in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), and enhanced SDRAM (ESDRAM), and the like.

[0102] The computer 1002 may also include removable / non-removable, volatile / non-volatile computer storage media. Fig.10For example, disk storage device 1014 is illustrated. Disk storage device 1014 includes, but is not limited to, devices such as disk drives, solid state disks (SSDs), flash memory cards, or memory sticks. In addition, disk storage device 1014 may include a separate storage medium or a storage medium combined with other storage media, including but not limited to an optical drive, such as a CD-ROM device (CD-ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), or a digital universal disk ROM drive (DVD-ROM). In order to facilitate connecting disk storage device 1014 to system bus 1008, a removable or non-removable interface, such as interface 1016, is generally used. It should be understood that disk storage device 1014 can store information related to a user. Such information can be stored at a server or provided to an application running on a server or user device. In one embodiment, the user can be notified (for example, by output device 1036) of the type of information stored in disk storage device 1014 or transmitted to a server or application. The user may be provided an opportunity to opt-in or opt-out of the collection or sharing of such information by the server or application (eg, via input from input device 1028).

[0103] It should be understood that Fig.10 Software is described that acts as an intermediary between a user and the basic computer resources described in an appropriate operating environment 1000. Such software includes an operating system 1010. The operating system 1010, which may be stored on a disk storage device 1014, is used to control and allocate the resources of the computer 1002. Application programs 1020 utilize the management of resources by the operating system 1010 through program modules 1024, and program data 1026 stored in the system memory 1006 or on the disk storage device 1014, such as a startup / shutdown transaction table. It should be appreciated that the claimed subject matter may be implemented with various operating systems or combinations of operating systems.

[0104] The user enters commands or information into the computer 1002 through input devices 1028. Input devices 1028 include, but are not limited to, pointing devices such as a mouse, trackball, stylus, touchpad, keyboard, microphone, joystick, game pad, satellite dish, scanner, TV tuner card, digital camera, digital video camera, webcam, etc. These and other input devices are connected to the processing unit 1004 through the system bus 1008 via interface ports 1030. Interface ports 1030 include, for example, serial ports, parallel ports, game ports, and universal serial bus (USB). Output devices 1036 use some of the same types of ports as input devices 1028. Thus, for example, a USB port can be used to provide input to the computer 1002 and output information from the computer 1002 to output devices 1036. Output adapters 1034 are provided to illustrate that there are some output devices 1036 such as monitors, speakers, and printers, as well as other output devices 1036 that require special adapters. By way of illustration and not limitation, output adapters 1034 include video and sound cards that provide a means of connection between output devices 1036 and system bus 1008. It should be noted that other devices or systems of devices provide input and output capabilities, such as remote computer 1038.

[0105] The computer 1002 can work in a networked environment using logical connections to one or more remote computers, such as remote computer 1038. The remote computer 1038 can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based device, a peer device, a smart phone, a tablet computer, or other network node, and typically includes many of the elements described relative to the computer 1002. For the purpose of simplicity, only a memory storage device 1040 is shown for the remote computer 1038. The remote computer 1038 is logically connected to the computer 1002 through a network interface 1042, and then connected via a communication connection 1044. The network interface 1042 covers a wired communication network or a wireless communication network, such as a local area network (LAN) and a wide area network (WAN) and a cellular network. LAN technologies include fiber distributed data interface (FDDI), copper distributed data interface (CDDI), Ethernet, token ring, etc. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks such as integrated services digital networks (ISDN) and variations thereon, packet switching networks, and digital subscriber lines (DSL).

[0106] The communication connection 1044 refers to the hardware / software used to connect the network interface 1042 to the bus 1008. Although the communication connection 1044 is shown internal to the computer 1002 for clarity of illustration, the communication connection can also be external to the computer 1002. For exemplary purposes only, the hardware / software required to connect to the network interface 1042 includes internal technology and external technology, such as modems, including conventional telephone grade modems, cable modems and DSL modems, ISDN adapters, as well as wired and wireless Ethernet cards, hubs and routers.

[0107] Fig.11 1 is a schematic block diagram of a sample computing environment 1100 with which the subject matter of the present disclosure can interact. System 1100 includes one or more clients 1102. Client 1102 (e.g., corresponding to client system 700 in some embodiments) can be hardware and / or software (e.g., threads, processes, computing devices). System 1100 also includes one or more servers 1104 (e.g., corresponding to supplier system 600 in some embodiments). Therefore, in addition to other models, system 1100 can correspond to a two-tier client server model or a multi-tier model (e.g., client, middle-tier server, data server). Server 1104 can also be hardware and / or software (e.g., threads, processes, computing devices). For example, server 1104 can accommodate threads to perform conversions by adopting the present disclosure. One possible communication between client 1102 and server 1104 can be in the form of data packets transmitted between two or more computer processes (e.g., including feedback information 132).

[0108] The system 1100 includes a communication framework 1106 that can be used to facilitate communication between the client 1102 and the server 1104. The client 1102 is operatively connected to one or more client data repositories 1108, which can be used to store information local to the client 1102. Similarly, the server 1104 is operatively connected to one or more server data repositories 1112, which can be used to store information local to the server 1104.

[0109] It should be noted that various aspects or features of the present disclosure can be utilized in substantially any wireless telecommunication or radio technology, for example, Wi-Fi; Bluetooth; Worldwide Interoperability for Microwave Access (WiMAX); Enhanced General Packet Radio Service (Enhanced GPRS); 3rd Generation Partnership Project (3GPP) Long Term Evolution (LTE); 3rd Generation Partnership Project 2 (3GPP2) Ultra Mobile Broadband (UMB); 3GPP Universal Mobile Telecommunications System (UMTS); High Speed ​​Packet Access (HSPA); High Speed ​​Downlink Packet Access (HSDPA); High Speed ​​Uplink Packet Access (HSUPA); GSM (Global System for Mobile Communications) EDGE (Enhanced Data Rates for GSM Evolution) Radio Access Network (GERAN); UMTS Terrestrial Radio Access Network (UTRAN); LTE Advanced (LTE-A); etc. In addition, some or all aspects described herein can be utilized in traditional telecommunication technologies (e.g., GSM). In addition, mobile as well as non-mobile networks (e.g., the Internet, data service networks such as Internet Protocol Television (IPTV), etc.) can utilize aspects or features described herein.

[0110] Although the subject matter has been described above in the general context of computer executable instructions of a computer program running on one and / or multiple computers, it will be appreciated by those skilled in the art that the present disclosure may also or may be implemented in combination with other program modules. In general, program modules include routines, programs, components, data structures, etc. that perform specific tasks and / or implement specific abstract data types. In addition, it will be appreciated by those skilled in the art that the method of the present invention may be practiced with other computer system configurations, including single-processor or multi-processor computer systems, small computing devices, large computers, and personal computers, handheld computing devices (e.g., PDAs, phones), microprocessor-based or programmable consumer or industrial electronics, etc. The illustrated aspects may also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices linked by a communication network. However, some (if not all) aspects of the present disclosure may be practiced on stand-alone computers. In a distributed computing environment, program modules may be located in local and remote memory storage devices.

[0111] As used in this application, the terms "component", "system", "platform", "interface", etc. may refer to and / or may include computer-related entities or entities associated with an operating machine having one or more specific functionalities. The entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a server and a server may be a component. One or more components may reside within a process and / or a thread of execution, and a component may be located on one computer and / or distributed between two or more computers.

[0112] As another example, the corresponding components can be executed according to various computer-readable media having various data structures stored thereon. The components can communicate via local and / or remote processes, such as according to signals having one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, and / or interacts with other systems across a network (such as the Internet) via a signal). As another example, a component can be a device having a specific function provided by a mechanical part operated by an electrical or electronic circuit system, which is operated by a software or firmware application executed by a processor. In such cases, the processor can be inside or outside the device and can execute at least a portion of the software or firmware application. As another example, a component can be a device that provides specific functionality through electronic components rather than mechanical parts, wherein the electronic component may include a processor or other devices for executing software or firmware that at least partially imparts functionality to the electronic component. On the one hand, the component can simulate the electronic component via a virtual machine, for example, in a cloud computing system.

[0113] In addition, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless specified otherwise or clear from the context, "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing cases. Furthermore, the articles "a" and "an" used in the specification and the drawings should generally be construed to mean "one or more" unless specified otherwise or clear from the context to be directed to the singular.

[0114] As used herein, the terms "example" and / or "exemplary" are used to mean serving as an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited to such examples. In addition, any aspect or design described herein as "example" and / or "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs, nor is it intended to exclude equivalent exemplary structures and techniques known to those of ordinary skill in the art.

[0115] Standard programming or engineering techniques can be used to implement various aspects or features described herein as methods, devices, systems, or products. In addition, various aspects or features disclosed in the present disclosure can be implemented by implementing at least one or more of the program modules of the methods disclosed herein, and the program modules are stored in a memory and executed at least by a processor. Other combinations of hardware and software or hardware and firmware can implement or implement the aspects described herein, including the disclosed methods. As used herein, the term "product" can cover a computer program that can be accessed from any computer-readable device, carrier, or storage medium. For example, a computer-readable storage medium can include, but is not limited to, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe ...), an optical disk (e.g., a compact disk (CD), a digital versatile disk (DVD), a blue optical disk (BD) ...), a smart card, and a flash memory device (e.g., a card, a stick, a key drive ...), etc.

[0116] As used in this specification, the term "processor" may refer to substantially any computational processing unit or device, including but not limited to a single-core processor; a single processor with software multithreaded execution capability; a multi-core processor; a multi-core processor with software multithreaded execution capability; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. In addition, a processor may refer to an integrated circuit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. In addition, the processor may utilize nanoscale architectures (such as, but not limited to, transistors, switches, and gates based on molecules and quantum dots) in order to optimize space usage or enhance the performance of user equipment. The processor may also be implemented as a combination of computational processing units.

[0117] In this disclosure, terms such as "repository," "storage," "data repository," "data storage," "data storage," "database," and substantially any other information storage component related to the operation and functionality of the component are used to refer to a "memory component," an entity embodied in a "memory," or a component that includes a memory. It should be understood that the memory and / or memory components described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory.

[0118] By way of illustration and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). For example, volatile memory may include RAM that may act as external cache memory. By way of illustration and not limitation, RAM may be provided in a variety of forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). In addition, the disclosed memory components of the systems or methods herein are intended to include, but are not limited to, including these and any other suitable types of memory.

[0119] It should be recognized and appreciated that components described with respect to a particular system or method may include the same or similar functionality as corresponding components described with respect to other systems or methods disclosed herein (eg, separately named components or similarly named components).

[0120] What has been described above includes examples of systems and methods that provide advantages of the present disclosure. Of course, it is not possible to describe every conceivable combination of components or methods for the purpose of describing the present disclosure, but one of ordinary skill in the art will recognize that many additional combinations and permutations of the present disclosure are possible. In addition, to the extent that the terms "including," "having," "having," and the like are used in the detailed description, claims, appendices, and drawings, such terms are intended to be inclusive in a manner similar to the term "including," as interpreted when "including" is used as a transitional word in a claim.

Claims

1. A system, comprising: a memory storing computer executable components; and a processor that executes the computer executable components stored in the memory, wherein the computer executable components include: a synthetic image generation component that generates a synthetic medical image based on feedback information associated with performance of a medical image inference model, the feedback information being received in association with application of the medical image inference model to a medical image in a deployment environment, wherein the feedback information does not include image data; and A refinement component updates the medical image inference model using the synthesized image and a model update process.

2. The system of claim 1, wherein the feedback information comprises non-image data regarding properties of at least some of the medical images, and wherein the feedback information does not include the medical images.

3. The system of claim 2, wherein at least some of the medical images correspond to a subset of the medical images for which the performance of the medical image inference model is associated with one or more negative performance criteria.

4. The system of claim 1, wherein the feedback information comprises non-image data related to the results of the medical image inference model.

5. A system according to claim 1, wherein the feedback information includes text data, and wherein the synthetic image generation component uses a synthetic image generation model machine learning model to generate the synthetic medical image, and the synthetic image generation model machine learning model generates the synthetic image based on the text data and without input data including the image data or additional image data.

6. The system of claim 1, wherein the model updating process comprises a machine learning process.

7. A system according to claim 1, wherein the medical image inference model includes a first machine learning model trained on a training data set including a first medical image, wherein the synthetic image generation component uses a second machine learning model to generate the synthetic medical image, the second machine learning model generates the synthetic medical image based on text description data and without input data including the image data or additional image data, and wherein the second machine learning model is trained on training text description data associated with the first medical image.

8. The system of claim 7, wherein the computer executable components further comprise: A training component, wherein the training component trains the first machine learning model and the second machine learning model.

9. A system according to claim 1, wherein the feedback information includes automatic assessment data regarding an assessment of at least one of: the performance, at least some of the medical images, or results of the medical image inference model, the automatic assessment data being generated using one or more artificial intelligence assessment models.

10. The system of claim 1, wherein the feedback information includes metadata associated with at least some of the medical images.

11. The system of claim 1, wherein the feedback information comprises clinical report data associated with at least some of the medical images.

12. The system of claim 1, wherein the feedback information comprises user-provided feedback regarding an assessment of at least one of: the performance, at least some of the medical images, or results of the medical image inference model.

13. The system of claim 12, wherein the computer executable components further comprise: A feedback component facilitates receiving feedback provided by the user in association with reviewing the results using the medical imaging application using one or more feedback tools associated with the medical imaging application.

14. A method comprising: generating, by a system including a processor, a synthetic medical image based on feedback information associated with performance of a medical image inference model, the feedback information received in association with application of the medical image inference model to a medical image in a deployment environment, wherein the feedback information does not include image data; and The medical image inference model is updated by the system using the synthesized image and a model update process.

15. A non-transitory machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations comprising: generating a synthetic medical image based on feedback information associated with performance of a medical image inference model, the feedback information received in association with application of the medical image inference model to a medical image in a deployment environment, wherein the feedback information does not include image data; and The medical image inference model is updated using the synthesized image and a model update process.

Images