Image analysis and insight generation

Abstract

An approach is provided for analyzing an input image and providing one or more outputs related to the input image. According to an example embodiment, the input image may be received and analyzed using a trained machine learning model to generate an inference related to the image. Based at least in part on the generated inference, one or more reports related to the inference may be generated and provided for presentation on a user device. A user may interact with the reports in a conversational manner with the computer system to generate additional reports or insights related to the input image.

Description

[Background technology]

[0001] Medical professionals may rely on imaging techniques to make more informed diagnoses of patients. Often, different medical professionals may have conflicting opinions on what a diagnosis may be. Additionally, human error may cause medical professionals to inadvertently overlook artifacts in images, resulting in inaccurate reports. Manually generated reports may also take longer to generate than may be required in an emergency situation. Additionally, conventional methods may utilize templates or rule-based algorithms, which may require manual input and may exclude artifacts not specified in the template or algorithm. Additionally, conventional methods may not allow a user to communicate with the system in a conversational manner to aid in decision making. Thus, there is a need for improved radiology techniques and systems to enable medical professionals to improve accuracy and efficiency over existing techniques. Summary of the Invention [Means for solving the problem]

[0002] In the following description, various embodiments are described. For purposes of explanation, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will also be apparent to one skilled in the art that the embodiments may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0003] According to an exemplary embodiment, medical imaging may include, but is not limited to, techniques related to X-rays, computed tomography (CT) scans, magnetic resonance imaging (MRI), nuclear medicine imaging positron emission tomography (PET), and ultrasound. Medical imaging may also refer to any imaging-based technique used to view humans from the inside, such as techniques used in colonoscopy and endoscopy. Such medical imaging may provide information about physical issues related to a patient that are not easily visible to the human eye. However, given demanding schedules and / or limited personnel, it may take several hours before a medical professional has time to analyze the images and generate a report related to the images. In such cases, the patient's condition may change rapidly while waiting for the medical professional to read the results. For example, a medical professional working in gastroenterology may take several hours to provide therapeutic guidance to a medical professional working in the emergency room. The patient's condition may worsen all the while the emergency room specialist is waiting for therapeutic guidance. Furthermore, one expert's opinion may differ from another's opinion, or an expert may inadvertently overlook important artifacts that may be present in the image. As a result, manually generated medical reports may not capture important details with a high degree of accuracy or consistency.

[0004] According to an example embodiment, an input image, e.g., an image acquired by using medical imaging technology, may be received. A request for at least one insight related to the input image may also be received. The input image may be analyzed to detect one or more anomalies present in the input image. At least one neural network may be utilized to infer information related to the input image. A plurality of reports related to the input image may be generated based at least in part on the inferred information, each of the plurality of reports having at least one associated metric.

[0005] According to another example embodiment, an input image may be received. Using a machine learning model, at least one inference related to the image may be generated. A first output may be provided for display based at least in part on the at least one inference. At least one interaction with the first output may be received. In response to the received at least one interaction, at least one second output may be provided for display.

[0006] According to yet another exemplary embodiment, a machine learning model may be trained with a set of training data, the set of training data including values ​​of individual tokens and one or more groups of values ​​of individual tokens. An input image may be received and analyzed using the trained machine learning model. An output related to the analyzed input image may be received or generated. At least a first subset of the received data in the output may be sampled, and a second subset of the data in the output is excluded from sampling. A report related to the input image may be generated based at least in part on the first subset of sampled data. [Brief description of the drawings]

[0007] Various embodiments according to the present disclosure will now be described with reference to the drawings.

[0008] [Figure 1] 1 illustrates example interfaces that can be used to implement aspects of various embodiments. [Diagram 2] 1 illustrates another example interface that can be used to implement aspects of various embodiments. [Diagram 3] 1 illustrates another example system that can be used to implement one or more aspects of various embodiments. [Figure 4] 1 illustrates an exemplary methodology that may be utilized in accordance with various embodiments. [Diagram 5] 1 illustrates another exemplary method that may be utilized in accordance with various embodiments. [Figure 6] 1 illustrates another exemplary method that may be utilized in accordance with various embodiments. [Figure 7] 1 illustrates a system that can be used to implement at least some or all of the techniques described herein, according to some embodiments. [Figure 8] 1 illustrates components of another exemplary environment in which aspects of various embodiments may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] FIG. 1 illustrates an exemplary interface 100 that can be used to implement aspects of various embodiments. As illustrated in FIG. 1, a medical image or set of medical images 110 can be provided as an input. As described above, according to an exemplary embodiment, medical images can include, but are not limited to, techniques related to x-rays, computed tomography (CT) scans, computed tomography (CAT) scans, magnetic resonance imaging (MRI), nuclear medicine imaging positron emission tomography (PET), and ultrasound. Medical imaging can also refer to any imaging-based technique used to view humans from the inside, such as techniques used in colonoscopy and endoscopy. Such medical imaging can provide information about physical problems associated with a patient that are not easily visible to the human eye.

[0010] According to an exemplary embodiment, the medical image(s) 110 may be provided in addition to a request, such as whether the expert desires a detailed report related to the image(s) or whether the expert has a high degree of accuracy in predicting a diagnosis. Artificial intelligence may be utilized, according to one or more embodiments, to assist in the analysis of the images and provide an interpretation of the images, in addition to confirming the expert's opinion, among other such options. As described and / or referenced herein, artificial intelligence may include, but is not limited to, the use of machine learning, including unsupervised machine learning, semi-supervised or enhanced machine learning, or supervised machine learning, as may be applicable. Machine learning as described and / or referenced herein may include the use of neural networks and deep learning.

[0011] As shown in FIG. 1, the system may be able to automatically identify that the input image(s) represent a chest exam, as indicated by display box 120. If the image is not very clear, or if the system cannot determine, at least with a threshold level of confidence, what the exam pertains to, the system may request additional input to clarify the information. According to an exemplary embodiment, machine learning may be utilized to analyze the input image, such as a scan acquired by medical imaging, to detect any abnormalities that may be present in the scan. In some exemplary embodiments, the input image may be compared to training data including medical history data, image data, and text data, among other such options. Medical history data may include various types of medical reports, including scans that are considered "healthy" or have associated abnormalities, and associated medical images. In some embodiments, healthy scans may be excluded from the set of training data to minimize the amount of stored data required to train the model. In such a case, the model may still operate with high accuracy while improving the processing speed of the computing device. The training data may also include image data corresponding to items, such as items that may contribute to the detection of an abnormality, at least in some exemplary embodiments. For example, a child may have swallowed a button or a coin. Images of the button or coin may be utilized as training data to help provide a potential diagnosis of the button or coin detected in the child's input image. In some embodiments, since imaging may not provide clear information regarding the features present in an anomalous object, the outline of the item may be compared to the training data to determine what the potential source of the anomaly may be.

[0012] Based on the analysis, a report may be generated according to an exemplary embodiment, as shown at 130. For example, if the initial request was to check a diagnosis, the report may indicate whether that diagnosis was found during the analysis. Alternatively, if the initial request was to get a summary of potential abnormalities or to generate a diagnosis, a report containing information related to the request may be provided in response to the request. According to an exemplary embodiment, the user may also prompt the system to provide a description or explanation regarding the identified portion of the input image. In this way, the user essentially has a conversational relationship with the computing system, which may better inform the user's decisions regarding the image. In this example of FIG. 1, the system was requested to get a report or summary of any findings based on the input image(s) 110. As shown, the system could generate findings based on clinical indications. Clinical indications, such as fever, may be provided by the expert, so that the system would have additional contextual information when generating the report. In this way, the compiled findings may be reconciled to provide a more accurate representation of any findings and other considerations that should be of concern to the expert. In addition to findings, the system may be able to provide a summary impression indicating whether immediate action is required or whether the patient's condition is stable. In at least some embodiments, the report may be generated utilizing an encoder / decoder framework, such as via a transducer model, including but not limited to a Bidirectional and Autoregressive (BART) model, a Bidirectional Encoder Representation from Transducers (BERT) model, or a large transducer-based language model, such as a Generative Pre-Trained Transducer (GPT) model.

[0013] As described herein, the training data may include output from multiple types of models, including but not limited to trained image models, trained text-based models, and models trained on medical history data such as multiple medical reports. The training data for the reports may describe medical problems that may be difficult to diagnose initially. For example, reports corresponding to more complex assessments may be utilized to help the model understand such complex cases. For example, a patient may have a small pneumothorax in one portion of the image, but the model may be able to identify a mediastinal shift in the trachea that may have been initially missed by the ordering medical professional. In this case, the professional may have ordered the image to confirm the pneumothorax, and thus may only focus on that portion of the image. Thus, the mediastinal shift may have been missed by the professional because he or she did not realize that there was a problem with the trachea that needed attention. Certainly, other medical personnel may spot the shift, but if the original professional did not catch the problem, there would be no reason to seek therapeutic guidance regarding the trachea. Thus, by utilizing a trained model as described herein, a more comprehensive assessment of the patient's imaging may be provided. After the system identifies a midline shift, the user can ask the system follow-up questions related to the shift, such as to generate additional reports related to that particular portion of the image, or by selecting other reports that may provide alternative explanations for the problem. In this way, the user can interact with the computing system to better understand the problem, as they would with other experts in the medical field. Individual reports can be provided with a certainty value or other metric that indicates the certainty of the confidence in the accuracy of the report. The user can look at the certainty value of a given report to determine how much weight to give that report when generating an overall diagnosis.Other metrics, according to an example embodiment, may include the average confidence score across all token values, the maximum confidence score for a single report, the minimum confidence score for a single report, the standard deviation of the confidence score(s), the number of highlight foci in a report / image pairing, among other such metrics.

[0014] Traditionally, a model may be optimized to reduce loss as much as possible. For example, a model may be optimized to not overfit in order to reduce loss. However, in certain contexts, such as the medical field, it may be beneficial to consider additional data points that may be excluded due to lack of certainty. For example, it may be beneficial to intentionally overfit a model to make the model more confident in its predictions so that the overall clinical performance is improved. In this example, the model may only need to be trained on passages of memorized phrases, rather than specific words or syllables, allowing the model to spend more resources on analyzing clinical details related to the request, instead of determining how to present a set of words that summarize the analysis.

[0015] According to an exemplary embodiment, when a set of reports is being generated for a given input image, a histogram may be generated and brought with certainty of accuracy for presentation. A histogram may be a graph with a series of peaks, where each peak may be associated with a possible frequency distribution. Conventional systems may focus only on a single peak that corresponds to the most accurate output, which is most frequently correct. However, in certain instances, such as in medical practice, it may be beneficial to analyze smaller peaks when there are additional or alternative diagnoses to consider. For example, in the medical field, it is common for various medical experts to disagree on the findings of an image. According to one or more exemplary embodiments, a user may click on one or more of the peaks to see other possible diagnoses, which are presented in the report or provided in the form of a label for the image. In this way, a user may have a representation of what a number of experts would predict for the image (e.g., 100 experts for 100 generated reports), providing the user with a more comprehensive overview of the predictions. If there is only one peak in the histogram, the user may be confident in his diagnosis. However, if there are multiple peaks (e.g., two or more), the user may determine that additional information is needed to settle on a diagnosis, or the user may suggest that each peak is a representative surrogate for what may be an underlying condition. This example is not intended to be limiting, and other forms of indication may be provided.

[0016] FIG. 2 illustrates another exemplary interface 200 that can be used to implement aspects of various embodiments. As shown in this example, one or more medical image(s) 210 can be provided to a computing system for analysis. The type of procedure, technique, medical history, and potential indications, among other such information, can be provided as contextual information for the computing system to consider. In accordance with one or more techniques described herein, the computing system can utilize artificial intelligence to analyze the input image(s) 210 and generate a report 230 that includes the contextual information as well as conclusions and instructions determined by the computing system. Alternatively or additionally, the user may be prompted by the computing system to view specific details as many times as desired, so that the computing system can provide multiple results that may vary with each iteration. In doing so, the user and the computing system have established a conversational approach to assist the user in making an overall diagnosis of what the images show. As shown in FIG. 2, the computing system could determine that "there is a radiopaque zipper protruding above the upper left part of the abdomen." In other words, the system detected a foreign body in the body and included the finding in the report 230 to alert the reader of the report. Based on this information, the reader may then take the necessary steps, such as ordering a scan of that particular area of ​​the body to ensure that the patient is not facing other problems besides the catheter. The user may further manually highlight or otherwise select portions of the report to formulate the computing system's next question as an indicator to the computing system that the next question is related to the highlighted portion of the report. Similarly, the user may be able to highlight or otherwise select portions of the input image to indicate to the computing system that the next question is related to a portion of the input image.According to an alternative embodiment, the user may be able to interact with a portion of the input image, and the system may automatically highlight a corresponding portion of the generated report that may be related to the interacted portion of the input image. The user may also request the system to generate a report related to a particular condition. For example, the report in FIG. 2 shows a radiopaque zipper, but the user may ask whether the system should check for any infections that may be present in the patient. According to an exemplary embodiment, the system may also generate recommendations on how to proceed when an abnormality is detected. Recommendations may include suggestions to order additional imaging for a specified area, or to request therapy guidance from a specific provider regarding a particular problem detected.

[0017] As shown in the exemplary system 300 of FIG. 3, a user may be enabled to use a client device 302 to submit a request including a query related to an input image 306 received in an environment over at least one network 304. The request may be received by a user, such as a medical professional, submitting the query via a provider environment 308. The query may be submitted in any suitable manner (e.g., a text query, a voice request, an image-based query, etc.). In this exemplary embodiment, the query may include a request to confirm a diagnosis associated with the input image or a request for a report including any abnormalities that may be present in the input image, among other such queries. Although a portable computing device (e.g., an e-book reader, a smartphone, or a tablet computer) is shown as the client device 302, it should be understood that any electronic device capable of receiving, determining, and / or processing inputs may be used in accordance with various embodiments described herein, where the device may include, for example, desktop computers, laptops, personal data assistants, video game consoles, television set-top boxes, wearable computers (e.g., smart watches and glasses), and portable media players, among others.

[0018] The at least one network 304 may include any suitable network, such as the Internet, an intranet, a local area network (LAN), a cellular network, a Wi-Fi network, etc. The request may be transmitted to an appropriate provider environment 308, which may provide one or more services, systems, or applications for processing such a request. A provider may be any source of digital or electronic content, such as may include a website provider, a hospital with electronic records, a content distribution company, etc.

[0019] In this example, the request is received by a network interface layer 310 of the provider environment 308. The network interface layer may include any suitable components known or used to receive requests over a network, such as may include one or more application programming interfaces (APIs) or other such interfaces for receiving such requests. The network interface layer 310 may be owned and operated by the provider or may be utilized by the provider as part of a shared resource or “cloud” offering. The network interface layer may receive and analyze the request from the client device 302 and direct at least a portion of the information in the request to an appropriate system or service, such as a server 312 (e.g., a web server or an application server), among other such options. In the case of a web page, for example, at least one server 312 may be used to generate code and transmit content for rendering the requested web page. If processing is to be performed, such as generating a result in response to the request, performing an operation on user input, validating information for the request, etc., the information may also be directed to at least one other server for processing, such as, for example, a request engine 316. A server or other component of the environment may access one or more data stores, such as an image data store 314, that contain image data as a form of training data that can be utilized by the request engine 316. In some exemplary embodiments, the image data store 314 may further include corresponding reports related to the image data that can be utilized as training data.

[0020] The request engine 316 can receive a request from a server and can determine one or more responses to the request. In at least some embodiments, the request engine 316 can consider information provided by the analysis component 320 and provide an analysis to the response generation component 322. The analysis component 320 can utilize one or more artificial intelligence techniques, including machine learning, to analyze the input image and determine any anomalies that may be present in the input image. For example, training data can be utilized to predict and / or identify any anomalies that may be present in the input image. Additionally, the analysis component can consider data, such as data present in a manually generated report, to enable it to determine whether the report is accurate given the input image. The response generation component 322 can consider the results of the analysis component to generate a report or instructions to provide to the request engine 316. For example, the response generation component 322 can utilize a language model to generate a report or provide an indication as to whether the diagnosis is accurate. The request engine 316 can interface with the analysis component 320 and the response generation component 322 in any suitable manner to perform the functions described herein. The data store 318 may include additional training data, such as linguistic or textual training data and medical history training data, language models, and / or results from the analysis component 320 and the response generation component 322. In at least some exemplary embodiments, the results from the analysis component 320 and / or the response generation component 322 may be utilized as training data for use in future requests.

[0021] 4 illustrates an example method 400 that may be utilized in accordance with various embodiments. For any process herein, unless specifically stated otherwise, it should be understood that there may be additional, fewer, or alternative steps that are performed in a similar or alternative order, or in parallel, within the scope of various embodiments.

[0022] In this exemplary embodiment, an input image may be received (410). In some examples, the input image may be a medical image acquired by a radiology department. In other examples, the input image may correspond to video data, such as video data acquired during a procedure, or other image and video data unrelated to the medical field. A request for at least one insight related to the input image may also be received (420). For example, a user, including but not limited to a medical professional, may submit a request for a report summarizing any findings related to the input image. The medical report, in at least some exemplary embodiments, may be generated automatically, such as by using artificial intelligence techniques. The input image may be analyzed to detect one or more anomalies that may be present in the input image (430). For example, the input image may be analyzed using machine learning or other artificial intelligence techniques. The underlying algorithms may be trained with one or more sets of training data, including but not limited to language models, image models, and medical history data, such as existing radiology data. Utilizing at least one neural network, information related to the input image may be inferred (540). A number of reports may be generated based at least in part on the inferred information (450).

[0023] In at least some embodiments, each of the multiple reports may have an associated confidence or metric level that corresponds to the average level of certainty of the given report. For example, one report may have a high level of confidence in certainty (e.g., above a threshold level of certainty) and therefore is more likely than a less accurate report and diagnosis. An individual report may be provided with a confidence value or other metric that indicates the confidence in the report's accuracy. A user may look at the confidence value of a given report to determine how much weight to give that report when generating an overall diagnosis. Other metrics may include, according to example embodiments, an average confidence score across all values ​​of the token, a maximum confidence score for a single report, a minimum confidence score for a single report, a standard deviation of the confidence score(s), a number of highlight foci in a report / image pairing, among other such metrics. In using such confidence scores and / or metrics, a user may feel confident that a diagnosis is based on the report. In other cases, a report may have a fairly high level of certainty (e.g., within a range of certainty below a threshold level of certainty but above a minimum level of certainty). In these cases, the user may review other reports generated to better understand what is happening with the input image. For example, a physician may receive reports related to the same image, with each related report providing a different interpretation of the image. Based on at least a subset of the interpretations, the physician may provide a more comprehensive diagnosis and / or order potential follow-up tests related to the potential diagnosis(es). In some embodiments, the generated report may be utilized by a trained professional to gain a better understanding of the image or to determine if there are any abnormalities that the professional may have missed in the initial diagnosis. According to an exemplary embodiment, multiple reports may be presented in a histogram such that each data point on the histogram corresponds to an individual report with a determined level of certainty. By presenting the reports to the user in a histogram format, the user can better understand the various findings of the reports and determine whether or not to base his or her opinion on an individual report.

[0024] 5 illustrates another example method 500 that may be utilized in accordance with various embodiments. For any process herein, unless specifically stated otherwise, it should be understood that there may be additional, fewer, or alternative steps that are performed in a similar or alternative order, or in parallel, within the scope of various embodiments.

[0025] In this exemplary embodiment, an input image may be received (510). In some examples, the input image may be a medical image acquired by a radiology department. In other examples, the input image may correspond to video data, such as video data acquired during a procedure, or other image and video data unrelated to the medical field. In accordance with one or more embodiments, a machine learning algorithm may be used to generate at least one inference related to the input image (520). The input image may be analyzed to detect one or more anomalies that may be present in the input image. For example, the input image may be analyzed using machine learning or other artificial intelligence techniques. The underlying algorithm may be trained with one or more sets of training data, including, but not limited to, language models, image models, and existing radiology data. Based at least in part on the inference, a first output may be provided for display (530). For example, in response to an initial query, a set of reports summarizing one or more findings of the input image may be provided for display. In this example, the findings may correspond to potential information that may contribute to a medical diagnosis.

[0026] At least one interaction with the first output may be received (540). According to an exemplary embodiment, the user may interact with one or more of the generated reports by highlighting features of the report or input image, asking follow-up questions, or querying the system to regenerate a set of reports regarding the highlighted feature(s), among other such interactions. In at least some exemplary embodiments, features present in the input image may be linked to one or more terms such that a detected anomaly links to a corresponding description. When a user hovers over or clicks on a detected anomaly, a corresponding section of text may be highlighted or otherwise provided in prominent display to the user. Similarly, when a user hovers over or clicks on a section of text, an indication may be presented on the image, such as a box surrounding the anomaly. Such linking may be performed automatically, using artificial intelligence techniques. The system may then provide at least one second output for display in response to the at least one interaction received (550). Based on the second output, a user may be able to use the information provided in the second output to better understand any findings in the generated report, e.g., to make a better medical diagnosis with a more comprehensive overview and analysis of the input images.

[0027] 6 illustrates another example method 600 that may be utilized in accordance with various embodiments. For any process herein, unless specifically stated otherwise, it should be understood that there may be additional, fewer, or alternative steps that are performed in a similar or alternative order, or in parallel, within the scope of various embodiments.

[0028] According to this exemplary embodiment, the machine learning model may be trained on at least one set of training data, the at least one set of training data including values ​​of individual tokens and one or more groups of values ​​of individual tokens (610). For example, the machine learning model may be trained on multiple words or syllables at a time, with each syllable or group of words corresponding to a value of a token, in contrast to traditional models that prevent memorization of words and syllables due to the possibility of overfitting the model. If the model is overfitted, it may be readily apparent that text generated by the overfitted model does not sound like natural language, but was actually generated by a computing system. In some exemplary embodiments, the values ​​of the tokens may correspond to values ​​of individual pixels of an image.

[0029] An input image may be received (620). In some examples, the input image may be a medical image acquired by a radiology department. In other examples, the input image may correspond to video data, such as video data acquired during a procedure, or other image and video data unrelated to the medical field. Using the trained machine learning model, the input image may be processed or analyzed (630). An output related to the analyzed input image may be received (640). According to an example embodiment, the output may include a probability distribution of the generated text. In at least some example embodiments, the probability distribution may be overfitted to the training data. As noted elsewhere herein, overfitting a model may produce undesirable results in conventional systems, including text that is unnatural to humans to read. At least a first subset of the data received at the output may be sampled, and a second subset of the data at the output is excluded from sampling (650). For example, at least a portion of the results of the probability distribution may be excluded from sampling based on a threshold applied to the results. Although the examples described herein refer to probability distributions, other types of distributions may be used according to various embodiments. A report associated with the input image can be generated based at least in part on the first subset of sampled data (660). By generating the report in this manner, the report is more natural for a human to read, even after the model has overfitted. With respect to overfitting, the text may be overfitted but the pixels may not, or the pixels may be overfitted but the text may not.

[0030] In some embodiments, a system implementing some or all of the techniques described herein may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media, such as computer system 700 illustrated in FIG. 7. In the illustrated embodiment, computer system 700 includes one or more processors 710A-710N coupled to a system memory 720 via an input / output (I / O) interface 730. Computer system 700 further includes a network interface 740 coupled to I / O interface 730. Although FIG. 7 illustrates computer system 700 as a single computing device, in various embodiments computer system 700 may include one computing device or any number of computing devices configured to operate together as a single computer system 700.

[0031] In various embodiments, computer system 700 may be a uniprocessor system including one processor 710A, or a multiprocessor system including several processors 710A-710N (e.g., two, four, eight, or another suitable number). Processors 710A-710N may be any suitable processor capable of executing instructions. For example, in various embodiments, processors 710A-710N may be general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs). In a multiprocessor system, each of processors 710A-710N may typically, but need not, implement the same ISA.

[0032] System memory 720 may store instructions and data accessible by processor(s) 710A-710N. In various embodiments, system memory 720 may be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated embodiment, program instructions and data implementing one or more desired functions, such as those methods, techniques, and data described above, are shown stored within system memory 720 as image analysis and result generation service code 725 and data 726.

[0033] In one embodiment, the I / O interface 730 may be configured to coordinate I / O traffic between the processor(s) 710A-710N, the system memory 720, and any peripheral devices within the device, including the network interface 740 or other peripheral interfaces. In some embodiments, the I / O interface 730 may perform any necessary protocol, timing, or other data conversions to convert data signals from one component (e.g., the system memory 720) into a format suitable for use by another component (e.g., the processor(s) 710A-710N). In some embodiments, the I / O interface 730 may include support for devices attached through various types of peripheral buses, such as, for example, the Peripheral Component Interconnect (PCI) bus standard, or variations of the Universal Serial Bus (USB) standard. In some embodiments, the functionality of the I / O interface 730 may be split into two or more separate components, such as, for example, a northbridge and a southbridge. Also, in some embodiments, some or all of the functionality of the I / O interface 730, such as the interface to the system memory 720, may be incorporated directly into the processor(s) 710A-710N.

[0034] Network interface 740 may be configured to allow data to be exchanged between computing system 700 and other devices 760 attached to network(s) 750. In various embodiments, network interface 740 may support communications over any suitable wired or wireless general data network, such as, for example, a type of Ethernet network. Additionally, network interface 740 may support communications over a telecommunications / telephony network, such as an analog voice network or a digital fiber communications network, over a storage area network (SAN), such as a Fibre Channel SAN, or over I / O, any other suitable type of network and / or protocol.

[0035] In some embodiments, computer system 700 includes one or more offload cards 770 (including one or more processors 775 and possibly one or more network interfaces 740) connected using an I / O interface 730 (e.g., a bus implementing a version of the Peripheral Component Interconnect Express (PCI-E) standard or another interconnect such as the QuickPath Interconnect (QPI) or UltraPath Interconnect (UPI)). For example, in some embodiments, computer system 700 can act as a host electronic device (e.g., operating as part of a hardware virtualization service) that hosts computing instances, and one or more offload cards 770 execute a virtualization manager that can manage the computing instances executing on the host electronic device. By way of example, in some embodiments, offload card(s) 770 can perform computing instance management operations such as pausing and / or unpausing computing instances, starting and / or terminating computing instances, performing memory transfer / copy operations, etc. These management operations, in some embodiments, may be performed by offload card(s) 770 in coordination with (e.g., in response to requests from) hypervisors executed by other processors 710A-710N of computer system 700. However, in some embodiments, the virtualization manager implemented by offload card(s) 770 may service requests from other entities (e.g., from the computing instance itself) and may not coordinate with (or service) any separate hypervisor.

[0036] In some embodiments, the system memory 720 may be an embodiment of a computer-accessible medium configured to store program instructions and data as described above. However, in other embodiments, the program instructions and / or data may be received, transmitted, or stored in different types of computer-accessible media. Generally speaking, the computer-accessible medium may include non-transitory storage or memory media, such as magnetic or optical media, e.g., disks or DVDs / CDs, coupled to the computing system 700 via the I / O interface 730. The non-transitory computer-accessible storage media may also include any volatile or non-volatile media, such as RAM (e.g., SDRAM, double data rate (DDR) SDRAM, SRAM, etc.), read-only memory (ROM), etc., that may be included in some embodiments of the computing system 700 as the system memory 720 or another type of memory. Additionally, the computer-accessible medium may include a transmission medium or a signal, such as an electrical signal, an electromagnetic signal, or a digital signal, transmitted over a communication medium, such as a network and / or a wireless link, such as may be implemented via the network interface 740.

[0037] FIG. 8 illustrates an exemplary environment 800 in which aspects of various embodiments may be implemented. In this example, a user may utilize a client device 802 to submit requests to a multitenant resource provider environment 806 over at least one network 804. The client device may include any suitable electronic device operable to transmit and receive requests, messages, or other such information over an appropriate network and to convey the information back to a user of the device. Examples of such client devices include personal computers, tablet computers, smartphones, notebook computers, and the like. The at least one network 804 may include any suitable network including an intranet, the Internet, a cellular network, a local area network (LAN), or any other such network, or combination, and communication through the network may be enabled via wired and / or wireless connections. The resource provider environment 806 may include any suitable components for receiving requests and returning information or performing actions in response to those requests. By way of example, the provider environment may include a web server and / or application server for receiving and processing requests and then returning data, web pages, video, audio, or other such content or information in response to the requests.

[0038] In various embodiments, the provider environment may include various types of resources that may be utilized by multiple users for a variety of different purposes. As used herein, computing and other electronic resources utilized in a network environment may be referred to as "network resources." These may include, for example, servers, databases, load balancers, routers, and the like that may perform tasks such as receiving, transmitting, and / or processing data and / or executable instructions. In at least some embodiments, all or a portion of a given resource, or set of resources, may be assigned to a particular user for at least a defined period of time or may be assigned to a particular task. The sharing of those multi-tenant resources from the provider environment is often referred to as resource sharing, web services, or "cloud computing," among other such terms, as well as depending on the particular environment and / or implementation. In this example, the provider environment includes multiple resources 814 of one or more types. Those types may include, for example, application servers operable to process instructions provided by users, or database servers operable to process data stored in one or more data stores 816 in response to user requests. As is known for such purposes, a user may also reserve at least a portion of the data storage in a given data store. Methods for enabling users to reserve various resources and resource instances are well known in the art, and as a result, a detailed description of the entire process, and a description of all possible components, will not be discussed in detail herein.

[0039] In at least some embodiments, a user desiring to utilize a portion of the resource 814 can submit a request that is received at an interface layer 808 of the provider environment 806. The interface layer can include an application programming interface (API) or other exposed interface that allows the user to submit a request to the provider environment. The interface layer 808 in this example can also include other components, such as at least one web server, a routing component, and a load balancer. When a request to provision a resource is received at the interface layer 808, information about the request can be directed to a resource manager 810 or other such system, service, or component configured to manage user accounts and information, resource provisioning and usage, and other such aspects. The resource manager 810 that receives the request can perform tasks such as authenticating the identity of the user submitting the request and determining whether the user has an existing account with the resource provider, and the account data can be stored in at least one data store 812 within the provider environment. The user can provide any of a variety of types of authentication information to the provider to authenticate the user's identity. These authentication information may include, for example, a username and password pair, biometric data, a digital signature, or other such information. The provider may validate this information against information stored for the user. If the user has an account with the proper permissions, state, etc., the resource manager may determine whether there are appropriate resources available suitable for the user's request, and if so, may provision the resources or allow access to the corresponding portions of those resources for use by the user for an amount specified by the request. This amount may include, for example, the ability to process a single request or perform a single task, a specified period of time, or a recurring / reconfigurable period of time, among other such values.If the user does not have a valid account with the provider, the user account does not allow access to resources of the type specified in the request, or another such reason prevents the user from gaining access to such resources, a communication may be sent to the user to enable the user to create or modify an account or change the resources specified in the request, among other such options.

[0040] Once a user has been authenticated, an account verified, and resources allocated, the user may utilize the allocated resource(s) for a specified capacity, amount of data transfer, duration, or other such value. In at least some embodiments, the user may provide a session token or other such authentication information with subsequent requests to allow those requests to be processed in the user session. The user may receive a resource identifier, specific address, or other such information, which may enable the client device 802 to communicate with the allocated resources without having to communicate with the resource manager 810, at least until an associated aspect of the user account is changed, until the user is no longer authorized to access the resource, or until another such aspect is changed, etc.

[0041] The resource manager 810 (or another such system or service) in this example can also function as a virtual layer of hardware and software components that handles control functions in addition to management actions, such as may include provisioning, scaling, replication, etc. The resource manager can utilize dedicated APIs in the interface layer 808, each of which may be provided to receive a request for at least one specific action to be performed with respect to the data environment, such as to provision, scale, clone, or hibernate an instance. Upon receiving a request to one of the APIs, the web services portion of the interface layer can parse or otherwise analyze the request to determine steps or actions required to act on or process the call. For example, a web services call may be received that includes a request to create a data repository.

[0042] The interface layer 808 in at least one embodiment includes a scalable set of user-facing servers that can provide various APIs and return appropriate responses based on the API specifications. The interface layer can also include at least one API services layer, which in one embodiment is made up of stateless, replicated servers that handle external-facing user APIs. The interface layer can be responsible for web services front-end functions such as authenticating users based on credentials, authorizing users, throttling user requests to the API server, verifying that user input is valid, and marshalling or unmarshalling requests and responses. The API layer can also be responsible for reading and writing database configuration data to and from an administration data store in response to API calls. In many embodiments, the web services layer and / or API services layer are the only components that are externally visible or visible to and accessible by users of the control services. The servers in the web services layer can be stateless and horizontally scaled, as known in the art. The API servers, as well as the persistent data stores, may be distributed, for example, across multiple data centers in a region, so that the servers are resilient to the failure of a single data center.

[0043] Various embodiments may also be implemented in a wide variety of operating environments, which in some cases may include one or more user computers or computing devices that may be used to operate any number of applications. User or client devices may include any of a number of general purpose personal computers, such as desktop or laptop computers running standard operating systems, as well as cellular, wireless, and handheld devices that run portable software and can support a number of networking and messaging protocols. Such systems may also include a number of workstations running any of a variety of commercially available operating systems and other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as dummy terminals, thin clients, gaming systems, and other devices capable of communicating over a network.

[0044] Most embodiments utilize at least one network to support communications using any of a variety of commercially available protocols, such as TCP / IP, FTP, UPnP, NFS, and CIFS, as would be familiar to one of ordinary skill in the art. The network may be, for example, a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, and any combination thereof. In embodiments utilizing a web server, the web server may run any type of server or mid-tier application, including an HTTP server, an FTP server, a CGI server, a data server, a Java server, and a business application server. The server(s) may also be capable of executing programs or scripts in response to requests from user devices, such as by executing one or more web applications, which may be implemented as one or more scripts or programs written in any programming language, such as Java, C, C#, or C++, or any scripting language, such as Perl, Python, or TCL, and combinations thereof. The server(s) may also include database servers, including, but not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, and IBM®.

[0045] The environment may include the various data stores and other memory and storage media mentioned above. These may reside in a variety of locations, such as on storage media that are local to (and / or contained within) one or more of the computers, or remote from any or all of the computers across a network. In a particular set of embodiments, the information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any files necessary to perform functions belonging to a computer, server, or other network device may be stored locally and / or remotely as needed. Where the system includes computerized devices, each such device may include hardware elements that may be electrically connected via a bus, including, for example, at least one central processing unit (CPU), at least one input device (e.g., a mouse, keyboard, controller, touch-sensitive display element, or keypad), and at least one output device (e.g., a display device, printer, or speaker). Such systems may also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices, such as random access memory (RAM) or read-only memory (ROM), as well as removable media devices, memory cards, flash cards, and the like. Such devices may also include a computer-readable storage media reader, a communication device (e.g., a modem, a network card (wireless or wired), an infrared communication device), and a working memory, as described above. The computer-readable storage media reader may be connected to or configured to receive storage media for containing, storing, transmitting, and retrieving computer-readable information on a temporary and / or even permanent basis, with computer-readable storage media representing remote, local, fixed, and / or removable storage devices.

[0046] The systems and various devices also typically include several software applications, modules, services, or other elements located within at least one working memory device, including an operating system and application programs such as client applications or web browsers. It should be understood that alternative embodiments may have numerous variations from those described above. For example, specialized hardware may also be used, and / or particular elements may be implemented in hardware, software (including portable software such as applets), or both. Additionally, connections to other computing devices, such as network input / output devices, may be used. Storage media and other non-transitory computer readable media embodying code or portions of code can include any suitable media known or used in the art, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data, including RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage, or any other media that can be used to store the desired information and that can be accessed by a system device. Based on the disclosure and teachings provided herein, a person skilled in the art will recognize other methods and techniques for implementing the various embodiments.

[0047] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense, however it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.

Claims

1. 1. A computer-implemented method, comprising: receiving an input image; receiving a request for at least one insight related to the input image; analyzing the input image to detect one or more anomalies present in the input image; utilizing at least one neural network to infer information related to the input image; generating a plurality of reports related to the input images based at least in part on the inferred information, each of the plurality of reports having at least one associated metric.

2. The computer-implemented method of claim 1 , wherein the at least one neural network is trained using at least one of an image model, a text model, and medical history data.

3. enabling a user to interact with at least a portion of one of the reports and the input image; and The computer-implemented method of claim 1 or 2, further comprising providing an indication regarding the other of the report and the input image corresponding to the interacted portion of the report.

4. 4. The computer-implemented method of claim 1, further comprising: allowing a user to interact with the plurality of reports, the interaction including one or more of: highlighting the input image or a report of the plurality of reports by a user, receiving a question from a user related to the plurality of reports, or interacting with a distribution generated for the plurality of reports.

5. The computer-implemented method of claim 1 , wherein the input image is obtained through processing of a medical image.

6. The computer-implemented method of claim 1 , wherein the plurality of reports is provided in a histogram, the histogram configured to provide a comprehensive presentation of the plurality of reports.

7. 1. A computer-implemented method, comprising: receiving an input image; using a machine learning model to generate at least one inference related to the image; providing a first output for display based at least in part on the at least one inference; receiving at least one interaction with the first output; and providing at least one second output for display in response to the received at least one interaction.

8. The computer-implemented method of claim 7 , wherein the first output comprises a set of reports related to the at least one inference.

9. 9. The computer-implemented method of claim 7 or 8, wherein the at least one second output comprises one or more of a response responsive to the at least one interaction and a new set of reports generated in response to the at least one interaction.

10. 10. The computer-implemented method of claim 7, wherein the at least one interaction includes one or more of highlighting in the first output by a user, receiving a question from a user, or interacting with a distribution generated for the first output.

11. enabling a user to interact with at least a portion of the report in the first output; and The computer-implemented method of claim 7 , further comprising: providing an indication regarding the input image that corresponds to the interacted portion of the report.

12. The computer-implemented method of claim 7 , wherein the input image is obtained through processing of a medical image.

13. 13. The computer-implemented method of claim 7, wherein the machine learning model is trained using at least one of an image model, a text model, and medical history data.

14. 14. The computer-implemented method of claim 7, wherein the first output is provided as a histogram, the histogram configured to provide a comprehensive presentation of a plurality of reports.

15. The computer-implemented method of claim 10 , wherein the distribution generated for the first output provides insight into a clinical diagnosis associated with the input image.

16. 1. A computer-implemented method comprising: training a machine learning model with a set of training data, the set of training data including values ​​of individual tokens and one or more groups of values ​​of the individual tokens; receiving an input image; analyzing the input image using the trained machine learning model; receiving an output related to the analyzed input image; sampling at least a first subset of data received at the output, wherein a second subset of data at the output is excluded from the sampling; and generating a report associated with the input image based at least in part on the first subset of the sampled data.

17. 17. The computer-implemented method of claim 16, wherein the values ​​of the individual tokens correspond to textual or image values ​​of the set of training data.

18. 18. A computer-implemented method according to claim 16 or 17, wherein the output corresponds to a distribution of values ​​of a text.

19. 19. The computer-implemented method of claim 16, wherein the output related to the analyzed image overfits the set of training data.

20. 20. The computer-implemented method of claim 19, wherein overfitting the output to one or more modalities in the set of training data increases a level of accuracy associated with the report.

Images