Ai-assisted medical image report generation display and workflow

Abstract

Disclosed herein are systems, media, and methods for medical report generation or evaluation. Natural language processing, computer vision, large language models, or other software may be used to facilitate accurate, efficient evaluation of contemporary or historical medical reports. The present disclosure may provide particular utility in patient health diagnostics, prognostics, treatment, or monitoring.

Description

AI-ASSISTED MEDICAL IMAGE REPORT GENERATION DISPLAY AND WORKFLOWCROSS REFERENCE TO OTHER APPLICATION(S)

[0001] This application claims benefit of U.S. Provisional Application No. 63 / 598,037, filed November 10, 2023, which is incorporated herein in its entirety by reference.BACKGROUND

[0002] Medicine is a multi-modal discipline informed by numerous modalities such as text, images, audio, video, and tabular data. Patient health may be comprehensively represented by these modalities, yet leveraging such complex, contextually dependent, temporally resolved patient health data is challenging by its very nature. Radiology is a discipline of particular difficulty, where imaging data is contextualized by a diversity of patient health data. A lapse in appropriate application of context to imaging data can limit the effectiveness, timeliness, and quality of diagnosis and treatment.SUMMARY

[0003] In an aspect disclosed herein is a method for generating a medical report, comprising: (a) displaying a medical image on a display; (b) while a user is viewing the display, receiving a user input from the user; (c) while receiving the user input, distributing a plurality of findings from the user input to a plurality of report sections of the medical report; and (d) providing the medical report to the user.

[0004] In some embodiments, the display comprises an augmented reality, virtual reality, or mixed reality device. In some embodiments, the display receives the user input. In some embodiments, the method further comprises modifying said medical image. In some embodiments, modifying comprises rotating, zooming in, zooming out, or moving said medical image. In some embodiments, the method further comprises displaying a finding of the plurality of findings in a report section of the plurality of report sections while receiving the user input. In some embodiments, a report section of the plurality of report sections comprises an impression section. In some embodiments, the impression section is automatically generated by a large language model. In some embodiments, the large language model is a multi-modal machine learning model. In some embodiments, the multi-modal machine learning model extracts at least one image finding included in the impression. In some embodiments, the impression section comprises a recommendation or a diagnosis. Insome embodiments, the method further comprises analyzing said user input using a machine learning model. In some embodiments, the method further comprises extracting the plurality of findings of the user input using a machine learning model. In some embodiments, the machine learning model is a large language model. In some embodiments, the method further comprises detecting a discrepancy in the plurality of findings. In some embodiments, the discrepancy is between a first finding and a second finding of the plurality of findings, the plurality of findings and a computer-generated findings, or a combination thereof. In some embodiments, detecting the discrepancy comprises comparing a finding of the plurality of findings with a finding from a medical record. In some embodiments, finding from the medical record is a text finding, an image finding, or a tabular finding. In some embodiments, the discrepancy is between a first finding a second finding, wherein one or both of the first and second finding is a permanent finding, a transient finding, a pertinent negative finding, or an incidental finding. In some embodiments, the method further comprises presenting the discrepancy to a user, wherein the user accepts, rejects, or edits a finding of the plurality of findings. In some embodiments, upon accepting, rejecting, or editing the finding of the plurality of findings, the medical reportis updated with an updated finding that reflects said accepting, rejecting, or editing. In some embodiments, the medical image comprises an x-ray, a CAT scan, a PET scan, an MRI, an ultrasound, a tomosynthesis scan, a fluoroscopy scan, or a combination thereof. In some embodiments, the user input is a verbal user input. In some embodiments, the verbal user input is received through an audio input device. In some embodiments, the method further comprises providing a link in said medical report, wherein the link corresponds to a finding of said plurality of findings, and selection of the link retrieves the medical image for viewing. In some embodiments, selection of the link retrieves at least one additional medical image co-registered with the medical image. In some embodiments, the method further comprises providing a link in said medical image, wherein the link corresponds to a finding of said plurality of findings, and selection of the link retrieves the medical report for viewing. In some embodiments, distributing of the plurality of findings into the plurality of report sections of the medical report is based at least in part on a medical report template selected by the user or for the user. In some embodiments, the user selects a medical report template. In some embodiments, the user interacts with the display solely through voice commands.

[0005] In an aspect disclosed herein is a method for reviewing a parameter of a finding, the method comprising: (a) receiving a first medical report and a second medical report; (b)processingthe first medical report and the second medical report to extract the finding from each of the first medical report and the second medical report using a machine learning model; (c) determining a first parameter of the finding in the first medical report and a second parameter of the finding in the second medical report; and (d) generating a summary comprising a statement regarding a difference of the first parameter of the finding and the second parameter of the finding.

[0006] In some embodiments, the first medical report was generated at a different time than the second medical report. In some embodiments, the difference of the first parameter and the second parameter finding shows a progression or treatment of a disease. In some embodiments, the summary comprises a sentence or statement regarding a change in the finding based on the difference of the first parameter of the finding and the second parameter of the finding. In some embodiments, the finding is a transient finding or a permanent finding. In some embodiments, the method further comprises displaying a portion of the first medical report comprising the finding. In some embodiments, the method further comprises displaying a portion of the second medical report comprising the finding. In some embodiments, the method further comprises displaying a medical image of the first medical report. In some embodiments, the method further comprises displaying a medical image of the second medical report. In some embodiments, the method further comprises coregistering the medical image of the first medical report and the medical image of the second medical report. In some embodiments, the finding describes a pathology or an anatomy. In some embodiments, the finding is an image finding or a text finding. In some embodiments, the first medical report and the second medical report were generated at least one day apart. In some embodiments, the machine learning model is a large language model. In some embodiments, the large language model is a multi-modal large language model. In some embodiments, the large language model comprises a computer vision model. In some embodiments, the parameter of the finding comprises a size, width, density, color, diameter, shape, thickness, severity, weight, or location of the finding. In some embodiments, the summary comprises a textual summary of the change of the finding, a graph of the parameter, or a table of the parameter. In some embodiments, a large language model generates said summary. In some embodiments, the method further comprises selecting a medical report of the at least two reports from a medical record of a patient. In some embodiments, said selecting is based at least in part on a similarity between a portion of the medical report and a user provided finding. In some embodiments, the selecting comprises the use of naturallanguage processing. In some embodiments, the selecting comprises the use of large language model.

[0007] In an aspect disclosed herein is a method of reviewing a medical report of a patient, comprising: (a) extracting a first finding from the medical report of the patient; (b) extracting a second finding from a medical record of the patient; (c) comparing, using a large language model, the first finding with the second finding to determine a discrepancy between the medical report of the patient and the medical record of the patient; (c) retrieving a study of the patient upon determination of the discrepancy; (d) extracting a third finding from the study of the patient; (e) evaluating, the discrepancy using the large language model by comparing the discrepancy with the third finding from the study of the patient; (f) outputting, from the large language model, an option for a user to accept, reject, or edit the first finding or the second finding.

[0008] In some embodiments, an impression section of the medical report comprises the first finding. In some embodiments, the method further comprises generating a sentence or statement comprising the first finding using a natural language processor. In some embodiments, the second finding and the third finding are the same. In some embodiments, the discrepancy is determined when the first finding and the second finding are different. In some embodiments, the method further comprises alerting a user of the discrepancy. In some embodiments, the large language model is used to determine the third finding. In some embodiments, the second finding is from a second medical report of the medical record. In some embodiments, the second finding is contained in DICOM data, HL7 data, or an electronic medical record (EMR) of the medical record. In some embodiments, the first finding, second finding, or both, is a text finding, an image finding, a tabular finding, or a combination thereof. In some embodiments, the first finding and the second finding comprise a different modality. In some embodiments, determining the discrepancy is based on a first parameter of the first finding and a second parameter of the second finding. In some embodiments, the first parameter, second parameter, or both is a size, width, density, color, diameter, shape, thickness, severity, weight, location, or a combination thereof. In some embodiments, the discrepancy is determined at least in part based on an input prompt used to configure the large language model. In some embodiments, the study of the patient comprises a medical image. In some embodiments, the third finding is an image finding of the medical image. In some embodiments, the large language model is a multi -model large languagemodel. In some embodiments, the method further comprises updating the medical report, the medical record, or both. In some embodiments, updating the medical report comprises modifying or changing the first finding. In some embodiments, a large language model is used to extract the first finding or second finding. In some embodiments, the large language model for comparing the first finding with the second finding to determine the discrepancy is a first large language model, and the large language model for evaluating the discrepancy by comparing the discrepancy with the third finding from the study of the patient is a second large language model.

[0009] In an aspect disclosed herein is a method for monitoring a change of a finding, said method comprising: (a) receiving a voice dictation from a user evaluating a medical image of a patient, wherein the voice dictation comprises a first parameter of a finding of the medical image; (b) retrieving at least one prior medical report of the patient; (c) obtaining a second parameter of the finding from the at least one prior medical report based on the finding of the medical image; (d) comparing the first parameter of the finding to the second parameter of the finding and detecting a difference between the first parameter of the finding and the second parameter of the finding; and (e) generating a statement pertaining to the difference tween the first parameter of the finding and the second parameter of the finding.

[0010] In some embodiments, the at least one prior medical report for the patient comprises textual data, image data, tabular data, or a combination thereof. In some embodiments, the finding of the medical image comprises a transient finding, permanent finding, incidental finding, or pertinent negative finding. In some embodiments, natural language processing is used to determine the finding of the medical image from the voice dictation. In some embodiments, the finding from the at least one prior medical report comprises an image finding, a textual finding, or a combination thereof. In some embodiments, obtaining the finding of the at least one prior medical report comprises extracting the finding from the at least one prior medical report using a machine learning model. In some embodiments, the machine learning model is a large language model (LLM). In some embodiments, the LLM comprises a multi-modal LLM. In some embodiments, the statement is output by a large language model. In some embodiments, a parameter of the finding of the medical image is compared to a parameter of the finding from the at least one medical report to provide a comparison between the findings. In some embodiments, the method further comprises presenting the statement to the user. In some embodiments, the method further comprisespresenting the user an option to accept, deny, or modify the impression. In some embodiments, the method further comprises receiving feedback from a user about the statement of the medical image. In some embodiments, the feedback comprises an acceptance, rejection, or edit of the impression. In some embodiments, the feedback is used in a training set for updating a machine learning model used to generate the impression. In some embodiments, the feedback and the difference between generated and submitted statement is used in the training set for updating the machine learning model used to generate the impression In some embodiments, the statement is presented in an augmented, virtual, or mixed reality environment. In some embodiments, the first parameter of the finding comprises an absence of the finding. In some embodiments, the second parameter of the finding comprises an absence of the finding.

[0011] In an aspect disclosed herein is a method for training a machine learning model, comprising: (a) generating a medical report from a medical record using a large language model; (b) presenting the medical report to a user; (c) receiving feedback from the user about the medical report; (d) generating a training set to train the large language model, wherein a datum of the training set comprises at least a portion of the medical record, the medical report, and the feedback about the medical report; and (e) training the machine learning model using the training set.

[0012] In some embodiments, the feedback comprises a user updated or filed medical report. In some embodiments, the method further comprises comparing the generated medical report to the user updated or filed medical report to determine a difference between the generated medical report and the user updated or filed medical report. In some embodiments, the difference is an addition, a removal, or an edit of a finding of the medical report. In some embodiments, the method further comprises generating an updated medical report using the large language model. In some embodiments, the method further comprises comparing a second medical report to the updated medical report. In some embodiments, a finding of the updated medical report is compared to a finding of the second medical report. In some embodiments, the method further comprises detecting a discrepancy between the finding of the updated medical report and the finding of the second medical report. In some embodiments, the second medical report was generated before the updated medical report. In some embodiments, the method further comprises generating a second medical report using the large language model. In some embodiments, the medical report comprises an impression.In some embodiments, the medical report comprises a summary of the medical record. In some embodiments, the medical report comprises a finding. In some embodiments, feedback is received via virtual, mixed, or augmented reality device. In some embodiments, the medical report is presented to the user via virtual, mixed, or augmented reality device. In some embodiments, the large language model is a multi-modal large language model. In some embodiments, the feedback from the user comprises an acceptance, a rejection, or an edit of the medical report. In some embodiments, the method further comprises detecting a discrepancy in the medical report and prompting the user to provide the feedback about the discrepancy. In some embodiments, the feedback is used to generate an updated medical record. In some embodiments, the feedback comprises a preference of a user. In some embodiments, the method further comprises generating a second medical report based at least in part on a preference of the user. In some embodiments, the preference comprises a display preference.

[0013] In an aspect disclosed herein is a method of reviewing a medical report, said method comprising: (a) reviewing the medical report using a machine learning model; (b) reviewing a second medical report using the machine learning model; (c) detecting a discrepancy between the medical report and the second medical report; and (d) generating one or more sentence or phrase describing or assessing said discrepancy.

[0014] In some embodiments, said machine learning model is a large language model. In some embodiments, the machine learning model is prompt engineered to determine a finding. In some embodiments, said finding is associated with an anatomic variant. In some embodiments, said finding is a permanent finding, a transient finding, or a pertinent negative finding. In some embodiments, the discrepancy is based on a difference in semantic similarity between a finding of said medical report and a finding of said second medical report. In some embodiments, the discrepancy is a difference between a finding of the medical report and a finding of the medical record. In some embodiments, the discrepancy is a difference between a first parameter of a finding of the medical report and a second parameter of a finding of the medical record. In some embodiments, the medical report and the second medical report are based on the same medical image, patient study, or a combination thereof. In some embodiments, one or more sentence or phrase describing or assessing said discrepancy is an output of a natural language processing algorithm. In some embodiments, said natural language processing algorithm comprises a large language model. In some embodiments, auser reviews said one or more sentence or phrase and accepts, rejects, or edits at least a portion of said one or more sentence or phrase before insertion of said one or more sentence or phrase into the medical report. In some embodiments, a medical record comprising the medical report is updated based on the user review. In some embodiments, the second medical report is a computer-generated medical report. In some embodiments, the computergenerated medical report is based at least in part on a medical image. In some embodiments, the medical report is based at least in part on said medical image. In some embodiments, the second medical report is generated by a human. In some embodiments, the reviewing is an overread of the medical report.

[0015] In as aspect described herein is a method of reviewing a medical report, said method comprising: (a) providing the medical report comprising a medical image and a medical report text; (b) detecting a textual finding in the medical report text; (c) detecting an image finding in the medical image; (d) comparing the textual finding with the image finding using a multi-modal machine learning model; and (e) detecting a discrepancy when the textual finding does not match the image finding.

[0016] In some embodiments, said medical reportis a contemporary medical report. In some embodiments, said medical report is a prior medical report. In some embodiments, said textual finding is a permanent finding, a pertinent negative finding, or a transient finding. In some embodiments, said image finding is a permanent finding, a pertinent negative finding, or a transient finding In some embodiments, multi-modal machine learning model is a multimodal large language model. In some embodiments, said large language model is a finetuned foundation model. In some embodiments, said medical summary comprises text data, image data, audio data, video data, tabular data, or a combination thereof. In some embodiments, said multi-modal machine learning model takes as input at least two of text data, image data, audio data, video data, or tabular data. In some embodiments, said multimodal machine learning model is trained to compare a first finding from a first data modality and a second finding from a second data modality. In some embodiments, said multi-modal machine learning model comprises an encoder at the first data modality and the second data modality. In some embodiments, said first data modality or second data modality is text, image, audio, video, or tabular. In some embodiments, said first data modality is different from said second data modality. In some embodiments, said multi-modal machine learning model is prompt engineered with a prompt to detect said discrepancy. In some embodiments,said prompt is used for in-context learning. In some embodiments, said prompt comprises at least one instruction. In some embodiments, the method further comprises alerting a user of the discrepancy. In some embodiments, the medical report text is in a summary of the medical report. In some embodiments, a discrepancy is detected when a parameter of the textual finding does not match a parameter of the image finding.

[0017] In an aspect disclosed herein is a computer-implemented method for generating a summary, the method comprising: (a) receiving a medical record and a summary template; (b) extracting a finding from the medical record; (c) determining a finding class and a relevancy category of the finding; and (d) generating the summary based at least in part on the summary template, the finding class and relevancy category of the finding.

[0018] In some embodiments, the summary template comprises a label for the finding class of the finding. In some embodiments, the label comprises a text color, a text highlighting, a text font, an icon, or any combination thereof. In some embodiments, the summary template is determined by a user. In some embodiments, the summary template is determined a large language model and is based at least in part on a preference of the user. In some embodiments, the finding class comprises a permanent finding class, a pertinent negative finding class, or a transient finding class. In some embodiments, the summary does not comprise the finding with a transient class. In some embodiments, the summary comprises the finding determined to have a transient finding class if a practitioner comment is received regarding the finding comprising the finding class comprising the transient finding. In some embodiments, the relevancy category comprises a specific relevancy category if the finding is related to an anatomy, a pathology, or both, associated with the summary. In some embodiments, the relevancy category is determined at least in part based on the finding class. In some embodiments, the summary template comprises a preferred order of one or more findings wherein the preferred order is based at least in part on: the determined finding class of each of the one or more findings, a user preference, or both. In some embodiments, the summary displays the preferred order of the one or more findings. In some embodiments, summary is further generated based on Digital Imaging and Communications in Medicine (DICOM) data, Health Level Seven (HL7) data, or both. In some embodiments, the method further comprises receiving a medical image and determining an anatomy shown in the medical image, wherein the relevancy category is further determined based at least in part on the anatomy. In some embodiments, the method further comprises receiving a feedback froma user regarding the summary, and wherein the summary is further generated based on the feedback. In some embodiments, the preference of the user is based on a specialty of the user. In some embodiments, the preference of the user is based on a previous user interaction with a summary of a prior medical record. In some embodiments, a second finding of the one or more findings is not included in the summary. In some embodiments, the method further comprises extracting a second finding from a medical image or the medical record. In some embodiments, the method further comprises determining a second finding class and a second relevancy category of the second finding, wherein the summary of the medical record is further based at least in part on the second finding class and the second relevancy category of the second finding. In some embodiments, the summary is a summary of the medical report.

[0019] In an aspect disclosed herein is a computer-based method for generating a medical report of a patient, the method comprising: (a) receiving a medical report template, a finding regarding the patient, and a prior medical report associated with a user, the patient, the finding, or any combination thereof; (b) reviewing the medical report template, the prior medical report, and the finding using a machine learning model; and (c) generating the medical report based at least in part on the medical report template, the prior medical report, and the finding.

[0020] In some embodiments, the medical report template comprises a plurality of fields. In some embodiments, a field of the plurality of fields comprises an impression. In some embodiments, the finding regarding the patient is extracted from a user input of the user. In some embodiments, the finding regarding the patient is extracted from an image study of the patient. In some embodiments, the medical report is generated by: determining one or more fields of the plurality of fields related to the finding, and inserting a sentence or phrase related to the finding into the one or more fields. In some embodiments, the medical report is generated by determining one or more fields of the plurality of fields that are represented by a portion of the prior medical report and inserting the portion of the prior medical report into the one or more fields. In some embodiments, determining one or more fields of the plurality of fields that are related to the finding comprises the use of a machine learning module. In some embodiments, a natural language processing module is used to generate the sentence or phrase related to said finding into the one or more fields. In some embodiments, the method further comprises receiving a report feedback from the user regarding the medical report, and wherein a second medical report or an updated medical report is generated based at least inpart on the report feedback. In some embodiments, the method further comprises receiving a medical record of the patient, wherein the medical reportis further generated based at least in part on the medical record. In some embodiments, the method further comprises determining a finding class of the finding, wherein the medical report is further generated based at least on the finding class. In some embodiments, the finding class comprises a permanent finding class, a pertinent negative class, or a transient finding class. In some embodiments, the medical report template is selected by a user. In some embodiments, the medical report template is selected through the use of a machine learning module. In some embodiments, the machine learning module selects the medical report template based at least in part on a prior action by the user or a preference of the user.

[0021] In an aspect disclosed herein is a computer-based method for generating a medical report, the method comprising: (a) receiving a finding regarding a patient; (b) storing the finding in a medical record of the patient; (c) receiving a request to generate a first medical report, and a first template; (d) generating the first medical report according to the first template based at least in part on the medical record and the finding; (e) receiving a second request to generate a second medical report and a second template; and (f) generating the second medical report according to the second template based at least in part on the medical record and the finding.

[0022] In some embodiments, the finding regarding the patient is received from a user. In some embodiments, the finding comprises a dictated finding from the user. In some embodiments, the first medical report and the second medical report comprise the finding. In some embodiments, the finding regarding the patient is a computer-generated finding. In some embodiments, the first template is based at least in part on a preference of a first user. In some embodiments, the first template is based at least in part on a preference of a second user. In some embodiments, the first template and the second template are different. In some embodiments, the first template and the second template each comprise a plurality of fields. In some embodiments, at least one field of the plurality of fields of the first template is different from at least one field of the plurality of fields of the second template. In some embodiments, the method further comprises determining one or more of the plurality of fields that is represented by the finding and further inputting a sentence or phrase comprising the finding into the determined one or more. In some embodiments, the method further comprises receiving a report feedback from a user regarding the first medical report and wherein thesecond medical report is further generated based on the report feedback. In some embodiments, the first template is different from the second template. In some embodiments, the first medical reportis different from the second medical report. In some embodiments, a machine learning module selects the first template, the second template, or both. In some embodiments, a user selects the first template. In some embodiments, a second user selects the second template. In some embodiments, the first template is based at least in part on a preference of a firstuser. In some embodiments, the second template is based at least in part on a preference of a second user.

[0023] In an aspect disclosed herein is a method of reviewing a medical report of a patient, comprising: (a) extracting a first finding from the medical report of the patient; (b) extracting a second finding pertaining to the patient from the medical report; (c) comparing, using a multi-modal language model, the first finding with the second finding to determine a discrepancy between the first finding and the second finding.

[0024] In some embodiments, the medical report comprises a lab result or imaging pertaining to the patient. In some embodiments, the first finding and the second finding comprise a gender of the patient. In some embodiments, the first finding and the second finding comprise an anatomy. In some embodiments, the discrepancy comprises an inconsistency between the first finding and the second finding. In some embodiments, the medical report comprises the second finding. In some embodiments, a medical image comprises the second finding. In some embodiments, a lab result comprises the second finding. In some embodiments, the method further comprises: retrieving a study of the patient upon determination of the discrepancy; extracting a third finding from the study of the patient; evaluating, the discrepancy using the model by comparing the discrepancy with the third finding from the study of the patient; outputting, from the large language model, an option for a user to accept, reject, or edit the first finding or the second finding.

[0025] In an aspect described herein is a computer system, comprising: (a) a processor; (b) a display; and (c) a non-transitory computer readable storage medium encoded with a computer program that causes said processor to perform any one of the methods of any one of the preceding embodiments. In some embodiments, the display comprises a virtual reality, augmented reality, or mixed reality display.

[0026] In an aspect disclosed herein is a non-transitory computer-readable storage media encoded with a computer program including instructions executable by a processor to createan application, the application configured to perform any one of the methods of any one of the preceding embodiments.INCORPORATION BY REFERENCE

[0027] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent publications and patents and patent applications incorporated by reference contradict the disclosure contained in the specification, the specification is intended to supersede and / or take precedence over any such contradictory material.BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The novel features of the inventive concepts are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present inventive concepts will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the inventive concepts are utilized, and the accompanying drawings (also “Figure” and “FIG.” herein), of which:

[0029] FIG. 1 shows an example of a report summarization pipeline.

[0030] FIG. 2 shows an example of a discrepancy detection pipeline.

[0031] FIG. 3 shows an example computing system for implementing methods as disclosed herein.

[0032] FIG. 4 shows an example method for generating a sentence or phrase.

[0033] FIG. 5 shows an example method for detecting a discrepancy.

[0034] FIG. 6 shows an example of findings presented in a template.

[0035] FIG. 7 shows an example of findings and a summary presented in a template.

[0036] FIG. 8 shows an example of a worklist.

[0037] FIG. 9 shows an example of a dashboard with findings presented in a template.

[0038] FIG. 10 shows an example dashboard with findings presented in a template.

[0039] FIG. 11 shows the example dashboard of FIG. 10 receiving a voice command.

[0040] FIG. 12 shows an example dashboard of FIG. 10 updated based on voice command of FIG. 11

[0041] FIG. 13 shows an example dashboard with a voice command.

[0042] FIG. 14 shows an example dashboard.

[0043] FIG. 15A shows an example of findings presented in paragraph form.

[0044] FIG. 15B shows an example of findings organized by anatomy and followed by an impression.

[0045] FIG. 15C shows an example of an impression formatted as a bulleted using an LLM prompt.

[0046] FIG. 16 shows an example of a computing device with one or more processors, memory, storage, and a network interface.

[0047] FIG. 17 shows an example platform comprising aspects disclosed herein.

[0048] FIG. 18 shows an example of a web / mobile application provision system providing browser-based and / or native mobile user interfaces.

[0049] FIG. 19 shows an example of a cloud-based web / mobile application provision system comprising an elastically load balanced, auto-scaling web server and application server resources as well synchronously replicated databases.

[0050] FIG. 20 shows an example of selecting a large language model (LLM).

[0051] FIG. 21 shows an example of prior reports summarization.

[0052] FIG. 22 shows an example of discrepancy detection.

[0053] FIG. 23 shows an example annotated finding.

[0054] FIG. 24 shows an example image link to medical finding.

[0055] FIG. 25 shows an example of discrepancy detection.

[0056] FIG. 26 shows an example anatomic navigator.

[0057] FIG. 27 shows an example anatomic navigator.

[0058] FIG. 28A shows an example of user interaction with an anatomic navigator.

[0059] FIG. 28B shows an example of user interaction with an anatomic navigator.

[0060] FIG. 29A shows an example dashboard.

[0061] FIG. 29B shows an example dashboard.

[0062] FIG. 30A shows an example medical report interaction.

[0063] FIG. 30B shows an example medical report interaction .

[0064] FIG. 30C shows an example medical report interaction.

[0065] FIG. 30D shows an example medical report interaction.

[0066] FIG. 31 shows an example phrase or sentence generation for insertion into a medical report.DETAILED DESCRIPTION

[0067] In some embodiments, provided herein are systems and methods for increasing the productivity of a healthcare provider. The systems and methods disclosed herein may be used to generate or evaluate a medical report. In some cases, the systems herein may execute machine-readable instructions stored on non -transitory computer readable media via one or more processors on one or more computing systems. This non -transitory computer readable media may comprise software instructions to execute the methods described herein or to deploy a system as described herein.

[0068] Generally, the generation or evaluation of a medical report may depend on disparate data types such as images (e.g., x-ray, MRI, PET Scan), text (e.g., portions of EMRs, DICOM metadata, HL7 data, patient jackets, patient summaries, impressions), audio (e.g., practitioner voice dictations), or tabular data (e.g., structured numerical data, patient statistics). Each of these disparate data types may contribute unique and valuable data to a holistic understanding of a patient’s health. While multi-modal data describing a patient’s health may in theory facilitate effective treatment and patient-centric care through nuanced representation of patient health, the abundance and complexity of such data may introduce challenges that limit its realized utility. The systems and methods disclosed herein may provide particular advantage in leveraging these disparate modalities to provide or inform practitioners of the full breadth of a patient’s health data, thereby promoting effective treatment and patient-centric care.

[0069] The systems and methods provided herein may incorporate data of a plurality of modalities as input into a machine learning model. This may be useful for ensuring the fullbreadth of patient data maybe used to inform outputs of machine learning models. Generally, these outputs facilitate understanding of patient health, an understanding that is key to effective medicine. In some cases, the plurality of modalities and the data they represent may be referred to as the medical record herein. In some cases, the medical record comprises a portion or an entirety of a patient’s medical history. In some cases, a medical record may comprise contemporary patient data. In some cases, a medical record may comprise recently updated patient data. In some cases, a medical record may be updated overtime as more data for a patient is gathered or generated. For example, the medical record may comprise all or a portion of data associated with a patient including the patient’s treatment history, electronic medical records, medication history, pre-existing conditions, demographic information, family history, mental health history, imaging history, genetics, prior impressions of the patient’s health, prior medical reports describing the patient, reasons for examinations, types of examinations performed, or any other data that would be accessible to a healthcare provider treating a patient. For example, the medical record of a patient may comprise an EMR, DICOM data, HL7 data, or other custom data type for representing patient data. In some cases, the medical record is compiled or pre-processed. In some cases, pre-processing comprises gathering relevant data based on findings of a recent study, user requests, preferences, or computational pre-processing operations such as converting images to vectors.

[0070] The systems and methods herein may facilitate improved patient health monitoring, patient treatment planning, administration of prophylactic medicine, and may generally lower the risk of myopic decision-making in patient care. Further, the systems and methods disclosed herein may provide improvements to computational tools directed to generating and evaluating medical reports. Machine learning tools disclosed herein may reduce computational overhead and improve computational efficiency through direct inference of trained machine learning models. For example, generative algorithms (e.g., LLMs) may excel at zero-shot, few-shot, or in context learning, where pre-trained algorithms may excel at disparate tasks without (or with minimal) explicit training. The field of medicine is particularly suited to implementing such techniques as experts (e.g., doctors, radiologists, medical researchers) are in the loop and able to correct, reject, or accept machine learning outputs, thereby providing sources of expert guided training data that may hone machine learning models over time. As such, the tools disclosed herein may provide significantadvantages for practitioner workflow while also providing systems and methods that may improve patient outcomes.

[0071] The systems and methods herein may be useful for facilitating interpretation and reporting of a medical image. The methods disclosed herein may facilitate interpretation of a medical record. The interpretation of a medical record can be facilitated through the automation or streamlining of aspects of medical report generation, evaluation, quality control, organization, traversal, summarization, or a combination thereof. In some cases, medical records comprise patient health records, imaging studies, doctor impressions, patient summaries, medical records, other document providing patient health data, or a combination thereof. In some cases, traversal herein comprises switching between a current report and a prior report. In some cases, traversal comprises switching between a medical image and a medical text. Switching between the medical image and medical text may be achieved via interaction (e.g., selection within) with the medical image or text. In some embodiments, interaction with the medical image causes the text to be displayed. In some embodiments, interaction with the text causes the medical image to be displayed. Medical images may be shown on a display. In some cases, the display comprises a screen of a computer, laptop, tablet, smart device, or an augmented, virtual, or mixed reality headset. A user may interact with the methods and systems disclosed herein. In some embodiments, the user interacts using tactile input devices, such as mouse and keyboard, buttons, or controllers. In some cases, the user interacts using audio input device, such as a microphone or transcription device. In some cases, the user interacts using a virtual input device, such as virtual, mixed, or augmented reality devices. The user input may allow the system to detect what portion of an image the user is selecting or looking at. The portion of the image may be segmented or labeled according to an image segmentation algorithm. In some cases, image segmentation delineates an anatomic variant or a finding. In addition, user dictation of an evaluation or interpretation of the selected portion (e.g., feature) of the image can be converted into text through audio detection and an analysis algorithm. The combination of user input and voice dictation may be used to generate a finding corresponding to the selected portion of an image. The finding may be constructed based on the context inferred from the user input at the time of the dictation (e.g., shared timestamp), for example, specifying the anatomic segment the user was “looking at” or “pointing at” (e.g., bone, organ, tissue, body part) when dictating an analysis. In some cases, the medical record comprises contemporaneous data of a patient. Forexample, the medical record may comprise a recent imaging study, a lab result, or a referring physician impression.

[0072] Systems and methods disclosed herein may be applied to the findings in a report to determine a discrepancy between findings within a report or between reports. A discrepancy may be between a computer-generated finding and a user finding, a computer-generated finding and a computer-generated finding, or a user finding and a user finding. In some cases, a notice of discrepancy is provided to the user when a discrepancy is determined. The notice may aid with amending, rejecting, or confirming a new finding when finalizing a medical report.

[0073] A medical report herein maybe configured to infer or link one or more findings to the medical record within which they were generated . For example, linking a finding to the most informative image or image segment used to inform the finding. This may facilitate a subsequent viewer of a text finding to interact with the text finding and visualize the same information used by the original user to generate the text finding.

[0074] Generally, a text finding herein may indicate the source of the finding is text. For example, a medical record may be rife with practitioner impressions, summaries of patient health, descriptions of patient symptoms or treatment, etc. . Similarly, an image finding may indicate the source of the finding is an image or frame of a video. In another example, a tabular finding may indicate the source of the finding is tabular data . In some cases, a tabular finding may capture numerical data, data contained in a table, data contained in a graph, data contained in an array, time-series data, or other data organized into rows or columns. For example, a table of tumor diameter overtime may be used to extract a finding describing that a tumor is growing, shrinking, stable, etc.. In another example, an audio finding may indicate the source of the finding is audio. For example, a finding may be extracted from a user voice dictation. Generally, any modality herein may be associated with findings extracted from data of a modality. Further, such findings may be additionally classified as other finding classes herein such as a permanent finding, a transient finding, a key findings, an incidental findings class, or a pertinent negative finding.Report Summarization and Priors Reportins

[0075] Often, a user herein may review a patient’s medical record after a long gap in interaction with the patient or for the first time for a new patient. In some cases, the user maybenefit from getting up to speed on the medical history of the patient. This may include reviewing the patient’s electronic medical record, reviewing images from past radiology exams, or reading past radiology reports. Of these, prior radiology reports may be important to obtain a visual understanding of patient history, but users may have limited time to review the reports and there may be many (e.g., hundreds) of reports for patients with chronic conditions.

[0076] Further, a user may have preferences regarding what they would like to see summarized from prior reports. One way to manage this may be to categorize historical findings into certain buckets and handle these buckets in different ways depending on radiologist preference. For instance, a permanent finding, such as whether a patient had a hysterectomy, may show up on all summaries. However, many findings may be transient and may not be shown in summaries. Bucketing, or grouping, findings based on a finding class may facilitate summarization of such reports. In some cases, a findings group may also facilitate accommodation of user preference. For example, a user may request each relevant permanent finding be presented in a bulleted list.

[0077] As an example of the utility of the systems and methods herein, radiologists (a user) may prefer to review findings relevant to a current case in summaries. Relevance may be related to anatomy (e.g., the “field of view” of a study), or date, or pertinent negatives (e.g., normal findings that are still meaningful in a particular context). For example, a radiologist may desire to review a chronologically ordered series of a tumor to view its growth or shrinkage. How summaries are displayed (e.g., bullets, paragraphs) and how radiologists can interact with those summaries are a part of whether they will be useful to the radiologist or not. In addition to categorization, the order in which findings are summarized may also be important for review.

[0078] As such, in some cases herein are systems and methods facilitated by LLMs to address some of the challenges faced by users when reviewing medical data. For example, systems and methods herein may receive a medical record and a summary template selection, extract a plurality of findings from the medical record, determine a finding class and a relevancy category for at least a portion of the plurality of findings, and generate the summary of the medical record based at least in part on the selected summary template selection, one or more of the plurality of findings, and the finding class and relevancy category of the one or more findings.

[0079] In some cases, the summary template selection comprises a label for the finding class of at least one of the one or more findings. In some cases, the label comprises a text color, a text highlighting, a text font, an icon, or any combination thereof. In some cases, the finding class comprises a permanent finding class, a pertinent negative finding class, an incidental finding class, a transient finding class, or any combination thereof. In some cases, the finding class comprises any other classification for determining which findings are of importance to the examination / patient. In some cases, the permanent finding class represents findings regarding a patientthat are persistent or are life-long (e.g., type 2 diabetes mellitus). In some cases, marking such findings helps a user to interpret the medical data in light of such findings, which being chronic, may be important towards their analysis. In some cases, pertinent negative findings relate to pathologies or illnesses that are not chronic, but that may be important to the subject of the summary or examination. Such a label allows a user to quickly view the most relevant findings to save time and ensure that such findings are addressed in their analysis. In some cases, the transient finding class represents an abnormality that is worthy of note but impertinent to the subject of the study / examination. Displaying the transient finding class label allows a user focus on more other pertinent findings, while remaining visible if found to be pertinent. In some cases, the summary of the medical record does not comprise the finding with a determined transient class. In some cases, the summary of the medical record comprises the finding determined to have a transient class if a user comment is received regarding that finding. In some cases, an incidental finding class represents an unexpected finding. Displaying the incidental finding class label may guide a user’s attention to a potential finding for further analysis.

[0080] In some cases, the relevancy category comprises the specific relevancy category if the finding is related to an anatomy, a pathology, or both, associated a medical report. In some cases, the medical reportis part of the medical record. In some cases, the relevancy category comprises the specific relevancy category if the finding is related to an anatomy, a pathology, or both, that is the subject of the medical report. For example, a title or other meta-data of a DICOM file may be used to determine a relevancy category. In some cases, the relevancy category comprises the specific relevancy category if the finding is related to an anatomy, a pathology, or both, associated with a title of the medical report. In some cases, the relevancy category comprises the specific relevancy category if the finding is related to an anatomy, a pathology, or both, that is the subject of the medical report. In some cases, the relevancy category comprises the specific relevancy category if the finding is related to an anatomy, apathology, or both, associated with a title of a prior medical report regarding the patient. In some cases, the relevancy category comprises the specific relevancy category if the finding is related to an anatomy, a pathology, or both, that is the subject of a prior medical report regarding the patient. In some cases, the use of the specific relevancy category enables the formation of a summary focused on a specific anatomy or pathology of the medical report.

[0081] In some cases, the relevancy category comprises a global category relevancy if the finding is determined to be of key importance, regardless of the related anatomy or pathology. A finding can be of key importance if the finding is life-threatening (e.g., can or is likely to cause death or serious complications within a year). In some cases, the use of the global category relevancy allows the formation of a summary that reviews all the patient’s findings and conditions for an overall understanding of their condition. In some cases, the relevancy category is determined at least in part based on the finding class. In some cases, a finding labeled as having a permanent finding class, a transient finding class, or both may also be labeled as having a global category relevancy.

[0082] In some cases, the summary template comprises: a preferred order for the one or more of the plurality of findings based at least in part on the determined finding class, which findings to display based at least in part on the determined finding class, or both. In one example, a summary template can include instructions to display only x-ray images and to show findings chronologically in bullet points.

[0083] In some cases, a summary is generated based on Digital Imaging and Communications in Medicine (DICOM) data, Health Level Seven (HL7) data, or both. In some cases, the DICOM data comprises DICOM metadata. Generally, medical imaging data herein may comprise meta-data that describes circumstances of the medical image. For example, metadata may include when an image was taken, where it was taken, who took the image, how the image was taken, or what the image is representing (e.g., anatomy). In some cases, medical images are received and an anatomy shown in the medical image is determined, wherein the relevancy category is further determined based the determined anatomy.

[0084] Summarizing new or existing patient data may be useful for providing informed treatment recommendations. Summarizing patient data may allow for a healthcare provider to quickly assess a patient and their health conditions. Providing a summary of information in a patient’s medical record may help a healthcare provider avoid spending time reading throughnumerous patient charts in order to gain an understanding of the new patient’s medical information. Summarization may be particularly useful when a new healthcare provider begins treating a patient or when a patient’s medical information has been updated by a third party such as another doctor or specialist. Generally, a user may want a summary of prior reports on the patient. For example, a user reviewing a chest x-ray may want a summary on all findings related to a user’s cardiovascular or respiratory health. To illustrate, the user may request of a system herein that a summary of prior treatments of respiratory issues for a patient.

[0085] In some cases, a patient may undergo a series of general tests intake operations, specific tests, or a combination thereof. Such testing and intake operations may result in large amount of multi-modal patient data may be difficult to evaluate in a short period of time. In some cases, general tests include tests that are received during a “physical” or general checkup. For example, a general test may comprise a blood panel, a urinalysis, an eye test, an ear test, a heart monitoring, or a blood pressure measurement. In some cases, patients may undergo other intake procedures such as demographic, behavioral, medical history, or symptom questionnaires. Using this abundance of data to inform diagnoses, prognoses, or treatment plans may prove challenging given the multi-modal, inhomogeneous nature of even relatively small amounts of patient health data. This challenge maybe exacerbated in patients with complex existing medical histories. Further, time, aging , doctor switching, specialist involvement, comorbidities, chronic ailments, or severe medical events may result in exorbitant amounts of distributed, complex, stylistically distinct forms of data for a single patient. As such, the generation and evaluation of summative medical reports, both in the general context (e.g., basic, governing aspects of overall health) and specific contexts (e.g., specialist visit, imaging study), may substantially facilitate every stage of patient care. Report summarization may be particularly useful for physician consultations. For example, when a consulting physician reviews a patient case, a summary of a referring physician’s findings or prior reports may be provided to a consulting physician so they may review the case in a timely manner.

[0086] The summarization of patient data may be of particular value to radiologists evaluating hanging protocols showing imaging data of a patient. In some cases, summarization comprises contextualizing dictated findings of a radiologist (e.g., verbalizing impression of imaging data) in a medical report. Contextualizing may comprise considerationor prior medical data (e.g., prior reports) or linking findings inserted into a medical report with the image such that the image may be efficiently retrieved based on interaction with the medical report. In some cases, contextualizing comprises presenting of findings in context. Context may include information relating to the patient, such as, for example, a finding related to a prior medical image or prior medical report. For example, a finding may be a simple sentence or fact describing a patient. In some cases, the finding describes a doctor’s observation, a doctor’s analysis of lab results, a doctor’s analysis of an image, or other commentary of a doctor that comprises commentary on the health of a patient. As such, a finding may be, for example, “MRI of brain reveals lesion in fontal cortex.” This finding may have context dependent meaning not reflected in the finding alone. As such, the systems and methods herein may present the finding in a context that clarifies, for example, the reason for the scan, a summary of a prior scan or patient evaluation, implications of the finding, or other information clarifying the finding. In some cases, the context may be sourced from the medical record. For example, the medical record may comprise information about prior evaluations of a brain lesion such that a report generated to comprise the finding may include details of the progression of the brain lesion .

[0087] In some cases, disclosed herein are systems and methods for generating a medical report. The generation of medical reports may be performed using aspects of natural language processing (NLP), computer vision (CV), or a combination thereof. In some cases, NLP comprises rules-based, ontological, or machine learning methods. In some cases, large language models (LLMs) perform or leverage NLP and CV tasks. For example, multi-modal LLMs may function as one or both an agent of NLP or CV. Additionally, or alternatively, LLMs may perform a portion of the operations disclosed herein and form a portion of an overall system or ontology implementing software comprising LLMs, CV, and NLP. In some cases, generating a medical report may comprise summarization of prior medical reports, summarization of findings of a contemporary study, or both.Illustrative Summarization Workflow

[0088] Report generation may utilize a process shown in FIG. 1. As shown, a summarization module may receive an input prompt. Generally, the summarization module may perform summary generation as disclosed herein. In some cases, the input prompt comprises imaging context. Imaging context may comprise information related to a medical image, such as, for example, a finding associated with a current image data, raw current image data, a findingassociated with a prior image, raw prior image data, pathology data, a prior patient report, a user preference, or a combination thereof. In some cases, the imaging context may be a portion of the medical record_of a patient. Generally, raw image data may refer to pixels of an image and their associated color value. In some cases, a color value may comprise a color contribution (e.g., RGB values), a greyscale value, a brightness, a hue, or an opacity . In some cases, an image finding, may comprise a medically relevant portion of an image. For example, a full body MRI may comprise small regions of medically relevant information. To illustrate, a full body MRI may reveal a plurality of gastrointestinal abnormalities that may form a plurality of image findings. In such a case, the plurality of gastrointestinal abnormalities may be the medically relevant portion of the image. In some cases, clinical context may comprise DICOM data, pathology reports, HL7 data, genomic data, and electronic medical records. Generally, clinical context may comprise non -image data. In some cases, one or both of clinical context or imaging context may comprise a portion of the medical record. The input prompt may be used as input into a multi-modal LLM model which may generate a medical summary. In some cases, the medical summary may be used in a patient jacket or medical report. In some cases, a patient jacket may comprise data of a patient organized into sections for review. For example, the patient jacket may contain a medical summary, a list of findings organized by type or anatomy, a list of links to prior reports, an image of the patient, etc.

[0089] In some cases, a medical report comprises the medical summary. In some cases, the medical summary comprises an impression. Generally, an impression integrates the patient’s clinical context, signs, symptoms, and history to describe potential underlying causes of observed radiologic findings. An impression may be a clinical summation of information or an opinion that is the result of a clinical assessment process, such as a preliminary diagnosis or working diagnosis. An impression can be based on a patient's signs, symptoms, medical history, or a combination thereof. The impression could describe what the medical provider believes is causing those signs and symptoms. In some cases, an impression provides a summary of findings. In some cases, the summary of findings is contextualized by a plurality of findings from a medical record. The systems and methods disclosed herein may provide particular utility in contextual enrichment of impressions or other medical summaries by historical patient data from a medical record. For example, an impression generated by summarization may be supplemented with a link to an imaging study comprising an image finding or supplemented with data from the medical record. In some cases, an impressionmay comprise contemporary study and prior report findings. In some cases, an impression may comprise recommendations. In some cases, an impression may comprise action items. For example, an impression may comprise a diagnosis of medical issue and a recommended treatment plan. In some cases, an impression may comprise information not found in a contemporary study or a prior report. For example, the impression may automatically recommend a treatment for a condition that is inferred to be present based on content within a contemporary study or a medical report. In some cases, the impression may comprise user dictated findings supplemented with a recommendation.

[0090] In some embodiments, the impression is automatically generated. In some embodiments, the impression is automatically generated based on a medical report. In some embodiments, the impression is generated according to a user. The systems and methods herein may change the impression based on a user preference or the specialty of the user. The impression may change even though the contents of the medical report do not change. For example, the impression may demonstrate a first finding and a second finding to a first user, and a second finding and a third finding to a second user. To further illustrate, an impression may recite a cardiovascular issue (e.g., heart trauma) to a cardiologist, but may recite a liver issue (e.g., liver trauma) to a hepatologist.

[0091] A report summarization module may provide a medical summary. In some cases, a summarization module may execute report summarization. The medical summary may comprise a finding. The medical summary may comprise a key finding, a permanent finding, a transient finding, an incidental finding, or a pertinent negative finding. In some cases, the medical summary comprises a plurality of findings of a plurality of modalities. A finding herein may be, or may be derived from, a medical image or a medical summary as disclosed herein. In some cases, a medical summary may be general. For example, a medical summary may provide a holistic summary of a medical record. In some cases, a medical summary may be specific. For example, a medical summary may be specific for a current study. For example, the medical summary may provide a summary of relevant image studies given a current image study.

[0092] A permanent finding may comprise information about a patient that may be not change for the remaining life of a patient or other extended period of time. For example, a permanent finding may comprise patient history information such as chronic conditions, prior surgeries, allergies, demographics, etc.. A permanent finding may be useful for providingcontextual information that may inform a treatment or diagnosis. For example, a hemophilic patient may not be eligible for certain treatments.

[0093] A transient finding may be a new finding or other finding that is prone to significant change. For example, a transient finding may comprise information describing a temporary state such as a sign or symptom, an acute condition, lab result, etc.. In some cases, a sign or symptom may comprise a patient reported symptom. In some cases, a sign or symptom may be a doctor measured symptom. For example, a sign or symptom may comprise a fever, swelling, headache, nausea, etc.. In some cases, an acute condition may comprise an infection, heart attack, broken bone, trauma, tumor, organ function disruption, etc.. The transient finding may describe a short-lived, temporary, or potentially resolvable patient condition.

[0094] A pertinent negative finding may be a finding that indicates the absence of an expected condition. For example, an expected bone break, colon polyp, tumor, etc. may be the basis of a pertinent negative finding. In some cases, a pertinent negative finding may be informed by a medical record or a current study. For example, a medical record may indicate that the user has a rectal polyp that requires monitoring. As such, an evaluation of the patient that see no polyps may be of importance to a reviewing physician. In some cases, the pertinent negative finding may be informed by a current study. For example, a lack of fractures in an x-ray of a patient receiving x-rays for an expected bone break may comprise a pertinent negative finding.

[0095] An incidental finding may be a finding that is unexpected given a medical record or a current study. For example, a chest x-ray evaluating a patient for a broken rib may reveal a lung tumor. Accordingly, an incidental finding may be generated to describe the lung tumor. This may be highlighted for a user such that they review and address the incidental finding. In some cases, machine learning may provide particular utility by mitigating issues of user bias by broadly evaluating an image for findings. For example, a user reviewing an image for a broken bone may be myopic in their review and miss incidental findings. Alternatively, a machine learning model may evaluate the image for findings and present all findings. In some cases, how the finding is presented may be dependent on a current study or a medical record. For example, in one context a lung tumor may be a transient finding that is a subject of a study while in another it may be an incidental finding that is unexpected for a given study. In some cases, an incidental finding may be contained in a different report section.

[0096] A key finding may comprise a permanent finding, a transient finding, an incidental finding, or a pertinent negative finding. In some cases, a finding may be a key finding due to the finding’s value or relevance to a current study or report. For example, a user generating an impression of a patient’s health status may provide or have generated (e.g., by an ML model) a plurality of findings about the patient’s health for inclusion in the impression. In some cases, during impression generation, the impression maybe supplemented by a findings of the medical record. As such, a measure of relevancy may be used to determine which findings in the plurality of findings in a medical record are key. For example, a permanent finding of osteoporosis may be a key finding when generating an impression for a user evaluating an x-ray and may be included in the impression regardless of the user’s commentary when reviewing the x-ray. In some cases, the measure of relevancy is a score output by a machine learning model trained to predict relevancy. In some cases, the relevancy is an output of an ontology of set of software instructions configured to determine relevancy of a given finding. For example, a machine learning model, ontology, or software may be configured to take as input a general context such as “x-ray evaluation of femur” and output the relevancy of any arbitrary finding. Illustrating the example, the software may output a high relevancy score for a finding reciting “patient has osteoporosis” and may output a low relevancy score for a finding reciting “patient is allergic to penicillin.”

[0097] In some cases, a key finding, a permanent finding, a transient finding, incidental finding, or a pertinent negative finding may be a computer-generated finding. Generally, a computer-generated finding may be a finding that is output by processors executing a set of instructions encoded on non-transient computer readable media. For example, a machine learning model may be used to take as input a prior report of a patient and output a list of key findings, permanent findings, transient findings, incidental findings, and pertinent negative findings. In some cases, the machine learning model is a multi-modal large language model.

[0098] In some cases, a permanent finding is used to generate a medical summary. In some cases, a machine learning (ML) model utilizes a permanent finding to provide a medical summary comprising a transient finding related to said permanent finding. For example, if a patient has a permanent finding of liver cirrhosis, the medical summary can include transient findings related to the liver cirrhosis, such as lab results related to liver function, medical image findings related to liver function, or a combination thereof. The permanent findings may describe the patient (e.g., male, diabetic, allergic to penicillin) while the transientfindings may provide information relating to acute conditions (e.g., patient presenting with enlarged spleen). As such, a permanent finding may contextualize a transient finding as in “a diabetic male presents with an enlarged spleen.” A transient finding may be relevant to a specific context and may not be presented in other iterations of a medical summary. In some cases, a transient finding presented to a first healthcare provider is not presented to a second healthcare provider. The determination of whether to present the transient finding may be dependent upon the specialty of a healthcare provider. In some cases, the first healthcare provider is a specialist in a first field, whereas a second healthcare provider is a specialist in a second field. For example, a first specialist in a first specialty may not be interested in a transient finding associated with a second specialty. To illustrate, a cardiologist may be interested in transient findings relevant to heart health but may not be interested in transient findings pertaining to a recent bone break. A transient finding or a permanent finding may be derived from an image finding. Generally, whether a finding is transient or permanent is independent of the modality type that relays the finding. Additionally, the permanent or transient finding may be a computer-generated or provided by a medical provider provided. In some cases, an ML model utilizes a transient finding to provide a medical summary comprising a second transient finding related to said transient finding. For example, if a patient has a transient finding of a myocardial infarction (MI), the medical summary can include a second transient finding related to said MI, such as a lab result related to heart health (e.g., a cardiac biomarker).

[0099] In some cases, a finding is scored. In some cases, the finding is scored based on relevancy. Relevancy may include a hard-coded rule based on an ontology, a score output by a machine learning model, or a measure of similarity in a latent space of trained machine learning models. For example, “broken bone” may be encoded into a similar region of a latent space as an image of a broken bone. As such, the two may be linked via the learned association of “broken bone” and x-rays of broken bones. In some cases, the relevance of a finding is scored by an algorithm. In some cases, an input prompt may be configured assign relevancy scores based on a specialty. For example, a cardiologist may use an input prompt that indicates to an LLM that the LLM should primarily consider findings in a medical record comprising cardiovascular related findings. In some cases, the input prompt may comprise specific instructions for a kind of study being performed or preferences for a given specialty or specialist. The input prompt may act as a template or otherwise guide the formatting of the medical summary. For example, the input prompt may dictate formatting preferencescomprising formatting of text, organization of text, display of text, display of images, or a combination thereof. In some cases, the input prompt dictates formatting of text. In some cases, the input prompt dictates organization of text. In some cases, the input prompt dictates a display of a medical image. In some cases, the relevancy of a finding is determined by a user. For example, a user may request the inclusion of a finding in a medical summary which may indicate a high degree of relevancy. In a further example, the user may remove or otherwise indicate that a finding is not relevant to a current study. Collection of such user indicators may be applied to reinforcement learning or other adaptive learning techniques to fine-tune models or input prompts. Further, the relevancy of a finding may be tied to an anatomy or a pathology that may be a focus of a study. The finding may come from a prior (e.g., earlier) medical report or medical summary. For example, a finding of a prior medical summary directed towards a broken foot may not be found relevant to a current study directed towards lung imaging. Additionally, a finding of a prior medical summary directed towards a liver biomarker may be found relevant to a current study directed towards liver imaging.

[0100] Relevancy may influence the presentation of findings, such as in, for example, a hanging protocol. A medical summary may indicate one or more image findings that may be relevant to display one or more images. The relevancy of an image finding may dictate its inclusion, exclusion, or position in a hanging protocol. For example, a first image may be presented in a first position of the hanging protocol and a second image may be presented in a second position of the hanging protocol if the first image is more relevant than the second image. In some cases, the relevancy of an image finding dictates its inclusion, exclusion, or position in a medical summary. For example, a first image finding may be presented earlier in a medical report or within a medical summary, whereas a second image finding be presented after the first image finding or excluded from the medical summary if the first image finding is more relevant than the second image finding. In some cases, the order of the images presented may be based on user preference, relevancy, or both. For example, a user m ay prefer that images are always presented chronologically. As such, images may be ordered based on a date the image was generated. In some cases, the order of the images can be based on relevancy and user preferences. For example, the user may prefer that the top five most relevant images are presented first regardless of chronology, while the remaining images are ordered chronologically.

[0101] In some cases, a link associated with a medical image is provided. In some cases, the link is provided in the medical image, a medical summary, a medical report, or a combination thereof. Inclusion of the link may be determined by the relevancy of the finding. The link may associate a medical image with an image finding. A link in a medical summary may be associated with a medical image, such that selection of the link retrieves the medical image. Additionally, a link in the medical image may be associated with the medical summary, such that selection of the link retrieves the medical summary or a report comprising the medical summary. Linking may be useful for allowing quick retrieval of relevant medical images from the medical summary. For example, a medical summary may comprise an image finding describing a malignancy on the liver. The image finding may be linked to the medical image from which the image finding was derived such that the image may be automatically or manually (e.g., upon selection by audio, tactile, or virtual input) retrieved.

[0102] The medical summary may comprise a portion of information from a prior medical summary or a portion of a prior medical report. The inclusion of the information from the prior medical summary or a portion of a prior medical report can be determined based on the relevancy of the information. Information from a prior medical summary or prior medical report may comprise prior image findings or prior text, tabular, audio, or other modality findings found in a medical summary or report. For example, user feedback may be incorporated into future summarizations. In some cases, a prompt or model parameter (e.g., which may be weighted through training) is updated based on user feedback. User feedback may hard-coded user feedback or learned user feedback. For example, hard-coded user feedback may comprise instructions to display information in a same or different way as desired by a user. In some cases, the hard-coded user feedback in incorporated directly into an input prompt to reflect a user preference. For example, the prompt may be configurable via user selectable options, via verbalized or written user requests, or via user actions (e.g., store a user approved or edited summary for use as later template).

[0103] In some cases, the summarization module comprises an LLM model. In some cases, the summarization module comprises a multi-modal LLM. The summarization module may be used to provide a summary of information, such as provided in an impression section or a medical summary. The input to the multi-modal LLM may be an input prompt comprising one or more of the prior patient reports, medical summaries, medical images, lab results, or a combination thereof. In some cases, the summarization module comprises a plurality ofcomputer vision or natural language processing sub-modules. For example, a first submodule may be responsible for extracting image findings via computer vision while a second sub-module extracts findings from text or other modality. In some cases, the prompt for use in the summarization module is generated entirely or in part by a pre-processing pipeline. A pre-processing pipeline may comprise an ontology, machine learning model, or software instructions that build an input prompt. For example, the same LLM may be used in a cardiovascular context as in a neurological context, the use of the LLM differing by an input prompt used to guide the LLM. For example, computer vision may be used to analyze a medical image. The computer vision may extract an image finding from the medical image such that the image finding may be placed into an input prompt as natural language or via a direct feed into a tokenized representation of the input prompt. In some cases, natural language processing is used to analyze a medical report. The NLP may extract entities, sentiment, groundings, findings, or summaries of aspects of the medical record or to extract findings for insertion into a new or updated medical summary or report.

[0104] The multi-modal LLM may be trained upon or otherwise implement a foundation model as disclosed herein. In some cases, a foundation model may be used via fine-tuning, transfer learning, in -context learning, model grafting, etc.. The use of foundation models may provide performance benefits by providing a model that is language aware. For example, a model trained on the English language may be tuned to a model that understands jargon used in the context of the English language. Furthering the example, radiology data (e.g., EMR, DICOM data, patient jackets, medical reports, medical summaries) may provide a domain (radiology) specific corpus from which the foundation model may learn. Additionally, or alternatively, a domain model may be trained on broader corpus of medical data such as patient jackets, patient communications, medical reports, medical papers, journal articles, etc.. This domain-aware foundation model may then be well suited for further fine-tuning to specific tasks such as summarization, impression generation, report generation, discrepancy detection, calculation of relevancy scores, finding extraction, or report analysis.

[0105] In some cases, report summarization as described herein may be configured to provide longitudinal evaluation across prior reports. For example, a medical record may comprise a plurality of datapoints regarding a condition of a patient. In some cases, there may be tens to hundreds of datapoints regarding a condition of a patient. For example, a permanent finding such as type 1 diabetes may result in a lifetime of evaluations of patientinsulin levels. As such, prior reports may be evaluated for findings describing a similar pathology, anatomy, illness, condition, etc. and used to generate a textual, graphical, or tabular summary of the findings. In another example, a medical record may be evaluated for a plurality of measurements of a lung nodule. Accordingly, a graph of the diameter of the lung module over time may be generated based on the findings describing the lung nodule diameter.

[0106] In some cases, longitudinal evaluation across prior reports may be used to compare at least two findings that were gathered at separate times. In some cases, a second finding was recorder in a medical record at least about one hour, one day, one week, one year, or longer after a first finding. In some cases, longitudinal evaluation may be used to compare a parameter of a finding. For example, a finding may describe an anatomy, pathology, or other condition. In some cases, a finding may comprise a parameter such as size, width, density, color, diameter, shape, thickness, severity, weight, location, or other property of an anatomy, pathology, or other condition. In some cases, an LLM as described herein may be used to determine a parameter of a finding.Discrepancy Detection Pipeline

[0107] In some embodiments, provided herein are systems and methods for detecting a discrepancy. In some cases, the discrepancy is detected in the medical report. In some cases, a medical summary as described herein may be subject to discrepancy detection. The discrepancy detection may be utilized to provide an over-read of a medical report. Providing an over-read may allow for a review of the medical report before it is included in the patient file or it may allow for a medical report to be updated after its inclusion in a patient file. The discrepancy detection provided herein may review a medical report and a medical image to confirm a finding in a medical report (e.g., a finding that is related to a medical image). Thus, the discrepancy detection may function as a second opinion, which may be useful for ensuring that user findings are accurate. Furthermore, since discrepancy detection may be automated, the process of reviewing the medical report may be expedited compared to a review conducted by a human peer. Additionally, the discrepancy may be detected in the medical report by comparing the medical report to a second medical report for the same patient case. Such a comparison may prove useful in training a new user of the system or a new physician. For example, a patient case may be evaluated by a new physician and an experienced physician. The medical report generated by each of the new physician and theexperienced physician may be compared in order to demonstrate where variations in the report exist. This may be particularly useful with the systems and methods that can compare findings of users who generated the findings using different language to describe the same finding.

[0108] In some cases, a discrepancy may indicate a the a first finding and a second finding in a medical report are inconsistent. For example, a first finding in an impression that a patient is in remission may be inconsistent with an incidental finding in a medical image of the patient that reflects a new or growing malignancy. In another example, an anatomy described in a finding that would not be visible in a current study, such mentioning foraminal stenosis or a vertebral level in an ankle exam, may result in a discrepancy. Generally, a discrepancy may be a logical or factual inconsistency between or among findings in a medical report, optionally among data modalities.

[0109] A machine learning pipeline (e.g., machine learning pipeline 200) as shown in FIG. 2 may be used with the systems and methods disclosed herein to provide discrepancy detection or over-reads. In some cases, an over-read may comprise a second opinion. In some cases, reports as disclosed herein may be used as a portion of an input into a large language model (e.g., large language model 205 (herafter LLM 205)) configured to detect the presence of a discrepancy of a report (e.g., medical report 210). For example, a prior report of a patient may contain contradictory information with respect to a report. Illustrating the example, a medical report may indicate that a patient presents with a lesion on the superior left lobe of the patient’s lungs when a prior report indicates the prior lobectomy of the superior left lobe . The input may further comprise a portion of other medical record data. The medical record data may comprise one or more data modalities. The medical record data may further comprise electronic medical records (EMR), DICOM data, HL7, proteomic data, pathology report data, genomic data, transcriptomic data, phenomic data, or other omic data. The one or more more data modalities may comprise a visual modality, an audio modality, tabular data, or other custom data datatype defined for a particular medical purpose. In some cases, the one or more data modalities may comprise data types used in software such as objects, dictionaries, lists, sets, arrays, matrices, etc.

[0110] The LLM 205 may be configured to provide discrepancy detection via an input prompt 220. In some cases, the input prompt may be provided prior to later iterations of input into the LLM. For example, an input prompt may be used as input into an LLM to provide itwith directions for behavior to use when processing later inputs. For example, an LLM may be instructed by an input prompt to extract findings of various types from later inputs into an LLM. In some cases, the input prompt may comprise a hard prompt, a soft prompt, a zeroshot prompt, a few-shot prompt, or a chain of thought prompt. The input prompt may comprise a portion of the medical record disclosed herein. The input prompt may further comprise one or more preferences of a practitioner using an LLM. In some cases, a portion of the prompt may be generated based on one or more rules such the input prompt is configured to include relevant information from the medical record. In some cases, retrieval augmented generation (RAG) may be used to gather relevant information from the medical record. For example, based on a medical report, relevant patient data of a patient’s medical record may be obtained through a RAG framework configured to retrieve findings pertinent to the medical report. The input prompt may comprise one or more instructions for how to detect a discrepancy between a medical report and the medical record. In some cases, a portion of the input prompt may comprise a trainable vector (e.g., soft prompt) such that the input prompt is configured to detect discrepancies through a training operation. In some cases, the summarization module disclosed herein may implement similar prompting methods as the LLM 205. An LLM may take as input separately or in combination an input prompt, a medical report, or the medical record. For example, a medical report may be input into an LLM along with a portion of a medical record prior to incorporating the medical report into the medical record. This may be performed for quality control and consistency to promote inclusion of findings into a medical record that are consistent. The output of an LLM may comprise one or more discrepancy in any suitable form. For example, the output of an LLM may comprise one or more of a label, a class, a real number value, a natural language output, a finding, or a latent representation of the discrepancy . In some cases, a latent representation may comprise a data encoding. For example, the encoded data within an LLM may be used directly as input or injected into tokens or encodings of another LLM. In some cases, the output of LLM may comprise a portion of a prompt used as input into another LLM. For example, an output of LLM 205 may be used as input into LLM 230. In some cases, an LLM may be trained on individual data or a collection of patient data. For example, an LLM may be trained or fine-tuned on a medical record of a patient. In another example, an LLM may be trained on a plurality of medical records of a plurality of patients. In some cases, an LLM may be trained on a corpus of data comprising medical records for a plurality of patients and fine-tuned on a patient’s medical record. In some cases, data used to train an LLM maycomprise radiology reports, DICOM data, HR7 data, etc.. In some cases, data used to train an LLM may be used in a pre-training operation to convert a foundation model into a domain aware model. In some cases, an LLM may be fine-tuned on learning tasks. For example, LLM 205 may be trained in a learning operation to detect one or more discrepancies from a training set of historical medical reports labelled with errors by a user. In some cases, an LLM configured or trained to detect discrepancies may be used to review a database of patient medical records to detect discrepancies of non -subject medical records. In this example, non-subject refers to a patient not actively undergoing evaluation by a practitioner.[oni] A discrepancy detected by a first LLM may be passed to a second LLM. For example, an output of LLM 205 may be passed to LLM 230. In some cases, an LLM may be a multimodal LLM that may take as input one or more more data modalities that may comprise image data, audio data, tabular data, or other custom datatype defined for a particular medical purpose. In some cases, any format of data desired for use as input may be applied through the machine learning method disclosed herein. For example, data may be used as input via model grafting, prompt engineering, generative pre-training etc.. The second LLM may be configured to assess the one or more discrepancy detected by a first LLM. The input into into the second LLM may comprise a portion of the input into the first LLM, a discrepancy, or additional data 225. In some cases, additional data 225 may comprise medical record data or contemporary study or report data. For example, a contemprorary imaging study may be used to evaluate a medical report for a discrepancy prior to inclusion into a patient’s medical record. To illustrate, a discrepancy may indicate an expected lesion on a liver image.Accordingly, a recent image of the liver may be evaluated for a finding comprising the lesion. Additional data used as input into a second LLM may comprise a result of a recent patient study comprising an image of the patient, a data of the patient, a medical report of the patient, or a pathology data of the patient. In some cases, data of the patient may comprise a lab result, such as a cell count, blood pressure, omics data, heart rate, body temperature, or other clinical feature. The second LLM may perform operations to determine the presence of the one or more discrepancies. In some cases, the second LLM comprises a single machine learning model. In some cases, the second LLM comprises one or more sub-modules to perform one or more operations. For example, a sub -module of the one or more sub-modules may comprise a portion trained to perform a specific generative, classification, or regression task. The second LLM may output a suggestion for how to handle the detected discrepancy.For example, the seocnd LLM may provide one or more suggestions for phrases to incorporate into a medical report.

[0112] In some cases, the output of the second LLM may be used to request input of a user 235. For example, a user may provide an acceptance or rejection of the one or more suggestion of the second LLM. Additionally, or alternatively, the second LLM may present a discrepancy along with an indication that the second LLM disagrees with the first LLM. In some cases the second LLM may provide a degree of confidence in its agreement or disagreement with a discrepancy predicted by the first LLM. In some cases, the second LLM may provide a confidence for a discrepancy predicted by the first LLM. For example, the second LLM may be prompted to provide a numerical or categorical estimate of its confidence in a discrepancy. As such, the user may choose to accept or reject each of the one or more discrepancies. Additionally, or alternatively, the user may provided an opportunity to edit one or more suggested phrases for incorporation into a medical report. Further, the second LLM may be configured to provide one or more most relevant images, prior medical reports, or other patient data to the user for manual discrepancy verification. The second LLM may provide the one or more most relevant images, prior medical reports, or other patient data to the user based on their semantic similarities to the one or more discrepancies predicted by the first LLM. In some cases, a semantic similarity comprises a similarity between two findings as determined by a machine learning model or ontology . Any of the one or more discrepancies selected for incorporation by the user may be used to generate an updated medical report text 240.

[0113] In some cases, the machine learning pipeline 200 may gather as feedback 245 a selection, a ranking, a rejection, an edit, or a preference of a user pertaining to one or more interactions of the user for use in reward modeling or updating an LLM . For example, the user may select, reject, or rank a plurality of suggestions from the second LLM, thereby providing indicators for how to augment, supplement, or otherwise change a medical report to account for a discrepancy. The collection of such feedback may provide particular utility for updating a machine learning model, LLM, or system as disclosed herein due to the highly reliable nature of the expert provided feedback.

[0114] In some cases, the machine learning pipeline 200 may provide particular utility in patient health monitoring. The ability to detect a discrepancy between or among reports and, in some cases, contemporary health data, provides the ability to monitor the progression orregression of a disease state. For example, a malignancy with an expected shape, width, density, or other visually identifiable parameter may be identified by checking a medical report for one or more discrepancies. A change in a finding of a medical report may indicate valuable health information that may be subtle or otherwise difficult to account for in typical, non-machine learning or non -LLM workflows. Further, the systems and methods disclosed herein that extract findings from images, text, or other data modality and insert them into a report provides particular utility in the persistence of findings over time for both machine learning and visual analysis. For example, automated detection of relevant findings in a medical record for addition to medical report may reduce risk of foregoing important past patient data in a medical report.

[0115] In some cases, the systems and methods herein may be configured to receive a medical record and a first prompt, generate a first medical report based on the medical record and the first input prompt receive a second input prompt and generate a second medical report using the second input prompt. This may provide particular utility in generating medical reports based on user preference. For example, the same LLM may be configured to output a different result for a same set of data based on an input prompt. T o illustrate, a first user may provide in a first prompt a direction to present lung findings in a bulleted list. Alternatively a second user may provide in a second prompt a direction to present lung findings in a paragraph style summary.Computer Systems

[0116] The methods disclosed herein may be implemented using example computing system 300 as shown in FIG. 3. The computing system 300 maybe a user device. A user device may be any device with a display for interacting with the systems and methods disclosed herein. For example, a user device may comprise a mobile device, a personal computing device , or a wearable device. In some cases, a wearable device comprises a smart watch, virtual reality device, augmented reality device, smart glasses device, or smart lenses device. The computing system 300 may be used for medical report generation or medical report evaluation. For example, the computing system 300 may implement the systems and methods described with respect to FIGs. 1 or 2.

[0117] The computing system 300 may comprise one or more processor 305 for executing machine-readable instructions directed to the computer-implemented methods disclosedherein. The computing system 300 may further comprise an input 310 for receiving data. For example, the input 310 may comprise a software module for interacting with an external API, a physical port for uploading data stored on an external storage device, a software module for receiving data from an external server or database , or a physical or virtual connection to one or more healthcare devices, databases, or platforms to obtain patient data. For example, the computing system 300 may implement or communicate with a PACS or NVA system. The input 310 may comprise a tactile, audio, or virtual input as disclosed herein. For example, the computing system 300 maybe in communication with a secondary device such as mouse and keyboard, a controller, a microphone, a voice audio recorder, smart glasses, smart lenses, or a wearable device.

[0118] The computing system 300 may comprise a display 315. The display 315 may be configured to a show a graphical user interface (GUI) 320 for evaluating a medical image. The GUI 320 may, by the one or more processor 305, show a dashboard 325. The dashboard 325 may comprise an anatomic navigator 330. The anatomic navigator 330 may be used to extract, store, or provide coordinates of findings. For example, the anatomic navigator may be used to show medical images and image findings or anatomic variants. The coordinates may be used to efficiently present an image or portion of an image from which one or more findings was extracted. For example, a finding in a medical summary may be interactable and comprise a link such that interaction with the link retrieves or displays an appropriate medical image. The medical image may be displayed such that a coordinate associated with a finding is readily visible. In some cases, readily visible may indicate that a coordinate or associated finding is enlarged, highlighted, annotated or zoomed-in. In some cases, the anatomic navigator facilitates the retrieval of a medical image based on a finding in medical summary. In some cases, the anatomic navigator facilitates the retrieval of a medical summary based on an image finding. In some cases, the anatomic navigator may use machine learning to navigate between one or more data modalities. For example, the anatomic navigator may be configured to retrieve an appropriate image or images based on a text finding. To illustrate, the image may show a hairline fracture in a bone and a text may describe the same hairline fracture. Additionally, or alternatively, the anatomic navigator may be configured to generate or retrieve text relevant to or associated with an image or images. In some cases, two trained encoders may learn a shared embedding space between at least two data modalities. This may provide a foundation for generating or retrieving images that most closely align with a text describing an image or findings of the image. For example, a natural language description ofa patient and a finding in an image of the patient may be used to find a relevant image or images in a patient image dataset as determined by a machine learning model trained to associate text and images. In the reverse situation, an image may be used to retrieve or generate text describing the image. For example, an image comprising a finding may comprise a latent representation similar to a latent representation of text in a report describing said image. In such a way, the report comprising the text with a latent representation similar to the image may be retrieved. Additionally, or alternatively, the text may be generated using the image as context for said generation. This may provide particular utility for generating summaries or reports by automating the incorporation of appropriate, grounded (e.g., in a medical record) findings in descriptions of medical images.

[0119] The anatomic navigator 330 may provide navigation among text and image representations of a finding. In some cases, the finding may describe an anomaly, pathology, malignancy, key finding, permanent finding, transient finding, incidental finding, or pertinent negative finding. In some cases, a pertinent negative finding may indicate a lack of an expected finding. For example, a lack of a tumor where one was previously found may be a pertinent negative finding. The anatomic navigator may present a text or image representation of the medical finding on the display 315. The anatomic navigator 330 may comprise a portion of a dashboard 325 comprising a medical image x-ray report, a summary, or an interactive function. The one or more interactive functions may comprise computer vision tools through which the user may highlight, interrogate, annotate, or segment image portions of interest. In some cases, the computer vision tools may be automatically applied based on a user preference or a similarity between a current case and prior similar cases. In the manual or automatic scenario, aspects of an image highlighted, interrogated, annotated, or segmented may be added to a report. The addition to the report may be generated automatically or manually through text or audio input. In some cases, automated reports may be provided by the systems and methods described herein. The addition to the report may comprise a finding and may comprise a link for insertion into one or both of the report or the image. The link may be used to navigate to an image via selection in the report or to the report via selection in the image. For example, an automatically or manually highlighted, interrogated, annotated, or segmented finding in an image may be selected through user interaction to retrieve a report where the finding is discussed. In some cases, user interaction may comprise a mouse click, a verbal command, or eye-tracking. In the case where the finding is not found in a report, the finding may be inserted into a report contemporaneous with analysis. In a further example, alink in a medical report may be used to retrieve one or more associated images for inspection. The one or more images may be provided to the user with any type of finding herein highlighted, annotated, or segmented. A highlighted finding may, for example, comprise a bounding box output by an object detection algorithm or drawn by a user. An annotated finding may, for example, comprise a label as predicted by a classification or generative algorithm or input manually by a user. A segmented finding may comprise a plurality of highlighted pixels resultant from a segmentation operation. The highlighted pixels may indicate a mask overlaid on the image that may be a result of, for example, a semantic, instance, or panoptic segmentation algorithm, or a result of manual selection by a user.

[0120] In some cases, a link as described herein may be used for co -registering two or more images, reports, or combinations thereof. For example, a finding in a report may be coregistered with a plurality of images. To illustrate, a user may read in a report that a malignancy on a kidney has progressively shrunk through the course of treatment. As such, a finding describing that malignancy may be linked to, or co-registered, with a plurality of medical images of the malignancy. Those images may be presented to the user to review the images summarized by the finding. In some cases, the images may be presented to a user. In some cases, prior reports associated with the images may be co-registered. In some cases, a user may request to be shown the images through a voice command. For example, a user may request “show me the imaging studies for this patient related to the kidney malignancy.” In some cases, a plurality of reports of a patient may be co-registered. For example, a finding in a medical report may describe a patient condition over time. To illustrate, a finding may describe a patient as complaining of frequently complaining of acute migraine headaches. Accordingly, a finding reflecting this complaint in a medical report may be used to access co - registered medical reports relevant to complaints of acute migraine headaches.

[0121] In some cases, the natural language processing, computer vision, or other machine learning operations disclosed herein may be executed by a machine learning module 335 of the computing system 300. The machine learning module 335 may perform training, fine- tuning, reinforcement, generative, or inference operations as disclosed herein. In some cases, the machine learning module 335 may comprise the summarization module disclosed herein . The machine learning module 335 may be configured to perform automatic or on-demand analysis of medical reports and images. For example, a user preference may indicate that the user desires a specific analysis to be automatically performed for a study type. In a furtherexample, on-demand analysis may comprise a requested analysis via user engagement with an image such as a verbal request to analyze a region of anatomy, a written request to perform an analysis, a manual selection of an analysis tool, or an interaction with a display of a wearable device.

[0122] In some cases, virtual reality as disclosed herein may indicate a fully virtual experience that comprises no aspects of a user’s true surroundings. In some cases, augmented reality may indicate an overlay of digital elements over a user’s true surroundings. For example, augmented reality may comprise a floating radiological analysis station similar to those of the visual component of a PACS system. In some cases, mixed reality may indicate an overlay of digital elements over a user’s true surroundings that may interact with a user’s true surroundings. For example, x-ray images may be projected onto a body of patient. In some cases, each of virtual reality, augmented reality, or mixed reality may be controllable, navigable, or interactable by eye-tracking or finger tracking. In some cases, each of virtual reality, augmented reality, or mixed reality may be implemented using glasses, lenses, goggles, or headsets. For example, a mixed reality hanging protocol may comprise displayed images that are readily interacted with. Furthering the example, an image of the hanging protocol may be subject to annotation, visual manipulation, analysis , inclusion in summarization or discrepancy detection, or other technique as disclosed herein without the need for a physical or non-virtual screen.

[0123] In some cases, a storage 340 of the computing system 300 may be used to provide directly (e.g., via local storage) or indirectly (e.g., retrieve from external storage) images, medical reports, medical records, or medical data. The direct or indirect provision of images, medical reports, medical records, or medical data may comprise communication with a picture archiving and communication system (PACS), a vendor neutral archive (VNA), radiological information system (RIS), hospital information system (HIS), secure cloud storage solution, secure local storage solution, or other data storage or data management system.Computer Implemented Method

[0124] The methods disclosed herein maybe implemented to provide an over-read or second opinion of a medical report. An example of this is shown in operations 400 of FIG. 4. Overreads may be conducted to verify or otherwise compare with previously evaluated studies.The methods disclosed herein may provide particular utility in providing automated overreads that may replace or supplement the use of second or third-party over-reads. An overread may be provided herein to confirm, propose, or challenge a finding of a user. This may facilitate the inclusion of accurate findings by leveraging machine learning models trained on a large corpus of medical data. Such machine learning models may be less prone to limitations of bias, experience, or attentiveness, providing safeguards on evaluation of medical data. Further, the machine learning models disclosed herein may have access to prior patient data and may be configured to automatically detect discrepancies, anomalies, or otherwise unexpected findings in a report or medical data.

[0125] The process of providing over-reads may comprise operation 405. Operation 405 may comprise reviewing a medical report using a first machine learning model. The first machine learning model may output a finding of a medical report. A finding of a medical report may comprise a key finding, a permanent finding, a transient finding, an incidental finding, or a pertinent negative finding. A finding may be a finding of a contemporary study inserted either partially or entirely by a practitioner or a machine learning model. Alternatively, or additionally, a finding may be a finding of a prior report or contained within a medical record. In some cases, a finding of a prior report may be updated based on a contemporary report prior to insertion into a medical report. Retention of a prior report finding in a contemporary report may facilitate patient monitoring or informed diagnosis or rational treatment recommendation. A finding may be derived from a number of data modalities disclosed herein. For example, a report may comprise findings from text data, image data, tabular data, or audio data. A finding from text data may be a manually entered finding from a user or a generated finding from a machine learning model. A finding from image data may be manually entered or a generated finding from a machine learning model. A finding from tabular data may be manually entered or a generated finding from a machine learning model. A finding from audio data may be manually entered or a generated finding from a machine learning model. In some cases, the first machine learning model may be an LLM or a multimodal LLM.

[0126] In some cases, a finding of operation 410 may comprise reviewing a second medical report using a second machine learning model. In some cases, the second machine learning model is the same machine learning model as used in operation 405. For example, an updated prompt or additional context from a prior report may be fed into the machine learning modelto facilitate its verification of a previously detected discrepancy. In some cases, the second machine learning model is a different machine learning model. A finding may be extracted from the second medical report in the same or similar fashion as with operation 405. In some cases, the second medical report may be a contemporary or prior medical report. In some cases, a difference in behavior or operation between the machine learning model of operation 405 and 410 may be due to prompt engineering. For example, an output of a machine learning model comprising an LLM may be influenced by an input prompt. To illustrate, a model using the same underlying model weights may output different answers if asked to extract findings versus when asked to detect discrepancies. In some cases, a difference between the machine learning model of operation 405 and 410 may be due to a difference in modalities or composition of training data used. In some cases, a difference between the machine learning model of operation 405 and 410 may be a difference in architecture. For example, layers may be added or removed from a neural network to alter or fine-tune its behavior. In some cases, the difference in architecture or training data may provide at least two distinct feature extraction methods such that machine learning models implementing the at least two distinct feature extraction methods are less prone to missing a finding or a discrepancy in findings. This may provide built-in over-reads ortriple checking (e.g., human in the loop), where the model of operation 405 may detect a discrepancy between a medical report and one or more prior medical report, and the model of operation 410 may further evaluate the medical report to assess for concurrence with the discrepancy proposed by the model of operation 405.

[0127] In some cases, the operation 415 may comprise detecting a discrepancy between the medical report and the second medical report. Each of the first and second machine learning model may provide one or more finding from the first and second medical report, respectively. The one or more finding of the first machine learning model may be derived from a medical report describing a patient. The medical report may comprise a description of one or more aspects of a medical record including medical history, demographic data, treatment history, imaging history, pathology history, prior medical summary, or other medically relevant data. In some cases, the medical report of 405 is a contemporary report describing a patient. In some cases, the operation 405 may result in one or more finding determined from the medical report. In some cases, the one or more finding determined from the medical report comprises a discrepancy between a finding and other medical data of the patient. In some cases, the other medical data may comprise medical record data such as priorEMR data, omics data, DICOM data, HL7 data, pathology reports, or other prior patient medical data. A discrepancy between a finding and the other medical data of the patient may indicate and error in the medical report or a novel finding in the medical report. An error may be resolved through user intervention. For example, a user may be directed to the error and asked to confirm, reject, or otherwise resolve the error. The detection of a discrepancy in this way may provide particular utility in early detection and treatment of patients. The operation 415 may comprise instructing an LLM through a prompt to compare the medical report with the other aspects of the medical record. For example, an LLM of operation 415 may be configured to evaluate data input into the LLM. For example, an input prompt may be configured to direct an LLM to detect a discrepancy in medical data.

[0128] In some cases, the operation 415 may be performed by a multi-modal LLM. A discrepancy may be evaluated for similarity with a plurality of findings extracted by the multi-modal LLM. The multi-modal LLM may compare the discrepancy with one or more of current (contemporary) image data, pathology data, or prior image data. A multi-modal LLM may evaluate the similarity of an embedding of the discrepancy with an embedding of each of one or more finding extracted from the current image data, the pathology data, or the prior image data. In some cases, an embedding may be alternatively referred to as a latent representation herein. A high embedding similarity may indicate that a discrepancy is a false positive or that the discrepancy requires further evaluation. For example, a discrepancy may not inherently indicate an incorrect finding. To illustrate, a finding of a malignancy marked as a discrepancy may indicate a lapse in cancer remission or a new cancer diagnosis for a patient rather than an error. In another example, a pertinent negative finding may result in a discrepancy. For example, a prior history of kidney stones following by imaging studies indicating a lack of kidney stones may trigger a discrepancy. In some cases, however, the discrepancy may indicate an error in the medical report that may be remedied by the user either manually or automatically update the medical report.

[0129] In some cases, operation 420 may comprise generating one or more sentence or phrase describing the discrepancy. The one or more sentence or phrase describing the discrepancy may be a portion of an output of a machine learning model used to detect the discrepancy between the medical report and the second medical report. A system, device, or software implementing operations 400 may be configured to provide the one or more sentence or phrase to the user. The user may edit, reject, or accept a sentence or phrase beforeincluding or excluding the suggestion for the medical report. In some cases, the user may edit, reject, or accept the sentence or phrase via an input modality comprising manual interaction, voice interaction, or eye or finger tracking based interaction.

[0130] The methods disclosed herein may be implemented to detect a discrepancy in a medical report as shown in operations 500 of FIG. 5. A discrepancy may indicate a difference in a finding from one source and a finding in another source. A source may comprise any of the data modalities or types of medical data disclosed herein. In some cases, the source may comprise past data or contemporary data. In some cases, the medical record may comprise the past data. The detection of discrepancies between a medical summary and a medical image may provide particular utility in patient monitoring, treatment, or diagnosis. For example, a misalignment of an expected and a measured finding may indicate a change in patient health status. For example, a user or machine learning model may expect a decrease in tumor size overtime due to treatment, but updated imaging studies may indicate an increase in tumor size. An indication of a change in patient health status may indicate numerous medically valuable insights, for example the effectiveness of a treatment, the remission of a patient condition, the inception of a patient condition, the progression of a patient condition, or the presence of an anomalous patient condition. In some cases, an early detection of a patient health status may substantially influence both a prognosis and course of treatment for a patient.

[0131] In some cases, an imaging study as disclosed herein may comprise analysis of one or more images. For example, an imaging study may comprise an image hanging or hanging protocol. An image study may comprise analysis of the one or more in the context of other data modalities such as text, audio, video, or tabular data. An imaging study may be conducted on any display disclosed herein.

[0132] In some cases, operation 505 may comprise providing a medical report comprising a medical image and a medical summary. The medical report may be provided by a user for analysis. Alternatively, the medical report may be automatically provided as part of a quality assurance, or over-read, workflow. For example, a system implementing the methods disclosed herein or operations 500 may analyze each medical report generated by the system for discrepancies between the medical report and prior reports. As such, the medical report may be provided by a user, another computer-implemented method, or another software module.

[0133] In operation 510, a medical summary portion of the medical report may be used for detecting a finding in the medical summary. In some cases, detection of the finding may be performed by a machine learning model trained to detect findings in a medical summary. In some cases, the machine learning model may comprise a natural language processing model. For example, a supervised training operation me be used to train a machine learning model to extract findings from medical summaries using labelled training data. In some cases, the detection of the medical summary finding may be performed by an LLM that is configured by prompt engineering to detect a finding in the medical summary. For example, an LLM may be provided one or more instructions for how to detect a finding in a medical summary. The instruction may be a simple command (e.g., hard prompting, soft prompting) or a series of operations (e.g., chain of thought prompting) to guide the LLM to detect a medical summary finding.

[0134] In operation 515, a medical image portion of the medical report may be used for detecting a finding in a medical image. In some cases, detection of the medical image finding may be performed by a machine learning model trained to detected image findings from a medical image. In some cases, the machine learning model may comprise a computer vision model. For example, a supervised training operation me be used to train a machine learning model to extract findings from medical images using labelled training data. In some cases, the detection of the medical image finding may be performed by an LLM that is configured by prompt engineering to detect a medical image finding in the medical image. For example, an LLM may be provided one or more instructions for how to detect a finding in a medical image. The instruction may be a simple command or a series of operations to guide the LLM to detect a medical image finding.

[0135] In some cases, the same machine learning model used to detect a finding in a medical summary and a finding in a medical image. For example, the machine learning model may be a multi-modal LLM model configured to extract features from at least image and text data. The multi-modal LLM may have a text encoder and an image encoder trained to provide embeddings of a correlated text and image pair that are organized proximal to one another in the latent space of the multi-modal LLM. In this way, an image, and a description of the image, may share a similar representation in the learned latent space of the multi-modal LLM. As such, the multi-modal LLM may have been trained to understand semantically orcontextually similar image and text pairs as representing a similar concept. In this way, a multi-modal LLM may be prompted to compare the instant data modalities.

[0136] Operation 520 may comprise comparing the medical summary finding with the medical image finding using a multi-modal machine learning model. In some cases, the multi-modal machine learning model may comprise one or more models working in sequence or in parallel. In operation 520, the medical summary finding and the medical image finding may be provided as input into a machine learning model (e.g., an LLM, multi-modal LLM, series of machine learning modules). In some cases, the summary finding and the medical image finding may be compared for similarity or distance in a latent space of a machine learning model. A threshold value of similarity between a medical summary finding embedding and a medical image finding embedding may be established to determine the presence or absence of a discrepancy. In some cases, a confidence score may be provided directly to a user, where the confidence score indicates a likelihood or probability that a discrepancy is present. In some cases, the threshold value may be used internally in an LLM to guide its output. In some cases, the machine learning model may be trained to detect one or more discrepancies between at least two portions of an input. In some cases, the input may further comprise the medical report. In some cases, the machine learning model may be prompt engineered to detect a discrepancy between at least two portions of an input. In some cases, the machine learning model may be both trained and prompt engineered to detect a discrepancy between at least two portions of an input.Systems and Methods for Report Summarization

[0137] The methods and systems disclosed below may comprise computer-implemented methods (e.g., as shown in FIGs. 1, 2, 4, 5), computer systems (e.g., as shown in FIG. 3), or machine executable instructions. For example, a technique embodied as a computer- implemented method may be executed by one or more processors of one or more computing systems. Additionally, or alternatively, a method embodied in a computer system may comprise one or more processor and a non -transitory computer readable storage medium encoded with a computer program that causes the one or more processors to perform operations of a method or technique disclosed herein. Further, the operations of a method or technique may be stored on a non -transitory computer-readable storage media encoded with a computer program including instructions executable by a processor to create an application for performing an action.

[0138] Disclosed herein are systems and methods for generating a summary of a medical record. Such a technique may comprise operations including (1) receiving the medical record and a summary template selection, (2) extracting a plurality of findings from the medical record, (3) determining a finding class and a relevancy category for at least a portion of the plurality of findings, and (4) generating the summary of the medical record based at least in part on the selected summary template selection, one or more of the plurality of findings, and the finding class and relevancy category of the one or more findings. In some cases, the use of templates to facilitate summary generation may address issues related to context switching For example, a user interacting with existing report software may constantly switch between tabs and images during dictation. This may break attention and challenge effective reporting.

[0139] Alternative to context switching, systems and methods herein may allow a radiologist to simply state a few key findings in the images, and then leverage all of the context known about that study, including a preferred template, prior reports and a patient’s medical record, to generate a well formatted, grammatically correct final report. This may save radiologists enormous amounts of time and cognitive effort. Further, prior reports and images can be infused into the final report with the additional context those data provide, for instance populating report fields with relevant data from prior reports. The ability for a computer to intelligently synthesize a report from just a few findings and relevant context has broad advantages.

[0140] FIG. 31 shows one illustrative embodiment of a flowchart for generating a priors - based medical report. As shown, a context dependent summarization model prompt receives imaging context comprising current image data, prior image data, and pathology data. The context dependent summarization model prompt further receives prior patient reports, user preferences, and clinical context comprising DICOM data, pathology reports, HL7 data, genomic data, and electronic medical records. The prompt is then sent to a multi-modal LLM model which generates a medical summary and a patient jacket.

[0141] In some cases, prior summaries are constructed using a context sensitive prompt to a multi-modal LLM model. Either a generalist multi-modal model or model grafting approach may be used with foundation model to optimize performance. Prior reports from the p atient may be analyzed in appropriate clinical context based on information summarized from imaging content and from a medical record. In some cases, prior patient data or a medical record may be evaluated in parallel such that findings or summaries from the prior patientdata or a medical record is readily available during user interaction. User feedback may be used to further optimize the context-dependent prompt and fine-tune a multi-modal model. In some cases, users are queried for preferences (e.g., formatting and information to be included or excluded) upon introduction to the tool and can update these preferences as necessary.

[0142] In some cases, the systems and methods herein maybe configured to receive a report template from a practitioner, a finding regarding the patient, and a prior medical report associated with the practitioner, the patient, the finding, or any combination thereof, and generate the medical report based at least in part on the report template, the prior medical report, and the finding. In some cases, the use of prior reports to generate a current report promotes repeatability of report generation and improved accuracy.

[0143] In some cases, the report template comprises a plurality of fields. In some cases, the medical report is generated by determining one or more of the plurality of fields is represented by the finding and inserting key finding into at least a portion of the one or more determined fields. In some cases, the medical report is generated by determining one or more of the plurality of fieldsis represented by a portion of the prior medical report and inserting the portion of the prior medical report into the one or more determined fields. In some cases, the finding is received verbally. In some cases, a template may be established for a particular audience. For example, a template may be established to present findings to specialist. In another example, a template may be established to present findings to a referring physician.

[0144] In some cases, a report feedback is received from a practitioner regarding the medical report wherein the medical report is further generatedbased on the report feedback. In some cases, a medical record of the patient is received wherein the medical report is further generated based on the medical record. In some cases, a finding class of a finding is determined wherein the medical report is further generated based on the finding class. In some cases, the finding class comprises a permanent finding class, a pertinent negative class, a transient finding class, an incidental finding class, or any combination thereof.

[0145] In some cases, systems and methods herein may receive a finding from a user regarding a medical record, store the finding in association to the medical record, receive a request to generate a first medical report and a selection of a first template, generate the first medical report according to the first template based at least in part on the medical record and the finding, receive a request to generate a second medical report and a selection of a secondtemplate, and generate the second medical report according to the second template based at least in part on the medical record and the finding.

[0146] In some cases, a report template may be a summary template. In some cases, the summary template comprises a label for the finding class of at least one of the one or more findings. For example, a finding associated with an anatomy, anatomic variant, a permanent finding , a pertinent negative finding, or a transient finding of interest may be associated with a label in a template for the anatomy or anatomic variant. The template may be used to structure a report or a portion of a report describing the one or more findings of the finding class. This may facilitate visual inspection of the one or more findings for a given study or report. In some cases, the label comprises a text color, a text highlighting, a text font, an icon, or any combination thereof.

[0147] As an example, a medical report may be generated according to a report template. In some cases, the report template may comprise a finding dictated by a user or generated by a machine learning model. For example, information about a pulmonary nodule may be extracted via image analysis shown. For example, the pulmonary nodule may be determined to comprise a 9.3 mm measurement. Further, because the system may access a patient’s medical record, the system maybe able to generate a finding relaying that the nodule is stable over time. In this example, the existence of this degenerative disease is considered to be stable. An LLM herein may adapt this language and style of the user and have a custom interaction or output so that the report content matches what a practitioner would prefer. In some cases, the systems herein further receive feedback, such as, for example, “the next time that I report growth over time for this type of examination, include the prior measurements.”

[0148] In some cases, the finding class comprises a permanent finding class, a pertinent negative finding class, a transient finding class, incidental finding class, or any combination thereof. In some cases, the summary of the medical record does not comprise the finding with a determined transient class. In some cases, the summary of the medical record comprises the finding determined to have a transient class if a practitioner comment is received regarding that finding. For example, a natural language processing algorithm may classify a finding as transient and temporarily forego inclusion or highlighting of the transient finding in a medical report. However, if a user comments on the transient finding, the transient finding may be deemed of value to include or highlight in the medical report. In some cases, natural language processing may be used to compare one or more entity extracted from user input orinteraction data to an entity of the transient finding to determine a match or alignment between a user input and a possible transient finding. In some cases, user input or interaction data comprises user generated audio, text, mouse clicks, keyboard actions, touches (e.g., with touch screen), or manipulation of AR / VR / MR visual outputs.

[0149] In some cases, the relevancy category comprises a specific relevancy category if the finding is related to an anatomy, a pathology, or both, associated with the summary. In some cases, the relevancy category comprises a global category relevancy if the finding is determined to be of key importance, regardless of the related anatomy or pathology. In some cases, the relevancy category is determined at least in part based on the finding class. For example, a finding associated with the permanent finding class may be more likely to be assigned to global relevancy due to a high likelihood of relevancy to another condition. Furthering the example, a chronic condition such as type 1 diabetes may be globally relevant. In some cases, importance may be determined by a natural language processing algorithm trained to assign importance to findings. For example, a sentiment analysis or similar algorithm may be used to evaluate user input for a sentiment of an input. In some cases, a degree of importance may be a sentiment predicted. Additionally, or alternatively, importance may be established by an ontology comprising one or more rule for assigning importance to a finding. For example, chronic conditions may be known to be relevant to overall health and may be identified via a rule or rules of an ontology as an aspect of patient health to incorporate in a ubiquitous or near ubiquitous manner when generating reports or evaluating for discrepancies. In some cases, such conditions may be relevant to nearly all specialists or doctors.

[0150] FIG. 6 shows one illustrative embodiment of a global summary of a medical report regarding a CT abdomen and pelvis scan with contrast having findings labeled with a relevancy category comprising the global relevancy. In some cases, as shown, the template input / selection further allows a user to provide feedback to improve the summary generation process. In this example, the feedback is to “next time include the previous measurements when suggesting growth of pulmonary nodules over time.” This final report is ready for review by a practitioner and has synthesized all of the current findings dictated by the radiologist with information in the prior reports to generate a final report ready for review.

[0151] In some cases, the summary template comprises (1) a preferred order for the one or more of the plurality of findings based at least in part on the determined finding class, (2) which findings to display based at least in part on the determined finding class, (3) or both.

[0152] In some cases, the summary is further generated based on Digital Imaging and Communications in Medicine (DICOM) data, Health Level Seven (HL7) data, or both. In some cases, the computer program further causes the processor to receive a medical image and determine an anatomy shown in the medical image, wherein the relevancy category is further determined based the determined anatomy. For example, a rib bone may be assigned a lower relevancy or to a different relevancy category in a chest x-ray than an unusual mass detected in a region associated with a lung. As such, relevancy may be determined in part by a reason or motivation for a study. In some cases, the relevancy may be calculated during analysis of the image based a computer vision algorithm. In some cases, the relevancy category may be updated based on annotation or comment by a user. In some cases, the relevancy category is an anatomy. For example, a finding may be related to the abdomen and will be assigned with findings associated with the abdomen. In some cases, the relevancy category may be augmented by a relevancy score. For example, a plurality of findings may receive a relevancy score and be ordered in part by or filtered in part by a relevancy score.

[0153] FIG. 7 shows one illustrative embodiment of a specific summary of a medical report having findings labeled with a relevancy category comprises the specific relevancy. As shown, the findings on the left-side of the image, based on prior studies shown in the bottomright, are converted into a summary at the top-right. In this example text 701 comprises a first color as a label for the transient finding class, a second color for text 702 as a label for permanent finding class, and a third color for text 703 as a label for key findings. In some cases, a key finding may be chronic or transient finding. In some cases, a key finding may comprise a high relevancy score. For example, a first permanent finding may have a high relevancy score and therefore be a key finding to be highlighted in a contemporary summary. In another example, in a different circumstance, the first chronic may receive a low relevancy score and be excluded from a contemporary study. Generally, any finding herein may at certain times be a key finding as dictated by a relevancy of a finding. In this example the transient class finding is a finding that the gallbladder is “significantly more distended than on the prior study. The wall is not thickened intrahepatic ducts are not distended.” In this example, the permanent finding is of a prior hysterectomy, and the key findings relate to a“6mm pulmonary nodule in the left lower lobe” and “grade 1 degenerative spondylolisthesisL4 in relation to L5.

[0154] In some cases, a finding may be color coded according to its type. For example, a transient finding may receive a first color, a permanent finding a second color, a pertinent negative finding a third color, and so on.

[0155] In some cases, the computer program further causes the processor to receive a summary feedback from a practitioner regarding the summary, and wherein the summary of the medical record is further generated based on the summary feedback.

[0156] Another technique disclosed herein may be for performing an action related to a medical report. As such, the technique may comprise operations including (1) receiving a verbal command, (2) applying a machine learning algorithm to determine an intent based on the verbal command, (3) and performing an action related to the medical report based on the intent. In some cases, an intent may comprise a purpose or objective of a user’s input. In some cases, intent may be explicit such as a direct command of a user. In some cases, intent may be implicit such as implied by context. For example, a radiologist requesting of a system herein to view an x-ray of a patient may be requesting the x-ray because the patient is at a hospital due to a recent bone break. As such, the system may implicitly understand that the user wants to analyze the x-ray for bone breaks.

[0157] In some cases, the intent, the action, or both may cause a change or adjustment in (1) an arrangement of one or more images or data displayed on the graphical user interface, (2) one or more image or data displayed in the graphical user interface, (3) a display setting of an image or data displayed in the graphical user interface, (4) determine an image or data that mentions a key word, (5) sign the medical report, (6) display another medical report, (7) transmit a copy of the medical report, (8) measure a distance in the medical report, (9) display a key finding in the medical report, or (10) any combination thereof.

[0158] The verbal command may, for example, be provided by a user commenting on or navigating (e.g., anatomic navigator) a system implementing the methods disclosed herein. For example, the user may wish to scroll, zoom, expand, or otherwise visually manipulate an image to improve interpretability. Additionally, or alternatively, the user may wish to command the system to perform an analysis. For example, the user may request that a portion of the image is evaluated. In some cases, the user may request the analysis of anatomicvariant. The user command may be used to resolve one or more question or knowledge request. For example, the user may wish to know if prior studies had been performed for a subject. In some cases, the user may command the system to incorporate a finding or impression into the medical report.

[0159] The extraction of one or more intent from a user’s verbal command may indicate an operation to perform. For example, a user intent may be determined by a natural language processing algorithm such that the intent is used to perform one of the analyses, generations, actions, or other processes as disclosed herein. In some cases, a natural language processing algorithm may comprise an LLM. In some cases, the computer program further causes the processor to receive an action feedback from a user regarding the performed action, and wherein the action is further performed based on the action feedback. For example, the user and the system may perform iterative actions based on a command of the user and effectuation of the command by the system. In some cases, the feedback of the user may provide a measure of user satisfaction with the performance of the system for use in improving the system or updating the system to accommodate user pref erence.

[0160] Another technique disclosed herein may be for generating a medical report regarding a patient. The technique may comprise operations including (1) receiving a report template selection from a practitioner, a key finding regarding the patient, and a prior medical report associated with the practitioner, the patient, the key finding, or any combination thereof, and (2) generate the medical report based at least in part on the report template, the prior medical report, and the key finding.

[0161] In some cases, the report template may comprise a plurality of fields. For example, the medical report may comprise a field for a type of finding as disclosed herein. A field may comprise a standard field for a type of study or a field inserted by a user in prior similar cases or earlier in a history of a current case. The field may be inserted contemporaneously to report generation via a voice or other command. In some cases, the medical report is generated by determining one or more of the plurality of fields is represented by the key finding and inserting the key finding into at least a portion of the one or more determined fields. In some cases, the medical report is generated by determining one or more of the plurality of fields is represented by a portion of the prior medical report and inserting the portion of the prior medical report into the one or more determined fields. In some cases, the key finding is received verbally.

[0162] In some cases, the computer program further causes the processor to receive a report feedback from a practitioner regarding the medical report, and wherein the medical report is further generated based on the report feedback as disclosed herein. In some cases, the computer program further causes the processor to receive a medical record of the patient (e.g., from a PACS system, EMR, etc.), and wherein the medical report is further generated based on a patient’s medical record. In some cases, the computer program further causes the processor to determine a finding class of the key finding and wherein the medical report is further generated based on the finding class. For example, in some cases, a finding may be first identified as relevant to a current study and later be classified as chronic or transient. In some cases, the finding class comprises a permanent finding class, a pertinent negative class, a transient finding class, incidental finding class, or any combination thereof as disclosed herein.

[0163] FIG. 8 shows one illustrative embodiment of a list of medical reports. FIG. 9 shows one illustrative embodiment of images and a summary of a medical report arranged in a graphical user interface. In this example, scan images of the patient are shown arranged on the left and the template input is shown on the right. In some cases, the arrangement of the images can be adjusted with a voice-generated prompt.

[0164] In some cases, a report template as disclosed herein may comprise a first template format. The systems and methods disclosed herein maybe used to fill the report template. In some cases, filling the report template may comprise placing relevant images, findings, summaries, impressions, or other aspect herein in an appropriate location of a template . For example, a finding relevant to a contemporary study may be inserted into an impression section. In another example, a permanent finding may be inserted into a general medical summary of the patient. In another example, a plurality of findings of different types may be organized and color coded and presented in a section of a medical report describing an aspect of a patient (e.g., as in FIG. 7). Generally, templates may be dependent on user, institution, or other factor that may render templates heterogeneous among users. Accordingly, the systems and methods disclosed herein may be used to provide translation among template formats. For example, the first template may be used as a source of data input into a second template. This may provide particular utility when a second user wishes to review patient data that has been previously reviewed by a first user. In another example, a same user may switch to the second template format. As such, the information may be translated from the first templateformatto the second template format. As disclosed herein, findings may be of a plurality of types, relevancies, or both that may provide sufficient context for properly dispersing information from one template to another. In some cases, this process may be mediated by a computer system as described herein implementing one or both of machine learning techniques or ontologies to perform translation between templates.

[0165] Another technique disclosed herein may be for generating a medical report. As such, the technique may comprise operations comprising (1 ) receiving a finding from a practitioner regarding a medical record, (2) storing the finding in association to the medical record, (3) receiving a request to generate a first medical report and a selection of a first template, (4) generating the first medical report according to the first template based at least in part on the medical record and the finding, (5) receiving a request to generate a second medical report and a selection of a second template, and (6) generating the second medical report according to the second template based at least in part on the medical record and the finding. In some cases, the finding comprises a dictated key finding. For example, user commentary through verbal, tactile, or virtual input may indicate that a finding is a key finding. A key finding may be additionally or alternatively be identified as a key finding by natural language processing. As such, a key finding may be automatically or manually determined to be a key finding. In some cases, a key finding may be determined to be a key finding according to a relevancy score, relevancy category, or both. For example, in generating an impression on a chest x-ray, a finding of tumor in the upper left lobe of a lung may be determined to be a key finding.

[0166] In some cases, the first template and the second template each comprise a plurality of fields. In some cases, storing the key finding comprises determining one or more of the plurality of fields is represented by the key finding and further associating the determined one or more fields with the key finding. In some cases, the computer program further causes the processor to receive a report feedback from a practitioner regarding the first medical report and wherein the second medical report is further generated based on the report feedback.

[0167] FIG. 10 shows one illustrative embodiment of a first arrangement of images and data displayed in a graphical user interface. Shown in the presented scan is a pulmonary module. FIG. 11 shows one illustrative embodiment of a voice command received regarding the first arrangement of images and data displayed in a graphical user interface. In this example the voice command is to “open previous study where the pulmonary nodule was first noted.” The systems and methods as described herein may be used to identify the study where the nodulewas first noted. In response to this voice command, a second arrangement of images and data is displayed in the graphical user interface, per FIG. 12. In some cases, an image presented to the user may be automatically evaluated for a finding.

[0168] FIG. 13 shows one illustrative embodiment of a voice command received related to the second arrangement of images and data displayed in a graphical user interface. In this example, the voice command is a key finding that a “small bowel obstruction from a right lingual hernia is noted, with signs of ischemia.” A third arrangement of images and data is displayed in a graphical user interface, per FIG. 14, to show that the key finding was recorded. Alternatively, In some cases, the dictated finding is stored and used to generate the final report. Alternatively, In some cases, the dictated finding is stored and used to generate the final report without changing the graphical user interface.

[0169] FIG. 15A shows an exemplary key finding. FIG. 15B shows an exemplary summary generated by a voice command and including the key finding of FIG. 15 A. FIG. 15C shows a voice instruction to write the impressions as a bulleted list, with the received output of the summary written in bullet point form. Similarly, voice instructions could be used to modify the tone or complexity of a report to accommodate different audiences.

[0170] In some cases, voice commands or voice dictation may be used to perform automatic report completion. In some cases, automatic report completion may comprise filling out a report or report template automatically. In some cases, report completion may comprise generating a summary. This may facilitate user focus while analyzing a medical image or other medical report. Generally, systems and methods disclosed herein for automated completion of medical report during dictation may provide particular utility during evaluation of diagnostic images. For example, the systems and methods disclosed herein may facilitate user evaluation of medical images without a break in concentration to switch between aspects of analysis software (e.g., an image viewer and a reporter). As such, the systems and methods herein may capture and distribute findings to portions of a medical report during dictation. Such automated reporting may be performed during execution of methods herein. For example, such as during navigation of the anatomic navigator.

[0171] In some cases, automatic report completion comprises matching of entities of dictated speech with appropriate locations in a medical report or report template. For example, dictated speech may be evaluated for insertion in one or more of a procedure, technique, comparison, impression, summary, findings, or other section of report template.

[0172] Another technique disclosed herein may be for generating a medical report. As such, the technique may comprise operations including (1) receiving a medical record, a first input prompt, and a first machine learning parameter, (2) generating a first medical report based on the medical record and the input prompt using the first machine learning parameter, (3) receiving a second machine learning parameter, and (4) generating a second medical report using the second machine learning parameter. In some cases, the computer program further causes the processor to receive a second input prompt. In some cases, the second medical reportis generated based on the medical record and the second input prompt. In some cases, the first machine learning parameter, the second machine learning parameter, or both, comprises a Large Language Machine (LLM) parameter.Voice Commands

[0173] Described herein are systems and methods compatible with navigation through voice commands. Navigation of medical reports, PACS systems, or other medical imaging technology via a voice command may improve efficiency and reduce physical and mental burnout by allowing for more natural, voice-based interactions.

[0174] As such, herein users may use voice commands to interact with software. In some cases, voice commands may be used to leverage LLM function calls. For example, users can ask for images to be hung in specific viewports according to preset hanging protocols and leveraging whatever criteria they desire, simply by asking the system to hang the relevant images. One illustrative command is to “hang the first study that mentions a lung nodule.” In this example the systems, methods, and computer media herein would search for prior reports or images and return the relevant study to be displayed per the voice command. Other illustrative commands include “adjust window level for readability,” and “sign the report and load the next study.” In some cases, voice commands may be processed via natural language processing. For example, entity extraction may be performed on a voice input to determine that a user is making a request. Accordingly, the request may be satisfied via the execution of a software function. In some cases, the execution of a software function via a voice command may be performed based on an ontology mapping between possible actions and actions derived from a voice command.

[0175] Further the systems, methods, and computer media herein can launch any functionality using voice commands. The systems and methods herein can launch theappropriate function given the context of the current study by leveraging the LLMs described herein, the current state of the viewports and app, eye tracking, or any combination thereof. In some cases, a voice command may be evaluated by a machine learning algorithm to determine an intent based on the voice command, and perform an action related to the medical report based on the intent.

[0176] In some cases, the machine learning algorithm is an LLM. In some cases, the intent, the action, or both, is to change an arrangement of images or data displayed on a graphical user interface, change an image or data displayed on a graphical user interface, adjust a display setting of an image or data displayed in the graphical user interface, determine an image or data that mentions a key word, sign the medical report, display another medical report, transmit a copy of the medical report, measure a distance in the medical report, display a finding in the medical report, or any combination thereof. In some cases, the intent, the action, or both, is to request information about a medical report, a patient, or any other data. In some cases, an action feedback is received from a user regarding the performed action, wherein the action is further performed b ased on the action feedback. In some cases, a user may request a summary, impression, an evaluation of a medical report for discrepancies, or any other action described herein.

[0177] In one example of voice commands, a user views an image and dictates that “The liver demonstrates a new hypodense lesion in the right lobe measuring approximately two centimeters, period.” In some cases, a voice command can be used to find a prior report mentioning a certain topic, such as for example, diverticulosis. Accordingly, sy stems and methods as described herein may facilitate comparison between the current images and one with, for example, diverticulosis. In another example, a practitioner dictates an “Increase in size of the previously noted 4 millimeter non obstructing right renal calculus to 6 millimeters.” The system herein then maps the dictation to a template and pulls relevant prior reports. The system then shows an image of a kidney related to the measurement.Large Language Models

[0178] Large language models (LLMs) as disclosed herein may comprise from-scratch models trained on a corpus of medically relevant data. In some cases, medically relevant data may comprise medical records as disclosed herein of a plurality of patients. In some cases, medically relevant data may comprise medical reports, medical images, DICOM data, HL7data, or EMR data. Additionally, or alternatively, LLMs may build upon a foundation model such that an LLM as disclosed herein may be trained using a corpus of context specific data using the weights of a foundation model for initialization of the context specific task. Context specific data may comprise labelled or unlabeled data depending on a training goal or goals. In some cases, a training may comprise a single or multi-task learning. In some cases, the context specific task may be domain learning comprising a generative pre-training operation. In some cases, the context specific data may comprise particular report generation, summarization, discrepancy detection, classification, segmentation, regression, or detection training datasets. Foundation models may provide a base layer of knowledge such that the base layer of knowledge may be supplemented with further training operations comprising unsupervised, supervised, semi-supervised, or self-supervised training. Examples of foundation models may comprise Azure OpenAI (complete and chat APIs for GPT-3, GPT- 3.5, and GPT-4, used in ChatGPT), OpenAI (complete and chat APIs for GPT-3, GPT-3.5 and GPT-4), Google Vertex Al (e g., PaLM, PaLM-2), Meta's LLaMa, as well as BLOOM, Ernie 3.0 Titan, and Anthropic's Claude 2, FLAN-T5, OpenAssistant, RoBERTa, MiniLM, MPNet, DALL-E, and BERT. While foundation models may comprise those listed previously, foundation models may generally comprise a transformer, self -attention, or other machine learning model that has been previously trained.

[0179] In some cases, LLMs may comprise transformers. Transformers may be applied to a variety of natural language processing tasks such as text classification, named entity recognition, summarization, translation, question answering, or text generation. For example, transformers may facilitate some embodiments of medical report generation, summarization, or discrepancy detection. Transformers may additionally be applied to other data modalities as with vision transformers (ViT). For example, a ViT trained on radiological images may facilitate machine learning performance for tasks such as feature extraction, segmentation, object detection, or other computer vision task. In some cases, a ViT may be used to extract image findings. Transformers may be facilitated by the self-attention mechanism. The selfattention mechanism may facilitate models implementing the self-attention mechanism to assign different weight to different portions of a sequence of an input. Self -attention may facilitate LLMs or transformer based models to leverage contextual information of an input to inform an output while reducing the risk of the model “forgetting” about earlier portions of an input. For example, a medical record including prior patient reports, medical data, medical records, or other general patient history information may result in complex, evolving datasets.These complex, evolving datasets may contain highly relevant details about a patient that have not been commented on in some time. Furthering the example, a prior imaging study from ten years prior to a current imaging study may be valuable information for contextualizing and understanding a current imaging study; however, in the context of prompts, the details of the prior imaging study may be included in a substantially earlier portion of a prompt as input into an LLM. Without self-attention to retain spatially disparate (e.g., beginning versus end of multi-paragraph prompt) information, the prior imaging study information may be forgotten in non-transformer / self-attention based models. In this way, implementations of the self -attention mechanism may provide particular utility in complex, multi-modal data that may benefit from comprehensive, contextual understanding of an input.

[0180] LLMs as disclosed herein may be configured to perform operations that an LLM may or may not have been explicitly trained on. Configuring LLMs to perform operations or resolve inputs on which they were not explicitly trained may comprise prompt-engineering. In some cases, the ability of an LLM to perform a task on which it was not explicitly trained may be considered a form of zero-shot learning. An input prompt into an LLM may comprise one or more data modalities including audio, image, video, tabular, or text data. The input prompt may comprise a natural language input. In some cases, a natural language input may indicate that the input is not used to adhere to a specific format and that input prompt may be arbitrarily structured for a given task. For example, an input prompt may comprise one or more of a textual input, an image input, an audio input, a video input, or a tabular input for use in indicating a desired output of the LLM. For example, the input may comprise an imperative, an interrogative, an exclamatory, or a declarative input. In some cases, the input prompt may comprise any arbitrary component that the LLM is capable of encoding into an embedding that may be compared against a learned latent space of the LLM. For example, audio, tabular, image, or video data may be encoded into an embedding space with a similar or same semantic organization as text data through the use of, for example, model grafting as disclosed herein.

[0181] An input prompt in the context of LLMs may comprise instructions. Instructions may comprise one or more operations to execute. In some cases, the operations may be ordered. Instructions may be used to guide an LLM to resolve an input into the LLM. In some cases, instructions may be applied to resolve complex or multi-stage problems. In some cases, the input prompt may comprise formatting, problem-solving, or organizational preferences. Forexample, an input prompt may include a request for medical summary findings to be extracted from a medical report by the operations of (1 ) break the medical report into chunks based on paragraph, (2) extract medical terms from each chunk, (3) generate a contextual description of each medical term, (4) provide a bulleted list titled “medical summary findings” with each of the medical terms, and (5) organize the medical summary findings in order of relevance to cardiology. Additionally, or alternatively, the input prompt may facilitate in-context or few-shot learning. In in -context or few-shot learning, a small number of examples may be provided to an LLM in the body of an input prompt. The examples provided in the input prompt may facilitate the LLM in predicting a never before resolved example. For example, a plurality of samples of severity scores (e.g., numerical scale or categorical scale such as mild, moderate, severe) tied to various findings may be used as a portion of an input prompt used as input into an LLM to guide the LLM to assigning severity scores to a never before seen medical report or summary. In some cases, a severity score may comprise a relevancy score. For example, a key finding may be a finding that is indicated as severe. In in-context learning, the examples contained in the input prompt may not be persistently stored in the LLM. For example, the input prompt may not be used to train the LLM.

[0182] In some cases, soft prompting may be used in prompting an LLM to resolve a user input. Soft prompting may provide an efficient route towards leveraging an LLMs ability to perform tasks when given instructions or guidance in the form of an input prompt. For example, a portion of an input prompt may be learned by training a vector representing a query portion of the input prompt. Furthering the example, the vector may be trained to prompt the LLM to perform a specific tasks such as finding extraction. Soft prompting may provide particular utility where other prompt engineering approaches are found to be unsuccessful or if a task is difficult to convey in natural language. Soft prompting may comprise attaching fixed length vectors, or tokens, to the beginning of an input before feeding the input into the model. The composition of a fixed length vector or vectors may be updated during a training operation such that the weights of the LLM are frozen during training and only the fixed length vector or vectors are updatable during training. In this manner, a prompt may be tuned to guide an LLM to resolve a user input in an automated learning process rather than an iterative human prompt engineering process. For example, an LLM may be soft- prompted to extract findings from a medical record.

[0183] LLMs may provide a knowledge base that may be readily adapted to new information or a specific task. LLMs may be trained on a large non-specific training sets in a generative pre-training operation. Generative pre-training may be used to establish a general understanding of a language such that a syntax, diction, vocabulary, or grammar of a language is learned by an LLM. In some cases, language may include natural language. In some cases, natural language may comprise vernacular English, vernacular Mandarin, vernacular Spanish. In some cases, language may include domain specific language. In some cases, domain specific language may comprise jargon dictated by a field of use such as in a branch of science or medicine. In some cases, generative pre-training may comprise a series of training operations. For example, an LLM may be exposed to or trained on different domains over time. In some cases, a natural language corpus may be used to train an LLM prior to a domain specific language training. In addition to language, generative pre-training may be performed for other modalities comprising image, video, tabular data, audio, or other data format.

[0184] LLMs may be trained to perform a discriminative task such as regression. For example, LLMs may be trained to provide relevancy scores or confidence scores as described herein. In some cases, LLMs may be trained to perform classification. For example, an LLM configured to take image inputs may classify images as normal or abnormal. In some cases, LLMs may be trained to perform other specific tasks such as a semantic search, visual question answering, or quality assurance. In some cases, semantic search may be used to find most relevant findings in a medical record or current study. For example, semantic search may be able to provide findings and their sources (e.g., reports, images) for user queries such as “find image most relevant to osteophytes.” In some cases, visual question answering may comprise providing natural language outputs to user queries. For example, a user may be provided an answer to the query “does this image contain any anatomic variants?” In some cases, quality assurance may comprise discrepancy detection.

[0185] In some cases, LLMs, or transformers, may be used as feature extractors. For example, a multi-modal input may be fed into a trained transformer such that the output of trained transformer layer, which may be frozen, is used as input into a trainable head. In some cases, the trainable head is one or more deep learning layer that may be trained on inputs into the trainable head to predict a value of interest. For example, visual information may be encoded into feature rich embeddings that may be used to provide feature denserepresentations of input data to trainable heads to perform tasks such as regression or classification.

[0186] Multi-modal LLMs may comprise the use of one or more encoders. An encoder for each data modality used in a multi-modal LLM may be used to generate embeddings for each modality used with multi-modal LLMs. In some cases, each non-text modality may be mapped via encoders into the same embedding space as text. In some cases, a multi-modal model may leverage a grafting or bootstrapping technique. Grafting or b ootstrapping may comprise connecting an encoder trained on a modality of interest to a model expecting another data modality. Connecting two or more modalities may comprise training models of multiple modalities to produce similar embeddings for similar findings of disparate modalities. For example, an audio description of a broken bone, an image of a broken bone, and a written description of a broken bone may be encoded into a similar embedding representation. Connecting the modalities may comprise training a querying transformer comprising a set of learnable query vectors to extract features from a first modality to pass to a model trained on a second modality. The querying transformer may have a smaller set of learnable parameters that may require less data and less computation for training than an end- to-end model training. The learnable query vectors may comprise one or more vectors that may be updated or learned during a training operation to extract features from an output embedding of a first modality encoder. For example, the learnable query vectors may be turned to extract relevant features of an image embedding output by an image encoder before passing the embedding of the relevant image features to a model trained on text. In this way, the relevant image features may be used with further transformer models such as encoder, decoder, or encoder-decoder models. For example, the latent representation of the relevant features may be used in a generative text decoder to generate appropriate text such as an impression to incorporate in a medical report. In some cases, the encoder used to extract relevant features of a first modality may be trained in a contrastive learning operation.Contrastive learning may comprise training one or more encoders to output embeddings of each modality of the one or more encoders such that the encoders output similar embeddings for similar inputs of disparate modalities. For example, an encoder may be trained to convert tabular data to text data. Furthering the example, the encoder may be trained to provide a text description of key details of tabular data for inclusion into a medical report or summary. Continuing the example, the text data may describe tabular data similar to how a user may describe the tabular data when graphed or otherwise presented to the user. Still furthercontinuing the example, the text description may describe ECG data with findings such as “normal heartbeat,” “abnormal heart rhythm,” “slow heartbeat,” etc.

[0187] Multi-modal LLMs may comprise the use of one or more encoders. An encoder for each data modality used in a multi-modal LLM may be used to generate embeddings for each modality used with multi-modal LLMs. In some cases, each non-text modality may be injected into a text vector or token sequence prior to feeding into an LLM. In some cases, a text input into an LLM may converted to a vector or token sequence prior to being fed to neural layers of an LLM. To marry non-text modalities with text, an encoder may be trained for each non-text modality to output vectors or tokens of non-text modalities that may be prefixed, interleaved, or post-fixed to text vectors or tokens. As such, non-text modalities may be injected into text vectors or tokens to generate vectors or tokens representing multimodal data. These vectors or tokens may, for example, then inform a generative operation such as text or image generation such that the generated output is dependent on each of the input modalities represented in the vector or tokens. For example, an LLM may convert a prompt into a text vector or token sequence before learning or inferencing based on the prompt. Other data modalities may leverage neural networks (e.g., CNNs, FCNs, transformers, RNNs, ResNets, etc.) to convert embeddings of the other data modalities to a vectorized or tokenized representation that may be combined with vector or token representations of text.

[0188] In some cases, reinforcement learning may be used to optimize models over time. Reinforcement learning may use human feedback to fine-tune models to output results in line with desired human or specific user preference. For example, approvals, rejections, edits to, rankings of outputs by LLMs as described herein may be used to build a training dataset comprising inputs, LLM outputs, and user desired outputs. One example may be reinforcement learning from human feedback (RLHF) may comprise operations including supervised fine-tuning of an LLM, building a separate reward model, and optimizing the LLM using the reward model. In some cases, the reward model may be trained to predicted human preference based on gathered human preference data. This may allow a deployed platform implementing the systems and methods herein to improve over time via usage . In some cases, the supervised fine-tuning may comprise training an LLM to provide outputs to prompts similar to those generated by a human. The supervised fine-tuning may comprise collecting human demonstration data, for example, summarization of medical data, examplesof findings discrepancies, anatomic variant detection, or any other output of interest. In some cases, the anatomic variant detection may comprise annotation data, segmentation data, bounding box data, or class labelled data. In some cases, an output of interest may comprise a regression, classification, or generative output. Building a separate reward model may comprise presenting a plurality of outputs of the fine-tuned LLM to a user for ranking. For example, a user may be provided summary options, label options, annotation options, format options, or organization options for ranking. The human ranking of the plurality of outputs may be used train the reward model to estimate a user’s preference for an output of the fine - tuned LLM. For example, an output of an LLM may comprise varying levels of detail or organization from which the user may select.

[0189] Encoders may be trained herein to output or generate embeddings that are similar for representations of findings among a plurality of modalities. For example, audio, visual, text, or tabular data may each comprise a finding describing a similar condition. For example, a medical report may comprise an image of a tumor, an audio dictation of a user describing the tumor, a written description of the tumor, and a table indicating a progression of tumor density over time. Each of these modalities may comprise a finding that tumor thickness is increasing over time. As such, an encoder for each modality may comprise an embedding representation of “tumor thickness increasing over time” that is similar among each of the types of modalities.

[0190] In some cases, encoders may be trained via contrastive learning. Contrastive learning may comprise a supervised or self-supervised learning paradigm. In self-supervised contrastive learning m (SSCL), the model implementing SSCL learns differentiate between similar and dissimilar input types automatically. Alternatively, supervised contrastive learning (SCL) may be used in training encoders (e.g., transformers) to generate similar embeddings for disparate data types (e.g., image and text, text and audio, audio and tabular data) that were assigned as a pair in a training set. Contrastive learning may be used for data- retrieval algorithms. For example, contrastive learning may be used to train a model to obtain representatives of a data modality with the greatest semantic similarity to an input into the model. In some cases, the semantic similarity may be a similarity of images takes of a patient and one or more finding of a patient (e.g., a medical summary finding). Contrastive learning may provide a powerful tool for semantic search applications. An implementation of semantic search may use natural language to retrieve images comprising semantically similarinformation or vice versa. For example, a user may request “most similar images” to a finding (e.g., enlarged gallbladder).

[0191] In some cases, the systems and methods disclosed herein may be implemented as a method for generating a medical report. In some cases, generating the medical report may comprise presenting at least one medical image on a display, obtaining a dictation of a user viewing the at least one medical image; and while obtaining the dictation, distributing a plurality of findings of the dictation to a plurality of report sections, thereby generating the medical report. Generating a medical report may be performed by a user engaged in an augmented reality, virtual reality, or mixed reality device. For example, a user may view a hanging protocol using an AR / VR / MR headset and dictate finding while viewing the hanging protocol. In some cases, either within or outside of an AR / VR / MR context, a user may interact with a medical image (e.g., scrolling, zooming, etc.). In some cases, a user may comment on, or dictate findings for, a medical image subject to their viewing. The findings may be used to complete either partially or entirely a plurality of report sections. In some cases, the plurality of report sections may comprise an impression section. In some cases, the plurality of findings may be determined by a machine learning model as disclosed herein. For example, a dictation of a user may be fed into a machine learning or large language model as disclosed herein and subsequently analyzed for findings. Continuing the example, a machine learning model may be configured to extract, predict, or otherwise determine relevant aspects of a dictation and categorize them as belonging to, for example, a finding class of the plurality of finding classes disclosed herein. In some cases, systems and methods described herein may be used to detect a discrepancy in the plurality of findings. For example, machine learning may be used to detect a discrepancy between least one finding of the plurality of findings with a historical finding. In some cases, the historical finding may be located or derived from a portion of a medical history of a subject. In some cases, the medical history of the subject comprises a historical medical image of the subject, a history medical report of the subject, demographic data of the subject, or any combination thereof. In some cases, upon detecting the discrepancy in the plurality of findings, at least one finding of the plurality of findings is identified in the medical report as comprising the discrepancy. For example, the discrepant finding may be labelled or otherwise identified (e.g., contained in a special section) in a medical report. As such, the user may be able to edit, reject, confirm, or otherwise evaluate a discrepant finding. In some cases, the at least one medical image comprises an x-ray, a CAT scan, a PET scan, an MRI, or an ultrasound. In some cases, thedictation is received through a microphone or voice audio recorder. In some cases, the method described may further comprise inserting a link between a finding of the dictation and a portion of the at least one medical image. For example, the link may facilitate anatomic navigation or linkage as described elsewhere herein. Continuing the example, the link may be inserted into the medical report, the at least one medical image, or both.AI-Assisted Image Segmentation and Labelins

[0192] Described herein are systems, software, and methods for facilitating interpretation and reporting of medical images. In some cases, the systems, software, and methods leverage computer vision such as image segmentation (e.g., semantic segmentation, instance segmentation, panoptic segmentation, watershed based segmentation, Mask R-CNN), image classification (e.g., using ResNet, FCNs, RNNs, CNNs, transformers), object detection (e.g., YOLO, faster-RCNN, SSD), or feature extraction (e.g., encoders). These processes may be implemented via a system, subsystem, or module, which can be stand-alone or part of a larger platform or system as disclosed herein. Medical images (e.g., x-ray, MRI) can be evaluated using an image segmentation algorithm to partition the image into one or more regions. The regions can be sets of pixels that correspond to objects or boundaries that share certain anatomic characteristics. For example, an x-ray image of human vertebrae can be segmented to partition the image into segments that correspond to the individual bones. Segmentation provides a representation of a medical image that can be medically relevant, and thus suitable for inclusion within an analysis or report. The segments in an image can be defined according to one or more labels assigned to the individual pixels. For example, semantic segmentation may be used to differentiate pixels belonging to different classes in an image (e.g., background, bone, and tissue). In a further example, instance segmentation may combine aspects of object detection to and semantic segmentation to identify unique objects in an image and assign pixels of each object as a unique member of a class (e.g., identify each bone and each tissue as unique instances of bone and tissue). In yet a further example, panoptic segmentation may segment pixels and assign a unique label to each pixel in an image regardless of detectability by object detection (e.g., identify amorphous classes such as x-ray background and identify each bone and each tissue as unique instances of bone and tissue). In some cases, image segmentation is performed by an image segmentation module. The image segmentation module may be a component module of an anatomic navigator module. Image segmentation may be applied to 2D or 3D medical images.

[0193] Accordingly, disclosed herein are systems, software, and methods for performing image segmentation using segmentation algorithms that are configured to detect anatomic variants. In some cases, an anatomic variant may comprise an anatomy with an abnormal or otherwise unexpected structure. Anatomic variants may be identified through machine learning algorithms as disclosed herein. For example, a machine learning model may be trained through a supervised approach to detect normal versus abnormal anatomy. In some cases, abnormal anatomy comprises anatomic variants. In some cases, the supervised approach may comprise one class of anatomy or multiple classes. In some cases, the anatomic variants may be indicated by a score as with a regression algorithm (e.g., 1 is highly abnormal, 10 is highly normal) or through a class (e.g., “normal” versus “abnormal”). For example, a regression algorithm may output a number corresponding to increasing abnormality on a 0 to 10 continuous scale. In another example, a class may comprise a label such as “normal,” “abnormal,” or “severely abnormal.” In some cases, the anatomic variants may be dependent on a proximity between a possible anatomic variant and one or more learned representations of the anatomy in a latent space . For example, an image of a diseased tissue may be determined to be a variant of healthy tissue based on a difference in embedding representation of the diseased versus healthy tissue.

[0194] Various types of images can be analyzed as described according to the present disclosure, for example, magnetic resonance imaging (MRI), CT (computerized tomography), CAT (computerized axial tomography), positron emission tomography (PET), single photon emission computed tomography (SPECT), ultrasound, x-ray images, and other types of medical images.

[0195] In some cases, each anatomy in a system or method herein may comprise a tag that describes the anatomy. In some cases, the tag may be a unique identifier such that the tag may be used by an ontology or software implementation of the systems and methods disclosed herein. For example, the tag may be used for keyword or other non -ML search. Additionally, or alternatively, the unique identifier may be used to organize patient data such that the relevant portions of patient data are retrieved for analysis by the systems and methods disclosed herein. This identifier may be used to a context for an imaging study, report generation, report evaluation, or other analysis. For example, the unique identifier may be used to select a most relevant embodiment of the systems and methods disclosed herein. Furthering the example. The unique identifier may be used to select models, input prompts,software, scripts, or macros to perform an appropriate analysis. In some cases, when a user selects (e.g., using the mouse or eye-tracking) a part of the anatomy shown on the image, the user is presented with a list possible clinical findings relevant to that part of the anatomy.

[0196] In some cases, the user may interact the anatomic navigator using commands such as "compare this." The anatomic navigator may subsequently compare the current and prior studies. The anatomic navigator may be configured to highlight or present any differences between a finding of a current study and a finding of a prior study. For example, if an anatomic variant has changed since its last evaluation, the discrepancy between or among the studies may be presented to the user. Furthering the example, the change in anatomic variant may indicate a change in structure as with growth or shrinkage of tumor. In addition to qualitative findings, quantitative measurements, quantitative comparisons (for instance, percent change) may be generated the text and inserted into the report automatically (e.g., a computer-finding that is a quantitative measurement).

[0197] In some cases, the anatomic navigator may comprise an image query function. The present systems, software, and methods allow for querying of specific regions of the image. For example, a user may, by one of the input modalities disclosed herein, solicit an analysis of an image or region of the image. Soliciting the analysis may comprise requesting or commanding via a tactile input, audio input, or virtual input device that the image or portion of the image be assessed (e.g., by a computer vision algorithm as disclosed herein). The possible findings can be generated using a computer vision algorithm or module comprising one or more algorithms or models for image analysis. The computer vision algorithm may be specific to a task (e.g., detect liver malignancies), to an anatomy (e.g., detect liver variants), to an imaging modality (e.g., evaluate MRI for anatomic variants), or a general image corpus (e.g., as with zero-shot techniques). In some cases, the computer vision algorithm may be used to generate one or more image finding as disclosed herein. In some cases, one or more image finding may be produced through using natural language tools as disclosed herein. For example, a computer vision model may be grafted onto an LLM to provide extracted image features to an LLM such that the LLM may decode the image features into the one or more image findings.Virtual, Augmented, or Mixed Reality

[0198] Alternatively, or in combination with mouse cursor detection, eye-tracking or fingertracking may be used to for interaction with the systems and methods disclosed herein. This may allow a user to interact with an image without requiring the user to manipulate a mouse, trackpad, keyword, stylus, touch screen, or other physical input device. The system may include an eye-tracking device for measuring eye position or eye movement (e.g., smart glasses, VR, AR, or MR headset). In some cases, the system utilizes an eye tracking algorithm to extrapolate the user’s gaze fixation point(s) on a medical image. The image may be displayed on a computer screen or display, or alternatively, on a projected image. The eye tracking device may be a camera or other optical sensor. In some cases, the system comprises a light emitting device that projects light (e.g., infrared) onto the eye(s) of a user. An optical sensor then detects the light that is reflected off the eye(s). The pattern of reflected light detected by the optical sensor can be used to calculate the corresponding gaze fixation point. The light can be a near-infrared (IR) light, which has the advantage of not being invisible to human eyes. A video-based eye tracking system can take an image of the corneal reflection of the reflected light and the pupil. The corneal reflection and pupil can be used as features to determine the eye position and / or movement for purposes of identifying the gaze fixation point. This system can be setup on a display or monitor, for example, the light emitting device and the optical sensor both built into or attached to the display.

[0199] In some cases, the eye-tracking system is configured to detect user commands based on eye tracking input. User commands can include various input commands that are available using conventional input devices such as, for example, a mouse, keyboard, trackpadjoystick, controller, or stylus. Examples of user commands that may be conveyed using eye tracking input include selection or de-selection of a GUI element, scrolling, magnification, switching between images, or opening or closing an image. The eye tracking input can be preprogrammed to correspond to specific commands. Additionally, or alternatively, an eye tracking input can be customized to correspond to commands specified by a user. The eye tracking input can be configured to respond to actions such as blinking, number of, gaze duration, or eye rolls. For example, a user may gaze in a particular direction or towards the edge of the medical image to scroll in that direction. The user may need to maintain the gaze for a minimum duration (e.g., 0.5s, Is, 2s, etc.) to initiate the scrolling or other command. Similarly, the system can be configured to require the blink be maintained for a minimum duration in order to effectuate the user command. Additionally, or alternatively to eye tracking, finger tracking may be used to substitute eye tracking for operations such asselecting, switching, opening, closing, scrolling, or magnification. Finger actions may include pointing, pinching, reverse-pinching, swiping, waving, pushing away, pulling in, or other gesture. Accordingly, eye tracking or finger tracking may be used for executing the operations disclosed herein.Machine Learning

[0200] In some cases, machine learning (ML) may generally involve identifying and recognizing patterns in existing data in order to facilitate making predictions for subsequent data. ML may include a ML model (which may include, for example, a ML algorithm). Machine learning, whether analytical or statistical in nature, may provide deductive or abductive inference based on real or simulated data. The ML model may be a trained model. ML may comprise one or more supervised, semi-supervised, self-supervised, or unsupervised ML techniques. For example, an ML model may be a trained model that is trained through supervised learning (e.g., various parameters are determined as weights or scaling factors). ML may comprise one or more of regression analysis, regularization, classification, dimensionality reduction, ensemble learning, meta learning, association rule learning, cluster analysis, anomaly detection, deep learning, or ultra -deep learning. ML may comprise: k- means, k-means clustering, k-nearest neighbors, learning vector quantization, linear regression, non-linear regression, least squares regression, partial least squares regression, logistic regression, stepwise regression, multivariate adaptive regression splines, ridge regression, principal component regression, least absolute shrinkage and selection operation (LASSO), least angle regression, canonical correlation analysis, factor analysis, independent component analysis, linear discriminant analysis, multidimensional scaling, non-negative matrix factorization, principal components analysis, principal coordinates analysis, projection pursuit, Sammon mapping, t-distributed stochastic neighbor embedding, AdaBoosting, boosting, gradient boosting, bootstrap aggregation, ensemble averaging, decision trees, conditional decision trees, boosted decision trees, gradient boosted decision trees, random forests, stacked generalization, Bayesian networks, Bayesian belief networks, naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, hidden Markov models, hierarchical hidden Markov models, support vector machines, encoders, decoders, auto -encoders, stacked auto encoders, perceptrons, multi-layer perceptrons, artificial neural networks, feedforward neural networks, convolutional neural networks, recurrent neural networks, long short-term memory, deep belief networks, deep Boltzmann machines, deep convolutional neuralnetworks, deep recurrent neural networks, large language models, vision transformers, or generative adversarial networks.

[0201] Training the ML model may include, in some cases, selecting one or more untrained data models to train using a training data set. The selected untrained data models may include any type of untrained ML models for supervised, semi-supervised, self-supervised, or unsupervised machine learning. The selected untrained data models may be specified based upon input (e.g., user input) specifying relevant parameters to use as predicted variables or other variables to use as potential explanatory variables. For example, the selected untrained data models may be specified to generate an output (e.g., a prediction) based upon the input. Conditions for training the ML model from the selected untrained data models may likewise be selected, such as limits on the ML model complexity or limits on the ML model refinement past a certain point. The ML model may be trained (e.g., via a computer system such as a server) using the training data set. In some cases, a first subset of the training data set may be selected to train the ML model. The selected untrained data models may then be trained on the first subset of training data set using appropriate ML techniques, based upon the type of ML model selected and any conditions specified for training the ML model. In some cases, due to the processing power requirements of training the ML model, the selected untrained data models may be trained using additional computing resources (e.g., cloud computing resources). Such training may continue, in some cases, until at least one aspect of the ML model is validated and meets selection criteria to be used as a predictive model.

[0202] In some cases, one or more aspects of the ML model may be validated using a second subset of the training data set (e.g., distinct from the first subset of the training data set) to determine accuracy and robustness of the ML model. Such validation may include applying the ML model to the second subset of the training data set to make predictions derived from the second subset of the training data. The ML model may then be evaluated to determine whether performance is sufficient based upon the derived predictions. The sufficiency criteria applied to the ML model may vary depending upon the size of the training data set available for training, the performance of previous iterations of trained models, or user-specified performance requirements. If the ML model does not achieve sufficient performance, additional training may be performed. Additional training may include refinement of the ML model or retraining on a different first subset of the training dataset, after which the new ML model may again be validated and assessed. When the ML model has achieved sufficientperformance, in some cases, the ML may be stored for present or future use. The ML model may be stored as sets of parameter values or weights for analysis of further input (e.g., further relevant parameters to use as further predicted variables, further explanatory variables, further user interaction data, etc.), which may also include analysis logic or indications of model validity in some instances. In some cases, a plurality of ML models may be stored for generating predictions under different sets of input data conditions. In some cases, the ML model may be stored in a database (e.g., associated with a server).Computer Systems

[0203] Referring to FIG. 16, a block diagram is shown depicting an exemplary machine that includes a computer system 1600 (e.g., a processing or computing system) within which a set of instructions can executefor causing a device to perform or execute any one or more of the aspects and / or methodologies for static code scheduling of the present disclosure. The components in FIG. 16 are examples only and do not limit the scope of use or functionality of any hardware, software, embedded logic component, or a combination of two or more such components implementing particular embodiments. The computer system 1600 may be configured to perform any of the methods disclosed herein. For example, the computer system 1600, may use the one or more processors 1601 to execute operations for medical report summarization, evaluation (e.g., discrepancy detection), or generation. Further, the one or more processors may be used for machine learning training or inference operations as disclosed herein. Yet further, the one or more processors may be used to control a display 1632 a to show the GUI 320 of computing system 300. In some cases, the computer system 1600 may comprise or interact with (e.g., via network 1630) the computing system 300. The computer system 1600 may be in communication with storage (e.g., via network 1630, storage devices 1635, storage media 1636) to retrieve images, medical reports, medical records, or medical data may comprise communication with a picture archiving and communication system (PACS), a vendor neutral archive (VNA), radiological information system (RIS), hospital information system (HIS), secure cloud storage solution, secure local storage solution, or other data storage or data management system.

[0204] In some cases, a GUI 320 may be configured to interact with a software implementing the systems and methods described herein. For example, as shown in FIG. 20, a user may select from a plurality of prompts depending on a task desired by the user. In some cases, the user my dictate to the software via a voice command. For example, a user may request a priorreports summarization (“Priors Summarization”), a discrepancy detection (“MedCheck”), or a report generation based on one or more finding (“Ubercomplete”). In some cases, the user may select among a plurality of machine learning models to execute a user request.

[0205] In some cases, a GUI 320 may be configured to display a summary to a user. For example, as shown in FIG. 21, a user may request a summary of prior reports. In this example, the user requests a description of the evolution of findings over time. In some cases, the systems and methods may be described herein may be used to extract a similar finding from a plurality of reports to evaluate the finding. In some cases, the plurality of findings may be presented or summarized as shown in FIG. 21. For example, a brief description of the finding over time may be “overall the reports indicate post-surgical changes, degenerative changes, and chronic small vessel ischemic changes in the patient.”

[0206] In some cases, a GUI 320 may be configured to display a discrepancy of a medical report as shown in FIG. 22. For example, a user may request a grammatical review, a findings discrepancy review, or both. As show, a machine learning model may process the request and return answers describing both grammatical and findings discrepancies.

[0207] The computer system 1600 may facilitate the deployment of a platform as shown in FIG. 17. FIG. 17 shows one illustrative embodiment of a flowchart for a multi-modal Large Language Model (LLM). As shown, a Sirona RADOS cloud receives user input, multi-modal clinical data integrations, and third-party API integration outputs, and provides application state and standardized multi-modal data, CV, NLP, and multi-modal models, analytics (e.g., data monitoring and user feedback), and an enhanced radiologist experience. In some cases, the application state and standardized multi-modal data provides model training, prompt refinement, and interference to the CV, NLP, and multi-modal models. In some cases, the analytics further interacts with the CV, NLP, and multi-modal models and helps enhance radiologist experience. In some cases, the multi-modal clinical data integrates information from current image data, electronic medical records, genomic and other data, DICOM data, pathology reports, HL7 data, and pathology data. In some embodiment the third-party API integration outputs receives data from RadAI, Koios, RevealDX, and other integrations.

[0208] In some cases, the methods, systems, and computer media herein further generate an impression per the methods, systems and media to generate a report summary. In some cases, the impression is auto-generated based on prior reports, DICOM and HL7 data, optionally with an artificial intelligence algorithm. In some cases, the impression section of a reportrepresents the radiologist's summary of the most clinically important findings discovered during a read.

[0209] Computer system 1600 may include one or more processors 1601, a memory 1603, and a storage 1608 that communicate with each other, and with other components, via a bus 1640. The bus 1640 may also link a display 1632, one or more input devices 1633 (which may, for example, include a keypad, a keyboard, a mouse, a stylus, etc.), one or more output devices 1634, one or more storage devices 1635, and various tangible storage media 1636. All of these elements may interface directly or via one or more interfaces or adaptors to the bus 1640. For instance, the various tangible storage media 1636 can interface with the bus 1640 via storage medium interface 1626. Computer system 1600 may have any suitable physical form, including but not limited to one or more integrated circuits (ICs), printed circuit boards (PCBs), mobile handheld devices (such as mobile telephones or PDAs), laptop or notebook computers, distributed computer systems, computing grids, or servers.

[0210] Computer system 1600 includes one or more processor(s) 1601 (e.g., central processing units (CPUs), general purpose graphics processing units (GPGPUs), or quantum processing units (QPUs)) that carry out functions. Processor(s) 1601 optionally contains a cache memory unit 1602 for temporary local storage of instructions, data, or computer addresses. Processor(s) 1601 are configured to assist in execution of computer readable instructions. Computer system 1600 may provide functionality for the components depicted in FIG. 16 as a result of the processor(s) 1601 executing non -transitory, processor-executable instructions embodied in one or more tangible computer-readable storage media, such as memory 1603, storage 1608, storage devices 1635, and / or storage medium 1636. The computer-readable media may store software that implements particular embodiments, and processor(s) 1601 may execute the software. Memory 1603 may read the software from one or more other computer-readable media (such as mass storage device(s) 1635, 1636) or from one or more other sources through a suitable interface, such as network interface 1620. The software may cause processor(s) 1601 to carry out one or more processes or one or more steps of one or more processes described or illustrated herein. Carrying out such processes or steps may include defining data structures stored in memory 1603 and modifying the data structures as directed by the software.

[0211] The memory 1603 may include various components (e.g., machine readable media) including, but not limited to, a random access memory component (e.g., RAM 1604) (e.g.,static RAM (SRAM), dynamic RAM (DRAM), ferroelectric random access memory (FRAM), phase-change random access memory (PRAM), etc.), a read-only memory component (e.g., ROM 1605), and any combinations thereof. ROM 1605 may act to communicate data and instructions unidirectionally to processor(s) 1601, and RAM 1604 may actto communicate data and instructions bidirectionally with processor(s) 1601. ROM 1605 and RAM 1604 may include any suitable tangible computer-readable media described below. In one example, a basic input / output system 1606 (BIOS), including basic routines that help to transfer information between elements within computer system 1600, such as during startup, may be stored in the memory 1603.

[0212] Fixed storage 1608 is connected bidirectionally to processor(s) 1601, optionally through storage control unit 1607. Fixed storage 1608 provides additional data storage capacity and may also include any suitable tangible computer-readable media described herein. Storage 1608 may be used to store operating system 1609, executable(s) 1610, data 1611, applications 1612 (application programs), and the like. Storage 1608 can also include an optical disk drive, a solid-state memory device (e.g., flash -based systems), or a combination of any of the above. Information in storage 1608 may, in appropriate cases, be incorporated as virtual memory in memory 1603.

[0213] In one example, storage device(s) 1635 may be removably interfaced with computer system 1600 (e.g., via an external port connector (not shown)) via a storage device interface 1625. Particularly, storage device(s) 1635 and an associated machine-readable medium may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for the computer system 1600. In one example, software may reside, completely or partially, within a machine-readable medium on storage device(s) 1635. In another example, software may reside, completely or partially, within processor(s) 1601.

[0214] Bus 1640 connects a wide variety of subsystems. Herein, reference to a bus may encompass one or more digital signal lines serving a common function, where appropriate. Bus 1640 may be any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures. As an example, and not by way of limitation, such architectures include an Industry Standard Architecture (ISA) bus, an Enhanced ISA (EISA) bus, a Micro Channel Architecture (MCA) bus, a Video ElectronicsStandards Association local bus (VLB), a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, an Accelerated Graphics Port (AGP) bus, HyperTransport (HTX) bus, serial advanced technology attachment (SATA) bus, and any combinations thereof.

[0215] Computer system 1600 may also include an input device 1633. In one example, a user of computer system 1600 may enter commands and / or other information into computer system 1600 via input device(s) 1633. Examples of an input device(s) 1633 include, but are not limited to, an alpha-numeric input device (e.g., a keyboard), a pointing device (e.g., a mouse or touchpad), a touchpad, a touch screen, a multi-touch screen, a joystick, a stylus, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), an optical scanner, a video or still image capture device (e.g., a camera), and any combinations thereof. In some cases, the input device is a Kinect, Leap Motion, or the like. Input device(s) 1633 may be interfaced to bus 1640 via any of a variety of input interfaces 1623 (e.g., input interface 1623) including, but not limited to, serial, parallel, game port, USB, FIREWIRE, THUNDERBOLT, or any combination of the above.

[0216] In particular embodiments, when computer system 1600 is connected to network 1630, computer system 1600 may communicate with other devices, specifically mobile devices and enterprise systems, distributed computing systems, cloud storage systems, cloud computing systems, and the like, connected to network 1630. Communications to and from computer system 1600 may be sent through network interface 1620. For example, network interface 1620 may receive incoming communications (such as requests or responses from other devices) in the form of one or more packets (such as Internet Protocol (IP) packets) from network 1630, and computer system 1600 may store the incoming communications in memory 1603 for processing. Computer system 1600 may similarly store outgoing communications (such as requests or responses to other devices) in the form of one or more packets in memory 1603 and communicated to network 1630 from network interface 1620. Processor(s) 1601 may access these communication packets stored in memory 1603 for processing.

[0217] Examples of the network interface 1620 include, but are not limited to, a network interface card, a modem, and any combination thereof. Examples of a network 1630 or network segment 1630 include, but are not limited to, a distributed computing system, a cloud computing system, a wide area network (WAN) (e.g., the Internet, an enterprise network), a local area network (LAN) (e.g., a network associated with an office, a building, acampus or other relatively small geographic space), a telephone network, a direct connection between two computing devices, a peer-to-peer network, and any combinations thereof. A network, such as network 1630, may employ a wired and / or a wireless mode of communication. In general, any network topology may be used.

[0218] Information and data can be displayed through a display 1632. Examples of a display 1632 include, but are not limited to, a cathode ray tube (CRT), a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic liquid crystal display (OLED) such as a passive-matrix OLED (PMOLED) or active-matrix OLED (AMOLED) display, a plasma display, and any combinations thereof. The display 1632 can interface to the processor(s) 1601, memory 1603, and fixed storage 1608, as well as other devices, such as input device(s) 1633, via the bus 1640. The display 1632 is linked to the bus 1640 via a video interface 1622, and transport of databetween the display 1632 and the bus 1640 can be controlled via the graphics control 1621. In some cases, the display is a video projector. In some cases, the display is a head-mounted display (HMD) such as a VR headset. In further embodiments, suitable VR headsets include, by way of non -limiting examples, HTC Vive, Oculus Rift, Samsung Gear VR, Microsoft HoloLens, Razer OSVR, FOVE VR, Zeiss VR One, Avegant Glyph, Freefly VR headset, and the like. In still further embodiments, the display is a combination of devices such as those disclosed herein.

[0219] In addition to a display 1632, computer system 1600 may include one or more other peripheral output devices 1634 including, but not limited to, an audio speaker, a printer, a storage device, and any combinations thereof. Such peripheral output devices may be connected to the bus 1640 via an output interface 1624. Examples of an output interface 1624 include, but are not limited to, a serial port, a parallel connection, a USB port, a FIREWIRE port, a THUNDERBOLT port, and any combinations thereof.

[0220] In addition or as an alternative, computer system 1600 may provide functionality as a result of logic hardwired or otherwise embodied in a circuit, which may operate in place of or together with software to execute one or more processes or one or more steps of one or more processes described or illustrated herein. Reference to software in this disclosure may encompass logic, and reference to logic may encompass software. Moreover, reference to a computer-readable medium may encompass a circuit (such as an IC) storing software for execution, a circuit embodying logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware, software, or both.

[0221] Those of skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality.

[0222] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0223] The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by one or more processor(s), or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

[0224] In accordance with the description herein, suitable computing devices include, by way of non-limiting examples, server computers, desktop computers, laptop computers, notebook computers, sub-notebook computers, netbook computers, netpad computers, set-top computers, media streaming devices, handheld computers, Internet appliances, mobilesmartphones, tablet computers, personal digital assistants, video game consoles, and vehicles. Computing devices may further comprise televisions, video players, and digital music players with optional computer network connectivity are suitable for use in the system described herein. Suitable tablet computers, in various embodiments, include those with booklet, slate, and convertible configurations, known to those of skill in the art.

[0225] In some cases, the computing device includes an operating system configured to perform executable instructions. The operating system is, for example, software, including programs and data, which manages the device’s hardware and provides services for execution of applications. A server operating system may include, by way of non-limiting examples, FreeBSD, OpenBSD, NetBSD®, Linux, Apple® Mac OS X Server®, Oracle® Solaris®, Windows Server®, and Novell® NetWare®. A suitable personal computer operating system may include, by way of non-limiting examples, Microsoft® Windows®, Apple® Mac OS X®, UNIX®, and UNIX-like operating systems such as GNU / Linux®. In some cases, the operating system is provided by cloud computing. A suitable mobile smartphone operating system include, by way of non -limiting examples, Nokia® Symbian® OS, Apple® iOS®, Research In Motion® BlackBerry OS®, Google® Android®, Microsoft® Windows Phone® OS, Microsoft® Windows Mobile® OS, Linux®, and Palm® WebOS®. Suitable media streaming device operating systems may include, by way of non-limiting examples, Apple TV®, Roku®, Boxee®, Google TV®, Google Chromecast®, Amazon Fire®, and Samsung® HomeSync®. Suitable video game console operating system may include, by way of non-limiting examples, Sony® PS3®, Sony® PS4®, Sony® PS5®, Microsoft® Xbox 360®, Microsoft® Xbox One, Microsoft® Xbox Series X, Microsoft® Xbox Series S, Nintendo® Wii®, Nintendo® Wii U®, Nintendo® Switch™, and Ouya®.

[0226] Another aspect of the disclosure herein describes a non -transitory, computer-readable medium comprising executable instructions, wherein when a processor, when executing the executable instructions, performs a method as described herein.Web application

[0227] In some cases, a computer program includes a web application. In light of the disclosure provided herein, a web application, in various embodiments, utilizes one or more software frameworks and one or more database systems. In some cases, a web application is created upon a software framework such as Microsoft® .NET or Ruby on Rails (RoR). Insome cases, a web application utilizes one or more database systems including, by way of non-limiting examples, relational, non-relational, object oriented, associative, XML, and document oriented database systems. In further embodiments, suitable relational database systems include, by way of non -limiting examples, Microsoft® SQL Server, mySQL™, and Oracle®. In some cases, a web application, in various embodiments, is written in one or more versions of one or more languages. A web application may be written in one or more markup languages, presentation definition languages, client-side scripting languages, server-side coding languages, database query languages, or combinations thereof. In some cases, a web application is written to some extent in a markup language such as Hypertext Markup Language (HTML), Extensible Hypertext Markup Language (XHTML), or extensible Markup Language (XML). In some cases, a web application is written to some extent in a presentation definition language such as Cascading Style Sheets (CSS). In some cases, a web application is written to some extent in a client-side scripting language such as Asynchronous JavaScript and XML (AJAX), Flash® ActionScript, JavaScript, or Silverlight®. In some cases, a web application is written to some extent in a server-side coding language such as Active Server Pages (ASP), ColdFusion®, Perl, Java™, JavaServer Pages (JSP), Hypertext Preprocessor (PHP), Python™, Ruby, Tel, Smalltalk, WebDNA®, or Groovy. In some cases, a web application is written to some extent in a database query language such as Structured Query Language (SQL). In some cases, a web application integrates enterprise server products such as IBM® Lotus Domino®. In some cases, a web application includes a media player element. In various further embodiments, a media player element utilizes one or more of many suitable multimedia technologies including, by way of non-limiting examples, Adobe® Flash®, HTML 5, Apple® QuickTime®, Microsoft® Silverlight®, Java™, and Unity®.

[0228] Referring to FIG. 18, in a particular embodiment, an application provision system comprises one or more databases 1800 accessed by a relational database management system (RDBMS) 1810. Suitable RDBMSs include Firebird, MySQL, PostgreSQL, SQLite, Oracle Database, Microsoft SQL Server, IBM DB2, IBM Informix, SAP Sybase, Teradata, and the like. In this embodiment, the application provision system further comprises one or more application severs 1820 (such as Java servers, .NET servers, PHP servers, and the like) and one or more web servers 1830 (such as Apache, IIS, GWS and the like). The web server(s) optionally expose one or more web services via app application programming interfaces (APIs) 1840. Via a network, such as the Internet, the system provides browser-based and / or mobile native user interfaces.

[0229] Referring to FIG. 19, in a particular embodiment, an application provision system alternatively has a distributed, cloud -based architecture 1900 and comprises elastically load balanced, auto -scaling web server resources 1910 and application server resources 1920 as well synchronously replicated databases 1930.Mobile application

[0230] In some cases, a computer program includes a mobile application provided to a mobile computing device. In some cases, the mobile application is provided to a mobile computing device at the time it is manufactured. In other embodiments, the mobile application is provided to a mobile computing device via the computer network described herein.

[0231] In view of the disclosure provided herein, a mobile application may be created by using hardware, languages, and development environments. Additionally, mobile applications may be written in several languages. Suitable programming languages include, by way of non-limiting examples, C, C++, C#, Objective-C, Java™, JavaScript, Pascal, Object Pascal, Python™, Ruby, Rails, VB.NET, WML, and XHTML / HTML with or without CSS, or combinations thereof.

[0232] Suitable mobile application development environments are available from several sources. Commercially available development environments include, by way of non -limiting examples, Airplay SDK, alcheMo, Appcelerator®, Celsius, Bedrock, Flash Lite, .NET Compact Framework, Rhomobile, and WorkLight Mobile Platform. Other development environments are available without cost including, by way of non -limiting examples, Lazarus, MobiFlex, MoSync, and Phonegap. Also, mobile device manufacturers distribute software developer kits including, by way of non-limiting examples, iPhone and iPad (iOS) SDK, Android™ SDK, BlackBerry® SDK, BREW SDK, Palm® OS SDK, Symbian SDK, webOS SDK, and Windows® Mobile SDK.

[0233] Several commercial forums may be available for distribution of mobile applications including, by way of non-limiting examples, Apple® App Store, Google® Play, Chrome WebStore, BlackBerry® App World, App Store for Palm devices, App Catalog for webOS, Windows® Marketplace for Mobile, Ovi Store for Nokia® devices, Samsung® Apps, and Nintendo® DSi Shop.Standalone application

[0234] In some cases, a computer program includes a standalone application, which is a program that is run as an independent computer process, not an add-on to an existing process, e.g., not a plug-in. A standalone applications may be compiled. A compiler is a computer program(s) that transforms source code written in a programming language into binary object code such as assembly language or machine code. Suitable compiled programming languages include, by way of non-limiting examples, C, C++, Objective-C, COBOL, Delphi, Eiffel, Java™, Lisp, Python™, Visual Basic, and VB .NET, or combinations thereof. Compilation is often performed, at least in part, to create an executable program. In some cases, a computer program includes one or more executable complied applications.Web browser plus-in

[0235] In some cases, the computer program includes a web browser plug-in (e.g., extension, etc.). In computing, a plug-in is one or more software components that add specific functionality to a larger software application. Makers of software applications support plugins to facilitate third-party developers to create abilities which extend an application, to support easily adding new features, and to reduce the size of an application. When supported, plug-ins facilitate customizing the functionality of a software application. For example, plugins are commonly used in web browsers to play video, generate interactivity, scan for viruses, and display particular file types. Several web browser plug-ins including, Adobe® Flash® Player, Microsoft® Silverlight®, and Apple® QuickTime®. In some cases, the toolbar comprises one or more web browser extensions, add-ins, or add-ons. In some cases, the toolbar comprises one or more explorer bars, tool bands, or desk bands.

[0236] In view of the disclosure provided herein, several plug-in frameworks are available that facilitate development of plug-ins in various programming languages, including, by way of non-limiting examples, C++, Delphi, Java™, PHP, Python™, and VB .NET, or combinations thereof.

[0237] Web browsers (also called Internet browsers) are software applications, designed for use with network-connected computing devices, for retrieving, presenting, and traversing information resources on the World Wide Web. Suitable web browsers include, by way of non-limiting examples, Microsoft® Internet Explorer®, Mozilla® Firefox®, Google® Chrome, Apple® Safari®, Opera Software® Opera®, and KDE Konqueror. In some cases, the webbrowser is a mobile web browser. Mobile web browsers (also called microbrowsers, mini- browsers, and wireless browsers) are designed for use on mobile computing devices including, by way of non-limiting examples, handheld computers, tablet computers, netbook computers, subnotebook computers, smartphones, music players, personal digital assistants (PDAs), and handheld video game systems. Suitable mobile web browsers include, byway of non-limiting examples, Google® Android® browser, RIM BlackBerry® Browser, Apple® Safari®, Palm® Blazer, Palm® WebOS® Browser, Mozilla® Firefox® for mobile, Microsoft® Internet Explorer® Mobile, Amazon® Kindle® Basic Web, Nokia® Browser, Opera Software® Opera® Mobile, and Sony® PSP™ browser.Software modules

[0238] In some cases, the platforms, systems, media, and methods disclosed herein include software, server, and / or database modules, or use of the same. In view of the disclosure provided herein, software modules may be created using machines, software, and languages. The software modules disclosed herein are implemented in a multitude of ways. In various embodiments, a software module comprises a file, a section of code, a programming object, a programming structure, a distributed computing resource, a cloud computing resource, or combinations thereof. In further various embodiments, a software module comprises a plurality of files, a plurality of sections of code, a plurality of programming objects, a plurality of programming structures, a plurality of distributed computing resources, a plurality of cloud computing resources, or combinations thereof. In various embodiments, the one or more software modules comprise, by way of non -limiting examples, a web application, a mobile application, a standalone application, and a distributed or cloud computing application. In some cases, software modules are in one computer program or application. In other embodiments, software modules are in more than one computer program or application. In some cases, software modules are hosted on one machine. In other embodiments, software modules are hosted on more than one machine. In further embodiments, software modules are hosted on a distributed computing platform such as a cloud computing platform. In some cases, software modules are hosted on one or more machines in one location. In other embodiments, software modules are hosted on one or more machines in more than one location.Databases

[0239] In some cases, the platforms, systems, media, and methods disclosed herein include one or more databases, or use of the same. In view of the disclosure provided herein, many databases are suitable for storage and retrieval of medical images, medical reports, medical summaries, medical data (e.g., DICOM data, HL7 data, EMR data, pathology reports, genomic data), or any combination thereof. In various embodiments, suitable databases include, by way of non-limiting examples, relational databases, non-relational databases, object oriented databases, object databases, entity -relationship model databases, associative databases, XML databases, document oriented databases, and graph databases. Further non limiting examples include SQL, PostgreSQL, MySQL, Oracle, DB2, Sybase, and MongoDB. In some cases, a database is Internet-based. In further embodiments, a database is web -based. In still further embodiments, a database is cloud computing based. In a particular embodiment, a database is a distributed database. In other embodiments, a database is based on one or more local computer storage devices.Data transmission

[0240] The subject matter described herein, including methods and systems as described herein and may be configured to be performed in one or more facilities at one or more locations. Facility locations are not limited by country and include any country or territory. In some instances, one or more steps are performed in a different country than another step of the method. In some cases, one or more method steps involving a computer system are performed in a different country than another step of the methods provided herein. In some cases, data processing and storage are performed in a different country or location than one or more steps of the methods described herein. In some cases, one or more products or data are transferred from one or more of the facilities to one or more different facilities for analysis or further analysis. Data includes, but is not limited to, information regarding the stratification of a subject, and any data produced by the methods disclosed herein. In some cases of the methods and systems described herein, the subject information is compiled, and a subsequent data transmission step will transmit or store the subject information.

[0241] In some cases, any step of any method described herein is performed by a software program or module on a computer. In additional or further embodiments, data from any step of any method described herein is transferred to and from facilities located within the same or different countries, including analysis performed in one facility in a particular location and the data shipped to another location or directly to an individual in the same or a differentcountry. In additional or further embodiments, data from any step of any method described herein is transferred to and / or received from a facility located within the same or different countries, including analysis of a data input, such as queries, objects, properties, types, filters, tables, or any combination thereof, performed in one facility in a particular location and corresponding data transmitted to another location.Definitions

[0242] Unless defined otherwise, all terms of art, notations and other technical and scientific terms or terminology used herein are intended to have the same meaning as is commonly understood by one of ordinary skill in the art to which the claimed subject matter pertains. In some cases, terms with commonly understood meanings are defined herein for clarity and / or for ready reference, and the inclusion of such definitions herein should not necessarily be construed to represent a substantial difference over what is generally understood in the art.

[0243] As used in this specification and the appended claims, when the term “at least,” “greater than,” or “greater than or equal to” precedes the first numerical value in a series of two or more numerical values, the term “at least,” “greater than” or “greater than or equal to” applies to each of the numerical values in that series of numerical values. For example, greater than or equal to 1, 2, or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

[0244] As used in this specification and the appended claims, when the term “no more than,” “less than,” or “less than or equal to” precedes the first numerical value in a series of two or more numerical values, the term “no more than,” “less than,” or “less than or equal to” applies to each of the numerical values in that series of numerical values. For example, less than or equal to 3, 2, or 1 is equivalent to less than or equal to 3, less than or equal to 2, or less than or equal to 1.

[0245] As used in this specification, “or” is intended to mean an “inclusive or” or what is also known as a “logical OR,” wherein when used as a logic statement, the expression “A or B” is true if either A or B is true, or if both A and B are true, and when used as a list of elements, the expression “A, B or C” is intended to include all combinations of the elements recited in the expression, for example, any of the elements selected from the group consisting of A, B, C, (A, B), (A, C), (B, C), and (A, B, C); and so on if additional elements are listed.As such, any reference to “or” herein is intended to encompass “and / or” unless otherwise stated.

[0246] As used in this specification and the appended claims, the indefinite articles “a” or “an,” and the corresponding associated definite articles “the” or “the,” are each intended to mean one or more unless otherwise stated, implied, or physically impossible. Yet further, it should be understood that the expressions “at least one of A and B, etc.,” “at least one of A or B, etc.,” “selected from A and B, etc.” and “selected from A or B, etc.” are each intended to mean either any recited element individually or any combination of two or more elements, for example, any of the elements from the group consisting of “A,” “B,” and “A AND B together,” etc.

[0247] As used in this specification and the appended claims “about” or “approximately” may mean within an acceptable error range for the value, which will depend in part on how the value is measured or determined, e.g., the limitations of the measurement system. For example, “about” may mean within 1 or more than 1 standard deviation, per the practice in the art. Alternatively, “about” may mean a range of up to 20%, up to 10%, up to 5%, or up to 1% of a given value. Where values are described in the application and claims, unless otherwise stated the term “about” meaning within an acceptable error range for the particular value may be assumed.EXAMPLES

[0248] The following examples are included for illustrative purposes only and are not intended to limit the scope of the inventive concepts.Example 1 - Image Annotation

[0249] A system implements the methods disclosed herein. An image as shown in FIG. 23 is retrieved for a radiologist to analyze. The radiologist selects spine analysis module that provides an image analysis component. The spine analysis component provides annotation of an anatomic variant within the image. The annotation, or medical image finding, reads “C2 - C3: Loss of disc height.” The radiologist is provided the option to accept, reject, or edit the finding annotated on the image. If the medical image finding is accepted (with or without editing), the medical image finding is inserted as a finding in a medical report relevant to the image.Example 2 - Linked Image Finding

[0250] A finding as disclosed herein is presented as text in a medical report or medical summary of a patient “Jane Smith” shown in FIG. 24. The finding is linked to an image from which the finding was extracted. The user interacts with (e.g., mouse, eye-tracking, verbal command) the finding presented to the user. Upon interaction, the image is presented to the user in an inset image. The image is further annotated to contextualize the finding with series (“Series 5”), image (“Image 18”), and anatomy (“C5-C6”) values. The user can click on the finding to retrieve the full image for further analysis.Example 3 - Discrepancy Detection

[0251] A user is analyzing a number of medical images following the workflow represented by FIG. 25. The user, a radiologist, evaluates the images and places a number of user image findings from the images into a medical report. The user image findings are added by dictation and manual (e.g., typed) user input. A natural language processing evaluation of the report extracts a number of findings from the report. The extracted, or digested, user image findings are compared to a number of computer vision (CV) image findings digested from computer vision algorithms used to analyze the same images. Any user image finding or CV image finding that does not have a sufficient semantic similarity with a CV image finding or user image finding, respectively, is flagged as a discrepancy. Any flagged finding is evaluated by the user before inclusion or exclusion from a medical report or summary.Example 4 - First Illustrative Anatomic Navigator

[0252] An illustrative configuration for an anatomic navigator is shown in FIG. 26. The anatomic navigator is implemented as one or more software modules in a system as disclosed herein. Images are retrieved from one or both of a PACS or VNA storage solution. The images are shown in a workspace (e.g., dashboard 325). The workspace, or image viewer, presents the images to a user for analysis. A finding of an image extracted either automatically or manually is recorded with information including the coordinates of the finding in the image. The coordinates are passed to the anatomic navigator. The anatomic navigator converts the finding to a text-based finding for insertion into a medical report or medical summary. The text-based finding is passed to a reporter system for insertion into a radiology information system (RIS). Additionally, the user dictates a finding corresponding to an image shown in the workspace. The anatomic navigator associates the dictated finding to an appropriate portion of an image shown in the workspace. The appropriate portion is associated with coordinates corresponding to the portion of the image referenced by thedictated findings. The anatomic navigator can be used for transitioning between text-based findings and image findings (e.g., anatomic variant) through data stored associating textbased findings with images (and image portions via coordinates).Example 5 - Second Illustrative Anatomic Navigator

[0253] A user analyzing a number of images uses an anatomic navigator as disclosed herein and embodied in FIG. 27. The user has a number of preferences for the type of image study currently being performed. Upon acquisition, the images are subjected to one or more segmentation algorithms for detecting a number of findings in the images (e.g., anatomic variants). The segmented images are subsequently labelled or annotated by an automatic region labelling process (e.g., semantic segmentation, panoptic segmentation). The user wishes to analyze the images manually. The images with pre-computed, segmented, and labeled regions are providedto the user. For each labeled region presented to the user, which correspond to a finding as disclosed herein, the user either accepts, rejects, or edits the presented labeled regions. An anatomic mapper module of the anatomic navigator is used to retain coordinates and anatomic descriptors for each accepted, or edited and accepted, finding such that they are inserted into a report and the images are retrievable upon interaction with a finding from the image in the report (e.g., Example 2).Example 6 - Image Study Using Anatomic Navigator

[0254] A user interacts with an anatomic navigator as disclosed herein and embodied in FIG. 28A and FIG. 28B. As shown in FIG. 28A, the user (e.g., a radiologist) analyzes a portion of an image of a spine. The user dictates that “[t]here is mild broad-based disc osteophyte complex.” The anatomic navigator is configured to show a number of relevant details directed to findings of the image. In FIG. 28A, the finding is an osteophyte on the L4- L5 disk of a patient. In addition to the finding as dictated by the user, the anatomic navigator presents a number of other options for each detail of the finding. As shown, a severity (e.g., mild, moderate, sever), a type (e.g., broad -based), and an anatomy (e.g., osteophyte) of the finding is presented and further configurable by the user. The alternatives to the details provided by the user (e.g., severities of moderate and severe) are provided by the ontology or other hard -coded rule(s) as disclosed herein. In FIG. 28B, the image of FIG. 28A is shown with updated severity next to a prior report showing the same portion of the spine at an earlierpointin time. The user is able navigate the user’s relevant medical history and prior findings. The user is able to monitor and track patient health over the course of time or treatment by comparing findings of a current dictation with findings of one or more prior report. As shown, the osteophyte progresses in severity from mild to moderate.Example 7 - Image Study Templates

[0255] A user interacts with a dashboard (e.g., dashboard 325) to analyze images of a subject. The user switches between view one as shown in FIG. 29A and FIG. 29B. The user has the viewport shown in FIG. 29A presented initially due to a prior demonstrated preference for the layout shown in FIG. 29A.Example 8 - Medical Report Interaction

[0256] A medical report containing an MRI of a subject is presented to a user. The MRI is associated with a one or more report or summary as disclosed herein. Initially, the user is presented with the MRI as shown in FIG. 30A. The user interacts with a liver portion of the MRI using an input as disclosed herein (e.g., input 310). Details (e.g., findings, summary, history) of the liver of the subject are presented upon interaction with the liver as shown in FIG. 30B. The user interacts with a finding related to the liver. The liver finding further highlights the specific finding in the MRI by zooming in on the liver as shown in FIG. 30C. Further, an abdominal MRI report is presented to the user providing contextual information to the user. The abdominal MRI report indicates that the subject also had kidney surgery previously. The user commands to be shown information pertaining to the kidney. As such, the user is presented with the kidney surgical report and the MRI showing the kidney as shown in FIG. 30D. The transition between various views shown in FIGs. 30A-D are facilitated in part by links among findings as disclosed herein.Example 9 - Report Generation

[0257] A user seeks to generates a report as disclosed herein. The user follows an illustrative workflow shown in FIG. 31. The user, a radiologist is presented with an image that has been pre-processed by a computer vision algorithm to be segmented to highlight and clearly differentiate various anatomies or anatomic variants shown in an image. Upon visualization of the image, the user interacts with the image using tactile (e.g., stylus, mouse and keyboard) or virtual inputs (e.g., AR, VR, MR). The user locates a region of interest in the image and zooms in and centers a possible anatomic variant present in the image. The user dictates, inmedical jargon, a finding. The medical jargon contains abbreviated medical terms and simplifications. The dictated findings are converted into a plain-text format for insertion in a medical report.Example 10 - User Interaction with A System Herein

[0258] A user interacts with a system implementing the disclosed herein. On the far -left monitor is a work list or task list for the radiologists where they find what study they're going to read next. Two center monitors are shown for manipulating and reviewing the images, wherein on the far right, is a speech to text reporting module as well as perhaps an EMR. The practitioner opens a patient's case and verbally requests the system for a summary of the patient’s medical history. The systems herein determine the clinical scenario for a patient and the clinical question the practitioner is trying to answer. Accordingly, the practitioner is provided a summary of the patient’s medical history, contextualized by findings from the medical record. In other PAC solutions, this is commonly very hard to do. When clicking in multiple locations one either sees just the impressions or you can look further to find the full reports. And so, for most cases, the radiologists will look at the most recent prior exam that's relevant to the current exam that they're reading or maybe they'll go to two or three, depending on the type of patient and reason the practitioner is evaluating the patient.

[0259] However, herein a medical record may be gathered, summarized, and have relevant findings distributed to a template, and presented to the practitioner. This helps ensure that the practitioner does not forego relevant patient data that may otherwise be simply overlooked for time purposes. Then once the practitioner begins their interpretation, they're commonly switching between the images that they're manipulating and the report they aim to generate. The practitioner, with existing systems, must spent substantial energy maintaining careful vigilance to ensure they are putting their dictation in the right field while maintaining accuracy. This breaks their chain of concentration as they do their interpretation. With the systems described herein, shown to the user is a clinical scenario using the worklist to open a current study. The system the user interacts with is embedded with Al capabilities for hanging protocols to ensure that all of the series and images both for the priors and the current study are hung in the right way. On the left-hand monitor there is a description of relevant patient data. Through the incorporation of large language models, which are skilled at creating a synthesis of large amounts of data, the system provides a patient summary for this radiologist which includes all of the imaging history. EMR clinical backgrounds arepresented as well so that the radiologist can very quickly see the most important things that are going on with this patient. This practitioner is reading an MRI of the brain. Because of the description of relevant patient data, the practitioner knows that this is a breast cancer patient and that the practitioner is looking for metastases. The user reviews the MRI and dictates that the MRI looks normal. Accordingly, a pertinent negative finding is inserted into a medical report generated based on the practitioner’s dictation.

[0260] While preferred embodiments of the present invention have been shown and disclosed herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. It is not intended that the invention be limited by the specific examples provided within the specification. While the invention has been described with reference to the aforementioned specification, the descriptions and illustrations of the embodiments herein are not meant to be construed in a limiting sense. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. Furthermore, it shall be understood that all aspects of the invention are not limited to the specific depictions, configurations or relative proportions set forth herein which depend upon a variety of conditions and variables. It should be understood that various alternatives to the embodiments of the invention disclosed herein may be employed in practicing the invention. It is therefore contemplated that the invention shall also cover any such alternatives, modifications, variations, or equivalents. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims

CLAIMSWhat is claimed is:

1. A method for generating a medical report, comprising: a. displaying a medical image on a display; b. while a user is viewing the display, receiving a user input from the user; c. while receiving the user input, distributing a plurality of findings from the user input to a plurality of report sections of the medical report; and d. providing the medical report to the user.

2. The method of claim 1, wherein the display comprises an augmented reality, virtual reality, or mixed reality device.

3. The method of claim 1, wherein the display receives the user input.

4. The method of claim 1, further comprising modifying said medical image.

5. The method of claim 4, wherein modifying comprises rotating, zooming in, zooming out, or moving said medical image.

6. The method of claim 1, further comprising displaying a finding of the plurality of findings in a report section of the plurality of report sections while receiving the user input.

7. The method of claim 1, wherein a report section of the plurality of report sections comprises an impression section.

8. The method of claim 7, wherein the impression section is automatically generated by a large language model.

9. The method of claim 8, wherein the large language model is a multi-modal machine learning model.

10. The method of claim 9, wherein the multi-modal machine learning model extracts at least one image finding included in the impression.

11. The method of claim 7, wherein the impression section comprises a recommendation or a diagnosis.

12. The method of claim 1, further comprising analyzing said user input using a machine learning model.

13. The method of claim 1 , further comprising extracting the plurality of findings of the user input using a machine learning model.

14. The method of claim 12 or 13, wherein the machine learning model is a large language model.

15. The method of claim 1, further comprising detecting a discrepancy in the plurality of findings.

16. The method of claim 15, wherein the discrepancy is between a first finding and a second finding of the plurality of findings, the plurality of findings and a computergenerated findings, or a combination thereof.

17. The method of claim 15, wherein detecting the discrepancy comprises comparing a finding of the plurality of findings with a finding from a medical record.

18. The method of claim 17, wherein finding from the medical record is a text finding, an image finding, or a tabular finding.

19. The method of claim 15, wherein the discrepancy is between a first finding a second finding, wherein one or both of the first and second finding is a permanent finding a transient finding, a pertinent negative finding, or an incidental finding.

20. The method of claim 15, further comprising presenting the discrepancy to a user, wherein the user accepts, rejects, or edits a finding of the plurality of findings.

21. The method of claim 20, wherein upon accepting, rejecting, or editing the finding of the plurality of findings, the medical report is updated with an updated finding that reflects said accepting, rejecting, or editing.

22. The method of claim 1 , wherein the medical image comprises an x-ray, a CAT scan, a PET scan, an MRI, an ultrasound, a tomosynthesis scan, a fluoroscopy scan, or a combination thereof.

23. The method of claim 1, wherein the user input is a verbal user input.

24. The method of claim 23, wherein the verbal user input is received through an audio input device.

25. The method of claim 1, further comprising providing a link in said medical report, wherein the link corresponds to a finding of said plurality of findings, and selection of the link retrieves the medical image for viewing.

26. The method of claim 25, wherein selection of the link retrieves at least one additional medical image co-registered with the medical image.

27. The method of claim 1, further comprising providing a link in said medical image, wherein the link corresponds to a finding of said plurality of findings, and selection of the link retrieves the medical report for viewing.

28. The method of claim 1, wherein distributing of the plurality of findings into the plurality of report sections of the medical report is based at least in part on a medical report template selected by the user or for the user.

29. The method of claim 1, wherein the user selects a medical report template.

30. The method of any one of the preceding claims, wherein the user interacts with the display solely through voice commands.

31. A method for reviewing a parameter of a finding, the method comprising: a. receiving a first medical report and a second medical report; b. processing the first medical report and the second medical report to extract the finding from each of the first medical report and the second medical report using a machine learning model; c. determining a first parameter of the finding in the first medical report and a second parameter of the finding in the second medical report; and d. generating a summary comprising a statement regarding a difference of the first parameter of the finding and the second parameter of the finding.

32. The method of claim 31 , wherein the first medical report was generated at a different time than the second medical report.

33. The method of claim 31 , wherein the difference of the first parameter and the second parameter finding shows a progression or treatment of a disease.

34. The method of claim 31, wherein the summary comprises a sentence or statement regarding a change in the finding based on the difference of the first parameter of the finding and the second parameter of the finding.

35. The method of claim 31, wherein the finding is a transient finding or a permanent finding.

36. The method of claim 31, further comprising displaying a portion of the first medical report comprising the finding.

37. The method of claim 36, further comprising displaying a portion of the second medical report comprising the finding.

38. The method of claim 31, further comprising displaying a medical image of the first medical report.

39. The method of claim 38, further comprising displaying a medical image of the second medical report.

40. The method of claim 39, further comprising co-registering the medical image of the first medical report and the medical image of the second medical report.

41. The method of claim 31, wherein the finding describes a pathology or an anatomy.

42. The method of claim 31, wherein the finding is an image finding or a text finding.

43. The method of claim 31, wherein the first medical report and the second medical report were generated at least one day apart.

44. The method of claim 31, wherein the machine learning model is a large language model.

45. The method of claim 44, wherein the large language model is a multi-modal large language model.

46. The method of claim 44, wherein the large language model comprises a computer vision model.

47. The method of claim 31 , wherein the parameter of the finding comprises a size, width, density, color, diameter, shape, thickness, severity, weight, or location of the finding.

48. The method of claim 31, wherein the summary comprises a textual summary of the change of the finding, a graph of the parameter, or a table of the parameter.

49. The method of claim 31, wherein a large language model generates said summary.

50. The method of claim 31, further comprising selecting a medical report of the at least two reports from a medical record of a patient.

51. The method of claim 50, wherein said selecting is based at least in part on a similarity between a portion of the medical report and a user provided finding.

52. The method of claim 50 or 51, wherein the selecting comprises the use of natural language processing.

53. The method of claim 50 or 51, wherein the selecting comprises the use of large language model.

54. A method of reviewing a medical report of a patient, comprising: a. extracting a first finding from the medical report of the patient; b. extracting a second finding from a medical record of the patient; c. comparing, using a large language model, the first finding with the second finding to determine a discrepancy between the medical report of the patient and the medical record of the patient; d. retrieving a study of the patient upon determination of the discrepancy; e. extracting a third finding from the study of the patient;f. evaluating, the discrepancy using the large language model by comparing the discrepancy with the third finding from the study of the patient; g. outputting, from the large language model, an option for a user to accept, reject, or edit the first finding or the second finding.

55. The method of claim 54, wherein an impression section of the medical report comprises the first finding.

56. The method of claim 54, further comprising generating a sentence or statement comprising the first finding using a natural language processor.

57. The method of claim 54, wherein the second finding and the third finding are the same.

58. The method of claim 54, wherein the discrepancy is determined when the first finding and the second finding are different.

59. The method of claim 58, further comprising alerting a user of the discrepancy.

60. The method of claim 54, wherein the large language model is used to determine the third finding.

61. The method of claim 54, wherein the second finding is from a second medical report of the medical record.

62. The method of claim 54, wherein the second finding is contained in DICOM data, HL7 data, or an electronic medical record (EMR) of the medical record.

63. The method of claim 54, wherein the first finding, second finding, or both, is a text finding, an image finding, a tabular finding, or a combination thereof.

64. The method of claim 54, wherein the first finding and the second finding comprise a different modality.

65. The method of claim 54, wherein determining the discrepancy is based on a first parameter of the first finding and a second parameter of the second finding.

66. The method of claim 65, wherein the first parameter, second parameter, or both is a size, width, density, color, diameter, shape, thickness, severity, weight, location, or a combination thereof.

67. The method of claim 66, wherein the discrepancy is determined at least in part based on an input prompt used to configure the large language model.

68. The method of claim 54, wherein the study of the patient comprises a medical image.

69. The method of claim 68, wherein the third finding is an image finding of the medical image.

70. The method of claim 54, wherein the large language model is a multi-model large language model.71 . The method of claim 54, further comprising updating the medical report, the medical record, or both.

72. The method of claim 71, wherein updating the medical report comprises modifying or changing the first finding.

73. The method of claim 54, wherein a large language model is used to extract the first finding or second finding.

74. The method of claim 54, wherein the large language model for comparing the first finding with the second finding to determine the discrepancy is a first large language model, and the large language model for evaluating the discrepancy by comparing the discrepancy with the third finding from the study of the patient is a second large language model.

75. A method for monitoring a change of a finding, said method comprising: a. receiving a voice dictation from a user evaluating a medical image of a patient, wherein the voice dictation comprises a first parameter of a finding of the medical image; b. retrieving at least one prior medical report of the patient; c. obtaining a second parameter of the finding from the at least one prior medical report based on the finding of the medical image; d. comparing the first parameter of the finding to the second parameter of the finding and detecting a difference between the first parameter of the finding and the second parameter of the finding; and e. generating a statement pertaining to the difference tween the first parameter of the finding and the second parameter of the finding.

76. The method of claim 75, wherein the at least one prior medical report for the patient comprises textual data, image data, tabular data, or a combination thereof.

77. The method of claim 75, wherein the finding of the medical image comprises a transient finding, permanent finding, incidental finding, or pertinent negative finding.

78. The method of claim 75, wherein natural language processing is used to determine the finding of the medical image from the voice dictation.

79. The method of claim 75, wherein the finding from the at least one prior medical report comprises an image finding, a textual finding, or a combination thereof.

80. The method of claim 75, wherein obtaining the finding of the at least one prior medical report comprises extracting the finding from the at least one prior medical report using a machine learning model.81 . The method of claim 80, wherein the machine learning model is a large language model (LLM).

82. The method of claim 81, wherein the LLM comprises a multi-modal LLM.

83. The method of claim 75, wherein the statement is output by a large language model.

84. The method of claim 75, wherein a parameter of the finding of the medical image is compared to a parameter of the finding from the at least one medical report to provide a comparison between the findings.

85. The method of claim 75, further comprising presenting the statement to the user.

86. The method of claim 85, further comprising presenting the user an option to accept, deny, or modify the impression.

87. The method of claim 85, further comprising receiving feedback from a user about the statement of the medical image.

88. The method of claim 87, wherein the feedback comprises an acceptance, rejection, or edit of the impression.

89. The method of claim 88, wherein the feedback is used in a training set for updating a machine learning model used to generate the impression.

90. The method of claim 89, wherein the feedback and the difference between generated and submitted statement is used in the training set for updating the machine learning model used to generate the impression91 . The method of claim 75, wherein the statement is presented in an augmented, virtual, or mixed reality environment.

92. The method of claim 75, wherein the first parameter of the finding comprises an absence of the finding.

93. The method of claim 75, wherein the second parameter of the finding comprises an absence of the finding.

94. A method for training a machine learning model, comprising: a. generating a medical report from a medical record using a large language model;b. presenting the medical report to a user; c. receiving feedback from the user about the medical report; d. generating a training set to train the large language model, wherein a datum of the training set comprises at least a portion of the medical record, the medical report, and the feedback about the medical report; and e. training the machine learning model using the training set.

95. The method of claim 94, wherein the feedback comprises a user updated or filed medical report.

96. The method of claim 95, further comprising comparing the generated medical report to the user updated or filed medical report to determine a difference between the generated medical report and the user updated or filed medical report.

97. The method of claim 96, wherein the difference is an addition, a removal, or an edit of a finding of the medical report.

98. The method of claim 94, further comprising generating an updated medical report using the large language model.

99. The method of claim 98, further comprising comparing a second medical report to the updated medical report.

100. The method of claim 99, wherein a finding of the updated medical report is compared to a finding of the second medical report.

101. The method of claim 100, further comprising detecting a discrepancy between the finding of the updated medical report and the finding of the second medical report.

102. The method of any one of claims 99-101, wherein the second medical report was generated before the updated medical report.

103. The method of claim 94, further comprising generating a second medical report using the large language model.

104. The method of claim 94, wherein the medical report comprises an impression.

105. The method of claim 94, wherein the medical report comprises a summary of the medical record.

106. The method of claim 94, wherein the medical report comprises a finding.

107. The method of claim 94, wherein feedback is received via virtual, mixed, or augmented reality device.

108. The method of claim 94, wherein the medical report is presented to the user via virtual, mixed, or augmented reality device.

109. The method of claim 94, wherein the large language model is a multi-modal large language model.

110. The method of claim 94, wherein the feedback from the user comprises an acceptance, a rejection, or an edit of the medical report.

111. The method of claim 94, further comprising detecting a discrepancy in the medical report and prompting the user to provide the feedback about the discrepancy.

112. The method of claim 94, wherein the feedback is used to generate an updated medical record.

113. The method of claim 94, wherein the feedback comprises a preference of a user.

114. The method of claim 113, further comprising generating a second medical report based at least in part on a preference of the user.

115. The method of claim 113, wherein the preference comprises a display preference.

116. A method of reviewing a medical report, said method comprising: a. reviewing the medical report using a machine learning model; b. reviewing a second medical report using the machine learning model; c. detecting a discrepancy between the medical report and the second medical report; and d. generating one or more sentence or phrase describing or assessing said discrepancy.

117. The method of claim 116, wherein said machine learning model is a large language model.

118. The method of claim 117, wherein the machine learning model is prompt engineered to determine a finding.

119. The method of claim 118, wherein said finding is associated with an anatomic variant.

120. The method of claim 118, wherein said finding is a permanent finding, a transient finding, or a pertinent negative finding.

121. The method of claim 116, wherein the discrepancy is based on a difference in semantic similarity between a finding of said medical report and a finding of said second medical report.

122. The method of claim 116, wherein the discrepancy is a difference between a finding of the medical report and a finding of the medical record.

123. The method of claim 116, wherein the discrepancy is a difference between a first parameter of a finding of the medical report and a second parameter of a finding of the medical record.

124. The method of claim 116, wherein the medical report and the second medical report are based on the same medical image, patient study, or a combination thereof.

125. The method of claim 116, wherein one or more sentence or phrase describing or assessing said discrepancy is an output of a natural language processing algorithm.

126. The method of claim 125, wherein said natural language processing algorithm comprises a large language model.

127. The method of claim 126, wherein a user reviews said one or more sentence or phrase and accepts, rejects, or edits at least a portion of said one or more sentence or phrase before insertion of said one or more sentence or phrase into the medical report.

128. The method of claim 116, wherein a medical record comprising the medical report is updated based on the user review.

129. The method of claim 116, wherein the second medical report is a computergenerated medical report.

130. The method of claim 129, wherein the computer-generated medical report is based at least in part on a medical image.

131. The method of claim 130, wherein the medical reportis based at least in part on said medical image.

132. The method of claim 116, wherein the second medical report is generated by a human.

133. The method of claim 116, wherein the reviewing is an overread of the medical report.

134. A method of reviewing a medical report, said method comprising: a. providing the medical report comprising a medical image and a medical report text; b. detecting a textual finding in the medical report text; c. detecting an image finding in the medical image; d. comparing the textual finding with the image finding using a multi-modal machine learning model; ande. detecting a discrepancy when the textual finding does not match the image finding.

135. The method of claim 134, wherein said medical report is a contemporary medical report.

136. The method of claim 134, wherein said medical report is a prior medical report.

137. The method of claim 134, wherein said textual finding is a permanent finding, a pertinent negative finding, or a transient finding.

138. The method of claim 134, wherein said image finding is a permanent finding, a pertinent negative finding, or a transient finding139. The method of claim 134, wherein multi-modal machine learning model is a multimodal large language model.

140. The method of claim 139, wherein said large language model is a fine-tuned foundation model.

141. The method of claim 134, wherein said medical summary comprises text data, image data, audio data, video data, tabular data, or a combination thereof.

142. The method of claim 134, wherein said multi-modal machine learning model takes as input at least two of text data, image data, audio data, video data, or tabular data.

143. The method of claim 134, wherein said multi-modal machine learning model is trained to compare a first finding from a first data modality and a second finding from a second data modality.

144. The method of claim 143, wherein said multi-modal machine learning model comprises an encoder at the first data modality and the second data modality.

145. The method of claim 143, wherein said first data modality or second data modality is text, image, audio, video, or tabular.

146. The method of claim 143, wherein said first data modality is different from said second data modality.

147. The method of claim 134, wherein said multi-modal machine learning model is prompt engineered with a prompt to detect said discrepancy.

148. The method of claim 147, wherein said prompt is used for in -context learning.

149. The method of claim 148, wherein said prompt comprises at least one instruction.

150. The method of claim 134, further comprising alerting a user of the discrepancy.

151. The method of claim 134, wherein the medical report text is in a summary of the medical report.

152. The method of claim 134, wherein a discrepancy is detected when a parameter of the textual finding does not match a parameter of the image finding.

153. A computer-implemented method for generating a summary, the method comprising: a. receiving a medical record and a summary template; b. extracting a finding from the medical record; c. determining a finding class and a relevancy category of the finding; and d. generating the summary based at least in part on the summary template, the finding class and relevancy category of the finding.

154. The method of claim 153, wherein the summary template comprises a label for the finding class of the finding.

155. The method of claim 154, wherein the label comprises a text color, a text highlighting, a text font, an icon, or any combination thereof.

156. The method of claim 153, wherein the summary template is determined by a user.

157. The method of claim 153, wherein the summary template is determined a large language model and is based at least in part on a preference of the user.

158. The method of claim 153, wherein the finding class comprises a permanent finding class, a pertinent negative finding class, or a transient finding class.

159. The method of claim 158, wherein the summary does not comprise the finding with a transient class.

160. The method of claim 158, wherein the summary comprises the finding determined to have a transient finding class if a practitioner comment is received regarding the finding comprising the finding class comprising the transient finding.

161. The method of claim 153, wherein the relevancy category comprises a specific relevancy category if the finding is related to an anatomy, a pathology, or both, associated with the summary.

162. The method of claim 153, wherein the relevancy category is determined at least in part based on the finding class.

163. The method of claim 153, wherein the summary template comprises a preferred order of one or more findings wherein the preferred order is based at least in part on: the determined finding class of each of the one or more findings, a user preference, or both.

164. The method of claim 163, wherein the summary displays the preferred order of the one or more findings.

165. The method of claim 163, wherein summary is further generated based on Digital Imaging and Communications in Medicine (DICOM) data, Health Level Seven (HL7) data, or both.

166. The method of claim 163, further comprising receiving a medical image and determining an anatomy shown in the medical image, wherein the relevancy category is further determined based at least in part on the anatomy.

167. The method of claim 163, further comprising receiving a feedback from a user regarding the summary, and wherein the summary is further generated based on the feedback.

168. The method of claim 163, wherein the preference of the user is based on a specialty of the user.

169. The method of claim 163, wherein the preference of the user is based on a previous user interaction with a summary of a prior medical record.

170. The method of claim 163, wherein a second finding of the one or more findings is not included in the summary.

171. The method of claim 153, further comprising extracting a second finding from a medical image or the medical record.

172. The method of claim 171, further comprising determining a second finding class and a second relevancy category of the second finding, wherein the summary of the medical record is further based at least in part on the second finding class and the second relevancy category of the second finding.

173. The method of claim 153, wherein the summary is a summary of the medical report.

174. A computer-based method for generating a medical report of a patient, the method comprising: a. receiving a medical report template, a finding regarding the patient, and a prior medical report associated with a user, the patient, the finding, or any combination thereof; b. reviewing the medical report template, the prior medical report, and the finding using a machine learning model; and c. generating the medical report based at least in part on the medical report template, the prior medical report, and the finding.

175. The method of claim 174, wherein the medical report template comprises a plurality of fields.

176. The method of claim 175, wherein a field of the plurality of fields comprises an impression.

177. The method of claim 174, wherein the finding regarding the patient is extracted from a user input of the user.

178. The method of claim 174, wherein the finding regarding the patient is extracted from an image study of the patient.

179. The method of claim 174, wherein the medical report is generated by: determining one or more fields of the plurality of fields related to the finding, and inserting a sentence or phrase related to the finding into the one or more fields.

180. The method of claim 175 or 179, wherein the medical report is generated by determining one or more fields of the plurality of fields that are represented by a portion of the prior medical report and inserting the portion of the prior medical report into the one or more fields.

181. The method of claim 179, wherein determining one or more fields of the plurality of fields that are related to the finding comprises the use of a machine learning module.

182. The method of claim 179, wherein a natural language processing module is used to generate the sentence or phrase related to said finding into the one or more fields.

183. The method of claim 174, further comprising receiving a report feedback from the user regarding the medical report, and wherein a second medical report or an updated medical report is generated based at least in part on the report feedback.

184. The method of claim 174, further comprising receiving a medical record of the patient, wherein the medical report is further generated based at least in part on the medical record.

185. The method of claim 174, further comprising determining a finding class of the finding, wherein the medical reportis further generated based at least on the finding class.

186. The method of claim 185, wherein the finding class comprises a permanent finding class, a pertinent negative class, or a transient finding class.

187. The method of claim 174, wherein the medical report template is selected by a user.

188. The method of claim 174, wherein the medical report template is selected through the use of a machine learning module.

189. The method of claim 188, wherein the machine learning module selects the medical report template based at least in parton a prior action by the user or a preference of the user.

190. A computer-based method for generating a medical report, the method comprising: a. receiving a finding regarding a patient; b. storing the finding in a medical record of the patient; c. receiving a request to generate a first medical report, and a first template; d. generating the first medical report according to the first template based at least in part on the medical record and the finding; e. receiving a second request to generate a second medical report and a second template; and f. generatingthe second medical report according to the second template based at least in part on the medical record and the finding.

191. The method of claim 190, wherein the finding regarding the patient is received from a user.

192. The method of claim 191, wherein the finding comprises a dictated finding from the user.

193. The method of claim 190, wherein the first medical report and the second medical report comprise the finding.

194. The method of claim 190, wherein the finding regarding the patient is a computergenerated finding.

195. The method of claim 190, wherein the first template is based at least in part on a preference of a first user.

196. The method of claim 190, wherein the first template is based at least in part on a preference of a second user.

197. The method of claim 196, wherein the first template and the second template are different.

198. The method of claim 190, wherein the first template and the second template each comprise a plurality of fields.

199. The method of claim 198, wherein at least one field of the plurality of fields of the first template is different from at least one field of the plurality of fields of the second template.

200. The method of claim 198, further comprising determining one or more of the plurality of fields that is represented by the finding and further inputting a sentence or phrase comprising the finding into the determined one or more.

201. The method of claim 190, further comprising receiving a report feedback from a user regarding the first medical report and wherein the second medical report is further generated based on the report feedback.

202. The method of claim 190, wherein the first template is different from the second template.

203. The method of claim 202, wherein the first medical report is different from the second medical report.

204. The method of claim 190, wherein a machine learning module selects the first template, the second template, or both.

205. The method of claim 190, wherein a user selects the first template.

206. The method of claim 205, wherein a second user selects the second template.

207. The method of claim 190, wherein the first template is based at least in part on a preference of a first user.

208. The method of claim 190, wherein the second template is based at least in part on a preference of a second user.

209. A method of reviewing a medical report of a patient, comprising: a. extracting a first finding from the medical report of the patient; b. extracting a second finding pertaining to the patient from the medical report; c. comparing, using a multi-modal language model, the first finding with the second finding to determine a discrepancy between the first finding and the second finding.

210. The method of claim 209, wherein the medical report comprises a lab result or imaging pertaining to the patient.

211. The method of claim 209, wherein the first finding and the second finding comprise a gender of the patient.

212. The method of claim 209, wherein the first finding and the second finding comprise an anatomy.

213. The method of claim 209, wherein the discrepancy comprises an inconsistency between the first finding and the second finding.

214. The method of claim 209, wherein the medical report comprises the second finding.

215. The method of claim 209, wherein a medical image comprises the second finding.

216. The method of claim 209, wherein a lab result comprises the second finding.

217. The method of claim 209, further comprising: retrieving a study of the patient upon determination of the discrepancy; extracting a third finding from the study of the patient; evaluating, the discrepancy using the model by comparing the discrepancy with the third finding from the study of the patient; outputting, from the large language model, an option for a user to accept, reject, or edit the first finding or the second finding.

218. A computer system, comprising: a. a processor; b. a display; and c. a non-transitory computer readable storage medium encoded with a computer program that causes said processor to perform any one of the methods of any one of the preceding claims.

219. The system of claim 218, wherein the display comprises a virtual reality, augmented reality, or mixed reality display.

220. A non-transitory computer-readable storage media encoded with a computer program including instructions executable by a processor to create an application, the application configured to perform any one of the methods of any one of the preceding claims.-HO-

Images