System and method for automated generation of a clinical report

Abstract

Disclosed herein are systems, methods, and devices for automated generation of a clinical report. A study data structure is generated to include study data representative of a radiological study of a particular patient, and study metadata. The study data structure is applied to a machine learning model to generate a draft report for the radiological study emulating a report style of the particular clinician. The draft report is provided to the particular clinician. A final report incorporating modifications made by the particular clinician is received. A reward for the machine learning model is generated by applying the draft report and the final report to a reward model such that the final report is ranked more favorably than the draft report. The reward is provided to the machine learning model.

Description

SYSTEM AND METHOD FOR AUTOMATED GENERATION OF A CLINICAL REPORTCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims all benefit including priority to U.S. Provisional Patent Application 63 / 540,452, filed September 26, 2023, and entitled “SYSTEM AND METHOD FOR AUTOMATED GENERATION OF A CLINICAL REPORT”; the entire contents of which are hereby incorporated by reference herein.FIELD

[0002] This disclosure relates to medical imaging, and more particularly relates to automated generation of clinical reports.BACKGROUND

[0003] Medical imaging plays a crucial role in the diagnosis and treatment of various medical conditions. Radiologists and other clinicians, as specialized medical professionals, interpret these images and provide detailed reports to guide clinical decisions. The process of creating clinical reports on radiological studies traditionally involves manual interpretation of images followed by the creation of textual reports. This process can be time-consuming and prone to human error.

[0004] Efforts have been made to improve and streamline the reporting process using technology. Computer-aided detection (CAD) systems have been developed to assist clinicians in identifying abnormalities within images, but these systems have limitations in generating relevant textual reports. Additionally, while speech recognition software has been employed to transcribe spoken interpretations, it does not assist in the analysis and synthesis of findings required for a clinical report.

[0005] Furthermore, the increasing demand for radiological services due to the growing prevalence of imaging studies, along with a shortage of clinicians in some regions, has created a pressing need for more efficient reporting solutions. Theexisting manual reporting process, often augmented by pre-defined templates and / or macro text, struggles to keep pace with the volume of imaging studies, potentially leading to delays in diagnosis and treatment.

[0006] Accordingly, there is a need for improved or alternate ways of assisting clinicians in creating clinical reports.SUMMARY

[0007] In accordance with one aspect, there is provided a computer-implemented method for automated generation of a clinical report. The method includes maintaining a machine learning model trained to generate clinical reports emulating report styles of particular clinicians and a reward model for providing rewards to the machine learning model; generating a study data structure including: study data representative of a radiological study of a particular patient, the study data including at least one of: a study image; or technologist notes associated with the study image; and study metadata including an identifier of a particular clinician. The method further includes applying the study data structure to the machine learning model to generate a draft report for the radiological study emulating a report style of the particular clinician; providing the draft report to the particular clinician; receiving a final report incorporating modifications made by the particular clinician; generating a reward for the machine learning model by applying the draft report and the final report to the reward model such that the final report is ranked more favorably than the draft report; and providing the reward to the machine learning model.

[0008] In accordance with another aspect, there is provided a computer- implemented system for automated generation of a clinical report. The system includes a processing subsystem that includes one or more processors and one or more memories coupled with the one or more processors. The processing subsystem is configured to cause the system to: maintain a machine learning model trained to generate clinical reports emulating report styles of particular clinicians anda reward model for providing rewards to the machine learning model; generate a study data structure including: study data representative of a radiological study of a particular patient, the study data including at least one of: a study image; or technologist notes associated with the study image; and study metadata including an identifier of a particular clinician; apply the study data structure to the machine learning model to generate a draft report for the radiological study emulating a report style of the particular clinician; provide the draft report to the particular clinician; receive a final report incorporating modifications made by the particular clinician; generate a reward for the machine learning model by applying the draft report and the final report to the reward model such that the final report is ranked more favorably than the draft report; and provide the reward to the machine learning model.

[0009] In accordance with a further aspect, there is provided a non-transitory computer-readable medium or media having stored thereon machine interpretable instructions which, when executed by a processing system, cause the processing system to perform method for automated generation of a clinical report. The method includes: maintaining a machine learning model trained to generate clinical reports emulating report styles of particular clinicians and a reward model for providing rewards to the machine learning model; generating a study data structure including: study data representative of a radiological study of a particular patient, the study data including at least one of: a study image; or technologist notes associated with the study image; and study metadata including an identifier of a particular clinician; applying the study data structure to the machine learning model to generate a draft report for the radiological study emulating a report style of the particular clinician; providing the draft report to the particular clinician; receiving a final report incorporating modifications made by the particular clinician; generating a reward for the machine learning model by applying the draft report and the final report to thereward model such that the final report is ranked more favorably than the draft report; and providing the reward to the machine learning model.

[0010] Many further features and combinations thereof concerning embodiments described herein will appear to those skilled in the art following a reading of the instant disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the figures,

[0012] FIG. 1 is a network diagram of a network environment of a clinical report generation system, in accordance with an embodiment;

[0013] FIG. 2 is a high-level schematic diagram of the clinical report generation system of FIG. 1, in accordance with an embodiment;

[0014] FIG. 3 is a high-level schematic diagram of a report generation engine and a reinforcement learning from human feedback engine of the clinical report generation system of FIG. 1, in accordance with an embodiment;

[0015] FIG. 4 is a high-level schematic diagram of a machine learning model of the clinical report generation system of FIG. 1, in accordance with an embodiment;

[0016] FIG. 5 is a data flow diagram of the clinical report generation system of FIG. 1, in accordance with an embodiment;

[0017] FIG. 6A and FIG. 6B show an example data structure containing study metadata for an x-ray imaging study, in accordance with an embodiment;

[0018] FIG. 7A shows example x-ray images of an x-ray imaging study, in accordance with an embodiment;

[0019] FIG. 7B shows x-ray images of prior imaging studies, in accordance with an embodiment;

[0020] FIG. 8 shows an example draft report for an x-ray imaging study, as generated by the clinical report generation system of FIG. 1, in accordance with an embodiment;

[0021] FIG. 9 shows an example revised draft report, as edited by a clinician, in accordance with an embodiment;

[0022] FIG. 10A and FIG. 10B show an example data structure containing study metadata for an ultrasound imaging study, in accordance with an embodiment;

[0023] FIG. 11 shows an example scanned image of a technologist’s notes for an ultrasound imaging study, in accordance with an embodiment;

[0024] FIG. 12 shows an example draft report for an ultrasound imaging study, as generated by the clinical report generation system of FIG. 1, in accordance with an embodiment; and

[0025] FIG. 13 is a schematic diagram for a computing device, in accordance with an embodiment.

[0026] These drawings depict exemplary embodiments for illustrative purposes, and variations, alternative configurations, alternative components and modifications may be made to these exemplary embodiments.DETAILED DESCRIPTION

[0027] FIG. 1 is a network diagram depicting a network environment of a clinical report generation system 100, in accordance with an embodiment. As detailed herein, clinical report generation system 100 is configured for automated generation of clinical reports for medical imaging studies using one or more machine learningmodels. These clinical reports are generated to emulate report styles of particular identified clinicians (e.g., a particular clinician conducting the medical imaging study).

[0028] Conveniently, the particular clinician is provided with a draft clinical report generated with content emulating his or her own reporting style. The clinician can adopt the draft report as a final report as is, or optionally upon making edits. Further, in accordance with this embodiment, in the event that the clinician makes edits to the draft report, these edits are utilized in a reinforcement learning from human feedback (RLHF) process, which may improve the aforementioned machine learning models over time.

[0029] A clinical report, as contemplated in this disclosure, refers to a structured document that conveys the results, interpretations, and diagnostic findings based on medical imaging data of an imaging study. Such medical imaging data may include, but is not limited to, X-ray images, computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, ultrasound images, positron emission tomography (PET) scans, or any other form of medical images used for diagnostic or monitoring purposes. A clinical report may be structured in various sections including, for example:

[0030] Patient Information: one or more patient identifiers and descriptors, such as name, date of birth, sex, medical record number, and examination date, or the like.

[0031] Clinical History: information about the patient’s medical history, symptoms, and any specific clinical questions or concerns provided by a referring physician.

[0032] Findings: the clinician’s observations, interpretations, and diagnostic conclusions based on the analysis of the medical images. It includes descriptions of anatomical structures, the presence or absence of abnormalities, measurements, and other relevant details.

[0033] Opinion: overall conclusions based on the findings. Recommendations for further actions, such as additional imaging studies, consultations, or treatments, may also be included here.

[0034] Of course, the structure and content of a clinical report may vary, and embodiments clinical report generation system 100 are compatible with various forms of clinical reports.

[0035] Continuing to refer to FIG. 1, the network environment includes clinical report generation system 100 interconnected with a picture archiving and communication system (PACS) 10, a radiology information system (RIS) 20, and a plurality of clinician devices 80, by way of a communication network 50.

[0036] PACS 10 is a system for storage and retrieval data of medical images and related documentation. Images may include archived medical images from various imaging studies. Such medical images may be stored, for example in a Digital Imaging and Communications in Medicine (DICOM) format or another suitable format. Related documentation may include, for example, technologists’ notes (e.g., in the form of a worksheet or otherwise) prepared when gathering particular medical images and stored in association therewith, requisitions for the medical study, or the like. Such related documentation may be stored as scanned images, or another format such as text, audio, or a combination of the foregoing. In some embodiments, related documentation are stored solely as scanned images without accompanying text data as may be obtained through an optical-character recognition process or similar.

[0037] RIS 20 is a system for storing data of clinical reports (e.g., for prior imaging studies). Such data include identifiers of the particular clinician who wrote each of the clinical reports.

[0038] Each clinician device 80 is a device operable by a clinician to view and edit draft reports generated by clinical report generation system 100 (or generation system 100 for brevity). For example, a clinician device 80 may be a personal computing device such as a desktop computer, a laptop computer, a tablet computer, a smartphone, each having an integrated display screen and input interface for inputting edits.

[0039] Network 50 may include a packet-switched network portion, a circuit- switched network portion, or a combination thereof. Network 50 may include wired links, wireless links such as radio-frequency links or satellite links, or a combination thereof. Network 50 may include wired access points and wireless access points. Portions of network 50 could be, for example, an IPv4, IPv6, X.25, IPX or similar network. Portions of network 50 could be, for example, a GSM, GPRS, 3G, LTE, 5G or similar wireless networks. Network 50 may include or be connected to the Internet. When network 50 is a public network such as the public Internet, it may be secured as a virtual private network.

[0040] FIG. 2 is a high-level schematic diagram of generation system 100, in accordance with an embodiment.

[0041] As depicted, generation system 100 includes a report generation engine 102, an input collector 104, a generation model trainer 106, a data extractor 108, a clinician input interface 110, and an RLHF engine 112.

[0042] Report generation engine 102 is configured to generate draft clinical reports. Report generation engine 102 includes one or more machine learning models configured to map input data (e.g., of medical images and / or technologists’ notes regarding medical images of a particular imaging study and study metadata) to output data (e.g., text data representing a draft report for the imaging study in the style of the particular clinician, as identified).

[0043] Study metadata may include one or more of an identifier of a particular clinician conducting the study, patient age, patient sex, what examinations are being done for the study, clinical indication (e.g., chest / hip pain).

[0044] The input data provided to the machine learning models may also include data relating to prior imaging studies, such as, for example, medical images and / or technologists’ notes for examinations performed prior to the current study, identifiers of the clinician(s) who conducted the prior imaging studies, the date of the studies, etc.

[0045] Input collector 104 is configured to generate one or more data structures containing input data to be provided to report generation engine 102. The data structures may include data structure suitable for storing text data such as JSON, XML, or the like. The data structures may include data structures suitable for storing image data. The image data may be in the form of binary data of images, or textual data of embeddings of images, or a combination thereof. The images may include medical images, and / or images representing scans of technologists’ notes or other related documentation.

[0046] Generation model trainer 106 is configured to train the one or more machine learning models of report generation engine 102. Generation model trainer 106 may train the one or more machine learning models using training data that comprises data pairs, each pair including (i) historical medical images of a particular imaging study, and / or a technologist’s notes associated with the medical images, and an identifier of a particular clinician conducting the study, and (ii) historical reports written for the particular imaging study by the particular clinician, as identified. In some embodiments, generation model 106 may be configured train one or more machine learning models of report generation engine 102 using an optimizer algorithm such as, for example, stochastic gradient descent or Adam, with a backpropagation of errors algorithm.

[0047] Data extractor 108 is configured to receive data from PACS 10 and RIS 20, and provide such data to input collector 104 and generation model trainer 106, as required. Such data may include, for example, data of medical image and related documentation received from PACS 10, and data of clinical reports received from RIS 20.

[0048] Clinician input interface 110 is configured to provide draft clinical reports generated by report generation engine 102 to a clinician device 80. For a clinical report drafted for an imaging study being conducted by a particular clinician (as identified), the draft report is transmitted to the clinician device 80 operated by that particular clinician. Clinician input interface 110 is also configured to receive revised clinical reports, each as edited by the particular clinician.

[0049] In some embodiments, clinician input interface 110 receives data (such as Unicode of ASCII text data) representing the revised report. In some embodiments, clinician input interface 110 receives data representing a delta between the draft report and the revised report. In some embodiments, clinician input interface 110 provides a user interface (e.g., graphical user interface providing a text editing tool) allowing the clinician to make edits.

[0050] RLHF engine 112 is configured to generate rewards based on one or more reward models. The rewards are provided to one or more of the machine learning models of report generation engine 102. In accordance with an embodiment depicted in FIG. 3, RLHF engine 112 includes a reward model 113 and report generation engine 102 includes a machine learning model 103. Reward model 113 generates a reward, which is provided to machine learning model 103 in the form of reward signal 114.

[0051] In the depicted embodiment, reward model 113 is configured to receive a pair of reports, namely, a draft report generated by report generation engine 102 emulating the reporting style of a particular clinician, and the same report in finalform as edited by the emulated clinician. Reward model 113 is configured to generate, for each of the two reports, a score reflective of a likelihood that the report was written by a human, and more specifically, the emulated clinician. Reward model 113 is configured to rank the edited report more favourably than the draft report, and to generate a reward based on such ranking. This generated reward is provided to machine learning model 103. In the depicted embodiment, machine learning model 103 is updated using the reward based on proximal policy optimization (PPO).

[0052] Conveniently, in the depicted embodiment, machine learning model 103 is trained using a combination of supervised learning (by operation of generation model trainer 106) and reinforcement learning (by operation of RLHF engine 112). The reinforcement learning may improve machine learning model 103 over time.Conveniently, the reinforcement learning is performed opportunistically as clinicians edit draft reports to prepare final reports.

[0053] FIG. 4 is a high-level schematic diagram of machine learning model 103, in accordance with an embodiment.

[0054] In the depicted embodiment, machine learning model 103 implements an adaption of the BLIP-2 architecture, as described in Li, Junnan, et al. “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.” arXiv preprint arXiv:2301 .12597 (2023).

[0055] As depicted, machine learning model 103 is a multi-modal encoderdecoder model. Machine learning model 103 includes a vision encoder 160, a query transformer 162, and a language decoder 164. Vision encoder 160 may be a pretrained frozen vision encoder. In various embodiments, vision encoder 160 may implement using one or more vision-language representation models such as ViT, BEiT, DeiT, Swin, or the like. Language decoder 164 may be a pretrained frozen language decoder. In various embodiments, language decoder 164 may implement using one or more language models such as RoBERTa, GPT, BERT, FLAN-T5, orthe like. Query transformer 162 is trained, alongside an image transformer for cooperation with vision encoder 160 and a text transformer for cooperation with language decoder 164, while enforcing a set of learned queries 170.

[0056] In some embodiments, adapters such as low-rank adapters may be added to machine learning model 103. In some embodiments, quantization may be employed to more efficiently train machine learning model 103.

[0057] When report generation engine 102 receives data structures containing input data from input collector 104, image data 246 (e.g., medical images or scanned images of the study) are provided to vision encoder 160. Further, study metadata 172 (e.g., text data) are provided to one or both of query transformer 162 and language decoder 164. Conveniently, in some embodiments, providing study metadata 172 to both query transformer 162 and language decoder 164 improves the generated draft report (e.g., improves the scoring as assessed by reward model 113).

[0058] Each of report generation engine 102, input collector 104, generation model trainer 106, data extractor 108, clinician input interface 110, and RLHF engine 112 may be implemented using conventional programming languages such as Java, J#, C, C++, C#, Perl, Visual Basic, Ruby, Python, Scala, etc. These components of system 100 may be in the form of one or more executable programs, scripts, routines, statically / dynamically linkable libraries, or the like.

[0059] In some embodiments, generation system 100 may include a vector database for storing embeddings of image data (e.g., of medical images or related documentation) or other study and report data. In such embodiments, input collector 104 may retrieve at least some image data from this vector database rather than from PACS 10.

[0060] The operation of generation system 100 is further described with reference to the data flow diagram of FIG. 5, in accordance with an embodiment.

[0061] Data extractor 108 performs study extraction 202 by retrieving imaging data of imaging studies 204 from PACS 10. Data extractor 108 also performs report extraction 206 by retrieving data of clinical reports 208 from RIS 20.

[0062] The extracted data of imaging studies 204 and data of clinical reports 208 are merged into a data structure at block 210 and prepared at block 212 to provide training data, as described herein. The training data is provided to generation model trainer 106 which applies training 220 to train one or more machine learning models (including machine learning model 103) of report generation engine 102.

[0063] Input collector 104 collects input data to be provided as input 248 to report generation engine 102. Input 248 may include data relating to, for example, clinical indication and patient history 240, previous reports 242, a clinician identifier 244, and any other study metadata disclosed herein. Input 248 may also include image data relating to, for example, medical images and / or related documentation 246 such as a technologist’s notes.

[0064] Input 248 is provided to report generation engine 102. Input 248 is applied to machine learning models of report generation engine 102 to generate a draft report 250, emulating the reporting style of the clinician identified by identifier 244.

[0065] Draft report 250 is provided to a particular clinician device 80 operated by the identified clinician via clinician input interface 110. Data reflective of edits 262 to the draft report from radiologist input 260 while preparing final report 270 are received from the clinician device 80 via clinician input interface 110.

[0066] Input 248, draft report 250, and final report 270 are merged into a data structure at block 280 and provided to RLHF engine 112. In the depictedembodiment, input to RLHF engine 112 is provided in the same form as it is provided to report generation engine 102. The draft and final reports are provided in an order such that the final report is ranked higher (more aligned) than the draft report as it pertains to emulating a clinician report. RLHF engine 112 applies the data structure to reward model 113 to generate a reward(i.e. logits or predictions corresponding to the likelihood of the reports being human generated). This reward is provided to the machine learning models of report generation engine 102, with further training 220, using, for example, proximal policy optimization.

[0067] The operation of generation system 100 is further described with reference to an example x-ray imaging study, in accordance with an embodiment.

[0068] Operation begins with collection of input 248 by input collector 104. Input 248 includes image data 246 for the x-ray imaging study and textual data of study metadata for the x-ray imaging study.

[0069] FIG. 6A and FIG. 6B depict an example JSON data structure 600 generated by input collector 104 to store study metadata for the x-ray imaging study. Referring to FIG. 6A, data structure 600 includes metadata 602 pertaining to the x- ray imaging study including a patient age, a patient sex, a date of the study, a description of the study, and a clinical indication. Importantly, metadata 602 includes an identifier of the clinician performing the imaging study, whose reporting style is to be emulated.

[0070] Data structure 600 also includes metadata 604A providing similar information for a prior study of the same patient. Of note, the clinician for a prior study may be the same or different from the clinician of the current study. Data structure 600 also includes metadata 604B and 604C (FIG. 6B), respectively providing similar information for two further prior studies of the same patient. Data structure 600 may include additional metadata 606 (e.g., for further prior studies or other metadata).

[0071] FIG. 7A depicts example image data 246 for the x-ray imaging study, including an image 702A of a frontal exam, and an image 704A of a lateral exam. Input images need not be confined to a single projection or body part.

[0072] In some embodiments, image data 246 also includes image data of images from prior imaging studies performed of the same patient, as described in data structure 600. For example, in accordance with such embodiments, FIG. 7B depicts three sets of images corresponding to three prior imaging studies, namely, (i) for a first prior study: an image 702B of a frontal exam, and an image 704B of a lateral exam, (ii) for a second prior study: an image 702C of a frontal exam, and an image 704C of a lateral exam; and (iii) for a third prior study: an image 702D of a frontal exam, and an image 704D of a lateral exam for a third prior study.

[0073] FIG. 8 shows an example draft report 250 for an x-ray imaging study, as generated by generation system 100, in accordance with an embodiment. FIG. 9 shows an example final report 270, representing a draft report 250 as edited by the clinician of the current x-ray imaging study.

[0074] The operation of generation system 100 is further described with reference to an example ultrasound imaging study, in accordance with an embodiment.

[0075] Operation begins with collection of input 248 by input collector 104. Input 248 includes image data 246 for the ultrasound imaging study and textual data of study metadata for the ultrasound imaging study.

[0076] FIG. 10A and FIG. 10B depict an example JSON data structure 1000 generated by input collector 104 to store study metadata for the ultrasound imaging study. Referring to FIG. 10A, data structure 1000 includes metadata 1002 pertaining to the ultrasound imaging study including a patient age, a patient sex, a date of the study, a description of the study, and a clinical indication. Importantly, metadata1002 also includes an identifier of the clinician performing the imaging study, whose reporting style is to be emulated.

[0077] Data structure 1000 also includes metadata 1004 providing similar information for a prior study of the same patient. Data structure 1000 may include additional metadata 1006 (e.g., for further prior studies or other metadata).

[0078] FIG. 11 depicts example image data 246 for the ultrasound imaging study, including image data for a scanned image of a technologist’s notes for the ultrasound imaging study. Of note, the language of the technologist’s notes may differ from a clinical report in its structure, content, use of terminology, etc.

[0079] FIG. 12 shows an example draft report 250 for the ultrasound imaging study, as generated by clinical report generation system 100, in accordance with an embodiment. For this example, draft report 250 was adopted as final report 270 without edits from the clinician.

[0080] In some embodiments, image data 246 may include a combination of medical images and technologists’ notes.

[0081] FIG. 13 is a schematic diagram of a computing device 1300 which may be used to implement at least some aspects of clinical report generation system 100, in accordance with an embodiment. In some embodiments, a computing device 1300 may be used to implement a clinician device 80.

[0082] As depicted, computing device 1300 includes at least one processor 1302, memory 1304, at least one I / O interface 1306, and at least one network interface 1308.

[0083] Each processor 1302 may be, for example, any type of general-purpose microprocessor or microcontroller, a digital signal processing (DSP) processor, anintegrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, a programmable read-only memory (PROM), or any combination thereof.

[0084] Memory 1304 may include a suitable combination of any type of computer memory that is located either internally or externally such as, for example, randomaccess memory (RAM), read-only memory (ROM), compact disc read-only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM), and electrically-erasable programmable read-only memory (EEPROM), Ferroelectric RAM (FRAM) or the like.

[0085] Each I / O interface 1306 enables computing device 1300 to interconnect with one or more input devices, such as a keyboard, mouse, camera, touch screen and a microphone, or with one or more output devices such as a display screen and a speaker.

[0086] Each network interface 1308 enables computing device 1300 to communicate with other components, to exchange data with other components, to access and connect to network resources, to serve applications, and perform other computing applications by connecting to a network (or multiple networks) capable of carrying data including the Internet, Ethernet, plain old telephone service (POTS) line, public switch telephone network (PSTN), integrated services digital network (ISDN), digital subscriber line (DSL), coaxial cable, fiber optics, satellite, mobile, wireless (e.g. Wi-Fi, WiMAX), SS7 signaling network, fixed line, local area network, wide area network, and others, including any combination of these.

[0087] For simplicity only, one computing device 1300 is shown but system 1000 may include multiple computing devices 1300. The computing devices 1300 may be the same or different types of devices. The computing devices 1300 may be connected in various ways including directly coupled, indirectly coupled via a network, and distributed over a wide geographic area and connected via a network (which may be referred

[0088] For example, and without limitation, a computing device 1300 may be a server, network appliance, set-top box, embedded device, computer expansion module, personal computer, laptop, personal data assistant, cellular telephone, smartphone device, LIMPC tablets, video display terminal, gaming console, or any other computing device capable of being configured to carry out the methods described herein.

[0089] The foregoing discussion provides many example embodiments of the inventive subject matter. Although each embodiment represents a single combination of inventive elements, the inventive subject matter is considered to include all possible combinations of the disclosed elements. Thus if one embodiment comprises elements A, B, and C, and a second embodiment comprises elements B and D, then the inventive subject matter is also considered to include other remaining combinations of A, B, C, or D, even if not explicitly disclosed.

[0090] The embodiments of the devices, systems and methods described herein may be implemented in a combination of both hardware and software. These embodiments may be implemented on programmable computers, each computer including at least one processor, a data storage system (including volatile memory or non-volatile memory or other data storage elements or a combination thereof), and at least one communication interface.

[0091] Program code is applied to input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices. In some embodiments, the communication interface may be a network communication interface. In embodiments in which elements may be combined, the communication interface may be a software communication interface, such as those for inter-process communication. In still other embodiments, there may be a combination of communication interfaces implemented as hardware, software, and combination thereof.

[0092] Throughout the foregoing discussion, numerous references will be made regarding servers, services, interfaces, portals, platforms, or other systems formed from computing devices. It should be appreciated that the use of such terms is deemed to represent one or more computing devices having at least one processor configured to execute software instructions stored on a computer readable tangible, non-transitory medium. For example, a server can include one or more computers operating as a web server, database server, or other type of computer server in a manner to fulfill described roles, responsibilities, or functions.

[0093] The technical solution of embodiments may be in the form of a software product. The software product may be stored in a non-volatile or non-transitory storage medium, which can be a compact disk read-only memory (CD-ROM), a USB flash disk, or a removable hard disk. The software product includes a number of instructions that enable a computer device (personal computer, server, or network device) to execute the methods provided by the embodiments.

[0094] The embodiments described herein are implemented by physical computer hardware, including computing devices, servers, receivers, transmitters, processors, memory, displays, and networks. The embodiments described herein provide useful physical machines and particularly configured computer hardware arrangements.

[0095] Of course, the above described embodiments are intended to be illustrative only and in no way limiting. The described embodiments are susceptible to many modifications of form, arrangement of parts, details and order of operation. The disclosure is intended to encompass all such modification within its scope, as defined by the claims.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for automated generation of a clinical report, the method comprising: maintaining a machine learning model trained to generate clinical reports emulating report styles of particular clinicians and a reward model for providing rewards to the machine learning model; generating a study data structure including: study data representative of a radiological study of a particular patient, the study data including at least one of: a study image; or technologist notes associated with the study image; and study metadata including an identifier of a particular clinician; applying the study data structure to the machine learning model to generate a draft report for the radiological study emulating a report style of the particular clinician; providing the draft report to the particular clinician; receiving a final report incorporating modifications made by the particular clinician; generating a reward for the machine learning model by applying the draft report and the final report to the reward model such that the final report is ranked more favorably than the draft report; and providing the reward to the machine learning model.

2. The method of claim 1 , wherein the machine learning model includes a multi-modal encoder-decoder model.

3. The method of claim 2, wherein the machine learning model includes a vision encoder, a query transformer, and a language decoder.

4. The method of claim 3, wherein the applying includes providing image data of the study data to the vision encoder.

5. The method of claim 3, wherein the applying includes providing the study metadata to the query transformer.

6. The method of claim 3, wherein the applying includes providing the study metadata to the language decoder.

7. The method of claim 1 , wherein the study metadata further includes at least one of: a patient age; a patient sex; an identifier of the study; a clinical indication of the study; a date of the study; a date of prior study; or a prior study report.

8. The method of claim 1 , wherein the machine learning model is trained using prior study data, prior reports written in connection with the prior study data, and identifiers of clinicians who wrote the prior reports.

9. The method of claim 1 , wherein the output of the reward model is used to update the machine learning model through proximal policy optimization.

10. A computer-implemented system for automated generation of a clinical report, the system comprising: a processing subsystem that includes one or more processors and one or more memories coupled with the one or more processors, the processing subsystem configured to cause the system to: maintain a machine learning model trained to generate clinical reports emulating report styles of particular clinicians and a reward model for providing rewards to the machine learning model; generate a study data structure including: study data representative of a radiological study of a particular patient, the study data including at least one of: a study image; or technologist notes associated with the study image; and study metadata including an identifier of a particular clinician;apply the study data structure to the machine learning model to generate a draft report for the radiological study emulating a report style of the particular clinician; provide the draft report to the particular clinician; receive a final report incorporating modifications made by the particular clinician; generate a reward for the machine learning model by applying the draft report and the final report to the reward model such that the final report is ranked more favorably than the draft report; and provide the reward to the machine learning model.

11. The system of claim 10, wherein the machine learning model includes a multi-modal encoder-decoder model.

12. The system of claim 11 , wherein the machine learning model includes a vision encoder, a query transformer, and a language decoder.

13. The system of claim 12, wherein image data of the study data is provided to the vision encoder.

14. The system of claim 12, wherein the study metadata is provided to the query transformer.

15. The system of claim 12, wherein the study metadata is provided to the language decoder.

16. The system of claim 10, wherein the study metadata further includes at least one of: a patient age;a patient sex; an identifier of the study; a clinical indication of the study; a date of the study; a date of prior study; or a prior study report.

17. The system of claim 10, wherein the machine learning model is trained using prior study data, prior reports written in connection with the prior study data, and identifiers of clinicians who wrote the prior reports.

18. The system of claim 10, wherein the output of the reward model is used to update the machine learning model through proximal policy optimization.

19. A non-transitory computer-readable medium or media having stored thereon machine interpretable instructions which, when executed by a processing system, cause the processing system to perform method for automated generation of a clinical report, the method comprising: maintaining a machine learning model trained to generate clinical reports emulating report styles of particular clinicians and a reward model for providing rewards to the machine learning model; generating a study data structure including: study data representative of a radiological study of a particular patient, the study data including at least one of:a study image; or technologist notes associated with the study image; and study metadata including an identifier of a particular clinician; applying the study data structure to the machine learning model to generate a draft report for the radiological study emulating a report style of the particular clinician; providing the draft report to the particular clinician; receiving a final report incorporating modifications made by the particular clinician; generating a reward for the machine learning model by applying the draft report and the final report to the reward model such that the final report is ranked more favorably than the draft report; and providing the reward to the machine learning model.

Images