On-the-fly improvement of artificial intelligence detected pathologies in medical images

Abstract

In a clinical finding recommender method, artificial intelligence (AI) clinical finding detectors are applied to at least one image of a medical imaging study being read to generate probabilities for a plurality of AI clinical findings. The clinical finding probabilities are adjusted based on statistical correlations between the AI clinical findings to generate adjusted clinical finding probabilities. One or more proposed clinical findings are selected for which the corresponding adjusted probabilities meet a selection criterion. The selected one or more proposed clinical findings are presented during the reading of the medical imaging study. On user review of a presented proposed clinical finding, its probability is set to a fixed value indicated by the user review, and the adjusting, selecting, and presenting are repeated to update the selected one or more proposed clinical findings.

Description

ON-THE-FLY IMPROVEMENT OF ARTIFICIAL INTELLIGENCE DETECTED PATHOLOGIES IN MEDICAL IMAGESFIELD

[0001] The following relates generally to the radiology arts, radiology report generation arts, clinical finding arts, artificial intelligence (Al) arts, and related arts.BACKGROUND

[0002] When a patient is injured, has a chronic medical condition, or otherwise requires medical imaging, an examination order is often transmitted from a Radiology Information System (RIS) or Hospital Information System (HIS) or the like to schedule a patient for radiological scanning. The patient is scanned with an imaging system (e.g., CT, x-ray, et cetera). The medical images are transmitted to a workstation (e.g., CT workstation, x-ray workstation, independent radiology workstation, physician computers, departmental workstations, e.g., cardiology department workstation, radiology department workstation, oncology department workstation, or the like), Picture Archive Communications System (PACS), or similar type of reading station.

[0003] By way of example, a PACS is a specialized apparatus and technology for medical imaging that provides support for a radiologist (or cardiologist in the case of cardiovascular imaging, but radiologist use cases will be exemplified more readily throughout this disclosure) in reading (e.g., accessing, processing, and / or analyzing) images created by different imaging modalities such as, but not limited to, Digital Radiography (DR), Computed Tomography (CT), Magnetic Resonance (MR), Ultrasound (US), and Nuclear Medicine (NM). A PACS can be designed in many different ways. An exemplary PACS technology infrastructure may include imaging device interfaces, storage devices, host computers, communication networks, and display systems often integrated by a flexible software package for supporting a radiologist in reading a patient case (or otherwise referred to as reading an image study) through a diagnostic workflow. Common specialized hardware components may include for example, patient data servers, data / modality interfaces, PACS controllers with database and archive, and display workstations connected by communication networks for handling and managing efficient data / image flow. A PACS is a synergy of specialized hardware and flexible software. The flexible software also includes various functions such as for example, but not limited to,measurement, segmentation, tumor or lesion identification, landmark detection, visualization, and reporting on clinical findings.

[0004] Medical images or image studies are typically transmitted to a PACS electronically / digitally e.g., via a communication channel and / or via the Digital Imaging and Communications in Medicine (DICOM) protocol, which includes a file format definition and a network communications protocol, and uses for example, Transmission Control Protocol (TCP) / Internet Protocol (IP) TCP / IP to communicate between systems. The images are assigned to a radiologist(s) and displayed in a worklist for a radiologist of the group who will read the images at the radiology reading station. The images to be read and other non-image patient information are often referred to collectively as an image study. The radiologist loads and reads the image study (including the images) at the radiology reading station, reviews regions of interest in the image(s), identifies clinical findings in the image(s), and creates a report with a diagnosis, which is saved (e.g., in the PACS or to some other information system such as a RIS, HIS, EMR, et cetera). As a nonlimiting illustrative example oncological imaging study, a primary region of interest may be a malignant tumor and surrounding tissue, and findings may include by way of nonlimiting illustrative example, physical dimensions of the tumor, metrics of tumor density, a finding of whether the cancer has metastasized (and if so metrics of the extent thereof), and / or so forth. The radiologist prepares a report (for example, called a radiology report or the like) summarizing these findings, the report may be transmitted to another information system via HL.7 and the images (or certain key image(s) identified by the radiology reading) may be transmitted via DICOM. Non-image data, such as a scanned document, may be incorporated, e.g., using formats such as Portable Document Format (PDF).

[0005] The PACS may also provide a reporting environment sometimes employing a report template that is filled in by the radiologist, an image viewer (integrated with the reporting environment and / or standalone), and so forth. To assist the radiologist in reading an image study, the PACS may provide various manual, semi-automated, or automated image analysis tools, such as: anatomy labeling; contouring (i.e., identifying the boundary of) a tumor or other region of interest; measurement of clinically significant metrics such as tumor dimensions and / or tumor density; comparison of a current image with a corresponding image of a previous study of the patient (e.g., to assess tumor growth or shrinkage). In the case of cardiology PACS, cardiovascular metrics may be assessed such as artery wall thickening; and / or so forth. Suchimage analysis tools reduce the time and effort by the radiologist (or cardiologist as the case may be) in reading an imaging study, and thereby greatly increase the throughput, efficiency, and clinical accuracy of study readings.

[0006] Furthermore, the PACS may include various artificial intelligence (Al) algorithms to analyze medical images to automatically detect various clinical findings, such as detecting a meniscus tear, a bone fracture, potentially malignant lesions, and so forth. In most jurisdictions, the radiology reading must be performed by a suitably trained and credentialed radiologist. Radiologists must often complete a review of an image study within a certain period of time, which is sometimes predetermined. Radiologists can become fatigued or “burned out” from having to read many image studies under time pressure (e.g., within a certain time period). Al finding detectors may serve in a supporting role, detecting proposed clinical findings that are then presented to the radiologist during the radiology reading for acceptance or rejection by the radiologist.

[0007] The following discloses certain improvements to overcome these problems and others.SUMMARY

[0008] In some embodiments disclosed herein, a radiology reading system includes a database storing medical imaging studies comprising medical images, a client device comprising at least one display and at least one user input device, and at least one electronic processor programmed to perform a clinical finding recommender method including: applying artificial intelligence (Al) clinical finding detectors to at least one image of the retrieved medical imaging study to generate clinical finding probabilities corresponding to a plurality of Al clinical findings; adjusting the clinical finding probabilities based on statistical correlations between the Al clinical findings of the plurality of Al clinical findings to generate adjusted clinical finding probabilities corresponding to the plurality of Al clinical findings; selecting one or more proposed clinical findings for which the corresponding adjusted clinical finding probabilities meet a selection criterion; and presenting the selected one or more proposed clinical findings on the at least one display of the radiology workstation.

[0009] In some embodiments disclosed herein, a non-transitory computer readable medium, stores a database with a plurality of medical images, and instructions executable by at least one electronic processor to: receive at least one medical image of a patient undergoing a medical imaging study; generate a plurality of finding probabilities indicative of Al generated findings inthe received at least one medical image; adjust the generated finding probabilities using a correlation between the Al generated findings; and generate a clinical report including a plurality of proposed findings from the adjusted proposed finding probabilities.

[0010] In some embodiments disclosed herein, a clinical finding recommender method includes: receiving at least one medical image of a patient undergoing a medical imaging study; generating a plurality of finding probabilities indicative of corresponding Al -generated findings in the received at least one medical image; adjusting the generated finding probabilities using a correlation between the Al generated findings; and generating a clinical report including a plurality of proposed findings selected from the Al generated findings using the adjusted proposed finding probabilities.

[0011] One advantage resides in reducing confusion during a radiology reading session.

[0012] Another advantage resides in reducing extraneous work for a radiologist (or other qualified medical professional, such as a cardiologist with radiology expertise reviewing cardiac images) during a radiology reading session.

[0013] Another advantage resides in increasing a confidence in Al-generated proposed clinical findings.

[0014] Another advantage resides in providing an improved graphical user interface (GUI) with correlated clinical findings for display during a radiology reading session.

[0015] Another advantage resides in providing an improved commercial technology of a radiology workstation operating in conjunction with a Picture Archiving and Communication System (PACS) or other medical imaging studies storage and analysis database.

[0016] A given embodiment may provide none, one, two, more, or all of the foregoing advantages, and / or may provide other advantages as will become apparent to one of ordinary skill in the art upon reading and understanding the present disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The disclosure may take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the disclosure.

[0018] FIGURE 1 diagrammatically illustrates a radiology reading system in accordance with the present disclosure.

[0019] FIGURE 2 diagrammatically illustrates a radiology report entry method and a clinical finding recommender method using the radiology reading system of FIGURE 1.

[0020] FIGURE 3 diagrammatically illustrates a system for recommending proposed clinical findings adjusted to account for finding cross-correlations.DETAILED DESCRIPTION

[0021] In presenting Al -proposed findings, a problem can arise in that there may be multiple proposed findings which are correlated. For example, if one proposed clinical finding is “bone fracture of left tibia” and another proposed clinical finding is “compound fracture of left tibia,” the compound fracture proposed finding is highly correlated (e.g., at or near 100%) with the bone fracture finding (the reverse will also have some correlation, albeit less than 100%). In this case, if the radiologist rejects the proposed “bone fracture of left tibia” finding and is then presented by the Al with the proposed “compound bone fracture of left tibia” finding, this will waste the radiologist’s time (as its obvious that based on the high correlation, the compound fracture does not make sense) and may cause the radiologist to have less confidence in the Al-proposed findings in general.

[0022] The following discloses a radiology reading system with an interactive correlationbased on-the-fly adjustment of a finding probability for a radiology report. The approach can operate with any set of Al clinical finding detectors that output finding probabilities for AI- generated findings. A correlation-based finding probability adjustment is added which adjusts the finding probabilities output by the Al finding detectors based on a priori-estimated correlations between the various findings (i.e., cross-correlations between the Al-generated findings). The adjusted probabilities are used to decide which Al-generated finding(s) to present to the user (e.g., if a certain probability of a finding meets a particular threshold, it may be proposed to the radiologist).

[0023] Furthermore, when the radiologist accepts an Al-proposed finding, then its probability is set to 100% (or another suitable value indicating the finding is certainly present). Conversely, when the radiologist rejects an Al-proposed finding, then its probability is set to 0% (or another suitable value indicating the finding is certainly not present). This new radiologistset value is then fed back to the correlation-based finding probability adjustment, which adjusts the probabilities of any correlated findings based on the new radiologist-set value of the radiologist-reviewed finding, and thresholding is again applied to the updated adjusted findingprobabilities to update the Al-proposed findings that are presented to the radiologist. This provides the interactive aspect of the disclosed interactive correlation-based on-the-fly adjustment of the proposed findings presented to the radiologist.

[0024] As an example, suppose one Al algorithm indicated “bone fracture of left tibia” with a probability that is high enough to present this Al-generated finding the radiologist (i.e., that it meets a particular threshold probability). Another Al algorithm indicated “compound bone fracture of left tibia” again with a probability high enough (or above a certain threshold) to present this Al-proposed finding the radiologist as well. These two findings have substantial correlation, so the correlation adjustment will not reduce the probabilities significantly. In this case, both proposed findings are presented to the radiologist (even after the correlation-based probability adjustment). If the pathologist then rejects the proposed “bone fracture of left tibia” finding, its probability is set to zero and so the correlation-based finding probability adjustment lowers the probability of the highly correlated “compound bone fracture of left tibia” to zero or near zero, resulting in the “compound bone fracture of left tibia” finding no longer being presented to the radiologist. This avoids the problem of the radiologist being confronted with a proposed finding (“compound bone fracture of left tibia”) that is inconsistent with the radiologist’s decision to reject the proposed finding of “bone fracture of left tibia.”

[0025] The correlation-based adjustment typically receives as input finding cross-correlation coefficients, for example in the form of a findings correlation matrix. The finding crosscorrelation coefficients can be determined statistically, e.g., by analysis of historical radiology reports to determine pairs of findings that frequently occur together in single radiology reports (strong positive correlation), and pairs of findings that rarely occur together in single radiology reports (strong negative correlation or anti-correlation). The finding cross-correlation coefficients can additionally or alternatively be determined manually, based on clinical expertise or first principles (e.g., a proposed compound bone fracture finding is inherently closely correlated with a proposed bone fracture finding, since the former is a sub-class of the latter.) In one variant embodiment, different correlation-based finding probability adjustments based on different cohort-specific correlation statistics may be used. To do so, different sets of finding cross-correlation coefficients are obtained for different cohorts, and when performing a radiology reading for a particular patient the finding cross-correlation coefficients for the closest-matching cohort to the patient who is the subject of the reading is used.

[0026] In the illustrative example, the correlation-based finding probability adjustment is an added processing layer that is applied after the Al algorithms output the finding probabilities. In a variant embodiment, the correlation-based finding probability adjustment can be integrated into the Al algorithms, for example as hidden interior neural network layers in the case of Al algorithms employing artificial neural networks (ANNs).

[0027] The disclosed technological improvement presents to the user (e.g., radiologist or other qualified radiological image interpreter) as an improved radiology reading workstation with a GUI providing improved and dynamically adjusted proposed findings. The disclosed interactive correlation-based on-the-fly adjustment is suitably implemented in some embodiments as a component of a PACS such as for example Philips Vue® PACS, or Philips Intellispace® PACS, Philips Cardiovascular Information System, and / or as a component of a radiology workstation used by a radiologist to perform radiology readings (and which may be connected with a PACS and / or implemented as a component of a PACS), and / or as a PACS or other radiological image archiving / communication component of a cardiovascular information system (CVIS) used by a radiologist or by a radiologically qualified cardiologist or other suitably qualified medical professional, et cetera.

[0028] FIGURE 1 illustratively shows a diagnostic system 10 as including a server 14 and a client 18, However, diagnostic system 10 may be alternatively embodied as a standalone computer, a mobile device (e.g., a tablet computer), or a built-in user interface that is built into a medical device or medical system (e.g., a patient monitor having a built-in display and built-in keypad, buttons, touch-sensitive display overlay, various combinations thereof, and / or the like). The diagnostic system 10 may also be a specialized technology such as a radiology workstation or a PACS. For illustrative purposes in Fig. 1, diagnostic system 10 includes typical components for enabling user interfacing, such as an electronic processor 20 (e.g., a microprocessor), at least one user input device (e.g., a mouse, a keyboard, a trackball, and / or the like) 22, and a display device 24 (e.g., an LCD display, plasma display, cathode ray tube display, and / or so forth). While one illustrative client 18 is shown in FIGURE 1, it will be appreciated that there may be a plurality of clients connected with the server 14. In some embodiments, such as a typical radiology workstation, the display 24 may include two or more display devices or screens, e.g. a high resolution display or screen to display clinical images and a (optionally lower resolution) display or screen for displaying a radiology report as it is being drafted by the user (e.g., aradiologist, qualified cardiologist, qualified oncologist, or other qualified radiological image interpreter). To display color-coded image(s) and / or data, the display device 24 should typically be (or include at least one display which is) a color display device, although a monochrome display is also contemplated.

[0029] The illustrative diagnostic system 10 further includes a non-transitory storage media 26 which may, by way of non-limiting illustrative example, include one or more of a magnetic disk, RAID, or other magnetic storage medium; a solid-state drive, flash drive, electronically erasable read-only memory (EEROM) or other electronic memory; an optical disk or other optical storage; various combinations thereof; or so forth; and may include for example a network storage, an internal hard drive of the client 18, various combinations thereof, or so forth. It is to be understood that any reference to a non-transitory medium or media 26 herein is to be broadly construed as encompassing a single medium or multiple media of the same or different types. Likewise, the electronic processor 20 may be embodied as a single electronic processor or as two or more electronic processors; moreover, the electronic processor 20 may include an electronic processor of the client 18, an electronic processor of the server 14, a combination thereof, or an otherwise-located electronic processor. The non-transitory storage media 26 stores instructions executable by the at least one electronic processor 20. The instructions include instructions to generate a visualization of a user interface (UI) 28, which is typically (although not necessarily) a graphical user interface (GUI) 28, for display on the display device 24. The non-transitory storage medium or media 26 suitably also stores medical images 30 and radiology reports 32.

[0030] In one exemplary embodiment, the illustrative diagnostic system 10 is implemented for example, as a PACS, in which the diagnostic system 10 is implemented as a client-server IT architecture including the server 14 and one or more clients 18 connected with the server 14 (one client 18 being illustrated as a representative example). In such a PACS embodiment, all functions (such as those described later in this application, e.g., report entry method 100, clinical finding recommender 200, pre-processing functions, image processing functions, diagnostic workflow, and so forth) are carried out by execution of instructions on the PACS and / or by the PACS with at least an electronic processor 20. The server 14 may be a server computer, a server cluster, a cloud-based computing resource, or the like. Each client 18 is suitably a computer or workstation or the like that includes an instance of the illustrative user input device(s) 22 and display(s) 24 (often two or even three or more displays in the case of a radiology workstation).The clients 18 may variously include (by way of nonlimiting illustrative example) radiology workstations, physician’s computers, departmental computers, and so forth. As previously discussed, the server 14 provides remote medical image storage to facilitate sharing images amongst a fleet of clients 18, image retrieval to display at a client 18, report entry at a client 18, various image analysis tools (e.g., anatomy labeling, automatic contouring, clinical metric measurements, et cetera), and various semi-automated or automated clinical finding detectors. The image analysis tools and automated clinical finding tools may be implemented locally at the client 18, at the server 14, or distributed between the server 14 and client 18. As used herein, an embodiment of the diagnostic system 10 as a PACS is to be understood to encompass PACS functionality or variants thereof, such as a DICOM imaging system, a department-specific imaging IT infrastructure such as a cardiovascular information system (CVIS) including PACS- like functionality, an integrated imaging IT infrastructure that combines PACS functionality with a radiology scheduling and / or administrative system, or the like.

[0031] While implementation of the diagnostic system 10 as a client-server IT infrastructure is illustrated, it is alternatively contemplated for the diagnostic system 10 to be implemented as a standalone computer. For example, in one embodiment, diagnostic system 10 may be implemented as a standalone PACS in which all functions (such as those described later in this application, e.g., report entry method 100, clinical finding recommender 200, pre-processing functions, image processing functions, diagnostic workflow, and so forth) are carried out by execution of instructions on the PACS and / or by the PACS with at least an electronic processor 20. In such an alternative implementation, the server 14 is suitably omitted, and the functionality of the server is integrated with the client computer 18.

[0032] The diagnostic system 10 further includes the non-transitory storage medium 26 that is diagrammatically shown in FIGURE 1 as storing medical images 30 and radiology reports 32. The non-transitory storage media 26 also stores instructions executable by the at least one electronic processor 20 of the client 18 and / or by an electronic processor (or processors) of the server 14 to perform an illustrative report entry method 100 via which a user enters a radiology report with respect to an imaging study, and an illustrative clinical finding recommender method 200 which supports the user in preparing the report via the report entry method 100 by generating proposed findings using illustrative artificial intelligence (Al) clinical finding detectors 38. As disclosed herein, the clinical finding recommender method 200 takes into account statisticalcorrelations between the Al clinical findings generated by the Al clinical finding detectors 38 to adjust the Al-proposed clinical findings presented to the user of the report entry method 100, thereby providing benefits such as improving efficiency of the client 18 and the report entry method 100. Report entry method 100 and clinical finding recommender method 200 could be part of a PACS workflow or a workflow in some other types of systems which are envisioned as examples of a diagnostic system 10.

[0033] Al clinical finding detector 38 may output a probability that the finding is present, and the probability is adjusted based on correlations with other Al -proposed clinical findings and compared with a threshold to decide whether to present the proposed finding to the radiologist. The Al -proposed clinical findings to be presented are displayed, and may be accepted or rejected by the user performing the report entry method 100. The display of the Al-proposed clinical findings is integrated with the GUI of the report entry method 100. In one suitable approach, Al- proposed clinical findings can be displayed in a corresponding window of the graphical user interface (GUI) of the report entry method 100, and the radiologist can accept a finding at which point it is removed from the window and transferred into the radiology report template; or reject the finding at which point it is removed from the window and is not transferred into the report template. In another suitable approach, Al-proposed clinical findings are automatically populated into the report template presented via the GUI of the report entry method 100, after which the radiologist reviews each proposed finding and edits the report to accept the finding by keeping it in the radiology report, or reject the finding (e.g., edit the report to remove it), or modify the finding as determined appropriate by the radiologist. In one approach, to ensure the radiologist reviews each such auto-populated proposed finding, it could be highlighted (e.g., in red or another distinctive color, boldfacing, or so forth) and acceptance of the proposed finding then removes the highlighting. The radiology reporting GUI can also be configured to not allow final archiving or saving of the completed radiology report unless the radiologist has taken affirmative action on every auto-populated Al-proposed clinical finding.

[0034] The illustrative functionality 100, 200 is a nonlimiting example, and other functionality not described herein may also be implemented (e.g., image receipt / storage, image analysis tools, et cetera). Additional functionalities may also be PACS functionalities.

[0035] It will be further appreciated that in the illustrative client-server architecture, the storage media 26 may include multiple storage media, e.g., a remote or cloud-based storage ofor associated with the server 14 and local storage medium or media of the client 18. Likewise, the execution of the instructions to implement the various functionality (e.g., the report entry method 100 and clinical finding recommender method 200) may be variously divided amongst the computing resources. For example, the server 14 may store the images 30 and radiology reports 32, while the report entry method 100 and the clinical finding recommender method 200 may be implemented as computer-readable instructions read from the storage medium 26 and executed at the client 18. The Al clinical finding detectors 38 may also be implemented at the client 18; or, if an Al clinical finding detector requires computationally complex computations, it may be implemented at the server 14 which typically has higher computational capacity. The specific division of processing could in some implementations be situationally dependent, e.g., may be implemented through some combination of locally and remotely performed steps depending on various infrastructure technology factors (e.g., on-premise or standalone system without server, slow connection between server and client such as in the case of mobile teleradiology, preference to processing in the cloud or on a server if there is a thin client with minimal processing capabilities, preference to processing by a specialized remotely accessible Al system operated by a third party, or there is otherwise a preference to distribute some of the processing load to a remote server or cloud processing, et cetera). For example, it may be advantageous to automate identification of potential clinical findings through Al on the server 14 prior to the user receiving / loading an image study so that the identified Al clinical findings can be transmitted and presented to the user concurrently with the user loading the image study at the client 18. However, later updating of clinical findings may be performed on the client 18. These are merely some nonlimiting illustrative implementational examples.

[0036] With reference to FIGURE 2, and with continuing reference to FIGURE 1, an illustrative embodiment of an instance of the report entry method 100 and a clinical finding recommender method 200 are diagrammatically shown as a flowchart. In some embodiments, the report entry method 100 and the clinical finding recommender method 200 (or parts thereof) may run concurrently, so that proposed findings from the recommender method 200 are updated in real-time as the user accepts or rejects proposed findings (and thereby modifies the correlation adjustments for other findings that may be correlated with the accepted or rejected proposed findings). The concurrent operation of the methods 100 and 200 thus provides interactivecorrelation-based on-the-fly adjustment of Al-proposed finding probabilities provided by the diagnostic system 10.

[0037] The report entry method 100 operates on medical images 30 retrieved from the non- transitory storage medium 26, and the medical images 30 are optionally preprocessed before or after the retrieval. The optional image preprocessing may, for example, employ image analysis tools that perform anatomy labeling, automatic contouring, automatic measurement of clinically significant metrics, automatic assessment of cardiovascular metrics such as artery wall thickening (in the case of cardiology applications), and / or so forth. The type(s) of optional preprocessing may depend on factors such as the imaged anatomy and the clinical question being addressed, e.g., if the imaged anatomy includes a tumor which is the subject of the clinical question then preprocessing related to the tumor may be performed, whereas if the imaged anatomy is the heart and coronary blood vessels then preprocessing related to artery wall thickening may be performed. The optional preprocessing may be a feature of in a PACS system or other types of diagnostic systems 10 as envisioned or exemplified in this application, and may be performed by the server 14 or at the client 18 or some combination thereof. By way of a few nonlimiting illustrative examples, the medical imaging study being reported on via the reporting method 100 may be a computed tomography (CT) study (whole body or an arm, torso, or other anatomical region, depending on the clinical purpose) and the medical image(s) 30 are CT images, a magnetic resonance imaging (MRI) study (e.g., of the brain or another organ or anatomical region) and the medical image(s) 30 are MRI images, a positron emission tomography (PET) study for angiography or other clinical purpose and the medical image(s) 30 are PET images, an ultrasound study and the images are ultrasound images, a multi-modality study (e.g., PET / CT including both PET and CT images), or so forth. The medical image(s) 30 may be single images, or may be cine sequences of images, e.g. to analyze inflow and washout of an intravascularly administered contrast agent in an angiography scan. The medical image(s) 30 may optionally include metadata annotated to the images, for example identifying imaging settings used in acquiring the medical image(s) 30.

[0038] At an operation 104 of the report entry method 100, which is a precursor to reporting, one or more medical images 30 of the medical imaging study is displayed on the display device 24 to be read by the user as part of a diagnostic workflow. The operation 104 may include diagnostic workflow in advance of reporting such as providing manually performed manipulationof the displayed image(s) such as zoom, pan, paging between images, and / or so forth, measurements of findings, annotation, and so forth. The operation 104 may also provide the ability for the user to manually apply image processing (in addition to any image processing automatically performed in the preprocessing) such as select and apply image filters and so forth.

[0039] At an operation 106 (which may be performed concurrently with the image display 104), a radiology report 32 pertaining to the retrieved medical imaging study (or a report template that the user fills out to create radiology report 32) is populated at the client device 18 via the at least one user input device 22. As the radiology report 32 is populated, it is displayed in an operation 108. Some fields of the report template may also be auto-populated, for example by preprocessing tools, and the operation 106 then entails the user reviewing the auto-populated information and accepting, rejecting, or modifying the auto-populated information. Process flow of the report entry method 100 iterates between operation 106 facilitating population of the radiology report 32 and operation 108 displaying the radiology report on the at least one display 24 of the client device 18 as the radiology report is received. Thus, the user sees the in-progress radiology report in real-time as it is populated. In some embodiments, the operation 106 may utilize a radiology report template that includes fillable fields for certain information commonly included in a radiology report, such as patient information, study type, reason for exam, and so forth. In such cases, the operation 106 includes (at least in part) receiving inputs for filling in (i.e., populating) the fillable fields via the at least one user input device 22.

[0040] Recommender method 200 operates in conjunction with and supports the report entry method 100 to present Al-proposed clinical findings in the GUI of the report entry method 100. The recommender method 200 includes operations 202, 204, 206, 208, and 210. The result or output of clinical finding recommender method 200 is selected clinical findings generated by the Al clinical finding detectors 38, which are proposed to the user for inclusion in the radiology report 32 being prepared via the report entry method 100. The recommender method 200 takes into account correlations between Al-proposed clinical findings, and makes adjustments “on the fly”, that is, as the user accepts or rejects various Al-proposed clinical findings, to provide optimized presentation of remaining (not-yet-accepted or rejected) proposed clinical findings to the user over the course of the report entry process. At an operation 202 of the clinical finding recommender method 200, a plurality of finding probabilities 34 for Al clinical finding 36 in the at least one medical image 30 are generated. The operation 202 suitably operates on the imagesthat are the subject of the report being prepared in the report entry method 100 (or a subset of those images). In the illustrative embodiment, in the operation 202 Al clinical finding detectors 38 are applied to at least one medical image 30 of the medical imaging study being read by the user to generate the Al clinical findings (that is, findings generated by the Al clinical finding detectors 38), along with clinical finding probabilities 34 corresponding to the respective Al clinical findings 36. In one embodiment, the Al clinical finding detectors 38 comprise either (i) an artificial neural network (ANN) having a plurality of outputs corresponding to the plurality of Al clinical findings 36; or (ii) a plurality of ANNs each having an output corresponding to one Al clinical finding of the plurality of Al clinical findings 36.

[0041] Examples of suitable Al algorithms used by Al clinical finding detectors 38 include, but are not limited to, different types of neural networks, deep neural networks or “deep learning” models, or for example, sequence to sequence models. Throughout this description, the term “neural network” is used to describe a plurality of processing nodes that are densely interconnected. Sometimes, the neural network is organized into layers of nodes, but it is not a requirement. In a layer model, for example, a node may be connected to one or more nodes in a lower layer, from which it receives data, and one or more nodes in a higher layer, to which it sends data. It should also be understood that a neural network is a subset of machine learning, which is a method of data analysis that automates analytical model building. Thus, throughout this description, it should be understood that use of a neural network is only exemplary, and any functions or operations described as being performed by a neural network may be performed by any type of machine learning and should not be interpreted as being limited to only a neural network. Some examples of neural networks include feed-forward neural network, Radial Basis Function (RBF) Neural Network, Multilayer Perceptron, Convolutional Neural Network (i.e. densely connected neural networks, residual neural networks, networks resulting from architecture search algorithms, capsule networks, et cetera.), Recurrent Neural Network (RNN), Modular Neural Network, and any similar types of algorithms which are known in the art or are suitable for the purpose.

[0042] The operation 202 could be performed as part of the image preprocessing before the images 30 are retrieved to the client 18 for viewing by the user at operation 104, and / or concurrently with the operations 104, 106, 108 performed at the client 18. The operation 202 typically outputs each Al clinical finding 36 with an associated probability 34 that is computedby the Al clinical finding detector for that Al clinical finding 36, without regard to probabilities of any of the other Al clinical findings. However, the different Al clinical findings 36 may have various cross-correlations, as previously discussed. The probabilities 34 determined in the operation 202 do not take these correlations into account.

[0043] At an operation 204, the generated finding probabilities 34 of the Al clinical findings 36 are adjusted using correlations between pairs of Al findings. The correlations may be represented as pairwise correlation coefficients. For example, two Al clinical findings X and Y may have respective probabilities Px and PY provided by the operation 202. There may be a correlation CXY indicating the correlation of Al finding Y with Al finding X, and a correlation CYX indicating the correlation of Al finding X with Al finding Y. CXY and CYX may have the same value, but in some embodiments may have different values. For example, if Al finding X is “bone fracture” and Al finding “Y” is “compound bone fracture,” then the correlation CYX=1, meaning that if Al finding Y: “Compound bone fracture” is present then the Al finding X: “Bone fracture” is definitely also present (since Al finding Y is a subset of Al finding X). But the converse is not true, so CXY<1. This is because even if Al finding X: “Bone fracture” is present, this does not necessarily mean that Al finding Y: “Compound bone fracture” is also present, since it could be a simple bone fracture rather than a compound bone fracture. As previously discussed, the correlation coefficients can be determined, for example, based on statistical correlations observed in historical radiology reports, and / or based on first principles or so forth. The operation 204 adjusts the finding probabilities of the Al findings 36 using the finding correlation coefficients to reflect statistical correlations between the Al clinical findings of the plurality of Al clinical findings 36 to generate adjusted clinical finding probabilities corresponding to the plurality of Al clinical findings 36. For example, the adjusting of the clinical finding probabilities 34 uses correlation coefficients 40 that represent the statistical correlations between the Al clinical findings 36. In another example, cohort-specific correlation coefficients 40 can be selected for use in the adjusting based on information about the patient who is the subject of the retrieved medical imaging study. In some embodiments, the adjusting operation 204 can be performed using the ANN, advantageously allowing the clinical finding probabilities 34 to be updated in real-time.

[0044] At an operation 206, one or more Al clinical findings 36 for which the corresponding adjusted clinical finding probabilities 34 meet a selection criterion are selected as proposedclinical findings (for example, by thresholding the adjusted probabilities and selecting any Al clinical finding whose adjusted probability equals or exceeds the threshold). At an operation 208, the one or more selected proposed clinical findings are proposed to the user, for example by being displayed on the display device 24. In some examples, a report template for the radiology report 32 can also be displayed, in which the one or more proposed clinical findings can be added to the displayed radiology report 32.

[0045] At an operation 210, a user review of the selected one or more proposed clinical findings is received via the at least one user input device 22. The user review comprises an input indicative of an acceptance or a rejection of a reviewed proposed clinical finding, which is received via the at least one user input device 22 from a user.

[0046] Responsive to the received input, the clinical finding probability 34 of the reviewed proposed clinical finding 36 is updated based on whether the input is indicative of an acceptance or a rejection for each proposed finding. In some embodiments, the update of the proposed clinical finding probability 34 includes fixing the clinical finding probability 34 to a fixed value indicated by the user review. For example, when the received input is indicative of an acceptance of the proposed finding, setting the finding probability 34 for that proposed finding to a first value (i.e., one), and when the received input is indicative of a rejection of the proposed finding, setting the finding probability 34 for that proposed finding to a second different value (i.e., zero). The operations 206-210 can then be repeated (as indicated by a feedback arrow connecting 210 to 204) to update the probabilities 34 of the one or more Al clinical findings 36 to generate updated proposed clinical findings 42. The repetition of the adjusting operation 206 uses the fixed value of the clinical finding probability 34 of the reviewed proposed clinical finding. The repetition can be performed each time an input is received at operation 210 via which the user accepts or rejects a proposed finding of the currently selected one or more proposed findings. Notably, due to the adjustment of the probabilities in each subsequent pass of the operation 204, each repetition of the operations 206-210 may select different proposed findings 36 for display in the operation 208. This provides on-the-fly adjustment of which of the Al clinical findings 36 are presented to the user.

[0047] With reference to FIGURE 3, the operations 202 and 204 of the clinical finding recommender method 200 are described in terms of components previously described with reference to FIGURE 1. As previously described, one or more clinical images 30 is input to theAl clinical findings detector 38 to generate clinical finding probabilities 34 corresponding to Al clinical findings 36. Each Al finding has a corresponding probability, e.g., an Al finding #1 has probability Pl, an Al finding #2 has probability P2, an Al finding #3 has probability P3, and so forth. In an operation 300, the findings (cross-)correlation coefficients 40 are applied to adjust the finding probabilities 34 to produce the adjusted proposed findings 42, e.g. as shown in FIGURE 3 Al finding #1 has adjusted probability Pl ’, Al finding #2 has adjusted probability P2’, Al finding #3 has adjusted probability P3’, and so forth. The operation 300 may operate by simple multiplication in some nonlimiting illustrative embodiments. For example, consider a pair of Al findings X and Y, with respective (unadjusted) probabilities Px and PY and a correlation CXY indicating the correlation of Al finding Y with Al finding X, and a correlation CYX indicating the correlation of Al finding X with Al finding Y. Then the adjusted probability Px’= PCXXXPX+PCYXXPY and the adjusted probability PY’= PCYYXPY+PCXYXPX. More complex correlation adjustments are also contemplated.

[0048] The ANN, in some embodiments, can comprise a convolutional neural network (CNN) that can be trained using a training dataset of annotated images for which findings are available. An example loss function for training can be the mean squared error of the resulting probabilities for each finding Fl, F2, ... FN with respect to the (binary) ground truth information (i.e., if a finding is present, then annotate the probability as 100%, and if a finding is not present, then annotate the probability as 0%).

[0049] The correlation coefficients 40 can be obtained, for example, from empirical correlations in historical radiology reports. A correlation coefficient of 100% for an example pair of Al findings FX and FY indicates that their presence is fully linked, i.e. the presence of FX in a radiology report always implies the presence of FY in that same radiology report. A correlation coefficient of 0% on the other hand indicates that the presence of FX is completely independent of the presence of FY. In some embodiments, the correlation coefficients may include negative or anti-correlations, e.g. if FX and FY are 100% negatively correlated, i.e., anti-correlated, then the presence of FX in a radiology report always implies the FY is not present in that same radiology report. The correlation coefficients 40 might be computed for a given cohort selected by a cohort selector 302. This recognizes that different cohorts may have different cross-correlations. The updated (i.e., correlation-corrected) probabilities 42 can also be used to further train the CNN (or other type of ANN).

[0050] After the training phase of the CNN, for a given new image 32 proposed findings are proposed for acceptance / correction by the user based on the correlation-corrected probabilities 42 and a given threshold. For example, if a first corrected probability Fl’ exceeds 50%, the corresponding first Al finding 36 is proposed to be present. Once the user has accepted the proposed finding, the value Fl will be replaced by 100%. If the user has rejected the proposed finding, it will be replaced with 0%, and the correlation-corrected probabilities 42 are updated for the Al findings that still need to be reviewed by the user. This way, information about accepted or rejected proposed findings will enhance the accuracy of residual information.

[0051] The CNN can include features, such as a number of layers, patch size, et cetera, that can be subject to the specific set of findings. As an alternative to the multi-task network structure other designs for the ANN, like a set of individual, unconnected classifiers might be used. Furthermore, pre-processing steps such as the segmentation of corresponding anatomical structures might be part of the ANN and alternative topologies such as individual networks per each finding might be used.

[0052] In some embodiments, the correlation coefficients 40 might be specified in a more complex way, such as computing correlation coefficients for different probability ranges of a specific finding. This way, non-linear correlation effects can be modelled.

[0053] In some embodiments, only the CNN (or other ANN) is employed. Here, after accepting / rej ection of a particular finding Fl, F2, ... FN, the CNN is partially retrained based on partially available ground truth information, i.e. backpropagation is applied for the corresponding findings. The correlations 40 between individual findings 42 are implicitly modelled by the CNN. While ANN and CNN has been described in some examples or embodiments herein, other forms of suitable Al algorithms can be used by Al clinical finding detectors 38 as previously explained (e.g., any form of suitable deep learning, machine learning, feed-forward neural network, Radial Basis Function (RBF) Neural Network, Multilayer Perceptron, Convolutional Neural Network (i.e. densely connected neural networks, residual neural networks, networks resulting from architecture search algorithms, capsule networks, et cetera), Recurrent Neural Network (RNN), Modular Neural Network, and any similar types of algorithms which are known in the art or are suitable for the purpose).

[0054] The disclosure has been described with reference to the preferred embodiments. Modifications and alterations may occur to others upon reading and understanding the precedingdetailed description. It is intended that the exemplary embodiment be construed as including all such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

CLAIMS:

1. A radiology reading system (10), comprising: a database storing medical imaging studies comprising medical images (30); a client device (18) comprising at least one display (24) and at least one user input device (22); and at least one electronic processor (20) programmed to perform a clinical finding recommender method (200) including: applying artificial intelligence (Al) clinical finding detectors (38) to at least one image of the retrieved medical imaging study to generate clinical finding probabilities (34) corresponding to a plurality of Al clinical findings (36); adjusting the clinical finding probabilities based on statistical correlations (40) between the Al clinical findings of the plurality of Al clinical findings to generate adjusted clinical finding probabilities corresponding to the plurality of Al clinical findings; selecting one or more proposed clinical findings for which the corresponding adjusted clinical finding probabilities meet a selection criterion; and presenting the selected one or more proposed clinical findings on the at least one display of the radiology workstation.

2. The radiology reading system (10) of claim 1, wherein the at least one electronic processor (20) is programmed to perform a report entry method (100) including: retrieving a medical imaging study from the database; displaying one or more images of the retrieved medical imaging study on the at least one display of the radiology workstation; and receiving entry of a radiology report on the retrieved medical imaging study via the at least one user input device of the radiology workstation and displaying the radiology report on the at least one display of the radiology workstation as entry of the radiology report is received; wherein the clinical finding recommender method (200) is performed during performance of the radiology report entry method.

3. The radiology reading system (10) of either one of claims 1 and 2, wherein the clinical finding recommender method (200) further includes: receiving, via the at least one user input device (22), a user review of a reviewed proposed clinical finding; fixing the clinical finding probability (34) of the reviewed proposed clinical finding to a fixed value indicated by the user review; and repeat the adjusting, selecting, and presenting to update the selected one or more proposed clinical findings, wherein the repetition of the adjusting uses the fixed value of the clinical finding probability of the reviewed proposed clinical finding.

4. The radiology reading system (10) of claim 3, wherein: the receiving of the user review includes receiving an acceptance or a rejection of the reviewed proposed clinical finding (36); and the fixed value of the clinical finding probability of the reviewed proposed clinical finding is zero if a user rejection is received or one if a user acceptance is received.

5. The radiology reading system (10) of any one of claims 1-4, wherein the Al clinical finding detectors (38) comprise one of: an artificial neural network having a plurality of outputs corresponding to the plurality of Al clinical findings (36); or a plurality of artificial neural networks each having an output corresponding to one Al clinical finding of the plurality of Al clinical findings.

6. The radiology reading system (10) of any one of claims 1-5, wherein the adjusting of the clinical finding probabilities (34) uses correlation coefficients (40) that represent the statistical correlations between the Al clinical findings (36).

7. The system (10) of claim 6, wherein the clinical finding recommender method (200) further includes: selecting cohort-specific correlation coefficients for use in the adjusting based oninformation about the patient who is the subject of the retrieved medical imaging study.

8. A non-transitory computer readable medium (14), storing a database with a plurality of medical images (32); and instructions executable by at least one electronic processor (20) to: receive at least one medical image (30) of a patient undergoing a medical imaging study; generate a plurality of finding probabilities (34) indicative of artificial intelligence (Al) generated findings (36) in the received at least one medical image; adjust the generated finding probabilities using a correlation (40) between the Al generated findings; and generate a clinical report (32) including a plurality of proposed findings from the adjusted proposed finding probabilities.

9. The non-transitory computer readable medium (14) of claim 8, wherein the instructions executable by the at least one electronic processor (20) include: display, on a display device (24), the generated clinical report (32); receive, via at least one user input device (22), an input indicative of an acceptance or a rejection of a reviewed proposed finding; and finalize the displayed clinical report.

10. The system (10) of claim 9, wherein the instructions executable by the at least one electronic processor (20) include: responsive to the received input, update the finding probabilities (34) based on whether the input is indicative of an acceptance or a rejection of the reviewed proposed finding; and repeat the adjusting of the generated finding probabilities.

11. The non-transitory computer readable medium (14) of claim 10, wherein the instructions executable by the at least one electronic processor (20) include: when the received input is indicative of an acceptance of the reviewed proposedfinding, setting the finding probability (34) for the reviewed proposed finding to a first value; and when the received input is indicative of a rejection of the reviewed proposed finding, setting the finding probability for the reviewed proposed finding to a second different value.

12. The non-transitory computer readable medium (14) of any one of claims 8-11, wherein the instructions executable by the at least one electronic processor (20) include: adjust the generated finding probabilities (34) using different cohort-specific correlation statistics.

13. The non-transitory computer readable medium (14) of any one of claims 8-12, wherein the generated finding probabilities (34) are generated with an Al clinical finding detector (38).

14. The non-transitory computer readable medium (14) of claim 13, wherein the generated finding probabilities (34) are adjusted using the Al clinical finding detector (38).

15. The non-transitory computer readable medium (14) of either one of claims 13 and 14, wherein the Al clinical finding detector (38) comprises an artificial neural network (ANN).

16. The non-transitory computer readable medium (14) of any one of claims 8-15, wherein the generated finding probabilities (34) are adjusted in real-time.

17. The non-transitory computer readable medium (14) of any one of claims 8-16, wherein the database comprises a database of a Picture Archiving and Communication System (PACS).

18. A clinical finding recommender method (200), comprising: receiving at least one medical image (30) of a patient undergoing a medical imaging study; generating a plurality of finding probabilities (34) indicative of corresponding AI- generated findings (36) in the received at least one medical image; adjusting the generated finding probabilities using a correlation (40) between the Al generated findings; andgenerating a clinical report (32) including a plurality of proposed findings selected from the Al generated findings using the adjusted proposed finding probabilities.

19. The method (200) of claim 18, further including: displaying, on a display device (24), the generated clinical report (32); receiving, via at least one user input device (22), an input indicative of an acceptance or a rejection of a reviewed proposed finding; and finalizing the displayed clinical report.

20. The method (200) of claim 19, further including: responsive to the received input, updating the finding probabilities (34) based on whether the input is indicative of an acceptance or a rejection of the reviewed proposed finding (36); and repeating the adjusting of the generated finding probabilities.

Images