Radiologist Fingerprinting

Abstract

The apparatus (10) for evaluating radiologist performance includes, during an image reading session in which a user is logged into a user interface (UI) (27), the user submits (98) a medical imaging study (31) via the UI, receives an examination report for the submitted medical imaging study via the UI, and files the examination report; and executes a tracking method (102, 202), the tracking method (102, 202) for (i) verifying a match between clinical findings contained in the examination report and corresponding computer-generated clinical findings for the submitted medical imaging study that are generated by a computer-aided diagnosis (CAD) process that is executed as a background process during the image reading session. and (ii) calculating (204) a quantified match score 34, and / or (ii) determining (208) an interpretation time 38 of the submitted medical imaging studies, wherein the interpretation time of each submitted medical imaging study is the time interval from the start of presentation of the medical imaging study via the user interface to the submission of the corresponding examination report; and generating (104) at least one time-dependent user performance metric 36 for the user based on the calculated match score and / or the determined interpretation time.

Description

[Technical Field]

[0001] The present invention relates generally to the fields of radiology, radiological examination interpretation, imaging workflow, computer-aided diagnosis (CAD), and related technologies. [Background technology]

[0002] In the past few years, machine learning (ML) or deep learning (DL) artificial intelligence (AI) solutions have reached or exceeded human-like performance levels for a variety of tasks, such as detecting relevant findings (e.g., detecting lung nodules in computed tomography (CT) scans, breast lesions in mammograms, pneumothorax in chest X-rays, etc.). However, for several reasons, most notably regulatory issues, such solutions have not been well integrated into clinical routine. Summary of the Invention [Problem to be solved by the invention]

[0003] At the same time, radiologist performance evaluation is increasingly required in radiology departments as a way to ultimately improve the throughput and accuracy of radiology examination interpretation, which can reduce costs while maintaining or improving interpretation quality.

[0004] One of the radiologist performance metrics is the radiology report turnaround time (TAT), which is defined as the time interval between the time when the clinical images are uploaded to the radiology information system after the radiology exam by the radiologist and the time when the staff radiologist completes the radiology report. TAT affects the patient, the referring physician, and the entire hospital facility. For the purpose of best patient care, radiologists must be able to avoid TAT. Note that TAT depends, at least in part, on factors outside the radiologist's control, such as a backlog of radiology exams to be read.

[0005] Of greater importance for evaluating radiologist performance is interpretation time, which is the time interval between when a radiologist opens a radiology study to perform interpretation and when the radiologist files the final radiology report containing the radiologist's findings. Interpretation time depends on both the radiologist and the procedure type. For example, interpretation time can be affected by the complexity of the imaging study (e.g., a complex 3D CT scan to assess cardiac health may take longer to interpret than a 2D X-ray to assess a possible fracture), the complexity of the patient context (e.g., if a patient has a complex medical history and / or numerous previous imaging studies, the radiologist is expected to review this patient's medical history to understand the patient context), and / or different work efficiencies of individual radiologists at different times of day and / or on different days of the week.

[0006] Currently, radiologists typically work within a Picture Archiving and Communication System (PACS)-driven workflow. PACS workstations have several worklists that are typically organized according to exam status, location, modality, and body part. Radiologists can select the next case to be reviewed from the worklist. This "cherry-picking" case selection can lead to some radiologists tending to pick less complex cases, which can lead to a buildup of complex cases that have not yet been reviewed at the end of the day or shift. Furthermore, this ad-hoc selection is not optimized for efficiency and quality. Furthermore, critical scans should be reviewed before non-critical scans, and urgency can be a factor in case selection.

[0007] Without global knowledge of how a radiologist's reading efficiency varies over the course of a day or week, aberrant reading performance cannot be identified and, as a result, cannot be managed dynamically to avoid possible study backlogs and / or affected reading quality. Additionally, a radiologist's accuracy in accurately reading selected cases is also an efficiency factor.

[0008] The following discloses specific improvements to overcome these and other problems. [Means for solving the problem]

[0009] In one aspect, an apparatus for evaluating radiologist performance has at least one electronic processor programmed to perform the following steps during an interpretation session in which a user is logged in to a user interface (UI): submitting a medical imaging study via the UI, receiving an examination report for the submitted medical imaging study via the UI, and filing the examination report; and executing a tracking method, the tracking method including at least one of: (i) calculating a match score that quantifies the match between clinical findings included in the examination report and corresponding computer-generated clinical findings for the submitted medical imaging study, which are generated by a computer-aided diagnosis (CAD) process executed as a background process during the interpretation session; and / or (ii) determining an interpretation time for the submitted medical imaging study, wherein the interpretation time for each submitted medical imaging study is the time interval from the start of submission of the medical imaging study via the user interface to the filing of the corresponding examination report; and generating at least one time-dependent user performance metric for the user based on the calculated match score and / or the determined interpretation time.

[0010] In another aspect, an apparatus for evaluating radiologist performance has at least one electronic processor programmed to perform the following steps: during an interpretation session in which a user is logged in to the UI, performing a presentation of a medical imaging examination via the UI, including displaying medical images of the medical imaging examination, and receiving clinical findings generated by the user for the presented medical imaging examination via the UI; and performing a tracking method, the tracking method performing a CAD process on the medical images of the presented medical imaging examination as a background process executed during the interpretation session to generate computer-generated clinical findings for the presented medical imaging examination; calculating a match score that quantifies the match between the computer-generated clinical findings for the presented medical imaging examination and corresponding user-generated clinical findings for the presented medical imaging examination; and generating time-dependent user performance metrics for the user based on the match score.

[0011] In another aspect, an apparatus for evaluating radiologist performance has at least one electronic processor programmed to execute a method during an interpretation session in which a user is logged in to a UI, the method comprising the steps of: providing a worklist of uninterpreted medical imaging studies via the UI; submitting medical imaging studies selected by the user from the worklist via the UI; receiving examination reports via the UI for the submitted medical imaging studies; filing the received examination reports, wherein the interpretation time for each submitted medical imaging study is determined as the time interval from the start of submission of the medical imaging study via the UI to the filing of the corresponding received examination report; and generating time-dependent user performance metrics for the user based on the determined interpretation times.

[0012] One advantage is comparing the performance of an individual radiologist performing one or more imaging studies with an AI-enabled algorithm running the same or similar imaging studies.

[0013] Another benefit is that it runs a background program that tracks the similarity between radiologists' performance and the AI-enabled algorithm.

[0014] Another advantage is that it does not use the results of AI-enabled algorithms in patient diagnosis.

[0015] Another advantage resides in tracking radiologist performance during an imaging study to obtain a benchmark level of radiologist performance.

[0016] Another advantage resides in tracking a radiologist's accuracy performance during an imaging study to obtain a benchmark accuracy level for the radiologist's performance.

[0017] Another advantage is to have a benchmark level of radiologist performance as an internal standard.

[0018] Another advantage resides in determining the efficiency of radiologists performing medical imaging exams based on their reading time.

[0019] Another advantage resides in updating a radiologist's schedule or workflow based on the radiologist's reading time.

[0020] A given embodiment may provide none of the aforementioned advantages, or may provide one, two, more, or all of the advantages, and / or may provide other advantages as will become apparent to those skilled in the art upon reading and understanding this disclosure.

[0021] The disclosure may take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for purposes of illustrating preferred embodiments and are not to be construed as limiting the disclosure. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a diagram that schematically illustrates an exemplary apparatus for evaluating radiologist performance according to the present disclosure; [Figure 2] 2A and 2B illustrate exemplary flow chart operations performed by the device of FIG. 1. DETAILED DESCRIPTION OF THE INVENTION

[0023] As used herein, the term "background process" (and variants thereof) refers to a computer process that runs autonomously behind the scenes of another process (such as an imaging review session) without user intervention.

[0024] As used herein, the term "concordance score" (and variants thereof) refers to the relationship between the results of a radiologist's image interpretation and the results generated by an artificial intelligence background process.

[0025] As used herein, the term "fingerprint" (and variants thereof) refers to the relationship between a radiologist's personal reading characteristics and potentially subtle differences compared to other radiologists.

[0026] As used herein, the term "user performance metrics" (and variants thereof) refers to the time-stamping or fitting process of fingerprints or match scores.

[0027] AI-based systems, such as Computer Aided Diagnostic (CAD) systems, are becoming highly accurate and could in principle be used for clinical diagnostic tasks. However, such use is hindered by non-technical considerations, such as regulatory frameworks that do not permit CAD for diagnostics, or, even if permitted, the incorporation of CAD requires costly re-certification of systems and processes for regulatory approval.

[0028] The following discloses that, in some embodiments, an AI CAD program runs in the background. AI CAD results are not used to provide or assist in an actual diagnosis. Rather, the AI ​​CAD results are compared with clinical findings contained in the radiology report generated by the radiologist to generate a concordance score (sometimes referred to as a fingerprint in these embodiments) for the radiologist, which measures how well the radiologist's clinical findings agree with the clinical findings generated by the AI ​​CAD. Assuming the AI ​​CAD is reasonably accurate, a higher concordance score can be expected to correlate with greater accuracy of the radiologist's interpretation of the radiograph. This remains true as long as the AI ​​CAD is reasonably accurate. Therefore, the AI ​​CAD does not need to be perfect or sufficiently accurate for clinical diagnosis. The concordance score for the radiologist may be calculated as a function of time and may be divided in various ways, for example, into different concordance scores for different types of interpretations.

[0029] The agreement score can have a variety of uses. It can be used to track radiologist performance over the course of a day to identify periods when a radiologist's accuracy may be declining (e.g., late afternoon due to fatigue). It can be used to compare radiologist performance across radiology departments or hospitals. A shift in agreement score can also be an indicator of a problem in the radiology interpretation process. For example, a decline in agreement score across all radiologists could be due to a change in imaging protocol or equipment malfunction (which could lead to reduced AI CAD accuracy). Advantageously, these embodiments leverage AI CAD in actual clinical workflows while avoiding regulatory or other non-technical concerns that have traditionally limited or prevented the use of AI CAD in the clinical diagnosis of actual patients.

[0030] In other (not necessarily mutually exclusive) embodiments disclosed herein, different types of radiologist fingerprints are provided to assess the efficiency of radiology interpretation. In these embodiments, the fingerprint is a metric of how often a radiologist fails to meet the expected interpretation time for a study. This assessment takes advantage of the fact that most PACS implementations timestamp the start of a radiology study interpretation (when the radiologist accesses the imaging study data) and the end of the interpretation (when the radiology report is filed), with the interpretation time falling between those two points. To establish an "expected" interpretation time (e.g., based on an individual radiologist), each radiologist's interpretation time is statistically analyzed to determine a typical interpretation time threshold that the radiologist typically meets. At a higher level of granularity, the reading time threshold is preferably determined for a particular reading task (e.g., the reading time threshold for a simple CT reading to detect a possible fracture may be much shorter than the reading time threshold for a complex PET scan reading to detect a possible lesion), and may also be determined for a particular day of the week, a particular part of the day, or other particular time period (e.g., a radiologist may be less efficient on Mondays compared to Tuesdays, or more efficient in the afternoon compared to the morning, or vice versa).

[0031] After this setup, the radiologist's reading time for each review is compared to the reading time threshold for that radiologist and that type of review (and optionally, for that day of the week, etc.). If more than a certain number of reviews per time block exceed the threshold (e.g., in one example, more than two reviews in a 30-minute period exceed the reading time threshold), the readings that exceed the threshold are evaluated with respect to patient context. If there is something in the patient context that justifies a longer reading time, this reading time that exceeds the threshold is discounted. If, after this patient context analysis, the number of reading times that exceed the threshold within a time block is still too high, dynamic management of the radiologist's workload is invoked.

[0032] Dynamic management can include, for example, assigning some easier readings to a radiologist. Alternatively, if a radiologist is performing well (no reading time above the threshold over the most recent time block), that radiologist can be assigned some more difficult readings because the reader has been indicated as a preferred reader for these types of images. More generally, radiologist above-threshold fingerprints can be used to intelligently distribute unread cases to available radiologists.

[0033] In today's radiology review systems, radiologists are typically presented with a queue of pending cases, which can lead to cherry-picking of easier cases. Dynamic management can additionally or alternatively be achieved by adjusting the queue of pending cases for each individual radiologist so that the radiologist is presented only with appropriate cases based on the radiologist's current review time performance for different types of reviews.

[0034] Referring to FIG. 1 , an exemplary apparatus 10 for evaluating a radiologist's performance in reviewing images generated by an image acquisition device (not shown) is shown. FIG. 1 also shows an electronic processing device 18, such as a workstation computer, or more generally, a computer. The electronic processing device 18 typically includes a radiology review workstation and may also include a server computer or multiple server computers interconnected to form, for example, a server cluster, cloud computing resources, etc., to perform more complex image processing or other complex computational tasks. The workstation 18 includes typical components such as an electronic processor 20 (e.g., a microprocessor), at least one user input device (e.g., a mouse, keyboard, trackball, etc.) 22, and a display device 24 (e.g., an LCD display, a plasma display, a cathode ray tube display, etc.). In some embodiments, the display device 24 can be a separate component from the workstation 18 or can include two or more display devices (e.g., a high-resolution display for presenting clinical images of a radiology study and a low-resolution display for providing text or low-resolution graphical content).

[0035] The electronic processor 20 is operatively connected to one or more non-transitory storage media 26. The non-transitory storage media 26 may include, by way of non-limiting example, one or more of a magnetic disk, RAID, or other magnetic storage medium, a solid-state drive, a flash drive, an electronically erasable read-only memory (EEROM), or other electronic memory, an optical disk or other optical storage device, various combinations thereof, and the like, such as a network storage device, an internal hard drive of the workstation 18, various combinations thereof, and the like. It should be understood that any reference herein to one or more non-transitory media 26 should be broadly interpreted to encompass a single medium or multiple media of the same or different types. Similarly, the electronic processor 20 may be embodied as a single electronic processor or as two or more electronic processors. The non-transitory storage media 26 store instructions executable by at least one electronic processor 20. The instructions include instructions for generating a visualization of a graphical user interface (GUI) 27 for display on the display device 24.

[0036] The apparatus 10 further includes, or is otherwise in operative communication with, a database 28 that stores a set 30 of images and / or medical imaging studies 31 to be reviewed. The database 28 may be any suitable database, including a radiology information system (RIS) database, a picture archiving and communication system (PACS) database, an electronic medical record (EMR) database, or the like. In particular, the database 28 typically comprises a PACS database or its functional equivalent. Alternatively, the database 28 may be implemented in one or more non-transitory media 26. The workstation 18 may be used to access the stored set 30 of images of the radiology studies 31 to be reviewed, along with imaging metadata stored, for example, in DICOM format.

[0037] The images 30 can be downloaded from the database 28 to the workstation 18 so that a radiologist can review the images and report findings (e.g., presence of a lesion, errors in the images, regions of interest in the images, etc.). In some embodiments, the at least one electronic processor 20 is further programmed to implement an AI component 32. The AI ​​component 32 is programmed to run one or more algorithms (e.g., CAD algorithms) on the image set 30 as the radiologist reviews the images to generate computer-generated clinical findings for the submitted medical imaging study 31. However, unlike typical CAD systems, the computer-generated clinical findings are not presented to the radiologist for consideration when performing an interpretation of the radiological study 31. Rather, the at least one electronic processor 20 is programmed to calculate a fingerprint or match score 34 based on a comparison between the radiologist's performance and the AI ​​component 32. From the match score 34, a user performance metric 36 for the radiologist is calculated. In this manner, AI component 32 plays no role in the clinical radiology reading process (e.g., the computer-generated clinical findings are not known to the reading radiologist and are not included in the filed radiology report.) As a result, AI component 32, and its use as disclosed herein, typically does not require regulatory approval by medical regulatory authorities.

[0038] In other (not necessarily mutually exclusive) embodiments, radiologist fingerprints are generated based on tracking of reading times and can be used, for example, in dynamic management of radiologist workload, as further described herein.

[0039] The apparatus 10 is configured to perform the radiology interpretation method 98 and the radiologist performance evaluation method or process 100, as described above. The non-transitory storage medium 26 stores instructions readable and executable by at least one electronic processor 20 to perform the disclosed computational operations, including performing the interpretation method 98 and the radiologist performance evaluation method or process 100. In some examples, one or both of the methods 98, 100 may be performed at least in part by cloud processing.

[0040] The radiology reading method 98 provides the radiologist with tools for reading radiology studies. In a typical workflow, the radiologist logs into the workstation 18 to conduct a reading session. Logging in may be accomplished by the radiologist entering their username and password. Other login approaches may use biometric-based login, such as using a fingerprint reader (not shown) that reads the radiologist's finger print, or using facial recognition, etc. Other typical login approaches may also be utilized, such as two-factor authentication, in which the radiologist enters a password, inserts a USB security key, and provides a computer-generated one-time passcode.

[0041] During a review session, a user (e.g., a radiologist) logs into the UI 27. The user selects a medical imaging study 31 from a work list provided by the UI 27, and the selected medical imaging study is presented via the UI 27. This presentation may include, for example, displaying clinical images 30 of the study on the display device 24 and allowing the user to zoom, pan, or otherwise manipulate the display of the images. The UI 27 may allow the user to manipulate an on-screen cursor to measure distances within the image or provide other functions, such as outlining lesions or other features of interest. Additionally, the UI 27 provides a user input window for receiving an examination report for the presented medical imaging study 31 via the UI 27. The user (e.g., a radiologist) writes up the radiology report, including providing the radiologist's clinical findings. Once the report is complete, the user files the examination report, for example, by uploading the final report to the PACS database 28. The radiology reading method 98 can be implemented, for example, as a commercially available radiology reading environment such as the IntelliSpace PACS Radiology reading environment (available from Koninklijke Philips NV, Eindhoven, the Netherlands).

[0042] In a typical radiology department, a radiologist logs into a workstation 18 at the start of each work shift and conducts a review session, which may include performing the review of several radiological studies. The radiologist logs out at the end of the work shift (and may also log out / log in at other intervals, such as for lunch). The radiologist thereby conducts successive review sessions that may span days, weeks, months, or years, depending on their tenure in the radiology department. The radiologist's performance in these successive review sessions is evaluated by a radiologist performance evaluation method 100, embodiments of which are described herein.

[0043] With continued reference to Figure 1 and further reference to Figure 2, an exemplary embodiment of a radiologist performance evaluation method 100 is shown generally as a flowchart 100 in Figure 2. In operation 102, at least one electronic processor 20 is programmed to execute the tracking method 200 during successive reading sessions in which a user is logged into the GUI 27 while radiological study readings are being performed according to the reading method 98.

[0044] In one embodiment, the tracking method 200 can include operations 202-206. In operation 202 (actually performed by the interpretation method 98), the medical imaging study 31 is presented on the GUI 27, which includes displaying the medical images of the imaging session. A user then inputs clinical findings (e.g., presence of a lesion, errors in the image, regions of interest in the image, etc.) for the medical imaging study 31 through the GUI 27 via at least one user input device 22.

[0045] In process 204, which runs in the background concurrently with process 202, at least one electronic processor 20 is programmed to perform a CAD process on the medical images of the presented medical imaging study 31. In some embodiments, the AI ​​component 32 executes process 204 as an AI-CAD process. The CAD process generates computer-generated clinical findings for the medical study presented to the user in process 202. Advantageously, the computer-generated clinical findings are not presented to the user when the user is logged into the GUI 27. Thus, the computer-generated clinical findings are not used for diagnosis.

[0046] In process 206, the at least one electronic processor 20 is programmed to extract the clinical findings input by the user by process 202 and calculate one or more match scores 34. The match scores 34 quantify the match (e.g., similarity) between the computer-generated clinical findings for the submitted medical imaging study 31 and the corresponding user-generated clinical findings for the submitted medical imaging study.

[0047] User-generated clinical findings can be identified in a variety of ways. In one approach, the radiology report entered by the user in operation 202 is processed to extract the user-generated clinical findings. The method for extracting the user-generated clinical findings from the report depends on the format of the report. If the findings have been entered in one or more structured data fields of the report designated for entry of findings, the user-generated clinical findings can be extracted by simply reading the clinical findings from the data fields designated for entry of clinical findings. On the other hand, if the findings have been entered in free-form entry fields of the report, extraction may involve natural language processing (NLP) techniques, such as detecting keywords associated with the clinical findings and / or performing semantic analysis of the text. For example, if the free-form text entry reads "Lesion size increased to 1.25 mm," the terms "lesion," "size," and "increased" can be detected to extract the finding "lesion size increasing," while the additional content "1.25 mm" may allow for the extraction of the finding "lesion size = 1.25 mm." These are merely non-limiting example embodiments. Once the match score 34 is calculated, the tracking method 200 is complete.

[0048] As process 104, the at least one electronic processor 20 is programmed to generate one or more user performance metrics 36 for the user based on the match scores 34 calculated over successive reading sessions. In some embodiments, the user performance metrics 36 are time-dependent. For example, the user performance metric 36 may be a time sequence of time-stamped match scores 34. In another example, the user performance metric 36 may include a post-processing operation, such as fitting the match scores 34 to a graphical representation, such as a polynomial function, as a function of time. In other embodiments, multiple finding-type-specific time-dependent user performance metrics 36 may be generated by performing the tracking method 200 using CAD processes specific to each different finding type that run as a background process. In other embodiments, the at least one electronic processor 20 is programmed to analyze the time-dependent user performance metrics 36 over daily time intervals to identify one or more time intervals during which the time-dependent user performance metric is below a threshold. If a user performance metric falls below a threshold, certain corrective actions can be taken (e.g., adjusting a radiologist's schedule, reviewing the tracking method 200 to see if a process error exists, etc.).

[0049] In some embodiments, the tracking method 200 can be repeated for multiple different radiologists, in which case individual user-specific time-dependent user performance metrics 36 can be generated. The at least one electronic processor 20 is programmed to compare the performance of each different user by displaying on the display device 24 a comparison (e.g., numerically, graphically, etc.) of the time-dependent user performance metrics 36 specific to each different user.

[0050] 1 and 2 , in another embodiment, instead of or in addition to running a background CAD process and performing operations 204, 206, the tracking method 200 can include determining the radiologist's reading time 38 of the medical imaging study 31. A fingerprint or user performance metric 36 can be generated for the radiologist based on reading times for previous readings, reading times based on procedure type, how reading times vary at different times during a work day or on different days during a week, the patient context of each patient in the medical imaging study, etc. As used herein, the term “patient context” (and variants thereof) refers to the complexity of various factors, such as different reasons for previous visits for a patient, the number of previous visits, and the number of previous scans taken for the same procedure type.

[0051] To determine the reading time 38, the tracking method 200 includes a process 208. In process 202, as previously described, medical studies are retrieved from the database 28 and presented via the GUI 27 as a working list of studies that have not yet been read. A user can select the studies for review. The reviewed study reports can be filed (e.g., stored) in the database 28. (Again, process 202 corresponds to the reading method or process 98 shown in FIG. 1.)

[0052] In operation 208, the at least one electronic processor 20 is programmed to determine a reading time 38 for each submitted medical imaging study 31 as the time interval between the start of the presentation of the medical imaging study via the GUI 27 and the filing of the corresponding received study report. The reading time 38 may be stored in the non-transitory computer-readable medium 26 and / or displayed on the display device 24.

[0053] In this embodiment, process 104 includes generating a time-dependent user performance metric 36 for a user based on the reading times 38 across successive reading sessions. In some embodiments, the user performance metric 36 is time-dependent. For example, the user performance metric 36 may be a time sequence of time-stamped match scores 34. In another example, the user performance metric 36 may include a post-processing operation, such as fitting the match scores 34 to a graphical representation, such as a polynomial function, as a function of time. In other embodiments, multiple finding-type-specific time-dependent user performance metrics 36 may be generated by performing tracking method 200 using the reading times 38 of different types of medical imaging exams 31. In some embodiments, tracking method 200 may be repeated for multiple different radiologists, in which case individual user-specific time-dependent user performance metrics 36 may be generated. At least one electronic processor 20 is programmed to compare the performance of different users by displaying a comparison (e.g., numerically, graphically, etc.) of the different user-specific time-dependent user performance metrics 36 on display device 24.

[0054] In a further embodiment, the at least one electronic processor 20 is programmed to analyze the time-dependent user performance metric 36 over daily time intervals, where such analysis is performed to identify one or more time intervals during which the time-dependent user performance metric is below or below a threshold based at least on the patient context of the images reviewed to generate the time-dependent user performance metric. For example, if the radiologist's reading time exceeds a predetermined threshold, the at least one electronic processor 20 is programmed to evaluate and automatically flag and trigger a check of the patient context. If the patient context is significantly complex, the at least one electronic processor 20 is programmed to determine that the radiologist's long reading time is due to complex patient context; if the patient context is not significantly complex, the at least one electronic processor determines that the radiologist's current reading performance is abnormal.

[0055] If a user performance metric falls below a threshold, certain corrective actions can be taken (e.g., adjusting a radiologist's schedule, reviewing the tracking method 200 to see if a process error exists, etc.). For example, after a predetermined number of instances of abnormal behavior are detected within a certain time period (e.g., two instances within 30 minutes), the at least one electronic processor 20 may be programmed to dynamically adjust the radiologist's reading schedule, such as assigning the radiologist fewer cases than usual, assigning less complex cases (e.g., chest x-rays), and / or adjusting other radiologists' reading assignments accordingly, as needed, in order to avoid slowing overall throughput.

[0056] In a particular example, for an imaging exam involving a CT scan of a patient's head without contrast, the maximum interpretation time for a particular radiologist from 8:00 AM to 10:00 AM on Mondays is 9 minutes. This maximum interpretation time is set as the detection threshold for this particular radiologist, and if the interpretation time at 9:00 AM on a particular Monday morning is 11 minutes, this performance is flagged as abnormal after confirmation that the patient's context is not significantly complex. After a predetermined number of instances of abnormal behavior are detected within a predetermined time period, the particular radiologist's schedule can be adjusted accordingly (e.g., to include fewer or less complex cases). Additionally, other radiologists' schedules can also be updated to account for changes in the particular radiologist's schedule.

[0057] In some examples, the AI ​​component 32 can be configured with a self-learning component in that the AI ​​component is configured to evaluate user performance metrics 36 for one or more radiologists based on imaging protocols, reading preferences, etc. For example, in the case of a spectral CT imaging protocol, the AI ​​component 32 is configured to update the user performance metrics 36 based on the radiologist's results (e.g., radiologist performance is more consistent with the AI-CAD process when MonoE images are reviewed as opposed to conventional CT images).

[0058] The present disclosure has been described with reference to preferred embodiments. Modifications and variations may occur to those skilled in the art upon reading and understanding the foregoing detailed description. It is intended that the exemplary embodiments be construed as including all such modifications and variations insofar as they come within the scope of the appended claims or equivalents thereof.

Claims

1. 1. An apparatus for evaluating radiologist performance, comprising: - during a reading session in which a user is logged into a user interface, submitting a medical imaging study via said user interface, receiving an examination report with clinical findings generated by said user for said submitted medical imaging study, and filing said examination report; Executing a tracking method, said tracking method comprising: (i) calculating a match score quantifying a match between clinical findings included in the examination report and corresponding computer-generated clinical findings for the submitted medical imaging examination generated by a computer-aided diagnosis process running as a background process during the image reading session, the match referring to a similarity between the computer-generated clinical findings for the submitted medical imaging examination and corresponding user-generated clinical findings for the submitted medical imaging examination, the computer-aided diagnosis comprising an artificial intelligence computer-aided diagnosis; and (ii) determining an interpretation time of the submitted medical imaging study, wherein the interpretation time of the submitted medical imaging study is the time interval from the start of submission of the medical imaging study via the user interface to the filing of the corresponding examination report; generating at least one time-dependent user performance metric for the user, the time-dependent user performance metric including an indication of the calculated match score and the determined interpretation time; 1. An apparatus having at least one electronic processor programmed to execute:

2. 2. The apparatus of claim 1, wherein the generating step includes generating a plurality of finding-type-specific time-dependent user performance metrics by executing the tracking method using computer-aided diagnostic processes specific to each different finding type that run as background processes.

3. 3. The device of claim 1, wherein the at least one electronic processor is not programmed to present the computer-generated clinical findings via the user interface during the reading session while the user is logged into the user interface.

4. 4. The device of claim 1, wherein the at least one electronic processor is further programmed to analyze the time-dependent user performance metric at daily time intervals to identify one or more time intervals in which the time-dependent user performance metric is below a threshold.

5. 5. The apparatus of claim 1, wherein the at least one electronic processor is programmed to repeat execution of the tracking method for each different user and generate user-specific time-dependent user performance metrics for the different users, and wherein the electronic processor is further programmed to compare performance of the different users by displaying a comparison of the user-specific time-dependent user performance metrics.

6. The at least one electronic processor: analyzing the time-dependent user performance metric to determine when the time-dependent user performance metric falls below a predetermined quality threshold based on a patient context of the images reviewed to generate the time-dependent user performance metric; modifying the user's work schedule if the time-dependent user performance metric falls below the predetermined quality threshold; 10. The apparatus of claim 1, programmed to:

7. The change is: Adding or removing cases from the user's work schedule; and generating the work schedule for the user based on the at least one time-dependent user performance metric for the user; 7. The apparatus of claim 6, comprising one or more of:

Images