System and method for automated standardization of surgical outcome data

Abstract

A method for automated determination of a surgical outcome comprises acquiring data comprising diagnosis codes and text notes related to a subject who underwent a surgery from a medical record database, selecting at least one surgical outcome from a set of surgical outcomes based on the diagnosis codes, searching the text notes for at least one keyword from a set of keywords, when a first keyword from the set of keywords is found in the text notes, validating the selected surgical outcome, when a second keyword from the set of keywords is found in the text notes, selecting at least one additional surgical outcome from the set of surgical outcomes, and adding the at least one surgical outcome to a database of surgical outcomes indexed to the subject and the surgery. A system for automated determination of a surgical outcome is also disclosed.

Description

Attorney docket # 047162-5363-00WO SYSTEM AND METHOD FOR AUTOMATED STANDARDIZATION OF SURGICAL OUTCOME DATA CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No.63 / 613,886 filed on December 22, 2023, incorporated herein by reference in its entirety. BACKGROUND OF THE INVENTION

[0002] Persistent variation in radical prostatectomy outcomes underscores the need for procedure-specific quality improvement (QI) efforts. [see Pooli, A., et al., In Urologic Oncology: Seminars and Original Investigations, 2020; Gore, J. L., et al., Cancer, 2012, 118(4), 987-996]. Discrepancies in post-acute care utilization also represent opportunities to improve cost- efficiency. [see Herrel, L. A., et al., Urology, 2016, 97, 105-110]. Furthermore, the recent addition of prostate cancer surgeries outcomes to the U.S. News & World Report Best Hospital Rating algorithm has incentivized QI initiatives to optimize prostatectomy quality metrics. [see U.S. News & World Report, L.P. (2022). Methodology U.S. News & World Report 2022-23 Best Hospitals: Specialty Rankings. Retrieved March 7, 2023 from U.S. News & World Report, L.P.]

[0003] Launched in 2004 by the American College of Surgeons (ACS), the National Surgical Quality Improvement Program (NSQIP) provides critical quality metric data reporting and national benchmarking. [see Fink AS, et al., Ann Surg.2002 Sep; 236(3):344-53; discussion 353-4]. NSQIP methodology involves standardized data collection performed by dedicated trained data abstractors; requires substantial financial investment to initiate and maintain; and may be cost prohibitive for resource-constrained systems. [see Eisenstein, S., et al., Clinics in colon and rectal surgery, 2019; 32(01), 041-053]. Procedure-specific data for urology cases, including prostatectomy, have only been captured by NSQIP since 2019. [American College of Surgeons National Surgical Quality Improvement Program®. (2022). User Guide for the 2021Attorney docket # 047162-5363-00WO Procedure Targeted Participant Use Data File (PUF). Retrieved March 7, 2023 from American College of Surgeons National Surgical Quality Improvement Program®.: facs.org / quality- programs / data-and-registries / acs-nsqip / participant-use-data-file / ].

[0004] Thus, there exists a need for reliable and accessible data collection to serve local procedure-specific QI efforts, including for prostatectomy [see Harvey, G., et al., BMJ Quality & Safety, 2003, 12(3), 210-214] Such efforts would make outcomes data more accessible to surgeons, as current knowledge of tracked metrics is poor [see Neuman, H., et al., Surgery, 2009, 145(1): 27-33], providing earlier recognition of QI opportunities. Surgical margin status, for example, has been shown to be correlated with performing surgeon [see Eastham, J., et al., Journal of Urology, 2003, 170(6 Pt 1):2292-5], demonstrating the need for surgeon-level metrics.

[0005] Administrative data, though more accessible, has reduced accuracy and reliability compared to registry data. [see Stey, A. M., et al., Surgery, 2015, 157(2), 381-395; and Lawson, E., et al., Annals of Surgery, 2012, 256 (6), 973-81] Prominent QI data registries in Urology include the Michigan Urological Surgery Improvement Collaborative (MUSIC) and the American Urological Association Quality (AQUA) Registry. Though MUSIC has been successful in catalyzing improvements in prostate cancer care across Michigan, it offers limited data management services to practices outside the region and requires trained abstractors for data collection. Furthermore, MUSIC capitalizes on a dominant single-payor system in Michigan, which may not be reproducible elsewhere. [see Montie, J. E., et al., (2014), Urology Practice, 20141(2), 74-78.; Reese, A. C., et al., Translational Andrology and Urology, 2021, 10(5), 2280] Similar regional collaboratives, such as the Pennsylvania Urologic Regional Collaborative (PURC), also depend on dedicated trained local data abstractors. AQUA uses an automated data collection methodology and has the capacity for national benchmarking, but relies on claims data and currently offers limited tracking for postoperative outcomes.

[0006] The monetary and work-hour cost of quality metric reporting is also significant, with the annual cost of data reporting to external organizations estimated to exceed $5 million and requiring >108,000 work hours in a study with a conservative estimate of total cost conducted at one academic hospital. [see Saraswathula, A., et al., JAMA, 2023, 329(21), 1840-1847], but electronic metrics make up only 0.1% and 0.2% of hours worked and cost, respectively. Thus,Attorney docket # 047162-5363-00WO development of automated data extraction from the electronic health record (EHR), and / or electronic medical record (EMR) would significantly reduce the cost and time burden in reporting quality metrics.

[0007] EMRs have become nearly universal in the US over the past decade with the passage of the 2009 Health Information Technology for Economic and Clinical Health Act [see Atasoy, H., et al., Annual review of public health, 2019, 40, 487-500], providing opportunities to expand the use of electronic metrics, standardize abstraction methodology and facilitate local QI efforts. Thus, there is a need in the art for a novel automated algorithm-driven institutional dashboard for surgical outcomes and quality metrics following robot-assisted laparoscopic radical prostatectomy (RALP) with significant inter-rater reliability with the institutional National Surgical Quality Improvement Program (NSQIP)-trained data abstraction team. SUMMARY OF THE INVENTION

[0008] In one aspect, a method for automated determination of a surgical outcome comprises acquiring data comprising diagnosis codes and text notes related to a subject who underwent a surgery from a medical record database, selecting at least one surgical outcome from a set of surgical outcomes based on the diagnosis codes, searching the text notes for at least one keyword from a set of keywords, when a first keyword from the set of keywords is found in the text notes, validating the selected surgical outcome, when a second keyword from the set of keywords is found in the text notes, selecting at least one additional surgical outcome from the set of surgical outcomes, and adding the at least one surgical outcome to a database of surgical outcomes indexed to the subject and the surgery.

[0009] In one embodiment, the diagnosis codes comprise CPT codes or ICD codes. In one embodiment, the surgery is a robot-assisted laparoscopic radical prostatectomy. In one embodiment, the step of searching the text notes for at least one keyword comprises searching for multiple keywords using at least one Boolean operator selected from AND and OR. In one embodiment, the step of searching the text notes for at least one keyword comprises searching for multiple keywords using add-on codes or modifiers for diagnosis codes. In one embodiment,Attorney docket # 047162-5363-00WO the add-on codes or modifiers comprise addition or subtraction of diagnosis codes. In one embodiment, the step of searching the text notes for at least one keyword comprises using natural language processing (NLP) to analyze the text notes. In one embodiment, the step of searching the text notes for at least one keyword comprises the step of selecting the at least one keyword from the set of keywords based on which provider supplied the text notes. In one embodiment, the step of validating the selected surgical outcome comprises a comparison to one or more institutional benchmarks or quality metrics. In one embodiment, the one or more one or more institutional benchmarks or quality metrics comprise any of National Surgical Quality Improvement Program (NSQIP), Positive predictive value (PPV), and negative predictive value (NPV).

[0010] In one aspect, a system for automated determination of a surgical outcome comprises a non-transitory computer readable medium with instructions stored thereon, which when executed by a processor, performs steps comprising a method as disclosed herein. In one embodiment, the system further comprises a display device, and the instructions further comprise presenting a graphical user interface comprising data from the database of surgical outcomes indexed to the subject and the surgery.

[0011] In one embodiment, the graphical user interface is configured to display one or more graphs of the data over time. In one embodiment, the graphical user interface is configured to display one or more quality scores or metrics associated with the data. In one embodiment, the database of surgical outcomes is further indexed to one or more providers associated with the subject and the surgery. In one embodiment, the graphical user interface is configured to display one or more graphs of surgical outcomes over time for the one or more providers. In one embodiment, the graphical user interface is configured to display one or more quality scores or metrics associated with one or more providers. In one embodiment, the database of surgical outcomes is further indexed to one or more institutions associated with the one or more providers. In one embodiment, the graphical user interface is configured to display one or more graphs of surgical outcomes over time for the one or more institutions. In one embodiment, the graphical user interface is configured to display one or more quality scores or metrics associated with one or more institutions.Attorney docket # 047162-5363-00WO BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The foregoing purposes and features, as well as other purposes and features, will become apparent with reference to the description and accompanying figures below, which are included to provide an understanding of the invention and constitute a part of the specification, in which like numerals represent like elements, and in which: Fig.1 is a diagram of an exemplary computing device. Fig.2 is a diagram of an exemplary graphical user interface (GUI) of the present disclosure. Fig.3 is a diagram depicting an exemplary method of the present disclosure. Fig.4 – Fig.10 show exemplary GUI screens of the present disclosure. DETAILED DESCRIPTION

[0013] It is to be understood that the figures and descriptions of the present invention have been simplified to illustrate elements that are relevant for a clear understanding of the present invention, while eliminating, for the purpose of clarity, many other elements found in related systems and methods. Those of ordinary skill in the art may recognize that other elements and / or steps are desirable and / or required in implementing the present invention. However, because such elements and steps are well known in the art, and because they do not facilitate a better understanding of the present invention, a discussion of such elements and steps is not provided herein. The disclosure herein is directed to all such variations and modifications to such elements and methods known to those skilled in the art.

[0014] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present invention, exemplary methods and materials are described.Attorney docket # 047162-5363-00WO

[0015] As used herein, each of the following terms has the meaning associated with it in this section.

[0016] The articles “a” and “an” are used herein to refer to one or to more than one (i.e., to at least one) of the grammatical object of the article. By way of example, “an element” means one element or more than one element.

[0017] “About” as used herein when referring to a measurable value such as an amount, a temporal duration, and the like, is meant to encompass variations of ±20%, ±10%, ±5%, ±1%, and ±0.1% from the specified value, as such variations are appropriate.

[0018] Throughout this disclosure, various aspects of the invention can be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 2.7, 3, 4, 5, 5.3, 6 and any whole and partial increments therebetween. This applies regardless of the breadth of the range.

[0019] In some aspects of the present invention, software executing the instructions provided herein may be stored on a non-transitory computer-readable medium, wherein the software performs some or all of the steps of the present invention when executed on a processor.

[0020] Aspects of the invention relate to algorithms executed in computer software. Though certain embodiments may be described as written in particular programming languages, or executed on particular operating systems or computing platforms, it is understood that the system and method of the present invention is not limited to any particular computing language, platform, or combination thereof. Software executing the algorithms described herein may be written in any programming language known in the art, compiled or interpreted, including but not limited to C, C++, C#, Objective-C, Java, JavaScript, MATLAB, Python, PHP, Perl, Ruby, or Visual Basic. It is further understood that elements of the present invention may be executedAttorney docket # 047162-5363-00WO on any acceptable computing platform, including but not limited to a server, a cloud instance, a workstation, a thin client, a mobile device, an embedded microcontroller, a television, or any other suitable computing device known in the art.

[0021] Parts of this invention are described as software running on a computing device. Though software described herein may be disclosed as operating on one particular computing device (e.g. a dedicated server or a workstation), it is understood in the art that software is intrinsically portable and that most software running on a dedicated server may also be run, for the purposes of the present invention, on any of a wide range of devices including desktop or mobile devices, laptops, tablets, smartphones, watches, wearable electronics or other wireless digital / cellular phones, televisions, cloud instances, embedded microcontrollers, thin client devices, or any other suitable computing device known in the art.

[0022] Similarly, parts of this invention are described as communicating over a variety of wireless or wired computer networks. For the purposes of this invention, the words “network”, “networked”, and “networking” are understood to encompass wired Ethernet, fiber optic connections, wireless connections including any of the various 802.11 standards, cellular WAN infrastructures such as 3G, 4G / LTE, or 5G networks, Bluetooth®, Bluetooth® Low Energy (BLE) or Zigbee® communication links, or any other method by which one electronic device is capable of communicating with another. In some embodiments, elements of the networked portion of the invention may be implemented over a Virtual Private Network (VPN).

[0023] Fig.1 and the following discussion are intended to provide a brief, general description of a suitable computing environment in which the invention may be implemented. While the invention is described above in the general context of program modules that execute in conjunction with an application program that runs on an operating system on a computer, those skilled in the art will recognize that the invention may also be implemented in combination with other program modules.

[0024] Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems,Attorney docket # 047162-5363-00WO microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0025] Fig.1 depicts an illustrative computer architecture for a computer 100 for practicing the various embodiments of the invention. The computer architecture shown in Fig.1 illustrates a conventional personal computer, including a central processing unit 150 (“CPU”), a system memory105, including a random access memory 110 (“RAM”) and a read-only memory (“ROM”) 115, and a system bus 135 that couples the system memory 105 to the CPU 150. A basic input / output system containing the basic routines that help to transfer information between elements within the computer, such as during startup, is stored in the ROM 115. The computer 100 further includes a storage device 120 for storing an operating system 125, application / program 130, and data.

[0026] The storage device 120 is connected to the CPU 150 through a storage controller (not shown) connected to the bus 135. The storage device 120 and its associated computer-readable media provide non-volatile storage for the computer 100. Although the description of computer- readable media contained herein refers to a storage device, such as a hard disk or CD-ROM drive, it should be appreciated by those skilled in the art that computer-readable media can be any available media that can be accessed by the computer 100.

[0027] By way of example, and not to be limiting, computer-readable media may comprise computer storage media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer.Attorney docket # 047162-5363-00WO

[0028] According to various embodiments of the invention, the computer 100 may operate in a networked environment using logical connections to remote computers through a network 140, such as TCP / IP network such as the Internet or an intranet. The computer 100 may connect to the network 140 through a network interface unit 145 connected to the bus 135. It should be appreciated that the network interface unit 145 may also be utilized to connect to other types of networks and remote computer systems.

[0029] The computer 100 may also include an input / output controller 155 for receiving and processing input from a number of input / output devices 160, including a keyboard, a mouse, a touchscreen, a camera, a microphone, a controller, a joystick, or other type of input device. Similarly, the input / output controller 155 may provide output to a display screen, a printer, a speaker, or other type of output device. The computer 100 can connect to the input / output device 160 via a wired connection including, but not limited to, fiber optic, Ethernet, or copper wire or wireless means including, but not limited to, Wi-Fi, Bluetooth, Near-Field Communication (NFC), infrared, or other suitable wired or wireless connections.

[0030] As mentioned briefly above, a number of program modules and data files may be stored in the storage device 120 and / or RAM 110 of the computer 100, including an operating system 125 suitable for controlling the operation of a networked computer. The storage device 120 and RAM 110 may also store one or more applications / programs 130. In particular, the storage device 120 and RAM 110 may store an application / program 130 for providing a variety of functionalities to a user. For instance, the application / program 130 may comprise many types of programs such as a word processing application, a spreadsheet application, a desktop publishing application, a database application, a gaming application, internet browsing application, electronic mail application, messaging application, and the like. According to an embodiment of the present invention, the application / program 130 comprises a multiple functionality software application for providing word processing functionality, slide presentation functionality, spreadsheet functionality, database functionality and the like.

[0031] The computer 100 in some embodiments can include a variety of sensors 165 for monitoring the environment surrounding and the environment internal to the computer 100. These sensors 165 can include a Global Positioning System (GPS) sensor, a photosensitiveAttorney docket # 047162-5363-00WO sensor, a gyroscope, a magnetometer, thermometer, a proximity sensor, an accelerometer, a microphone, biometric sensor, barometer, humidity sensor, radiation sensor, or any other suitable sensor.

[0032] The persistent variation in prostate cancer surgery outcomes as well as recent inclusion of prostate cancer surgery outcomes in the quality metric algorithm for national hospital rating demonstrate the need for institutional quality improvement efforts to optimize prostatectomy outcomes. High-reliability quality data collection is vital to quality improvement (QI) initiatives in Urology, but available systems are ill-equipped to address institutional needs. NSQIP has facilitated surgical QI for nearly two decades. However, NSQIP services are limited by rigidly defined outcomes variables that are less procedure-specific, and high overhead costs which preclude its use across resource-poor hospitals. More accessible registries for Urology practices, such as AQUA, rely on administrative data and currently lack granular clinical outcomes data to adequately support local procedure-specific QI initiatives. Disclosed herein is a novel, EMR- based, automated algorithm-driven institutional dashboard for NSQIP-defined surgical outcomes following RALP that achieves >90% sensitivity for all variables except rectal injury and met or exceeded the specificity of the institutional NSQIP abstractors for all variables. Additionally, it extracts pathology variables not tracked by NSQIP with 100% sensitivity and specificity, making this valuable data more accessible. Moreover, the dashboard demonstrated substantial agreement with NSQIP across most variables further supporting its status as an alternative to the constrained processes of NSQIP without compromising quality of data.

[0033] Other studies have also demonstrated discrepancies between local databases and NSQIP. Epelboym et al found that postoperative events for pancreatectomy cases were often misclassified by NSQIP and observed discordance in 29.3% of cases. [see Epelboym, I., et al., World journal of surgery, 2014, 38, 1461-1467]. In a comparison of National Surgical Quality Improvement Program-Pediatric (NSQIP-P) to an institutional research database for appendectomy cases, Anderson et al observed significant differences in rates of emergency room visits, readmissions, and composite morbidity between the two sources. [see Anderson, K. T., et al., Journal of pediatric surgery, 2019, 54(1), 97-102]. While no clear cause for these discrepancies could be identified, the risk of variation in outcomes reporting is likely increased by the inherent subjectivity of individual data abstraction.Attorney docket # 047162-5363-00WO

[0034] Among the most significant advantages of disclosed system is its resource efficiency. A recent review of healthcare quality reporting found that the annual cost of electronically tracked data, per metric, was < 1% of that incurred by either claims-based or chart-abstracted metrics representing possible savings of millions of dollars for a single hospital. Moreover, the number of personnel hours required to implement and maintain automated electronic data collection was similarly minute compared to alternatives. [see Saraswathula, A., et al., JAMA, 2023, 329(21), 1840-1847] In light of this, the upfront investment of time in developing this dashboard and subsequent upkeep ranges in the hundreds of hours compared to the over 100,000 hours annually spent on data extraction and validation at the aforementioned study’s single hospital. Costs are nominal compared to the reported cost of claims-based and chart-abstracted metrics. Investments in automated quality reporting systems can therefore substantially reduce the overall costs of institutional QI initiatives. The disclosed system has the potential to fill this void for radical prostatectomy outcomes by providing low-cost, highly reliable retrospective delivery of quality metrics and outcomes data.

[0035] The disclosed system capitalizes on the rich store of high-precision data in the EMR to obviate the burden of individual abstractors. Regional clinical data registries, such as MUSIC and PURC, similarly require institutional data abstractors and local surgeon champions. Many hospitals may not be able to afford the initial investment and maintenance expenses of these services. Implementation of procedure-specific automated EMR data extraction algorithms, like the disclosed system, may obviate the need for a dedicated team of trained data abstractors. Moreover, the granularity of the data provided by automated algorithms will continue to increase if ICD-10 and CPT codes continue to become more specific and EMR intermediate variables grow standardized. This granularity is already evidenced in the disclosed system by the superior sensitivity across the ICD-10 code-based outcomes of Clostridium difficile colitis and pneumonia, and 100% specificity across nearly all outcomes. Additionally, the scope of the data abstracted can be expanded through the addition of new logic to the algorithm that allows finer tuning to better represent the quality of a procedure while also immediately retroactively applying to previous cases, providing the efficient flexibility that individual EMR abstraction lacks. This extracted data can then be risk adjusted against national benchmarks, similar to NSQIP’s practice.Attorney docket # 047162-5363-00WO

[0036] The disclosed system was developed in collaboration with the Joint Data Analytics Team (JDAT) of an academic hospital system who provided clinical and research analytics through the EMR data warehouse system. The disclosed algorithm was developed to automatically abstract surgical outcomes and quality metrics retrospectively from the EPICTMEMR for all RALP procedures and report them via a browser-based dashboard with variable filters to facilitate QI initiatives. Although certain examples referred to herein may reference one particular EHR, for example EPICTM, it is understood that the systems and methods disclosed herein may be used with any EHR system, including but not limited to ModMed, eClinicalWorks, EHR Your Way, NextGen Healthcare, Pabau, SimplePractice, Cerner, or any other EHR software. Similarly, although certain examples disclosed herein may refer to RALP procedures as the target surgical procedure for determining outcomes, it is understood that the systems and methods disclosed herein may be used to automatically determine and categorize outcomes for a wide variety of surgical and non-surgical procedures based on EMR data.

[0037] The automated prostatectomy outcomes dashboard (APOD) disclosed herein, also referred to as Algorithm for Automated Surgical Outcomes Data Abstraction (AASODA), has flexible data export tools and supports filtering of outcomes and quality metric variables by date and to the surgeon, hospital site, delivery network, department, or hospital system levels. All EMR data are in some embodiments directly abstracted from HELIXTM, the EPICTMEMR warehouse, for data visualization in a TableauTMServer. In some embodiments, a visualization may be configured to automatically output surgeon-specific run charts for the outcome measures with running average event rates for both the surgeon and the overall department overlayed. Charted data may be linked to source cases in the EMR by medical record number (MRN) to facilitate direct auditing. In some embodiments, a user interface and / or dashboard as disclosed herein may comprise a random case selection function to facilitate generation of case lists for auditing.

[0038] In some embodiments, a method or system disclosed herein may collect data from one or more EHRs. Data collected may include one or more of patient name, date of birth, medical record number (MRN), contact serial number (CSN) of the surgical encounter, procedure date, primary surgeon, and procedure description. Age at time of surgery may be collected or may be calculated as the difference between the extracted date of birth and date of surgery. Length ofAttorney docket # 047162-5363-00WO stay may be collected directly or may be defined as the difference in days between the admission date and the discharge date for a procedure. Defined surgical outcomes, for example NSQIP- defined surgical outcomes and / or pathology data may be collected, with or without data related to frequencies of surgical site infection (SSI) and venous thromboembolism (VTE). NSQIP- defined patient comorbidities and cancer treatment history may also be collected where available.

[0039] Some pathology data fields may be populated for example by text recognition of keywords which occur before and after the outcome of interest in templated pathology reports and extracting the outcome note text which occurs between these boundaries for display in the APOD. In some embodiments, natural language processing (NLP) may be used to collect terms of interest. Selected text preceding the variables may vary between pathology report templates across delivery networks or EHRs, which may require the use of different keywords in different contexts.

[0040] In some embodiments, a system or method as disclosed herein may comprise the step of acquiring data having different formats, for example data from different EMRs or EHRs, or data stored in multiple different formats in the same EMR or EHR, then standardizing the data to use as an input to an algorithm for automated determination of one or more surgical outcomes. Graphical User Interface

[0041] Fig.2 is a diagram of an exemplary graphical user interface (GUI) of the present disclosure that may comprise one or more dashboards. In some embodiments, a system as disclosed herein may comprise a graphical user interface (GUI) comprising for example at least one dashboard. Fig.2 depicts an exemplary dashboard 200 for the GUI comprising a navigation bar 210 and content area 220. In some embodiments, dashboard 200 further comprises a search tool 230 and drop down menu or selector 240 (e.g., a content and / or outcome selector). In some embodiments, dashboard 200 provides one or more tabs and / or lists with organized widgets, modules, alerts, graphs, and / or reports displayed in content area 220 of the GUI that give insight patient-related medical data. In some embodiments, a GUI may comprise a Prostatectomy Dashboard, comprising a list of prostatectomies with associated outcome and descriptive flags. Filters are available to sort the data. In some embodiments, a GUI may comprise an EncounterAttorney docket # 047162-5363-00WO Counts dashboard, comprising a graph of encounters by provider over time. An outcome selector may be used to highlight encounters by different outcomes. In some embodiments, a GUI may comprise a Metric Graphs Dashboard, comprising a graph of outcome percent over time. For each provider and year a global average may be shown, along with a provider all-time average and a provider yearly average. The Metric selector may be used to change the outcome being viewed. In some embodiments, a GUI may comprise an Average Length of Stay Dashboard, comprising a graph of average length of stay over time. For each provider and year a global average may be shown, along with a provider all-time average and a provider yearly average.

[0042] In some embodiments, the GUI displays various data from a database of surgical outcomes indexed to the subject and / or the surgery. In some embodiments, the GUI is configured to display one or more graphs of the data over time. In some embodiments, the GUI is configured to display one or more quality scores or metrics associated with the data, as discussed herein and in the example below. Aspects of the present disclosure relate to providing a score or rating for a provider or institution based on surgical outcomes. In some embodiments, the database of surgical outcomes is further indexed to one or more providers associated with the institution, subject and / or the surgery. In some embodiments, the GUI is configured to display one or more graphs of surgical outcomes over time for the one or more providers. In some embodiments, the GUI is configured to display one or more quality scores or metrics associated with one or more providers. In some embodiments, the database of surgical outcomes is further indexed to one or more institutions, and / or one or more institutions associated with the one or more providers. In some embodiments, the GUI is configured to display one or more graphs of surgical outcomes over time for the one or more institutions. In some embodiments, the GUI is configured to display one or more quality scores or metrics associated with one or more institutions.

[0043] The Dashboards and GUIs disclosed herein may display any data as disclosed below. In some embodiments, diagnosis information is taken from both the patient problem list and the encounter diagnoses. The noted date or date of entry is used as an indicator for the purposes of capturing time of diagnosis. All outcomes marked with an ‘*’ are captured from interpreting pathology report text. Exemplary GUI screens and dashboards are also shown in Fig.4 – Fig.10.Attorney docket # 047162-5363-00WO

[0044] Outcomes and Patient Encounter information:

[0045] 30d Mortality YN - Patient mortality within 30 days of the index surgery

[0046] LOS (days) - Length of hospital stay in days for the index surgical encounter

[0047] 30 day readmission - Presence of additional encounters for hospital admission following discharge from the index surgical encounter within 30 days of the index surgery date

[0048] OR Return w / in 30 days - Presence of additional surgeries performed within 30 days following the index surgery

[0049] Ventilator 48h+ 30d - Cumulative recorded time for mechanical ventilation exceeding 48hr within 30 days of the index surgery (does not include the index surgery itself)

[0050] Unplanned Intubation - Additional intubations / airway access device placements performed within 30 days of the index surgical encounter following initial extubation.

[0051] Ureteral Obstruction - Diagnosis of ureteral obstruction or placement of ureteral stent / nephrostomy tube within 30 days following the index surgery.

[0052] Anastomotic bowel leak - All of the following conditions are met: A nasogastric tube (NGT) was present during the surgical encounter and was removed at least one day after the surgery, or within 30 days of the index surgery the patient was on NPO diet orders, removed from those NPO orders, and then had those NPO orders reinstated; a procedure code indicating rectal injury was found within 30 days of the index surgery; a procedure code indicating drainage was found within 30 days of the index surgery; IV antibiotics were given to the patient after the surgery date and during the surgical encounter; and TPN was indicated within 30 days of the index surgery.

[0053] Prolonged NGT / NPO - One or all of the following conditions are met: NGT was present during the surgical encounter and was removed at least 4 days after surgery; No return to the OR was founding within 30 days of the index surgery and one of the following are true: An NPO order lasting at least 4 days from the index surgery; Within 30 days of the index surgery an NPOAttorney docket # 047162-5363-00WO order was found, the patient was removed from that NPO order, and then placed on NPO orders again at least 4 days post index surgery.

[0054] Dialysis 30d - Need for renal replacement procedure (hemodialysis, CCVH, CRRT) within 30 days of the index surgery.

[0055] Renal Insufficiency – Diagnosis of acute kidney injury or rise in creatinine of 0.3mg / dl or 50% from previous over a 48 period at any point within 30 days of the index surgery

[0056] Positive Surgical Margins* - Presence of positive surgical margins on final pathology specimen

[0057] Bowel Injury - A diagnosis indicating Bowel Injury within 30 days of index surgery or presence of a procedure code indicating rectal injury within 30 days of index surgery

[0058] Bladder Neck Contracture - Diagnosis of bladder neck contracture or need for postoperative procedure intervention for bladder neck contracture at any time following index surgical procedure**

[0059] Pelvic Fluid Collection – Presence of a procedure code indicating pelvic fluid collection within 30 days of index surgery

[0060] Lymphoceles - Diagnosis indicating Lymphoceles or a procedure code indicating Sclerotherapy within 30 days of index surgery

[0061] Urine Leak - One or both of the following conditions are true: Presence of two Cystogram orders within 60 days of the index surgery such that the second Cystogram order was at least 1 day after the first; Presence of two Fluoro orders within 60 days of the index surgery such that the second Fluoro order was at least 1 day after the first.

[0062] Cardiac Event 30d - Presence of a Cardiac Cath order or a Troponin result 3x higher than the reference high value within 30 days of the index surgery

[0063] UTI - Presence of an abnormal urine culture within 30 days of the index surgeryAttorney docket # 047162-5363-00WO

[0064] DeNovo Erectile Dysfunction - A new diagnosis of Erectile Dysfunction 6 months post surgery with no Erectile Dysfunction diagnosis prior to surgery

[0065] DeNovo Urinary Incontinence - A new diagnosis of Urinary Incontinence 6 months post surgery with no Urinary Incontinence diagnosis prior to surgery

[0066] Sepsis 30d - One or both of the following conditions are true: A diagnosis indicating sepsis noted between 30 days prior to the index surgery and 30 days after the index surgery; Both of the following are true: Within 30 days of the index surgery the patient meets Sepsis SIRS conditions, defined meeting at least two of the following conditions within 24 hours of each other: A temperature below 96.8 or above 100.4; A heart rate over 90; A respiratory rate over 20; A WBC below 4000 or above 12000; The patient has an abnormal respiratory, blood, urine, abscess, or wound culture within 5 days of a Sepsis SIRS conditions being met.

[0067] CDiff 30d - An abnormal Cdiff lab result or an indication of CDiff by infection control within 30 days of the index surgery

[0068] Pneumonia 30d - An abnormal respiratory culture or a diagnosis indicating pneumonia within 30 days of the index surgery

[0069] Stage - Cancer stage according to the AJCC criteria

[0070] Core Ct Positive - Number of prostate biopsy cores for which cancer cells were identified on pathological analysis

[0071] Core Ct - Total number of cores sampled during prostate biopsy

[0072] Gleason Max - The higher of the two most common Gleason grades as determined by pathological analysis of biopsy specimen

[0073] Gleason Min - The lower of the two most common Gleason grades as determined by pathological analysis of biopsy specimen

[0074] Gleason Total - The Gleason score, represented as the sum of ‘Gleason Max’ and ‘Gleason Min’Attorney docket # 047162-5363-00WO

[0075] Percent Cores Positive - Percentage of total cores sampled during prostate biopsy for which cancer cells were identified on pathological analysis

[0076] Extraprostatic Extension - Presence of cancer cells outside of the prostate as determined by final pathological analysis of surgical specimens

[0077] Seminal Vesicle invasion - Presence of cancer cells in seminal vesicle tissue as determined by final pathological analysis of surgical specimens

[0078] Perineural Invasion - Presence of cancer cells within or along nerve tissue as determined by final pathological analysis of surgical specimens

[0079] Lymphovascular Invasion - Presence of cancer cells within an endothelium-lined space as determined by final pathological analysis of surgical specimens

[0080] Surgical Margins - Presence of cancer cells at the resection margin as determined by final pathological analysis of surgical specimens

[0081] Lymph Nodes Examined - Number of lymph node specimens submitted for final pathological analysis

[0082] Lymph Nodes Percent - Percentage of lymph node specimens submitted for final pathological analysis for which cancer cells were identified on final pathological analysis

[0083] Aspects of the present disclosure relate to a system and method for automated determination of a surgical outcome. Disclosed is a system for automated determination of a surgical outcome comprising a non-transitory computer readable medium with instructions stored thereon, which when executed by a processor, performs any disclosed methods, procedures and / or steps. In some embodiments, the system may further comprise a display device, the instructions further comprising presenting a user interface (e.g., a GUI) comprising data from the database of surgical outcomes indexed to the subject and the surgery.

[0084] Fig.3 is a diagram depicting an exemplary method 300 for automated determination of at least one surgical outcome comprising the steps of 301 acquiring data comprising diagnosis codes and text notes related to a subject who underwent a surgery from a medical recordAttorney docket # 047162-5363-00WO database, 302 selecting at least one surgical outcome from a set of surgical outcomes based on the diagnosis codes, 303 searching the text notes for at least one keyword from a set of keywords, 304 when a first keyword from the set of keywords is found in the text notes, validating the selected surgical outcome, 305 when a second keyword from the set of keywords is found in the text notes, selecting at least one additional surgical outcome from the set of surgical outcomes, and 306 adding the at least one surgical outcome to a database of surgical outcomes indexed to the subject and the surgery.

[0085] In some embodiments, the surgery is a robot-assisted laparoscopic radical prostatectomy. In some embodiments, the step of searching the text notes for at least one keyword comprises searching for multiple keywords using at least one Boolean operator, which may be selected from AND and OR. In some embodiments, the step of searching the text notes for at least one keyword comprises using natural language processing (NLP) to analyze the text notes. In some embodiments, the step of searching the text notes for at least one keyword comprises the step of selecting the at least one keyword from the set of keywords based on which provider supplied the text notes.

[0086] Any known medical coding add-on codes and / or modifiers may be used with the disclosed methods. In some embodiments, the step of searching the text notes for at least one keyword comprises searching for multiple keywords using add-on codes or modifiers for diagnosis codes. In some embodiments, the add-on codes or modifiers comprise addition or subtraction of diagnosis codes. In some embodiments, the step of validating the selected surgical outcome comprises a comparison to one or more institutional benchmarks or quality metrics. In some embodiments, the one or more one or more institutional benchmarks or quality metrics comprise any of National Surgical Quality Improvement Program (NSQIP), Positive predictive value (PPV), and negative predictive value (NPV). EXPERIMENTAL EXAMPLES

[0087] The invention is further described in detail by reference to the following experimental examples. These examples are provided for purposes of illustration only, and are not intended toAttorney docket # 047162-5363-00WO be limiting unless otherwise specified. Thus, the invention should in no way be construed as being limited to the following examples, but rather, should be construed to encompass any and all variations which become evident as a result of the teaching provided herein.

[0088] Without further description, it is believed that one of ordinary skill in the art can, using the preceding description and the following illustrative examples, make and utilize the system and method of the present invention. The following working examples therefore, specifically point out the exemplary embodiments of the present invention, and are not to be construed as limiting in any way the remainder of the disclosure. Automated Electronic Medical Record Abstraction Algorithm for Radical Prostatectomy Outcomes

[0089] Variation in outcomes following radical prostatectomy and inclusion of prostate cancer surgery metrics in hospital ratings signal need for procedure-specific quality improvement (QI) efforts. The disclosed novel electronic medical record (EMR)-based, automated algorithm-driven dashboard for surgical outcomes and quality metrics following robot-assisted laparoscopic radical prostatectomy (RALP) demonstrates >90% sensitivity and specificity and significant inter-rater reliability (IRR) with National Surgical Quality Improvement Program (NSQIP) abstraction.

[0090] In some embodiments, the disclosed algorithm is configured to automatically abstract RALP outcomes and quality metrics retrospectively from the EMR. Pathology results were abstracted through text extraction; surgical outcomes were abstracted using ICD-10 codes, CPT codes, and EMR data variables. Sensitivity, specificity, and IRR between the algorithm and NSQIP-abstraction were assessed using Cohen’s kappa with statistical significance set p<0.05.

[0091] In this example, a total of 927 cases were mutually tracked. IRR was highest for mortality (k=1.00) and lowest for dialysis and ureteral obstruction (k=0.00). IRR was fair for: sepsis (k=0.28), renal insufficiency (k=0.32), and prolonged NGT / NPO (k=0.39); moderate: UTI (k=0.50) and stage (k=0.53); substantial: surgical margins (k=0.94), urine leak (k=0.60), C-Diff (k=0.67), pneumonia (k=0.80). Sensitivity of the algorithm was > 90% for all mutually tracked outcomes except rectal injury (0%) and specificity was >97%.Attorney docket # 047162-5363-00WO

[0092] This disclosed algorithm for RALP outcomes matches or exceeds sensitivity and specificity of institutional NSQIP abstraction for all but one variable. Substantial agreement between the algorithm and NSQIP supports automated extraction of outcome metrics as an acceptable replacement for trained abstractors, and broader application provides opportunities to facilitate and reduce cost of outcomes and quality metric benchmarking. Developmental Overview

[0093] Disclosed herein is an algorithm to automatically abstract surgical outcomes and quality metrics retrospectively from the EPICTM(Epic Systems Corporation, Verona, WI) EMR for all RALP procedures and report them via a browser-based dashboard (e.g., a user interface). The resulting algorithm, known in some examples as Algorithm for Automated Surgical Outcomes Data Abstraction (AASODA) or Automated Prostatectomy Outcomes Dashboard (APOD), has flexible data export tools and supports filtering of outcomes and quality metrics. In this example, all EMR data were abstracted from the EPICTMEMR database ClarityTMand processed with SQL ServerTM(Microsoft Corporation, Redmond, WA) and RTM (R Core Team, Vienna, Austria), for visualization in a Tableau ServerTM(Tableau Software, Inc., Mountain View, CA) with automatically generated run charts. Charted data were linked to source cases in the EMR by medical record number (MRN). Patient Denominator

[0094] Source cases included all RALPs performed between January 2013—May 2023 by urologists employed by the hospital system as identified by CPT code and primary surgeon name. Pathology results were abstracted through recognition of exact-text matches from pathology reports while NSQIP-defined surgical outcomes were abstracted using a combination of ICD-10 codes, CPT codes, and other EMR intermediate data variables. Internal and external validation of the algorithm was performed through an iterative 15% sampling audit of the EMR for procedures performed at the hospital system main delivery network between January 2013- May 2023 and procedures performed at the hospital system outer delivery networks between September 2018-April 2023, respectively. Matched cases between the APOD and NSQIP data abstraction for procedures performed at all delivery networks between October 2015-December 2022 were used to compare sensitivity, specificity, and IRR in a sample that included all casesAttorney docket # 047162-5363-00WO identified by the APOD or NSQIP as having an event and a random sampling of negative cases to total 15%. NSQIP abstractors for the main delivery network review all major urologic oncology cases. At the outside delivery network, 5 Urology cases per 8-day cycle are reviewed.

[0095] The algorithm searched operating room (OR) logs for free text and / or provider identification numbers containing institutional urologists as the primary surgeon, followed by CPT codes for prostatectomy. Inclusion parameters were defined with explanatory variables such as ICD-10 codes, CPT codes, vital sign values, laboratory values, were used to build the component logic of the APOD algorithm for each outcome (see Table 1, containing a list of CPT and diagnostic codes used by the disclosed algorithm). Procedures were included in the dashboard if they meet these inclusion criteria. Procedures were excluded if the CPT codes or keywords in the procedure description field included other major procedures such as cystectomy.Outcome ComponentCPT and Diagnosis CodesAnastomotic Need for CPT: 10030, 49062, 49323, 49405, 49406, 49407 Bowel Leak percutaneous drainage Procedure for bladder neck CPT:52275, 52276,53600, 52500, 52640 Bladder Neck contracture Contracture ICD-10: N35, N32.0, Q64.31, Q64.39; Exclude N35.9 Diagnosis of bladder neck contracture ICD-9: 596.0, 753.6, 598.2 Diagnosis of bowel ICD-10: S36.4, S36.5, S36.6, S36.8, S36.9, K91.7, K83.9 injury Bowel Injury CPT: 44120, 44605, 44602, 44603, 44604, 44615, 44640, 44650, 44680, Need for procedure 44227, 45562, 45540, 44120, 44605, 44602, 44603, 44604, 44615, 44640, for rectal injury 44650, 44680, 44227, 45562, 45540 Cardiac Diagnosis of Cardiac ICD-10: I21, I22, I24, I46 Complications Complication ICD-10: N52.0, N52.1, N52.2, N52.3, N52.31, N52.32, N52.33, N52.34, Erectile Diagnosis of Erectile N52.35, N52.36, N52.37, N52.39, N52.8, N52.9 Dysfunction Dysfunction ICD-9: 607.84, 302.72 Pelvic Fluid Need for percutaneou CPT: 10030, 49062, 49323, 49405, 49406, 49407 Collection s drainage DiagnosisICD-10: A48.1, J09.119, J11.00, J12.0, J12.1, J12.2, J12.81, J12.89, J12.9,PneumoniaofpneumoniaJ13, J14, J15.0, J15.1, J15.20, J15.21, J15.29, J15.3, J15.4, J15.5, J15.6, J15.7, J15.8, J15.9, J16.0, J16.8, J18.0, J18.1, J18.9, Z16, A01.03, A02.22, A37.00,Attorney docket # 047162-5363-00WO A37.01, A37.10, A37.11, A37.80, A37.81, A37.90, A37.91, A54.84, B01.2, B05.2, B06.81, B77.81, B95.3, B96, B96.1, J09.X1, J10.00, J10.01, J10.08, J11.00, J11.08, J12.0, J12.1, J12.2, J12.3, J12.81, J12.89, J12.9, J13, J14, J15.0, J15.1, J15.20, J15.211, J15.212, J15.29, J15.3, J15.4, J15.5, J15.6, J15.7, J15.8, J15.9, J16.0, J16.8, J17, J18.1, J18.2, J18.8, J18.9, J20.0, J84.111, J84.116, J84.117, J84.2, J85.1, J85.2, J95.851, A48.1 ICD-9:481, 482.0, 482.1, 482.2, 482.30, 482.31, 482.32, 482.39, 482.40, 482.41, 482.42, 482.49, 482.82, 482.83, 482.84, 482.89, 482.9, 483.0, 483.1 ICD-10: A02.1, A36.84, A40.0, A40.1, A40.3, A40.8, A40.9, A41.01, A41.02, A41.1, A41.2, A41.3, A41.50, A41.51, A41.52, A41.81, A41.9, B37.7, E86.1, G93.41, N05.7, N05.8, N10, N13.6, N14.11, N17.0, N17.1, Renal Diagnosis of acute N17.2, N17.9, N18.1, N18.2, N18.30, N18.31, N18.32, N18.4, N18.5, N18.9, Insufficiency kidney injury N28.0, N99.0, N99.89, O03.82, O04.82, O75.3, O85, P36.0, P36.4, R57.9, R65.20, R65.21, S37.009A, T50.8X5A, T65.91XA, T79.5XXA, T79.5XXD, T79.5XXS, U07.1, Z99.2 ICD-10: A02.1, A22.7, A26.7, A32.7, A39.0, A39.2, A39.4 A40 A40.0 A40.1, A40.3, A40.8, A40.9, A41, A41.0, A41.01, A41.02, A41.1, A41.2, A41.3, A41.4, A41.5, A41.50, A41.51, A41.52, A41.53, A41.59, A41.8, A41.81, A41.89, A41.9, A42.7, A48.0, A48.4, A54.86, A74.9, B00.7, B04, B20, B37.7, B48.8, B49, B97.89, B99.9, D65, D69.59, D70.9, E87.20, G62.81, G72.81, G93.40, G93.41, I50.1, I50.9, I51.5, I95.9, J18.9, J95.02, J96.00, J96.01, J96.02, J96.90, K12.1, K28.9, K29.60, K63.9, K72.00, Sepsis Diagnosis of sepsis K72.01, K83.09, K92.2, L03.90, L08.9, N10, N17.0, N17.1, N17.2, N17.8, N17.9, N39.0, N71.9, O03.37, O03.87, O07.37, O08.82, O66.9, O75.3, O85, O86.04, O86.19, O98.819, O99.53, P07.30, P36, P36.0, P36.1, P36.10, P36.19, P36.2, P36.3, P36.30, P36.39, P36.4, P36.5, P36.8, P36.9, P37.5, P39.8, R06.03, R65.2, R65.20, R65.21, S27.309A, T80.211A, T80.211D, T80.211S, T80.29XA, T80.29XD, T80.29XS, T81.44XA, T81.44XD, T81.44XS, T82.7XXA, T82.7XXD, T82.7XXS, T85.79XA, T85.79XD, T85.79XS, U07.1, Z16.21, Z90.81, IMO0001, IMO0002 Diagnosis of ureteral ICD-10: Q62.11, Q62.39, N13.0, N13.1, N13.3, N13.4, N13.5, N13.6 Ureteral obstruction Obstruction Renal decompression CPT: 52282, 52332, 52334 procedure ICD-10: R32, N39.3, N39.42, N39.43, N39.44, N39.45, N39.46, N39.48, Urinary Diagnosis of Urinary N39.8, N39.9, N39.490, N39.491, N39.492 Incontinence Incontinence ICD-9: 788.32, 788.33, 788.37 Table 1: List of CPT and Diagnosis Codes Used by the Algorithm Outcome Measures

[0096] The primary outcome measured was sensitivity and specificity of the AASODA and NSQIP review compared to individual EMR review as the gold standard. The secondary outcome of agreement between the AASODA and NSQIP abstraction was assessed with Cohen’sAttorney docket # 047162-5363-00WO kappa. Patient name, date of birth, medical record number (MRN), contact serial number (CSN) of the surgical encounter, procedure date, primary surgeon, and procedure description were collected. Sensitivity and specificity of the pathology variables abstracted by AASODA but not by NSQIP were also examined. Positive predictive value (PPV) and negative predictive value (NPV) were calculated for all variables. Age at time of surgery was calculated as the difference between the extracted date of birth and date of surgery. Length of stay (LOS) was defined as the difference between the admission time / date and the discharge time / date. NSQIP-defined surgical outcomes and pathology data were extracted except frequencies of surgical site infection (SSI) and venous thromboembolism (VTE) as they are subjects of dedicated institutional quality dashboards. NSQIP-defined patient comorbidities and cancer treatment history were not included in AASODA.

[0097] The defined oncologic pathology outcomes are shown in Table 2 below and surgical outcomes are shown in Table 3. All data are reported using Boolean expression except length of stay (LOS), operative time and pathology count variables, which are expressed as a continuous variables. Outcome Definition Biopsy Metrics Core Ct Number of cores sampled Core Ct Positive Number of cores with identifiable cancer cells Surgical Pathology Metrics Extraprostatic Extension Presence of cancer cells outside of the prostate Gleason Grade Group Gleason Grade Group score Lymphovascular Invasion Presence of cancer cells within an endothelium-lined space Lymph Nodes Examined Number of lymph nodes in the surgical specimen Lymph Nodes Positive Percentage of lymph nodes with identifiable cancer cells Perineural Invasion Presence of cancer cells within or along nerve tissue Seminal Vesicle invasion Presence of cancer cells in seminal vesicle tissue Stage As defined by the AJCC 8th Edition criteria Surgical Margins Presence of cancer cells at the resection margin Table 2: Tracked Oncologic Pathology Outcomes

[0098] Explanatory variables such as ICD-10 codes, CPT codes, vital sign values, laboratory values, were used to build the component logic of the AASODA algorithm for each outcomeAttorney docket # 047162-5363-00WO Table 1. The Biopsy metrics in Table 2 were derived from pathology result documentation of preoperative prostate biopsy. The surgical pathology metrics in Table 2 below are derived from pathology result documentation of surgical specimens.

[0099] Pathology data fields were populated by text recognition of keywords which occur before and after the outcome of interest in templated pathology reports and extracting the outcome note text which occurs between these boundaries for display in the AASODA. Keywords surrounding the outcomes varied between pathology report templates across delivery networks within the system requiring distinct keywords. Pathology reports were generated with CoPath and EPICTMAP Beaker module software. Incidence of each outcome of interest from January 2013-May 2023 was calculated using all cases identified by the AASODA as being positive for that outcome. False negatives and positives identified on audit were used to adjust the number of positive cases appropriately.

[0100] With reference to Table 3 below, the entries denoted1indicate that the data location may be site-specific. The entries denoted2indicate that the logic is adapted from separate algorithms. The entries denoted3indicate that they are defined in Table 1. Outcome Description Component Logic Flag Flag Criteria Data Source Within 30 days of Any procedures ordered surgery: Drainage Proc CPT3within 30 days of surgery date Need for percutaneous NGT placement after NGT > 4 days 72 hours LDA fl1Anastom Need for drainage owsheet otic treatment of AND postoperatively Bowel bowel leak within Need for: [NGT 30 days of the placement OR NPONPO > 4 days NPO order after 72 Order historyLeakhours postoperativelyindex surgery status] IV Antibiotics given AND IV ABX after 24 hours MAR Need for IV antibiotics postoperatively ANDTPN access catheter1Need for TPNTPN placement LDA flowsheetDiagnosis or Within 5 years of Bladder neck Any procedures ordered postoperative surgery: contracture CPT3within 5 years of surgery Bladder procedure for proc date Neck bladder neck Procedure for bladder Contractu contracture at any neck contracture Hospital problem lie time following OR Bladd st r er neck cal Diagnosis of bladdercontracICD-103OR index surgiture DXEncounter Diagnoses procedure neck contractureAttorney docket # 047162-5363-00WO Within 30 days of Bowel injury Hospital problem list Diagnosis or surgery:DXICD-103OR procedural Encounter Diagnoses Bowel management of Diagnosis of bowel Injury bowel injury injury Any procedures o 30 days of OR Rectal rdered within Injury3within 30 days of s Need for procedure forProCPTurgery index surgerycdate rectal injuryWithin 30 days of Internal case request surgery: Cardiac cath code IDs: CATH01, Case request orders CATH69 Cardiac Need for cardiac Elev > 3 times the upper Cardiac ca ated morbid theterization normal limit of Laboratory flowsheet Complica ity within troponins 30 days OR Troponin lab values tions of the index surgery Critical elevation in troponin enzymes C Hospital problem list OR ardiac DXICD-103OR Diagnosis of CardiacflagsEncounter Diagnoses Complication Within 30 days of C. diff colitis surgery: Abnormal results for C-Diff within 30 days of Positive dia C-diff labs any of the C. difficile Laboratory flowsheet the index surgery gnostic laboratory results for c. laboratory assays diff colitis Within 30 days of surgery: Renal Need for inpatient replacement hemodialysis ialysis procedure within ORInternAny procedures ordered Dal code IDs forNeed for continuousHemodialysis diawithin 30 days of surgery 30 days of thelysis and CVVHveno-venous date index surgery hemofiltration (CVVH) or continuous renal replacement therapy (CRRT)Pre-Op: Diagnosis of erectile dysfunction prior to date of prostatectomy Erectile OR Erectile dysfunction Post-Op: Diagnosis of Erectile Hospital problem list Dysfuncti diagnosed before erectile dysfunction 6 Dysfunction ICD-103OR on or after months or greater after DX Encounter Diagnoses prostatectomy date of prostatectomy AND De Novo: If present >6 months post-op but not present pre-op Patient mortality Mortality within 30 days of the index N / A Mortality N / A Patient Registry Data surgeryAttorney docket # 047162-5363-00WO Diagnosis or Within 30 days of Lymphoc Hospital problem list procedural surgery: ele DXICD-10: I89.8OR Lymphoc management of Encounter Diagnoses ele lymphocele Diagnosis of post- procedural lymphocele Sclerotherapy in within 30 days of Sclerotherapy procedure display ***Encounter index surgery OR proc procedures*** Need for sclerotherapy name Presence of additional surgeri Within 30 days of OR es performed withi surgery:Presence of a non-Return n Return to ORcanceOR logs 30 dayslled OR logfollowing the Return to OR index surgery Procedural Within 30 days of Pelvic intervention for surgery: Fluid pelvic fluid within Need for percutaneousDraina 3 ***EncounterCollectio collectionge proc CPTprocedures***n 30 days of index drainage of pelvic fluid surgery collection Within 30 days of Pneumon Pneumonia surgery: Hospital problem list within 30 days of Pne3ia umonia DXICD-10 OR index surgery Diagnosis of Encounter Diagnoses pneumonia Need for bowel Within 30 days of NGT placement after rest or surgery: NGT > 4 days 721Prolonge hours LDA flowsheet d nasogastric tube postoperatively NGT / NP placement within Need for NGT O 30 days placement following index ORNPO > 4 days NPO order after 72hours postoperativelyOrder historysurgeryNeed for NPO statusPresence of additional encounters for Within 30 days of hospital surgery: admission Readmiss following Urgent or Emergent Elective ion discharge from hospital admission Readmission Hospitalizations Admission History the index surgical following discharge excluded encounter within from index surgical 30 days of the encounter index surgery date Within 30 days of Hospital problem list sis or surgery: AKI DX3Diagno ICD-10 OR laboratory Encounter Diagnoses Renal evidence of renal Di Creatinine increase of Insufficie agnosis of acute insufficiency k 0.3mg / dl or 50% from ncy idney injury within 30 days of OR Creatinine Rise previous over a 481evidence of period at an LDA flowsheet index surgery laboratory y point renal dysfunction within 30 days of the index surgeryAttorney docket # 047162-5363-00WO Hospital problem list n 30 days of Sepsis DX I3Withi CD-10 OR surgery: Encounter Diagnoses When at least 2 of the Diagnosis of sepsis following conditions OR are met: [Within a 5-day Temp < 96.8 or > Sepsis within 30 Vitals and laboratory period SIRS 100.4 Sepsis days of the index : flowsheet Patient Heart Rate > 90 surgery meets SIRS criteria Resp Rate > 20 AND WBC < 4000 or WBC Patient has an > 12000 abnormal respiratory, Within 5 days of blood, urine, abscess, Abnormal surgery: or wound culture] Culture Resp, blood, urine, Laboratory flowsheet abscess, or wound cultures Additional intubations / airwa y access device Unplanne placements Within 30 days after erformed within surg Placement of: d p ery: Airway endotracheal tube, LDA flowshe1Intubatio 30 days of the lar et An airwa yngeal mask airway, n index surgical y access device is pl or tracheostomy tube encounter aced following initial extubation Diagnosis of With Hospital problem list ureteral in 30 days of Obstruction3bstruction or surgICD-10OR o ery:DXEncounter Diagnoses Ureteral placement of Obstructi ureteral Diagnosis of ureteral on stent / nephrostom obstruction y tube within 30 OR Stent / NT Proc CPT3Encounter Procedures days following Renal decompression the index surgery procedure is performed Pre-Op: Diagnosis of urinary incontinence prior to date of prostatectomy Urinary OR Urinary Incontinence Post-Op: Diagnosis of contine diagnosed before erectile dy Urinary Hospital problem list In sfunction 63months o Incontinence ICD-10 OR nce or after r greater after the dat DX Encounter Diagnoses prostatectomy e of prostatectomy AND De Novo: If present >6 months post-op but not present pre-op Urine Detection of Within 60 days of Internal code IDs for Leak urine leak duringsurgery:Cystogramcystograms Encounter ProceduresAttorney docket # 047162-5363-00WO routine postoperative Presence of two follow up within Cystogram orders such 30 days of index that the second surgery Cystogram order was at least 1 day after the first Internal code IDs for OR Fluoro outpatient fluoroscopy Encounter Procedures Presence of two procedures Fluoroscopy orders such that the second Fluoro order was at least 1 day after the first Urinary tract Within 30 days of surge UTI infection 30 ry: Abnormal Critical value for urine within days of urine culturecuLaboratory flowsheetPatient has a positilturethe index surgery ve urine culture Cumulative recorded time for mechanical >48hr on ventilation Ventilato exceeding 48hr LDA flowsh1within 30 eet r2days of the index surgery (does not include the index surgery itself) Length of hospital stay (in LOS days) for the Encounter Data index surgical encounter Table 3: Tracked Non-Oncologic Outcomes Data Validation and Statistical Analysis

[0101] During development, data retrieval was evaluated via iterative EMR chart audits. Internal and external validation of the algorithm was performed through iterative 15% sampling audit of the EMR for procedures performed at the hospital system main delivery network between January 2013-May 2023 and procedures performed at the hospital system outer delivery networks between September 2018-April 2023, respectively. Matched cases between the AASODA and NSQIP data abstraction for procedures performed at all delivery networks between October 2015-December 2022 were used to compare sensitivity, specificity, and IRR in a sample that included all cases identified by the AASODA or NSQIP as having an event and a random sampling of negative cases to total 15%.Attorney docket # 047162-5363-00WO

[0102] For stage data, 160 RALP procedures were randomly selected. For margin data, all 141 procedures reviewed by NSQIP were selected and assessed for EMR concordance. The algorithm was optimized through inclusion of new coding logic including addition / subtraction of ICD-10 and CPT codes. Modifications and audits were iteratively performed until sensitivity and specificity met the a priori-defined acceptable threshold of >90% or were as optimized as possible. Sensitivity and specificity were calculated using Microsoft ExcelTM(Microsoft Corporation, Redmond, WA).

[0103] Inter-rater reliability between the AASODA and the NSQIP data abstraction team was assessed for all mutually tracked outcomes using the Cohen’s kappa statistic. Interpretation of Cohen’s Kappa statistic was guided by conventional criteria of agreement (0-0.19: slight, 0.2- 0.39: fair, 0.4-0.59: moderate, 0.6-1.0: substantial) [see Blackman, N. J. M., et al., Statistics in medicine, 2000, 19(5), 723-741]. Statistical significance was set at p<0.05. Cohen’s kappa analyses were performed using SPSSTM (International Business Machines CorporationTM, Armonk, New York). Outcomes that were not defined by both systems in compatible terms were excluded from the inter-rater reliability analysis. These included pelvic fluid collection, LOS, lymphocele, and anastomotic bowel leak. When no events were detected by either NSQIP or AASODA for a given outcome, inter-rater reliability could not be assessed. These included: > 48 hours of mechanical ventilation, unplanned re-intubation, cardiac complications, and bowel injury. NSQIP data were not available for the following: LOS, readmission within 30 postoperative days, return to the OR within 30 postoperative days, and bladder neck contracture. NSQIP did not begin tracking prolonged use of nasogastric tube (NGT) or nil per os (NPO), ureteral obstruction, or urine leak until January 2019. The only pathology outcomes tacked by both systems were surgical margin status and stage. However, surgical margins were not tracked by NSQIP until May 2022. Results

[0104] 1586 RALP procedures between January 2013-May 2023 were identified by the AASODA. Institutional NSQIP reports identified 997 RALP procedures between October 2015- December 2022. Of these 997 NSQIP-identified RALP procedures, 70 procedures were appropriately excluded by the AASODA: 44 cystectomy cases, 2 open simple prostatectomies,Attorney docket # 047162-5363-00WO and 24 performed by surgeons not tracked by the AASODA.927 RALP procedures were tracked by both NSQIP and AASODA.

[0105] Cohen’s Kappa statistic for assessed outcomes ranged between 0.00-1.00 (Table 4). Inter- rater reliability (IRR) was substantial for urine leak (0.60; 95% CI: 0.43, 0.76), Clostridium difficile colitis (0.67; 95% CI: 0.05, 1.28), pneumonia (0.80; 95% CI: 0.42, 1.19), surgical margin status (0.94; 95% CI: 0.86, 1.02), and mortality (1.00; 95% CI: 1.00, 1.00). IRR was 0 for dialysis and ureteral obstruction.

[0106] Sensitivity for both the AASODA and NSQIP abstraction in matched case analysis ranged from 0.00-100% (Table 4). Sensitivity was greater for the AASODA compared to NSQIP for pneumonia, prolonged NGT / NPO, renal insufficiency, sepsis, surgical margins, stage, urine leak, and UTI. Conversely, sensitivity was less for the AASODA for rectal injury. Specificity for NSQIP ranged from 95.8-100% and for the AASODA ranged from 97.9-100% (Table 4). Specificity was greater in the AASODA for prolonged NGT / NPO, rectal injury, ureteral obstruction, and UTI and equal for all remaining outcomes. Specificity was equal between the AASODA and NSQIP abstraction for sepsis (99.36%) and equal at 100% for the following outcomes: cardiac arrest / CPR, Clostridium difficile colitis, dialysis, mortality, pneumonia, renal insufficiency, surgical margins, stroke, and urine leak. Incidence of each outcome since implementation of the EMR was also examined (Table 4). Sensitivity and specificity of the AASODA (Table 5) and NSQIP abstraction (Table 6) in non-matched cases demonstrated similar superiority of the AASODA in sensitivity and specificity. Table 2 also demonstrates the performance of variables tracked by the AASODA but not NSQIP. Sensitivity and specificity of operative time and pathology variables tracked only by the AASODA were 100% except core count (sensitivity 68.9%).

[0107] Inter-rater reliability analysis was performed for all eligible outcomes tracked by both systems, with the results shown in Table 4 below. With reference to Table 4, *** denotes that there were no events of this outcome, † denotes that the result was not calculated due to incompatible definitions, and †† denotes a non-dichotomous variable, and a correct identification rate was calculated instead of sensitivity and specificity.Attorney docket # 047162-5363-00WO Cohen’s Outcome Sensitivity Specificity PPV NPV Weighted Kappa (95% CI) AASODA NSQIP AASODA NSQIPAASODAASOD NSQI ANSQIPAP0.04 ( age †† 100% 94.20% 100% 83.30% 100% 99.30% 100%35.- St70% 0.03, 0.10) Surgical 8 0.94 Margins98.40% 80.30% 100% 100% 100% 100% 98.70%5.20 % (0.86- 1.02) Prolonged 95 0.39NGT / NPO100% 38.50% 100% 99.40% 100% 83.30% 100%.10 % (0.11, 0.66) Rectal Injury0% 100% 100% 99.40% *** 66.70% 98.80% 100% †Ureteral 0.00Obstruction*** *** 100% 99.40% *** 0% 100% 100%(0.00, 0.00) rine Leak 90.00% 60.00% 100% 100% 100% 100% 98.60%90.60U4.60% (0.43, 0.76) CardiacArrest / CPR*** *** 100% 100% *** *** 100% 100% †C-Diff 0.67 Infection100% 100% 100% 100% 100% 100% 100% 100%(0.05, 1.28) 0.00 Dialysis *** *** 100% 100% *** *** 100% 100% (0.00,0.00 ) 1.00 Mortality 100% 100% 100% 100% 100% 100% 100% 100% (1.00, 1.00) ia 100% 0% 100% 100% 100% *** 100%99.0.80Pneumon40% (0.42, 1.19) Renal nsufficienc100% 16.10% 100% 100% 100% 100% 100%70.32 I6.00% (0.19, y 0.44) .00% 25.00% 99.40% 99.40% 95.00% 83.30% 99.40%91.20.28 Sepsis 950% (0.06, 0.50) 0.28 Stroke *** *** 100% 100% *** *** 100% 100% (0.06, 0.50) 97.90% 95.90% 90.90% 75.00% 97.90%900.50 UTI 90.90% 54.60%.30% (0.34, 0.65) Table 4: Sensitivity, Specificity, and Cohen’s kappa of the AASODA and NSQIP and Incidence of Outcomes in Matched CasesAttorney docket # 047162-5363-00WO Incidence Outcome Sensitivity Specificity PPV NPV(Jan.2013- May 2023) Core Count Examined †69.00% † † † 100% reportedCore Count Positive †69.00% † † † 100% reportedExtraprostatic Extension100% 100% 100% 100% 100% reportedGleason Grade Group †100% † † † 100% reportedLymph Nodes Examined †100% † † † 100% reportedLymph Nodes Positive †100% † † † 100% reportedLymphovascular Invasion100% 100% 100% 100% 100% reportedPerineural Invasion100% 100% 100% 100% 100% reportedPositive Surgical Margins 97.10% 100% 100% 83.30% 37.00%Seminal Vesicle Invasion100% 100% 100% 100% 100% reportedStage † 99.50% † † † 100% reported Bladder Neck Contracture88.90% 100% 100% 100% 2.90%Erectile Dysfunction 95.30% 100% 100% 95.10% 16.10%(pre-op) Erectile Dysfunction 74.10% 100% 100% 65.00% 40.30%(post-op) Erectile Dysfunction 82.20% 91.40% 92.50% 80.00% 33.40%(De Novo) Operative Time 100% † † † 100% reported Unplanned Intubation100% 100% 100% 100% 1.40%Urine Leak 93.80% 100% 100% 99.20% 8.20%Urinary Incontinence 97.90% 95.30% 96.90% 96.80% 3.30%(pre-op) Urinary Incontinence 93.20% 92.90% 98.40% 74.30% 17.40%(post-op) Urinary Incontinence 92.40% 89.30% 97.60% 71.40% 16.70%(De Novo)Attorney docket # 047162-5363-00WO Ureteral Obstruction100% 100% 100% 100% 0.40%Ventilator >48h 100% 100% 100% 100% 0.20% Anastomotic Bowel Leak*** 100% *** 100% 0%Cardiac Arrest / CPR100% 100% 100% 100% 0.30%C-Diff Infection 100% 100% 100% 100% 0.20% Dialysis 100% 100% 100% 100% 0.20% Mortality 100% 100% 100% 100% 0.30% Pneumonia 100% 100% 100% 100% 0.70% Prolonged NGT / NPO100% 100% 100% 100% 2.60%Rectal Injury 33.33% 100% 100% 99.30% 0.20%Renal Insufficiency 100% 100% 100% 100% 6.00% Sepsis 96.00% 100% 96.00% 99.70% 2.20%Stroke *** 100% *** 100% 0% UTI 89.70% 98.60% 92.10% 98.10% 3.70%Table 5: Sensitivity, Specificity, PPV, NPV, and Incidence of Outcomes in the AASODA in All Audited Cases Outcome Sensitivity Specificity PPV NPVStage† 80.30% † † † Surgical Margins 95.20% 100% 96.40% 85.90%Prolonged NGT / NPO 63.20% 99.40% 92.30% 95.70%Rectal Injury 100% 99.40% 75.00% 100%Ureteral Obstruction*** 98.80% *** 98.80%Urine Leak 61.90% 100% 100% 94.60% Cardiac Arrest / CPR100% 100% 100% 100%C-Diff Infection100% 100% 100% 100%Dialysis *** 100% *** 100% Mortality 100% 100% 100% 100% Pneumonia 50.00% 92.30% 50.00% 99.40%Renal Insufficiency22.40% 100% 100% 76.60%Sepsis 42.30% 99.40% 91.70% 91.90%Stroke *** 100% *** 100% UTI 64.30% 96.40% 81.80% 91.50%Table 6: Sensitivity and Specificity of NSQIP in All Audited CasesAttorney docket # 047162-5363-00WO

[0108] With reference to Table 5 and Table 6, *** denotes that there were no events of that outcome and † denotes non-dichotomous variable, and a correct identification rate was calculated instead of sensitivity and specificity. Discussion

[0109] Persistent variation in prostate cancer surgery outcomes as well as recent inclusion of these outcomes in national hospital ratings demonstrate need for institutional QI efforts to optimize prostatectomy outcomes. NSQIP has facilitated surgical QI for nearly two decades; however, NSQIP services are limited by rigidly defined, less procedure-specific outcomes, and high overhead costs. More accessible registries for Urology practices, such as AQUA, rely on administrative data and currently lack granular clinical outcomes data to adequately support local procedure-specific QI initiatives. This example presented AASODA, a novel, EMR-based automated algorithm-driven institutional dashboard for NSQIP-defined surgical outcomes following RALP that achieves >90% sensitivity for all variables except rectal injury and met or exceeded the specificity of the institutional NSQIP-abstractors for all variables. Additionally, AASODA extracts variables not tracked by NSQIP with similar sensitivity and specificity, making this valuable data more accessible. Moreover, the AASODA demonstrated substantial agreement with NSQIP across most variables, supporting it as an alternative to the constrained processes of NSQIP without compromising data quality.

[0110] For the outcomes where agreement between AASODA and NSQIP was lower, such as dialysis and ureteral obstruction, audits favored the AASODA as more accurate; however, the low incidence of these events prevents a more accurate comparison. Discrepancies for UTI and rectal injury were largely due to cases of treatment at outside hospitals and reporting of diagnoses only in note text that is currently inaccessible to AASODA given inability to perform natural language data processing (NLP). Similarly, low sensitivity of the algorithm for ED and SUI is driven by reporting these outcomes only in note text without use of appropriate ICD-10 codes. The low sensitivity of the algorithm for core count represents a subset of pathology notes that are unreadable to the algorithm, either because they are a scanned image or are in a note template not compatible with the algorithm’s code. Sensitivity was 100% for core count in pathology notes readable by the algorithm. Low sensitivity of NSQIP abstraction for prolongedAttorney docket # 047162-5363-00WO NGT / NPO, renal insufficiency, sepsis, and UTI is likely driven by difference in what a reviewer sets as a threshold for these outcomes and a less strict application of the outcome definitions. Moreover, the algorithm can examine all lab reports, flowsheets, and ICD-10 and CPT codes instantly which likely represents a more thorough review than is feasible for an abstractor.

[0111] Other studies have also demonstrated discrepancies between local databases and NSQIP. Epelboym et al found pancreatectomy outcomes were misclassified by NSQIP in 29.3% of cases. [Epelboym, I., Gawlas, I., Lee, J. A., et al. World Journal of Surgery 2014; 38: 1461-1467] Comparison of NSQIP-Pediatric to an institutional database for appendectomy cases by Anderson et al demonstrated significant differences in rates of emergency room visits, readmissions, and composite morbidity between the two sources. [Anderson, K. T., Bartz- Kurycki, M. A., Austin, M. T., et al. Journal of Pediatric Surgery 2019; 54(1): 97-102.] While no clear cause for these discrepancies was identified, the risk of variation in outcomes reporting is likely increased by the inherent subjectivity of individual data abstraction.

[0112] Among the most significant advantages of the AASODA is its resource efficiency with potential to substantially reduce costs of institutional QI initiatives. A recent review of healthcare quality reporting found annual cost of electronically tracked data, per metric, was < 1% of that incurred by claims-based or chart-abstracted metrics representing possible savings of millions of dollars for a single hospital. Moreover, the number of personnel hours required to implement and maintain automated electronic data collection was similarly minute. [Saraswathula, A., Merck, S. J., Bai, G., et al. Journal of the American Medical Association 2023; 329(21): 1840-1847] In this context, the time investment in developing this dashboard and subsequent upkeep likely ranges in the hundreds of hours compared to the over 100,000 hours annually spent on data extraction and validation at the aforementioned study’s single hospital. While cost was not evaluated in this study, it is likely to be nominal compared to the reported cost of claims-based and chart- abstracted metrics.

[0113] The disclosed procedure-specific automated EMR data extraction algorithm capitalizes on the rich store of data in the EMR to remove the burden of individual abstractors. Regional clinical data registries, such as MUSIC and PURC, require institutional data abstractors and local surgeon champions which many hospitals may not be able to afford. In contrast, AQUA is a toolAttorney docket # 047162-5363-00WO available to Urology practices that utilizes automated data abstraction and fulfills Centers for Medicare & Medicaid Services (CMS) Merit-Based Incentive Payment System (MIPS) reporting which could magnify the scope of our algorithm. However, AQUA users cannot define their own outcomes, many of the AQUA outcomes are claims-based to comply with Medicare quality benchmarking standards, and many current users use multiple EMRs in their practices which may not inter-communicate. [American Urological Association: AUA Quality Registry. Available at: auanet.org / research-and-data / aua-quality-(aqua)-registry. Retrieved August 30, 2023] Moreover, the granularity of the data provided by automated algorithms such as the disclosed example will continue to improve if ICD-10 and CPT codes become more specific and EMR intermediate variables are standardized. Additionally, the scope of algorithm-abstracted data can expand through addition of new logic while also immediately retroactively applying to past cases, providing flexibility that individual EMR abstraction lacks. This extracted data can then be risk adjusted against national benchmarks, similar to NSQIP’s practice. The utility of the disclosed algorithm as an adjunct to AQUA is evidenced by the AASODA’s excellent specificity and flexibility.

[0114] AASODA possesses key limitations to consider. While EPICTMis the most widely used EMR with a 35% market share [Sorace, J., Wong, H., DeLeire, T., Xu, D., Handler, S., Garcia, B., & MaCurdy, T. International Journal of Informatics 2020; 136], the algorithm has only been tested with EPICTMwhich limits the algorithm’s applicability. The algorithm’s accuracy is also limited by the accuracy of intermediate variables used in the code and errors in CPT or ICD-10 code entry. Additionally, some EMR intermediate variable flowsheets may be institution- specific. This limitation may be circumvented if updated EMR versions include standardized flowsheets as part of its universal build. Moreover, the algorithm has not yet been validated outside the healthcare system used in this example. In some embodiments, AASODA does not perform machine learning text extraction such as NLP, and instead uses explicit trigger words and text manipulation to extract textual data from templated notes manually, or uses a combination of NLP and manual extraction. Thus, diagnoses in a radiology report or outside hospital note that might be missed by the algorithm but easily identified by an abstractor. In fact, nearly all incorrectly identified outcomes by the AASODA may be remedied with NLP. Additionally, IRR between the AASODA and NSQIP varied widely which may reflect the limitations of Cohen’s kappa: Cohen’s kappa conservatively estimates agreement due to itsAttorney docket # 047162-5363-00WO assumption that random allocation of responses comprises some portion of the agreement between two ratings. In the disclosed dataset, the much greater number of negative events causes a larger estimation of random allocation, thereby decreasing apparent agreement.

[0115] Overall, AASODA’s success represents the significant potential of automated clinical data extraction. If externally validated, inter-institutional reliability can be assessed and drive algorithm optimization. Applications of the disclosed algorithm can also target different surgical procedures, outcomes of interest, and patient populations. Benchmark outcomes can be established and collected for centralized processing and comparative performance feedback through outside institution participation. Ultimately, improved access to accurate and reliable clinical database services through automated data extraction could facilitate cost-effective QI initiatives nationwide.

[0116] Automated algorithms for clinical data extraction offer the opportunity to produce efficient, high-reliability quality metric databases. AASODA, a novel automated algorithm- driven institutional dashboard for RALP surgical outcomes and quality metrics, achieves >90% sensitivity and specificity for all variables except rectal injury, and met or exceeded the specificity of the institutional NSQIP-abstraction for all variables. Substantial agreement between the AASODA and NSQIP supports automated EMR data extraction as a tenable replacement for trained abstractors. AASODA provides high resource efficiency, real-time event reporting, and ability to integrate with existing EMRs. Applying AASODA to more hospital settings and procedures provides a promising opportunity to facilitate, standardize, and reduce the cost of local and national outcomes and quality metric benchmarking.

[0117] The disclosures of each and every patent, patent application, and publication cited herein are hereby incorporated herein by reference in their entirety. While this invention has been disclosed with reference to specific embodiments, it is apparent that other embodiments and variations of this invention may be devised by others skilled in the art without departing from the true spirit and scope of the invention. The appended claims are intended to be construed to include all such embodiments and equivalent variations. ReferencesAttorney docket # 047162-5363-00WO

[0118] The following publications are incorporated herein by reference in their entirety:

[0119] Pooli, A., Salmasi, A., Johnson, D. C., Lenis, A. T., Faiena, I., Lebacle, C., ... & Chamie, K. (2020, January). Positive surgical margins at radical prostatectomy in the United States: Institutional variations and predictive factors. In Urologic Oncology: Seminars and Original Investigations (Vol.38, No.1, pp.1-e17). Elsevier.

[0120] Gore, J. L., Wright, J. L., Daratha, K. B., Roberts, K. P., Lin, D. W., Wessells, H., & Porter, M. (2012). Hospital‐level variation in the quality of urologic cancer surgery. Cancer, 118(4), 987-996.

[0121] Herrel, L. A., Syrjamaki, J. D., Linsell, S. M., Miller, D. C., & Dupree, J. M. (2016). Identifying drivers of episode cost variation with radical prostatectomy. Urology, 97, 105-110.

[0122] U.S. News & World Report, L.P. (2022). Methodology U.S. News & World Report 2022-23 Best Hospitals: Specialty Rankings. Retrieved March 7, 2023 from U.S. News & World Report, L.P.:

[0123] Fink AS, Campbell DA Jr, Mentzer RM Jr, Henderson WG, Daley J, Bannister J, Hur K, Khuri SF. The National Surgical Quality Improvement Program in non-veterans administration hospitals: initial demonstration of feasibility. Ann Surg.2002 Sep;236(3):344-53; discussion 353-4. doi: 10.1097 / 00000658-200209000-00011. PMID: 12192321; PMCID: PMC1422588.

[0124] Eisenstein, S., Stringfield, S., & Holubar, S. D. (2019). Using the National Surgical Quality Improvement Project (NSQIP) to perform clinical research in colon and rectal surgery. Clinics in colon and rectal surgery, 32(01), 041-053.

[0125] American College of Surgeons National Surgical Quality Improvement Program®. (2022). User Guide for the 2021 Procedure Targeted Participant Use Data File (PUF). Retrieved March 7, 2023 from American College of Surgeons National Surgical Quality Improvement Program®.: facs.org / quality-programs / data-and-registries / acs-nsqip / participant-use-data-file /

[0126] Montie, J. E., Linsell, S. M., & Miller, D. C. (2014). Quality of care in urology and the Michigan Urological Surgery Improvement Collaborative. Urology Practice, 1(2), 74-78.Attorney docket # 047162-5363-00WO

[0127] Neuman, H., Michelassi, F., Turner, J., & Bass, B. (2009) Surrounded by quality metrics: what do surgeons think of ACS NSQIP?. Surgery, 145(1): 27-33.

[0128] Eastham, J., Kattan, M., Riedel, E., Begg, C., Wheeler, T., Gerigk, C., Gonen, M., Reuter, V., & Scardino, P. (2003). Variations among individual surgeons in the rate of positive surgical margins in radical prostatectomy specimens. Journal of Urology, 170(6 Pt 1):2292-5

[0129] Stey, A. M., Russell, M. M., Ko, C. Y., Sacks, G. D., Dawes, A. J., & Gibbons, M. M. (2015). Clinical registries and quality measurement in surgery: a systematic review. Surgery, 157(2), 381-395.;

[0130] Lawson, E., Louie, R., Zingmond, D., Brook, R., Hall, B., Han, L., Rapp, M., & Ko, C. A comparison of clinical registry versus administrative claims data for reporting of 30-day surgical complications. Annals of Surgery, 256 (6), 973-81.

[0131] Harvey, G., & Wensing, M. (2003). Methods for evaluation of small scale quality improvement projects. BMJ Quality & Safety, 12(3), 210-214.

[0132] Reese, A. C., & Ginzburg, S. (2021). The past, present, and future of urological quality improvement collaboratives. Translational Andrology and Urology, 10(5), 2280.

[0133] Health Care Improvement Foundation: Pennsylvania Urologic Regional Collaborative. Available at: https: / / hcifonline.org / purc /

[0134] American Urological Association: AUA Quality Registry. Available at: https: / / www.auanet.org / research-and-data / aua-quality-(aqua)-registry

[0135] Saraswathula, A., Merck, S. J., Bai, G., Weston, C. M., Skinner, E. A., Taylor, A., ... & Berry, S. A. (2023). The Volume and Cost of Quality Metric Reporting. JAMA, 329(21), 1840- 1847.

[0136] Atasoy, H., Greenwood, B. N., & McCullough, J. S. (2019). The digitization of patient care: a review of the effects of electronic health records on health care quality and utilization. Annual review of public health, 40, 487-500.Attorney docket # 047162-5363-00WO

[0137] Blackman, N. J. M., & Koval, J. J. (2000). Interval estimation for Cohen's kappa as a measure of agreement. Statistics in medicine, 19(5), 723-741.

[0138] Epelboym, I., Gawlas, I., Lee, J. A., Schrope, B., Chabot, J. A., & Allendorf, J. D. (2014). Limitations of ACS-NSQIP in reporting complications for patients undergoing pancreatectomy: underscoring the need for a pancreas-specific module. World journal of surgery, 38, 1461-1467.

[0139] Anderson, K. T., Bartz-Kurycki, M. A., Austin, M. T., Kawaguchi, A. L., Kao, L. S., Lally, K. P., & Tsao, K. (2019). Room for “quality” improvement? Validating National Surgical Quality Improvement Program-Pediatric (NSQIP-P) appendectomy data. Journal of pediatric surgery, 54(1), 97-102.\

[0140] Sorace, J., Wong, H., DeLeire, T., Xu, D., Handler, S., Garcia, B., & MaCurdy, T. (2020). Quantifying the competitiveness of the electronic health record market and its implications for interoperability. International Journal of Informatics, 136.

Claims

Attorney docket # 047162-5363-00WO CLAIMS What is claimed is:

1. A method for automated determination of a surgical outcome, comprising: acquiring data comprising diagnosis codes and text notes related to a subject who underwent a surgery from a medical record database; selecting at least one surgical outcome from a set of surgical outcomes based on the diagnosis codes; searching the text notes for at least one keyword from a set of keywords; when a first keyword from the set of keywords is found in the text notes, validating the selected surgical outcome; when a second keyword from the set of keywords is found in the text notes, selecting at least one additional surgical outcome from the set of surgical outcomes; and adding the at least one surgical outcome to a database of surgical outcomes indexed to the subject and the surgery.

2. The method of claim 1, wherein the diagnosis codes comprise CPT codes or ICD codes.

3. The method of claim 1, wherein the surgery is a robot-assisted laparoscopic radical prostatectomy.

4. The method of claim 1, wherein the step of searching the text notes for at least one keyword comprises searching for multiple keywords using at least one Boolean operator selected from AND and OR.

5. The method of claim 1, wherein the step of searching the text notes for at least one keyword comprises searching for multiple keywords using add-on codes or modifiers for diagnosis codes.

6. The method of claim 5, wherein the add-on codes or modifiers comprise addition or subtraction of diagnosis codes.Attorney docket # 047162-5363-00WO 7. The method of claim 1, wherein the step of searching the text notes for at least one keyword comprises using natural language processing (NLP) to analyze the text notes.

8. The method of claim 1, wherein the step of searching the text notes for at least one keyword comprises the step of selecting the at least one keyword from the set of keywords based on which provider supplied the text notes.

9. The method of claim 1, wherein the step of validating the selected surgical outcome comprises a comparison to one or more institutional benchmarks or quality metrics.

10. The method of claim 9, wherein the one or more one or more institutional benchmarks or quality metrics comprise any of National Surgical Quality Improvement Program (NSQIP), Positive predictive value (PPV), and negative predictive value (NPV).

11. A system for automated determination of a surgical outcome, comprising: a non-transitory computer readable medium with instructions stored thereon, which when executed by a processor, performs steps comprising the method of any of claims 1-6.

12. The system of claim 11, further comprising a display device, the instructions further comprising presenting a graphical user interface comprising data from the database of surgical outcomes indexed to the subject and the surgery.

13. The system of claim 11, wherein the graphical user interface is configured to display one or more graphs of the data over time.

14. The system of claim 11, wherein the graphical user interface is configured to display one or more quality scores or metrics associated with the data.Attorney docket # 047162-5363-00WO 15. The system of claim 11, wherein the database of surgical outcomes is further indexed to one or more providers associated with the subject and the surgery.

16. The system of claim 15, wherein the graphical user interface is configured to display one or more graphs of surgical outcomes over time for the one or more providers.

17. The system of claim 16, wherein the graphical user interface is configured to display one or more quality scores or metrics associated with one or more providers.

18. The system of claim 15, wherein the database of surgical outcomes is further indexed to one or more institutions associated with the one or more providers.

19. The system of claim 18, wherein the graphical user interface is configured to display one or more graphs of surgical outcomes over time for the one or more institutions.

20. The system of claim 19, wherein the graphical user interface is configured to display one or more quality scores or metrics associated with one or more institutions.

Images