AI-enabled echo confirmation workflow environment

Abstract

In an echocardiogram analysis method, a diagnostic code for an echocardiogram including a set of echocardiogram loops is received or generated. A plurality of different subsets are selected from the echocardiogram. Each subset consists of one or more echocardiogram loops from the set of echocardiogram loops. For each subset, a confidence score indicating a relevance of the subset to the diagnostic code is determined using an artificial intelligence (AI) engine operating on the subset, wherein the AI engine is trained on historical echocardiograms labeled with the diagnostic code. Relevant echocardiogram loop groups are identified based on the determined confidence scores for the respective subsets indicating a relevance of the respective subsets to the diagnostic code. An echocardiogram reading user interface is presented including displaying the diagnostic code associated with the echocardiogram loops of the relevant echocardiogram loop groups.

Description

TECHNICAL FIELD

[0001] The following generally relates to the field of cardiology, the field of echocardiography, the field of cardiac imaging, and related fields. BACKGROUND

[0002] Echocardiography is a common imaging modality for cardiac disease. Compared to general radiology, the number of diagnoses that can be derived from echocardiography is relatively small. From an echocardiography study, approximately 50 diagnoses can be derived. In an echocardiography exam, a technician acquires echocardiography loops, which are short time sequences of ultrasound images, typically spanning one or a few heartbeats. These are acquired at different positions (i.e., "views") of the ultrasound transducer (relative to the heart). There are several standard views (e.g., approximately 8), and for each standard view, various cardiology panels can identify up to several dozen sub-views. Each loop is typically acquired with an ultrasound transducer probe that is manually positioned by the ultrasound technician on the patient's torso at an angle relative to the heart that is appropriate for imaging the desired view / sub-view. The image quality of any given loop can vary significantly depending on the stability with which the technician holds the ultrasound transducer probe and the extent to which the patient can move during acquisition of the loop. In addition, the "quality" of a view depends on how accurately the technician positioned the ultrasound transducer probe to obtain that view. The technician can observe the acquired loops on a display of the ultrasound machine and can acquire multiple loops of a given view depending on the technician's subjective opinion of the loop image quality and how well the acquired loop visualizes the desired view.

[0003] The technician sometimes makes a preliminary report by reviewing the ultrasound loops and entering a diagnosis code or diagnosis language that is clinically easily understood. The technician is not a trained cardiologist and therefore will typically avoid making non-trivial diagnoses. The loops are stored with the metadata, such as the preliminary diagnosis, at a picture archiving and communication system (PACS), cardiovascular information system (CVIS), and / or other electronic database(s). At a later time, a cardiologist reviews the echocardiography, often using any preliminary diagnosis provided by the technician as a starting point.

[0004] Jeffrey Zhang et al., "Fully Automated Echocardiogram Interpretation in Clinical Practice: Feasibility and Diagnostic Accuracy," Circulation, vol. 138, no. 16, 16 October 2018 (2018-10-16), pp. 1623-1635, regarding automated cardiac image interpretation.

[0005] Certain improvements are disclosed below. SUMMARY

[0006] In some non-limiting illustrative embodiments disclosed herein, a non-transitory storage medium stores instructions readable and executable by an electronic processor to perform an echocardiogram analysis method including generating or receiving a diagnostic code for an echocardiogram comprising a set of echocardiogram loops. A plurality of different subsets are selected from the echocardiogram. Each subset is composed of one or more echocardiogram loops of the set of echocardiogram loops. For each subset, a confidence score is determined that indicates a relevance of the subset to the diagnostic code. The confidence score is determined using an artificial intelligence (AI) engine operating on the subset, wherein the AI engine is trained on historical echocardiograms labeled with the diagnostic code. Relevant echocardiogram loop groups are identified based on the determined confidence scores for the respective subsets that indicate a relevance of the respective subsets to the diagnostic code. An echocardiogram reading user interface is presented including displaying the diagnostic code associated with the echocardiogram loops of the relevant echocardiogram loop groups.

[0007] In some non-limiting illustrative embodiments disclosed herein, an echocardiogram analysis device includes an electronic processor and a non-transitory storage medium storing instructions readable and executable by the electronic processor to analyze an echocardiogram comprising a set of echocardiogram loops. The stored instructions include: AI engine instructions implementing an AI engine trained to output a confidence level for a diagnostic code in response to receiving an input set of one or more echocardiogram loops; relevant loop identification instructions implementing relevant loop identification including determining, by inputting different subsets of the set of echocardiogram loops to the AI engine, confidence scores for the different subsets for a preliminary diagnostic code, and identifying relevant echocardiogram loop groups based on the determined confidence scores for the respective subsets; and echocardiogram reading instructions implementing an echocardiogram reading user interface on a client system including displaying, on a display of the client system, the preliminary diagnostic code associated with the echocardiogram loops of the relevant echocardiogram loop groups.

[0008] In some non-limiting illustrative embodiments disclosed herein, an echocardiogram analysis method utilizing an Al engine trained to output a confidence level for a diagnostic code in response to receiving an input set of one or more echocardiogram loops is disclosed. The echocardiogram analysis method includes generating or receiving a preliminary diagnostic code for an echocardiogram comprising a set of echocardiogram loops, determining a confidence level for the preliminary diagnostic code for different subsets of the set of echocardiogram loops by inputting different subsets of all of the set of echocardiogram loops to the Al engine, and identifying a relevant set of echocardiogram loops related to the preliminary diagnostic code based on the determined confidence scores for the respective subsets, and presenting an echocardiogram reading user interface including displaying the preliminary diagnostic code associated with the echocardiogram loops in the relevant set of echocardiogram loops on a display of a client system.

[0009] One advantage resides in providing a more efficient user interface for echocardiogram interpretation environments.

[0010] Another advantage resides in providing a user interface for echocardiogram interpretation environments that reduces the number of echocardiogram loops that must be reviewed in order to formulate or confirm a given diagnostic code.

[0011] Another advantage resides in providing a user interface via which a cardiologist can confirm a previously generated diagnosis rather than creating a diagnosis from scratch.

[0012] Another advantage resides in providing a user interface for echocardiogram interpretation environments that reduces the number of less relevant or irrelevant loops that a cardiologist or other user considers in formulating or confirming a given diagnostic code.

[0013] Another advantage resides in providing an automated preliminary diagnostic capability for analyzing echocardiograms in echocardiogram interpretation environments.

[0014] Another advantage resides in providing a user interface for echocardiogram interpretation environments that reduces the likelihood of failing to consider relevant loops that are mislabeled as views (or sub-views) in making a clinical diagnosis.

[0015] A given embodiment can not provide the foregoing advantages, provide one, two, more than two, or all of the foregoing advantages, and / or can provide other advantages, as will become apparent to those of ordinary skill in the art upon reading and understood the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0016] The present application can take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for the purpose of illustrating preferred embodiments and are not to be construed as limiting the present application.

[0017] Figure 1 FIG. 1 diagrammatically illustrates an echocardiogram acquisition and reading system.

[0018] Figure 2 FIG. 1 diagrammatically illustrates a flowchart of a process for automatically analyzing one or more cycles of an echocardiogram to detect whether the cycle(s) indicate a certain diagnostic code, the process being suitably performed by the system of Figure 1 .

[0019] Figure 3 FIG. 1 diagrammatically illustrates a flowchart of a process for reviewing and confirming a diagnostic code during review of an echocardiogram, the process being suitably performed by the system of Figure 1 .

[0020] Figure 4 FIG. 1 diagrammatically illustrates a user interface display, the user interface display being suitably presented by the system of Figure 3 when performing the method of Figure 1 .

[0021] Figure 5 FIG. 1 diagrammatically illustrates a flowchart of a process for identifying or ranking cycles of an echocardiogram corresponding to a relevance to a diagnostic code, the process being suitably performed by the system of Figure 1 . DETAILED DESCRIPTION

[0022] Reference is made to Figure 1, an echocardiogram acquisition and reading system is diagrammatically shown. The illustrative system includes an echocardiogram acquisition device 10, which is typically a cardiac ultrasound imaging system (i.e., echocardiograph) 10, programmed to acquire echocardiogram loops (also referred to herein for brevity as "loops"), which are short time sequences of ultrasound images of the heart (and / or major heart vessels), typically spanning one or a few heartbeats. These are acquired at different positions (i.e., "views") of an ultrasound transducer probe (not shown) relative to the heart. Typically, each loop is acquired with the ultrasound transducer probe positioned on the patient's torso by an ultrasound technician at an angle relative to the heart appropriate to image the desired view / subview. The echocardiograph 10 includes at least one display 12, 14 (in a non-limiting illustrative example, a main display 12 for displaying ultrasound images or playing back ultrasound loops and a user interface display 14 for presenting operational parameter values, etc.) and one or more user input devices 16, such as a keyboard, touchpad, touch-sensitive overlay of one or both displays 12, 14, etc. In a typical echocardiogram study acquisition session, the ultrasound technician places the probe in the appropriate position on the patient's chest to acquire a view of the heart. The acquired echoes are played out on the display 12. If the technician is not satisfied with the loop, he or she can reposition the probe and / or ask the patient to hold still, and then acquire another loop that captures the view. This can be repeated multiple times for a given view, and / or the probe position can be adjusted to acquire various subviews of the view. Typically, all acquired loops are saved to the PACS 20 (and / or CVIS or other electronic database), unless they are clearly of non-clinical quality.

[0023] At the same time as the loops are acquired and / or after the loops have been acquired, the technician typically performs a preliminary analysis to arrive at preliminary findings and / or diagnoses. These are typically expressed using a standard notation in which a given finding or diagnosis is represented by a diagnosis code (also referred to herein for brevity as "code"). The diagnosis code is a unique identifier corresponding to a certain diagnosis or finding, which can include a disease and / or its severity (e.g., normal / mild / moderate / severe). The diagnosis code can also be qualified to express that a certain diagnosis cannot be reliably assessed, e.g., due to limited image quality. The preliminary diagnosis codes are stored in the PACS 20 in the appropriate echocardiogram study data packet along with the echocardiogram loops, and the preliminary diagnosis codes are subsequently confirmed, rejected, or modified by a cardiologist who reviews the echocardiogram and can also add additional diagnosis codes representing additional findings or diagnoses made by the cardiologist.

[0024] In a common implementation, the echocardiogram interpretation environment is hosted by a server computer 22, which can also optionally host the PACS 20 and / or other electronic databases storing echocardiogram loops. In general, the server computer 22 can be a single server computer (or a desktop computer or other computer with sufficient processing power) or a plurality of computers configured as a compute cluster, cloud computing resources, etc. The echocardiogram interpretation environment is accessed by client systems such as the illustrative echocardiogram machine 10, a cardiologist’s workstation 30, etc. The illustrative cardiologist’s workstation 30 includes a computer 32 having at least one display 34 and one or more user input devices such as a keyboard 36, a touchpad 38 (and / or a mouse and / or other pointing device), etc. The client systems 10, 30 are illustrative examples, and a typical cardiology department of a hospital or other medical facility can have dozens or more devices that are able to access the echocardiogram interpretation environment as client systems. In general, most of the data processing to implement the echocardiogram interpretation environment is performed at the server 22, although some less computationally intensive processes (e.g., rendering of user interface displays) and / or specialized processes (e.g., controlling the ultrasound hardware of the echocardiogram machine 10 to acquire echocardiogram loops) can be performed locally at the client systems 10, 30. The user interface of the echocardiogram interpretation environment presented on a given client system can depend on a number of factors such as the credentials of the client system (e.g., the user identity of the user logged into the client system, typically with an associated access / privilege level) and the capabilities of the client system (e.g., a user accessing the echocardiogram interpretation environment through the echocardiogram machine 10 can be provided with user interfaces supporting loop acquisition; whereas the cardiologist’s workstation 30 can not provide these loop acquisition user interface aspects). In general, the cardiologist’s workstation 30 will provide a user (e.g., a cardiologist) with access to any preliminary diagnosis codes entered by a technician at the echocardiogram machine 10. By way of non-limiting illustrative example, the echocardiogram interpretation environment can be the Philips IntelliSpace Cardiovascular (ISCV)™ echocardiogram interpretation environment provided by Koninklijke Philips N.V. (with additional features provided by as disclosed herein).

[0025] To perform a review of the preliminary diagnostic codes determined by the technologist, a cardiologist must review various acquired loops of echocardiograms to identify the most relevant loops for confirming or rejecting each preliminary diagnostic code. While loops can be annotated by view, and possibly also by subview, this is not sufficient for a cardiologist to efficiently and comprehensively select the best view for confirming a diagnostic code. Loop acquisition is a semi- manual process because the technologist manually places the ultrasound transducer probe on the outside of the patient's torso, and the image quality of any given loop depends on the precise placement of the transducer probe and the amount of motion blur (motion can be instability of the hand-held probe or patient motion or both). Thus, for any given view, the technologist can acquire several loops that can have different image quality and can or can not provide an ideal vantage point for the view (e.g. the transducer probe can record some loops at less than ideal positions for capturing the intended view). Furthermore, the view annotations assigned to individual loops can be in error. As a result, selecting the "best" view for confirming a given diagnosis is tedious, can be subjective, and can be prone to error.

[0026] Artificial intelligence (AI), such as a convolutional neural network (CNN) trained on echocardiograms annotated with diagnostic codes provided by readings performed by cardiologists, has the potential to improve echocardiogram readings. However, such AI can not be reliable enough to enable fully automated diagnosis of echocardiograms. Unless AI can perform fully automated diagnosis reliably enough, cardiologists will still need to review preliminary diagnostic codes (whether human or AI generated), and thus the above problems will not be alleviated. Given the life-critical nature of ensuring accurate echocardiogram readings, it can be impractical in many cases to replace the expertise of a cardiologist in reading echocardiograms with AI analysis of echocardiograms.

[0027] In view of the above, embodiments disclosed herein utilize AI analysis of echocardiograms in a manner that provides improved efficiency and accuracy of echocardiogram reading workflows without relying on AI to provide definitive diagnoses.

[0028] In one aspect disclosed herein, AI is used to generate preliminary diagnostic codes, instead of or in addition to preliminary analysis performed by a technologist. Optionally, the AI can also assign a confidence score to the generated diagnostic codes. Since these are only preliminary diagnostic codes, there is less concern about the possibility of errors by the AI because the preliminary diagnostic codes are reviewed by a skilled cardiologist.

[0029] In another aspect disclosed herein, the AI is used to select the most relevant loops for examination of the preliminary diagnosis code. In other words, starting from a preliminary diagnosis code (generated by a human (such as an ultrasound technician) or AI), the candidate loops are input to the AI, which outputs the diagnosis code (if indeed the AI associates the loop with the diagnosis code) and also outputs a confidence score for the diagnosis code. One or more of the metrics quantify the relevance of the loop to the diagnosis code. For example, if the code is output by the AI for a loop with a high confidence, the loop is highly relevant; however, if the code is not output by the AI at all, or is output by the AI but with a low confidence score, the loop is not very relevant.

[0030] Through such metrics, the loops of an echocardiogram can be ranked according to their relevance to a given diagnosis code. In one form of approach, a table of preliminary diagnosis codes is generated, and for each code, the relevant loops are represented as, for example, a set of at most N loops that are relevant to the code above some threshold. In some different embodiments, the relevant loop selection operates iteratively. For example, if there are 99 loops in an echocardiogram, each loop is initially input to the AI with a diagnosis code, and the most relevant loop to that diagnosis code is identified. Then, the most relevant loop is combined in turn with each of the remaining 98 loops to identify the second most relevant loop. Then, the two most relevant loops are combined in turn with each of the remaining 97 loops to identify the third most relevant loop. This can be repeated to generate a ranked list of (generally N) most relevant loops.

[0031] The operation of the AI can take various inputs and outputs. For example, in some embodiments, the diagnosis code is input to the AI with one or more loops, and only a single metric is output by the AI for that input code, namely the confidence score for the code. In this approach, the code is input to the AI, and a sufficiently low confidence score output by the AI for the code can be understood by a human reviewer as indicating that the AI does not recommend the diagnosis code.

[0032] In other illustrative embodiments, the diagnosis code is not input, but only the one or more loops are input to the AI. In these embodiments, the AI processes the loop(s) to determine whether they support various diagnosis codes, and only outputs a diagnosis code when its confidence score is above some threshold, along with its confidence score. In these embodiments, the diagnosis code is output, not input.

[0033] Further, it should be understood that, in accordance with the AI analysis of the cycle(s), the confidence scores used herein with respect to the diagnostic codes are broadly interpreted as a measure of how likely it is that the patient imaged by the processed cycle(s) has the diagnosis or finding indicated by the diagnostic code. The confidence score can be represented, for example, in different ways as a percentage (e.g., within the range [0%, 100%]) or a probability (e.g., within the range [0, 1]), where a confidence score close to 100% (or a probability close to 1) corresponds to a very high likelihood that the patient has the diagnosis or finding indicated by the code, while a score close to 0% (or a probability close to 0) corresponds to a very low likelihood. As another example, the confidence score for a code can monotonically increase with the likelihood that the patient has the diagnosis or finding indicated by the code, but the range of potential confidence score values can not be 0% to 100%. These are just illustrative examples.

[0034] The following two aspects are independent: (1) providing preliminary diagnostic codes by AI analysis and (2) selecting the most relevant cycles by AI analysis. For example, in some embodiments, only the first aspect is employed: AI is used to generate preliminary codes, but not to select the most relevant cycles for the preliminary codes generated by the AI. In other embodiments, only the second aspect is employed: the preliminary diagnostic codes are manually generated (e.g., by a technician reviewing the cycles), and the AI is only used to select the most relevant cycles for the manually generated preliminary diagnostic codes.

[0035] In other embodiments, the first and second aspects are employed together. That is, some or all of the preliminary diagnostic codes can be generated by the AI in accordance with the first aspect, and then the AI is used to select the most relevant cycles for the preliminary diagnostic codes (whether the preliminary diagnostic codes were manually selected or selected by the AI in accordance with the first aspect). Advantageously, in such embodiments that incorporate both the first and second aspects, the same AI engine can be employed for both the first aspect (generating some or all of the preliminary diagnostic codes) and the second aspect (selecting the most relevant cycles for manual review of the diagnostic codes).

[0036] Embodiments that employ AI to select the most relevant loops for a given preliminary diagnosis code can optionally employ an improved user interface for the echocardiogram interpretation environment via which the cardiologist reviews the preliminary diagnosis code. In the user interface, the preliminary diagnosis code is listed, each associated with the most relevant loops for that code. The association of relevant loops with the code can be direct, e.g., displaying the most relevant loops for that code, or can be associated by displaying a hyperlink to the most relevant loops stored in the PACS 20 (where the hyperlink can be represented by a thumbnail image or thumbnail video representing the loops). The confirmation user interface also provides a "confirm" button (or other suitable user-selectable dialog element) by which the cardiologist can select to confirm the preliminary diagnosis code. In embodiments that display hyperlinks or thumbnails to represent the relevant loops, the cardiologist can click on a loop to bring up a larger video of the loop displayed at its full resolution. If the cardiologist selects the "confirm" button, the confirmed diagnosis code is added to the echocardiogram report as the diagnosis code and / or as a corresponding natural language textual statement of the finding or diagnosis represented by the diagnosis code. Optionally, the link to the most relevant loops can also be automatically annotated to the diagnosis code in the report.

[0037] Some embodiments can also employ AI to select the most relevant loops for a diagnosis code selected by the cardiologist (or technician at the preliminary diagnosis stage). For example, the cardiologist (or technician) can be reviewing a given loop or set of loops and can enter a diagnosis code based on the cardiologist's reading of those loops (or at the preliminary stage, a preliminary code can be entered based on the technician's reading). The diagnosis code entered by the cardiologist or technician is then immediately input to a relevant loop selection engine that identifies the most relevant loops for that diagnosis code. If the highly relevant loops selected by the AI are not currently being reviewed by the cardiologist (or technician), a user dialog is opened suggesting review of that additional loop.

[0038] In another contemplated aspect, the AI engine can also be trained to identify loops that are highly discordant with a given diagnosis code (i.e., trained to identify loops that strongly suggest that the diagnosis code is not appropriate for that echocardiogram study), and if highly discordant loops are identified, a user dialog is opened suggesting review of that discordant loop by the cardiologist.

[0039] With continuing reference to Figure 1Figurative diagram illustrating an echocardiographic interpretation environment 40, suitably implemented by instructions stored on a non-transitory storage medium 20, read and executed by electronic processors 10, 22, 30. In a typical setting, the server 22 performs more complex processing to implement the interpretation environment 40, such as implementing neural networks, support vector machine (SVM) classifiers or other complex Al, while a local processor at the client system (or including the client system), such as the echocardiograph 10 or the cardiologist’s workstation 30, performs user interface dialog drawing operations, local control of echocardiographic image acquisition hardware, data entry of electrocardiographic reports, etc. The echocardiographic interpretation environment 40 includes an echocardiographic reading user interface (UI) 42, which by way of one non-limiting example can be implemented as the UI of the Philips IntelliSpace Cardiovascular (ISCV)™ echocardiographic interpretation environment. The echocardiographic interpretation environment 40 also includes one or more diagnostic Al engines 44, which receive one or more loops and in response output a diagnostic code and an associated confidence score (or in different embodiments receive one or more loops and a diagnostic code and output only a confidence score for that diagnostic code). In embodiments including Al-generated preliminary diagnostic codes, implementation of such preliminary Al diagnoses requires input of one or more loops 46 to the Al engine(s) 48 and in response, retrieval of one or more preliminary diagnostic codes 50 (preferably together with corresponding confidence scores).

[0040] The echocardiogram interpretation environment 40 also includes (most relevantly) a cycle selection component 50 that identifies the most relevant cycle in the echocardiogram for a given input diagnostic code 52. To this end, the cycle selection component 50 inputs one or more cycles 54 to the AI engine(s) 44, optionally along with the diagnostic code (in embodiments where the AI receives the code as input) or along with a task identification (in embodiments where the AI does not receive the code, but has multiple AI engines trained to perform different tasks, the provided task identification effectively determines which AI engine or engines should be employed). As a non-limiting illustrative example, the task can be a left ventricular assessment task, and the corresponding AI engine 44 can be trained to output an appropriate diagnostic code for a given input cycle(s), such as one of the codes selected from the following group: LV 1 : left ventricular performance is normal; LV 2: left ventricular performance is mildly reduced; LV 3: left ventricular performance is moderately reduced; LV 4: left ventricular performance is severely reduced; LV Q: left ventricular performance cannot be assessed due to low image quality; and LV I: left ventricular performance cannot be assessed due to incomplete study. In response to the input(s) 54 provided by the cycle selection component 50, the AI engine(s) 44 return a result 56 that includes one or more diagnostic codes generated by the AI along with a corresponding confidence score(s) (or in alternative embodiments where the code is input, the result 56 can simply be the confidence score determined by the AI for that code). For task-oriented embodiments of the AI engine(s) 44, the cycle selection component 50 appropriately includes a lookup table 58 of diagnostic codes to task conversions. For example, if the received code 52 is LV-2, then in the previous example, the table 58 would associate the code LV-2 with the left ventricular performance task. Based on the returned confidence scores 56 for the input diagnostic code 52, the cycle selection component 50 identifies the most relevant cycle for the code 52. By performing this process for various preliminary diagnostic codes, the cycle selection component 50 builds a lookup table 60 that associates each preliminary diagnostic code with a set of most relevant cycles. A confirmation UI 62 (or sub-UI 62) of the echocardiogram reading UI 42 presents the preliminary diagnostic codes to the cardiologist at the workstation 30 along with the most relevant cycles for each code identified in the lookup table 60 constructed by the cycle selection component 50.

[0041] Referring to Figure 2 , one possible illustrative workflow is shown for using Figure 1The AI engine(s) 44 generate preliminary diagnosis codes. In operation 70, echocardiogram loops are received (e.g., acquired by the echocardiograph 10 for a patient undergoing an electrocardiogram). In operation 72, the loops are labeled as views (and optionally also as sub-views). Operation 72 can be a manual operation performed by an ultrasound technician, or automated labeling using a loop classifier, or some combination thereof. In operation 74, a diagnostic task is selected, and the appropriate view(s) identified, and the echocardiogram loops labeled with such view(s) at 72 are selected. To perform the preliminary diagnosis, the output of operation 74 is Figure 1 the input 46 shown in FIG. 1. For example, if the ultrasound technician requires AI assistance in performing a left ventricular (LV) assessment, the technician selects the LV assessment task, and references a lookup table to identify the standard views for performing an LV assessment. The echocardiogram loops labeled with such views then form the input 46 to the AI engine(s) 44. In Figure 2 At operation 76, the loop(s) selected at operation 74 are input to the AI engine(s) 44, if this is the input to the AI engine(s) 44, possibly along with the identification of the task (e.g., selecting to use a particular AI engine trained to perform that task). At operation 78, the diagnosis code (or codes) generated by the AI engine(s) 44 in response to the input 76 are received, and displayed on the display 14 of the echocardiograph 10, for example, for the technician to consider. Alternatively, the AI-generated code(s) output at operation 78 can be automatically entered into a preliminary echocardiogram report, or can be so entered after being confirmed by the ultrasound technician. In the case of automatic insertion into a report, such automatic insertion can only be done if the confidence score is above some minimum threshold (e.g., >80% confidence).

[0042] With reference to Figure 3 and Figure 4 , an illustrative confirmation process is described via which a cardiologist confirms the preliminary diagnosis codes with the assistance of the confirmation UI 62 supported by the loop selection component 50. In operation 80, an echocardiogram study is retrieved for the cardiologist’s workstation 30. The retrieved echocardiogram includes various preliminary diagnosis codes generated by the ultrasound technician and / or by the AI engine(s) 44, e.g., via the method of Figure 2 Such information is displayed on the display 34 of the cardiologist’s workstation 30 as display 42D of the echocardiogram reading UI 42, for example. The cardiologist can then review the echocardiogram and the preliminary diagnosis codes, and confirm or modify the codes as appropriate. The confirmed diagnosis codes are then output to the echocardiogram report 44, and are inserted into the report 44 as the final diagnosis codes for the echocardiogram study. Figure 4the non-limiting illustrative display example shown in FIG. 42D). The display 42D includes a display window 62D generated by the confirmation UI 62. To produce this display 62D, in operation 82, the preliminary code to be confirmed is selected; in operation 84, the lookup table 60 is referenced to identify the most relevant loops for the code selected at 82 (again, the lookup table is generated by the loop selection component 50, as previously described with reference to Figure 1 the non-limiting illustrative display example shown in FIG. 42D). The display 42D includes a display window 62D generated by the confirmation UI 62. To produce this display 62D, in operation 82, the preliminary code to be confirmed is selected; in operation 84, the lookup table 60 is referenced to identify the most relevant loops for the code selected at 82 (again, the lookup table is generated by the loop selection component 50, as previously described with reference to Figure 4 the non-limiting illustrative display example shown in FIG. 42D). The display 42D includes a display window 62D generated by the confirmation UI 62. To produce this display 62D, in operation 82, the preliminary code to be confirmed is selected; in operation 84, the lookup table 60 is referenced to identify the most relevant loops for the code selected at 82 (again, the lookup table is generated by the loop selection component 50, as previously described with reference to Figure 4 the non-limiting illustrative display example shown in FIG. 42D). The display 42D includes a display window 62D generated by the confirmation UI 62. To produce this display 62D, in operation 82, the preliminary code to be confirmed is selected; in operation 84, the lookup table 60 is referenced to identify the most relevant loops for the code selected at 82 (again, the lookup table is generated by the loop selection component 50, as previously described with reference to

[0043] With continuing reference to Figure 3 and Figure 4 if, at operation 90, the cardiologist selects a selection button representing one of the preliminary codes, then, in operation 92, code text describing the findings or diagnosis corresponding to the selected preliminary (and now confirmed) diagnostic code is entered into the echocardiogram report. In the illustrative example of FIG. 42D, the window 94D in the echocardiogram reading UI display 42D shows an illustrative report, which graphically indicates that the selection button 96 for the code representing mildly reduced LV performance was selected. Figure 4

[0044] With continuing reference to Figure 1 and now also with reference to Figure 5 an illustrative example of a method is shown, the method being performed by the loop selection component 50, in response to a user selection of a preliminary code from the echocardiogram reading UI display 42D, to generate a confirmation display 62D for the selected preliminary code, the confirmation display 62D including a display window 62D generated by the confirmation UI 62, the display window 62D including a selection button for each of the preliminary code, the selection button for each of the preliminary code being associated with a most relevant loop for the preliminary code, the most relevant loop for each of the preliminary code being identified by the loop selection component 50 in response to the user selection of the preliminary code.​Figure 1 The cycle selection component 50 suitably performs a loop selection procedure for selecting the most relevant loop for the input diagnosis code 52. In this embodiment, the AI engine 44 comprises a plurality of different AI engines, each trained to perform a particular echocardiogram diagnosis or finding task (e.g. the illustrative LV performance assessment task such as the illustrative example herein). Thus, in operation 100, the code to task lookup table 58 is consulted to identify the task to which the input code 52 belongs. At operation 102, it is determined whether the set of AI engines 44 includes an AI engine trained to perform the task identified at 100. If no AI engine is available for this task, then at 104 the cycle selection component 50 ends processing of the code 52 without associating any most relevant loop with the code 52. (This can be represented in the lookup table 60 as a null entry for the code 52, for example). On the other hand, if at decision operation 102 it is determined that an AI engine trained to perform the task associated with the code 52 is available, then at operation 104 a number of candidate loops are selected from the set of echocardiogram loops of the patient’s echocardiogram for testing. For example, in one embodiment, each loop of the echocardiogram is processed independently as a candidate loop (e.g. if the echocardiogram comprises 100 loops, then initially each of the 100 loops is tested individually). At operation 106, each of the loops selected at 104 is input to an AI engine in the AI engine 44 trained to perform the task identified at 102. At operation 108, the most relevant loop (or subset of loops) or N most relevant loops (or N most relevant subsets of loops) is identified using the confidence scores returned by the AI engine for each loop of the code 52 (or more generally, for each subset of one or more loops of the echocardiogram selected at 104) (where N is some positive integer). At stop criterion 110, it is determined whether more testing should be performed. If the stop criterion 110 is not met, then flow proceeds to operation 112, where the candidate loops (or loop subsets) are updated, and flow proceeds back to operation 106, where the updated loops or loop subsets are tested by the AI, and the updated most relevant loop is selected at 108, and so on, until the stop criterion 110 is met.

[0045] For example, in one approach, operation 104 selects each individual cycle of the echocardiogram to be tested. Thus, if there are N cycles in the echocardiogram, this will generate N tests, each consisting of a single cycle. In this example, at operation 108, the single cycle with the best confidence score is accepted as the most relevant cycle. Then, at operation 112, an updated cycle subset is generated, each of which is a double cycle subset that includes the accepted most relevant cycle and one other cycle. Thus, if there are N cycles in the echocardiogram, this will generate N-1 double cycle subsets. Each of these N-1 double cycle subsets is tested per operation 106, and at the next round of operation 108, the most relevant double cycle subset is selected. The flow then goes again to operation 112, where each remaining cycle is added to this most relevant double cycle subset to generate N-2 triple cycle subsets. This process can continue to identify any selected number of most relevant cycles, e.g., if the stopping criterion 110 stops after this second round, the final output 114 will be the three most relevant cycles for code 52.

[0046] Referring again to Figure 5 Another example of a suitable most relevant cycle selection process that can be performed by cycle selection component 50 is as follows. At operation 100, a diagnostic code is mapped to a code family (e.g., echocardiogram task). For example, the code "LV-2: mild reduction in left ventricular performance" is mapped to the family "left ventricular performance." This mapping is suitably performed by way of lookup table 58. The family can have an AI engine associated with it for cycle selection. For any given diagnostic code, if at operation 102 it is determined that there is no AI engine associated with its family, then no cycles are selected and the process terminates at 104. On the other hand, if there is an AI engine, then cycle selection component 50 utilizes it to select a subset of cycles that sufficiently support the diagnostic code. This can be implemented in various ways. In this illustrative example, a gradient ascent selection process is used. In a gradient ascent search, a function F is maximized by iteratively traversing a multidimensional grid and picking as successors the nodes that maximally increase the function score. Here, the nodes correspond to subsets of the cycle set of the echocardiogram. In the grid, one can traverse from one node to the next after adding one cycle to any one node if the cycle subset associated with the cycle is the same.

[0047] For each node n, the value F(n) is determined by two values generated by the AI engine: (1) the likelihood of the diagnostic code (e.g., "LV-2") according to the AI engine's inputs; and (2) the confidence score output by the AI engine for that code. The function F combines these inputs to generate a composite value.

[0048] In one processing sequence following this approach, the initial iteration starts with an empty loop set, and the search will find the one loop that maximally supports the likelihood as indicated by the confidence score output for the individual loop for the inputted diagnosis code (coded by F). Then, the likelihood and confidence in the inputted diagnosis code will typically increase by iteratively adding more and more loops. The search can terminate if one of the following conditions (i.e., illustrative examples of the stopping criteria 100) holds: (i) the difference (delta) between F(n) and F(n') (where n is the previous node and n' is the current node) does not exceed a predetermined threshold; or (ii) whenever F(n) exceeds a predetermined threshold or a threshold specifically determined for this search.

[0049] In a more generalized workflow, the loop selection component 50 can operate as follows. For an echocardiogram that includes an echocardiogram loop set, a diagnosis code 52 is generated (e.g., by the AI engine 44 utilized to determine a preliminary diagnosis code) or received (e.g., received via user input 16 of the echocardiogram machine 10). A plurality of different subsets are selected from the echocardiogram, where each subset is composed of one or more echocardiogram loops of the echocardiogram loop set. For each subset, a confidence score is determined that indicates a relevance of the subset to the diagnosis code 52. The confidence score is determined using an AI engine 44 that operates on the subset, where the AI engine that determines the confidence score is trained on historical echocardiograms that are labeled with the diagnosis code. Based on the determined confidence score for the respective subset, the score indicating a relevance of the respective subset to the diagnosis code, a relevant echocardiogram loop group is identified. An echocardiogram reading user interface 62D (see FIG. 6D) is presented that includes a display of the diagnosis code associated with the echocardiogram loops of the relevant echocardiogram loop group. Figure 4

[0050] In this generalized workflow, the echocardiogram analysis method can iterate between selecting a subset and determining a confidence score. In such an iterative embodiment, at least one iteration of the selecting is based at least in part on the confidence score determined for a subset selected in a previous iteration of the selecting.

[0051] ​As another more specific example of this generalized workflow, selecting the subset and determining the confidence score can include: selecting one or more first subsets from the echocardiogram; determining, for the one or more first subsets, a confidence score indicative of a relevance of the respective first subset to the diagnostic code; selecting one or more second subsets by expanding the first subset with additional echocardiogram loops; and determining, for the one or more second subsets, a confidence score indicative of a relevance of the respective first subset to the diagnostic code. This iterative process can continue by additional iterations of: selecting one or more additional (e.g., third) subsets by expanding the (last, e.g., second) subset with additional echocardiogram loops; and determining, for the one or more additional (e.g., third) subsets, a confidence score indicative of a relevance of the respective third subset to the diagnostic code.

[0052] In presenting the echocardiogram reading user interface 42, this can include displaying an echocardiogram report (e.g., via a window 94D of the echocardiogram reading user interface 42D). In response to receiving a user selection of one of the relevant echocardiogram loop group’s echocardiogram loops via the echocardiogram reading user interface 42D’s (of the confirmation window 62D), the user-selected echocardiogram loop is displayed. In response to receiving a user selection of the diagnostic code via the echocardiogram reading user interface 42D’s (of the confirmation window 62D), text corresponding to the diagnostic code is inserted into the echocardiogram report. Figure 4

[0053] Optionally, the echocardiogram analysis method further includes ranking the relevant echocardiogram loop group’s echocardiogram loops based on the determined confidence scores for the respective subsets. In this case, the relevant echocardiogram loop group’s echocardiogram loops can be associated with the diagnostic codes ordered by the ranking (e.g., in the confirmation window 62D display).

[0054] In another method for ordering the displayed most relevant loops, for each of the relevant echocardiogram loop group’s echocardiogram loops, a loop confidence score is determined using the AI engine operating on the echocardiogram loop, indicative of a relevance of the echocardiogram to the diagnostic code. In this method, the relevant echocardiogram loop group’s echocardiogram loops are associated with the diagnostic codes ordered by the loop confidence scores.

[0055] ​In the illustrative example, each AI engine 44 comprises an artificial neural network (ANN). More generally, however, the AI engines 44 can be any type of classifier that outputs a confidence score for a diagnostic code in response to processing an echocardiogram cycle or subset of cycles. As some non-limiting illustrative examples, one or more of the AI engines 44 can comprise one or more of: an ANN trained on historical echocardiograms labeled with diagnostic codes; a support vector machine (SVM) trained on historical echocardiograms labeled with diagnostic codes; another type of machine learning (ML) component trained on historical echocardiograms labeled with diagnostic codes; a logistic regression classifier trained on historical echocardiograms labeled with diagnostic codes, etc.

[0056] The application has been described with reference to the preferred embodiments. Modifications and alterations can occur to others upon reading and understanding the preceding detailed description. It is intended to include all such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. A non-transitory storage medium storing instructions readable and executable by an electronic processor to perform an echocardiographic analysis method, the method comprising: generating or receiving a primary diagnosis code for an echocardiogram comprising a set of echocardiogram cycles, the diagnosis code being a unique identifier for a specific diagnosis or finding; selecting a plurality of different subsets from the echocardiogram, wherein each subset consists of one or more echocardiogram cycles from the set of echocardiogram cycles; determining, for each subset, a confidence score indicating a relevance of the subset to the diagnosis code, wherein the confidence score is determined using an artificial intelligence (AI) engine operating on the subset, wherein the AI ​​engine is trained on historical echocardiograms comprising echocardiographic cycles and annotated with diagnosis codes, wherein one or more candidate cycles are input to the AI ​​engine, and the AI ​​engine outputs the diagnosis code and the confidence score for the diagnosis code; and wherein said selecting and said determining include: selecting one or more first subsets from the echocardiograms; determining, for the one or more first subsets, a confidence score indicating a relevance of the respective subset to the diagnostic code; selecting one or more second subsets by extending the first subset with additional echocardiographic cycles; and determining, for the one or more second subsets, a confidence score indicating a relevance of the corresponding second subset to the diagnosis code; identifying a relevant set of echocardiographic cycles based on the determined confidence scores for the respective subsets indicating relevance of the respective subsets to the diagnostic code; and An echocardiogram reading user interface is presented including displaying the diagnosis code associated with the echocardiogram cycle in the group of related echocardiogram cycles.

2. The non-transitory storage medium according to claim 1, wherein: The echocardiographic analysis method comprises: iterating between the selecting and the determining; and The selecting of at least one iteration is based at least in part on a confidence score determined for a subset selected in a previous iteration of the selecting.

3. The non-transitory storage medium according to any one of claims 1 to 2, wherein the presentation of the echocardiogram reading user interface further comprises: Display echocardiogram report; in response to receiving a user selection of one of the echocardiographic cycles in the group of associated echocardiographic cycles via the echocardiographic reading user interface, displaying the user-selected echocardiographic cycle; In response to receiving a user selection via the echocardiogram reading user interface confirming the diagnosis code, text corresponding to the diagnosis code is inserted into the echocardiogram report.

4. The non-transitory storage medium according to any one of claims 1 to 2, wherein the echocardiography analysis method further comprises: ranking the echocardiographic cycles in the group of correlated echocardiographic cycles based on the confidence scores determined for the respective subsets; wherein the echocardiographic cycles in the group of related echocardiographic cycles are associated with the diagnosis codes ordered by the ranking.

5. The non-transitory storage medium according to any one of claims 1 to 2, wherein the echocardiography analysis method further comprises: determining, for each echocardiographic cycle in the set of correlated echocardiographic cycles, a cycle confidence score indicating a correlation of the echocardiographic cycle with the diagnostic code, wherein the cycle confidence score is determined using the AI ​​engine operating on the echocardiographic cycle; and The echocardiographic cycles in the group of related echocardiographic cycles are associated with the diagnosis codes ordered by the cycle confidence scores.

6. The non-transitory storage medium of any one of claims 1 to 2, wherein the generating or receiving of the diagnostic code comprises: The diagnostic code is automatically generated using the AI ​​engine operating on a subset of the set of echocardiographic cycles.

7. The non-transitory storage medium of any one of claims 1 to 2, wherein the generating or receiving of the diagnostic code comprises: using an echocardiograph to acquire said echocardiogram comprising said set of echocardiogram cycles; as well as The diagnostic code is received via user input of the echocardiograph.

8. The non-transitory storage medium according to any one of claims 1 to 2, wherein the AI ​​engine comprises an artificial neural network.

9. The non-transitory storage medium according to any one of claims 1 to 2, wherein: The AI ​​engine includes a plurality of task-specific AI engines, each of the plurality of task-specific AI engines being trained on historical echocardiograms annotated with diagnostic codes for a corresponding clinical task, and The confidence score indicating the relevance of the subset to the diagnosis code is determined by identifying a clinical task corresponding to the diagnosis code and using the task-specific AI engine operating on the subset for the identified clinical task.

10. An echocardiographic analysis device comprising: electronic processors (10, 22, 30); as well as a non-transitory storage medium storing instructions readable and executable by the electronic processor to analyze an echocardiogram comprising a set of echocardiogram cycles, the stored instructions comprising: AI engine instructions implementing an artificial intelligence (AI) engine trained to output a confidence level for a diagnostic code in response to receiving an input set of one or more echocardiographic cycles; Related cycle identification instructions that implement related cycle identification, the related cycle identification comprising determining confidence levels for preliminary diagnosis codes of different subsets of the set of echocardiographic cycles by inputting the different subsets into the AI ​​engine, the diagnosis code being a unique identifier for a specific diagnosis or finding, and identifying related groups of echocardiographic cycles based on the determined confidence levels for the respective subsets; and echocardiogram reading instructions to implement an echocardiogram reading user interface on a client system, including displaying on a display of the client system the preliminary diagnosis code associated with the echocardiogram loop of the related echocardiogram loop group; wherein the related loop identification includes (i) initially selecting different subsets, (ii) determining a confidence level for the preliminary diagnosis code for the different subsets by inputting the different subsets to the AI engine, and (iii) at least one iteration of selecting an additional subset by adding an echocardiogram loop to the subset with the highest confidence level determined in step (ii) and then repeating step (ii) for the additional subset.

11. The echocardiogram analysis device of claim 10, wherein the implementing of the echocardiogram reading user interface includes: displaying an echocardiogram report on the display of the client system; and in response to receiving a user selection confirming the preliminary diagnosis code via the echocardiogram reading user interface, inserting text corresponding to the preliminary diagnosis code into the echocardiogram report.

12. The echocardiogram analysis device of any one of claims 10-11, wherein the echocardiogram analysis method further includes: ranking the echocardiogram loops of the related echocardiogram loop group based on the confidence scores for the preliminary diagnosis code determined by inputting subsets of the related echocardiogram loop group to the AI engine; wherein the echocardiogram loops of the related echocardiogram loop group are associated with the preliminary diagnosis codes ordered by the ranking.

13. The echocardiogram analysis device of any one of claims 10-11, wherein the instructions further include: analysis instructions to generate the preliminary diagnosis code by operations including inputting a subset of one or more echocardiogram loops of the echocardiogram set to the AI engine.

Images