Advanced loop selection system and method for supporting efficient echo comparison

Abstract

A system and method for selecting image sequences of an echocardiographic study for visual comparison by a user. The method includes receiving image sequences from a current study and a previous study to be compared. The method includes determining relevant views of the image sequences in the current study and the previous study, and selecting image sequences of relevant views in the one of the current study and the previous study for comparison. The method includes displaying image sequences from one of the current study and the previous study to a user via an interactive comparison panel. The method includes: determining which image sequence in relevant views of the other of the current study and the previous study most closely matches the selected image sequence; and displaying the image sequence from the other of the current study and the previous study that most closely matches the selected image sequence via the interactive comparison panel.

Description

Background Art

[0001] Echocardiography is commonly used to diagnose and treat patients with different heart conditions. In some cases, a cardiologist may be required to compare a current echocardiographic study with a previous echocardiographic study of the patient. However, a typical echocardiographic acquisition may result in multiple image acquisitions of the same view due to, for example, subtle differences (e.g., at different depths) or because the quality of the initial acquisition was deemed non-diagnostic or non-optimal. Therefore, during the comparison of the current study and the previous study, a large number of image sequences may be presented to the cardiologist for review, resulting in a time-consuming and inefficient process.

[0002] WO 2018 / 095792 discloses a medical imaging device, which includes a radiology workstation having a workstation display and one or more workstation user input devices. The medical imaging device controller includes a controller display and one or more controller user input devices.

[0003] US 2016 / 232658 discloses methods and arrangements for automatic ground truth generation for a collection of medical images. Aspects include receiving a plurality of imaging studies, wherein each imaging study includes one or more images and a text report associated with the one or more images. Other aspects also include selecting a key image from each of the plurality of imaging studies and each of the one or more images, and extracting one or more distinguishing image features from a region of interest within the key image.

[0004] US 2018 / 108125 discloses methods and systems for automatically detecting disease based on analysis of echocardiographer's findings in echocardiogram videos. Summary of the invention

[0005] The invention is defined by the claims.

[0006] An exemplary embodiment relates to a method comprising: receiving image sequences from a current study and a previous study to be compared, each image sequence in the image sequences comprising data corresponding to a view of the image sequence; determining relevant views of the image sequences in the current study and the previous study; selecting image sequences in relevant views of one of the current study and the previous study for comparison; displaying the image sequences from one of the current study and the previous study to a user via an interactive comparison panel; determining which image sequence in relevant views of the other of the current study and the previous study most closely matches the selected image sequence; and displaying the image sequence from the other of the current study and the previous study that most closely matches the selected image sequence via the interactive comparison panel.

[0007] An exemplary embodiment relates to a system comprising: a non-volatile computer-readable storage medium storing an executable program; and a processor executing the executable program so that the processor: receives image sequences from a current study and a previous study to be compared, each of the image sequences comprising data corresponding to a view of the image sequence; determines relevant views in the current study and the previous study; selects image sequences in the relevant views of one of the current study and the previous study for comparison; displays the image sequences from one of the current study and the previous study to a user via an interactive comparison panel; determines which image sequence in the relevant views of the other of the current study and the previous study most closely matches the selected image sequence; and displays the image sequence from the other of the current study and the previous study that most closely matches the selected image sequence via the interactive comparison panel.

[0008] An exemplary embodiment relates to a non-volatile computer-readable storage medium comprising a set of instructions capable of being executed by a processor, which, when executed by the processor, causes the processor to perform operations including: receiving image sequences from a current study and a previous study to be compared, each of the image sequences comprising data corresponding to a view of the image sequence; determining relevant views of the current study and the previous study; selecting image sequences in the relevant views of one of the current study and the previous study for comparison; displaying the image sequences from one of the current study and the previous study to a user via an interactive comparison panel; determining which image sequence in the relevant views of the other of the current study and the previous study most closely matches the selected image sequence; and displaying the image sequence from the other of the current study and the previous study that most closely matches the selected image sequence via the interactive comparison panel. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic diagram of a system according to an exemplary embodiment is shown.

[0010] Figure 2 A flow chart of a method according to an exemplary embodiment is shown.

[0011] Figure 3 An exemplary comparison panel comparing relevant views of a current and previous echocardiographic studies is shown in accordance with an exemplary embodiment, wherein sequences in the current examination that are considered most likely to be selected by a user for visual comparison are highlighted.

[0012] Figure 4 According to an exemplary embodiment, Figure 3 An exemplary comparison panel in which a user selects a sequence from a current study for visual comparison causing matching sequences from a previous echocardiographic study to be highlighted.

[0013] Figure 5 According to an exemplary embodiment, Figure 3 and 4 An exemplary comparison panel in which a user selects a matching sequence from a previous echocardiographic study for visual comparison with a selected sequence from a current echocardiographic study. DETAILED DESCRIPTION

[0014] The exemplary embodiments may be further understood with reference to the following description and accompanying drawings, in which similar elements are represented by the same reference numerals. The exemplary embodiments relate to systems and methods for selecting image sequences of relevant views of current and previous echocardiographic studies for visual comparison by a user. The image sequences selected from each relevant view are presented to the user in an interactive comparison panel, so that the user can further select which sequences the user wishes to visually compare from the current and previous studies. Those sequences determined by the system to be most likely to be selected by the user for visual comparison may be presented to the user in the interactive comparison panel. However, the user may select any available sequence for visual comparison. These user selections may then be stored and used to improve the sequence selection process so that only the most relevant and / or most likely selected sequences are presented to subsequent users of the interactive comparison panel.

[0015] As will be described in more detail below, the exemplary embodiments improve the operation of the system in a number of ways. In a first example, storing user selections to improve the sequence selection process increases the speed at which relevant sequences are displayed to the user and reduces the search time for relevant sequences. In another example, as will be described in more detail below, storing sequences with contextual information and other information can more efficiently store and search for sequences, thereby improving system performance.

[0016] like Figure 1As shown in , according to an exemplary embodiment of the present disclosure, a system 100 selects image sequences of a current and previous echocardiographic study for visual comparison by a user (e.g., a cardiologist), thereby presenting a sequence of most relevant views to the user in the context of a comparison indication. The comparison indication may include a condition or anatomical structure being compared, such as left ventricular performance or mitral regurgitation. The system 100 includes a processor 102, a user interface 104, a display 106, and a memory 108. The processor 102 may include a sequence selection engine 110, a sequence comparison engine 112, and a machine learning engine 114. An interactive comparison panel 116 is displayed on the display 106, and the interactive comparison panel 116 shows image sequences in the current study and the previous study that can be used for visual comparison. Using the user interface 104, a user can select an image sequence from the interactive comparison panel 116 for visual comparison.

[0017] The sequence selection engine 110 receives a series of echocardiographic image sequences from the current study and selects one or more sequences from the current study that the sequence selection engine 110 believes are most likely to be selected by the user for visual comparison. The user can then select the image sequence selected by the sequence selection engine 110, or select a different image sequence from the current study for visual comparison. The sequence comparison engine 112 then selects (one or more) image sequences from the previous study that most closely match the (one or more) image sequences selected by the user. Again, the user can select (one or more) image sequences selected by the sequence comparison engine 112, or select a different image sequence for visual comparison with the (one or more) image sequences selected by the user for the current study. All of these user-system interactions and contextual information are stored in an interaction database 118 of the memory 108. The information stored in the interaction database can be used via the machine learning engine 114 to create a model for improving the sequence selection process of the sequence selection engine 110 and / or the sequence comparison engine 112 based on previous user selections.

[0018] Those skilled in the art will appreciate that while the exemplary embodiment illustrates and describes the sequence selection engine 110 as selecting (one or more) image sequences from a current study, the sequence selection engine 110 may similarly select (one or more) image sequences from a prior study, wherein the sequence comparison engine 112 selects (one or more) image sequences from the current study that most closely match the image sequences selected from the prior study. Those skilled in the art will also appreciate that the image sequences for the current study and the prior study may be accessed from a patient's medical record, which may be stored in the memory 108.

[0019] The interactive comparison panel 116 can show all available image sequences for the current study and the previous study, indicating which image sequences from the current study and the previous study are most likely to be selected by the user for visual comparison. In particular, in an embodiment involving the interactive comparison panel 116, the image sequences most likely to be selected by the user for visual comparison can be based on the comparison indication indicated and / or selected by the user. For example, the image sequences selected by the sequence selection engine 110 and the sequence comparison engine 112 can be highlighted on the display 106. Alternatively, as more user-system interaction data is stored in the interaction database 118, the sequence selection engine 110 and the sequence comparison engine 112 are able to run the model created by the machine learning engine 114 to better predict the image sequences that will be selected by the user for visual comparison based on the current study and the previous study, and the interactive comparison model 116 can only display those image sequences that are most likely to be selected by the user.

[0020] Those skilled in the art will appreciate that engines 110-114 may be implemented by processor 102, for example, as lines of code executed by processor 102, as firmware executed by processor 102, as functions of processor 102 as an application specific integrated circuit (ASIC), and so on.

[0021] Figure 2 An exemplary method 200 is shown for selecting image sequences of a current and previous echocardiographic study to be visually compared by a user, and presenting a sequence of most relevant views to the user in the context of a comparison indication. In 210, the sequence selection engine 110 receives image sequences from the current and previous echocardiographic studies and structural comparison indications from the user. The sequence from the echocardiographic examination can be retrieved from a patient medical record (e.g., a picture archiving and communication system). The structural comparison indication can be input by the user via the user interface 104 and can include an indication such as "compare LV performance" or "compare mitral regurgitation." In 220, the sequence selection engine 110 determines relevant views of the echocardiographic study that show the comparison indication entered by the user in 210. The sequence selection engine 110 can determine, for example, that the PLAX, SLAX, AP2, and AP4 views show LV performance. The echocardiographic views showing the comparison indication can be determined, for example, by a lookup table.

[0022] At 230, the sequence selection engine 110 selects one or more image sequences of relevant views of the current echocardiographic study that are most likely to be selected by the user for visual comparison. The sequence selection engine 110 processes the sequences of relevant views including contextual information, including, for example, the order of the sequences (based on acquisition timestamps) and an indication to generate an output for each sequence in the relevant view, the output indicating the likelihood of the sequence being selected by the user for visual comparison. The output may be a binary "yes" or "no", or a number within a predetermined range (e.g., 0-1), where a high number indicates a high likelihood of being selected and a low number indicates a low likelihood of being selected. The sequence selection engine 110 may run one or more mathematical, statistical, or rule-based predictive models. The sequence selection engine 110 may utilize a library of known and proprietary image features. For example, rule-based techniques may perform high-level reasoning, such as:

[0023] - If the sequence is the first sequence in multiple views in the same view, split it

[0024] Configured as output 0.

[0025] - If this sequence is the last in the sequence, then apply the prediction model X.

[0026] In another example, particularly during earlier iterations of method 200, the rule-based technique may select the last sequence in each view as the most likely to be selected by the user for visual comparison by default, because in many cases, the sequences are imaged until the best sequence has been obtained. In yet another example, the predictive model may consider the order of the image sequences. For example, the predictive model may consider the sequence to be the fourth sequence in a sequence of five sequences. The selected (one or more) image sequences in the current study are displayed to the user in the interactive comparison panel 116.

[0027] At 240, the processor 102 receives a user input selecting one or more sequences of relevant views of the current echocardiographic study that the user wants to view for comparison. The user may select a sequence different from the image sequence selected by the sequence selection engine 110 at 230. The user may input the user selection via the user interface 104.

[0028] In 250, the sequence comparison engine 112 selects one or more sequences from the previous study that closely match the sequence selected by the user in 240. The sequence comparison engine 112 compares each image sequence of the same relevant view from the previous study with the user-selected image sequence in the current study and assigns a score (e.g., between 0-1) to each image sequence of the same relevant view in the previous study to mark its similarity. In an exemplary embodiment, the sequence comparison engine 112 may perform a difference operation on the frames of the sequences to be compared and generate a weighted difference score across all frames. In another exemplary embodiment, the sequence comparison engine 112 may first align the heartbeats of the sequences before acquiring the differences of the frames, potentially utilizing ECG signals recorded with the sequence image data in a dedicated DICOM field. In yet another exemplary embodiment, the sequence comparison engine 112 may first reduce the resolution before acquiring the differences of the frames. Once an image sequence of the previous study that matches the user-selected sequence is determined, the matching image sequence is displayed to the user via the interactive comparison panel 116.

[0029] At 260, the processor 102 receives a user input selecting a sequence of a previous study to be visually compared to the user-selected sequence of the current study in 240. The user selection may be made via the user interface 104. As will be described in further detail below, after selecting an image sequence from a previous study, the user may provide further input indicating a desire to view the selected image sequence in the current study in 240 and the selected image sequence in the previous study in 260, side by side for visual comparison.

[0030] Although 230 and 240 describe selecting an image sequence from the current study, and 250 and 260 describe selecting an image sequence from a previous study, those skilled in the art will appreciate that the sequence selection engine 110 may similarly select an image sequence from a previous study so that the sequence comparison engine 112 selects a matching image sequence from the current study. Those skilled in the art will also appreciate that, if desired, 230-260 may be repeated as needed to select an image sequence from the current study and the previous study for each different relevant view.

[0031] At 270, all user-system interactions are received from the interactive comparison panel 116 and stored in the interaction database 118. For example, the sequences selected by the sequence selection engine 110 and the sequence comparison engine 112, the sequences selected by the user, and any contextual information (e.g., sequence order, structural comparison indications) may be stored in the interaction database 118. Although the user-system interactions are shown and described as being stored in 270, those skilled in the art will appreciate that the user-system interactions and any contextual information for the sequences may be stored in the interaction database 118 as they occur. In particular, the interactions may be stored in the interaction database 118 as information is received from the interactive comparison panel.

[0032] At 280, the machine learning engine 114 accesses the interaction database 118 to create a machine learning, statistical and / or rule-based model for inclusion in the sequence selection engine 110 and the sequence comparison engine 112. To create a model for the sequence selection model, for example, the machine learning engine 114 may use information such as a series of image sequences, one or more sequences selected (ground truth), an order of the sequences, and a comparison indication. To create a model for the sequence comparison engine 112, the machine learning engine 114 may use information such as two series of sequences being compared (image data), one or more selected sequences in the series of the first sequence (ground truth), an order of the two sequences, and a comparison indication.

[0033] Models can be created for each of the sequence selection engine 110 and the sequence comparison engine 112 using known or new image processing feature libraries using techniques such as convolutional neural networks, random forests, support vector machines, logistic regression, etc., with the goal of maximizing the performance of predicting the true sequence from other sequences. According to another exemplary embodiment, the model creation engine can also utilize a view detection engine that determines the view of the sequence (e.g., PLAX, SLAX, AP2, AP3, AP4), that is, the position of the probe relative to the heart. The machine learning engine 114 can be called according to a predetermined schedule, or alternatively, manually called via a system administrator. For example, the machine learning engine 114 can be called every night or every month to improve the models of the sequence selection engine 110 and the sequence comparison engine 112. In another embodiment, a fixed set of annotated instances can be used to track the performance of the newly created instances to determine whether the performance of the new model is better than the currently used model. If the performance is determined to be excellent, the currently used model can be replaced with the newly created model.

[0034] Figure 3-5 An exemplary interactive comparison panel according to system 100 and method 200 is shown. Figure 3As shown in 210, the interactive comparison panel 116 shows the sequences of the current echocardiographic study and the previous echocardiographic study in each relevant view according to the structural comparison indication received by the sequence selection engine 110 in 210. For example, for the LV performance indication in which the relevant views are determined to be PLAX, SLAX, AP2, and AP4 views in 220, the interactive comparison panel 116 shows the sequences from the current study and the previous study in each relevant view, disabling other views that are considered irrelevant. However, the interactive comparison panel 116 allows the user to easily enable other views if desired. Likewise, the interactive comparison panel allows the enabled views to be easily disabled if desired.

[0035] Figure 3 The interactive comparison panel 116 of the embodiment shows all available sequences in each relevant view of the current study and the previous study, and the sequence in the current study selected by the sequence selection engine 110 in 230 is highlighted on the display 106. The highlighting indicates to the user which sequence is determined to be most likely to be selected by the user for visual comparison. Although the exemplary embodiment displays the selected sequence as highlighted, those skilled in the art will understand that the interactive comparison panel 116 can use other methods to indicate the selection of the sequence selection engine 110. In addition, although Figure 3 Only the sequences of the current study selected by the sequence selection engine 110 are shown, but the interactive comparison panel 116 can also be labeled to highlight sequence selections from a previous study or both the current study and the previous study.

[0036] although Figure 3 The interactive comparison panel 116 of 230 shows all available sequences for each relevant view, but in another embodiment, the interactive comparison panel 116 may only show those sequences selected by the sequence selection engine 110 as being most likely to be selected by the user for visual comparison in 230. In this embodiment, the interactive comparison panel 116 is not required to highlight (or provide any other indication of selection) any sequence, since only those sequences that have been selected are shown. Alternatively, the interactive comparison panel 116 may show any sequence that produces an output above a predetermined threshold, highlighting the sequence with the highest generated output.

[0037] As described above, the user may select the sequence selected by the sequence selection engine 110, or alternatively, choose to select a different sequence. Figure 4 In the example shown, the user selects a highlighted sequence for the current study (i.e., selected by the sequence selection engine 110) in one of the related views (e.g., PLAX). Selecting a sequence causes a previously studied sequence in the same view to be highlighted. The highlighted sequence displays the sequence determined by the sequence comparison engine 112 as the closest match to the sequence selected by the user in 2650. Figure 5 As shown, the user can then select a sequence from a previous study that the user wants to visually compare with the user-selected sequence of the current study. After selecting two sequences that the user wants to compare, the user can press, for example, a compare button of an interactive comparison panel to generate a side-by-side visual comparison of the two selected sequences.

[0038] Those skilled in the art will appreciate that the exemplary embodiments described above may be implemented in a variety of ways, including as separate software modules, as a combination of hardware and software, etc. For example, the sequence selection engine 110, the sequence comparison engine 112, and the machine learning engine 114 are programs comprising lines of code that, when compiled, may be executed on the processor 102.

[0039] It will be apparent to those skilled in the art that various modifications may be made to the disclosed exemplary embodiments, methods and alternatives without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is intended to cover modifications and variations that fall within the scope of the appended claims and their equivalents.

Claims

1. A computer-implemented method for selecting an image sequence of echocardiographic views in a current echocardiographic study and a previous echocardiographic study to be visually compared by a user, comprising: receiving an image sequence from each of a current echocardiographic study and a previous echocardiographic study to be compared, each of the image sequences comprising data corresponding to a view of the image sequence; receiving a comparison indication from a user, wherein the comparison indication includes the condition or anatomical structure being compared; determining the echocardiographic views in the image sequences of the current echocardiographic study and the previous echocardiographic study showing the comparison indication; selecting an image sequence in the echocardiographic view showing the comparison indication in one of the current echocardiographic study and the previous echocardiographic study for comparison; displaying to a user via an interactive comparison panel a selected sequence of images from one of the current echocardiographic study and the previous echocardiographic study; determining which image sequence in the echocardiographic view showing the comparison indication in the other of the current echocardiographic study and the previous echocardiographic study most closely matches the selected image sequence; and The image sequence from the other of the current echocardiographic study and the previous echocardiographic study that most closely matches the selected image sequence is displayed via the interactive comparison panel.

2. The method according to claim 1, further comprising: User input corresponding to the selected sequence of images is received.

3. The method according to claim 1, further comprising: A user input corresponding to a matching image sequence selected by the user is received for visual comparison with the selected image sequence.

4. The method according to claim 1, further comprising: Creating at least one of a machine learning, statistical, and rule-based model for selecting the image sequence from one of the current echocardiographic study and the previous echocardiographic study, and determining the image sequence from the other of the current echocardiographic study and the previous echocardiographic study that most closely matches the selected image sequence.

5. The method according to claim 1, wherein: Selecting the image sequence from the one of the current echocardiographic study and the previous echocardiographic study includes analyzing an acquisition order of each of the image sequences in the one of the current echocardiographic study and the previous echocardiographic study.

6. The method according to claim 1, wherein: Selecting the image sequence from the one of the current echocardiographic study and the previous echocardiographic study includes generating, for each image sequence in the one of the current echocardiographic study and the previous echocardiographic study, an output corresponding to a likelihood that the sequence will be selected by the user for visual comparison.

7. The method according to claim 6, wherein: Image sequences in the current echocardiographic study and the one of the previous echocardiographic studies having an output greater than a predetermined value are displayed via the interactive comparison panel.

8. The method according to claim 1, wherein: Determining which image sequence in the echocardiographic view showing the comparison indication in the current echocardiographic study and the other one of the previous echocardiographic studies most closely matches the selected image sequence includes: for each image sequence in the echocardiographic view showing the comparison indication in the current echocardiographic study and the other one of the previous echocardiographic studies, outputting a score corresponding to the level of similarity between each image sequence in the current echocardiographic study and the other one of the previous echocardiographic studies and the selected image sequence.

9. The method according to claim 8, wherein: Image sequences in the current echocardiographic study and the other of the previous echocardiographic studies having an output score greater than a predetermined value are displayed via the interactive comparison panel.

10. A system (100), comprising: a non-transitory computer-readable storage medium storing an executable program; as well as A processor (102) that runs the executable program to cause the processor to: receiving an image sequence from each of a current echocardiographic study and a previous echocardiographic study to be compared, each of the image sequences comprising data corresponding to a view of the image sequence; receiving a comparison indication from a user, wherein the comparison indication includes the condition or anatomical structure being compared; determining an echocardiographic view in the current echocardiographic study and the prior echocardiographic study showing the comparative indication; selecting an image sequence in the echocardiographic view showing the comparison indication in one of the current echocardiographic study and the previous echocardiographic study for comparison; displaying to a user via an interactive comparison panel (116) a selected image sequence from one of the current echocardiographic study and the previous echocardiographic study; determining which image sequence in the echocardiographic view showing the comparison indication in the other of the current echocardiographic study and the previous echocardiographic study most closely matches the selected image sequence; and The image sequence from the other of the current echocardiographic study and the previous echocardiographic study that most closely matches the selected image sequence is displayed via the interactive comparison panel (116).

11. The system according to claim 10, wherein: The processor (102) runs the executable program so that the processor: User input corresponding to the selected sequence of images is received.

12. The system according to claim 10, wherein: The processor (102) runs the executable program so that the processor (102): A user input corresponding to a matching image sequence selected by the user is received for visual comparison with the selected image sequence.

13. The system according to claim 10, wherein: The processor runs the executable program so that the processor: At least one of a machine learning, statistical, and rule-based model is created based on user-system interaction to select the image sequence from one of the current echocardiographic study and the previous echocardiographic study, and to determine the image sequence from the other of the current echocardiographic study and the previous echocardiographic study that most closely matches the selected image sequence.

14. The system according to claim 10, wherein: Determining which image sequence in the echocardiographic view showing the comparison indication in the current echocardiographic study and the other one of the previous echocardiographic studies most closely matches the selected image sequence includes: for each image sequence in the echocardiographic view showing the comparison indication in the current echocardiographic study and the other one of the previous echocardiographic studies, outputting a score corresponding to the level of similarity between each image sequence in the current echocardiographic study and the other one of the previous echocardiographic studies and the selected image sequence.

15. A non-transitory computer-readable storage medium comprising a set of instructions executable by a processor, the set of instructions, when executed by the processor, causing the processor to perform the operations of the method of claims 1-9.

Images