Method and system for predicting button push sequences during ultrasound examinations - Patents.com

Abstract

The method includes obtaining a sequence of button pushes performed by a user via a control interface during an examination workflow, distinguishing components of the examination workflow based on the button push sequence, extracting a user-specific workflow trend including most frequently used button push sequences, detecting a probing motion of a transducer probe during an ultrasound examination, and predicting a next button push string using a predictive model based on previous button pushes in the user-specific workflow trend, where predicting the next button push string is triggered by the detected probing motion, and outputting a macro button on the control interface corresponding to the predicted string of the next button push, where selecting the macro button executes the corresponding predicted string of the next button push.

Description

[Technical field]

[0001] For ultrasound imaging, button pushes and keystrokes entered by a user (e.g., sonographer, physician) are logged and stored in a service log file as a series of events that captures the workflow description of an ultrasound imaging exam from start to finish. [Background technology]

[0002] Thus, the log file provides insightful information about the examination workflow. Analysis of the log file can help designers improve the control panel layout of the ultrasound imaging system, improving the workflow and custom / dynamic keyboard and touch screen design that mimics user preferences. However, personalized and smart workflows are still required by users for faster and more efficient ways of working during ultrasound imaging examinations. Summary of the Invention [Problem to be solved by the invention]

[0003] Currently, ultrasound imaging systems are not configurable to user settings and preferences, and no intelligence is applied to speed up the imaging exam workflow with respect to suggested lists or sequences of buttons to press to complete a scan protocol. Expert users may have specific ways of working for specific clinical applications and / or specific patient populations, making generic workflow solutions feasible, but not user specific. For example, expert users may have different image optimization workflows, while novice users may deviate from a better and faster workflow due to lack of experience. [Means for solving the problem]

[0004] According to an exemplary embodiment, a method is provided for performing an ultrasound examination using an ultrasound imaging system that includes a transducer probe and a control interface for controlling the acquisition of ultrasound images during the ultrasound examination. The method includes the steps of: acquiring a sequence of button pushes performed by a user via the control interface during an examination workflow, each sequence of button pushes having a corresponding sequence length defined by a number of button pushes in the sequence of buttons; distinguishing components of the examination workflow based on the sequence of button pushes, the distinguished components depending on a clinical application of the ultrasound examination; performing sequence pattern mining of the sequence of button pushes based on the distinguished components of the examination workflow to extract a user-specific workflow trend, the user-specific workflow trend having a plurality of most frequently used sequences of button pushes; detecting a probe movement of the transducer probe during the ultrasound examination; and predicting a next button push string using a predictive model based on at least one previous button push in the user-specific workflow trend, respectively, wherein predicting the next button push string is triggered by the detected probe movement; and outputting at least one macro button on the control interface corresponding to the predicted next button push string, wherein selection of at least one macro button by the user executes the corresponding predicted next button push string.

[0005] According to another exemplary embodiment, a system for performing an ultrasound examination is provided, the system comprising: an ultrasound imaging system having a transducer probe and a control interface for controlling acquisition of ultrasound images during the ultrasound examination, a display configured to display the ultrasound images, and at least one processor coupled to the ultrasound imaging system and the display, and a method for, when executed by the at least one processor, causing the at least one processor to acquire a sequence of button pushes performed by a user via the control interface during an examination workflow, each sequence of button pushes having a corresponding sequence length defined by a number of button pushes in the sequence of buttons, and distinguishing components of the examination workflow based on the sequence of button pushes, the distinguished components depending on a clinical application of the ultrasound examination. performing sequence pattern mining of the sequences of button pushes based on distinguished components of the inspection workflow to extract a user-specific workflow trend, the user-specific workflow trend having a plurality of most frequently used sequences of button pushes; predicting a next button push string using a predictive model, each based on at least one previous button push in the user-specific workflow trend; and outputting at least one macro button on the display corresponding to the predicted next button push string, wherein selection of at least one macro button by the user executes the corresponding predicted next button push string.

[0006] According to another representative embodiment, a non-transitory computer readable medium storing instructions for performing an ultrasound examination is provided. When executed by at least one processor, the instructions cause the at least one processor to perform the steps of: obtaining a sequence of button pushes performed by a user during an examination workflow via a control interface configured to interface with a transducer probe during an ultrasound examination, each sequence of button pushes having a corresponding sequence length defined by a number of button pushes in the sequence of buttons; distinguishing components of the examination workflow based on the sequence of button pushes, the distinguished components depending on a clinical application of the ultrasound examination; performing sequence pattern mining of the sequences of button pushes based on the distinguished components of the examination workflow to extract a user-specific workflow trend, the user-specific workflow trend having a plurality of most frequently used sequences of button pushes; predicting a next button push string using a predictive model based on at least one previous button push in each of the user-specific workflow trends; and outputting at least one macro button on the display corresponding to the predicted next button push string, wherein selection of at least one macro button by the user executes the corresponding predicted next button push string.

[0007] The illustrative embodiments are best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that the various features are not necessarily drawn to scale. In fact, dimensions may be arbitrarily increased or decreased for clarity of discussion. Where applicable and practical, like reference numerals refer to like elements. [Brief description of the drawings]

[0008] [Figure 1]FIG. 1 is a simplified block diagram of an ultrasound imaging system for predicting a button push sequence during an ultrasound examination, in accordance with a representative embodiment. [Diagram 2] FIG. 1 is a flow diagram illustrating a method for predicting a button push sequence during an ultrasound examination using an ultrasound imaging system, according to a representative embodiment. [Figure 3A] 1 is a first portion of a schematic diagram illustrating an example of the distinction of examination workflow components during an ultrasound examination, according to a representative embodiment. [Figure 3B] FIG. 2 is a second portion of a schematic diagram illustrating an example of the distinction of examination workflow components during an ultrasound examination, according to a representative embodiment. [Figure 4] 1 is a plan view of a control interface in an ultrasound imaging system including macro buttons corresponding to strings of predicted next button pushes in accordance with a representative embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the following detailed description, for purposes of explanation and not limitation, representative embodiments disclosing specific details are described to provide a thorough understanding of the embodiments according to the present teachings. Descriptions of known systems, devices, materials, methods of operation, and methods of manufacture may be omitted to avoid obscuring the description of the representative embodiments. Nevertheless, systems, devices, materials, and methods within the scope of those skilled in the art may be within the scope of the present teachings and used in accordance with the representative embodiments. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting. Defined terms are in addition to the technical and scientific meaning of the defined terms as commonly understood and accepted in the art of the present teachings.

[0010] It should be understood that although terms such as first, second, third, etc. may be used herein to describe various components or components, these components or components should not be limited by these terms. These terms are used only to distinguish one component or component from another component or component. Thus, a first element or component described below can be called a second element or component without departing from the teachings of the inventive concept.

[0011] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used in this specification and the appended claims, the singular forms of the terms "a," "an," and "the" are intended to include both the singular and the plural unless the context clearly dictates otherwise. In addition, the terms "comprises," "comprises," and / or similar terms specify the presence of stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0012] Unless otherwise stated, when a component or component is said to be "connected," "coupled," or "adjacent" to another component or component, it will be understood that the component or component may be directly connected or coupled to the other component or component, or there may be intervening components or components. That is, these and similar terms encompass the cases where one or more intermediate components or components may be used to connect the two components or components. However, when a component or component is said to be "directly connected" to another component or component, this only encompasses the cases where the two components or components are connected to each other without any intermediate or intervening components or components.

[0013] Thus, the present disclosure is intended to derive one or more of the advantages as specifically described below through one or more of its various aspects, embodiments, and / or specific features or subcomponents. For purposes of explanation and not limitation, exemplary embodiments disclosing specific details are described to provide a thorough understanding of embodiments according to the present teachings. However, other embodiments consistent with the present disclosure that depart from the specific details disclosed herein remain within the scope of the appended claims. Furthermore, descriptions of known devices and methods may be omitted so as not to obscure the description of the exemplary embodiments. Such methods and devices are within the scope of the present disclosure.

[0014] Generally, various embodiments described herein provide systems and methods for improving ultrasound imaging workflow by combining log file analysis with ultrasound image and probe tracking to intelligently predict the next string of button pushes. The improved ultrasound imaging workflow is dynamically adapted to each user's working methods and preferences, real-time examination workflow, and clinical application (reason for ultrasound examination).

[0015] FIG. 1 is a simplified block diagram of an ultrasound imaging system for predicting a button push sequence during an ultrasound examination, according to a representative embodiment.

[0016] 1, the ultrasound imaging system 100 includes a workstation 130 for implementing and / or managing the processes described herein. The workstation 130 includes one or more processors, represented by processor 120, one or more memories, represented by memory 140, a user interface 122, and a display 124. The user interface 122 and the display 124 may be integrated into a control interface 125 operable by a user to control the ultrasound images according to an examination workflow described below. The memory 140 stores instructions executable by the processor 120. When executed, the instructions cause the processor 120 to implement, for example, one or more processes for predicting a button push sequence during an ultrasound examination, as well as a process for controlling the performance of the ultrasound images, described below with reference to FIG. 2. For illustrative purposes, the memory 140 is shown to include software modules, each of which includes instructions corresponding to associated capabilities of the ultrasound imaging system 100, as described below.

[0017] Processor 120 represents one or more processing devices, which may be implemented using any combination of hardware, software, firmware, hardwired logic circuitry, or combinations thereof, such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a general purpose computer, a central processing unit, a computer processor, a microprocessor, a microcontroller, a state machine, a programmable logic device, or combinations thereof. Any processing unit or processor herein may include multiple processors, parallel processors, or both. Multiple processors may be included in or combined in a single device or multiple devices. The term "processor" as used herein encompasses electronic components capable of executing programs or machine-executable instructions. A processor may also refer to a collection of processors within a single computer system or distributed among multiple computer systems, such as in a cloud-based or other multi-site application. A program has software instructions executed by one or more processors, which may be within the same computing device or distributed across multiple computing devices.

[0018] Memory 140 may include main memory and / or static memory, which may communicate with each other and with processor 120 via one or more buses. Memory 140 may be implemented by any number, type, and combination of random access memory (RAM) and read-only memory (ROM), for example, and may store various types of information, such as software algorithms, artificial intelligence (AI) machine learning models, and computer programs, all of which are executable by processor 120. The various types of ROM and RAM may include any number, type, and combination of computer-readable storage media, such as disk drives, flash memory, Electrically Field Programmable Gate Array Read Only Memory (EPROM), Electrically Erasable Field Programmable Gate Array Read Only Memory (EEPROM), registers, hard disks, removable disks, tape, compact disk read only memory (CDROM), digital versatile disks (DVDs), floppy disks, Blu-ray disks, universal serial bus (USB) drives, or any other form of storage media known in the art. Memory 140 is a tangible storage medium that stores data and executable software instructions, and is non-transient while the software instructions are stored therein. As used herein, the term "non-transient" should be interpreted as a characteristic of a state that persists over a period of time, rather than as a permanent characteristic of a state. The term "non-transient" specifically negates fleeting characteristics, such as characteristics of a carrier wave or signal, or other formation that exists only temporarily at any place at any time. Memory 140 may store software instructions and / or computer readable code that enable the performance of various functions. Memory 140 may be secure and / or encrypted, or non-secure and / or unencrypted.

[0019] The ultrasound imaging system 100 further includes or interfaces with one or more log file databases, as indicated by the log file database 116, for storing information that may be used by various software modules in the memory 140. The log file database 116 may be implemented by any number, type, and combination of RAM and ROM, for example. The various types of ROM and RAM may include any number, type, and combination of computer-readable storage media, such as disk drives, flash memory, EPROM, EEPROM, registers, hard disks, removable disks, tapes, CD-ROMs, DVDs, floppy disks, Blu-ray disks, USB drives, or any other form of storage media known in the art. The log file database 116 is a tangible storage medium for storing data and executable software instructions, and is non-transient while the data and software instructions are stored. The log file database 116 may be secure and / or encrypted, or non-secure and / or encrypted. For purposes of illustration, the log file database 116 is shown as a separate database, but it is understood that it may be combined with and / or included within the memory 140 without departing from the scope of the present teachings. The log file database 116 may be established as a routine subject at one or more facilities providing clinical care, storing at least patient demographic and clinical information.

[0020] The ultrasound imaging system 100 further includes a transducer probe 160. The transducer probe 160 may include a transducer array comprising a two-dimensional array of transducers that can be scanned in two or three dimensions to transmit ultrasound waves into a subject (patient) 165 and receive echo information in response thereto. The transducer array may include, for example, capacitive micromachined ultrasound transducers (CMUT) or piezoelectric transducers formed from materials such as PZT or PVDF. The transducer array is coupled to a microbeamformer in the transducer probe 160 to control reception of signals by the transducers.

[0021] The memory 140 includes a probe interface module 141 for interfacing the transducer probe 160 with the processor 120 to control the acquisition of ultrasound images of the subject 165. The probe interface module 141 may include a transmit / receive (T / R) switch coupled to a microbeamformer of the transducer probe 160 by a probe cable. The T / R switch may, for example, switch between transmit and receive modes under the control of the processor 120 and / or the user interface 122. The processor 120 also controls the direction in which the beam is steered and focused via the probe interface module 141. The beam may be steered straight ahead (orthogonal) from the transducer array or at a different angle for a wider field of view. The processor 120 may also include a main beamformer that provides the final beamforming following digitization. In general, the transmission of ultrasound and the reception of echo information are well known and therefore further details in this regard are not included herein.

[0022] Processor 120 may include or have access to an AI engine or module, which may be implemented as software that provides artificial intelligence, such as natural language processing (NLP) algorithms and applies machine learning, such as neural network modeling, as described herein. The AI ​​engine may reside in any of a variety of components in addition to or other than processor 120, such as, for example, memory 140, an external server, and / or the cloud. If the AI ​​engine is implemented in the cloud, such as, for example, a data center, the AI ​​engine may be connected to processor 120 via the Internet using one or more wired and / or wireless connections.

[0023] The user interface 122 is configured to provide to a user information and data output by the processor 120 and / or memory 140 and / or information and data input by a user to the processor 120 and / or memory 140. That is, the user interface 122 allows a user to input data, control or manipulate aspects of the processes described herein, and control or manipulate aspects of ultrasound imaging. The user interface 122 also allows the processor 120 to indicate to the user the effect of the user's controls or manipulations.

[0024] All or part of the user interface 122 may be implemented by a graphical user interface (GUI), such as, for example, a GUI 128 on a touch screen 126 of the display 124, discussed below. The user interface 122 includes push buttons that can be operated (pressed) by a user to initiate various commands for manipulating the displayed images, making measurements and calculations, and the like, during an ultrasound examination. The push buttons may be displayed by the GUI 128 on the touch screen 126, or may be physical buttons, an example of which is shown in FIG. 4, discussed below. Buttons pressed by the user are logged and stored in the log file database 116 as workflows and usage scripts. The user interface 122 may further include any other compatible interface device for performing an ultrasound examination, such as, for example, a mouse, keyboard, trackball, joystick, microphone, video camera, touchpad, or voice or gesture recognition captured by a microphone or video camera.

[0025] Display 124 can be any compatible monitor for displaying ultrasound images, such as a computer monitor, a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, or a solid state display for viewing internal images of subject 165. Display 124 includes a touch screen 126 and a GUI 128 to allow a user to interact with the displayed images and features.

[0026] With reference to the memory 140, the imaging module 142 is configured to receive, process, and display on the display 124 the ultrasound images of the subject 165 received from the transducer probe 160 and the probe interface module 141. The processing may include, for example, band-pass filtering, decimation, I and Q component separation, and harmonic signal separation, as well as signal enhancement, for example, speckle reduction, signal combining, and noise removal. As the user manually manipulates the transducer probe 160 on the surface of the subject 165, the current image is displayed on the touch screen 126. This allows the user to analyze the current image to adjust the position of the transducer probe 160 to obtain a desired image, and to perform measurements and calculations on the touch screen 126.

[0027] The predictive model module 143 is configured to train and apply a predictive model to identify an examination workflow of an ultrasound imaging examination and predict one or more next button pushes of a button push sequence in the examination workflow based on one or more initial button pushes, as described below. The predictive model takes into account the context of the ultrasound examination during which the button push sequence was obtained, including different clinical applications.

[0028] The predictive model module 143 may be, for example, a deep learning neural network such as a convolutional neural network (CNN), a recurrent neural network (RNN), an artificial neural network (ANN), or a transformer network. The predictive model may be trained using log file analysis of the log files in the log file database 116. During training, the predictive model learns sequences of button pushes associated with input ultrasound images to accomplish various tasks in an examination workflow for one or more clinical applications, as discussed below. Training may be performed by obtaining sequences of button pushes performed by a user during the examination workflow of a previous ultrasound examination.

[0029] The NLP module 144 is configured to execute one or more NLP algorithms using word embedding techniques to identify button pushes extracted from the button push sequence entered by the user. The button pushes are typically identified by descriptive text and must be converted into computer readable data. NLP is well known and may include, for example, syntax and semantic analysis, as well as deep learning with accumulation of data to improve understanding by the NLP module 144, as will be appreciated by those skilled in the art. The identified button pushes may be provided to a predictive model module 143 for training and / or to a button prediction module 145, described below, to convert a natural language identification of the button push into computer readable data. In various embodiments, all or part of the processes provided by the NLP module 144 and / or the predictive model module 143 may be implemented by an AI engine executed by, for example, the processor 120.

[0030] The button prediction module 145 is configured to obtain a sequence of button pushes performed by a user via the control interface 125 during an examination workflow, distinguish components of the examination workflow based on the sequence of button pushes, perform sequence pattern mining of the sequence of button pushes based on the distinguished components of the examination workflow to extract user-specific workflow trends of the most frequently used button push sequences, predict a next button push string using a predictive model from the predictive model module 143 based on one or more previous button pushes, and create and output a macro button corresponding to the predicted next button push string on the touch screen 126 of the control interface 125. The distinguished components of the examination workflow depend on the clinical application of the particular ultrasound examination. Once the output macro buttons are provided on the touch screen 126, the user can select them to execute the corresponding predicted next button push string, respectively, for example, by touching a designated area of ​​the touch screen 126. The functionality of the button prediction module 145 is described in more detail with reference to FIG. 2 below.

[0031] The probe motion module 146 is configured to detect probe motion of the transducer probe 160 during an ultrasound examination. For example, the transducer probe 160 may include one or more sensors, such as electromagnetic (EM) sensors or inertial measurement unit (IMU) sensors. The probe motion module 146 then monitors the probe position of the transducer probe 160 by tracking position data provided by the sensors on the transducer probe 160. The probe motion may be used to trigger sequence pattern mining and / or prediction of the next button push without further input from the user.

[0032] 2 is a flow diagram illustrating a method for predicting a button push sequence during an ultrasound examination by a user using an ultrasound imaging system, according to a representative embodiment. The method may be performed, for example, by the ultrasound imaging system 100 described above under control of the processor 120 executing instructions stored as various software modules in the memory 140.

[0033] 2, the method includes initially training a predictive model in block S211 by obtaining a sequence of button pushes performed by a user during an examination workflow of a prior ultrasound examination. As described above, the sequence of button pushes may be obtained from a previously stored log file in a log file database (e.g., log file database 116) and corresponding images from the examination workflow. Each sequence has a corresponding sequence length defined by the number of button pushes in that button push sequence. The prior ultrasound examination may have been performed by the current user and / or by a different user, all of whom are preferably experts in the ultrasound imaging arts.

[0034] The predictive model, which may be a deep learning neural network, may be trained using log file analysis. During training, the predictive model learns sequences of button pushes associated with input images to accomplish various tasks in the examination workflow for one or more clinical applications. The training data includes button push sequences extracted from the examination workflow log files and represented by the NLP module 144. The predictive model uses information of the examination situation as an auxiliary input signal, and through hierarchical calculations of the neural network that allow information extraction, processing, and combination, it may take the formation of a conditional generator that uses a previous button push sequence (a training sequence with a later portion masked) as a primary signal to generate a button push sequence in a later portion of the training data (a training sequence with a later portion masked). The training process is performed by backpropagating the gradient of the neural network weights against a predefined objective function that measures the difference between the predicted output push sequence and the ground truth button push sequence, calculated and executed by the processor 120. The learning enables the predictive model to identify relevant examination workflows and predict one or more next button pushes of a button push sequence in those examination workflows based on one or more initial button pushes in the same sequence, taking into account the ultrasound examination context in which the button push sequences were acquired, including different clinical applications. At inference time, the predictive model propagates forward through the neural network and obtains one or more lengths of examination context information and the current button push sequence performed by the user as input signals for calculating predictions of one or more next button pushes.

[0035] Notably, although shown as the first step of the button push sequence prediction procedure, it is not necessary to first train the prediction model for every ultrasound examination to be performed. For example, training may be performed periodically to update the prediction model, which may then be applied to several different ultrasound examinations without further modification. Also, in one embodiment, the results of each ultrasound examination may be used to train the prediction model during subsequent training sessions.

[0036] In block S212, sequences of button pushes performed by a user via a control interface (e.g., control interface 125) during the current inspection workflow are captured. Each button push sequence corresponds to a component or task of the inspection workflow and has a sequence length defined by the number of button pushes in the button push sequence required to complete the task. The sequence length of the button push sequence may be determined by specific gates, such as the user freezing or acquiring an image, to isolate a set of related events.

[0037] The button pushes may be obtained from the control interface and / or from the log files of the ultrasound imaging system. For example, the button pushes may be obtained by accessing a log file database and pairing the ultrasound images with the log files based on the current examination workflow and the transition states of the ultrasound imaging system discussed below. Thus, the button push sequences are retrieved and organized such that the context of the current examination workflow / images and the transition states are attached to the extracted button push sequences. The transition states are also relevant to the component distinctions discussed below, as they indicate which state the system is in and identify how the button push sequence is being used. The button push sequences are used to infer or predict future button pushes using a previously trained predictive model in block S211, as described below. The examination workflow depends on the ultrasound image content and the clinical application of the ultrasound examination, as described below.

[0038] In block S213, components (or tasks) of the examination workflow are differentiated based on the button push sequences during the examination workflow and the transition states of the ultrasound imaging system. Differentiating the components includes converting the button push sequences obtained in block S212 into corresponding encoded vectors, clustering the encoded vectors according to usage, and separating the clustered encoded vectors into one of two transition states. The two transition states include a "live state" and a "frozen state", in which in the live state, the ultrasound image is displayed in real time, allowing the user to observe the target anatomical structure and interactively adjust the positioning of the transducer probe and / or the acoustic settings of the ultrasound imaging system in real time to obtain a desired ultrasound image. In the frozen state, the currently displayed ultrasound image is frozen, for example, allowing the user to evaluate the ultrasound image and / or measure a part of the anatomical structure in the ultrasound image. The user may also perform calculations based on the measurements.

[0039] With respect to the encoded vectors, the buttons and / or button pushes are effectively "words" to correspond to user interactions via the control interface. These words are converted into encoded vectors to become computer-readable sequences. The length of each encoded vector is determined by the number of button pushes in the corresponding sequence of button pushes. To convert the button push sequences into encoded vectors, the ultrasound images are first paired with a log file based on the examination workflow and the transition state of the ultrasound imaging system. The button push sequences are then converted into encoded vectors, for example, using the NLP-based techniques described above. In one embodiment, the user's user ID may also be extracted from the log file, so that the examination workflow may be customizable.

[0040] The component distinction is influenced by the clinical application of ultrasound examination. Generally, clinical applications include an associated scan protocol based on the anatomical structure to be examined and the purpose of the examination, and although the components of the examination workflow correspond to steps in the scan protocol, the specific examination workflow / button push may vary to complete the protocol. For example, an examination workflow for examining the liver follows a different scan protocol than an examination workflow for examining the bladder. Also, particularly with regard to liver imaging, a scan protocol for routine studies is different from a scan protocol for quantification of fibrosis stage, for example.

[0041] 3A and 3B provide a schematic diagram illustrating an example of distinguishing components of an examination workflow during an ultrasound examination, according to a representative embodiment.

[0042] 3A illustrates an exemplary button push sequence 311 with a set of words representing respective strings of commands entered by a user to interact with an ultrasound imaging system. Each set of words enclosed in a pair of brackets corresponds to a component of an examination workflow. The button push sequence 311 is converted into a corresponding encoded vector 312, indicated by arrow A1. As shown, the encoded vector 312 is a single row (or column) matrix of numbers each representing a set of words in the button push sequence 311. The encoded vector 312 is computer readable, enabling subsequent steps in distinguishing between components.

[0043] The encoded vectors 312 are clustered into clusters 313, indicated by arrow A2 in FIG. 3B. The clustering process is performed using an unsupervised dimensionality reduction algorithm that projects the high-dimensional data into a two-dimensional manifold while maximally preserving the topological structure of the original set of data points. The demonstrated relative topological structure may include two or more clusters, indicating different uses or components, depending on the specific clinical application. In a simplified general case for illustration purposes, two clusters 313, indicated as Seqs_upper_cluster and Seqs_bottom_cluster in FIG. 3B, correspond to live state 314 and frozen state 315, respectively. The horizontal and vertical axes of the clusters 313 represent the x and y coordinates of the latent vectors in the two-dimensional manifold, and thus the respective values ​​are relative. In various other cases where button push sequence data is collected across different test situations or applications, the encoded vectors may be clustered according to two or more different uses or components.

[0044] The clusters 313 are separated into frozen state 314 and live state 315, indicated by arrow A3 in FIG. 3B. This is done by collecting the corresponding original sequence data according to the clusters 313 and counting the frequency of the unique button strings per sequence from each cluster, after which the clusters 313 can be separated into different states by visualizing and analyzing the frequency plots. In each of the frozen state 314 and live state 315, the respective button pushes are indicated by frequency, and for illustration purposes (in each of the two distinct components), the most frequently used button pushes are labeled. Thus, for example, in the frozen state 314, the most frequently used button pushes are Bm_Icon, EVTMGR_CP_ Depth_CHANGE, EVTMGR_CP_FREEZE_PUSH, and Knob_Bm_RotateProbe, and in the live state 315, the most frequently used button pushes are EVTMGR_CP_FREEZE_PUSH, TGC, Depth, and Gain. For each condition, the horizontal axis represents the index of each unique button name, and the vertical axis represents the frequency per sequence in terms of the average number of incidents of that button name per sequence.

[0045] Returning to FIG. 2, in block S214, sequence pattern mining of button push sequences is performed based on the distinguished components of the examination workflow. Sequence pattern mining extracts patterns of the most frequently used consecutive strings of button pushed during an ultrasound examination to identify user-specific workflow trends. Thus, sequence pattern mining can identify K workflow trends from the K most frequently used patterns, where K is a positive integer. The user-following workflow trends can be identified based on the button pushes and real-time imaging context of the clinical application. Sequence pattern mining is performed on the latest dataset of user-specific button pushing sequences from past examinations, which can be updated after a certain number of new examinations are completed.

[0046] As an example, for a clinical application involving abdominal ultrasound examination of the aortic bifurcation, sequence pattern mining of button push sequences may yield the following 10 workflow trends, ranked from most frequently used to least frequently used: 1:[ 'EVTMGR_CP_FREEZE_PUSH', 'EVTMGR_CP_ANNOTATE_PUSH', 'SublLabel_Label', 'EVTMGR_C_FREEZE_PUSH'] 2:['EVTMGR_CP_FREEZE_PUSH', 'Bm_Icon', 'Bm_Icon', 'EVTMGR_C-_FREEZE_PUSH'] 3:['EVTMGR_CP_FREEZE_PUSH', 'EVTMGR_CP-ACUIRE1_PUSH', 'EVTMGR_CP-ACUIRE1_PUSH', 'EVTMGR_CP_FREEZE_PUSH'] 4:['EVTMGR_CP_FREEZE_PUSH', 'SublLabel_Label', 'SublLabel_Label', 'EVTMGR_CP_FREEZE_PUSH'] 5:['EVTMGR_CP_FREEZE_PUSH', 'EVTMGR_CP_DEPTH_CHANGE', 'EVTMGR_CP_DEPTH_CHANGE', 'EVTMGR_CP_FREEZE_PUSH'] 6:['EVTMGR_CP_FREEZE_PUSH', 'SublLabel_Label', 'Btn_Annot_Canned', 'EVTMGR_CP_FREEZE_PUSH'] 7:['EVTMGR_CP_FREEZE_PUSH', 'SublLabel_Label', 'EVTMGR_CP_ ACUIRE1_PUSH', 'EVTMGR_CP_FREEZE_PUSH'] 8:['EVTMGR_CP_FREEZE_PUSH', 'Btn_Annot_Canned', 'SublLabel_Label', 'EVTMGR_CP_FREEZE_PUSH'] 9:['EVTMGR_CP_FREEZE_PUSH', 'Bm_Icon', 'Bm_Icon', 'Bm_Icon'] 10:['EVTMGR_CP_FREEZE_PUSH', 'Bm_Icon', 'EVTMGR_CP_DEPTH_CHANGE', 'EVTMGR_CP_FREEZE_PUSH']

[0047] In one embodiment, the log file database may need to be projected into a smaller space of subsequences to apply pattern constraints, such as gap constraints and sequence length constraints, to sequence pattern mining. The gap constraint ensures that each of the most frequent workflow trends contains gaps between buttons that cannot be selected consecutively by a user in a real ultrasound examination scenario. For example, the Freeze_Push button is frequently used by a user to transition from a live state of an ultrasound imaging system to a frozen state, and vice versa. However, even if the Freeze_push button is frequently used during an examination, it is rare for a user to press it multiple times consecutively. Adding the gap constraint eliminates duplicate or outlier sequences that may contain multiple identical button strings consecutively, which are less meaningful from a clinical perspective, thereby improving sequence pattern mining accuracy. The sequence length constraint applies a preselected minimum sequence length to sequence pattern mining based on user input and clinical usage context. In general, smaller button push sequences generate more usage patterns, but are less meaningful in workflow. Thus, sequences of button pushes less than the preselected minimum are not considered when determining the most frequent workflow trends.

[0048] In another embodiment, sequence pattern mining can be performed using weighted button pushes. The weighting can be based on user preferences that depend on the clinical context to reflect different exam workflow approaches within a selected workflow trend. For example, a user can prefer cross-sectional views of certain organs and longitudinal views of other organs. The extracted patterns of button pushes are weighted based on the clinical application as well as the clinical context, for example, the clinical application can refer to the general exam type being performed and the clinical context can refer to a more specific imaging environment that the system can recognize. For example, when a user is performing an abdominal exam for screening purposes, different weights can be assigned to different button pushes compared to a follow-up abdominal exam. The different weights are applied as subjects of sequence pattern mining to identify preferred trends.

[0049] In block S215, probe motion of the transducer probe is detected during the ultrasound examination. When performing an ultrasound examination according to a scan protocol, a user moves the ultrasound transducer probe over the subject's body surface while viewing the image content to explore the anatomical structures and find the best imaging view. When probe tracking is enabled, the motion of the transducer probe is detected, for example, using a sensor attached to the transducer probe as described above.

[0050] A predefined motion or pattern of motion can be used to trigger a prediction of the next sequence of button pushes for an ultrasound examination. For example, as the user approaches a view of interest of the subject, the probe motion becomes more stable (e.g., slower, less frequent motion over a shorter distance). Also, the probe motion typically stops at an anatomical landmark(s) indicative of the anatomical structure of interest being investigated where qualitative and quantitative analysis is performed. Thus, a prediction can be triggered based on a detected more stable and / or stopped probe motion.

[0051] In block S216, a string of next button pushes is predicted based on at least one previous button push each in the user-specific workflow trend using the predictive model trained in block S211. In general, the predictive model predicts the next M buttons to be pressed in sequence for an ultrasound exam based on N previous button pushes, where M and N are positive integers.

[0052] The predictive model may be, for example, an autoregressive model and may be used together with a decoder. The decoder is configured to decode the sequence of computer-readable predictive vectors (in the form of a matrix) into a sequence of human-interpretable button strings or names. The log files and button pushes associated with the workflow trends provided by sequence pattern mining are divided into smaller subsets (batches) of button push sequences, so that individual button pushes of each button push sequence can be considered. The division of the log files and buttons into subsets may be "learned" by the predictive model or "customized" by the user, for example during a training process. That is, the predictive model or the user may select (or pre-select) the length of the string (the number of next button presses to predict) for the prediction task. In one embodiment, the user pre-selection may be provided based on user preferences identified using the user ID described above and real-time image content extraction. Alternatively, the user pre-selection may be retrieved from the default settings of the ultrasound imaging system or may be directly entered by the user from a control interface, for example using a dedicated knob or other input.

[0053] For example, if a user presses a preselection of N=3 previous buttons, the predictive model predicts the next button push based on the last three previous button pushes by the user. The accuracy and confidence with which each next sequence of button presses is predicted depends on the N previous button press values ​​considered as inputs and the M predicted next button press values ​​for the length of the predicted sequence. In general, the larger the number of previous button presses N and the smaller the number of predicted next button presses M, the greater the prediction accuracy and confidence level, and vice versa. Predictions can be triggered by certain probe movements, such as the transducer probe becoming more stable or stopping, as described above.

[0054] At block S217, a macro button corresponding to the predicted next button push string is output on the control interface. For example, the macro button may be created and displayed on a touch screen of the control interface with a label indicating the associated function of the predicted next button push. A user can then select the macro button to execute the corresponding string of the predicted next button push. The use of macro buttons makes clinical application specific workflow instruction execution faster and more consistent. In one embodiment, the macro buttons may appear in different colors, for example, representing different confidence levels associated with the predicted next button string.

[0055] 4 is a plan view of a control interface in an ultrasound imaging system including macro buttons corresponding to strings of predicted next button pushes, according to a representative embodiment. With reference to FIG. 4, a control interface 425, which may correspond to the control interface 125, for example, includes a touch screen 420 that displays macro buttons corresponding to strings of predicted next button pushes. The control interface 425 also includes standard slider controls 1-8 and push buttons 9-35 that correspond to predetermined functions. In the illustrated example, the touch screen 420 provides a first macro button 421 labeled Freeze 2D Depth change, a second macro button 422 labeled FreezAcquire1 Push, and a third macro button 423 labeled Freeze Annotation push. Also, in the illustrated example, the first and second macro buttons 421 and 422 are displayed in a first color (e.g., green) indicated by light shading to indicate a high confidence level in the associated string of the predicted next button, while the third macro button 423 is displayed in a second color (e.g., yellow) indicated by darker shading to indicate a medium confidence level in the associated string of the predicted next button.

[0056] According to various embodiments of the present disclosure, the methods described herein may be implemented using a hardware computer system executing a software program stored on a non-transitory storage medium. Furthermore, in exemplary non-limiting embodiments, implementations may include distributed processing, component / object distributed processing, and parallel processing. A virtual computer system process may implement one or more of the methods or functions described herein, and the processors described herein may be used to support a virtual processing environment.

[0057] Although predicting button push sequences has been described with reference to exemplary embodiments, it should be understood that the words used are words of description and illustration, rather than words of limitation. Changes may be made within the scope of the appended claims, as presently stated and as amended, without departing from the scope and spirit of intervention procedure optimization in its aspects. Also, while predicting button push sequences has been described with reference to particular means, materials, and embodiments, the intention is not to be limited to the details disclosed, but rather the embodiments extend to all functionally equivalent structures, methods, and uses, as are within the scope of the appended claims.

[0058] The description of the embodiments described herein is intended to provide a general understanding of the structure of the various embodiments. The illustrations are not intended to serve as a complete description of all of the elements and features of the present disclosure described herein. Many other embodiments may be apparent to those skilled in the art upon review of the present disclosure. Other embodiments may be utilized and derived from the present disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. In addition, the descriptions are merely representational and may not be drawn to scale. Certain proportions in the figures are exaggerated, while others are minimized. Thus, the present disclosure and drawings should be considered illustrative rather than limiting.

[0059] One or more embodiments of the present disclosure may be referred to herein, individually and / or collectively, by the term "invention" merely for convenience and without any intention to spontaneously limit the scope of the present application to any particular invention or inventive concept. Moreover, although specific likenesses have been illustrated and described herein, it should be understood that any subsequent configurations designed to achieve the same or similar purpose may be substituted for the specific likenesses shown. The present disclosure is intended to cover any and all subsequent adaptations or modifications of the various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those skilled in the art upon review of the description.

[0060] The Abstract of the Disclosure is provided for purposes of compliance with 37 CFR §1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the foregoing Detailed Description, various features may be grouped together or described in a single embodiment for the purpose of streamlining the disclosure. This disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may be directed to less than all of the features of any of the disclosed embodiments. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as defining separately claimed subject matter.

[0061] The foregoing description of the disclosed embodiments is provided to enable those skilled in the art to practice the concepts described in this disclosure. Therefore, the subject matter disclosed above should be considered as illustrative and not limiting, and the appended claims are intended to encompass all such modifications, enhancements, and other embodiments that fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent permitted by law, the scope of the present disclosure should be determined by the broadest permissible interpretation of the following claims and their equivalents, and should not be limited or restricted by the foregoing detailed description.

Claims

1. 1. A method for performing an ultrasound examination using an ultrasound imaging system having a transducer probe and a control interface for controlling acquisition of ultrasound images during the ultrasound examination, the method comprising: acquiring a sequence of button pushes performed by a user via the control interface during an inspection workflow, each sequence of button pushes having a corresponding sequence length defined by a number of button pushes in the sequence of button pushes; distinguishing components of the examination workflow based on the sequence of button pushes, the distinguished components depending on the clinical application of the ultrasound examination; performing sequence pattern mining of the button push sequences based on the distinguished components of the inspection workflow to extract a user-specific workflow trend, the user-specific workflow trend comprising a plurality of most frequently used button push sequences; detecting probe motion of the transducer probe during the ultrasound examination; predicting a next string of button pushes using a predictive model based on at least one previous button push, each in the user-specific workflow trend, wherein predicting the next string of button pushes is triggered by the detected probe movement; outputting at least one macro button on the control interface corresponding to the predicted next button push string, wherein selection of the at least one macro button by the user executes the corresponding predicted next button push string; A method comprising:

2. The step of obtaining a sequence of button pushes comprises: accessing a log file of the ultrasound imaging system; Pairing ultrasound images with the log file based on the state transition of the ultrasound imaging system and the examination workflow.

2. The method of claim 1, comprising:

3. 3. The method of claim 2, wherein the transition states of the ultrasound imaging system include a live state for providing ultrasound images in real time and a frozen state for freezing the ultrasound image, allowing the user to evaluate and / or measure portions of the ultrasound image.

4. The step of distinguishing components of the inspection workflow includes: converting each sequence of button pushes into an encoded vector; clustering the encoded vectors; separating the clustered encoded vectors into the transition states; 3. The method of claim 2, comprising:

5. The step of performing sequence pattern mining includes: applying gap constraints to the extracted user-specific workflow trends to skip irrelevant buttons in the sequence of button pushes; 2. The method of claim 1, comprising:

6. The step of performing sequence pattern mining includes: applying a sequence length constraint to the extracted user-specific workflow trends to exclude each of the most frequently used sequences of button pushes that is less than a predetermined minimum sequence length; 2. The method of claim 1, comprising:

7. The step of performing sequence pattern mining includes: assigning different weights to the user-specific workflow trends based at least in part on a clinical application of the ultrasound examination; 2. The method of claim 1, comprising:

8. 2. The method of claim 1, wherein detecting probe motion of the transducer probe comprises monitoring the transducer probe using an external camera and determining the probe motion from an image provided by the external camera.

9. 2. The method of claim 1, wherein detecting probe movement of the transducer probe comprises monitoring the transducer probe using at least one sensor on the transducer probe and determining the probe movement from position data provided by the at least one sensor.

10. The method of claim 9 , wherein the at least one sensor comprises at least one of an electromagnetic (EM) sensor or an inertial measurement unit (IMU) sensor.

11. The method of claim 1 , wherein the control interface comprises a touchscreen display.

12. The method of claim 11 , wherein the at least one macro button is displayed on the touchscreen in one of a plurality of different colors corresponding to a plurality of trust levels associated with the at least one macro button.

13. 1. A system for performing an ultrasound examination, comprising: an ultrasound imaging system having a transducer probe and a control interface for controlling the acquisition of ultrasound images during the ultrasound examination; a display configured to display the ultrasound image; at least one processor coupled to the ultrasound imaging system and the display; When executed by the at least one processor, the at least one processor acquiring a sequence of button pushes performed by a user via the control interface during an inspection workflow, each sequence of button pushes having a corresponding sequence length defined by a number of button pushes in the sequence of buttons; distinguishing components of the examination workflow based on the sequence of button pushes, the distinguished components depending on the clinical application of the ultrasound examination; performing sequence pattern mining of the button push sequences based on the distinguished components of the inspection workflow to extract a user-specific workflow trend, the user-specific workflow trend comprising a plurality of most frequently used button push sequences; predicting a string of next button pushes using a predictive model based on at least one previous button push, each in said user-specific workflow trend; outputting at least one macro button on the display corresponding to the predicted next button push string, wherein selection of at least one macro button by the user executes the corresponding predicted next button push string; a non-transitory memory for storing instructions for executing the A system having:

14. The instructions further include causing the at least one processor to: detecting probe motion of the transducer probe during the ultrasound examination; triggering a prediction of the next button push string in response to the detected probe movement; The system of claim 13, wherein the system executes the following:

15. 15. The system of claim 14, wherein the instructions cause the at least one processor to detect probe movement of the transducer probe by monitoring the transducer probe using at least one sensor on the transducer probe and determining the probe movement from position data provided by the at least one sensor.

16. The instructions to the at least one processor: accessing a log file of the ultrasound imaging system; Pairing ultrasound images with the log file based on the state transition and examination workflow of the ultrasound imaging system. The system of claim 13, wherein the sequence of button pushes is captured by

17. 17. The system of claim 16, wherein the transition states of the ultrasound imaging system include a live state for providing the ultrasound image in real time and a frozen state for freezing the ultrasound image, allowing the user to evaluate and / or measure portions of the ultrasound image.

18. The instructions to the at least one processor: converting each sequence of button pushes into an encoded vector; clustering the encoded vectors; separating the clustered encoded vectors into the transition states; The system of claim 16, wherein components of the inspection workflow are differentiated by

19. When executed by at least one processor, the method causes the at least one processor to: acquiring a sequence of button pushes performed by a user during an examination workflow via a control interface configured to interface with a transducer probe during an ultrasound examination, each sequence of button pushes having a corresponding sequence length defined by a number of button pushes in the sequence of buttons; distinguishing components of the examination workflow based on the sequence of button pushes, the distinguished components depending on the clinical application of the ultrasound examination; performing sequence pattern mining of the button push sequences based on the distinguished components of the inspection workflow to extract a user-specific workflow trend, the user-specific workflow trend comprising a plurality of most frequently used button push sequences; predicting a string of next button pushes using a predictive model based on at least one previous button push, each in said user-specific workflow trend; outputting at least one macro button on the display corresponding to the predicted next button push string, wherein selection of at least one macro button by the user executes the corresponding predicted next button push string; A non-transitory computer-readable medium storing instructions for performing an ultrasound examination, the instructions causing the computer to execute:

20. The instructions further include for the at least one processor to: detecting probe motion of the transducer probe during the ultrasound examination; triggering a prediction of a string of next button pushes in response to the detected probe movement; 20. The non-transitory computer-readable medium of claim 19,

Images