Method and system for predicting button press sequence during ultrasound examination
By analyzing the user button press sequence and prediction model in the ultrasound imaging system, combined with probe movement, personalized and intelligent button press prediction of ultrasound examination is achieved, which solves the problem that the existing system cannot adapt to the needs of users and clinical applications, and improves the problem of Check efficiency and accuracy.
Patent Information
- Application Number
- CN202380029818.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-24
- Filing Date
- 2023-03-21
- Publication Date
- 2024-11-08
AI Technical Summary
Existing ultrasound imaging systems cannot intelligently configure user settings and preferences, resulting in users not being able to achieve faster and more efficient workflows during ultrasound examinations, especially for different clinical applications and user groups. Unable to meet personalized needs.
By using button press sequence analysis and prediction models in ultrasound imaging systems, identify user-specific workflow trends and predict the next button press string based on probe motion, dynamically adjust the control interface to adapt to user preferences and clinical applications, providing personalization macro button operation.
The ultrasound examination process is personalized and intelligent, and the button press sequence is dynamically adjusted to improve work efficiency, adapt to the needs of different users and clinical applications, and improve the speed and accuracy of the examination.
Smart Images

Figure CN118922133A_ABST
Abstract
Description
Background Art
[0001] In ultrasound imaging, button presses and keystrokes input by users (e.g., sonographers, physicians) are recorded and stored in service log files as a series of events that capture the workflow of an ultrasound imaging examination from start to finish. Thus, log files provide insightful information about the examination workflow. Log file analysis of log files helps designers improve the control panel layout of ultrasound imaging systems. This improves workflows and customized / dynamic keyboard and touch screen designs that mimic the user's preferences. However, users still seek personalized and intelligent workflows to achieve a faster and more efficient way of working during ultrasound imaging examinations.
[0002] Currently, ultrasound imaging systems are not configurable for user settings and preferences, and no intelligence is applied to speed up the imaging exam workflow with respect to the suggested list or sequence of buttons that should be pressed to complete a scan protocol. Expert users may have a way of working that is specific to certain clinical applications and / or specific patient populations, such that general workflow solutions are feasible but not user-specific. For example, an expert user may have a different image optimization workflow, while a novice user may deviate from a better and faster workflow due to lack of experience. Summary of the invention
[0003] According to a representative embodiment, a method is provided for performing an ultrasound examination using an ultrasound imaging system comprising: a transducer probe; and a control interface for controlling acquisition of ultrasound images during the ultrasound examination. The method includes: obtaining a sequence of button presses performed by a user via the control interface during an examination workflow, each button press sequence having a corresponding sequence length defined by the number of button presses in the button sequence; distinguishing components of the examination workflow based on the button press sequences, wherein the distinguished components depend on the clinical application of the ultrasound examination; performing sequence pattern mining of the button press sequences based on the distinguished components of the examination workflow to extract user-specific workflow trends, wherein the user-specific workflow trends include a plurality of most frequently used button press sequences; detecting probe motion of an ultrasound transducer probe during the ultrasound examination; predicting a next string of button presses using a prediction model based on at least one previous button press in the user-specific workflow trends, respectively, wherein the next string of button presses is predicted to be triggered by the detected probe motion; and outputting at least one macro button corresponding to the predicted next string of button presses on the control interface, wherein selection of the at least one macro button by the user executes the corresponding predicted next string of button presses.
[0004] According to another representative embodiment, a system for performing an ultrasound examination is provided. The system includes: an ultrasound imaging system including a transducer probe and a control interface for controlling the acquisition of ultrasound images during an ultrasound examination; a display configured to display the ultrasound images; at least one processor coupled to the ultrasound imaging system and the display; and a non-transitory memory for storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the following operations: obtain a sequence of button presses performed by a user via the control interface during an examination workflow, each sequence of button presses having a corresponding sequence length defined by the number of button presses in the sequence of button presses; distinguish components of the examination workflow based on the sequence of button presses, wherein the sequence of button presses is used to perform the following operations: wherein the distinguished components depend on the clinical application of the ultrasound examination; performing sequential pattern mining of the button press sequence based on the distinguished components of the examination workflow to extract user-specific workflow trends, wherein the user-specific workflow trends include the most frequently used button press sequences; using a prediction model to predict a next string of button presses based on at least one previous button press in the user-specific workflow trends, respectively; and outputting at least one macro button corresponding to the predicted next string of button presses on the control interface on the display, wherein selection of the at least one macro button by the user executes the corresponding predicted next string of button presses.
[0005] According to another representative embodiment, a non-transitory computer-readable medium storing instructions for performing an ultrasound examination is provided. When executed by at least one processor, the instructions cause the at least one processor to perform the following operations: obtaining a button press sequence performed by a user via a control interface during an examination workflow, the control interface being configured to interface with a transducer probe during the ultrasound examination, each button press sequence having a corresponding sequence length defined by the number of button presses in the button press sequence; distinguishing components of the examination workflow based on the button press sequence, wherein the distinguished components depend on a clinical application of the ultrasound examination; performing sequential pattern mining of the button press sequence based on the distinguished components of the examination workflow to extract a user-specific workflow trend, wherein the user-specific workflow trend includes the most frequently used button press sequence; predicting a next button press string using a prediction model based on at least one previous button press in the user-specific workflow trend, respectively; and outputting at least one macro button corresponding to the predicted next button press string on the control interface on a display, wherein the user's selection of the at least one macro button executes the corresponding predicted next button press string. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] When with Figure 1 The exemplary embodiments are best understood from the following detailed description when read together. It is emphasized that the various features are not necessarily drawn to scale. In fact, for the clarity of discussion, the dimensions may be increased or decreased arbitrarily. Whenever applicable and practical, the same reference numerals represent the same elements.
[0007] Figure 1 is a simplified block diagram of an ultrasound imaging system for predicting a sequence of button presses during an ultrasound examination, according to a representative embodiment.
[0008] Figure 2 is a flow chart illustrating a method of predicting a sequence of button presses during an ultrasound examination using an ultrasound imaging system, according to a representative embodiment.
[0009] Figure 3A is a first portion of a schematic diagram illustrating an example of components of an inspection differentiation workflow during an ultrasound inspection according to a representative embodiment.
[0010] Figure 3B is a second portion of a schematic diagram illustrating an example of components of an inspection differentiation workflow during an ultrasound inspection according to a representative embodiment.
[0011] Figure 4 is a plan view of a control interface in an ultrasound imaging system including macro buttons corresponding to a predicted next string of button presses according to a representative embodiment. DETAILED DESCRIPTION
[0012] In the following detailed description, for the purpose of explanation rather than limitation, representative embodiments that disclose specific details are set forth in order to provide a thorough understanding of the embodiments according to the present teachings. The description of known systems, equipment, materials, operating methods, and manufacturing methods may be omitted to avoid blurring the description of the representative embodiments. Nevertheless, systems, equipment, materials, and methods within the scope of those of ordinary skill in the art are within the scope of the present teachings and may be used according to the representative embodiments. It should be understood that the terms used herein are only used for the purpose of describing specific embodiments and are not intended to be restrictive. The defined terms are attached to the technical and scientific meanings of the defined terms that are generally understood and accepted in the technical field of the present teachings.
[0013] It should be understood that although the terms first, second, third, etc. can be used to describe various elements or components in this article, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another element or component. Therefore, without departing from the teaching of the present invention, the first element or component discussed below can be referred to as the second element or component.
[0014] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to be restrictive. As used in the specification and the appended claims, unless the context clearly indicates otherwise, the singular form of the terms "one", "an" and "said" is intended to include both the singular and the plural form. In addition, the terms "include", "comprise" and / or similar terms specify the presence of the features, elements and / or components, but do not exclude the presence or addition of one or more other features, elements, components and / or their groups. As used herein, the term "and / or" includes any and all combinations of one or more items in the associated listed items.
[0015] Unless otherwise specified, when an element or component is referred to as being "connected to," "coupled to," or "adjacent to" another element or component, it will be understood that the element or component may be directly connected or coupled to the other element or component, or there may be intervening elements or components. That is, these and similar terms cover situations where one or more intermediate elements or components may be employed to connect two elements or components. However, when an element or component is referred to as being "directly connected" to another element or component, this only covers situations where two elements or components are connected to each other without any intermediate or intervening elements or components.
[0016] Therefore, the present disclosure is intended to provide one or more advantages specifically mentioned below through one or more of its various aspects, embodiments and / or specific features or subassemblies. For the purpose of explanation and not limitation, example embodiments that disclose specific details are set forth in order to provide a thorough understanding of the embodiments according to the present teachings. However, other embodiments consistent with the present disclosure that depart from the specific details disclosed herein are still within the scope of the appended claims. In addition, descriptions of well-known devices and methods may be omitted so as not to obscure the description of the example embodiments. Such methods and devices are within the scope of the present disclosure.
[0017] In general, various embodiments described herein provide a system and method for improving ultrasound imaging exam workflow by combining log file analysis with ultrasound images and probe tracking to intelligently predict the next string of button presses. The improved ultrasound imaging exam workflow dynamically adapts to each user's work style and preferences, real-time exam workflow, and clinical application (reason for the ultrasound exam).
[0018] Figure 1 is a simplified block diagram of an ultrasound imaging system for predicting a sequence of button presses during an ultrasound examination, according to a representative embodiment.
[0019] refer to Figure 1, the ultrasound imaging system 100 includes a workstation 130 for implementing and / or managing the processes described herein. The workstation 130 includes one or more processors indicated by the processor 120, one or more memories indicated by the memory 140, a user interface 122, and a display 124. The user interface 122 and the display 124 can be integrated into a control interface 125, which can be operated by a user to control ultrasound imaging according to an examination workflow, as described below. The memory 140 stores instructions that can be executed by the processor 120. When executed, the instructions cause the processor 120 to implement one or more processes for predicting a sequence of button presses during an ultrasound examination (e.g., as described below with reference to Figure 2 For illustrative purposes, the memory 140 is shown as including software modules, each of which includes instructions corresponding to the associated performance of the ultrasound imaging system 100, as discussed below.
[0020] Processor 120 represents one or more processing devices and can be implemented by a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a general purpose computer, a central processing unit, a computer processor, a microprocessor, a microcontroller, a state machine, a programmable logic device, or a combination thereof using any combination of hardware, software, firmware, hard-wired logic circuits, or a combination thereof. Any processing unit or processor herein may include multiple processors, parallel processors, or both. Multiple processors may be included in a single device or multiple devices or coupled to a single device or multiple devices. The term "processor" as used herein encompasses an electronic component capable of executing a program or machine executable instructions. A processor may also refer to a collection of processors within a single computer system or distributed between multiple computer systems (such as in a cloud-based or other multi-site application). A program has software instructions that are executed by one or more processors that may be distributed within the same computing device or across multiple computing devices.
[0021] The memory 140 may include a main memory and / or a static memory, wherein such memories may communicate with each other and with the processor 120 via one or more buses. For example, the memory 140 may be implemented by any number, type, and combination of random access memory (RAM) and read-only memory (ROM), and may store various types of information, such as software algorithms, artificial intelligence (AI) machine learning models, and computer programs, all of which may be executed by the processor 120. The various types of ROM and RAM may include any number, type, and combination of computer-readable storage media, such as disk drives, flash memory, electrically programmable read-only memory (EPROM), electrically erasable and programmable read-only memory (EEPROM), registers, hard disks, removable disks, magnetic tapes, compact disk read-only memory (CD-ROM), digital versatile disks (DVD), floppy disks, Blu-ray disks, universal serial bus (USB) drives, or any other form of storage media known in the art. The memory 140 is a tangible storage medium for storing data and executable software instructions, and is non-transitory during the time the software instructions are stored therein. As used herein, the term "non-transitory" is not to be construed as a permanent characteristic of a state, but rather as a characteristic of a state that will persist for a period of time. The term "non-transitory" specifically denies transient characteristics, such as characteristics of a carrier wave or signal or other form that only exists momentarily at any place at any time. Memory 140 may store software instructions and / or computer readable code capable of performing various functions. Memory 140 may be secure and / or encrypted, or unsecure and / or unencrypted.
[0022] The ultrasound imaging system 100 also includes or interfaces with one or more log file databases (indicated by log file database 116) for storing information that can be used by various software modules of the memory 140. For example, the log file database 116 can be implemented by any number, type, and combination of RAM and ROM. The various types of ROM and RAM can include any number, type, and combination of computer-readable storage media, such as disk drives, flash memory, EPROM, EEPROM, registers, hard disks, removable disks, tapes, CD-ROMs, DVDs, floppy disks, Blu-ray disks, USB drives, or any other form of storage media known in the art. The log file database 116 is a tangible storage medium for storing data and executable software instructions, and is non-transitory while the data and software instructions are stored therein. The log file database 116 can be secure and / or encrypted, or unsecure and / or unencrypted. For purposes of illustration, the log file database 116 is shown as a separate database, but it is understood that the log file database 116 may be combined with and / or included in the memory 140 without departing from the scope of the present teachings. The log file database 116 may be constructed as an item of routine at one or more facilities providing clinical care that stores at least patient demographic and clinical information.
[0023] The ultrasound imaging system 100 also includes a transducer probe 160. The transducer probe 160 may include a transducer array, which includes a two-dimensional transducer array capable of two-dimensional or three-dimensional scanning, for transmitting ultrasound waves into an object (patient) 165 and receiving echo information as a response. For example, the transducer array may include a capacitive micromachined ultrasound transducer (CMUT) or a piezoelectric transducer formed of a material such as PZT or PVDF. The transducer array is coupled to a microbeamformer in the transducer probe 160, which controls the transducer's reception of signals.
[0024] The memory 140 includes a probe interface module 141, which is used to interface the transducer probe 160 with the processor 120 to control the acquisition of an ultrasound image of the object 165. The probe interface module 141 may include a transmit / receive (T / R) switch of a microbeamformer coupled to the transducer probe 160 via a probe cable. For example, the T / R switch switches between a transmit mode and a receive mode under the control of the processor 120 and / or the user interface 122. The processor 120 also controls the direction in which the beam is steered and focused via the probe interface module 141. The beam can be steered to face directly in front of the transducer array (orthogonal to the transducer array), or steered at different angles to achieve a wider field of view. The processor 120 may also include a main beamformer that provides final beamforming after digitization. In general, the transmission of ultrasound waves and the reception of echo information are well known, so additional details about this aspect are not described in detail herein.
[0025] Processor 120 may include or have access to an AI engine or module that may be implemented as software that provides artificial intelligence (such as natural language processing (NLP) algorithms) and applies machine learning (such as neural network modeling), as described herein. For example, in addition to or different from processor 120, the AI engine may reside in any of a variety of components, such as memory 140, an external server, and / or a cloud. For example, when the AI engine is implemented in the cloud (such as at a data center), the AI engine may be connected to processor 120 via the Internet using one or more wired and / or wireless connections.
[0026] The user interface 122 is configured to provide information and data output by the processor 120 and / or the memory 140 to the user and / or to provide information and data input by the user to the processor 120 and / or the memory 140. That is, the user interface 122 enables the user to input data and control or manipulate various aspects of the processes described herein, and control or manipulate various aspects of ultrasound imaging. The user interface 122 also enables the processor 120 to indicate to the user the effects of the user's control or manipulation.
[0027] For example, all or a portion of the user interface 122 may be implemented by a graphical user interface (GUI), such as a GUI 128 on a touch screen 126 of the display 124, as discussed below. The user interface 122 includes push buttons that can be operated (pressed) by a user to initiate various commands for manipulating displayed images, performing measurements and calculations, etc. during an ultrasound examination. The push buttons may be displayed by the GUI 128 on the touch screen 126, or may be physical buttons, as discussed below. Figure 4An example of which is shown in . The buttons pressed by the user are recorded and stored in the log file database 116 as workflow and application scripts. For example, the user interface 122 may also include any other compatible interface device for performing ultrasound examinations, such as a mouse, keyboard, trackball, game controller, microphone, camera, touchpad, or voice or gesture recognition captured by a microphone or camera.
[0028] For example, the display 124 may be any compatible monitor for displaying ultrasound images, such as a computer monitor, a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, or a solid state display, for viewing internal images of the object 165. The display 124 includes a touch screen 126 and a GUI 128 to enable a user to interact with the displayed images and features.
[0029] With reference to the memory 140, the imaging module 142 is configured to receive and process the ultrasound image of the object 165 received from the transducer probe 160 and the probe interface module 141 for display on the display 124. For example, the processing may include, for example, bandpass filtering, decimation, I and Q component separation and harmonic signal separation, and signal enhancement, such as speckle reduction, signal compounding, and noise elimination. When the user manually manipulates the transducer probe 160 on the surface of the object 165, the current image is displayed on the touch screen 126. This enables the user to analyze the current image, adjust the position of the transducer probe 160 to acquire the desired image, and perform measurements and calculations on the touch screen 126.
[0030] The prediction model module 143 is configured to train and apply a prediction model that identifies an exam workflow for an ultrasound imaging exam and predicts the next one or more button presses in a sequence of button presses in the exam workflow based on one or more initial button presses, as discussed below. The prediction model takes into account the context of the ultrasound exam during which the sequence of button presses is obtained, including different clinical applications.
[0031] For example, the prediction model module 143 can be a deep learning neural network, such as a convolutional neural network (CNN), a recurrent neural network (RNN), an artificial neural network (ANN), or a transformer network. The prediction model can be trained using log file analysis of log files in the log file database 116. During training, the prediction model learns button press sequences associated with ultrasound images that are input to complete various tasks in an examination workflow of one or more clinical applications, as discussed below. Training can be performed by obtaining button press sequences performed by a user during an examination workflow of a previous ultrasound examination.
[0032] The NLP module 144 is configured to execute one or more NLP algorithms using word embedding techniques to identify button presses extracted from a sequence of button presses input by a user. Button presses are typically identified by descriptive text and must be converted into computer-readable data. NLP is well known and may include, for example, grammatical and semantic analysis and deep learning to improve the NLP module 144's understanding of the accumulation of data, which is apparent to those skilled in the art. The identified button presses may be provided to the prediction model module 143 (for training) and / or to the button prediction module 145, as discussed below, to convert the natural language identification of the button presses into computer-readable data. In various embodiments, for example, all or part of the processes provided by the NLP module 144 and / or the prediction model module 143 may be implemented by an AI engine executed by the processor 120.
[0033] The button prediction module 145 is configured to: obtain a sequence of button presses performed by a user via the control interface 125 during an examination workflow; distinguish components of the examination workflow based on the button press sequence; perform sequence pattern mining of the button press sequence based on the distinguished components of the examination workflow to extract user-specific workflow trends of the most frequently used button press sequences; predict the next string of button presses based on one or more previous button presses using a prediction model from the prediction model module 143; and, create and output a macro button corresponding to the predicted next string of button presses on the touch screen 126 of the control interface 125. The distinguished components of the examination workflow depend on the clinical application of the specific ultrasound examination. Once the output macro buttons are provided to the touch screen 126, the user can select them, for example, by touching designated areas of the touch screen 126 respectively, to execute the corresponding predicted next string of button presses. Refer to the following Figure 2 The functionality of the button prediction module 145 is described in more detail.
[0034] The probe motion module 146 is configured to detect probe motion of the transducer probe 160 during an ultrasound examination. For example, the transducer probe 160 may include one or more sensors, such as an electromagnetic (EM) sensor or an inertial measurement unit (IMU) sensor. The probe motion module 146 monitors the probe position of the transducer probe 160 by tracking position data provided by the sensors on the transducer probe 160. Detecting motion can be used to trigger sequential pattern mining and / or prediction of the next button press without further input from the user.
[0035] Figure 2is a flow chart showing a method for predicting a button press sequence during an ultrasound examination by a user using an ultrasound imaging system according to a representative embodiment. For example, the method may be implemented by the ultrasound imaging system 100 described above under the control of the processor 120 executing instructions stored in the memory 140 as various software modules.
[0036] refer to Figure 2 , the method includes: in box S211, initially training the prediction model by obtaining a sequence of button presses performed by a user during an examination workflow of a previous ultrasound examination. As discussed above, the button press sequence can be obtained based on previously stored log files in a log file database (e.g., log file database 116) and corresponding images from the examination workflow. Each sequence has a corresponding sequence length defined by the number of button presses in the button press sequence. The previous ultrasound examination may have been performed by the current user and / or by different users, all of which are preferably experts in ultrasound imaging technology.
[0037] The prediction model, which may be a deep learning neural network, may be trained using log file analysis. During training, the prediction model learns button press sequences associated with images, which are input to complete various tasks in the inspection workflow of one or more clinical applications. The training data includes button press sequences extracted and represented by the NLP module 144 from the inspection workflow log file. The prediction model may take the form of a condition generator that uses information about the inspection situation as an auxiliary input signal and uses an earlier button press sequence (a training sequence with a subsequent portion masked) as a primary signal to generate a button press sequence in a subsequent portion of the training data (a training sequence with an earlier portion masked) through hierarchical calculations of a neural network that enables information extraction, processing, and combination. The training process is achieved by back-propagating gradients of the neural network weights relative to a predefined objective function that measures the difference between the predicted output and the true button press sequence, which is calculated and executed by the processor 120. Learning enables the predictive model to identify relevant exam workflows and predict one or more next button presses of a sequence of button presses in those exam workflows based on one or more initial button presses earlier in the same sequence (considering the case of ultrasound exams, during which sequences of button presses are obtained, including different clinical applications). At inference time, the predictive model takes as input signals the exam case information and the length of one or more current button press sequences performed by the user to propagate forward through the neural network and compute a prediction of one or more next button presses.
[0038] It is noteworthy that, although depicted as the first step in the button press sequence prediction process, the prediction model need not be initially trained to perform each ultrasound examination. For example, training may be performed periodically to update the prediction model, which may then be applied to multiple different ultrasound examinations without further modification. Furthermore, in embodiments, the results of each ultrasound examination may be used to train the prediction model during subsequent training.
[0039] In block S212, a sequence of button presses performed by a user via a control interface (e.g., control interface 125) during a current inspection workflow is obtained. Each button press sequence corresponds to a component or task of an inspection workflow and has a sequence length defined by the number of button presses required to complete the task. The sequence length of a button press sequence may be determined by a specific gate, such as a user freezing or acquiring an image to isolate a set of related events.
[0040] Button presses may be obtained from a control interface and / or from a log file of the ultrasound imaging system. For example, button presses may be obtained by accessing a log file database and pairing an ultrasound image with a log file based on the current inspection workflow and the transition state of the ultrasound imaging system, as discussed below. Thus, a sequence of button presses is retrieved and organized so that the context of the current inspection workflow / image and transition state is attached to the extracted sequence of button presses. The transition states are also relevant for the component distinctions discussed below because they indicate which state the system is in in order to identify how to use a sequence of button presses. The sequence of button presses will be used to infer or predict future button presses using a predictive model previously trained in box S211, as discussed below. The inspection workflow depends on the ultrasound image content and the clinical application of ultrasound examination, as discussed below.
[0041] In box S213, components (or tasks) of the inspection workflow are distinguished based on the button pressing sequence and conversion state of the ultrasound imaging system during the inspection workflow. Distinguishing the components includes: transforming the button pressing sequence obtained in box S212 into a corresponding encoding vector; clustering the encoding vector according to the purpose; and dividing the clustered encoding vector into one of two conversion states. The two conversion states include a "live state" and a "frozen state". In the live state, the ultrasound image is displayed in real time, so that the user can observe the anatomical structure of the object and interactively adjust the positioning of the transducer probe and / or the acoustic settings of the ultrasound imaging system in real time to obtain the desired ultrasound image. In the frozen state, the currently displayed ultrasound image is frozen, so that the user can, for example, view the ultrasound image and / or measure a portion of the anatomical structure in the ultrasound image. The user can also perform calculations based on the measurements.
[0042] With respect to encoding vectors, buttons and / or button presses are actually "vocabulary" for providing interaction with a user via a control interface. These vocabularies are transformed into encoding vectors so that they become computer-readable sequences. The length of each encoding vector is determined by the number of button presses in the corresponding button press sequence. In order to transform the button press sequence into an encoding vector, the ultrasound image is first paired with a log file based on the inspection workflow and the conversion state of the ultrasound imaging system. The button press sequence is then transformed into an encoding vector using, for example, NLP-based techniques, as discussed below. In an embodiment, the user ID of the user can also be extracted from the log file, so that the inspection workflow can be customizable.
[0043] The component distinction is influenced by the clinical application of the ultrasound examination. Typically, based on the anatomy to be examined and the purpose of the examination, the clinical application includes an associated scan protocol in which the components of the examination workflow correspond to steps in the scan protocol, respectively, but the specific examination workflow / button presses may vary in order to complete the protocol. For example, an examination workflow for examining the liver follows a different scan protocol than an examination workflow for examining the bladder. Furthermore, in particular for liver imaging, the scan protocol used for routine studies is different from the scan protocol used, for example, for fibrosis grading quantification.
[0044] Figure 3A and 3B A schematic diagram is provided that illustrates an example of differentiating components of an examination workflow during an ultrasound examination according to a representative embodiment.
[0045] Figure 3A An illustrative button press sequence 311 is shown, which is a vocabulary set indicating a corresponding command string entered by a user for interacting with an ultrasound imaging system. Each group of vocabulary enclosed by a pair of brackets corresponds to a component of the inspection workflow. The button press sequence 311 is transformed into a corresponding encoding vector 312, as shown by arrow A1. As shown, the encoding vector 312 is a single row (or column) digital matrix representing the vocabulary groups in the button press sequence 311 respectively. The encoding vector 312 is computer readable, which enables subsequent steps in distinguishing the components.
[0046] The encoded vectors 312 are clustered into clusters 313, consisting of Figure 3B The clustering process is performed using an unsupervised dimensionality reduction algorithm that projects high-dimensional data onto a two-dimensional manifold while maximally preserving the topological structure of the original data point set. The relative topological structures shown (which may include two or more clusters) indicate different uses or components, depending on the specific clinical application. In the simplified general case used for illustration, Figure 3BThe two clusters 313 represented as Seqs_upper_cluster and Seqs_bottom_cluster in FIG. 3 correspond to the live state 314 and the frozen state 315, respectively. The horizontal and vertical axes of clusters 313 represent the x-coordinates and y-coordinates of the eigenvectors in the two-dimensional manifold, so the corresponding values are relative. In other various cases where button press sequence data is collected across different examination situations or applications, the encoding vectors can be clustered according to more than two different uses or components.
[0047] The cluster 313 is divided into a frozen state 314 and a live state 315, by Figure 3B 314 and the live state 315. Indicated by arrow A3 in. This is done by collecting the corresponding raw sequence data according to clusters 313 and counting the frequency of the unique button string from each sequence of each cluster, after which clusters 313 can be divided into different states by visualization and analysis of frequency graphs. In each state in frozen state 314 and live state 315, the corresponding button presses are indicated by frequency, and for the purpose of illustration (in each of the two distinguished components), the most frequently used button presses have been marked. Therefore, for example, the most frequently used button presses in frozen state 314 are Bm_Icon, EVTMGR_CP_DEPTH_CHANGE, EVTMGR_CP_FREEZE_PUSH and Knob_Bm_RotateProbe, and the most frequently used button presses in live state 315 are EVTMGR_CP_FREEZE_PUSH, TGC, Depth and Gain. For each state, the horizontal axis represents the index of each unique button name, and the vertical axis represents the frequency of each sequence, which is the average number of events of the button name of each sequence.
[0048] return Figure 2 , in box S214, sequential pattern mining of button press sequences is performed based on the distinguished components of the examination workflow. Sequential pattern mining extracts the patterns of the most frequently used strings of consecutive button presses during ultrasound examinations in order to identify user-specific workflow trends. Therefore, sequential pattern mining can identify K workflow trends from the K most frequent usage patterns, where K is a positive integer. The workflow trends followed by users can be identified based on the button presses and the real-time imaging context of clinical applications. Sequential pattern mining is performed on a recent dataset of user-specific button press sequences from past examinations, which can be updated after a certain number of new examinations are completed.
[0049] As an example, for a clinical application involving abdominal ultrasound examination of the aortic bifurcation, sequential pattern mining of button press sequences might produce the following ten workflow trends, ranked from most frequently used to less frequently used:
[0050] 1:[‘EVTMGR_CP_FREEZE_PUSH’,‘EVTMGR_CP_ANNOTATE_PUSH’,‘SublLabel_Label’,‘EVTMGR_C_FREEZE_PUSH’]
[0051] 2:[‘EVTMGR_CP_FREEZE_PUSH’,‘Bm_Icon’,‘Bm_Icon’,‘EVTMGR_C-_FREEZE_PUSH’]
[0052] 3:[‘EVTMGR_CP_FREEZE_PUSH’,‘EVTMGR_CP-ACUIRE1_PUSH’,‘EVTMGR_CP-ACUIRE1_PUSH’,‘EVTMGR_CP_FREEZE_PUSH’]
[0053] 4:[‘EVTMGR_CP_FREEZE_PUSH’,‘SublLabel_Label’,‘SublLabel_Label’,‘EVTMGR_CP_FREEZE_PUSH’]
[0054] 5:[‘EVTMGR_CP_FREEZE_PUSH’,‘EVTMGR_CP_DEPTH_CHANGE’,‘EVTMGR_CP_DEPTH_CHANGE’,‘EVTMGR_CP_FREEZE_PUSH’]
[0055] 6:[‘EVTMGR_CP_FREEZE_PUSH’,‘SublLabel_Label’,‘Btn_Annot_Canned’,‘EVTMGR_CP_FREEZE_PUSH’]
[0056] 7:[‘EVTMGR_CP_FREEZE_PUSH’,‘SublLabel_Label’,‘EVTMGR_CP_ACUIRE1_PUSH’,‘EVTMGR_CP_FREEZE_PUSH’]
[0057] 8:[‘EVTMGR_CP_FREEZE_PUSH’,‘Btn_Annot_Canned’,‘SublLabel_Label’,‘EVTMGR_CP_FREEZE_PUSH’]
[0058] 9:['EVTMGR_CP_FREEZE_PUSH','Bm_Icon','Bm_Icon','Bm_Icon']
[0059] 10:['EVTMGR_CP_FREEZE_PUSH','Bm_Icon','EVTMGR_CP_DEPTH_CHANGE','EVTMGR_CP_FREEZE_PUSH']
[0060] In one embodiment, it may be necessary to project the log file database into a smaller space of subsequences in order to apply pattern constraints (such as gap constraints and sequence length constraints) to sequence pattern mining. The gap constraint ensures that each of the most frequent workflow trends will include gaps between buttons that cannot be selected continuously by the user in a real ultrasound examination scenario. For example, the Freeze_Push button is the frequency with which the user uses to convert the ultrasound imaging system from the live state to the frozen state, and vice versa. However, even when the Freeze_Push button is frequently used during the examination, the user rarely presses it multiple times in a row. Adding a gap constraint eliminates repeated or abnormal sequences that may contain multiple consecutive identical button strings (which is not very reasonable from a clinical perspective), thereby improving the accuracy of sequence pattern mining. The sequence length constraint applies a pre-selected minimum sequence length to sequence pattern mining based on user input and clinical application context. Typically, smaller button press sequences produce more usage patterns, but may not be very meaningful in the workflow. Therefore, when determining the most frequent workflow trends, sequences with fewer button presses than the pre-selected minimum are not considered.
[0061] In another embodiment, sequential pattern mining can be performed using weighted button presses. Weighting can be based on user preferences, which depends on the clinical context, to reflect different examination workflow approaches within the selected workflow trend. For example, a user may prefer transverse views of certain organs and longitudinal views of other organs. The extracted button press patterns are weighted based on the clinical application as well as the clinical context, for example, the clinical application can refer to the general type of examination being performed, and the clinical context can refer to the more specific imaging environment that the system can recognize. For example, different weights can be assigned to different button presses when the user is performing an abdominal examination for screening purposes compared to a follow-up abdominal examination. Different weights are applied to sequential pattern mining to identify favorable trends.
[0062] In block S215, probe motion of the transducer probe is detected during the ultrasound examination. When performing an ultrasound examination according to a scanning protocol, a user moves the ultrasound transducer probe over the body surface of the subject while viewing image content to investigate anatomical structures and find the best imaging view. When probe tracking is enabled, movement of the transducer probe is detected, for example, using (one or more) sensors attached to the transducer probe, as discussed above.
[0063] Certain predetermined movements or movement patterns can be used to trigger predictions of the next button press sequence for the ultrasound examination. For example, as the user gets closer to a view of interest in the subject, the probe motion becomes more stable (e.g., slower, less frequent movements over smaller distances). In addition, the probe motion typically stops at (one or more) anatomical landmarks that indicate the anatomical structure of interest being studied, where qualitative and quantitative analysis is to be performed. Thus, a prediction can be triggered based on the detection of more stable and / or stopped probe motion.
[0064] In block S216, the prediction model trained in block S211 is used to predict the next string of button presses based on at least one previous button press in the user-specific workflow trend, respectively. Typically, the prediction model predicts the next M buttons to be pressed in sequence to perform an ultrasound examination based on the N previous button presses, where M and N are positive integers.
[0065] For example, the prediction model can be an autoregressive model and can be used with a decoder. The decoder is configured to decode a computer-readable sequence of prediction vectors (in matrix form) into a human-interpretable sequence of button strings or names. The log files and button presses associated with the workflow trends provided by the sequence pattern mining are divided into smaller subsets (batches) of the button press sequences so that the individual button presses of each button press sequence can be considered. The division of the log files and button presses into subsets can be "learned" by the prediction model, for example, during a training process or "customized" by the user. That is, the prediction model or the user can select (or pre-select) the length of the string for the prediction task (the number of next button presses to be predicted). In an embodiment, the user's pre-selection can be provided based on the user preferences identified using the user ID and real-time image content extraction as discussed above. Alternatively, the user's pre-selection can be retrieved from the default settings of the ultrasound imaging system, or the user's pre-selection can be directly input by the user from the control interface (e.g., using a dedicated knob or other input).
[0066] For example, when the user makes a preselection of N=3 previous button presses, the prediction model predicts the next button press based on the user's most recent three previous button presses. The accuracy and confidence of predicting each next sequence of button presses depends on the values of the N previous button presses (considered as input) and the values of the predicted M next button presses for the length of the predicted sequence. In general, the larger the number of previous button presses N and the smaller the number of predicted next buttons M, the greater the prediction accuracy and confidence level, and vice versa. As discussed above, the prediction can be triggered by certain probe movements, such as the transducer probe making a more stable movement or about to stop.
[0067] In frame S217, output the macro button corresponding to the predicted next button press string on the control interface.For example, a macro button can be created and displayed on the touch screen of the control interface together with a label indicating the associated function of the predicted next button press.Then, the user can select the macro button to execute the corresponding predicted next button press string.Use the macro button to cause the faster and more consistent execution of the clinical application specific workflow.In an embodiment, for example, the macro button can be presented as different colors, which represents the different confidence levels associated with the predicted next button string.
[0068] Figure 4 is a plan view of a control interface in an ultrasound imaging system, the control interface including a macro button corresponding to a predicted next string of button presses, according to a representative embodiment. Figure 4 , a control interface 425 (e.g., which may correspond to the control interface 125) includes a touch screen 420 that displays macro buttons corresponding to the predicted next string of button presses. The control interface 425 also includes standard slider controls 1-8 and buttons 9-35 corresponding to predetermined functions. In the depicted example, the touch screen 420 provides a first macro button 421 labeled Freeze 2D Depth change, a second macro button 422 labeled FreezeAcquire1 Push, and a third macro button 423 labeled Freeze Annotation push. In addition, in the depicted example, the first macro button 421 and the second macro button 422 are displayed in a first color (e.g., green) indicated by a light shade to show a high confidence level of the associated predicted next string of button presses, while the third macro button 423 is displayed in a second color (e.g., yellow) indicated by a darker shade to show a medium confidence level of the associated predicted next string of button presses.
[0069] According to various embodiments of the present disclosure, the methods described herein may be implemented using a hardware computer system that executes a software program stored on a non-transitory storage medium. In addition, in an exemplary, non-limiting embodiment, implementations may include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing may implement one or more of the methods or functions described herein, and the processors described herein may be used to support a virtual processing environment.
[0070] Although the predictive button press sequence has been described with reference to exemplary embodiments, it should be understood that the words that have been used are words of description and illustration, rather than of limitation. Changes may be made within the scope of the appended claims, as presently stated and amended, without departing from the scope and spirit of the interventional process optimization in its various aspects. Moreover, although the predictive button press sequence has been described with reference to specific units, materials, and embodiments, it is not intended to be limited to the details disclosed; rather, the embodiments extend, for example, to all functionally equivalent structures, methods, and uses within the scope of the appended claims.
[0071] The illustrations of the embodiments described herein are intended to provide a general understanding of the structures of various embodiments. These illustrations are not intended to be used as a complete description of all elements and features of the disclosure described herein. After reading the disclosure, many other embodiments may be apparent to those skilled in the art. Other embodiments may be utilized and derived from the disclosure so that structural and logical replacements and changes may be made without departing from the scope of the disclosure. In addition, the illustrations are merely representative and may not be drawn to scale. Certain ratios within the illustrations may be exaggerated, while other ratios may be minimized. Therefore, the disclosure and the accompanying drawings are considered to be illustrative rather than restrictive.
[0072] One or more embodiments of the present disclosure may be referred to herein individually or collectively by the term "invention", which is intended to limit the scope of the present application to any particular invention or inventive concept only for the purpose of convenience. Moreover, although specific embodiments are illustrated and described herein, it should be appreciated that any subsequent arrangement designed to achieve the same or similar purpose may be replaced for the specific embodiments shown. The present disclosure is intended to cover any and all subsequent adjustments or variations of the various embodiments. After reading the description, combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art.
[0073] The Abstract of the Disclosure is provided to comply with 37.FR §1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, various features may be grouped together or described in a single embodiment for the purpose of simplifying the disclosure. This disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than those expressly recited in the claims. On the contrary, as reflected in the appended claims, the inventive subject matter may involve less than all of the features of any disclosed embodiment. Therefore, the appended claims are incorporated into the Detailed Description, with each claim independently defining a separately claimed subject matter.
[0074] The foregoing description of the disclosed embodiments is provided to enable those skilled in the art to practice the concepts described in the present disclosure. Therefore, the subject matter disclosed above is to be considered illustrative rather than restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments that fall within the true spirit and scope of the present disclosure. Therefore, to the maximum extent permitted by law, the scope of the present disclosure will be determined by the broadest permissible interpretation of the appended claims and their equivalents, and shall not be restricted or limited by the foregoing detailed description.
Claims
1. A method for performing an ultrasound examination using an ultrasound imaging system (100), the ultrasound imaging system (100) comprising: Transducer probe (160); and, a control interface (125) for controlling the acquisition of ultrasound images during the ultrasound examination, the method comprising: obtaining a sequence of button presses performed by a user via the control interface during the inspection workflow, each button press sequence having a corresponding sequence length defined by the number of button presses in the button sequence (S212); Distinguishing components of the examination workflow based on the button pressing sequence, wherein the distinguished components depend on the clinical application of the ultrasound examination (S213); performing sequential pattern mining of the button pressing sequences based on the distinguished components of the inspection workflow to extract user-specific workflow trends, wherein the user-specific workflow trends include a plurality of most frequently used button pressing sequences (S214); detecting probe movement of the transducer probe during the ultrasound examination (S215); using a prediction model to predict a next string of button presses based on at least one previous button press in the user-specific workflow trend, respectively, wherein the next string of button presses is predicted to be triggered by the detected probe motion (S216); and At least one macro button corresponding to the predicted next string of button presses on the control interface is output, wherein selection of the at least one macro button by the user executes the corresponding predicted next string of button presses (S217).
2. The method according to claim 1, wherein: Obtaining the button pressing sequence includes: accessing a log file of the ultrasound imaging system; and Ultrasound images are paired with the log file based on the examination workflow and a transition state of the ultrasound imaging system.
3. The method according to claim 2, wherein: The conversion states of the ultrasound imaging system include: a live state for providing an ultrasound image in real time; and a freeze state for freezing the ultrasound image so that the user can view and / or measure a portion of the ultrasound image.
4. The method according to claim 2, wherein: The components that distinguish the inspection workflow include: Transform each button press sequence into an encoding vector; Clustering the encoding vectors; and The clustered code vectors are divided into the transition states.
5. The method according to claim 1, wherein: Executing the sequential pattern mining includes: A gap constraint is applied to the extracted user-specific workflow trends to skip buttons that are less meaningful in the button press sequence.
6. The method according to claim 1, wherein: Executing the sequential pattern mining includes: A sequence length constraint is applied to the extracted user-specific workflow trend to exclude each button press sequence of the most frequently used button press sequences that is less than a predetermined minimum sequence length.
7. The method according to claim 1, wherein: Executing the sequential pattern mining includes: Different weights are assigned to the user-specific workflow trends based at least in part on the clinical use of the ultrasound examination.
8. The method according to claim 1, wherein: Detecting the probe motion of the transducer probe includes: monitoring the transducer probe using an external camera; and determining the probe motion based on an image provided by the external camera.
9. The method according to claim 1, wherein: Detecting the probe motion of the transducer probe includes: monitoring the transducer probe using at least one sensor on the transducer probe; and determining the probe motion based on position data provided by the at least one sensor.
10. The method according to claim 9, wherein: The at least one sensor includes at least one of an electromagnetic (EM) sensor or an inertial measurement unit (IMU) sensor.
11. The method according to claim 1, wherein: The control interface includes a touch screen display.
12. The method according to claim 11, wherein: The at least one macro button is displayed on the touch screen in a color among a plurality of different colors, the plurality of different colors corresponding to a plurality of confidence levels associated with the at least one macro button.
13. A system for performing an ultrasound examination, comprising: An ultrasound imaging system (100) comprising: a transducer probe (160); and a control interface (125) for controlling the acquisition of ultrasound images during said ultrasound examination; a display (124) configured to display the ultrasound image; at least one processor (120) coupled to the ultrasound imaging system and the display; and A non-transitory memory (140) for storing instructions which, when executed by the at least one processor, cause the at least one processor to: obtaining a sequence of button presses performed by a user via the control interface during the inspection workflow, each button press sequence having a corresponding sequence length defined by the number of button presses in the button sequence (S212); Distinguishing components of the examination workflow based on the button pressing sequence, wherein the distinguished components depend on the clinical application of the ultrasound examination (S213); performing sequential pattern mining of the button pressing sequences based on the distinguished components of the inspection workflow to extract user-specific workflow trends, wherein the user-specific workflow trends include a plurality of most frequently used button pressing sequences (S214); using a prediction model to predict a next string of button presses based on at least one previous button press in the user-specific workflow trend, respectively (S216); and At least one macro button corresponding to the predicted next string of button presses on the control interface is output on the display, wherein selection of the at least one macro button by the user executes the corresponding predicted next string of button presses (S217).
14. The system according to claim 13, wherein: The instructions further cause the at least one processor to: detecting probe motion of the transducer probe during the ultrasound examination; and In response to detected probe motion, the prediction of the next string of button presses is triggered.
15. The system of claim 14, wherein: The instructions cause the at least one processor to detect the probe motion of the transducer probe by: monitoring the transducer probe using at least one sensor on the transducer probe; and determining the probe motion based on position data provided by the at least one sensor.
16. The system of claim 13, wherein: The instructions cause the at least one processor to obtain the button press sequence by: accessing a log file of the ultrasound imaging system; and Ultrasound images are paired with the log file based on the examination workflow and a transition state of the ultrasound imaging system.
17. The system of claim 16, wherein: The conversion states of the ultrasound imaging system include: a live state for providing an ultrasound image in real time; and a freeze state for freezing the ultrasound image so that the user can view and / or measure a portion of the ultrasound image.
18. The system of claim 16, wherein: The instructions cause the at least one processor to distinguish the components of the inspection workflow by: Transform each button press sequence into an encoding vector; Clustering the encoding vectors; and The clustered code vectors are divided into the transition states.
19. A non-transitory computer readable medium (140) storing instructions for performing an ultrasound examination, which when executed by at least one processor causes the at least one processor to: obtaining a sequence of button presses performed by a user via a control interface during an examination workflow, the control interface being configured to interface with a transducer probe during the ultrasound examination, each sequence of button presses having a corresponding sequence length defined by a number of button presses in the sequence of button presses; The components of the inspection workflow are distinguished based on the button press sequence, wherein: The components distinguished depend on the clinical application of the ultrasound examination in question; performing sequential pattern mining of the button press sequences based on the distinguished components of the inspection workflow to extract user-specific workflow trends, wherein the user-specific workflow trends include a plurality of most frequently used button press sequences; using a prediction model to predict a subsequent string of button presses based on at least one previous button press in the user-specific workflow trend, respectively; and At least one macro button corresponding to the predicted next string of button presses on the control interface is output on a display, wherein selection of the at least one macro button by the user executes the corresponding predicted next string of button presses.
20. The non-transitory computer readable medium of claim 19, wherein: The instructions further cause the at least one processor to: detecting probe motion of the transducer probe during the ultrasound examination; and In response to detected probe motion, the prediction of the next string of button presses is triggered.
