Systems and methods for automated state estimation for current imaging examination using user actions on console

Abstract

A non-transitory computer readable medium (26s) stores instructions executable by at least one electronic processor (14s) to perform a method (100) that provides assistance during a medical imaging examination performed using a medical imaging device (2). The method comprises: acquiring a video (17) of the medical imaging examination; determining, using a state machine (40) implemented in the at least one electronic processor, a current state of the imaging examination from the captured video; and displaying an indication (46) of the determined current state of the imaging examination on an electronic processing device (8).

Description

Technical Field

[0001] The following generally relates to imaging technology, remote imaging assistance technology, remote imaging inspection monitoring technology and related technologies. Background Art

[0002] Medical imaging (such as computed tomography (CT) imaging, magnetic resonance imaging (MRI), positron emission tomography (PET) imaging, fluoroscopic imaging, etc.) is a key component of providing medical care and is used in a wide range of medical fields, such as cardiology, oncology, neurology, orthopedics, etc. The operator of the medical imaging equipment used to acquire the medical images is typically a trained technician, and the interpretation of the medical images is typically handled by a medical specialist such as a radiologist. The interpretation of the radiology report or findings by the radiologist may be handled by the patient's general practitioner (GP) or a medical specialist such as a cardiologist, oncologist, orthopedic surgeon, etc.

[0003] Currently, diagnostic imaging is in high demand. As the world population grows, the need for fast, safe, high-quality imaging will only continue to grow, placing further pressure on imaging centers and their staff. In this context, errors are inevitable but often costly. One way for imaging centers to improve efficiency and grow operations without additional labor costs is through a Radiology Operations Command Center (ROCC) system. A Radiology Operations Command Center enables teams to work across a network of imaging sites, providing their expertise as needed and remotely assisting less experienced technicians in performing high-quality scans. Remote technicians or experts can monitor the local operator during the scanning process through cameras installed in the scanning area (or from other sources such as sensors (including radar sensors), console video feeds, microphones connected to Internet of Things (IoT) devices, etc.). In addition, these sources can be supplemented by other data sources such as Health-Level 7 (HL7), Digital Imaging and Communications in Medicine (DICOM), Electronic Health Record (EHR) databases, etc.

[0004] It is contemplated that a remote technician (i.e., a "super technician"; also referred to herein as an "expert technician" or remote expert) is assigned simultaneously to assist multiple different imaging bays at different sites, which may be located in different cities or different states. However, in practice, a super technician focuses on only a single imaging bay at any given time. A super technician will typically assist a local technician who proactively requests super technician support. However, situations may arise where the assistance of a super technician would be beneficial, but the local technician is unaware of the need for super technician assistance, or chooses not to request such assistance.

[0005] During image acquisition using MR or CT, a user (or technician) may perform a wide range of activities, including planning scans, reviewing images from current / past exams, adding / repeating sequences based on the patient and the context of the current exam. The asynchronous nature of these activities makes it difficult to determine the current status of the exam. The lack of current status of the exam prevents expert users from proactively providing assistance to remote / local technicians.

[0006] Certain improvements that overcome these and other problems are disclosed below. Summary of the invention

[0007] In one aspect, a non-transitory computer-readable medium stores instructions executable by at least one electronic processor to perform a method of providing assistance during a medical imaging examination performed using a medical imaging device. The method includes: acquiring a video of the medical imaging examination; determining a current state of the imaging examination based on the acquired video using a state machine implemented in the at least one electronic processor; and displaying an indication of the determined current state of the imaging examination on an electronic processing device.

[0008] On the other hand, a method for providing assistance during a medical imaging examination performed using a medical imaging device comprises: acquiring a video of the medical imaging examination; tracking the progress of the medical imaging examination using a state machine representing a workflow of the medical imaging examination, the progress being tracked based at least on matching information extracted from the acquired video with state information of the state machine; and performing an assistance action based on the tracked progress of the medical imaging examination to provide assistance during the medical imaging examination.

[0009] In another aspect, a non-transitory computer-readable medium stores instructions executable by at least one electronic processor to perform a method of providing assistance during a medical imaging examination performed using a medical imaging device. The method includes: acquiring a video of the medical imaging examination; determining a current state of the imaging examination based on the acquired video using a state machine implemented in at least one electronic processor; determining an event of the medical imaging examination that triggers a transition from the current state of the medical imaging examination to the next state of the medical imaging examination based on a state transition of the state machine from the current state to a next state; and displaying an indication of the determined current state of the imaging examination on an electronic processing device.

[0010] One advantage resides in providing alerts to a remote expert of events that occur during a procedure that may be performed by a local operator.

[0011] Another advantage resides in determining the status of an imaging examination.

[0012] Another advantage resides in providing an automated method of capturing the status of an imaging examination based on actions performed by a user on an imaging device console.

[0013] Another advantage resides in providing tracking the progress of a medical imaging examination using a state machine representing the workflow of the medical imaging examination, thereby enabling fine-grained detection of complex events that may occur during a given medical imaging examination.

[0014] Another advantage resides in providing assistance to a local operator performing a medical imaging examination based on such tracked progress.

[0015] Another advantage resides in collecting data regarding the performance of a local operator performing a medical imaging examination based on such tracking progress.

[0016] Another advantage resides in analyzing time series of exam status and patient / exam characteristics to provide reliable quantification of technologist expertise, which enables operation managers to not only effectively use their technologist pool, but also adhere to standard practices followed at their respective medical facilities.

[0017] Another advantage resides in improving efficiency in handling patients during imaging examinations.

[0018] Another advantage resides in that a technician checks compliance with a policy for an imaging exam while performing the imaging exam.

[0019] A given embodiment may provide none of the aforementioned advantages, provide one, two, more, or all of the aforementioned advantages, and / or may provide other advantages that will be apparent to one of ordinary skill in the art upon reading and understanding the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The disclosure may take form in various components and arrangements of components, and in various steps and arrangements of steps.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the disclosure.

[0021] Figure 1 An illustrative apparatus for providing remote assistance in accordance with the present disclosure is schematically shown.

[0022] Figure 2 Shown by Figure 1 An example flow chart of operations appropriately performed by a device.

[0023] Figure 3 An illustrative simplified state machine representing a medical imaging examination workflow is schematically shown. DETAILED DESCRIPTION

[0024] The following relates to a ROCC framework developed to enable communication between a local imaging technician performing an imaging examination and a remote expert, wherein a state machine is used to determine the current state of the examination.

[0025] The state machine may be constructed manually, and is typically specific to a particular imaging device brand and model (or model series, if that may be appropriate), software version (if that may be appropriate), and a particular imaging workflow (e.g., brain scan, whole body scan, etc.). Manual construction of the state machine is feasible because there are a relatively small number of states in any given workflow, and the computerized workflow implemented by the imaging device controller transitions between well-defined states with corresponding user interface (UI) dialogs and / or UI dialog content. Since the states will be evaluated based on information collected from captured controller screens, state transitions are identified by changes in displayed content, such as switching from one UI dialog to another, detecting the opening of a pop-up window, identifying newly displayed text, etc. Each state is characterized by a vector of values ​​of state variables.

[0026] During an imaging session, the captured screens are monitored. An initial state may be identified by a known startup UI dialog, e.g., via which an imaging technician enters patient and scan information. From the initial state, transitions between states of a state machine representing a medical imaging exam workflow are identified based on changes in the captured screens, as just described. For increased efficiency, various pipelines for detecting changes in display content corresponding to exam state changes may be filtered, e.g., applying OCR only to text that has changed.

[0027] The resulting real-time tracking of the progress of the medical imaging examination provides a rich source of information that can be used in real time during the examination to guide the ROCC operation. For example, if the examination remains in a patient loading state for too long, this may trigger an alert to the remote expert to contact the imaging technician to see if assistance is needed. For another example, if the medical imaging examination makes a "backward" transition through the state machine, such as from a state representing image acquisition to a state representing patient positioning, this can indicate an event, such as a problem that occurred in the medical imaging examination. Such detected events can be used to trigger auxiliary actions to assist the local operator, such as establishing a natural communication path (e.g., a video call) between the local operator performing the medical imaging examination and the remote expert, or automatically providing text, graphics, video and / or multimedia guidance about the determined event.

[0028] Real-time tracking of the progress of a medical imaging examination in terms of states traversed, duration in each state, transitions between states, etc. can be recorded for most or all imaging examinations performed by a radiology department or other entity and can be stored and subsequently mined for various purposes, such as detecting performance deficiencies of a particular imaging technician to identify where the technician requires further training, identifying where the radiology laboratory workflow is inefficient (e.g., if torso imaging exams frequently cycle back to re-acquisition of images requiring the patient to hold their breath, this may indicate that the workflow used to guide patients in breath holding should be reviewed), etc.

[0029] refer to Figure 1 , shows an apparatus 1 for providing assistance from a remote medical imaging expert RE (or super technician) to a local technician operator LO. Figure 1 As shown, a local operator LO operating a medical imaging device (also referred to as an image acquisition device, imaging device, etc.) 2 is located in a medical imaging device compartment 3, and a remote expert RE is provided in a remote service location or center 4. It should be noted that the "remote expert" RE may not necessarily directly operate the medical imaging device 2, but rather provide assistance to the local operator LO in the form of advice, guidance, instructions, etc. The remote location 4 may be a remote service center, a radiologist's office, a radiology department, etc. The remote location 4 may be in the same building as the medical imaging device compartment 3 (for example, this may be the case when the "remote operator or expert" RE is a radiologist responsible for peri-examination image review), but more typically, the remote service center 4 and the medical imaging device compartment 3 are in different buildings, and may actually be located in different cities, different countries, and / or different continents. Typically, the remote location 4 is remote from the imaging device compartment 3 in the sense that the remote expert RE cannot directly visually observe the imaging device 2 in the imaging device compartment 3 (thus optionally providing a video feed as further described herein).

[0030] The image acquisition device 2 may be a magnetic resonance (MR) image acquisition device, a computed tomography (CT) image acquisition device; a positron emission tomography (PET) image acquisition device; a single photon emission computed tomography (SPECT) image acquisition device; an X-ray image acquisition device; an ultrasound (US) image acquisition device; or another modality of medical imaging device. The imaging device 2 may also be a hybrid imaging device, such as a PET / CT or SPECT / CT imaging system. Although in Figure 1A single image acquisition device 2 is shown by way of illustration, but more typically a medical imaging laboratory will have multiple image acquisition devices, which may have the same and / or different imaging modalities. For example, if a hospital performs many CT imaging examinations and relatively few MRI examinations and even fewer PET examinations, the hospital's imaging laboratory (sometimes referred to as a "radiology laboratory" or some other similar term) may have three CT scanners, two MRI scanners, and only a single PET scanner. This is merely an example. In addition, the remote service center 4 may provide services to multiple hospitals. A local operator controls the medical imaging device 2 via an imaging device controller 10. The remote operator is located at a remote electronic processing device 12 (or more generally, an electronic controller 12).

[0031] To provide optional contrast-enhanced imaging, an optional contrast medium injector 11 is configured to inject contrast medium into a patient. The contrast medium injector 11 is a configurable automatic contrast medium injector with a display 13. A user (typically an imaging technician) loads a bottle or tube of contrast medium (or two or more bottles of different contrast medium components) into the contrast medium injector 11 and configures the contrast medium injector 11 by inputting contrast medium injector settings (such as flow rate, volume, time delay, injection duration, etc.) via a user interface (UI) of the contrast medium injector 11. The UI can be a touch-sensitive overlay and / or physical buttons, keypads, and / or other of the display 13. In a variant embodiment, the contrast medium injector 11 is integrated with the imaging device controller 10 (e.g., via a wired or wireless data connection), and the contrast medium injector 11 is controlled via the imaging device controller 10, including displaying the contrast medium injector settings in an (optionally selectable) window on the display of the imaging device controller 10.

[0032] As used herein, the term "medical imaging equipment bay" (and variations thereof) refers to a room containing a medical imaging device 2 and any adjacent control room containing a medical imaging device controller 10 for controlling the medical imaging device. For example, with reference to an MRI device, the medical imaging equipment bay 3 may include a radio frequency (RF) shielded room containing the MRI device 2, and an adjacent control room housing the medical imaging device controller 10, as is understood in the art of MRI equipment and procedures. On the other hand, for other imaging modalities such as CT, the imaging device controller 10 may be located in the same room as the imaging device 2, such that there is no adjacent control room, and the medical bay 3 is simply the room containing the medical imaging device 2. Furthermore, although Figure 1A single medical imaging equipment bay 3 is shown, but it should be understood that the remote service center 4 (and more specifically the remote electronic processing device 12) communicates with multiple medical bays via a communication link 14, which typically includes the Internet enhanced by a local area network at the remote expert RE and the local operator LO for electronic data communication. In addition, although Figure 1 A single remote service centre 4 is shown, but it will be appreciated that the medical imaging device bay 3 communicates with a plurality of medical bays via the communication link 14 .

[0033] like Figure 1 As schematically shown in FIG. 1 , in some embodiments, a camera 16 (e.g., a video camera) is arranged to capture a video stream or feed (i.e., video) 17 of a portion of a workspace of the medical imaging device compartment 3, the portion of the workspace including at least the area of ​​the imaging device 2 where the local operator LO interacts with the patient, and optionally may also include the imaging device controller 10. In some embodiments, an optional microphone 15 is arranged to capture an audio stream or feed 18 of the workspace, which includes audio noise occurring within the medical imaging device compartment 3 (e.g., verbal instructions of the local operator LO, questions from the patient, etc.). The video 17 and / or audio 18 are sent to the remote electronic processing device 12 via the communication link 14, for example as a streaming video feed received via a secure Internet link.

[0034] The communication link 14 also provides a natural language communication path 19 for oral and / or textual communication between the local operator and the remote operator. For example, the natural language communication link 19 can be a Voice-Over-Internet-Protocol (VOIP) telephone connection, an online video chat link, a computerized instant messaging service, etc. Alternatively, the natural language communication path 19 can be provided by a dedicated communication link separate from the communication link 14 providing data communications 17, 18, for example, the natural language communication path 19 can be provided via a landline telephone. In some embodiments, the natural language communication link 19 allows the local operator LO to call a selected remote expert RE. As used herein, a call can refer to an audio call (e.g., a telephone call), a video call (e.g., Skype or Facetime or other screen sharing program), or an audio-video call. In another example, the natural language communication path 19 may be provided via a ROCC device 8, such as a mobile device (e.g., a tablet computer or smart phone), or may be a wearable device worn by the local operator LO, such as an augmented reality (AR) display device (e.g., AR goggles), a projector device, a head-up display (HUD) device, etc., each of which has a display device 36. For example, an “app” may be run on the ROCC device 8 (operable by the local operator LO) and the remote electronic processing device 12 (operable by the remote expert RE) to allow communication (e.g., audio chat, video chat, etc.) between the local operator and the remote expert.

[0035] Figure 1Also shown is a remote electronic processing device 12, such as a workstation, workstation computer, or more generally a computer, included in the remote service center 4, which is operably connected to receive and present a video feed 17 and / or an audio feed 18 of the medical imaging device compartment 3 from the camera 16. Additionally or alternatively, the remote electronic processing device 12 can be embodied as a server computer or multiple server computers, for example interconnected to form a server cluster, cloud computing resources, etc. The electronic processing device 12 includes typical components such as an electronic processor 20 (e.g., a microprocessor), at least one user input device (e.g., a mouse, keyboard, trackball, etc.) 22 and at least one display device 24 (e.g., an LCD display, a plasma display, a cathode ray tube display, etc.). In some embodiments, the display device 24 can be a separate component from the electronic processing device 12. The display device 24 can also include two or more display devices. The electronic processor 20 is operably connected to one or more non-transitory storage media 26. As non-limiting illustrative examples, the non-transitory storage medium 26 may include one or more of the following: a disk, a RAID or other magnetic storage medium; a solid-state drive, a flash drive, an electrically erasable read-only memory (EEROM) or other electronic memory; an optical disk or other optical storage device; various combinations thereof, etc.; and may be, for example, a network storage device, an internal hard drive of the electronic processing device 12, various combinations thereof, etc. It should be understood that any reference to one or more non-transitory media 26 herein will be broadly interpreted to cover a single medium or multiple media of the same or different types. Similarly, the electronic processor 20 may be embodied as a single electronic processor or two or more electronic processors. The non-transitory storage medium 26 stores instructions that can be executed by at least one electronic processor 20. The instructions include instructions for generating a graphical user interface (GUI) 28 for display on a remote operator display device 24. The video feed 17 from the camera 16 may also be displayed on the display device 24, and the audio feed 18 may be output on the remote electronic processing device 12 via a speaker 29. In some examples, audio feed 18 may be the audio component of an audio / video feed (eg, as recorded by a video cassette recorder (VCR) device operating).

[0036] Figure 1An illustrative local operator LO and an illustrative remote expert RE (e.g., super technology) are shown. However, in a radiology operation command center (ROCC) as contemplated herein, the ROCC provides super technicians that can be used to assist local operators LO at different hospitals, radiology laboratories, etc. Each remote expert RE can operate a corresponding remote electronic processing device 12. The ROCC can be housed in a single physical location, or can be geographically distributed. For example, in one contemplated embodiment, remote expert REs are recruited from the United States and / or internationally to provide a wide range of expertise for super technicians in various imaging modalities and various imaging procedures targeting various imaging anatomical structures. The server computer 14s can communicate with the medical imaging compartment 3 and the remote service center 4 using one or more non-transitory storage media 26s. As a non-limiting illustrative example, the non-transitory storage medium 26s may include one or more of the following: a disk, a RAID or other magnetic storage medium; a solid-state drive, a flash drive, an electrically erasable read-only memory (EEROM) or other electronic memory; an optical disk or other optical storage device; various combinations thereof, etc.; and can be, for example, a network storage device, an internal hard drive of the server computer 14s, various combinations thereof, etc. It should be understood that any reference herein to one or more non-transitory media 26 will be broadly interpreted to cover a single medium or multiple media of the same or different types. Similarly, the server computer 14s can be embodied as a single electronic processor or two or more electronic processors. The non-transitory storage medium 26s stores instructions that can be executed by the server computer 14s.

[0037] The medical imaging device controller 10 in the medical imaging device compartment 3 also includes components similar to the remote electronic processing device 12 disposed in the remote service center 4. Unless otherwise specified herein, features of the medical imaging device controller 10 (including the local electronic processing device 12') disposed in the medical imaging device compartment 3 (similar to features of the remote electronic processing device 12 disposed in the remote service center 4) have common reference numerals followed by a "'" symbol, and descriptions of the components of the medical imaging device controller 10 will not be repeated. In particular, the medical imaging device controller 10 is configured to display a GUI 28' on a display device or controller display 24' that presents information related to the control of the medical imaging device 2, such as a configuration display for adjusting configuration settings, an alarm 30 perceptible at a remote location when status information about a medical imaging examination meets an alarm criterion of the imaging device 2, imaging acquisition monitoring information, presentation of acquired medical images, etc. It should be understood that the screen mirroring data stream 18 carries content presented on the display device 24' of the medical imaging device controller 10. The communication link 14 allows screen sharing between the display device 24 in the remote service center 4 and the display device 24' in the medical imaging device compartment 3. The GUI 28' includes one or more dialog screens, such as an exam / scan selection dialog screen, a scan setup dialog screen, an acquisition monitoring dialog screen, etc. The GUI 28' can be included in the video feed 17 and displayed on the remote electronic processing device display 24 at the remote location 4.

[0038] Furthermore, as disclosed herein, the server 14s executes a method or process 100 for providing assistance during a medical imaging examination performed using the medical imaging device 2 (i.e., by assisting a local operator LO of the corresponding medical imaging device 2 during the medical imaging examination by a remote expert RE). Instructions for executing the method 100 are stored in the non-transitory computer-readable medium 26 of the remote electronic processing device 12.

[0039] refer to Figure 2 , and continue to refer to Figure 1 , an illustrative embodiment of the method 100 is schematically shown as a flow chart. To begin the method 100, a medical imaging examination is started by a local operator LO using the medical imaging device 2. During the examination, events may occur that require assistance from a remote expert RE. Although a single medical imaging examination is described, it should be understood that an instance of the examination progress tracking method 100 may generally be performed for each medical imaging examination monitored by the camera of the ROCC device 1.

[0040] At operation 102, video 17 of a medical imaging examination (captured by one or more cameras 16) is collected and routed to a server computer 14s for analysis. At operation 104, a state machine 40 implemented in the server computer 14s is used to determine the status of one or more imaging examinations based on the collected video 17 (and optionally also the audio feed 18). At operation 106, an indication 46 of the determined status of the one or more imaging examinations is displayed on the display device 36 of the ROCC device 8. At operation 108, the process loops back so that operations 104 and 106 are iterated to track the progress of the medical imaging examination as the medical imaging examination transitions between the states of the state machine. At any time during the imaging examination, tracking of the progress of the imaging examination can result in operation 110, where an examination event requiring an auxiliary action is detected, in which case the auxiliary action is performed at operation 112. For example, the auxiliary action can include establishing a natural communication path 19 between a local operator LO performing the medical imaging examination and a remote expert RE. Establishing the communication may also include providing an indication of the detected event to the remote expert RE, so that the remote expert is given situational awareness of the event in the imaging examination. In another example, the auxiliary action may include automatically providing text, graphics, video and / or multimedia guidance about the determined event to the local operator LO. For example, if the detected event is that the state of the imaging examination changes from image data acquisition to patient positioning (which is a fallback step), this can trigger the user to be presented with information on how to position the patient for the specific imaging sequence being performed on a locally positioned display of text, graphics, video and / or multimedia guidance. Advantageously, this latter auxiliary action can facilitate the local operator LO to resolve the event without spending the valuable time of the remote expert RE. Additionally or alternatively, at operation 114, data collected from iterative tracking 108 about the progress of the examination can be stored for subsequent data mining.

[0041] The state machine 40 may include a plurality of states of the imaging examination. In order to determine the current state of the imaging examination performed by the local operator LO, an initial state of the imaging examination is determined. The indication 46 displayed on the ROCC device 8 may include the initial state, and the initial state may include patient information and imaging examination information input to the ROCC device 8 by the local operator LO. The state machine 40 may be used to identify a transition from the determined initial state to a subsequent state of the imaging examination. When such a transition occurs, the display device 36 of the ROCC device 8 may be updated to display the indication 46 as a transition from the previous state of the imaging examination to the updated state of the imaging examination.

[0042] In some embodiments, an expected duration for each state of the imaging examination may be determined, and an alert 30 may be output via the ROCC device 8 indicating that one of the states of the imaging examination exceeds the corresponding expected duration.

[0043] In some embodiments, the performance of a local operator LO performing an imaging examination may be monitored and the performance data may be stored in the server computer 14s.

[0044] In some embodiments, an indication 46 of the determined status of the imaging examination may be displayed on the remote electronic processing device 12. Additionally, an alert 30 indicating that one of the statuses of the one or more imaging examinations exceeds the corresponding expected duration may be output via the remote electronic processing device 12. In another example, a natural communication path 19 between the local operator LO and the remote expert RE may be established based on the determined status of the imaging examination (e.g., whether the local operator LO requires assistance from the remote expert RE, whether the alert 30 is output, etc.).

[0045] refer to Figure 3 , schematically shows an illustrative simplified state machine 40 representing the workflow of a medical imaging examination. The illustrative state machine 40 includes states for "PATIENT ARRIVAL", "PATIENT LOADING", "SCOUTING IMAGING", "SCAN SETUP", "IMAGING DATA ACQUISITION", and "PATIENT UNLOADING". The state machine 40 also includes state transitions represented by directed arrows connecting the states, where the direction of the arrow is from the current state to the next state. The state transitions indicated by straight arrows indicate the expected workflow: these include: the transition from "PATIENT ARRIVAL" to "PATIENT LOADING"; the transition from "PATIENT LOADING" to "SCOUTING IMAGING"; the transition from "SCOUTING IMAGING" to "SCAN SETUP"; the transition from "SCAN SETUP" to "IMAGING DATA ACQUISITION"; and, the transition from "IMAGING DATA ACQUISITION" to "PATIENT UNLOADING". In addition to this linear workflow, another expected transition from "IMAGING DATA ACQUISITION" to "SCAN SETUP" may typically occur, representing a transition from one scheduled scan to the next scheduled scan. These transitions are expected and therefore do not typically constitute events of a medical imaging examination that may require ancillary actions (e.g., based on Figure 2 Operations 110 and 112).

[0046] On the other hand, other possible state transitions may constitute events that require auxiliary actions. For example, the transition from "scout imaging" to "patient loading" may indicate an event that the patient is mispositioned. The transition from "image acquisition" to "scan setup" may indicate that the local operator LO has rejected the clinical image and is adjusting the scan settings for a rescan. The transition from "patient unloading" to "scan setup" (or to "patient loading", although this transition is not in Figure 3) may indicate that the reviewing radiologist has rejected the image. Such state transitions are "backwards" in the sense that the workflow is not progressing as expected and some remedial action is being performed. Therefore, these state transitions may trigger operation 110, which detects an examination event based on the detected back state transition, resulting in operation 112 performing an auxiliary action.

[0047] In some embodiments, the next state itself may trigger operations 110 and 112 regardless of what transitions lead to that state. Figure 3 An example is shown as an additional state of "Radiation Warning", which can be reached by transitioning from "Reconnaissance Imaging" or "Image Acquisition". (Note that Figure 3 These transitions are depicted using dashed arrow lines.) A "radiation warning" state may occur during imaging with ionizing radiation (such as CT) because the calculated radiation dose delivered to the patient (e.g., calculated based on the X-ray tube current and exposure duration) exceeds a predefined safety limit. Regardless of how the "radiation warning" state is reached, it will typically trigger operations 110 and 112 to perform some auxiliary actions.

[0048] The detection of an event in operation 110 may also depend on other information. For example, a single instance (or perhaps even two or three repetitions) of a transition from "scout imaging" to "patient loading" may not trigger an event in operation 110, because the local operator LO may typically need to iteratively position the patient, perform a scout scan, and reposition the patient to achieve optimal patient positioning. However, if there are more than a certain threshold N instances of the transition from "scout imaging" to "patient loading", then operation 110 may detect an event, because such an excessive number of repetitions may indicate that the local operator LO is having difficulty positioning the patient. Similarly, although the "scout imaging" state is a normal state of the workflow, if a medical imaging examination remains in the "scout imaging" state for longer than the expected duration (e.g., expected based on how long it typically takes the local operator to perform a scout scan), then this may trigger an event in operation 110, because this indicates that the local operator LO is having some difficulty. Again, these are merely further non-limiting illustrative examples.

[0049] It should also be understood that Figure 3The state diagram 40 is highly simplified, and a state machine representing an actual medical imaging examination workflow may include many more states and state transitions. As a non-limiting example, the "patient loading" state may actually be decomposed into multiple states, such as "loading patient onto patient support", "positioning patient on patient support", "raising patient support to bore level", "translating patient support into scanner bore", etc., wherein there may be various transitions between these states. Similarly, the "scan setup" state may be decomposed into various states representing stages of the scan setup process, etc. There may also be various imaging modality-specific or imaging examination-specific states, such as a state representing the initiation of contrast agent bolus delivery.

[0050] Various methods may be used to track the progress of a medical imaging examination using the state machine 40. For example, detecting a transition from a current state to a next state may involve: detecting the transition as a change in the content of the acquired video 17 from a first user interface (UI) dialog screen corresponding to the current state in the state machine 40 (e.g., a scout imaging UI dialog corresponding to the "scout imaging" state) to a second UI dialog screen (which is different from the first UI dialog screen); and determining the next state of the medical imaging examination by matching the second UI dialog screen to the next state in the state machine 40 (e.g., matching the second UI screen to a "scan settings" UI screen corresponding to the "scan settings" state). In another example, detecting a transition from the current state to the next state may involve determining a change in the content of the acquired video 17; and detecting a transition of the medical imaging examination from the current state of the medical imaging examination to the next state of the medical imaging examination based on the detected change in the content of the acquired video 17 and an allowed transition outside of the current state in the state machine 40. These are merely non-limiting illustrative examples.

[0051] Example

[0052] The following describes in more detail Figure 1 and Figure 2 Some additional embodiments of the apparatus 1 and method 100. The server computer 14s may include one or more modules configured to perform the operations of the method 100. In one example, a module for parsing the console video 17 into images is provided. The module is capable of parsing images captured from both the live video feed 17 or the recorded video at a selected frequency.

[0053] A module for estimating changes between images may also be provided. This module uses a reference image to identify changes in the current image. In the current implementation, the reference image is selected as the previous image. In other words, two consecutive images are used to detect changes. This module also handles pre-processing of the two images, such as thresholding, in order to detect differences. In addition, it filters minor changes such as mouse movements. This module can be extended to classify changes into two groups: new to the current image; and, removed from the current image.

[0054] A module may be provided for determining events of imaging examinations based on detected changes between images. The module analyzes the detected changes and classifies the changes as pop-ups. It uses a combination of image processing techniques and machine learning methods to perform this analysis. The module may also be extended to use templates to discern events that have occurred previously.

[0055] A module is also provided for estimating the imaging examination context based on the determined events. There is a typical sequence of console states experienced by any imaging examination, from patient registration to pushing images to PACS. The asynchronous nature of these examination states makes it challenging to derive the examination context. This module uses console events together with the state machine 40 for a robust estimate of the current state.

[0056] This module can be extended in such a way that inputs from other sensors such as cameras can be used to estimate the inspection state. Among other things, it is responsible for pushing these events on the server computer 14s along with the corresponding timestamps. This determination of the inspection background can be implemented as follows by using a state machine 40. Typically, the state will be given by a multidimensional state vector:

[0057] S=(s 1 ,s 2 ,…,s N ),

[0058] where N is the number of independent or partially independent state variables. For example, the state variable s 1 The inspection state can be represented as one of the following [“Not Started”, “Survey Acquisition”, “Geometry Planning”, “Image Acquisition”, “Complete”], and the state variable s 2 The screen display mode can be expressed as one of the following ["view mode", "scan mode", "planning mode"].

[0059] The state machine 40 manages the transitions between the state vectors S→S' based on the received determined imaging examination events. For each received event e, depending on the event and according to s i →s' i =R iThe current state of all state variables of (e,S), and the new state variables are determined by the rule set R.

[0060] In the disclosed apparatus 1, there are multiple pipelines such as optical character recognition (OCR) that are resource intensive to extract information from the console screen 10. The computationally lightweight nature of the proposed method allows it to be implemented at a higher frequency than resource intensive pipelines, and the insights derived by the method can help in the optimal use of resource intensive pipelines. For example, if the screen is idle for a long time or if the screen undergoes changes in a specific area, then the OCR does not need to run and the OCR is only run on that portion. Checking the background can be used to trigger the OCR pipeline as follows: for each state transition S→S', the filter

[0061]

[0062] Determines if an immediate pipeline run should be triggered. Independent of state changes, the pipeline will also be triggered at regular intervals by a timer.

[0063] The present disclosure has been described with reference to the preferred embodiments. Modifications and alterations may occur to others upon reading and understanding the preceding detailed description. It is intended that the exemplary embodiments be interpreted as including all such modifications and alterations as long as they fall within the scope of the appended claims or their equivalents.

Claims

1. A non-transitory computer readable medium (26s) storing instructions executable by at least one electronic processor (14s) to perform a method (100) of providing assistance during a medical imaging examination performed using a medical imaging device (2), the method comprising: Acquiring a video of the medical imaging examination (17); determining a current state of an imaging examination based on the acquired video using a state machine (40) implemented in the at least one electronic processor; as well as An indication (46) of the determined current status of the imaging examination is displayed on an electronic processing device (8).

2. The non-transitory computer readable medium (26s) according to claim 1, wherein: The method (100) further comprises: A transition from the current state of the medical imaging examination to a next state of the imaging examination is identified based at least on the acquired video (17) and the state machine (40).

3. The non-transitory computer readable medium (26s) according to claim 2, wherein: Identifying the transition includes: detecting the transition as a change in content of the captured video (17) from a first user interface (UI) dialog screen corresponding to the current state in the state machine (40) to a second UI dialog screen; and The next state of the medical imaging examination is determined by matching the second UI dialog screen to the next state in the state machine.

4. The non-transitory computer readable medium of claim 2, wherein: Identifying the transition includes: determining changes in the content of the captured video (17); and Based on detected changes in the content of the captured video and allowed transitions outside of the current state in the state machine (40), detecting the transition of the medical imaging examination from the current state of the medical imaging examination to the next state of the medical imaging examination.

5. The non-transitory computer readable medium (26s) according to any one of claims 2 to 4, wherein: The method (100) further comprises: An event of the medical imaging examination that triggers the transition from the current state of the medical imaging examination to the next state of the medical imaging examination is determined based on a state transition of the state machine (40) from the current state to the next state.

6. The non-transitory computer readable medium (26s) according to claim 5, wherein: The method (100) further comprises: An assistance action is performed based on the determined event to provide assistance during the medical imaging examination.

7. The non-transitory computer readable medium (26s) according to any one of claims 2 to 6, wherein: The auxiliary action includes establishing a natural communication path (19) between a local operator (LO) performing the medical imaging examination and a remote expert (RE).

8. The non-transitory computer readable medium (26s) according to claim 6, wherein: The auxiliary actions include automatically providing text, graphics, video and / or multimedia guidance regarding the determined event.

9. The non-transitory computer readable medium (26s) according to any one of claims 2 to 8, wherein: The current state of the medical imaging examination is defined by a state variable, the value of which is determined based on information determined from an acquired video (17), the acquired video (17) including at least patient information and imaging examination information input into the electronic processing device (8) by a local operator (LO).

10. The non-transitory computer readable medium (26s) according to any one of claims 1 to 9, wherein: The method (100) further comprises: An alarm is outputted via the electronic processing device (8) indicating that the duration of the current state of the imaging examination exceeds an expected duration of the current state of the medical imaging examination.

11. The non-transitory computer readable medium (26s) according to any one of claims 1 to 10, wherein: The method (100) further comprises: detecting transitions of the medical imaging examination between states of the state machine (40) by iteratively repeating the determination, thereby collecting data regarding the performance of a local operator (LO) performing the medical imaging examination; and The collected data related to the performance of the local operator is stored.

12. The non-transitory computer readable medium (26s) according to any one of claims 1 to 11, wherein: The display of the indication includes: The indication (46) of the determined current status of the medical imaging examination is displayed on a remote processing device (12) operable by a remote expert (RE).

13. The non-transitory computer readable medium (26s) according to any one of claims 1 to 12, wherein: The method (100) further comprises: Based on the determined current status of the medical imaging examinations, a natural communication path (19) is established between a local operator (LO) performing the one or more imaging examinations and the remote expert (RE).

14. A method (100) of providing assistance during a medical imaging examination performed using a medical imaging device (2), the method comprising: Acquiring a video of the medical imaging examination (17); tracking the progress of the medical imaging examination using a state machine (40) representing a workflow of the medical imaging examination, the progress being tracked based at least on matching information extracted from the acquired video with state information of states of the state machine; as well as An assistance action is performed based on the tracked progress of the medical imaging examination to provide assistance during the medical imaging examination.

15. The method (100) according to claim 14, wherein: The tracking includes: determining a current state of the medical imaging examination based on a user interface (UI) screen detected in the acquired video (17) of the medical imaging examination, the user interface (UI) screen corresponding to a UI screen associated with the current state in the state machine (40); detecting a transition of the medical imaging examination from the current state of the medical imaging examination by detecting a change of the acquired video feed from the first UI dialog screen to a second UI dialog screen; and A next state of the medical imaging examination is determined based on matching the second UI dialog screen with a UI screen associated with a next state in the state machine for which the state machine has a transition from the current state to the next state.

16. The method (100) of claim 14, wherein: The tracking includes: determining changes in the content of the captured video (17); and A transition of the medical imaging examination from a current state of the medical imaging examination to a next state of the medical imaging examination is detected based on a detected change in the content of the acquired video feed.

17. The method (100) according to any one of claims 15 and 16, further comprising: The auxiliary action to be performed is determined based on the determined next state and / or the detected transition.

18. The method (100) of claim 17, wherein: The auxiliary action includes establishing a natural communication path (19) between a local operator (LO) performing the medical imaging examination and a remote expert (RE).

19. The method (100) of claim 17, wherein: The auxiliary action includes automatically providing text, graphics, video and / or multimedia guidance related to the determined event.

20. A non-transitory computer readable medium (26s) storing instructions executable by at least one electronic processor (14s) to perform a method (100) of providing assistance during a medical imaging examination performed using a medical imaging device (2), the method comprising: Acquiring a video of the medical imaging examination (17); determining a current state of an imaging examination based on the acquired video using a state machine (40) implemented in the at least one electronic processor; determining an event of the medical imaging examination that triggers the transition from the current state of the medical imaging examination to the next state of the medical imaging examination based on a state transition of the state machine (40) from the current state to a next state; as well as An indication (46) of the determined current status of the imaging examination is displayed on an electronic processing device (8).

Images