Computing architecture for remote imaging examination monitoring to provide accurate, robust and real-time events

Abstract

A method (100) of monitoring a medical imaging examination is described. The method comprises: receiving one or more video feeds (17) of at least the imaging compartment (3); detecting, from the one or more video feeds, whether a medical procedure is being performed in the imaging compartment; in response to the detection indicating that a medical procedure is being performed in the imaging compartment, controlling a local electronic processing device (8) assigned to the imaging compartment to process the one or more video feeds to extract and present information about the medical procedure being performed in the imaging compartment; and in response to the detecting indicating that the medical procedure is not being performed in the imaging compartment, controlling the local electronic processing device not to process the one or more video feeds.

Description

Technical Field

[0001] The following generally relates to imaging technology, remote imaging assistance technology, remote imaging inspection monitoring technology and related technologies. Background Art

[0002] Medical imaging (such as computed tomography (CT) imaging, magnetic resonance imaging (MRI), positron emission tomography (PET) imaging, fluoroscopic imaging, etc.) is a key component in providing many types of medical care and is used in a wide range of medical fields, such as cardiology, oncology, neurology, orthopedics, etc. The operators of the medical imaging equipment used to acquire the medical images are typically trained technicians, while the interpretation of the medical images is typically handled by a medical specialist such as a radiologist. The radiologist's interpretation of the radiology report or findings and the application of those findings to the patient's specific clinical case may be handled by the patient's general practitioner (GP) or a medical specialist such as a cardiologist, oncologist, orthopedic surgeon, etc.

[0003] Diagnostic imaging is in high demand today. As the world population grows, the need for fast, safe, high-quality imaging will only continue to grow, placing further pressure on imaging centers and their staff. One way imaging centers can improve efficiency and grow operations without the accompanying increase in labor costs is through a Radiology Operations Command Center (ROCC) system. A Radiology Operations Command Center enables teams to work across a network of imaging sites, providing their expertise as needed and remotely helping less experienced technicians perform high-quality scans. Remote technicians or experts can monitor the local operator during the scanning process through cameras installed in the scanning area or from other sources such as sensors (including radar sensors), console video feeds, microphones connected to Internet of Things (IoT) devices, etc. In addition, these sources can be supplemented by other data sources such as Health Level 7 (HL7) medical data feeds, medical images stored in Digital Imaging and Communications in Medicine (DICOM) format, electronic health record (EHR) databases, etc.

[0004] ROCC realizes telepresence via audio-video connection, and provides real-time access to the imaging scanner console screen and video camera feed from the scanner room to the remote command center. Expert users at the command center provide virtual shoulder support to local technicians and staff who perform imaging examinations. ROCC can also provide automatic assistance to remote experts and / or local technicians. The computing algorithms on the ROCC tablet hardware process real-time event data from multiple channels including consoles and cameras, and provide executable insights to expert technicians in real time. ROCC provides solutions for a wide range of scanning equipment (such as MRI, CT, X-ray and ultrasound), and the duration of the patient in the scanner room varies from a few minutes to a few hours. Importantly, the calculations running in ROCC run in a way that reliably generates these events in real time across different imaging modalities. Such capabilities enable expert technicians to actively intervene based on events, and support local technicians to handle their daily problems.

[0005] Certain improvements that overcome these and other problems are disclosed below. Summary of the invention

[0006] The invention is defined by the independent claims. The dependent claims indicate advantageous embodiments.

[0007] In one aspect, a non-transitory computer-readable medium stores instructions executable by at least one electronic processing device to perform a method for monitoring a medical imaging examination. The method includes: receiving one or more video feeds of at least an imaging bay; detecting whether a medical procedure is being performed in the imaging bay based on the one or more video feeds; in response to the detection indicating that a medical procedure is being performed in the imaging bay, controlling a local electronic processing device assigned to the imaging bay to process the one or more video feeds to extract and present information about the medical procedure being performed in the imaging bay; and, in response to the detection indicating that a medical procedure is not being performed in the imaging bay, controlling the local electronic processing device to not process the one or more video feeds.

[0008] In another aspect, a method of monitoring a medical imaging examination is described, the method comprising the steps of:

[0009] receiving one or more video feeds of at least an imaging compartment;

[0010] detecting whether a medical procedure is being performed in the imaging bay based on the one or more video feeds;

[0011] responsive to the detecting indicating that a medical procedure is being performed in the imaging bay, controlling a local electronic processing device assigned to the imaging bay to process the one or more video feeds to extract and present information about the medical procedure being performed in the imaging bay; and

[0012] In response to the detecting indicating that a medical procedure is not being performed in the imaging bay, the local electronic processing device is controlled to not process the one or more video feeds.

[0013] It should be understood that method and computer-readable medium for performing the method may be considered synonyms, and if the applicant states a computer-readable medium, it may relate to the method, and vice versa, when the applicant states a method, it may relate to a computer-readable medium for performing the method.

[0014] In another aspect, a support device for medical imaging includes: a server computer; and a local electronic processing device, which is assigned to a corresponding medical imaging compartment and is programmed to apply a machine learning (ML) model to a video feed received from the corresponding imaging compartment to which it is assigned to extract information about a medical imaging procedure performed in the corresponding imaging compartment to which it is assigned. The server computer and / or the local electronic processing device are programmed to determine whether a medical imaging procedure is being performed in the corresponding medical imaging compartment. The server computer is programmed to perform training of the ML model, including allocating ML model training tasks among the local electronic processing devices based on whether the medical imaging procedure is being performed in the assigned corresponding medical imaging compartment, and receiving results of the allocated ML model training tasks from the local electronic processing device.

[0015] In another aspect, a non-transitory computer-readable medium stores instructions executable by at least one electronic processing device to perform a method for monitoring a medical imaging examination. The method includes: receiving one or more video feeds of at least an imaging bay; detecting whether a medical procedure is being performed in the imaging bay based on the one or more video feeds; in response to detecting that a medical procedure is being performed in the imaging bay, controlling a local electronic processing device assigned to the imaging bay to process the one or more video feeds to extract and present information about the medical procedure being performed in the imaging bay; and, in response to detecting that a medical procedure is not being performed in the imaging bay, controlling the local electronic processing device to perform one or more training tasks for a machine learning (ML) component.

[0016] Further described is a computer program comprising instructions which, when executed by a computer, cause the computer to perform the steps of any embodiment of the invention.

[0017] One advantage resides in faster processing of images during a medical imaging examination.

[0018] Another advantage resides in saving computing power when a medical imaging examination is not being performed.

[0019] Another advantage lies in training machine learning (ML) models to perform imaging analysis tasks.

[0020] A given embodiment may provide none, one, two, more, or all of the aforementioned advantages, and / or may provide other advantages that will be apparent to one of ordinary skill in the art upon reading and understanding the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The disclosure may take form in various components and arrangements of components, and in various steps and arrangements of steps.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the disclosure.

[0022] Figure 1 An illustrative apparatus for providing remote assistance according to the present disclosure is schematically shown.

[0023] Figure 2 Shown by Figure 1 An example flow chart of operations appropriately performed by a device.

[0024] Figure 3 and 4 Shows Figure 1 Different embodiments of the device.

[0025] Figure 5 Shown by Figure 1 An example flow chart of operations appropriately performed by a device. DETAILED DESCRIPTION

[0026] As previously described, the ROCC can provide automated assistance to remote experts and / or local technicians. This automated assistance can be provided in the form of machine learning (ML)-based image analysis performed on medical images immediately at the time of medical image acquisition. This ML-based analysis can provide early detection of imaging artifacts, incorrectly selected fields of view, incidental findings beyond the reason for the imaging examination, and other problems. With early detection of such problems, additional or replacement images can be acquired during the imaging examination, eliminating the need for recall examinations. ML-based detection of incidental findings can also have tremendous clinical value by providing early detection of treatable medical conditions. The ROCC can utilize other types of ML models, such as ML models applied to cubicle camera videos to detect events occurring in the imaging cubicle.

[0027] However, the ML models used in such ML-based image analysis require training, which is a computationally complex process. A large number of ML models can be envisioned to provide detection of various different types of artifacts, non-optimal imaging conditions, incidental findings, etc. However, training such a set of ML models can be burdensome even for a server computer or server network. Furthermore, it may be desirable to train some types of ML models dynamically, i.e., to periodically update the training using recently acquired medical images, in order to ensure that the ML models are well tuned to the peculiarities of images acquired by newer models of medical imaging equipment and newer techniques in medical imaging (such as the use of new types of contrast agents, new imaging sequences, etc.).

[0028] The following discloses the use of a layered information technology (IT) architecture to better utilize available computing resources and improve ML-based analysis of complex images and audio analysis to extract actionable information from compartment camera video and potentially other video cameras (in-bore, contrast injector, etc.) and from microphones in the imaging compartment. Such ML analysis is computationally heavy and prone to error if the training data set is insufficient.

[0029] The layered IT architecture includes sensors, "edge" devices, a cloud computing layer, and a centralized application layer. For example, an "edge" device can be a tablet computer used by a local imaging technician to interface with a centralized application layer. For example, the cloud computing layer corresponds to the hospital's computing IT network, while the centralized application layer is maintained by the supplier of medical imaging equipment. In this architecture, edge devices carry a large amount of workload, including camera feed analysis, imaging scanner console capture screen acquisition and analysis, microphone audio analysis, and conversion of executable events using various ML models. This is problematic because edge devices (e.g., tablet computers) typically have limited resources, such as computing power. On the other hand, although edge devices have limited computing power, they are numerous in large ROCC networks, and as recognized herein, they can be effectively utilized overall to assist complex computing tasks.

[0030] In one aspect, the training orchestrator tracks the usage of edge devices and reallocates the computing power of those devices that experience downtime (e.g., between exams, during night shifts, etc.) to ML model training tasks. The orchestrator also coordinates ML model training tasks, such as storing an index of ML models and assigning training to edge devices that utilize those models, and tracking the success / failure rates of ML models to assign or prioritize which models to train. This aspect generally utilizes edge devices for the secondary task of assisting ML model training without adversely impacting their primary task of assisting imaging technicians and / or experts in performing imaging exams.

[0031] In another aspect, optical character recognition (OCR) and information extraction from the console display feed are divided into mandatory and conditional categories. Mandatory OCR tasks are those tasks associated with fields that may typically change throughout an imaging exam, such that OCR needs to be run throughout the scan. Conditional OCR tasks are those tasks associated with fields that change only during clearly defined phases of the exam. Examples of conditional OCR tasks are those tasks related to patient demographics, which are only input during the exam startup phase, and the conditional OCR tasks that remain unchanged thereafter are only executed during their (one or more) associated exam phases, thereby reducing the computational burden on the edge device. This aspect improves the efficiency of the edge device in performing its primary task of assisting imaging technicians and / or experts in performing imaging exams by removing the unnecessary processing burden caused by static OCR of text on a significant portion of the imaging exam.

[0032] In another aspect, rather than applying a single ML model to extract information, an ensemble of ML models may be applied and the outputs of the ensemble may be combined using techniques such as voting to obtain a final result. This aspect improves the effectiveness of ML models in assisting imaging technicians and / or experts in performing imaging examinations.

[0033] refer to Figure 1 , shows a support device 1 for providing assistance from a remote medical imaging expert RE (or super technician) to a local technician LO. Figure 1 As shown, a local operator LO operating a medical imaging device (also referred to as an image acquisition device, imaging device, etc.) 2 is located in a medical imaging device compartment 3, and a remote expert RE is provided in a remote service location or center 4. It should be noted that the "remote expert" RE may not necessarily directly operate the medical imaging device 2, but rather provide assistance to the local operator LO in the form of advice, guidance, instructions, etc. The remote location 4 may be a remote service center, a radiologist's office, a radiology department, etc. The remote location 4 may be in the same building as the medical imaging device compartment 3 (for example, this may be the case of a "remote operator or expert" RE, which is a radiologist responsible for peri-examination image review), but more typically, the remote service center 4 and the medical imaging device compartment 3 are in different buildings, and may actually be located in different cities, different countries, and / or different continents. Typically, the remote location 4 is remote from the imaging device compartment 3 in the sense that the remote expert RE cannot directly visually observe the imaging device 2 in the imaging device compartment 3 (thus optionally providing a video feed as further described herein).

[0034] The image acquisition device 2 may be a magnetic resonance (MR) image acquisition device; a computed tomography (CT) image acquisition device; a positron emission tomography (PET) image acquisition device; a single photon emission computed tomography (SPECT) image acquisition device; an X-ray image acquisition device; an ultrasound (US) image acquisition device; or another modality of medical imaging device. The imaging device 2 may also be a hybrid imaging device, such as a PET / CT or SPECT / CT imaging system. Although in Figure 1 A single image acquisition device 2 is diagrammatically represented in , but more typically a medical imaging laboratory will have multiple image acquisition devices, which may have the same and / or different imaging modalities. For example, if a hospital performs many CT imaging examinations and relatively few MRI examinations and even fewer PET examinations, the hospital's imaging laboratory (sometimes referred to as a "radiology laboratory" or some other similar term) may have three CT scanners, two MRI scanners, and only a single PET scanner. This is merely an example. In addition, the remote service center 4 may provide services to multiple hospitals. A local operator controls the medical imaging device 2 via an imaging device controller 10. The remote operator is located at a remote workstation 12 (or more generally, an electronic controller 12).

[0035] Some types of imaging modalities may utilize intravascular contrast agents. For example, MR may utilize gadolinium-based contrast agents. In order to provide contrast-enhanced imaging, a contrast agent injector 11 is configured to inject contrast agents into a patient. The contrast agent injector 11 is a configurable automatic contrast agent injector with a display 13. A user (typically an imaging technician) loads a vial or syringe of contrast agent (or two or more vials of different contrast agent components) into the contrast agent injector 11, and configures the contrast agent injector 11 by inputting contrast agent injector settings (such as flow rate, volume, time delay, injection duration, etc.) via a user interface (UI) of the contrast agent injector 11. The UI may be a touch-sensitive overlay of the display 13, and / or a physical button, keypad, etc. In a variant embodiment, the contrast agent injector 11 is integrated with the imaging device controller 10 (e.g., via a wired or wireless data connection), and the contrast agent injector 11 is controlled via the imaging device controller 10, including displaying the contrast agent injector settings in an (optionally selectable) window on the display of the imaging device controller 10. In such an embodiment, the dedicated physical syringe display 13 of the contrast media injector may be optionally omitted (or, alternatively, the dedicated physical syringe display 13 may be retained and the contrast media settings may be displayed at both the dedicated physical syringe display 13 and the imaging device controller 10). In general, the automatic contrast media injector 11 may employ any suitable mechanical configuration for delivering contrast media (or multiple contrast media), such as a syringe injector, a dual syringe injector, a pump-driven injector, etc., and may include hardware for performing advanced functions, such as saline dilution of contrast media, priming and / or flushing contrast media injection lines with saline, etc.

[0036] As used herein, the term "medical imaging equipment bay" (and variations thereof) refers to a room containing a medical imaging device 2 and any adjacent control room containing a medical imaging device controller 10 for controlling the medical imaging device. For example, with reference to an MRI device, the medical imaging equipment bay 3 may include a radio frequency (RF) shielded room containing the MRI device 2, and an adjacent control room housing the medical imaging device controller 10, as is understood in the art of MRI devices and procedures. On the other hand, for other imaging modalities such as CT, the imaging device controller 10 may be located in the same room as the imaging device 2, such that there is no adjacent control room, and the medical bay 3 is simply the room containing the medical imaging device 2. Furthermore, although Figure 1 A single medical imaging equipment bay 3 is shown, but it should be understood that the remote service center 4 (and more specifically the remote workstation 12) communicates with multiple medical bays via a communication link 14, which typically includes the Internet for electronic data communication enhanced by a local area network at the remote expert RE and the local operator LO. In addition, although Figure 1A single remote service centre 4 is shown, but it will be appreciated that the medical imaging device bay 3 communicates with a plurality of medical bays via the communication link 14 .

[0037] like Figure 1 Schematically shown, in some embodiments, a camera 16 (e.g., a video camera) is arranged to capture a video stream or feed 17 of a portion of the workspace of the medical imaging device compartment 3, which includes at least the area of ​​the imaging device 2 where the local operator LO interacts with the patient, and optionally may also include the imaging device controller 10. Although one camera 16 is shown, there may be multiple cameras, for example, typically one camera provides a feed of the imaging compartment, another camera provides a video feed of the display 36 of the contrast medium injector 11, and so on. In other embodiments, a microphone 15 is arranged to capture an audio stream or feed 18 of the workspace, which includes audio noise occurring within the medical imaging device compartment 3 (e.g., verbal instructions of the local operator LO, questions from the patient, etc.). The video stream 17 and / or the audio stream 18 are sent to the remote workstation 12 via the communication link 14, for example as a streaming video feed received via a secure Internet link.

[0038] The communication link 14 also provides a natural language communication path 19 for oral and / or textual communication between the local operator and the remote operator. For example, the natural language communication link 19 can be a Voice-Over-Internet-Protocol (VOIP) telephone connection, an online video chat link, a computer instant messaging service, etc. Alternatively, the natural language communication path 19 can be provided by a dedicated communication link separate from the communication link 14 providing data communications 17, 18, for example, the natural language communication path 19 can be provided via a landline telephone. In some embodiments, the natural language communication link 19 allows the local operator LO to call a selected remote expert RE. As used herein, a call can refer to an audio call (e.g., a telephone call), a video call (e.g., Skype or Facetime or other screen sharing program), or an audio-video call. In another example, the natural language communication path 19 may be provided via a ROCC device 8, such as a mobile device (e.g., a tablet computer or a smart phone), or may be a wearable device worn by the local operator LO, such as an augmented reality (AR) display device (e.g., AR glasses), a projector device, a head-up display (HUD) device, etc., each of which has a display device 36. For example, an "app" may be run on a ROCC device 8 (operable by a local operator LO) and a remote workstation 12 (operable by a remote expert RE) to allow communication (e.g., audio chat, video chat, etc.) between the local operator and the remote expert. In some examples, a plurality of imaging device bays 3 may each include a corresponding ROCC device 8.

[0039] Figure 1 Also shown in the remote service center 4 is a remote workstation 12, such as an electronic processing device, a workstation computer, or more generally a computer, which is operably connected to receive and present a video feed 17 and / or an audio feed 18 of the medical imaging device compartment 3 from the camera 16. Additionally or alternatively, the remote workstation 12 can be embodied as a server computer or multiple server computers, for example interconnected to form a server cluster, cloud computing resources, etc. The workstation 12 includes typical components such as an electronic processor 20 (e.g., a microprocessor), at least one user input device (e.g., a mouse, keyboard, trackball, etc.) 22 and at least one display device 24 (e.g., an LCD display, a plasma display, a cathode ray tube display, etc.). In some embodiments, the display device 24 can be a separate component from the workstation 12. The display device 24 can also include two or more display devices. The electronic processor 20 is operably connected to one or more non-transitory storage media 26. As non-limiting illustrative examples, the non-transitory storage medium 26 may include one or more of the following: a disk, RAID or other magnetic storage medium; a solid-state drive, a flash drive, an electrically erasable read-only memory (EEROM) or other electronic memory; an optical disk or other optical storage device; various combinations thereof, etc.; and, for example, may be a network storage device, an internal hard drive of the workstation 12, various combinations thereof, etc. It should be understood that any reference to one or more non-transitory media 26 herein will be broadly interpreted to cover a single medium or multiple media of the same or different types. Similarly, the electronic processor 20 may be embodied as a single electronic processor or two or more electronic processors. The non-transitory storage medium 26 stores instructions that can be executed by at least one electronic processor 20. The instructions include instructions for generating a graphical user interface (GUI) 28 for display on the remote operator display device 24. The video feed 17 from the camera 16 may also be displayed on the display device 24, and the audio feed 18 may be output on the remote workstation 12 via the speaker 29. In some examples, the audio feed 18 may be the audio component of the audio / video feed (e.g., to be recorded as a video recorder device).

[0040] Figure 1An illustrative local operator LO and an illustrative remote expert RE (e.g., a senior technician) are shown. However, in a radiology operation command center (ROCC) as contemplated herein, the ROCC provides a senior technical staff that can be used to assist the local operator LO at different hospitals, radiology laboratories, etc. Each remote expert RE can operate a corresponding remote workstation 12. The ROCC can be housed in a single physical location, or can be geographically distributed. For example, in one contemplated embodiment, remote expert REs are recruited from the United States and / or internationally to provide senior technicians with extensive expertise in various imaging modalities and in various imaging processes targeting various imaged anatomical structures. The server computer 14s can communicate with the medical imaging compartment 3 and the remote service center 4 through one or more non-transitory storage media 26s. As a non-limiting illustrative example, the non-transitory storage medium 26s may include one or more of the following: a disk, a RAID or other magnetic storage medium; a solid-state drive, a flash drive, an electrically erasable read-only memory (EEROM) or other electronic memory; an optical disk or other optical storage device; various combinations thereof, etc.; and, for example, may be a network storage device, an internal hard drive of the server computer 14s, various combinations thereof, etc. It should be understood that any reference herein to one or more non-transitory media 26s will be broadly interpreted to cover a single medium or multiple media of the same or different types. Likewise, the server computer 14s may be embodied as a single electronic processor or two or more electronic processors. The non-transitory storage medium 26s stores instructions that can be executed by the server computer 14s.

[0041] The medical imaging device controller 10 in the medical imaging device compartment 3 also includes components similar to the remote workstation 12 provided in the remote service center 4. Unless otherwise specified herein, features of the medical imaging device controller 10 including the local workstation 12' provided in the medical imaging device compartment 3 (similar to the features of the remote workstation 12 provided in the remote service center 4) have common reference numerals followed by a "superscript" symbol, and the description of the components of the medical imaging device controller 10 will not be repeated. In particular, the medical imaging device controller 10 is configured to display a GUI 28' on a display device or controller display 24', which presents information related to the control of the medical imaging device 2, such as a configuration display for adjusting configuration settings, an alarm 30 perceptible at a remote location when status information about a medical imaging examination meets an alarm criterion of the imaging device 2, imaging acquisition monitoring information, presentation of acquired medical images, etc. It should be understood that the screen mirroring data stream 18 carries content presented on the display device 24' of the medical imaging device controller 10. The communication link 14 allows screen sharing between a display device 24 in the remote service center 4 and a display device 24' in the medical imaging device bay 3. The GUI 28' includes one or more dialog screens, including, for example, an exam / scan selection dialog screen, a scan setup dialog screen, an acquisition monitoring dialog screen, etc. The GUI 28' may be included in the video feed 17 and displayed on the remote workstation display 24 at the remote location 4.

[0042] Furthermore, as disclosed herein, the server 14s executes a method or process 100 for monitoring a medical imaging examination performed using a medical imaging device 2 (i.e., by assisting a local operator LO of the respective medical imaging device 2 during a medical imaging examination performed by a remote expert RE). Instructions for executing the method 100 are stored in a non-transitory computer-readable medium 26s of the server computer 14s.

[0043] refer to Figure 2 , and continue to refer to Figure 1, in one aspect, an illustrative embodiment of method 100 is schematically shown as a flow chart. In this regard, an edge device 11 is utilized to perform the ML training task. To begin method 100, an imaging examination is started by a local operator LO using a medical imaging device 2. An event may occur during an examination that requires assistance from a remote expert RE. At operation 102, a video feed 17 (collected by one or more cameras 16) and / or an audio feed 18 (collected by one or more microphones 15) are routed to a server computer 14s for analysis. At operation 104, the server computer 14s analyzes the video feed 17 and / or the audio feed 18 to detect whether a medical procedure is being performed in the imaging compartment 3. For example, this may be based on whether a patient support is visible and whether a patient is loaded thereon. In another embodiment, detection 104 may be based on information from a captured imaging device controller 10, for example, if an idle screen is detected, it is determined that the imaging device is not currently in use.

[0044] At operation 106, in response to detecting (i.e., detecting operation 104) indicating that a medical imaging procedure is being performed in the imaging bay 3, the server computer 14s is configured to control a local electronic processing device (i.e., the ROCC device 8) assigned to the imaging bay 3 to process one or more video feeds 17 or audio feeds 18 to extract and present information about the medical procedure being performed in the imaging bay 3.

[0045] In another disclosed aspect of operation 106, the one or more video feeds include a video feed 17 that includes a captured controller screen video feed of a medical imaging device controller 10 used in a medical procedure being performed in the imaging booth 3. Control operation 106 can then include identifying text regions containing text in the captured controller screen video feed and classifying the text regions as quasi-static or dynamic. An optical character recognition (OCR) process can be performed to continuously extract the content of the dynamic text regions during the performance of the medical procedure in the imaging booth 3, and another OCR process can be performed to extract the content of the quasi-static text regions only when the medical procedure is being performed in the imaging booth 3, wherein the content of the quasi-static text regions can change.

[0046] These are merely illustrative examples and should not be construed as limiting.

[0047] At operation 108, in response to the detection indicating that a medical procedure is not being performed in the imaging bay 3, the server computer 14s is configured to control the ROCC device 8 to not process one or more video feeds 17 and / or audio feeds 18. For example, the control operation 108 may include controlling the operation of at least one camera 16 and / or microphone 15 to not operate to prevent the generation of the video feed 17 and / or audio feed 18. Optionally, during these times when the ROCC device 11 is not being used for its primary purpose of supporting imaging examinations, it may be used for an auxiliary purpose of performing ROCC secondary tasks, such as training an ML model used by the ROCC.

[0048] In some embodiments, the control operation 108 may include controlling the ROCC device 8 to apply at least one machine learning (ML) component 40 to extract information about a medical imaging procedure performed in the imaging bay 3 from images acquired by the medical imaging device (e.g., detecting suboptimal imaging settings, detecting incidental findings, etc.). For example, a plurality of ML models 40 may be retrieved from the server computer 14s, and training of at least one of the plurality of ML models 40 may be assigned to the ROCC device 8. In another example, when a plurality of ROCC devices 8 are provided in the corresponding medical imaging device bay 3, each ROCC device 8 may be assigned one or more ML models 40 for training. In another example, a first ML model 40 may be applied to extract a first type of information from one or more video feeds 17 and / or audio feeds 18. A plurality of second ML models 40 may be applied to extract a second type of information from one or more video feeds 17 and / or audio feeds 18. The first type of information and the second type of information may then be combined (e.g., by a voting process) to extract information presented about the medical procedure being performed in the imaging bay 3.

[0049] The results of the training of the ML models 40 may then be received at the server computer 14s from the ROCC device 8. The server computer 14s is then programmed to receive feedback from the ROCC device 8 indicating the performance of each ML model in extracting information about the medical imaging procedure performed in the imaging bay 3. The server computer 14s is also programmed to distribute ML model training tasks among the one or more ML models 40 based on the feedback received from the ROCC device(s) 8 indicating the performance of each ML model 40.

[0050] In some embodiments, the ROCC device 8 is configured to provide a communication interface (ie, a natural language communication path 19 ) between a local operator LO and a remote expert RE.

[0051] refer to Figure 3, shows a high-level representation of the layered computing architecture of the ROCC, including the ROCC application layer 110, the cloud computing layer 112, the edge computing layer 114, and the sensing layer 116. The bottom layer is the sensing layer 116, which processes the raw data from the (one or more) cameras 16 and the controller display. This pre-processed data from the sensing layer 116 is fed to the ROCC tablet or other edge device 11 as part of the edge computing layer 114. Note that each scanner room or imaging compartment typically has a dedicated ROCC tablet 11, which contains deep learning (DL) or other ML models (or sets of ML models) suitable for its environment (particularly the scanner console and scanner room camera configuration). The edge computing layer 114 uses the ML model to process the data from the sensing layer 116 into events. These events are streamed to the cloud account of the cloud computing layer 112 specific to the imaging center. The cloud computing layer 112 uses events to measure workflow metrics and provide status about rooms, exams, etc. This layer 112 also hosts a local repository of ML models used at a specific imaging center. Inferences from the cloud computing layer 112 are pushed to the ROCC application layer 110, where it is relayed to expert users who can make appropriate imaging examination support decisions. The ROCC application layer 110 also contains a complete list of ML models used across the ROCC platform to facilitate distributed training and update training of ML models.

[0052] refer to Figure 4 , showing that according to Figure 3 The hierarchical computing ROCC architecture is Figure 2 An example of utilizing edge devices to perform auxiliary ROCC secondary tasks (such as training or updating of ML models used by the ROCC) during operation 106 of the embodiment of the present invention. Processing is divided between cloud computing layer process 120 performed by cloud computing layer 112 and process 122 performed by edge computing layer 114. Cloud computing process 120 includes a computing resource manager 124 that manages available computing resources. For example, it can receive Figure 2 The output 102 of the method 100 is used to determine which edge devices 11 are available to perform secondary support processing at any given time. The ML model building scheduler 126 schedules the ML models to be trained, and the ML model training orchestrator assigns ML training tasks to edge devices based on the availability of ML training tasks (from the resource manager 124) and the ML training tasks to be performed (from the scheduler 126). Then, according to Figure 2 In operation 106 , the assigned edge device performs the task.

[0053] In the edge computing layer processing 122, at operation 130, the edge device 11 uses the trained ML model to perform ROCC inspection support tasks. For example, operation 130 can use the trained ML model to identify non-optimal imaging scan settings, accidental findings, etc. in the acquired images to provide early warnings of such problems. Operation 130 can also use the trained ML model applied to the frames of (one or more) video feeds 17 to detect events that occurred during the imaging inspection. When applying the ML model, the edge device can use model weights, configuration parameters, etc. that are specific to the imaging scanner 2 or camera 16 used for the imaging inspection. These applications of the ML model in operation 130 generate new data, which can optionally be filtered 134 in various ways to generate (new) historical data 136. A fault detection model 138 is applied to detect whether the ML model produces correct results. Operation 138 can be run in various ways, such as comparing the ML model output with information that confirms or contradicts the result. As an example, if the ML model outputs a warning that the imaging settings may be incorrect, and the technician adjusts the settings and reacquires the data, this is a confirmation; on the other hand, if the technician saves the file without rescanning, this is a contradiction. Therefore, operation 138 provides a "ground truth" label as to whether the ML model output is correct or incorrect. Optionally, this feedback can be collected by providing a short feedback dialog to the imaging technician, such as asking: "Is this information useful?" The labeled historical data is then fed back to the ML model building scheduler 126 to perform model update training appropriately. (For example, if the model has more than a certain threshold percentage of contradictions, it can be updated and trained).

[0054] Example

[0055] The support device 1 is described in more detail below. The server computer 14s can receive data from the sensing layer configured to process the video feed 17 and the audio feed 18. This pre-processed data can be input to the ROCC device 8 (including the edge computing layer), which can include one of the ML models 40 configured to process events during the medical imaging examination. The processed data is transmitted to the server computer 14s and analyzed to measure workflow metrics and provide status updates about the medical imaging examination. The server computer 14s also stores the ML model 40.

[0056] The ML models may include You Only Look Once (YOLO) or another object detection algorithm to generate the event stream. In order to generate a reliable and accurate event stream, these models 40 need to be trained regularly. Training these models takes about 200 hours on a dataset containing 5000 images on the ROCC device 8. In addition, the number of ML models 40 running on the ROCC device 8 is proportional to the number of variants in the imaging compartment 3. Given this setting, training is a repetitive process involving intensive computing activities.

[0057] The computing resources available in the edge computing layer can be used to perform different services, such as ML model 40 training, converting sensory data into events, etc. The computing resource manager tracks the services (e.g., sensing, training, etc.) provided by each resource. The console and camera pipelines are triggered to shut down when there is no activity on the console screen and scanner room, respectively. This usually happens when there is a long gap between two consecutive inspections and night shifts. During these time periods, the service presented by the computing resources on the network of the ROCC device 8 is switched to training the ML model 40. This information is provided to the training orchestrator module implemented in the server computer 14s.

[0058] The training orchestrator module takes inputs from the ML model building scheduling module and the computing resource manager implemented in the server computer 14s. The ML model building scheduling module identifies the ML model 40 that needs to be trained, while the computing resource manager identifies the available computing resources within the network of ROCC devices 8 installed in the imaging center. The training orchestrator module uses these inputs to run the training of the ML model 40. However, to initiate the training of the ML model 40, the training orchestrator module compares the current ML model 40 located in the local model repository implemented in the ROCC device 8 with the corresponding ML model 40 in the global model repository implemented in the server computer 14s. Training is initiated only when the two ML models 40 are the same. To improve performance, the training work of the ML model 40 is scheduled based on the current performance metric and the target performance metric (e.g., the number of failures seen in the results obtained from the currently deployed ML model 40 and the time since the ML model 40 was last trained). Once the training is completed, the old ML model 40 is replaced with the new trained ML model 40 in the local DL model repository and the global DL model repository. The compute resource manager selects available resources in a way that balances the workload between the compute resources while reducing the waiting time for running tasks in the queue. To achieve this, it deploys optimization models such as gradient descent, genetic algorithms. Workload balancing requires pausing ML model training, saving / loading current training state, and restarting ML training without compromising the performance of any tablet to provide high priority services such as audio-video telepresence, running console and camera algorithms once activity is detected on those input channels, etc.

[0059] refer to Figure 5 ,exist Figure 2In the aspect of operation 106 of , the OCR of the text of the captured controller display 24 'can be limited based on the content. To this end, the process receives the captured controller display feed 150, and captures frames for processing at, for example, one frame per second in operation 152. (The capture rate is selected to balance the processing load reduced by using a lower capture frequency with the temporal resolution or responsiveness increased by using a higher capture frequency). In operation 154, the display is monitored to detect and classify text fields. For example, this can be done by applying a priori known controller display templates, such as templates for patient intake display, scan setting display, acquisition monitoring display, image viewing display, etc. These templates can be specific to the brand / model of the medical imaging device. Additionally or alternatively, the current frame can be compared with the last captured frame to detect changes. Each text area is then classified as a mandatory OCR field 156 or as an optional OCR field 158. As a specific example, a ROCC console event stream is generated by the OCR and passive monitor pipelines. The OCR pipeline runs on multiple fields on the console screen of the medical imaging device controller 10. Each field takes about 1-2 seconds to execute the OCR pipeline. This run can be performed sequentially, and the time taken by the OCR pipeline imposes a limit on the frequency of sensing from the console feed. To address this problem, the server computer 14s applies Figure 4 The process of performing OCR on the two categories (mandatory and optional) is performed to classify all fields into two categories (mandatory and optional), and OCR is used on the two categories via parallel processing to reduce computation time. Optional fields are those fields that undergo changes during the start of the inspection but remain the same throughout the inspection. Running OCR on these fields is conditional on certain events such as the start of the inspection. The remaining fields belong to the mandatory category. Techniques such as multi-processing and multi-threading are used to run OCR on mandatory fields and optional fields in parallel. In order to further reduce the computational burden, a passive monitor 154 is introduced. The passive monitor 154 uses fast and simple image analysis (e.g., template matching or comparison with the last captured frame) to determine whether there is a pop-up window in the image or whether there is any change between the current image and the previous image. The passive monitor triggers the relevant OCR pipeline (mandatory 156 or optional 158 or both, i.e., different fields for a single captured controller display frame) based on its findings so that the OCR pipeline does not run unnecessarily.

[0060] The YOLO ML model 40 is used to detect regions of interest (ROIs) for given OCR fields in an image. Changes in the text background and the font size of these fields translate in the ROI detection of the YOLO model. It is very important to correctly identify the ROI of each OCR field. The current implementation relies on a single YOLO model, making it prone to misdetection. To address this issue, multiple YOLO models (i.e., five) can be used to detect fields on a given image. Consensus on the detection of the fields is derived by using a mechanism such as a voting process.

[0061] Unlike a single model implementation, the best ROI is identified by multiple ML models 40. Different merging strategies such as overlapping ROIs or weighted combination of F1 scores of ROIs can be used to determine the best ROI. This way of determining the bounding box using a collection of ML models 40 makes ROI detection robust. Running multiple ML models 40 on an image requires additional computational time. To overcome this, these ML models 40 are run using parallel processing and trained using a service switching process.

[0062] The computer system may also include a processor. The processor executes instructions to implement some or all aspects of the methods and processes described herein. The processor is tangible and non-transitory. As used herein, the term "non-transitory" should be interpreted as not a permanent characteristic of a state, but a characteristic of a state that will last for a period of time. The term "non-transitory" specifically negates transient characteristics, such as characteristics of a carrier or signal or other forms that only exist temporarily anywhere at any time. The processor is an article and / or machine component. The processor is configured to run software instructions to perform functions as described in various embodiments herein. The processor may be a general-purpose processor or may be part of an application-specific integrated circuit (ASIC). The processor may also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device. The processor may also be a logic circuit, including a programmable gate array (PGA), such as a field programmable gate array (FPGA), or another type of circuit including discrete gating and / or transistor logic. The processor may be a central processing unit (CPU), a graphics processing unit (GPU), or both. Additionally, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in or coupled to a single device or multiple devices.

[0063] As used herein, the term "processor" encompasses an electronic component capable of executing a program or machine-executable instructions. References to a computing device that includes a "processor" should be interpreted as including more than one processor or processing core, such as in a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed among multiple computer systems. The term "computing device" should also be interpreted to include a collection or network of computing devices, each of which includes one or more processors. A program has software instructions that are executed by one or more processors, which may be within the same computing device or may be distributed across multiple computing devices.

[0064] The computer system also includes a main memory and a static memory, wherein the memories in the computer system communicate with each other and with the processor via a bus. Either or both of the main memory and the static memory can be considered as representative examples of the memory of the controller, and store instructions for implementing some or all aspects of the methods and processes described herein. The memory described herein is a tangible storage medium for storing data and executable software instructions, and is non-transitory during the period when the software instructions are stored therein. As used herein, the term "non-transitory" should be interpreted as not a permanent characteristic of a state, but a characteristic of a state that will last for a period of time. The term "non-transitory" specifically negates transient characteristics, such as characteristics of carriers or signals or other forms that only exist temporarily anywhere at any time. The main memory and the static memory are articles of manufacture and / or machine components. The main memory and the static memory are computer-readable media from which a computer (e.g., a processor) can read data and executable software instructions. Each of the main memory and the static memory may be implemented as one or more of the following: random access memory (RAM), read-only memory (ROM), flash memory, electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, removable disk, magnetic tape, compact disk read-only memory (CD-ROM), digital versatile disk (DVD), floppy disk, Blu-ray disk, or any other form of storage medium known in the art. The memory may be volatile or non-volatile, secure and / or encrypted, unsecure and / or unencrypted.

[0065] "Memory" is an example of a computer-readable storage medium. Computer memory is any memory that is directly accessible to a processor. Examples of computer memory include, but are not limited to, RAM memory, registers, and register files. References to "computer memory" or "memory" should be interpreted as potentially being multiple memories. A memory may, for example, be multiple memories within the same computer system. A memory may also be multiple memories distributed among multiple computer systems or computing devices.

[0066] As shown, the computer system also includes a video display unit, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, or a cathode ray tube (CRT). In addition, the computer system includes an input device (such as a keyboard / virtual keyboard or a touch-sensitive input screen or a voice input with voice recognition) and a cursor control device (such as a mouse or a touch-sensitive input screen or a tablet). The computer system also optionally includes an optical drive unit, a signal generating device (such as a speaker or a remote controller), and / or a network interface device.

[0067] In one embodiment, the optical disc drive unit includes a computer readable medium in which one or more sets of software instructions are embedded. The software instruction set is read from the computer readable medium for execution by the processor. In addition, the software instructions, when executed by the processor, perform one or more steps of the methods and processes as described herein. In an embodiment, the software instructions reside in the main memory, static memory, and / or processor in whole or in part during execution by the computer system. In addition, the computer readable medium may include software instructions or receive and execute software instructions in response to a propagated signal so that a device connected to the network transmits voice, video, or data through the network. The software instructions may be transmitted or received through the network via a network interface device.

[0068] In an embodiment, dedicated hardware implementations such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic arrays, and other hardware components are constructed to implement one or more methods described herein. One or more embodiments described herein may use two or more specific interconnected hardware modules or devices with related control and data signals that can be transmitted between modules and through modules to implement functions. Therefore, the present disclosure covers software, firmware, and hardware implementations. Nothing in this application should be interpreted as being implemented or achievable using only software rather than hardware (such as tangible non-transient processors and / or memories).

[0069] According to various embodiments of the present disclosure, the methods described herein can be implemented using a hardware computer system that executes a software program. In addition, in an exemplary non-limiting embodiment, the implementation can include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing can implement one or more of the methods or functions described herein, and the processor described herein can be used to support a virtual processing environment.

[0070] The illustrations of the embodiments described herein are intended to provide a general understanding of the structures of various embodiments. These illustrations are not intended to be used as a complete description of all elements and features of the disclosure described herein. After reading this disclosure, many other embodiments will be apparent to those skilled in the art. Other embodiments may be utilized and derived from this disclosure so that structural and logical replacements and changes may be made without departing from the scope of this disclosure. Additionally, the illustrations are representative only and may not be drawn to scale. Certain ratios within the illustrations may be exaggerated, while other ratios may be minimized. Therefore, this disclosure and the accompanying drawings are considered to be illustrative rather than restrictive.

[0071] One or more embodiments of the present disclosure may be referred to herein individually and / or collectively by the term "invention", which is merely for convenience and is not intended to actively limit the scope of the present application to any particular invention or inventive concept. In addition, although specific embodiments have been shown and described herein, it should be understood that any subsequent arrangements designed to achieve the same or similar purposes may replace the specific embodiments shown. The present disclosure is intended to cover any and all subsequent adaptations or variations of the various embodiments. After reading the specification, the combination of the above-mentioned embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art.

[0072] The foregoing description of the disclosed embodiments is provided to enable those skilled in the art to practice the concepts described in the present disclosure. Therefore, the subject matter disclosed above is to be considered illustrative rather than restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments that fall within the true scope of the present disclosure. Therefore, to the maximum extent permitted by law, the scope of the present disclosure will be determined by the broadest permissible interpretation of the appended claims and their equivalents, and shall not be limited or restricted by the foregoing detailed description.

[0073] The present disclosure has been described with reference to the preferred embodiments. Modifications and alterations may occur to others upon reading and understanding the preceding detailed description. It is intended that the exemplary embodiments be interpreted as including all such modifications and alterations as long as they fall within the scope of the appended claims or their equivalents.

Claims

1. A method (100) for monitoring a medical imaging examination, the method comprising the following steps: receiving one or more video feeds (17) of at least the imaging compartment (3); detecting whether a medical procedure is being performed in the imaging bay based on the one or more video feeds; In response to the detecting indicating that a medical procedure is being performed in the imaging bay, controlling a local electronic processing device (8) assigned to the imaging bay to process the one or more video feeds to extract and present information about the medical procedure being performed in the imaging bay; as well as In response to the detecting indicating that a medical procedure is not being performed in the imaging bay, the local electronic processing device is controlled to not process the one or more video feeds.

2. The method (100) according to claim 1, further comprising the following steps: receiving an audio feed (18) picked up by at least one microphone (15) disposed in the imaging compartment (3); Wherein, controlling the electronic processing device (8) not to process the one or more video feeds further comprises controlling not to operate the at least one camera or the microphone to prevent the generation of the video feed and the audio feed.

3. The method according to any one of claims 1 and 2, wherein: The method (100) further comprises: In response to the detecting indicating that a medical procedure is not being performed in the imaging bay (3), the local electronic processing device (8) is controlled to perform one or more training tasks for a machine learning (ML) component (40).

4. The method according to claim 3, wherein: The method (100) further comprises: Retrieving a plurality of ML models (40); and Distributing the training of at least one of the plurality of ML models to the local electronic processing device (8).

5. The method according to any one of claims 1 to 4, wherein: The one or more video feeds include a video feed (17) including a captured controller screen video feed of a medical imaging device controller (10) used in the medical procedure being performed in the imaging bay, and controlling the local electronic processing device (8) to process the one or more video feeds to extract and present information about the medical procedure being performed in the imaging bay includes: identifying a text region containing text in the grab controller screen video feed; and classifying the text region as quasi-static or dynamic; performing optical character recognition (OCR) to continuously extract content of the dynamic text region during the medical procedure being performed in the imaging bay; and OCR is performed to extract the content of the quasi-static text region only during the time when the medical procedure is being performed in the imaging booth when the content of the quasi-static text region may change.

6. The method according to any one of claims 1 to 5, wherein: Controlling the local electronic processing device (8) to process the one or more video feeds (17) to extract and present information about the medical procedure being performed in the imaging bay (3) comprises: applying a first machine learning (ML) model (40) to extract first information from the one or more video feeds (17); applying a plurality of second ML models to extract second information from the one or more video feeds; and The first information and the second information are combined to extract the information presented about the medical procedure being performed in the imaging bay.

7. The method according to claim 6, wherein: Combining the first information and the second information includes using a voting process.

8. The method according to any one of claims 1 to 7, wherein: The local electronic processing device (8) is also programmed to provide a communication interface between a user of the local electronic processing device and a remote expert located remote from the imaging bay to which the local electronic processing device is assigned.

9. A support device for medical imaging, the support device comprising: Server computers (14s); as well as a local electronic processing device (8) assigned to a respective medical imaging bay (3) and programmed to apply a machine learning (ML) model (40) to a video feed (17) received from the respective imaging bay to which it is assigned to extract information about a medical imaging procedure performed in the respective imaging bay to which it is assigned; wherein the server computer and / or the local electronic processing device is programmed to determine whether a medical imaging procedure is being performed in the corresponding medical imaging bay; and The server computer is programmed to perform training of the ML model, including: allocating ML model training tasks among the local electronic processing devices based on whether a medical imaging procedure is being performed in the corresponding assigned medical imaging compartment; and receiving results of the allocated ML model training tasks from the local electronic processing devices.

10. The support device according to claim 9, wherein: The server computer (14s) receives feedback from the local electronic processing device (8) indicating the performance of each of the ML models (40) in extracting the information about the medical imaging procedure performed in the assigned corresponding imaging bay (3); and The server computer is also programmed to distribute the ML model training tasks among the ML models based on the feedback received from the local electronic processing device indicating the performance of each ML model.

11. The support device according to any one of claims 9 and 10, wherein: Each local electronic processing device (8) is also programmed to provide a communication interface (19) between a user of the local electronic processing device and a remote expert located remote from the imaging booth (3) to which the local electronic processing device is assigned.

12. A non-transitory computer-readable medium (26s) storing instructions executable by at least one electronic processing device (8, 14s) to perform a method (100) of monitoring a medical imaging examination, the method comprising: receiving one or more video feeds (17) of at least the imaging compartment (3); detecting whether a medical procedure is being performed in the imaging bay based on the one or more video feeds; In response to the detecting indicating that a medical procedure is being performed in the imaging bay, controlling a local electronic processing device (8) assigned to the imaging bay to process the one or more video feeds to extract and present information about the medical procedure being performed in the imaging bay; as well as In response to the detecting indicating that a medical procedure is not being performed in the imaging bay (3), the local electronic processing device (8) is controlled to perform one or more training tasks for a machine learning (ML) component (40).

13. The non-transitory computer readable medium (26s) according to claim 12, wherein: The non-transitory computer readable medium (26s) further comprises executing the following steps: receiving an audio feed (18) picked up by at least one microphone (15) disposed in the imaging compartment (3); Wherein, controlling the electronic processing device (8) not to process the one or more video feeds further comprises controlling not to operate the at least one camera or the microphone to prevent the generation of the video feed and the audio feed.

14. The non-transitory computer readable medium (26s) according to any one of claims 12 and 13, wherein: The non-transitory computer readable medium (26s) further comprises executing the following steps: In response to the detecting indicating that a medical procedure is not being performed in the imaging bay, the local electronic processing device is controlled to not process the one or more video feeds.

15. The non-transitory computer-readable medium (26s) according to any one of claims 12 to 14, wherein: The method (100) further comprises: Retrieving a plurality of ML models (40); and Distributing the training of at least one of the plurality of ML models to the local electronic processing device (8).

16. A computer program comprising instructions which, when said program is run by a computer, cause said computer to perform the steps according to any one of claims 1 to 15.

Images