Vendor agnostic remote control screen overlay for collaboration in virtualized radiation environments

Abstract

A remote expert assists a local operator of a medical imaging device during a medical imaging examination. A user interface (UI) is provided on an assist electronic device operable by a remote expert. The UI displays a controller video output by an electronic controller of the medical imaging device. A graphical annotation for updating the controller video is received via at least one user input device of the assist electronic device. The controller video superimposed with the graphical annotation is displayed on a controller display of the medical imaging device. In one method, this entails splitting a controller video into a plurality of feeds, transmitting one of the feeds to assist electronics, and transmitting another of the feeds to a screen overlay processor, the screen overlay processor also receives a graphical annotation from an assist electronic device and generates the controller video overlaid with the graphical annotation.

Description

Technical Field

[0001] The following generally relates to imaging technology, remote imaging assistance technology, remote imaging inspection monitoring technology and related technologies. Background Art

[0002] Medical imaging (such as computed tomography (CT) imaging, magnetic resonance imaging (MRI), positron emission tomography (PET) imaging, fluoroscopic imaging, etc.) is a key component of providing medical care and is used in a wide range of medical fields, such as cardiology, oncology, neurology, orthopedics, etc. The operators of medical imaging equipment used to acquire medical images are typically trained technicians, while the interpretation of medical images is typically handled by medical specialists such as radiologists. The radiologist's interpretation of the radiology report or findings may be handled by the patient's general practitioner (GP) or a medical specialist such as a cardiologist, oncologist, orthopedic surgeon, etc.

[0003] Currently, diagnostic imaging is in high demand. As the world's population grows, the demand for fast, safe, high-quality imaging will only continue to grow, placing further pressure on imaging centers and their staff. In this situation, mistakes will occur, and mistakes are often very costly. One way for imaging centers to improve efficiency and grow operations without the accompanying increase in labor costs is through the Radiology Operations Command Center (ROCC) system. The Radiology Operations Command Center enables teams to work across a network of imaging sites, provide their expertise as needed, and remotely assist less experienced technicians in performing high-quality scans. Remote technicians or experts can monitor the local operator of the scanning process through cameras installed in the scanning area (or from other sources, such as sensors (including radar sensors), console video feeds, microphones connected to Internet of Things (IoT) devices, etc.). In addition, these sources can be supplemented by other data sources, such as Health Level 7 (HL7) medical data feeds, Digital Imaging and Communications in Medicine (DICOM), Electronic Health Record (EHR) databases, etc.

[0004] When collaborating via the ROCC platform, local and remote technicians can communicate via audio or video calls on a tablet computer. In addition, the remote technician can see the local console screen via video capture and transmission. Thus, the remote technician can observe control or other actions taken by the local technician, and can also observe the mouse pointer or other on-display user interface elements controlled by the local technician.

[0005] In many cases, due to safety or security regulations, the remote technician will only be allowed to view the local screen and not interact with the scanner system or local input devices. In addition, interpretation of user interface (UI) interactions or identification of specific areas in the image must be done via audio communications because the remote operator cannot interact with the local console screen. This makes assisted sessions with inexperienced local staff particularly difficult.

[0006] Certain improvements that overcome these and other problems are disclosed below. Summary of the invention

[0007] In one aspect, an assistance system is disclosed for providing remote assistance to a local operator of a medical imaging device having an electronic imaging device controller and a controller display. The assistance system includes an assistor electronic device, the assistor electronic device including an electronic processor, an assistor display, and at least one user input device. The assistance system also includes a video router, the video router being configured to: receive a controller video from the electronic imaging device controller; receive overlay content or video from the assistor electronic device; output the controller video to the assistor electronic device; and output an annotation video to the controller display, whereby the annotation video is displayed on the controller display. The assistor electronic device is programmed to provide a graphical user interface (GUI) operable to display the controller video on the assistor display and receive graphical annotations of the controller video from the at least one user input device. The assistor electronic device is also programmed to generate overlay content or video including at least the graphical annotations. The annotation video includes the controller video with the graphical annotations superimposed thereon.

[0008] In another aspect, a method of providing assistance from a remote expert to a local operator of a medical imaging device during a medical imaging examination is disclosed. The method includes providing a user interface (UI) on an assistor electronic device operable by the remote expert. The UI displays a controller video output by an electronic controller of the medical imaging device. The method also includes receiving one or more graphical annotations via at least one user input device of the assistor electronic device to update the controller video. The controller video with the graphical annotations superimposed is displayed on a controller display of the medical imaging device.

[0009] In another aspect, an assistance system is disclosed for providing remote assistance to a local operator of a medical imaging device having an electronic imaging device controller and a controller display. The assistance system includes an assistor electronic device and a video router. The assistor electronic device includes an electronic processor, an assistor display, and at least one user input device. The video router is configured to: receive a controller video from the electronic imaging device controller; receive graphical annotations from the assistor electronic device; output the controller video to the assistor electronic device; and, output an annotation video to the controller display, whereby the annotation video is displayed on the controller display. The assistor electronic device is programmed to: provide a GUI operable to display the controller video on the assistor display; and, receive the graphical annotations of the controller video from the at least one user input device. The annotation video includes the controller video with the graphical annotations superimposed thereon.

[0010] One advantage resides in providing a remote specialist or radiologist assisting a technician in performing a medical imaging examination with the ability to annotate a screen operable by the technician.

[0011] Another advantage is that a remote specialist or radiologist assists a technician in performing a medical imaging examination by being able to annotate a screen operable by the technician without interfering with the local imaging system or host computer.

[0012] Another advantage is that screen overlays generated by the remote specialist or radiologist are created directly on the video signal of the local console monitor.

[0013] Another advantage resides in creating such a screen overlay that is controlled in real time by a remote operator.

[0014] A given embodiment may provide none of the aforementioned advantages, provide one, two, more, or all of the aforementioned advantages, and / or may provide other advantages that will be apparent to one of ordinary skill in the art upon reading and understanding the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The disclosure may take form in various components and arrangements of components, and in various steps and arrangements of steps.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the disclosure.

[0016] Figure 1 An illustrative apparatus for providing remote assistance in accordance with the present disclosure is schematically shown.

[0017] Figure 2 Shows Figure 1 Another aspect of the device.

[0018] Figure 3 Shown by Figure 1 An example flow chart of operations suitably performed by an apparatus.

[0019] Figure 4 Some examples of highlighting overlay elements are schematically shown.

[0020] Figure 5 , Figure 6 and Figure 7 Shows Figure 2 Alternative embodiments to the illustrated embodiment. DETAILED DESCRIPTION

[0021] In some current implementations of a remote operations control center (ROCC), the remote expert receives a copy of a captured controller screen and can therefore see what the local operator sees. The captured controller screen viewed by the remote expert typically includes a mouse pointer or other user interface element(s) controlled by the local operator so the local operator can point to features on the screen during communications with the remote expert. However, there is no mechanism by which the remote expert can point to features on the controller screen in a manner that the local operator can see.

[0022] Embodiments disclosed herein address this deficiency by providing a mechanism for a remote expert to draw annotations on a controller display that are visible to a local operator as an overlay on the controller display. To make this vendor agnostic, in some embodiments, the method operates on a video signal, intercepting it as it passes from an imaging device controller to a display viewed by a local operator, and modifying the video signal to add the overlay.

[0023] Some existing ROCC implementations use a keyboard, video and mouse (KVM) switch to act as a transmitter of a video signal, and a video splitter is configured to split the video signal into two paths: a path of a video display seen by a local operator, and a second path of a captured screen that is sent to a remote expert. In order to achieve the overlay disclosed herein, an additional screen overlay processor is inserted in the path from the KVM switch (or more generally, a video router) to the local operator display. The operation of the screen overlay processor depends on the nature of the video signal. In some embodiments, for digital video signals such as HMDI, DisplayPort or digital DVI, the screen overlay processor can be implemented as a digital signal processing (DSP) component, such as using a graphics processing unit (GPU) and hardware acceleration to provide sufficient speed. On the other hand, if the video signal is analog, such as an analog DVI signal, an additional analog-to-digital (A / D) front end and a digital-to-analog (D / A) back end are appropriately adopted, wherein the digitized display content output by the A / D front end is executed DSP before the subsequent D / A back end. In some embodiments, annotated overlay elements may be received from a remote expert as vectorized or symbolic graphic elements, such as represented as lines or shapes depicted by coordinates, along with color and transparency values. The DSP then converts these into screen overlays that are fused with the received controller display video.

[0024] At the remote expert workstation, appropriate hardware and software are added to implement a screen annotation user interface (UI). For example, the workstation may have a touch-sensitive display so that the remote expert can draw on a window of the presentation controller display, and / or a mouse pointer, touchpad, etc. may be used for this purpose. As described above, in some embodiments, the remote expert workstation converts these annotations into vectorized or symbolic graphic elements that are sent to the screen overlay processor via the Internet.

[0025] One potential problem is that annotations added by the remote expert may confuse or distract the local operator. To address this potential problem, in some embodiments, a "kill switch" is provided on the local operator side, which can enable the local operator to turn off the annotation overlay for safety. As another contemplated variation, in addition to the annotations created directly by the remote expert, additional annotations may be overlaid on the local operator screen, such as a red box around the entire controller display, indicating that the remote operator is viewing it.

[0026] refer to Figure 1 , shows an apparatus 1 for providing assistance from a remote medical imaging expert RE (or super technician) to a local technician operator LO. Figure 1As shown, a local operator LO operating a medical imaging device (also referred to as an image acquisition device, imaging device, etc.) 2 is located in a medical imaging device compartment 3, and a remote expert RE is provided in a remote service location or center 4. It should be noted that the "remote expert" RE may not necessarily directly operate the medical imaging device 2, but rather provide assistance to the local operator LO in the form of advice, guidance, instructions, etc. The remote location 4 may be a remote service center, a radiologist's office, a radiology department, etc. The remote location 4 may be in the same building as the medical imaging device compartment 3 (for example, this may be the case where the "remote operator or expert" RE is a radiologist responsible for performing peri-examination image review), but more typically, the remote service center 4 and the medical imaging device compartment 3 are in different buildings, and may actually be located in different cities, different countries, and / or different continents. Typically, the remote location 4 is remote from the imaging device compartment 3 in the sense that the remote expert RE cannot directly visually observe the imaging device 2 in the imaging device compartment 3 (thus optionally providing a video feed as further described herein).

[0027] The image acquisition device 2 may be a magnetic resonance (MR) image acquisition device, a computed tomography (CT) image acquisition device; a positron emission tomography (PET) image acquisition device; a single photon emission computed tomography (SPECT) image acquisition device; an X-ray image acquisition device; an ultrasound (US) image acquisition device; or another modality of medical imaging device. The imaging device 2 may also be a hybrid imaging device, such as a PET / CT or SPECT / CT imaging system. Although in Figure 1 A single image acquisition device 2 is shown schematically in FIG, but more typically a medical imaging laboratory will have multiple image acquisition devices, which may have the same and / or different imaging modalities. In addition, the remote service center 4 may provide services to multiple hospitals. A local operator controls the medical imaging device 2 via an electronic imaging device controller 10. The remote operator is located at an auxiliary electronic device 12 (or more generally, a remote workstation 12 or electronic controller 12).

[0028] To provide contrast-enhanced imaging in certain imaging modalities, an optional contrast injector 11 is configured to inject contrast media into a patient. The contrast injector 11 is a configurable, automated contrast injector having a display 13. A user (typically an imaging technician) loads a vial or syringe of contrast media (or two or more vials of different contrast media components) into the contrast injector 11, and configures the contrast injector 11 by inputting contrast injector settings (such as flow rate, volume, time delay, injection duration, etc.) via a user interface (UI) of the contrast injector 11. The UI can be a touch-sensitive overlay of the display 13, and / or a physical button, keypad, etc. In a variant embodiment, the optional contrast injector 11 is integrated with the imaging device controller 10 (e.g., via a wired or wireless data connection), and the contrast injector 11 is controlled via the imaging device controller 10, including displaying the contrast injector settings in an (optionally selectable) window on the display of the imaging device controller 10.

[0029] like Figure 1 As shown, in some embodiments, a camera 16 (e.g., a video camera) is arranged to capture a video stream or feed 17 of a portion of the workspace of the medical imaging device compartment 3, the portion including at least the area where the local operator LO of the imaging device 2 interacts with the patient, and optionally may also include the imaging device controller 10. In some embodiments, a microphone 15 is arranged to capture an audio stream or feed 18 of the workspace so that the remote expert also has auditory situational awareness. The video stream 17 and / or audio stream 18 is sent to the auxiliary electronic device 12 via the communication link 14, for example as a streaming video feed received via a secure Internet link.

[0030] The communication link 14 also provides a natural language communication path 19 for verbal and / or textual communication between the local operator and the remote operator. For example, the natural language communication link 19 can be a Voice over Internet Protocol (VOIP) telephone connection, an online video chat link, a computerized instant messaging service, etc. Alternatively, the natural language communication path 19 can be provided by a dedicated communication link separate from the communication link 14 providing data communications 17, 18, for example, the natural language communication path 19 can be provided via a landline telephone. In some embodiments, the natural language communication link 19 allows the local operator LO to call a selected remote expert RE. As used herein, a call can refer to an audio call (e.g., a telephone call), a video call (e.g., Skype or Facetime or other screen sharing program), or an audio-video call. In another example, the natural language communication path 19 may be provided via a local electronic processing device, for example, including a ROCC device 8, such as a mobile device (e.g., a tablet computer or a smart phone), or may be a wearable device worn by a local operator LO, such as an augmented reality (AR) display device (e.g., AR glasses), a projector device, a head-up display (HUD) device, etc., each of which has a display device 36. For example, an “app” may be run on the ROCC device 8 (operable by the local operator LO) and the auxiliary electronic device 12 (operable by the remote expert RE) to allow communication (e.g., audio chat, video chat, etc.) between the local operator and the remote expert.

[0031] Figure 1Also shown is an auxiliary electronic device 12, such as an electronic processing device, a workstation computer, or more generally a computer, included in the remote service center 4, which is operably connected to receive and present a video feed 17 and / or an audio feed 18 of the medical imaging device compartment 3 from the camera 16. Additionally or alternatively, the auxiliary electronic device 12 can be embodied as a server computer or multiple server computers, for example interconnected to form a server cluster, cloud computing resources, etc. The workstation 12 includes typical components such as an electronic processor 20 (e.g., a microprocessor), at least one user input device (e.g., a mouse, keyboard, trackball, etc.) 22 and at least one display device 24 (e.g., an LCD display, a plasma display, etc.). In some embodiments, the display device 24 can be a separate component from the workstation 12. The display device 24 can also include two or more display devices. The electronic processor 20 is operably connected to one or more non-transitory storage media 26. As non-limiting illustrative examples, the non-transitory storage medium 26 may include one or more of the following: a disk, RAID or other magnetic storage medium; a solid-state drive, a flash drive, an electrically erasable read-only memory (EEROM) or other electronic memory; an optical disk or other optical storage device; various combinations thereof, etc.; and, for example, may be a network storage device, an internal hard drive of the workstation 12, various combinations thereof, etc. It should be understood that any reference to one or more non-transitory media 26 herein will be broadly interpreted to cover a single medium or multiple media of the same or different types. Similarly, the electronic processor 20 may be embodied as a single electronic processor or two or more electronic processors. The non-transitory storage medium 26 stores instructions executable by at least one electronic processor 20. The instructions include instructions for generating a graphical user interface (GUI) 28 for display on the remote operator display device 24. The video feed 17 from the camera 16 may also be displayed on the display device 24, and the audio feed 18 may be output on the auxiliary electronic device 12 via the speaker 29. In some examples, audio feed 18 may be the audio component of an audio / video feed (eg, a video recorder (VCR) device running as a recording).

[0032] The medical imaging device controller 10 in the medical imaging device compartment 3 also includes components similar to the auxiliary electronic device 12 provided in the remote service center 4. Unless otherwise specified herein, features of the medical imaging device controller 10 including the local workstation 12' provided in the medical imaging device compartment 3, which are similar to the features of the auxiliary electronic device 12 provided in the remote service center 4, have common reference numerals, accompanied by "superscript" symbols, and the description of the components of the medical imaging device controller 10 will not be repeated. In particular, the medical imaging device controller 10 is configured to display a GUI 28' on a display device or controller display 24', which presents information related to the control of the medical imaging device 2, such as a configuration display for adjusting configuration settings, an alarm 30 perceptible at a remote location when status information about a medical imaging examination meets an alarm criterion of the imaging device 2, imaging acquisition monitoring information, presentation of acquired medical images, etc. It should be understood that the screen mirroring data stream 18 carries content presented on the display device 24' of the medical imaging device controller 10. The communication link 14 allows screen sharing between a display device 24 in the remote service center 4 and a display device 24' in the medical imaging device bay 3. The GUI 28' includes one or more dialog screens, including, for example, an exam / scan selection dialog screen, a scan setup dialog screen, an acquisition monitoring dialog screen, etc. The GUI 28' may be included in the video feed 17 and displayed on the auxiliary electronic device display 24 at the remote location 4.

[0033] refer to Figure 2 , and continue to refer to Figure 1 , the device 1 includes a video router 40 (in Figure 1 2 and 3. The video router 40 may be a video router or a video controller. The video router 40 may include a keyboard, video, monitor (KVM) switch 41. In other examples, the video router 40 may be comprised of a set of discrete video signal splitter and combiner components. The video router 40 is configured to receive the controller video 17 from the electronic imaging device controller 10 and also receive overlay content or video 42 from the assistor electronic device 12. The controller video 17 is output to the assistor electronic device 12, where it is displayed on the display device 24 via the GUI 28 to allow the assistor to annotate the controller video displayed on the assistor electronic device 12.

[0034] The assistor electronic device 12 used by the remote expert RE is then programmed to provide input of a graphical annotation 44 indicative of the controller video 17 via at least one user input device 22. For example, the graphical annotation 44 may include one or more of a vectorized or symbolic graphic element, a bitmap graphic element, a color element, or a transparency value. Thus, the assistor electronic device 12 is configured to generate an overlay content or video 42 that is or is used to generate an annotation video 46 that includes at least the graphical annotation 44 to the controller display 24' of the electronic imaging device controller 10. Thus, the annotation video 46 includes the controller video 17 with the graphical annotation 44 superimposed.

[0035] In one embodiment, the overlay content or video 42 is generated at the assistive electronic device 12 as an annotation video 46 including the controller video 17 with the graphical annotation 44 overlaid thereon, and the video router 40 is configured to output the annotation video 46 received from the assistive electronic device 12 to the controller display 24'. That is, in this embodiment, the operation of combining the controller video 17 and the graphical annotation 44 is performed at the assistive electronic device 12 located at the remote expert RE.

[0036] In another embodiment, the overlay content or video 42 is generated as a graphical annotation 44 without the controller video 17, and the video router 40 includes a digital signal processor (DSP) 48 (in Figure 2The video router 40 includes a DSP 48 (identified as a screen overlay processor) that is programmed to generate an annotation video 46 by overlaying graphical annotations 44 received from the assistive electronic device 12 onto the controller video 17. That is, in this embodiment, the operation of combining the controller video 17 and the graphical annotations 44 is performed at the video router 40 located at the local operator LO. In this embodiment, the overlay content or video 42 received at the video router 40 represents the graphical annotations 44. In one embodiment, the overlay content or video 42 transmitted from the assistive electronic device 12 to the video router 40 is an overlay content that includes the graphical annotations 44 represented as vectorized or symbolic graphic elements (e.g., represented as lines or shapes depicted by coordinates) together with color and transparency values. In this embodiment, the DSP 48 of the video router 40 then converts these received vectorized or symbolic graphic elements into a screen overlay, which is then merged with the controller video 17 to produce the annotation video 46. In another embodiment, the overlay content or video 42 transmitted from the auxiliary electronic device 12 to the video router 40 includes an overlay video structured as a graphical annotation 44, which is converted into a bitmap video containing only the graphical annotation. In this case, the DSP 48 of the video router 40 then fuses the overlay video (which here contains only the annotation 44) and the controller video 17 to produce the annotation video 46. In either of these two embodiments, a kill switch 50 can be implemented at the video router 40 by stopping the fusion of the received overlay content or video 42 with the controller video 17.

[0037] exist Figure 2 In the illustrative example of , the video router 40 is configured to receive the overlay content or video 42 via the ROCC device 8 via the facilitator electronic device 12. However, in other embodiments, the video router 40 may receive the overlay content or video 42 directly from the network 14 without the ROCC device 8 acting as a relay.

[0038] In some embodiments, the video router 40 also includes a manual overlay disable switch 50 that can be operated by the local operator LO to switch the video router 40 from outputting the annotation video 46 to the controller display 24' to outputting the controller video 17 to the controller display 24'. In a variant embodiment where the controller video 17 and the graphical annotation 44 are combined at the auxiliary electronic device 12 located at the remote expert RE, the implementation of the kill switch 50 will require sending a kill signal to the auxiliary electronic device 12, and the auxiliary electronic device 12 will not add the annotation 44 to the controller video 17 in response to the activation of the kill switch 50.

[0039] Furthermore, as disclosed herein, the ROCC device 8, the assist electronic device 12, and the video router 40 are configured to perform a method or process 100 for providing assistance during a medical imaging examination performed using the medical imaging device 2 (i.e., by assisting a local operator LO of the corresponding medical imaging device 2 during a medical imaging examination by a remote expert RE). Instructions for performing the method 100 are stored in the non-transitory computer-readable medium 26 of the assist electronic device 12, in the video router 40, and in the ROCC device 8.

[0040] refer to Figure 3 , and continue to refer to Figure 1 and Figure 2 , an illustrative embodiment of method 100 is schematically shown as a flow chart. To begin method 100, an imaging examination is started by a local operator LO using a medical imaging device 2. An event may occur during the examination that requires assistance from a remote expert RE. The controller video feed (captured by one or more cameras 16) is routed to a video router 40.

[0041] At operation 102, a GUI 28 is provided on the assistor electronic device 12, and a controller video 17 is displayed thereon. At operation 104, one or more graphical annotations 44 are received by the assistor electronic device 12 (i.e., input by the remote expert RE via at least one user input device 22) and superimposed onto the controller video 17 to generate an annotation video 46. In some embodiments, the controller video 17 superimposed with the graphical annotations 44 is generated by a screen overlay component 48 that is part of the video router 40 and includes a digital signal processing (DSP) component.

[0042] At operation 106, the annotation video 46 is displayed on the controller display 24' of the medical imaging device controller 10. In some embodiments, the controller video 17 is split, i.e., via the KVM switch 41, or using a discrete video splitter, to split the controller video 17 into multiple feeds. One of the feeds is transmitted to the assistive electronic device 12. Another of the feeds is transmitted to the screen overlay processor 48, which also receives the graphical annotations 44 from the assistive electronic device 12, and the screen overlay processor 48 generates the controller video 17 with the graphical annotations 44 overlaid to generate the annotation video 46.

[0043] In another embodiment, the controller video 17 is an analog video, and the controller video 17 is converted to a digital controller video, which is processed by the DSP component 48 using an analog-to-digital converter (ADC) 52 implemented in the video router 40. The controller video 17 superimposed with the graphic annotations 44 (i.e., the annotation video 46) is converted to an analog signal via a digital-to-analog converter (DAC) 54 implemented in the video router 40.

[0044] In another embodiment, in response to the local operator LO activating the overlay disable switch 50 , the display of the controller video 17 with the graphical annotation 44 overlaid on the controller display 24 ′ is switched to displaying the controller video 17 without the graphical annotation 44 on the controller display 24 .

[0045] In another embodiment, the annotated video 46 is displayed on the display device 36 of the ROCC device 8 and the graphical annotation(s) 44 are received from the local operator LO, wherein the annotated video 46 includes the graphical annotation(s) 44 received from the local operator LO.

[0046] The system may provide the local operator LO with a kill switch 50 to toggle the screen overlay on and off. The kill switch 50 may be implemented by a touch or click button or switch displayed on a separate screen or by a physical manual button or switch 50. The kill switch 50 enables the local operator LO to avoid distraction by stopping the process of superimposing the annotation 44 on the controller video 17.

[0047] refer to Figure 4 , schematically showing some examples highlighting overlay elements drawn by a remote expert RE. Figure 4 Magnetic resonance angiography (MRA) images 60 and 62 are shown, such as may be displayed on the controller display 24' during an MRA imaging session. It should be understood that this is merely a non-limiting illustrative example, and the disclosed method for remote expert annotation may be more generally used in any medical imaging context (MR, CT, PET, etc.). The display may also include text fields, such as the non-limiting illustrative examples of "Patient: John Doe," "Age: 57," "Weight: 307 lb," and "Examination: Cardiac MRA." Figure 4 An example of an annotation video 46 is shown, including a combination of the controller video 17 and graphical annotations 441, 442, 443, 444, and 445 drawn by the remote expert RE using the assistive electronic device 12. Overlay elements 441 and 442 are areas highlighted by the remote expert RE, while annotation 443 is Figure 4, but other types of position markers are contemplated. For example, the area overlay elements 441 and 442 may be hand-drawn by the remote expert RE using a mouse pointer, a touch-sensitive display, or the like. Annotation 444 is an overlaid graphical annotation cursor 444 that may be operated by the remote expert RE to point out an object. In the illustrative example, the remote expert RE is using an overlaid graphical annotation cursor 444 to highlight the text "Weight: 307 lb," perhaps as part of explaining that the weight of the imaged object may explain some aspect of the imaging examination. It can be seen that the various graphical annotations 441, 442, 443, and 444 drawn by the remote expert RE enable the remote expert to identify features of an image, text, or other content shown on the controller display 24', thereby facilitating the communication of information to the local operator LO. The overlay elements are preferably displayed in a transparent manner so that information about the ROCC device 8 beneath the overlay elements remains visible to the local operator LO. Figure 4 Some examples of functional overlays shown in FIG. 4 include a boundary annotation 445 indicating that the remote expert RE is viewing the ROCC device 8 and a pointer symbol 444 already described, wherein the boundary annotation 445 indicates that the remote expert RE is viewing the ROCC device 8 and the pointer symbol 444 already described simulates the mouse movement of the remote expert RE on the shared screen. Figure 4 It is schematically shown as a dashed line, but may be implemented as, for example, a red border to indicate what the RE is observing.

[0048] The remote expert RE viewing the medical imaging device controller 10 via the KVM 41 will need to interact with the video feed 17 to define screen overlay elements. To this end, the auxiliary electronic device 12 includes a drawing tool for screen overlay elements. The drawing tool allows the remote expert RE to draw overlay elements directly onto the image of the shared screen.

[0049] The software then converts the screen coordinates of the element to the screen coordinates of the medical imaging device controller 10 (if the screen resolution or aspect ratio is different). Alternatively, the software calculates relative coordinates with respect to the screen extension and leaves the determination of absolute coordinates to the ROCC device 8 or screen overlay processor 48.

[0050] All information required to generate the overlay is transmitted in real time to the ROCC device 8 via the network connection 14. This information includes, but is not limited to: coordinates, shapes, colors, and transparency levels for all overlay elements; coordinates, shapes, colors, and transparency levels of the pointer; status information of additional functional overlays, such as screen sharing activity, etc.

[0051] refer to Figure 5-7 , presents Figure 2 Some alternative embodiments of the illustrated embodiment. Each of these embodiments includes a similar Figure 2Components of the components are indicated by the same reference numerals. Figure 2 The same as the example, Figure 5-7 Each embodiment includes a video router 40, which receives a controller video 17 from an electronic imaging device controller 10 and overlay content or video 42 from an auxiliary electronic device 12, and the video router 40 outputs the controller video 17 to the auxiliary electronic device 12, and also outputs an annotation video 46 to the controller display 24', whereby the annotation video 46 is displayed on the controller display 24'.

[0052] exist Figure 5 In an embodiment of the present invention, the overlay content or video 42 is constructed prior to transmission via the KVM 41. This has the advantage that a remote operator using the assistive electronic device 12 sees exactly what the local operator sees on the controller display 24'. Figure 5 A possible disadvantage of the approach is that drawing elements may appear to have some delay due to network latency.

[0053] Figure 6 An embodiment similar to Figure 5 However, in Figure 6 In the embodiment of the present invention, the overlay content or video 42 is in the form of UI inputs (e.g., keyboard and mouse inputs 42) at the assistive electronic device 12, which are made by the operator of the assistive electronic device 12 when drawing, typing, or otherwise creating the graphical annotations 44. These UI inputs that form the overlay content 42 are transmitted through the KVM 41 and fed to the ROCC device 8, which generates the overlay content or video 42 and then inputs it to the video router 40.

[0054] exist Figure 7 In the embodiment, it is shown that Figure 6 A similar system, except that Figure 7 The UI input forming the overlay content 42 is directly input to the screen overlay processor 48 (rather than the ROCC device 8), and the screen overlay processor 48 directly interprets the UI input forming the overlay content 42 to generate the annotation video 46, which is then fed to the controller display 24' via the KVM 41.

[0055] In the following, some further variant embodiments are described.

[0056] In some embodiments, the graphic overlay element has additional parameters to control animation, blinking, etc. to increase visibility.

[0057] In some embodiments, the remote expert RE or ROCC device 8 must request sharing of the overlay. The local operator LO must then confirm the request before the screen overlay function is activated.

[0058] In some embodiments, the local operator LO can use a local pointer device (mouse, etc.) to interact with overlay elements on the ROCC device 8. To accomplish this, the movement of the local pointer device must be processed by the ROCC device 8 or the screen overlay processor 48. A KVM 41 connected not only to the annotation(s) 44 but also to the pointer device can facilitate this functionality.

[0059] Aspects of the present invention can be implemented in a computer program product, which can be a set of computer program instructions that can be executed by a computer and stored on a computer-readable storage device. Instructions of the present invention can be any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs) or Java classes. Instructions can be provided as complete executable programs, partial executable programs, modifications (e.g., updates) to existing programs, or extensions (e.g., plug-ins) to existing programs. In addition, the parts of the processing of the present invention can be distributed on multiple computers or processors.

[0060] The processor or processing unit may include or consist of a controller for implementing the control method. The controller can be implemented in a variety of ways using software and / or hardware to perform the various functions required. A processor is an example of a controller that uses one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. However, the controller can be implemented with or without a processor, and can also be implemented as a combination of dedicated hardware for performing some functions and a processor (e.g., one or more programmed microprocessors and additional circuits) for performing other functions.

[0061] Examples of controller components that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0062] In various embodiments, a processor or controller may be associated with one or more storage media, such as volatile (transient) and non-volatile (non-transient) computer memory, such as RAM, PROM, EPROM, and EEPROM. The storage medium may be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform the desired functions. Various storage media may be fixed within the processor or controller, may be transportable, or may be available on demand (e.g., via the cloud) so that one or more programs stored thereon may be loaded into the processor or controller. Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description are to be considered illustrative or exemplary rather than restrictive; the present invention is not limited to the disclosed embodiments.

[0063] The present disclosure has been described with reference to the preferred embodiments. Modifications and changes may be made by others upon reading and understanding the foregoing detailed description. It is intended that the exemplary embodiments be interpreted as including all such modifications and changes as long as they fall within the scope of the appended claims or their equivalents. By studying the drawings, the disclosure and the appended claims, those skilled in the art may understand and implement other changes to the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "one" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously. The computer program may be stored / distributed on a suitable medium, for example, an optical storage medium or solid-state medium provided together with or as part of other hardware, but may also be distributed in other forms, for example, via the Internet or other wired or wireless telecommunications systems. Any reference numerals in the claims should not be interpreted as limiting the scope.

Claims

1. An assistance system (1) for providing remote assistance to a local operator (LO) of a medical imaging device (2) having an electronic imaging device controller (10) and a controller display (24'), the assistance system comprising: Auxiliary electronic device (12) comprising a processor (20), an auxiliary display (24), and at least one user input device (22); A video router (40) configured to: receiving controller video (17) from the electronic imaging device controller; receiving overlay content or video (42) from the auxiliary electronic device; outputting the controller video to the auxiliary electronic device; as well as outputting an annotated video (46) to the controller display, thereby displaying the annotated video on the controller display; Wherein, the assistive device electronic device is programmed to: providing a graphical user interface (GUI) (28) operable to display said controller video on said assistor display; and receiving a graphical annotation (44) of the controller video from the at least one user input device, the assistive electronic device further being programmed to generate the overlay content or video (42) including at least the graphical annotation; and Wherein, the annotated video includes the controller video with the graphic annotation superimposed by the video router (40).

2. The assistance system (1) according to claim 1, wherein: The assist electronic device (12) is programmed to generate the overlay content or video (42) as the annotated video (46) including the controller video (17) with the graphical annotation (44) superimposed thereon, and the video router (40) is configured to output the annotated video received from the assist electronic device to the controller display (24').

3. The assistance system (1) according to claim 1, wherein: The assist electronic device (12) is programmed to generate the overlay content or video (42) including the graphical annotation (44) without the controller video (17), and the video router (40) includes a digital signal processor (48) programmed to generate the annotation video (46) by combining the overlay content or video (42) received from the assist electronic device with the controller video (17).

4. The assistance system (1) according to claim 3, wherein: The overlay content or video (42) received from the assistive electronic device (12) is overlay content in which the graphic annotation (44) is represented as vectorized or symbolic graphic elements, and the digital signal processor (48) is programmed to generate the annotation video (46) by: converting the vectorized or symbolic graphic elements to a screen overlay; and, merging the screen overlay with the controller video (17) to produce the annotation video (46).

5. The assistance system (1) according to claim 3, wherein: The overlay content or video (42) received from the assistive electronic device (12) is an overlay video in which the graphical annotation (44) is represented as an overlay video, and the digital signal processor (48) is programmed to generate the annotation video (46) by fusing the overlay video with the controller video (17) to produce the annotation video (46).

6. The assistance system (1) according to any one of claims 4 to 5, wherein: The video router (40) includes an overlay disable switch (50) operable by the local operator (LO) to cause the video router (40) to stop combining the overlay content or video (42) received from the auxiliary electronic device with the controller video (17).

7. The assistance system (1) according to any one of claims 1 to 6, further comprising: A local electronic device (8) operable by the local operator (LO), wherein the video router (40) is configured to receive the overlay content or video (42) from the auxiliary electronic device (12) via the local electronic device, and output the controller video (17) to the auxiliary electronic device via the local electronic device.

8. The assistance system (1) according to any one of claims 1 to 7, wherein: The video router (40) includes a keyboard, video, monitor (KVM) switch (41).

9. A method (100) of providing assistance from a remote expert (RE) to a local operator (LO) of a medical imaging device (2) during a medical imaging examination, the method comprising: receiving a controller video (17) output between an assistive device electronic device (12) and an electronic controller (10) of the medical imaging device; providing a user interface (UI) (28) on an assistant electronic device (12) operable by the remote expert, the UI displaying a controller video (17); receiving overlay content via at least one user input device (22) of the assistive electronic device; combining the overlay content with the controller video (17) between the electronic controller (10) and the auxiliary electronic device (12); and The controller video is displayed with graphical annotations superimposed on a controller display (24') of the medical imaging device.

10. The method (100) according to claim 9, further comprising: Splitting the controller video (17) into multiple feeds; transmitting one of the feeds to the ancillary electronics (12); as well as Another of the feeds is transmitted to a screen overlay processor (48) which also receives the graphical annotation (44) from the assistive electronic device and generates the controller video (17) with the graphical annotation overlaid thereon.

11. The method (100) according to claim 10, wherein: The splitting includes employing a keyboard, video and mouse (KVM) switch (41) to split the feed (17) into the plurality of feeds.

12. The method (100) according to any one of claims 9 to 11, wherein: The assistive electronic device (12) implements a screen overlay component (48) configured to generate the controller video (17) with the graphical annotation (44) overlaid thereon.

13. The method (100) according to any one of claims 9 to 12, wherein: The controller video (17) with the graphical annotation (44) superimposed thereon is generated by a screen overlay component (48) including a digital signal processing (DSP) component.

14. The method (100) according to claim 13, wherein: The controller video (17) is an analog signal, and the method further comprises: converting the controller video using an analog-to-digital converter (ADC) (52) to a digital controller video processed by the DSP component (48); and The controller video with the graphical annotation (44) superimposed thereon is converted using a digital-to-analog converter (DAC) (54) to an analog signal for display by the controller display (24').

15. The method (100) according to any one of claims 9 to 14, wherein: The graphical annotation (44) includes one or more of a vectorized or symbolic graphical element, a color element, or a transparency value.

16. The method (100) according to any one of claims 9 to 15, further comprising: In response to the local operator (LO) activating an overlay disable switch (50), switching from displaying the controller video (17) with the graphical annotation (44) overlaid on the controller display (24') to displaying the controller video without the graphical annotation on the controller display.

17. The method (100) according to any one of claims 9 to 16, wherein: The controller display (36) includes a touch screen, and the method (100) further includes: One or more graphical annotations are received from the local operator (LO) via the touch screen, wherein the controller video (17) is further overlaid with the graphical annotations from the local operator.

18. An assistance system (1) for providing remote assistance to a local operator (LO) of a medical imaging device (2) having an electronic imaging device controller (10) and a controller display (24'), the assistance system comprising: Auxiliary electronic device (12) comprising a processor (20), an auxiliary display (24), and at least one user input device (22); A video router (40) configured to: receiving controller video (17) from the electronic imaging device controller; receiving a graphical annotation (44) from the assistive electronic device; outputting the controller video to the auxiliary electronic device; as well as outputting an annotated video (46) to the controller display, thereby displaying the annotated video on the controller display; Wherein, the assistive device electronic device is programmed to: providing a graphical user interface (GUI) (28) operable to display said controller video on said assistor display; and receiving the graphical annotation of the controller video from the at least one user input device (44); and Wherein, the annotated video includes the controller video with the graphic annotation superimposed by the video router (40).

19. The assistance system (1) according to claim 18, wherein: The assist electronic device (12) is programmed to represent the graphical annotation (44) as vectorized or symbolic graphic elements, and the video router (40) includes a digital signal processor (48) programmed to generate the annotation video (46) by combining the vectorized or symbolic graphic elements received from the assist electronic device with the controller video.

20. The assistance system (1) according to any one of claims 18-19, wherein: The video router (40) includes an overlay disable switch (50) operable by the local operator (LO) to stop the overlay of the controller video (17) with the graphical annotation (44).

21. The assistance system (1) according to any one of claims 18 to 20, further comprising: A local electronic device (8) operable by the local operator (LO), wherein the video router (40) is configured to receive the graphic annotation (44) from the auxiliary electronic device (12) via the local electronic device and output the controller video (17) to the auxiliary electronic device via the local electronic device.

Images