Ghosting mode glasses for servicing medical devices and for servicing workflows using same
Using historical services to call video assist engineers in on-site maintenance of medical equipment through augmented reality technology, the problem of viewpoint mismatch is solved and maintenance accuracy and efficiency is improved.
Patent Information
- Application Number
- CN202380029820.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-12
- Filing Date
- 2023-03-06
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
During on-site maintenance of medical equipment, it is difficult for field service engineers to accurately understand and perform maintenance actions through traditional text descriptions and drawing manuals, especially due to mismatch in viewpoints.
Using augmented reality technology, obtain videos of current service through AR-HUD devices and cameras, select and provide historical service call videos on display devices to overlap with the engineer's field of view, providing transparent or translucent overlays, guiding and assisting repair actions .
Significantly improve the accuracy and efficiency of on-site maintenance, enable engineers to perform maintenance actions more accurately through real-time visual assistance, reduce the possibility of errors, and provide an efficient maintenance method.
Smart Images

Figure CN118974748A_ABST
Abstract
Description
Technical Field
[0001] The following generally relates to the fields of medical devices, medical device maintenance, augmented reality (AR), and related fields. Background Art
[0002] Medical devices sometimes fail, which requires maintenance. Depending on the failure, the problem can sometimes be handled remotely by a remote service engineer (RSE). The RSE can perform remote diagnostics by analyzing log data generated from the failed medical device or by performing diagnostic tests remotely. In some cases, the RSE can successfully resolve the problem completely remotely.
[0003] However, for some faults, remote resolution is not possible. In such cases, in addition to remote diagnosis, a field service engineer (FSE) visits the physical location of the medical device and performs on-site maintenance. The FSE typically has a report of the main findings from the RSE, and based on this information, they may need to perform additional troubleshooting steps to discover the root cause of the problem, and / or replace parts of the machine and / or perform other types of maintenance or repairs.
[0004] When performing field service of a medical device, the FSE performs inspection and repair actions on the device, such as running diagnostic tests, inspecting the device for physical damage, replacing parts, etc. Typically, the FSE relies on comparing the medical device in service with text descriptions and / or drawings of the actions to be performed, which are provided in a maintenance manual (paper or electronic, such as pdf format). This can be challenging because the text descriptions need to be translated by the FSE to the physical device under observation, and even drawings may not be sufficient because they typically show a single viewpoint (e.g., in perspective) or again require translation by the FSE to the physical device under observation (e.g., in front view / top view / side view).
[0005] Certain improvements are disclosed below to overcome these and other problems. Summary of the invention
[0006] In one aspect, a non-transitory computer-readable medium stores: data including recorded historical service call videos for one or more medical imaging devices, the videos annotated with information about the service calls; and instructions capable of being read and executed by at least one electronic processor to: control a camera to acquire video of a current medical device being serviced in a current service call; select one of the historical service call videos based on the acquired video; and provide the selected historical service call video on a display device.
[0007] In another aspect, a system for providing maintenance videos for one or more medical devices includes an augmented reality head-up display (AR-HUD) device, the augmented reality head-up display device including a camera and at least one display device. A server computer is configured to: control the camera to acquire a video of a current medical device being serviced in a current service call; select one or more historical service call videos based on the acquired video, the historical service call videos including pre-recorded historical service call videos for one or more medical imaging devices, the videos being annotated with information about the service call; and provide the selected one or more historical service call videos as a transparent or semi-transparent overlay on the display device.
[0008] In another aspect, a system for providing maintenance videos for one or more medical devices includes an AR-HUD device having at least one display device. A server computer is configured to: select one or more historical service call videos, the historical service call videos including pre-recorded historical service call videos for one or more medical imaging devices, the videos annotated with information about the service calls; and provide the selected one or more historical service call videos as a transparent or semi-transparent overlay on the display device.
[0009] One advantage resides in assisting engineers in performing maintenance on medical devices using pre-recorded videos of maintenance actions presented on an augmented reality (AR) device. Another advantage resides in providing an efficient method for creating AR content for such AR-assisted maintenance.
[0010] Another advantage is that pre-recorded video of maintenance actions on the equipment is displayed to overlap with the engineer's field of view, guide the engineer's actions or suggest possible actions.Another advantage is that pre-recorded video of maintenance actions on the equipment is used to replace maintenance AR technology content.
[0011] It will become apparent to one of ordinary skill in the art upon reading and understanding the present disclosure that a given embodiment may provide none, one, two, more, or all of the aforementioned advantages, and / or may provide other advantages. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The present disclosure may be carried out in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the present disclosure.
[0013] Figure 1 An illustrative system for selecting videos for display according to the present disclosure is schematically shown.
[0014] Figure 2Shows Figure 1 An exemplary flow chart operation of a system. DETAILED DESCRIPTION
[0015] In the embodiments disclosed herein, the services of the medical device benefit from support for being able to display actions to be performed, in that the displayed information overlaps with the engineer's field of view, thereby guiding the engineer's actions or suggesting possible actions. The disclosed embodiments are implemented through augmented reality (AR), where relevant digital content is added as AR content in real time overlaid on the FSE viewpoint by using AR glasses, goggles, etc.
[0016] More specifically, the following discloses the use of AR glasses or an AR head-mounted kit to provide visual assistance to field service engineers (FSEs) when repairing medical imaging equipment. A database of recorded historical service call videos is maintained, and the historical service call videos are annotated with information about the service call. Each video is acquired by the FSE, for example using a forward-looking camera of the AR glasses or AR head-mounted kit worn by the FSE performing the historical service call. (In a variant embodiment, the historical video can be acquired using a forward-looking camera mounted on the head of the FSE but without a camera mount that provides AR rendering capabilities). Thus, a large database of historical service call videos can be created, with minimal archiving work limited to selecting the "best" service call videos to include in the database (for example, failed service calls where the FSE failed to resolve the problem will preferably be excluded from the database), and adding information metadata, such as the brand / model and service type of the medical imaging equipment shown in the video. The video retrieval component uses the metadata to select the most similar video, which is then presented to the FSE handling the current service call. In an illustrative example, the service call video is presented by the AR glasses or head-mounted kit as a transparent or translucent overlay (also referred to herein as a "ghost" display due to its transparent or translucent nature). Since the video will show the (historical) imaging device from the vantage point of the historical FSE that recorded it, this may not match the vantage point from which the current FSE is viewing the (current) imaging device, but this can be resolved by having the current FSE physically move and / or tilt his or her head to match the vantage point of the video. Such movement is also advantageous because it places the current service FSE performing the current medical imaging device in the same physical position as the current medical imaging device as captured by the FSE in the pre-recorded video when performing the action.
[0017] In some variant embodiments, rather than the database storing historical service calls for actual service of customer-installed medical imaging equipment, it may store historical service calls that were simulated service calls, e.g., performed not to provide actual service, but to generate AR content. This may have the advantage of being an "ideal" service call performed by an expert service technician in a well-controlled environment. In another variant, both historical service calls and simulations may be stored.
[0018] In other embodiments, some Video Chaptering Similarly, a video can be segmented into chapters to enable the current FSE to jump to the relevant part of the video. Typical video playback controls such as rewind and pause are provided appropriately, and optionally also zoom capabilities are provided.
[0019] Other disclosed aspects relate to selecting a pre-recorded video to be presented to assist with a particular current service call. In one variation, a third party (such as a remote service engineer (RSE)) that issues the current FSE can pre-select a video for the current FSE. In another variation, an automatic video selector selects a video to assist with the current service call based on, for example, comparing the content of the work order for the current service call with metadata associated with a database of pre-recorded videos. If the automatic video selector identifies more than one approaching video, the system can provide a user interface via which the current FSE can select a video to be played back, for example, using tactile controls on the AR glasses or head-mounted kit or using an in-AR menu, for example driven by the FSE focusing on the displayed menu option and using an eye tracker built into the AR glasses or head-mounted kit to detect this.
[0020] In yet another embodiment, the AR may provide other information, such as a summary of the current instance.
[0021] refer to Figure 1, an illustrative apparatus or system 10 for providing maintenance video of one or more medical devices 12 (e.g., MRI devices or other imaging modality devices) in a hospital or other medical facility is shown. To this end, the apparatus 10 includes a stereo camera 14, which is configured to acquire stereo images or videos of the medical devices 12 and / or the locations of structures within the medical facility. The stereo camera 14 typically includes a plurality of lenses or lens assemblies, wherein there is a separate sensor for each lens, which forms an image on a digital detector array (e.g., a CCD imaging array, a CMOS imaging array, etc.) to capture a 3D image. The stereo camera 14 preferably (but not necessarily) has color video capabilities, for example, by having the imaging array have pixels that are sensitive to red light, green light, and blue light (or another set of colors that substantially span the visible spectrum (e.g., 400-700nm)). The stereo camera 14 may optionally include other typical features, such as depth detection, a built-in flash (not shown), and / or an ambient light sensor (not shown) for setting the aperture, ISO, and / or exposure time. Figure 1 Also shown is an electronic processing device 18, such as a workstation computer, a smart phone, a tablet computer, etc. Additionally or alternatively, the electronic processing device 18 may be implemented as a server computer or multiple server computers, e.g., interconnected to form a server cluster, a cloud computing resource, etc. In other embodiments, the electronic processing device 18 may be integrated into an augmented reality head-up (AR-HUD) device 16. Various combinations or distributions of processing hardware are also contemplated.
[0022] The electronic processing device 18 includes typical components such as an electronic processor 20 (e.g., a microprocessor), at least one user input device (e.g., a mouse, keyboard, trackball, etc.) 22, and at least one display device 24 (e.g., an LCD display, an OLED display, a touch-sensitive display, a plasma display, a cathode ray tube display, etc.). In some embodiments, the display device 24 may be a separate component from the electronic processing device 18. The display device 24 may also include two or more display devices.
[0023] The electronic processor 20 is operably connected to one or more non-transient storage media 26. As non-limiting illustrative examples, the non-transient storage media 26 may include one or more of the following: a disk, a RAID or other magnetic storage media; a solid-state drive, a flash drive, an electrically erasable read-only memory (EEROM) or other electronic memory; an optical disk or other optical storage device; various combinations thereof; etc.; and may be, for example, a network storage device, an internal hard drive of the electronic processing device 18, various combinations thereof, etc. It should be understood that any reference to one or more non-transient media 26 herein should be broadly interpreted to cover a single medium or multiple media of the same or different types. Similarly, the electronic processor 20 can be implemented as a single electronic processor or two or more electronic processors. The non-transient storage medium 26 stores instructions that can be executed by at least one electronic processor 20. The instructions include instructions for generating a graphical user interface (GUI) 28 for display on a display device 24.
[0024] In some embodiments, the stereo camera 14 includes an AR-HUD device 16 having one or more AR-HUD displays including a display device 24. The illustrative design employs left-eye and right-eye displays 24, but alternatively, the display may be a single large window across both eyes. In some examples, the stereo camera 14 is mounted to the AR-HUD device 16 to provide a “first-person perspective” in order to align the AR content with the actual view seen through the (one or more) transparent displays 24 of the AR-HUD. In some examples, the AR-HUD device 16 may be configured as a helmet, headband, glasses, goggles, or other suitable embodiments to be worn on the user's head. The stereo camera 14 is mounted to the AR-HUD device 16 (e.g., to cover the user's forehead, or include two stereo cameras disposed on the lenses of the glasses).
[0025] Figure 1 Also shown is an AR-HUD device 16 in communication with a server computer 30 that includes a non-transitory computer-readable medium. The server computer 30 may include a database that stores a plurality of recorded historical service call videos 32 for one or more medical imaging devices (i.e., including the medical device 12). As used herein, the term "recorded historical service calls" (and variations thereof) may refer to recorded service calls performed on a "model" medical imaging device for providing assistance to a user of the AR-HUD device 16. The historical service call videos 32 may be annotated with information about the service calls. The historical service call videos 32 may also be pre-recorded videos (i.e., before a service call is generated). In some embodiments, the recorded historical service call videos 32 include recorded videos of historical service calls that are simulated service calls that depict "ideal" service calls.
[0026] Furthermore, as disclosed herein, the non-transitory storage medium 26 (or the server computer 30) performs a method or process 100 for providing a maintenance video for one or more medical devices 12. Figure 2 , and continue to refer to Figure 1 , the server computer 30 is configured as described above to perform the method 100 for providing a maintenance video for one or more medical devices 12. The server computer 30 stores instructions that can be read and executed by the server computer 30 to perform the disclosed operations, including executing the method or process 100. In some examples, the method 100 can be performed at least in part by cloud processing.
[0027] In some embodiments, the apparatus 10 may be implemented using an AR-HUD device 16 to acquire images of the medical device 12, and the electronic processing device 18 may be implemented as a workstation computer (or tablet) that processes the acquired images to determine a maintenance video. In other embodiments, the electronic processing device 18 may be implemented in the AR-HUD device 16, in which case at least one user input device 22 may, for example, include a gaze tracker (not shown) built into the AR-HUD device 16, which is combined with a selection button, menu, or other user interface dialog presented as AR content and selectable by a user who directs his or her gaze at a button, menu option, etc. and stares at it for a predetermined selection time.
[0028] At operation 102, the camera 12 is controlled (e.g., by the server computer 30) to acquire video 34 of the current medical device (e.g., the medical device 12) being serviced in the current service call, and to curate historical video and add it to a historical video database on the server computer 30. For example, an experienced FSE records video while performing a service task for which the FSE is particularly well qualified. The resulting recorded video may be reviewed by the FSE or another qualified individual (e.g., a service department manager) to determine whether it is of sufficiently high quality to be included in the server computer 30. If so, appropriate video editing may be performed, such as extracting the video clip recording the service action from the overall video (which may include events recorded before and / or after the actual service action), optionally adding chapter points or segment points to enable a user to jump to a specific portion of the video, adding text metadata describing the recorded action, recording the brand / model of the imaging device it serviced (and possibly other similar brands / indications that may apply to the video), date information, etc. These actions may generally be performed using any suitable video editing software. The resulting pre-recorded video is converted to AR content, e.g., by assigning the left and right channels of the stereoscopic video to corresponding left and right AR channels, reformatting the video into an AR format used by the AR-HUD device 16, optionally compressing the video to reduce its file size and thus the bandwidth used when sending the video, etc. It should be understood that operation 102 is suitably applied to a wide range of videos acquired during the normal course of service of medical imaging devices in a fleet of medical imaging devices maintained by a service provider, such that over time the database accumulates a large amount of AR content that overlays a wide range of service actions across a range of brands / models of devices. Advantageously, the amount of additional effort beyond the service calls already performed during the normal course of service operations is limited to selecting the "best" videos and curating these best videos by adding text descriptions, chapters, or other metadata and running automatic video format conversions to AR content.
[0029] At operation 104, one (or more) of the historical service call videos 32 is selected based on the acquired video 34. In one embodiment, the recorded historical service call videos 32 include metadata related to the type of medical device and the type of service performed that was added during the curation of operation 102. The selection operation 104 includes analyzing the metadata of the recorded historical service call videos 32 and selecting the historical service call video 32 based on having metadata that is most similar to the metadata of the acquired video 34.
[0030] At operation 106, the selected historical service call video 32 is provided on the display device 24. In one example, the selected historical service call video 32 is provided on the display device 24 of the electronic processing device 18. In another example, the selected historical service call video 32 is provided as augmented reality content on the display device 24 of the AR-HUD device 16. The selected historical service call video 32 can be displayed on the display device 24 as a transparent or semi-transparent overlay. Optionally, the transparency of the augmented reality content can be adjustable, such as by adjusting the alpha channel transparency value in the case of using alpha transparency. In some examples, a summary of the current service call can be provided on the display device 24.
[0031] At operation 108, the AR-HUD device 16 receives one or more inputs. In one embodiment, the RSE wearing the AR-HUD device 16 may provide input including a selection of one or more of the historical service call videos 32 (i.e., via an AR user interface that the user interacts with via gaze selection). Alternatively, the input may be provided by the electronic processing device 18 via at least one user input device 22 (e.g., a mouse click, a keystroke, etc.). In another example, a plurality of historical service call videos 32 are displayed on the display device 24, and the user may provide input indicating a selection of one of the historical service call videos 32.
[0032] At optional operation 109 , the selected historical service call video 32 may be segmented into one or more sub-portions, and the received input may indicate fast forwarding or rewinding to a different sub-portion of the selected historical service call video 32 .
[0033] In some embodiments, the AR-HUD device 16 may detect whether the gaze of the user of the AR-HUD device 16 matches the vantage point of the displayed historical service call video 32. Instructions may then be provided on the display device 24 for the user to change the gaze to match the vantage point of the displayed historical service call video 32.
[0034] exist Figure 2 , a linear workflow is shown, wherein historical video acquisition and curation operation 102 precedes selection and playback operations 104, 106, 108, 109. However, while selection operation 104 may only select from pre-recorded videos generated by operation 102, it should be understood that operation 102 may be considered ongoing, as each service call performed may itself be a source of additional video content for inclusion in the database via a corresponding instance of operation 102. Thus, as a medical imaging device vendor (or other service provider) provides service to a population of medical imaging devices, the database of AR content continues to grow over time.
[0035] Example
[0036] The system 10 and method 100 are described in more detail below. The system 10 includes a wearable display device that includes an AR-HUD device 16 that can be used by a FSE while performing field maintenance service of a physical system (e.g., a faulty medical device 12). The camera 14 of the AR-HUD device 16 captures a first-person video of the field maintenance operation. The AR-HUD device 16 displays a pre-recorded video 32 with an adjustable transparency level to allow the FSE to overlay the current video 34 with the real scene to match the video 34 and the state of the device area and the medical device 12 in the real environment. The videos 32 can be records of actual maintenance actions taken while handling actual situations, or they can be records of simulations of all known maintenance actions.
[0037] Given a case being processed, the relevant videos 32 to be displayed are obtained by a retrieval mechanism implemented by the server computer 30. This can work on a collection of historical cases, retrieving the cases most similar to the current case, and selecting the associated videos 32. Alternatively, the mechanism can use a collection of metadata information associated with each simulated video 32, and perform matching based on these. The number of relevant videos 32 retrieved can vary from one to multiple.
[0038] The diagnostic session for the current service case c may include, for example, customer input (i.e., an initial customer query regarding a system failure), remote diagnostic findings (i.e., the results of remote diagnostic actions performed by the RSE), and / or medical device 12 information (i.e., any information automatically generated by the system itself regarding the status and history of the medical device 12 being repaired, such as log files), etc.
[0039] The server computer 30 stores pre-recorded videos 32 and is also configured to implement function f to retrieve the most relevant video 32 given the current case information. This will match the current case c with the historical cases, resulting in a list of N relevant related cases. The video 32 corresponding to the selected case will then be sent to the AR-HUD device 16.
[0040] The server computer 30 stores the actual video file, plus a mapping function between the video and all related case details. The historical case details may be stored in any information management system (e.g., a database) with query capabilities on different attributes of the case. The mapping may be achieved by any association mechanism, including but not limited to the same database, where the video file path and case details are aligned in the same table or collection, or in different tables that share a common index, such as using a case unique identifier as a key.
[0041] The AR-HUD device 16 can be any wearable device that incorporates some lenses (as a single piece or in multiple parts) that are placed in front of the FSE's eyes when worn, such as glasses. The display device 24 includes lenses that display video, so that the FSE can watch the video through the lenses. The camera 14 attached or incorporated into the device frame is used to record the video 34 of the maintenance action when the maintenance action is performed. The camera 14 can be a device separate from the lens 24 and bound to it by some means. The information displayed by the lens 24 is controlled by an interface I (including but not limited to keys and knobs) to adjust the video stream and transparency, and to display, navigate and interact with other available information sources. The interface can be integrated into the lens 24, or implemented in a separate device connected by the lens, such as a tablet computer connected to the lens via a wireless communication system.
[0042] The repository of historical cases H is historical cases h 1 ,…,h n A set of h i Several case details are incorporated, including but not limited to (1) a customer query regarding the fault being presented, (2) an RSE report of diagnostic findings and actions performed so far, (3) information generated by the medical device 12 (i.e., log data), (4) the type of medical device 12 and detailed hardware configuration, (5) the installation type (e.g., right / left panel, cabinet order, ceiling rail or not, etc.) - this can be a textual description, or a collection of images that visually describe the system being processed, (6) the handedness of the engineer performing the on-site intervention, etc. The current case c being processed (processed by the RSE before the FSE intervention) can contain a subset or all of the details specified in the historical cases. Given these prerequisites, the video retrieval function is defined in Equation 1:
[0043] f (c, H) = S (1)
[0044] in, is an ordered set where historical cases are ranked according to their similarity relevance scores with c computed by f itself. Depending on design choices, the size of S can be zero or larger. Due to the mapping described previously, videos 32 corresponding to cases in S are retrieved. f can be implemented as a combination of one or more information retrieval methods, textual and / or visual, that utilizes one or more case information to evaluate the similarity of c with any h. i The similarities between.
[0045] The selected video 32 is then prepared for streaming on the lens 24, where the FSE uses the device interface I to select which video to play.
[0046] Advantageously, the FSE will be able to review video 32 of a maintenance action similar or identical to the one being performed. Additionally, due to ghosting mode, such video (i.e., first person view) can be overlaid with the FSE field of view, making it easier for the FSE to identify sensitive areas, follow and mimic specific maintenance procedures, etc. Ultimately, this support will allow the FSE to operate more accurately and faster.
[0047] In some embodiments, instead of recording a real case, the video 32 may be a live recorded simulation of a repair action for a known fault. Instead of recording each repair action performed by the FSE, it is possible to have an expert FSE perform a real simulation of the repair action for all known faults that require on-site maintenance. For this embodiment, the AR-HUD device 16 will not require the camera 14 to always be mounted on the lens frame. However, the video of the simulated repair action is still a first-person shot with a wearable camera.
[0048] To retrieve relevant videos for the current case, each simulation video 32 can be associated with all historical cases of the maintenance action described for its selection. The retrieval function f will work as described above, except that the case-video association is many-to-one, i.e., several cases will point to the same video.
[0049] Alternatively, each video 32 may be associated with a set of meta-information (e.g., a short description of the video content, system type, type of fault solved, etc.) (rather than having direct case information), and the matching function f will use such meta-information to match the current case with relevant videos.
[0050] The main technical effect of this embodiment is to greatly reduce the number of videos stored and the number of videos to search in. Potentially, for each known repair action, there will be only one video 32, so that the set of alternatives will be less noisy in terms of selection. In addition, the videos 32 will be more streamlined in what they represent, because they will only show the required actions without any other possible off-track intervention of the FSE, such as doing something wrong.
[0051] In some embodiments, both real case videos and simulated videos can be implemented. To this end, two different video repositories can be used. One repository contains videos that are records of actual maintenance actions performed by the FSE, and one repository contains simulated videos. In this case, each historical case h i Can be paired with both actual video and simulated video. Video retrieval function f will work the same as previous embodiment. Alternatively, if meta information is used for simulated video, two separate retrieval functions can be implemented, one for actual maintenance action video and one for simulation.
[0052] This embodiment will prevent those cases where errors occur when recording real maintenance actions, such as video not being recorded for some reason, wrong steps being taken during maintenance, etc. In these cases, the simulated video will be used as a backup.
[0053] In some embodiments, the historical video 32 can be segmented in time, and each segment can be tagged with information about the part being processed to allow retrieval of shorter videos. The maintenance action video 32 can be segmented in time so that specific segments can be retrieved independently, showing sub-actions in the entire maintenance video. In this case, each sub-video will still be paired with the relevant historical case and corresponding information, but a lot of additional information will also be associated with it, including but not limited to, the unique identifier of the part being manipulated / replaced (e.g., part 12NC), the type of part, the action performed, etc. Function f will also use such information to evaluate the similarity with the current case.
[0054] In this embodiment, it is possible to retrieve only segments of the video, rather than the entire video. This will help in having a more refined action selection. For example, suppose the case being processed will require the execution of action a 1 、a 2 、a 3 . Let us assume that the three actions are not included together in a single video, but that they exist in three separate videos. Without this embodiment, three entire maintenance videos would be retrieved and sent to the FSE. In turn, the FSE would have to browse through the three videos, looking for a single action. In contrast, with this embodiment, the FSE would still receive three videos, but these videos would be shorter and already labeled with the action type and component details, making them easy to identify.
[0055] In some embodiments, the selection of which video(s) 32 to display may be performed by the RSE. The selection of the video(s) 32 to be played on the FSE lens 24 may be performed by the RSE before the case is actually sent to the FSE. The RSE will select the most relevant video(s) 32 for the case being processed either manually (e.g., the RSE selects based on telediagnosis findings and her / his experience) or semi-automatically (e.g., applying a search function f to generate a video list S, which is then additionally filtered by the RSE based on her / his findings).
[0056] Furthermore, the RSE can also operate a selection from a remote command control center, where the RSE can examine the live feed from the FSE camera 16, i.e. the FSE viewpoint overlaid with the selected video. In this way, the RSE can actively support the FSE as a complement to the visual references from the video.
[0057] Advantageously, the selection of relevant videos 32 also incorporates the experience of the RSE. This may result in a shorter and more accurate selection of videos, as it will not rely on an automated process. Furthermore, additional real-time support from the RSE to the FSE is provided, potentially making field maintenance operations more accurate.
[0058] In some embodiments, each historical video 32 can have multiple additional information layers on top (additional information added manually or automatically), for example, labels of sensitive or interesting parts of the medical imaging device superimposed in the video, or reference points on the medical device 12. The maintenance action video 32 can be enhanced with several additional information annotated thereon, including but not limited to visual labels of sensitive parts, descriptions of the actions being performed, other important visual landmarks about the equipment and / or actions. Such labels can be introduced manually (e.g., through a maintenance video labeling process performed by the FSE itself or by other experts) or automatically introduced by means of dedicated image processing and computer vision models. A hybrid approach to such labeling is also possible, for example, first automatically annotating the video 32 and performing manual refinement in a second stage.
[0059] The additional information annotated on the video 32 will provide additional support to the FSE. Important parts or processes will be highlighted, making the FSE intervention sharper and faster.
[0060] In some embodiments, in addition to displaying the video 32, the lens 24 will also allow for superimposition of current case salient information for quick access by the FSE. Possibly, the interface I on the lens 24 may also be used to display and navigate related historical case information. Details about historical cases related to the video 32 being displayed may also be reported. Whether such information is displayed and how much will be controlled by the FSE through the same interface I used to control the video stream. The FSE will be able to more quickly review case details and find salient references to scenes seen, actions to be performed, and physical equipment.
[0061] The system 10 can be implemented for field assistance of any medical imaging system of any modality (interventional X-ray (iXR), digital X-ray (DXR), magnetic resonance (MR), computed tomography (CT), advanced medical imaging (AMI), and ultrasound). However, it can be applied to any equipment where field maintenance is required and possible.
[0062] The present disclosure has been described with reference to the preferred embodiments. Modifications and alterations may occur to others after reading and understanding the foregoing detailed description. The exemplary embodiments are intended to be interpreted as including all such modifications and alterations as long as they fall within the scope of the appended claims or their equivalents.
Claims
1. A non-transitory computer-readable medium (30) storing: Data comprising recorded historical service call videos (32) for one or more medical imaging devices (12), the videos annotated with information about the service calls; and Instructions capable of being read and executed by at least one electronic processor (20) to perform the following operations: controlling a camera (14) to acquire video (34) of a current medical device being serviced in a current service call; selecting one of the historical service call videos based on the acquired video; as well as The selected historical service call video is provided on a display device (24).
2. The non-transitory computer readable medium (30) of claim 1, wherein: The information about the service call includes at least one of: visual markings or reference points indicating components of the medical imaging device represented in the video, a description of one or more service actions being performed on the medical imaging device represented in the video, the brand and / or model of the imaging device for which the service was recorded, the date of recording, logos on the medical imaging device represented in the video, and / or one or more segmentation points that segment the video into chapters.
3. The non-transitory computer readable medium (30) according to any one of claims 1-2, wherein: The display device (24) includes an augmented reality head-up display (AR-HUD) device (16), and the selected historical service call video is provided as AR content on the display device of the AR-HUD device.
4. The non-transitory computer readable medium (30) according to any one of claims 1 to 3, wherein: The recorded historical service call videos (32) include metadata related to the type of medical device and the type of service performed, and selecting one or more of the historical service call videos based on the acquired video (34) includes: analyzing the metadata of the recorded historical service call videos; and One or more recorded historical service call videos having metadata most similar to the metadata of the acquired video are selected.
5. The non-transitory computer readable medium (30) according to any one of claims 1 to 4, wherein: Selecting one or more of the historical service call videos (32) based on the acquired videos (34) includes: An input is received from a remote service engineer (RSE) including a selection of one or more of the historical service call videos.
6. The non-transitory computer readable medium (30) according to any one of claims 1 to 5, wherein: Selecting one or more of the historical service call videos (32) based on the acquired videos (34) includes: Select multiple historical service call videos; providing a user interface (UI) (28) showing the selected historical service call video on the display device (24), and An input indicating a selection of one of the historical service call videos shown on the UI is received.
7. The non-transitory computer readable medium (30) according to any one of claims 1 to 6, wherein: The instructions also include: Segmenting the selected historical service call video (32) into one or more sub-portions; and An input is received indicating a fast forward or a rewind to a different sub-portion of a selected historical service call video.
8. The non-transitory computer readable medium (30) according to any one of claims 1 to 7, wherein: The recorded historical service call video (32) includes recorded video of the historical service call as a simulated service call, the simulated service call depicting an ideal service call.
9. The non-transitory computer readable medium (30) according to any one of claims 1 to 8, wherein: The instructions also include: A summary of the current service call is provided on the display device (24).
10. An augmented reality head-up display (AR-HUD) device (16), comprising: Camera (14); Display device (24); as well as The non-transitory computer readable medium (30) according to any one of claims 1-8; The selected historical service call video (32) is provided on the display device of the AR-HUD device as augmented reality content including a transparent or semi-transparent overlay.
11. The AR-HUD device (16) according to claim 10, wherein: The instructions also include: detecting whether a gaze of a user of the AR-HUD device (16) matches a vantage point of a displayed historical service call video (32); and Instructions are provided on the display device to cause the user to change the gaze to match the vantage point of the displayed historical service call video.
12. A system (10) for providing a maintenance video for one or more medical devices, the system comprising: An augmented reality head-up display (AR-HUD) device (16) comprising a camera (14) and at least one display device (24); as well as A server computer (30) configured to: controlling the camera to acquire video of the current medical device being serviced in the current service call (34); selecting one or more historical service call videos (32) based on the acquired video, the historical service call videos comprising pre-recorded historical service call videos for one or more medical imaging devices, the videos annotated with information about the service call; as well as The selected one or more historical service call videos are provided as a transparent or semi-transparent overlay on the display device.
13. The system (10) of claim 12, wherein: The recorded historical service call video (32) includes metadata related to the type of medical device and the type of service performed, and the server computer (30) is configured to: analyzing metadata of the recorded historical service call videos; as well as One or more recorded historical service call videos having metadata most similar to the metadata of the acquired video are selected.
14. The system (10) according to any one of claims 12 and 13, wherein: The server computer (30) is configured to: An input is received from a remote service engineer (RSE) including a selection of one or more of the historical service call videos (32).
15. The system (10) according to any one of claims 12 to 14, wherein: The server computer (30) is configured to: Select multiple historical service call videos; providing a user interface (UI) (28) for displaying the selected historical service call video on the display device (24), and An input is received indicating a selection of one of the historical service call videos displayed on the UI.
16. The system (10) according to any one of claims 12 to 15, wherein: The server computer (30) is configured to: Segmenting the selected historical service call video (32) into one or more sub-portions; and An input is received indicating a fast forward or a rewind of a different sub-portion of a selected historical service call video.
17. The system (10) according to any one of claims 12 to 16, wherein: The recorded historical service call video (32) includes recorded video of the historical service call as a simulated service call, the simulated service call depicting an ideal service call.
18. The system (10) according to any one of claims 12 to 17, wherein: The server computer (30) is configured to: A summary of the current service call is provided on the display device (24).
19. The system (10) according to any one of claims 12 to 18, wherein: The server computer (30) is configured to: detecting whether a gaze of a user of the AR-HUD device (16) matches a vantage point of a displayed historical service call video (32); as well as Instructions are provided on the display device to cause the user to change the gaze to match the vantage point of the displayed historical service call video.
20. A system (10) for providing a maintenance video for one or more medical devices (12), the system comprising: An augmented reality head-up display (AR-HUD) device (16) having at least one display device (24); as well as A server computer (30) configured to: selecting one or more historical service call videos (32), the historical service call videos comprising pre-recorded historical service call videos for one or more medical imaging devices, the videos annotated with information about the service calls; as well as The selected one or more historical service call videos are provided as a transparent or semi-transparent overlay on the display device.
