Augmented reality service instruction library based on self-expansion

Abstract

An augmented reality (AR) service guide uses a mobile device (30, 90) with a display and a camera. Computer vision (CV) processing (102) such as simultaneous localization and mapping (SLAM) is performed to align AR content (80) with a live video feed (96) captured by the camera. The aligned AR content is displayed on the display of the mobile device. In one example, the mobile device is a heads-up display (HUD) (90) with a transparent display (92, 94) and a camera (20). In another example, the mobile device is a mobile phone or tablet computer (30) with a front display (34) and a rear camera (36). The AR content (80) is generated by CV processing (54) to align (70) the AR content with a recorded video (40) of a service call. The alignment (72) includes aligning a location of interest (LOI) identified by user input in the recorded video of the service call.

Description

Technical Field

[0001] The following relates generally to the field of field services, and more particularly to the field of medical imaging equipment and system services, to the field of enhanced vision, and to related fields. Background Art

[0002] Medical imaging devices include very complex systems such as magnetic resonance imaging (MRI) devices, transmission computed tomography (CT) imaging devices, emission imaging systems (such as positron emission tomography (PET) imaging devices and gamma cameras for single photon emission computed tomography (SPECT) imaging), hybrid systems that provide multiple modalities in a single device (such as PET / CT or SPECT / CT imaging devices), and imaging devices designed to guide biopsies or other interventional medical procedures, often referred to as image-guided therapy (iGT) devices. These are merely illustrative examples.

[0003] Modern medical imaging equipment and systems present unusual challenges from a maintenance perspective. They typically include a large number of subsystems for widely varying purposes, such as: mechanical and electromechanical systems for loading patients, transporting to the imaging examination area, and subsequently unloading; X-ray tube assemblies and X-ray detector arrays in the case of CT imaging equipment; again, in the case of CT, rotating gantry; in the case of PET imaging equipment, annular rings of radiation detector modules; in the case of gamma cameras, radiation detector heads and associated robotic hardware for manipulating the radiation detector heads around patients; in the case of MRI, superconducting magnets and associated liquid helium tanks and / or other cryostat hardware; user interface components, such as computer controllers, control buttons, switches, touch-sensitive displays, etc.; and so on. Maintenance of medical imaging equipment must usually be performed on-site, i.e., in the hospital or other medical institution where the medical imaging equipment is deployed. In addition, since hospitals and other medical institutions expect these devices to be available almost continuously to provide imaging or image-guided therapy to patients, it is very desirable to minimize maintenance downtime.

[0004] A further difficulty from a repair perspective is that medical imaging devices are typically under near-continuous development by manufacturers and marketed with a variety of optional components and component packages, so that different deployed installations of the same make / model of medical imaging devices, depending on when they were manufactured and the options the customer chose to purchase, may have different components.

[0005] These considerations make it particularly challenging for service personnel to have a comprehensive understanding of every potential maintenance task that may be encountered in the field. Service personnel can consult written service manuals and / or computer-aided design (CAD) drawings; however, these may become outdated over time or may not include specific optional parts. In the case of CAD drawings, it may be difficult for service personnel to determine the correct drawing (part, revision, etc.) to reference. Furthermore, if the service manual is not written by a field service personnel, the manual may not address aspects that are only apparent when performing field service: for example, a particular part may be difficult to access in the field, but the design engineer who drafted the service manual may not be aware of this difficulty.

[0006] Another resource for the service personnel in the field may be telephone or video conferencing with access to an expert who specializes in the specific part, equipment, etc. that is being serviced. This takes up the expert's valuable time and requires coordination between the field service personnel and the expert - either the "expert" must be on standby during service hours, or the date / time of the call must be scheduled in advance. In addition, the expert may provide assistance based on limited information. This information typically includes a verbal description of the part and its environment provided by the service personnel, and perhaps also includes a video of the part provided using the built-in camera of a cellular phone (cell phone) or other mobile device. Important information may be lost in transmitting contextual data from the service personnel to the expert and / or conveying the expert's advice to the service personnel.

[0007] New and improved systems and methods are disclosed below. Summary of the invention

[0008] In one disclosed aspect, an augmented reality (AR)-based service guidance device includes: a mobile device including a display and a camera; and an electronic processor programmed to perform computer vision (CV) processing to align AR content with a live video feed captured by the camera of the mobile device and display the AR content aligned with the live video feed on the display of the mobile device.

[0009] In another disclosed aspect, a service guidance method based on augmented reality includes: using a computer to create AR content, including the following operations: receiving user input on the computer, the user input identifying at least one location of interest (LOI) in a recorded video of a service call, and performing CV processing to align the AR content with the recorded video of the service call, including aligning a portion of the AR content with the at least one LOI; and using an electronic processor and a mobile device including a display and a camera, presenting the created AR content, including using the electronic processor to perform CV processing to align the created AR content with a live video feed generated by a camera of the mobile device, and displaying the created AR content aligned with the live video feed on the display screen of the mobile device.

[0010] In another disclosed aspect, an AR content creation device is disclosed for creating AR content for service guidance. The device includes a computer and a non-transitory storage medium storing computer-readable and executable instructions to perform an AR content creation method, the method comprising: receiving user input on the computer, the user input identifying at least one LOI in a recorded video of a service call; and performing CV processing to align the AR content with the recorded video of the service call, including aligning a portion of the AR content with the at least one LOI.

[0011] One advantage resides in providing real-time automated assistance to service personnel.

[0012] Another advantage resides in providing real-time augmented reality (AR) content to assist in service calls.

[0013] Another advantage resides in providing real-time AR content to assist in service calls made from the service personnel's perspective that are tightly integrated with the service operations to be performed.

[0014] Another advantage resides in providing real-time AR content to assist in service calls, combined with the ability to switch to a preview mode that incorporates AR content.

[0015] Another advantage resides in providing real-time AR content utilizing a live video feed to automatically determine appropriate AR content.

[0016] Another advantage is leveraging existing service calls to provide data for authoring AR-based service instructions.

[0017] Another advantage resides in leveraging existing service calls to provide data to efficiently author updates to existing AR-based service instructions.

[0018] A given embodiment may provide none of the aforementioned advantages, provide one, two, more, or all of the aforementioned advantages, and / or may provide other advantages, as will become apparent to one of ordinary skill in the art upon reading and understanding this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The invention may take form in various components and arrangements of components, and in various steps and arrangements of steps.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0020] Figure 1 An embodiment for performing maintenance on a medical imaging device using a service call to provide data for authoring augmented reality (AR)-based service instructions, and a schematic diagram of an AR content authoring device using the data are diagrammatically illustrated.

[0021] Figure 2 Diagrammatically illustrates Figure 1 A more detailed embodiment of an AR content creation device.

[0022] Figure 3 Diagrammatically illustrates an AR-based service instruction delivery device for delivering Figure 1 and 2 AR content created by AR content creation devices. DETAILED DESCRIPTION

[0023] In the improvements disclosed herein, video recordings (optionally including audio) of service calls for performing maintenance on medical imaging equipment are utilized to provide a basis for creating augmented reality (AR) content for AR-based service guidance. The service call can be recorded using a conventional camera, or a stereo camera and / or a range-finding camera can be used to provide depth information. If a single (non-range-finding) camera is used, depth information can be extracted from different vantage points as the technician moves around. The camera can be head-mounted to provide a "first-person" view for the service personnel. During the service call, an indicator can optionally be used to mark the location of interest (LOI), and / or voice annotations of the call can be provided for recording.

[0024] The authoring system is used to convert recorded video of a service call into AR content to support future service calls for the same or similar maintenance tasks. The authoring system can use computer vision (CV) techniques (such as simultaneous localization and mapping (SLAM) algorithms) to simultaneously map the location of the service personnel and the significant environment (e.g., the service parts and possible adjacent parts). The extracted map can be used to align the parts in the recorded video with the computer-aided design (CAD) drawing, and the LOI is spatially located in three-dimensional space. The human editor can then add a verbal explanation overlay (or can be done using a text-to-speech transcription of a written service manual). The human editor can also appropriately add an overlay showing the LOI marker, and can optionally add other AR content that helps the service technician, such as part numbers, pop-ups showing CAD drawings, CAD animations or other visual effects, color highlights of key parts, and so on. To facilitate use in the field, various stop points can be added to the AR content, and / or the AR content can be segmented into scenes that can be played back in different orders (i.e., non-linear playback) to allow processing services to perform calls in different orders.

[0025] To use the authorized AR-based service manual content, the service technician must wear a heads-up display (HUD) with a transparent display screen (e.g., a see-through display) that displays the AR overlay on the transparent screen of the HUD so that it is superimposed on the service technician's actual view of the part being serviced. The HUD also includes a camera that captures the technician's line of sight and applies a SLAM algorithm to align the AR content with this real-time video feed so that the AR content overlay (and audio playback, if provided) matches the service technician's real-time view.

[0026] The service personnel can select a preview mode in which a pre-recorded video of the service call used to author the AR content is displayed, along with the AR content superimposed. In this way, the service personnel can preview how the steps of the maintenance task are performed. Advantageously, the AR processing remains unchanged in this preview mode, except that: (i) SLAM processing is performed to align the AR content with the preview video instead of the live video feed, and (ii) the pre-recorded video content is displayed as an underlay on the transparent display.

[0027] In another contemplated variation, a live video feed may be processed to identify specific components or sub-components so that appropriate AR content can be automatically retrieved and presented.

[0028] In an illustrative embodiment, the HUD uses a see-through display that is advantageously transparent except for the AR content overlay, thereby enabling the service personnel to directly observe the part in real time. In a variant embodiment, the HUD employs an opaque display over which live video is displayed as an underlay of the AR content (referred to as video augmented reality) - in this variant, the service personnel indirectly "sees" the part being serviced by virtue of viewing the live video feed display.

[0029] As another contemplated variation, instead of a HUD, the AR content can be displayed on the display of a phone or other mobile device using the built-in camera provided by most phones and other mobile devices to generate a live video feed. In this case, the device operates in video augmented reality by displaying live video on the phone display as the underlying layer.

[0030] The disclosed method can also be used to update AR-based service manuals. For example, if a service person encounters an unexpected problem, such as due to the use of a substitute part in a component of a particular deployed medical imaging device being serviced, the live feed provided by the HUD camera can be combined with the authoring system to produce updated AR content suitable for the substitute part.

[0031] refer to Figure 1, a system for recording a service call is illustrated. The object of the illustrative service call is an imaging device 10, which in this illustrative example is a hybrid imaging device including a transmission computed tomography (CT) imaging gantry 12 and a positron emission tomography (PET) imaging gantry 14, i.e., the illustrative imaging device is a CT / PET imaging device 10. More generally, the object of the service call can be any type of imaging device, such as a stand-alone PET or stand-alone CT imaging device, a magnetic resonance imaging (MRI) device, a gamma camera for single photon emission computed tomography (SPECT) imaging, an image-guided therapy (iGT) device, etc. More generally, the disclosed AR-based service guidance device and method can be applied to any type of system or equipment maintenance performed in the field, such as radiation therapy delivery equipment, research equipment such as magnets, cryogenic systems, factory robotic systems, processing furnaces, etc. The service call is recorded using a mobile camera 18 that is capable of recording video and preferably has a convenient form factor. In the illustrative example, the mobile camera 18 includes a stereo camera 20 mounted on glasses 21. The stereo camera 20 includes a left-eye camera 22 and a right-eye camera 24, and thus advantageously captures stereoscopic video to simulate binocular human vision. By mounting the stereo camera 20 on the glasses 21, the mobile camera 18 provides a "first person" view - if the glasses 21 are worn by a service person performing (or participating in performing) the service, the recorded service call video is advantageously viewed from the service person's viewpoint or vantage point. Instead of the illustrative mobile camera 18 using the glasses 21 as a structural support, another eyeglass form factor, such as goggles, or a camera mounted on a headband or helmet, etc., can provide similar advantages to record service video calls. By using the stereo camera 20, the recorded video is advantageously binocular in nature and can provide depth information for extracting three-dimensional (3D) information. However, in an alternative embodiment, a monocular camera is used - in this case, computer vision can generally extract 3D information based on different vantage points provided by the natural movement of the camera (due to the movement of the service person's head during maintenance). As another expected variation, a conventional optical camera can be extended by a range-finding camera to provide depth information.

[0032] A difficulty with the mobile camera 18 of the illustrative eyewear-mounted camera 20 is that it may provide limited support structure for mounting an electronic processor, that is, it may be difficult to integrate a microprocessor or microcontroller with sufficient processing power into the eyewear 21 to handle video generation and optional processing. In the illustrative embodiment, this problem is solved by having the mobile camera 18 communicate wirelessly with a mobile device 30, such as an illustrative cellular phone (sometimes referred to as a smartphone when equipped with an executable application or "app") or a tablet computer. The wireless connection 32 may be by way of non-limiting example, Bluetooth TM A wireless connection, a WiFi connection or other short-range or medium-range wireless connection. It is also contemplated that a wired connection, such as a USB cable, can physically connect the mobile camera 18 to the mobile device 30. The illustrative mobile device 30 includes typical components, such as an opaque display 34 disposed on the front face of the mobile device 30, and a rear-view camera 36 disposed on the opposite back face of the mobile device 30 (on the left). Figure 1 As is typical for such mobile devices, display 34 is preferably a touch-sensitive display, such as a capacitive touch-sensitive display or a surface acoustic wave (SAW) display, so that a user can interact through touch with mobile device 30 to run applications ("apps"), provide input, operate a "soft" keyboard, etc.

[0033] The recorded video 40 is transmitted from the glasses-based mobile camera 18 to the mobile device 30 via the wireless connection 32. The mobile device 30 also typically includes a microphone 38, which can be used to provide an audio component of the recorded video 40, or the microphone can be mounted on the glasses 21 (not shown); as another alternative, the recorded video 40 can be without recorded sound (only video). During the recording of the service call, the mobile device 30 can optionally perform other tasks, such as by running other apps. For example, the mobile device 30 in the form of a mobile phone can be used to discuss the service call by telephone with a remote expert 42 (e.g., a human expert with specialized knowledge about the specific service performed and / or the specific device 10 or parts of the device undergoing service).

[0034] As described below, the recorded video 40 provides the basis for authoring augmented vision (AR) content for use during subsequent service calls. Figure 1As shown, the recorded video 40 is transmitted to the AR content creation device 50 in the form of computer 52 readable instructions from a non-transitory storage medium (not shown—e.g., a hard drive, RAID or other magnetic storage medium; flash memory, solid-state drive or other electronic storage medium; an optical disk or other optical storage medium; various combinations thereof; etc.) to perform the AR content creation method 54. AR content creation can, for example, use computer vision (CV) processing (e.g., simultaneous localization and mapping (SLAM) processing) to align AR content (e.g., part number annotations extracted from a CAD drawing 56, a picture-in-picture (PIP) window that shows a CAD drawing or a portion thereof, or an animation generated from the CAD drawing 56, etc.) with the recorded video 40, thereby spatially registering the recorded video 40 with the computer-aided design (CAD) drawing 56.

[0035] It should be understood that Figure 1 Only one illustrative arrangement of the video 40 for capturing the service call is shown. As already mentioned, other types of glasses or headwear (e.g., goggles, headbands, helmets, etc.) can be used as a support structure for the mobile camera 18 to provide a first-person perspective of the repair process; in addition, as already mentioned, the stereo camera 20 can be replaced by another type of camera, such as a monocular camera, a combination of a camera plus a rangefinder camera, etc. In other embodiments, the recorded video of the service call may not be from the first-person perspective of the service personnel. For example, in an alternative embodiment, the mobile camera includes a mobile device 30, whose rear camera 36 is arranged on the back of the mobile device 30, used as a camera for recording the service call. In this case, the advantage will not be the first person but the service personnel (or assistant) can hold the mobile device 30 so as to appropriately guide the rear-view camera 36 to record the entire service process or only the part of the service process of interest (e.g., the difficult part).

[0036] To assist in authoring AR content, in some embodiments, a service person performing service captured by recorded video 40 of a service call may actively point to a location of interest (LOI) 60 using a pointer 62 or other unique pointing mechanism (which in some examples may simply be the service person's finger or hand). In other embodiments, the LOI is marked after the recorded video 40 is recorded, such as by a user operating one or more user input devices of computer 52 (e.g., illustrative keyboard 64 and / or mouse 66) to mark the LOI in a frame of the recorded video 40.

[0037] Reference now Figure 2, describes an illustrative embodiment of an AR content creation device 50 implemented by a computer 52 executing instructions read from a non-transitory storage medium (not shown). The AR creation process 54 receives as input the recorded video 40 and information from which the AR content is to be generated, such as a CAD drawing 56 (which may optionally include data such as part numbers, 3D renderings, animations, and / or the like of the components). In operation 70, various components of the imaging device 10 (see FIG. 1 ) are mapped using computer vision (CV) processing. Figure 1 ; more generally, individual components of a system or device undergoing service) are mapped to the recorded video 40. In a preferred embodiment, the mapping operation 70 is performed using a simultaneous localization and mapping (SLAM) process that employs Bayesian reasoning or another machine learning technique to optimally map the position of the service personnel (assuming first-person video; or, if the recorded video 40 is captured by the mobile device 30, the position of the mobile device) and reference points on the component or device 10, on the other hand, to the content of the observation sequence provided by successive frames of the recorded video 40. Optionally, the mapping 70 can be improved using a priori known information such as the spatial layout of the component reference points provided by the CAD drawing 56. The SLAM processing task can be implemented by the probabilistic formula P(c1(t), c2(t), …, c n (t),p(t)|f1,…,f t ) to mathematically formulate, where c1(t),c2(t),…,c n (t) is the position of a reference point of the component or device 10 at time t (where n reference points are assumed without loss of generality), p(t) is the position of the service personnel at time t, and f1,…,f t is a frame of the video 40 recorded up to time t. Various SLAM algorithms suitable for robot vision mapping, autonomous vehicle navigation techniques, etc. are known to implement the mapping operation 70. In operation 72, the LOI is received from the user via (one or more) user interface devices 64, 66 (e.g., by displaying a video frame from the recorded video 40 and providing an LOI for the user to click using a mouse 66) or automatically extracted from the recorded video 40 during the SLAM process 70. The latter method requires the computer 52 to perform SLAM or other CV processing to detect the user's actions captured by the recorded video 40 (e.g., pointing using a pointer 62, see Figure 1 ), which indicates that at least one LOI in the recorded video 40 is captured (e.g., Figure 1 The indicated LOI 60 then becomes one of the reference points ((c LOI (t)) will be mapped into the SLAM process 70.

[0038] In operation 74, the AR content is aligned with the recorded video 40 using the results of the SLAM process 70 and the LOI designation 72. For example, an annotation (e.g., a part number annotation, a CAD drawing or a portion of a CAD drawing displayed as a PIP window, or a CAD animation displayed as a PIP window) may be added at or immediately adjacent to the LOI, thereby "marking" the LOI with the part number of the CAD information. In a more advanced approach, a wireframe drawing of a key part or part combination or assembly may be extracted from the CAD drawing 56 and overlaid as AR content, again using the mapping output of the SLAM process 70 and the LOI designation 72 to align with the actual image of the (one or more) parts shown in the recorded video 40. The AR content thus created may optionally be color-coded (e.g., using different colors to distinguish different parts of a part assembly) or otherwise highlighted. AR content in the form of a verbal narrative may also be added. In this case, it is assumed that the AR content has a time sequence consistent with the time sequence of frames constituting the recorded video 40, and the narrative is added to synchronize in a timely manner with the performance of various service tasks in the video. In optional operation 76, the AR content may be time segmented by adding stopping points and / or segmenting the AR content into separate episodes. For example, if a repair involves removing a first component from the device 10 to reveal a second component that requires repair, the removal of the first component may be one segment, while the repair of the revealed second component may be a separate and distinct segment. Stopping points may be added to stop the temporal evolution of the AR content to allow for manual manipulation—for example, if a part requires oiling before installation, it may make sense to add a stopping point during the oiling process.

[0039] The resulting created AR content forms the maintenance process AR library component 80. In some embodiments, as will be described later herein, the AR library component 80 may include or have access to the stored recorded video 40 (optionally after segmentation and / or adding stop points similar to the AR content creation operation 76) as a preview video 41, such as Figure 2 The preview video 41 , if provided, is used to provide a preview of the service process including the superimposed AR content upon request of the service personnel.

[0040] As part of the overall AR maintenance process library, the AR library component 80 can optionally be linked in various ways with other AR library components in operation 82 to provide AR support for a reasonable service operation sequence. Figure 1The process of repairing an X-ray tube of a CT gantry 12. Repair requires opening the CT gantry 12 to access the X-ray tube, removing the X-ray tube, perhaps performing some setup or configuration on the new X-ray tube, installing the new X-ray tube, and then reassembling the CT gantry 12. In one approach, each step can be supported by a separate AR library component, which are then linked together as appropriate. Thus, a service person can initially load and view an AR content library component with the CT gantry opened - upon completion of which, the next library component (e.g., configuring a new X-ray tube) can be automatically linked and called, and so on. In a variant embodiment, the linking operation 82 can be performed automatically based on the video content, for example, during playback of the AR content, CV processing can be used to analyze the live video feed (referenced below). Figure 3 ) to detect the specific model, type, or other specific component being serviced, and can then call the correct AR library component for that specific model, type, or other specific component.

[0041] refer to Figure 3 , by reference Figure 2 The AR library component 80 generated by the AR content creation described is appropriately presented to the service personnel performing the subsequent service call (i.e., the service call performed after the subsequent service call, which is recorded to generate the recorded video 40, as shown in reference Figure 1 The AR content is presented using a suitable mobile device, such as an illustrative heads-up display (HUD) 90 in the form of glasses 21 on which a stereo camera 20 is mounted. Thus, the HUD 90 is similar to that already described with reference to Figure 1 The glasses-based mobile camera 18 described. However, the HUD 90 also includes a transparent display, which in the illustrative case includes a left transparent display 92 mounted on the left eye frame of the glasses 21 and a right transparent display 94 mounted on the left eye frame of the glasses 21. The HUD 90 thus constitutes a see-through AR display, wherein the user (e.g., a service person) sees the actual scene 94 directly by looking through the transparent display 92, and the AR content can be superimposed on the actual seen scene by displaying the AR content on the transparent displays 92, 94. The exemplary glasses-based HUD 90 is an example, and many other HUD designs are contemplated. In a variant embodiment, the HUD employs goggles having a single lens extending between the left eye and the right eye-in this embodiment, the transparent display can be a single transparent display that also extends between the left eye and the right eye.

[0042] A transparent AR display has significant advantages because the service person can directly see the actual scene (e.g., actually see the part being repaired), thereby having the maximum visual acuity provided by the person's eyesight. (If the service person's eyes are nearsighted or otherwise require prescription glasses or contact glasses, the service person will appropriately either wear prescription contact glasses in conjunction with the use of HUD 90, or optionally the glasses can form part of the transparent display 92, 94 modified to incorporate the ophthalmic prescription). However, in an alternative embodiment, it is envisioned to use a video AR display. In this alternative embodiment, the transparent displays 92, 94 are replaced by an opaque display that displays a video feed of the actual scene captured by the stereo camera 20. The displayed video source will then serve as the basis for the displayed AR content. A disadvantage of this approach using a video AR display is that the video display will typically have a coarser resolution and / or may have other optical performance degradations compared to the direct viewing of the actual scene provided by the illustrative perspective HUD 90.

[0043] In order to provide AR content for supporting the service process, the live video feed 96 is transmitted via Bluetooth TM or another short-range to medium-range wireless communication protocol (or, alternatively, see wired USB or other wired connection) to a mobile device 30 (e.g., a cell phone or tablet) and the mobile device 30 in turn relays the live video feed 96 to a server computer 100, such as via a cellular communication protocol (e.g., 4G) or via a WiFi link to an Internet Service Provider (ISP) / or hospital electronic network, etc. The server computer 100 executes instructions stored on a non-transitory computer storage medium (not shown—e.g., a hard drive, RAID or other magnetic storage medium; flash memory, solid-state drive or other electronic storage medium; optical disk or other optical storage medium; various combinations thereof; etc.) to perform alignment 102 to map the AR content of the AR library component 80 to the live video feed 96 and position the agent (in this embodiment, the HUD 90). The mapping 70 described previously (see Figure 2 ) to perform the alignment 102. The AR content mapped to the live video feed 96 suitably includes AR content defined in the AR library component 80, such as part number annotations, annotated picture-in-picture (PIP) windows in which a CAD drawing or portion thereof and / or a CAD animation is displayed, accompanying verbal narration, etc. For example, the aligned AR content may include markers aligned with the LOI identified in the live video feed 96 by the SLAM processing. As another example, the aligned AR content may include annotations (e.g., part number annotations, CAD drawings or portions of CAD drawings, or CAD animations) aligned with the LOI identified in the live video 96 feed by the CV processing 102.

[0044] The obtained AR content 104 is sent to the mobile device 30 via 4G, WiFi / ISP / Internet or other communication paths, and then transmitted from the mobile device 30 via Bluetooth TM or other short-range to medium-range wireless communications to the HUD 90. At the HUD 90, the AR content 104 is displayed as opaque or semi-transparent text and / or images on the transparent displays 92, 94, which the attendant visually perceives as being superimposed on the view of the actual scene seen by the attendant when looking through the transparent displays 92, 94. (In an alternative embodiment employing a video AR display, a live video feed 96 is displayed as an underlying layer with the AR content 104 superimposed thereon).

[0045] The HUD 90 optionally includes a user interface for control. In one illustrative embodiment, this is implemented through an AR user interface (UI) application ("AR UIapp") 106 that can run on the mobile device 30. Through the AR UIapp 106, the service personnel can perform various control operations, such as: turning the AR content on or off (for example, if the service personnel is sure that he or she does not need the assistance of the AR content, and / or if the AR content blocks the view of the part being serviced, the latter is useful); selecting to execute a specific AR content segment; pausing the AR content (again, it is usually presented as a time series synchronized with the live video feed 96); adjusting the brightness, transparency and / or other display characteristics of the AR content; turning on or off audio narration; etc. In some embodiments, the AR UIapp 106 can provide the service personnel with one or more mechanisms for interacting with the AR content, for example, if the HUD 90 includes gaze tracking technology, a suitable control mechanism can be to point the gaze at an AR content element (e.g., a PIP window, a part number annotation, etc.), and then speak a command, which is detected by the microphone 38 and processed by speech recognition to interpret the verbal command. Instead of via verbal commands, user commands may instead be input via soft controls (buttons, switches, soft keyboards, etc.) displayed on the touch-sensitive display 34 of the mobile device 30. These are merely illustrative user interface capabilities and control input mechanisms; more generally, substantially any type of user interface capabilities and / or control input mechanisms suitable for controlling the presentation of AR content may be employed.

[0046] Continue to refer Figure 3 In some embodiments, a preview of the service may be provided. As already indicated, a preview video 41 may be provided for this purpose, which comprises the recorded video 40, which is used as a preview of the video after some optional processing (e.g., inserting stop points and / or splitting into segments, see Figure 2 Operation 76) to create AR content 80 (as previously referred to in Figure 21). To adopt the preview mode, the user requests a preview in operation 110, for example, using the AR UI application 106. In response to operation 112, the server computer 100 is instructed to switch to preview mode (e.g., a command sent along the same 4G or other communication path used to transmit the live video feed 96 to the server 100; optionally, when switching to preview mode, the communication of the live video feed to the server is turned off). Command 112 causes the server to perform a modification 114 on the AR alignment process 102. The modification includes replacing the preview video 41 with the live video feed 96 so that the AR content is aligned with the preview video 41, and adding the preview video 41 as the basis for the AR content. This combination of the AR content aligned with the preview video and the underlying preview video then becomes the AR content 104, which is sent back to the mobile device 30. At the mobile device 30, in operation 116, the preview video with the superimposed AR content is displayed on the display 34 of the mobile device (while turning off the AR content fed to the HUD 90). The service personnel can then view the preview on the display 34 of the mobile device 30 without being obstructed by the extraneous AR content displayed on the transparent displays 92, 94. It is generally preferred to display the preview on the display 34 of the mobile device 30 because the content is no longer spatially aligned with the live video feed from the stereo camera 20; however, in another embodiment, it is contemplated to display the preview on the transparent displays 92, 94.

[0047] Figure 3 The illustrative embodiment employs a server computer 100 to perform computationally complex SLAM processing 102, which is beneficial if the mobile device 30 and HUD 90 have limited processing power. Similarly, using a mobile device 30 running an AR UI app 106 to provide a user interface utilizes a touch-sensitive display 34 and optionally other features of the mobile device 30 (e.g., a microphone 38) to provide a user-friendly interface. In alternative embodiments, these operations / processing may be distributed in other ways. In an intended embodiment, all processing and user interfaces are integrated with the HUD, and in this intended embodiment, the HUD includes flash memory, etc. to store the AR library component 80 (or download this content from a cloud server) to provide a service guide device based on a single unit AR. As another variant, the AR library component 80 can be used to perform operation 102 at the mobile device 30 when downloaded as a whole to the mobile device 30 or from the cloud server AR library component 80. These are also merely illustrative variations.

[0048] In addition, the illustrative HUD 90 can be replaced by another mobile device that presents AR content. For example, a mobile device 30, such as a mobile phone or tablet, can be used to present AR content (in this variant embodiment, HUD 90 is omitted). In order to use the mobile device 30 to present AR content that supports a service call, the service personnel points the rear camera 36 to the component being serviced, and the live video feed 96 is recorded by the rear camera 36. The server computer processes the live video feed via operation 102 as already described to generate AR content, and sends it back to the mobile device 30, which displays the live video feed 96, where the AR content is superimposed (or alternatively, the superposition of the AR content on the live video feed 96 is performed on the server computer 100, and then the server computer 100 combines the AR content with the underlying live video feed and sends it back). Advantageously, the maintenance personnel can view the display 34 of the mobile device 30 with the rear camera 36 pointing to the component to be repaired, so that the mobile device 30 is used as a video AR display.

[0049] By comparing reference Figure 1 and Figure 3 It will be appreciated that in some embodiments, the HUD 90 used in AR content support may also be used as a recording Figure 1 The mobile camera 18 of the recorded video 40, that is, the HUD 90 can replace Figure 1 This interoperability has valuable advantages: the live video feed 96 captured by the HUD 90 during a service call can optionally be used as Figure 1 The recorded video 40 is used as the basis Figure 2 The basis for performing further AR authoring to generate updated or additional AR content. For example, in the previous example of replacing the X-ray tube of the CT gantry 12, assume that the service personnel, while providing AR content using the HUD 90, has a new model X-ray tube that is different from the model installed in the service call that was recorded for creating the AR content. In this case, the service personnel can record a live video feed 96 while the new model X-ray tube is installed. As previously described with reference to Figure 1 As described, this may involve obtaining telephone advice from a remote expert 42. The recorded live video feed 96 of the installation of the new model X-ray tube then becomes available for use in executing Figure 2 The AR content is created based on the recorded video 40. Thereafter, when a new model X-ray tube is installed in a subsequent service call, the newly created AR content suitable for the new model X-ray tube can be used to provide AR-based guidance. Figure 2When creating updated AR content for an X-ray tube, a link may be added in operation 82 that, for a given subsequent service call, loads AR content for supporting installation of an older version or AR content for supporting installation of a newer model of X-ray tube, depending on which model of X-ray tube is being installed. This link may be manual (e.g., a user enters the X-ray tube model to load the correct AR support content via the AR UI 106) or automatically generated (e.g., when the service person first looks at the X-ray tube, the model or model distinguishing features are detected by the CV process and used to link the correct AR content for the X-ray tube being installed).

[0050] In an illustrative embodiment, Figure 2 The AR content creation method 54 processes a recorded video 40 of a service call performed to repair or maintain a medical imaging device in order to generate AR content 80 for subsequent use in the delivery of AR-based service instructions, as described with reference to Figure 3 However, Figure 2 The AR content authoring method 54 may additionally or alternatively be applied to process the recorded video 40 for other purposes, such as creating annotated AR log entries to reflect maintenance or repair actions performed on the imaging device 10. The authored AR content may be used as a record of maintenance inspections to verify maintenance and provide a tool for correlating planned maintenance needs.

[0051] In maintenance or repair record applications, the AR content log entry is typically included with the recorded video 40 with the annotations of the AR content created in operations 72, 74, and the AR content log entry (including the recorded video 40 and the AR annotations) is appropriately stored in the electronic service log of the imaging device 10. The LOI designation operation 72 can be performed by the service personnel immediately after completing a service call to identify the location of his or her service, and the operation 74 can require the service personnel to add AR content, such as part number annotations, verbal and / or textual (optional speech-to-text) and / or textual descriptions of the service performed and / or any abnormal aspects of the service, inspection observations made by the service personnel to supplement the recorded video of the inspection component, and / or the like. In these embodiments, the user operations 72, 74 can be performed, for example, on a tablet computer, a cellular phone, or other portable electronic device carried by the service personnel. Some annotated AR content can be automatically generated, for example, adding timestamp information, identifying the service personnel (or the AR content can be added manually), automatic identification of CV-based components or other objects appearing in the recorded video 40, and the like.

[0052] The present invention has been described with reference to the preferred embodiments. By reading and understanding the aforementioned detailed description, those skilled in the art may conceive of various modifications and variations. The purpose is that the present invention is understood to include all such modifications and variations as long as they fall within the scope of the claims or their equivalents.

Claims

1. A service guidance system based on augmented reality (AR), comprising: A mobile device including a display and a camera; an electronic processor programmed to perform computer vision (CV) processing (102) based on a recorded video of a service call to generate augmented reality content (80), wherein the augmented reality content comprises: a part number annotation taken from a computer-aided design (CAD) drawing of a component, a picture-in-picture (PIP) window showing the computer-aided design drawing or a portion of the computer-aided design drawing, or a picture-in-picture (PIP) window showing a computer-aided design animation of a component; wherein the electronic processor is further programmed to perform computer vision (CV) processing (102) including simultaneous localization and mapping (SLAM) processing to add the augmented reality content at or in close proximity to at least one location of interest (LOI) identified in a live video feed (96) captured by a camera of the mobile device, and to display the augmented reality content added to the live video feed on the display of the mobile device; and The electronic processor is further programmed to perform a service preview by performing the following operations: performing the computer vision CV processing (102) to add the augmented reality content (80) to a preview video (41) that replaces the live video feed (96) generated by the camera of the mobile device at or in close proximity to the at least one location of interest LOI and simultaneously displaying both the augmented reality content added to the preview video and the preview video displayed as an underlying layer of the augmented reality content on the display (34) of the mobile device, wherein the preview video includes the recorded video used as a basis for creating the augmented reality content, and the preview video provides a preview of the service process including the superimposed augmented reality content.

2. The service guidance system based on augmented reality according to claim 1, wherein: The added augmented reality content includes a marker added at or proximate to a location of interest (LOI) identified in the live video feed (96) by the computer vision processing (102).

3. The service guidance system based on augmented reality according to claim 1 or 2, wherein: The added augmented reality content includes annotations added at or near a location of interest (LOI) identified in the live video (96) feed by the computer vision processing (102).

4. The service guidance system based on augmented reality according to claim 1 or 2, wherein: The mobile device comprises a heads-up display (HUD) (90), in which the display is a transparent display, and the camera (20) is arranged to capture the live video feed (96), the live video feed corresponding to a viewpoint of a user of the heads-up display looking through the transparent display; as well as A processor is programmed to display the augmented reality content added to the live video feed on the transparent display of the heads-up display without displaying the live video feed on the transparent display.

5. The service guidance system based on augmented reality according to claim 1 or 2, wherein: The mobile device (30) comprises a cellular phone, a mobile phone, or a tablet computer, wherein the display is an opaque display (34) disposed on a front surface of the mobile phone or tablet computer, and the camera is a rear camera (36) disposed on an opposite rear surface of the mobile phone or tablet computer; and The processor of the mobile device is programmed to display the augmented reality content added to the live video feed (96) captured by the rear camera (36) and the live video feed as an underlying layer of the augmented reality content on the opaque display of the phone or tablet; Therein, the mobile device is used to communicate with a remote expert.

6. The service guidance system based on augmented reality according to claim 1 or 2, wherein: The electronic processor comprises: a server computer (100) programmed to perform the computer vision processing (102) to add the augmented reality content (80) to the live video feed (96) generated by the camera of the mobile device; and A processor of the mobile device is programmed to transmit the live video feed generated by the camera of the mobile device to the server computer, receive the augmented reality content added to the live video feed from the server computer (104), and display the augmented reality content added to the live video feed on the display of the mobile device.

7. The service guidance system based on augmented reality according to claim 1 or 2, further comprising: A computer (52) programmed to perform computer vision processing (54) to generate the augmented reality content (80) by adding (70) the augmented reality content to a recorded video (40) of a service call, including adding (72) at least a portion of the augmented reality content at or proximate to at least one location of interest (LOI) identified in the recorded video of the service call by user input.

8. A service guidance method based on augmented reality (AR), comprising: Using an electronic processor to perform computer vision (CV) processing (54) based on the recorded video of the service call to generate augmented reality content (80), wherein the augmented reality content includes: a part number annotation taken from a computer-aided design (CAD) drawing of the component, a picture-in-picture (PIP) window showing the computer-aided design drawing or a portion of the computer-aided design drawing, or a picture-in-picture (PIP) window showing a computer-aided design animation of the component; Using the electronic processor and a mobile device including a display and a camera, presenting the authored augmented reality content, including using the electronic processor to perform the computer vision (CV) processing (102) including simultaneous localization and mapping (SLAM) processing to add the authored augmented reality content (80) at or in close proximity to at least one location of interest (LOI) identified in a live video feed (96) generated by the camera of the mobile device, and displaying the authored augmented reality content added to the live video feed on the display of the mobile device; and Using the electronic processor and a mobile device including a display and a camera, the authored augmented reality content is presented, including performing a service preview by performing the following operations: performing the computer vision CV processing (102) to add the augmented reality content (80) at or adjacent to the at least one location of interest LOI to a preview video (41) that replaces the live video feed (96) generated by the camera of the mobile device and simultaneously displaying both the augmented reality content added to the preview video and the preview video displayed as an underlying layer of the augmented reality content on the display (34) of the mobile device, wherein the preview video includes the recorded video used as a basis for authoring the augmented reality content, and the preview video provides a preview of the service process including the superimposed augmented reality content.

9. The service guidance method based on augmented reality according to claim 8, wherein: The mobile device comprises a heads-up display (HUD) (90), wherein the display is a transparent display, and the camera (20) is arranged to capture the live video feed (96) corresponding to a viewpoint of a user of the heads-up display looking through the transparent display, and the presentation of the authored augmented reality content comprises: The augmented reality content (80) added to the live video feed is displayed on the transparent display of the heads-up display without displaying the live video feed on the transparent display.

10. The service guidance method based on augmented reality according to claim 8, wherein: The mobile device (30) comprises a cellular phone, a mobile phone, or a tablet computer, wherein the display is an opaque display (34) disposed on a front surface of the mobile phone or tablet computer, and the camera is a rear camera (36) disposed on an opposite rear surface of the mobile phone or tablet computer, and the presentation of the authored augmented reality content comprises: displaying on the opaque display of the phone or tablet both the augmented reality content (80) added to the live video feed (96) captured by the rear camera (36) and the live video feed as an underlying layer of the augmented reality content; Therein, the mobile device is used to communicate with a remote expert.

11. The service guidance method based on augmented reality according to any one of claims 8 to 10, wherein: The electronic processor includes: a server computer (100) used to perform the computer vision processing (102) to add the authored augmented reality content (80) to the live video feed (96) generated by the camera; and a processor of the mobile device used to display the authored augmented reality content added to the live video feed on the display of the mobile device, the augmented reality-based service boot method further comprising: transmitting the live video feed (96) from the mobile device to the server computer; and The authored augmented reality content (104) added to the live video feed is transmitted from the server computer to the mobile device.

Images