Surgical procedure guidance and training device, method, and system

Abstract

To provide a surgical procedure guidance and training device, method, and system.SOLUTION: In one embodiment, a surgical procedure guidance and training device, method, and system allow a surgical trainee to receive special clinical training during an actual operation, or practice a surgical technique on an anatomical model, in a realistic manner using an augmented realistic headset making audio communication and visual communication with a remotely located trainer having a similar AR headset. In a unique function of the system, supplying a live image or still image of a surgical portion to the AR headsets of the trainee and trainer is included. Imaging is supplied through a device medium hub from a medical device or other device, for example, ultrasound. Since the image is displayed on a field of view of a surgeon, the surgeon can see an own hand, a patient or model, and the imaging simultaneously.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 271,153, filed October 23, 2021, and U.S. Provisional Patent Application No. 63,341,234, filed May 12, 2022, both of which are incorporated by reference in their entireties.

[0002] The present invention relates to devices, systems, and methods for providing procedural instruction to a practitioner and / or training between at least two people, referred to herein as a "trainee" and a "trainer." While the present invention is broadly applicable to a wide variety of technology fields, examples of which will be further described below, in one embodiment of the present invention, a device, system, and method is provided for remote surgical training, proctoring, and / or clinical instruction utilizing a human computer interface ("HCI"), such as, by way of non-limiting example, an augmented reality ("AR") headset with surgical site imaging and telecommunications software. The trainee, in the training embodiment of the present invention, may be, by way of non-limiting example, a medical student, a surgical resident, or an experienced surgeon learning a new procedure who is not necessarily skilled or simply wants to refresh their skills, and the "trainer" may be, by way of non-limiting example, a surgeon, professor, or medical salesperson skilled in the procedure being performed. The trainees and trainers may or may not be in different geographic locations, and optionally multiple trainees and / or trainers may work simultaneously within the system. In a clinical teaching embodiment of the present invention, a single practitioner may use the devices, systems, and methods of the present invention to teach a procedure without having to take their eyes off the patient. [Background technology]

[0003] Surgical training is currently performed in a variety of ways, including on artificial body and organ models, and directly on both living and dead humans and animals. Despite the need for expert instruction of surgical trainees, it is clear that there are deficiencies in currently available surgical training methods, and there is a need for more efficient and effective surgical training methods.

[0004] It is known that there are numerous data points that a medical practitioner must continually monitor during a surgical or other medical procedure. In surgery, surgical team members such as anesthesiologists and nurses may periodically verbally present the patient's vital signs and status to the surgeon. The surgeon may receive such verbal information in such cases without having to look away from the surgical site. Other patient data, such as imaging of the patient's organs that has been performed, or imaging performed on the patient in the operating room during surgery, that may be available for the surgeon's reference pre-operatively, prior to the procedure, or broadcast to the surgeon during the procedure during minimally invasive surgery, must be viewed by the surgeon personally during the procedure. Since the imaging is typically displayed on a computer monitor or similar, the surgeon must look away from the surgical site when viewing the monitor (this is also true when one practitioner is performing a procedure on one patient). This is not ideal as it can lead to unnecessarily long surgical times and potential surgical errors due to the surgeon having to look away from the patient and hands to view the images on the monitor and thus being intermittently distracted from the surgical site. Summary of the Invention [Means for solving the problem]

[0005] In an exemplary embodiment of the present invention, a surgical training apparatus, method and system is provided that utilizes three main components in the form of a human-computer interface: in a preferred embodiment, an augmented reality (AR) headset with a see-through optical display and imaging array, a hub for the imaging device electronics, and software that enables electrical communication and control of the AR display.

[0006] Although one embodiment of the invention is used by a single practitioner, e.g., in a clinical setting, on an artificial model or on a real patient, the invention will be described herein with reference to a training embodiment of the invention involving a trainer and one or more trainees.

[0007] The trainer and one or more trainees each have their own respective AR headset, imaging device hub, and software. The trainer and each trainee also have their own imaging device and surgical site, which are all substantially the same such that they are all performing the same surgical procedure using the same type of imaging device.

[0008] The trainer and each trainee can view their own surgical site through their own AR headsets, each of which includes a see-through optical panel and has one or more modifiable image viewing areas on or adjacent to the see-through optical panel.

[0009] The trainer can control the image display of each trainee's AR headset. The trainer can choose to have the trainee see what the trainee sees in the trainer's AR headset during the procedure. The trainer can also choose to display what the trainer sees at the trainer's surgical site (the "trainer's view") in the trainee's AR headset, allowing the trainee to see how the trainer is demonstrating the procedure. The trainer can also choose to send other video and / or textual material to one or more of the trainee's AR headsets. The trainer can, in certain circumstances, choose to display both the trainer's and the trainee's images side-by-side in the field of view (FOV), allowing both to see each other's work being done simultaneously.

[0010] The AR headset may include a speaker and a microphone to allow two-way communication between the trainee and the trainer. An electronically controlled pointing device may also be included, allowing the trainer to selectively highlight the trainer's hands, the surgical site, and / or any of a number of devices that require instruction to the trainee. The trainee may also be included with a pointing device, allowing the trainee to highlight an image of an area of ​​interest for discussion with the trainer. The pointing device may be attached to the headset and may highlight external objects or locations, or may be electronically superimposed by either the trainer or trainee, by a controller, onto a live image that is typically being generated by a device that displays the image on a monitor.

[0011] Live or static imaging of the surgical site can be displayed on the trainee's and trainer's AR headsets, with imaging coming from a medical device or other device (e.g., ultrasound) through a device hub. Images are displayed in the trainer's and / or trainee's field of view so that the trainer and / or trainee can simultaneously see their own hands, the patient (or model), and the imaging. Control typically allows the imaging on each AR headset to be switched on command by the trainer, so that the trainer can, for example, deliver an image of the trainer's surgical site (e.g., on the trainer's own model) to the trainee's AR headset to demonstrate the correct procedure. Once the trainee has viewed the trainer's imaging of the correct procedure, the imaging source can be switched back to the trainee's surgical site, allowing the trainee to continue with the procedure.

[0012] The procedure can be recorded by teaching or live clinical assistance during surgery or on a model, and the recordings can be separated by machine learning models and classes, allowing for semi-supervised and / or unsupervised learning. After the machine learning models are created, automatic guidance and / or warnings can be given during surgery. Such recordings can additionally be used for certification or quality assurance, and can be recorded using line of sight (LOS) technology, described below, and the position of the image (e.g., raw ultrasound superimposed on the hand) generated in software and controlled by a trainer or other user. It should be noted that the term "superimposed" as used herein is not meant to mean a superimposed arrangement of two or more viewing fields, but rather a side-by-side arrangement of two or more viewing fields (e.g., in any direction, such as vertically, horizontally, diagonally, etc.).

[0013] The device connectivity hub allows for plug-and-play connectivity of one or more medical imaging sources or other non-imaging devices such as EKGs (including multiple at the same time), cameras, or other peripherals used during surgery or other complex procedures, either wired or via wireless connectivity using any number of technologies, including Quick Response or "QR" codes to initiate a connection.

[0014] The trainee's live (real-time) interactions with their connected devices (imaging array data) can be broadcast via the trainee's respective AR headset to a remotely located trainer's AR headset, which sees the trainee's actions through a reflective or refractive portion of the trainer's AR headset's see-through optical display, a traditional computer display, or a head-worn monitor. These additional displays can be viewed at the trainer's, the trainee's, or at a remote site.

[0015] The software (sometimes referred to herein as the "remote training platform (RTP)" or "RTP software") allows for the simultaneous display of the aforementioned images, videos, and / or other feeds that can be selectively positioned by controls exercised at or through the RTP software or through the controller so as not to obstruct the view of the surgical field, thereby maximizing trainer and trainee interaction with such devices. All functionality described herein as being included in the RTP software can be provided through video communication or contained within the controller.

[0016] The RTP software also enables the trainer to selectively send and / or play key video, imaging, and / or text / annotation content of the surgical scenario to one or more trainees' AR displays.

[0017] Images or other selected content can be presented on the AR headset, thus presenting a lifelike view. For example, currently available telecommunications software such as ZOOM allows video images to be displayed as horizontal or vertical tiles. According to the present invention, the trainer has the ability to position the various images, with controls provided by the software, so that the device image is above and out of the way of the trainer's view of the hand. This allows the surgeon to view (internal) images that would otherwise be hidden by the exposed portion of the surgical site, while also being able to see the surgeon's own hand performing the procedure without ever having to look away from the surgical site. In this last case scenario, the live image is visually positioned (overlaid) to be displayed above the hand, so that both clinicians have a see-through view (ultrasound image of the model or patient that generates this internal image) and actually see their own hand while performing the procedure. This spatial arrangement gives both the trainer and the trainee the impression of performing an open surgery.

[0018] The imaging array of the AR headset, in one embodiment of the invention, may be operable to reconstruct a 360 degree view around the trainer's or trainee's field of view ("FOV"), allowing the trainer or trainee to virtually "walk" around the trainer's or trainer's location, respectively, thereby observing in real time what others are working and what they are seeing. This functionality virtually places the trainer or trainee in a room with the others.

[0019] Additional objects, advantages, and novel features of the present invention will be set forth in part in the description which follows, and in part will become apparent to those who practice the invention upon examination of the accompanying drawings.

[0020] The above mentioned features and advantages of the present invention, as well as other features and advantages, and the manner in which the same are accomplished, will become apparent and better understood by referring to the following description of the invention taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a schematic diagram of an embodiment of the present invention. [Diagram 2] FIG. 1 is a simplified diagram showing a top view of a practitioner, trainer, or intermediary equipped with an AR headset with laser illumination. [Diagram 3] 1 is a schematic diagram of a practitioner, trainer, or intern performing a surgical procedure utilizing the devices, systems, and methods of one embodiment of the present invention; [Figure 4] FIG. 1 illustrates various possible layouts of the viewing area of ​​an AR headset. [Diagram 5] FIG. 1 illustrates various possible layouts of the viewing area of ​​an AR headset. [Figure 6] FIG. 1 illustrates various possible layouts of the viewing area of ​​an AR headset. [Figure 7] FIG. 1 illustrates various possible layouts of the viewing area of ​​an AR headset. [Figure 8] This is a simplified diagram showing one trainer training multiple trainees simultaneously. [Figure 9] A simplified diagram showing a trainer and trainee, each equipped with an AR headset, performing a surgical procedure using the trainee view mode. [Figure 10] A simplified diagram showing a trainer and a trainee, each equipped with an AR headset, performing a surgical procedure using the trainer view mode. [Figure 11] FIG. 8 is a simplified diagram illustrating an example of the AR headset of FIG. [Figure 12] A simplified diagram showing the FOV of an AR headset as seen through the AR headset and a laser spot pointing to an AOI within the FOV. [Figure 13] FIG. 1 is a simplified diagram showing bracketed AOIs projected in the Line of Sight Field Of View (LOSFOV). [Figure 14]FIG. 1 is a simplified diagram showing an AOI centered within the FOV and also showing that the Line of Sight instruction point (LOSIP) is centered within the FOV, such that a person wearing a head worn display has the AOI perfectly centered within the FOV. [Figure 15] A simplified diagram showing that the AOI is off-center in the FOV, in the lower left, and that the LOSIP is in the lower left quadrant of the FOV, indicating that the person wearing the head-worn display is not centrally located in the FOV. [Figure 16] A simplified diagram showing how mechanical tracking of a barcode can be used to center an AOI in the FOV. [Figure 17] A simplified diagram of the entire field of view, showing a subregion representing a zoomed-in view of the AOI being presented to the headset wearer. [Figure 18] A simplified diagram showing the mechanical shade mechanism retracted (top half of the diagram), meaning the wearer is viewing in AR mode, while the bottom half of the diagram shows the shade deployed, meaning the wearer is utilizing virtual reality (VR) mode. [Figure 19] 1 is a simplified diagram showing the vergence angle of an AR waveguide and the distance to an angle of interest. [Figure 20] FIG. 2 is a simplified diagram showing convergence points where distance to the FOV is sensed by the distance sensor shown in the figure. [Figure 21] FIG. 1 is a simplified diagram showing a belt-worn embedded controller that can control a laser illuminator and, in VR mode, can guide a pointer to an area of ​​interest in the XY directions by turning a knob CW or CCW, and also to stay within and out of the Z dimension by pressing and turning the knob CW and CCW, respectively. [Figure 22]FIG. 1 is a simplified diagram showing a pointer that can be displayed on images collected from a digital camera attached to an endoscope, allowing the trainer to highlight specific areas on the video image and can be controlled by software or from a dedicated location on the controller. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] 1, in a preferred embodiment of the present invention, a surgical training apparatus, system, and method is provided, generally designated 10, that utilizes three major components in the form of an augmented reality (AR) headset 12 with a see-through optical display and imaging array, a device electronics hub 14, and telecommunications software 16. In an embodiment involving a single practitioner at a clinical site, all that is required is a single AR headset with an associated device connectivity hub, and a computer running RTP software that connects to the AR headset and device connectivity hub. In a training embodiment, a trainer with one or more trainees are each provided with an AR headset, a device connectivity hub, and a computer running RTP software, as described more fully below.

[0023] The AR headset 12 allows one or more trainees T2, T3, ~TN, each wearing an AR headset, to see what the trainer T1 sees through the trainer T1's imaging array and also allows each trainee T2, T3, ~TN to view the training material and / or the trainer T1's perspective.

[0024] Each AR headset 12 (one for each of T1, T2, T3, ∼TN) may include a speaker 12a and a microphone 12b to allow two-way communication between the trainer T1 and each trainee. Each trainer / trainee AR headset is identified in the drawings with a notation such as 12-T1, 12-T2, 12-T3 to indicate the AR headset of the trainer T1 and each trainee T2, T3, ∼TN, respectively.

[0025] Each AR headset 12 is coupled to a respective device connection hub or "DCH" 14, seen in Figure 2. Each DCH 14 allows for plug-and-play connectivity for one or more medical or other devices (which may include two or more devices operating simultaneously), cameras, imaging devices, or other peripheral devices used during surgery or other complex procedures. The device, in the particular embodiment shown and described herein, is an ultrasound wand 16, seen in Figure 2, connected to a plug receptacle 14a of the DCH 14 that receives a device plug 16a.

[0026] The DCH 14 may preferably be an embedded device running local software, especially in a healthcare facility, where it may be behind a firewall due to standard computer security protocols in the healthcare facility. The DCH software may connect to Cloud C to receive periodic downloads of the DCH software, which is updated through a CI / CD (Continuous Integration / Continuous Development) server hosted in Cloud C, as seen in Figure 2.

[0027] The live (real-time) interactions (imaging array data) of the trainees (one or more of T2, T3, ∼TN) with connected devices such as device 16 can be broadcast via the trainees' respective AR headsets 12-T2 to the AR headset 12-T1 of the remotely located trainer T1, who views the actions of the trainee T2 through an optical display or a conventional computer display of the trainer's AR headset 12-T1.

[0028] The RTP software allows for the simultaneous display of the aforementioned images and / or video feeds, which may be selectively positioned within the AR headset to unobstruct the view of the operative field and thereby maximize trainer and trainee interaction with the device.

[0029] The RTP software also enables the trainer to selectively send and / or play back key video, imaging, and / or text / annotation content of the surgical scenario to the trainee's AR display.

[0030] The imaging array of the AR headset, in one embodiment of the invention, may be operable to reconstruct a 360 degree view around the trainer's or trainee's field of view, allowing the trainer or trainee to virtually "walk" around the trainer's or trainer's location, respectively, thereby observing in real time what others are working and what they are seeing. This functionality virtually places the trainer or trainee in the room with the others.

[0031] The AR headset 12 may include any one or more of the following features or their equivalents, but preferably includes all of them: Projection display waveguide with 1K, 2K, 4K, 8K or higher resolution monocular or binocular see-through optics WiFi and Bluetooth connectivity (or other connectivity methods such as ultrasound, thermal imaging, sonic waves, or other connectivity methods within the electromagnetic spectrum) Digital Camera Single or multi-core embedded CPU or GPU Right or left eye monocular or pair of binoculars Haptic feedback Voice Control Embedded OS Noise-canceling microphone On-board video recording medium with marking and playback functions

[0032] The AR headset 12 can be connected, wired or wirelessly, to a computing device 17, which can be in the form of a computer, smart phone, tablet, or other computing device, running the RTP software 16, which is, for example, a downloaded software application ("app"). The RTP software 16 can also be provided remotely in the "cloud" C and offered to the user as Software as a Service (SaaS). Any other type of computer can be used, such as a tablet, laptop, desktop, virtual desktop, smart phone, etc., to install the RTP software or access it as SaaS. The RTP software 16 can be programmed to present the user with a login screen on the device 17, a separate monitor (not shown), and / or the AR headset 12, and the user can have a password-protected data file that stores data of the user's live clinical or surgical training session for later retrieval and / or playback. The RTP software 16 can connect to other servers and / or networks, thereby connecting the user's RTP software files to the user's personal (e.g., student) data files hosted, for example, on the user's medical school or medical device company's server, such that the time the user spends on the simulated surgical training can be recorded for the user's course credit or other purposes.

[0033] Exemplary embodiments of the overall architecture comprising the elements of device connectivity hub 14 include, but are not limited to, the following:

[0034] Use case 1 - The AR headset 12 is a peripheral to a laptop / desktop computer, the see-through optics are an augmented display, and the camera in the headset is like a webcam.

[0035] 1A.1 (Headset): Internet AR Cloud with two-way communication wired or wireless link, two-way communication with laptop / desktop computer, and two-way communication with AR headset.

[0036] 1A.2 (Hub): Internet cloud with two-way communication wired or wireless link, two-way communication with laptop / desktop computer, and two-way communication with device connection hub.

[0037] Use Case 2 - Without a laptop / desktop computer, the AR headset is connected directly to the Hub.

[0038] 2A: Internet cloud with two-way communication with a wired or wireless link, with two-way communication with a wired or wireless link, with two-way communication with a hub, with an embedded computer, with two-way communication with an AR headset.

[0039] 2B: Internet cloud with two-way communication with a wired or wireless link, with an embedded computer with two-way communication with a wired or wireless link, with two-way communication with a hub, and with two-way communication with an AR headset.

[0040] 2C.1: An Internet cloud with two-way communication with a hub having an embedded computer and with two-way communication with a wired or wireless link.

[0041] 2C.2: Internet cloud with two-way communication with a wired or wireless link and with two-way communication with an AR headset.

[0042] Use Case 3 - A standalone clinical medical device with a direct connection in degraded mode to an internal facility server.

[0043] 3A: A server inside the facility (assuming a firewalled internet connection is available) with two-way communication with a wired or wireless link, with an embedded computer and two-way communication with a wired or wireless link, with two-way communication with a hub, and with two-way communication with an AR headset.

[0044] 3B: A server inside the facility (assuming a firewalled internet connection is available) with an embedded computer and two-way communication with a wired or wireless link, two-way communication with a wired or wireless link hub, and two-way communication with an AR headset.

[0045] 3C: A server inside the facility (assuming a firewall-protected internet connection is available) with two-way communication with a wired or wireless link, with an embedded computer and two-way communication with a wired or wireless link, with two-way communication with a hub, with an embedded computer, with two-way communication with an AR headset.

[0046] 3D.1: A server inside the facility (assuming a firewall-protected Internet connection is available) with two-way communication with the AR headset, equipped with an embedded computer and with two-way communication with a wired or wireless link.

[0047] 3D.2: A server inside the facility (assuming a firewall-protected Internet connection is available) with two-way communication with a hub, with an embedded computer, with two-way communication with a wired or wireless link.

[0048] Use Case 4 - Standalone clinical medical device with direct connection in degraded mode to a federated server (a server where anonymous images and / or data can be stored or retrieved; primarily used for building machine learning models, but other applications possible).

[0049] 4A: An integration server (assuming a firewalled internet connection is available) with two-way communication with a wired or wireless link, with an embedded computer and two-way communication with a wired or wireless link, with two-way communication with a hub, and with two-way communication with an AR headset.

[0050] 4B: Integration server (assuming a firewalled internet connection is available) with two-way communication with wired or wireless links, with an embedded computer and two-way communication with wired or wireless links, with two-way communication with a hub, and with two-way communication with an AR headset.

[0051] 4C: An integration server (assuming a firewalled internet connection is available) with two-way communication with a wired or wireless link, with an embedded computer and two-way communication with a wired or wireless link, with two-way communication with a hub, with an embedded computer, with two-way communication with an AR headset.

[0052] 4D.1: An integrated server (assuming a firewall-protected internet connection is available) with two-way communication with an AR headset, with an embedded computer, with two-way communication with a wired or wireless link.

[0053] 4D.2: Integrated server with two-way communication with a wired or wireless link and with an embedded computer and with two-way communication with a hub (assuming a firewall-protected Internet connection is available).

[0054] Use Case 5 - A standalone bedside medical device enabling see-through viewing as described above and with a direct connection in degraded mode to an integrated server.

[0055] 5A: Integration and internal server (assuming a firewalled internet connection is available) with two-way communication with wired or wireless link, with embedded computer and two-way communication with wired or wireless link, with two-way communication with hub, with two-way communication with AR headset.

[0056] 5B: Integration and internal server (assuming a firewalled internet connection is available) with two-way communication with wired or wireless links, with embedded computer and two-way communication with wired or wireless links, with two-way communication with a hub, with two-way communication with an AR headset.

[0057] 5C: Integration and internal server (assuming a firewalled internet connection is available) with two-way communication with a wired or wireless link, with an embedded computer and two-way communication with a wired or wireless link, with two-way communication with a hub, with an embedded computer, with two-way communication with an AR headset.

[0058] 5D.1: Integration and internal server with two-way communication with a wired or wireless link and with an embedded computer, with two-way communication with an AR headset (assuming a firewall-protected Internet connection is available).

[0059] 5D.2: Integration and internal servers with two-way communication with wired or wireless links and with embedded computers, with two-way communication with a hub (assuming a firewall-protected Internet connection is available).

[0060] Imaging devices capable of providing real-time image video feeds to the device connectivity hub include, for example, ultrasound, endoscopes, laparoscopes, etc. These same devices may also provide still image capture, if desired. Devices capable of providing still images, which may not be available in the surgical suite for a variety of reasons, and the use of such real-time and still images in connection with the present invention are discussed further below.

[0061] Image displays on an AR headset can be controlled and positioned differently for various procedures / specialties. For example, device images can be positioned adjacent to the top edge of the AR view panel while the see-through optics are positioned adjacent to the bottom edge of the AR view panel (see Trainee T2 in FIG. 9 and Trainer T1 in FIG. 10). The device images and see-through optics may be positioned side-by-side (see FIG. 6). Additionally, multiple device images (e.g., from two different devices) may be displayed on the view panel (see FIG. 7).

[0062] The controls provided by the software may enable the trainee and / or trainer (or a single practitioner) to switch between various device images and other visual feeds (e.g., a video feed sent to the trainee by the trainer or retrieved from Database D (FIG. 1)), and image view placement options for see-through optics and separate video feeds.

[0063] Such controls may enable a user to perform image adjustments such as, for example, image cropping, resizing, resolution adjustment, color balancing, zooming, mirroring, rotation, and horizontal or vertical image flipping.

[0064] For devices that cannot be used during the procedure, images may be acquired pre- or during the procedure and transmitted electronically to a device connectivity hub or otherwise downloaded and retrieved upon request of the trainee and / or trainer. Additionally, some imaging devices are mobile and may be brought into the surgical suite and used during pauses in the procedure (e.g., x-ray machines and scanners).

[0065] Controls may be provided to allow the trainee and / or trainer to select one or both modalities of still images and real-time image feed.

[0066] The present invention therefore provides a system for communication between a trainee and a remote procedure trainer, the system comprising: a) first and second headsets wearable by a trainee and a remote trainer, respectively, each of the headsets having a viewing pane operable to simultaneously enable visualization by the wearer of at least two viewing areas, the first viewing area being a see-through lens that allows the wearer to view a live procedure site located adjacent to the wearer, and the second viewing area being an image received by the headset as an electronic signal; b) first and second device connection hubs connected to the first and second headsets, respectively, each of the hubs operable to receive images from at least one imaging device; and and c) a software program having telecommunications capabilities, accessible to each of the first and second headsets, and operable to project an image into the second viewing area.

[0067] The first and second viewing ranges may be positioned adjacent to one another either vertically or horizontally within the viewing pane of the AR headset.

[0068] The first and second device connectivity hubs and / or the respective AR headsets can be coupled to software operable to perform signal processing on images received from the at least one imaging device.

[0069] The procedure being performed may be a surgical procedure on the patient.

[0070] The software may be operable to enable a trainer to transmit images from the trainer's headset to one or more trainees' headsets.

[0071] The software may be operable to enable a trainer to transmit electronic data files to one or more trainee headsets, where the electronic data files may include one or both of text and video.

[0072] The software may be operable to enable a trainer to selectively view either the first or second viewing range of one or more trainees' headsets.

[0073] The software may be operable to enable the trainer to cause one or more viewing areas of a selected trainee to be displayed in the viewing areas of the headsets of non-selected trainees.

[0074] The headset may be an augmented reality headset, or a combination of virtual reality and augmented reality.

[0075] The software may be operable to allow selective voice communication between the trainee and the trainer, which may be routed through respective headsets. a) One or more cameras may be electronically connected to one or both headsets. b) One or more cameras may be located on each headset. c) Three or more headsets may be provided for use by three or more trainees. Software in the AR headset may be provided operable to selectively control one or more of the image resolution, color balance, image cropping, resizing, positioning, scaling, mirroring, rotation, and horizontal and vertical flipping of the AR headset.

[0076] Attention now turns to FIGS. 12-21, the present invention in a further embodiment enhances the use of an augmented reality (AR) headset with a two-dimensional (2D) or three-dimensional (3D) video display in the exemplary case of remote surgical training by providing a means for highlighting an area of ​​interest (AOI) within the line of sight (LOS) of the surgical field. Additionally, a method for tracking the AOI so that it is always within the line of sight of the surgical field is transmitted to a remote student or instructor. Further enhancements to the AR headset are necessary since the headset and glasses are used to illuminate and / or magnify the surgical field or AOI, and only one single-function headset can be worn at a time. Further enhancements include the ability to convert the AR headset into a virtual reality (VR), which has several important advantages when performing tasks while viewing a remote camera such as an endoscope, or when viewing instructional videos for the task at hand before performing the procedure locally. These scopes of functionality are typically mutually exclusive.

[0077] Since the surgeon already wears the eye-front display and the gaze camera, these functions can be combined and modified, which increases the usefulness and functionality of the head-worn display. The present invention therefore includes a multi-function headset that realizes these functions in further embodiments. These functions described above and further described below are designed to enhance the working environment for any kind of procedure, for example, during maintenance, manufacturing sites, law enforcement situations, or strictly sequenced procedures. Furthermore, the controller can have software for specific applications in addition to the native RTP software that serves as a function of the embodiments described above. For example, dental digital reconstruction software can be embedded in the controller, allowing the user to use the headset as an attached monitor with see-through vision capability to operate a dental scanner to generate a digital reconstruction of the mouth for the purpose of making appliances.

[0078] 3 and 12, the AR headset 12 can include a laser pointer 23 with a laser spot 24 pointing to an area of ​​interest (AOI) within the field of view (FOV) seen through the AR headset 12. A physical laser projection method mounted on the headset 12 and aligned with the headset camera 13 presents both the trainer and the trainee with a visual indicator that the FOV of a similarly mounted 2D or 3D camera is within the targeted area, so that they can point to each other the area of ​​interest (AOI). To provide instruction, the center of the LOS can be illuminated as a single point and the wearer can be visually instructed to look at the center of the 2D or 3D FOV of the AOI. This further allows the instructor / trainer wearing the headset 12 to point to specific elements in the environment, AOI, and FOV to improve instruction.

[0079] It is desirable to ensure that the FOV of the head mounted camera is over the object and that the instructor / trainer can see that the FOV is being transmitted to the remote student / trainee. The physically attached pointing mechanism can be achieved by a number of different physical techniques. The physically attached pointing mechanism can take the form of, for example, visible light, sound, or heat means, in which case the headset includes a cue receiver operable to convert the pointing mechanism into the visible spectrum of the see-through optics to provide appropriate feedback to the wearer.

[0080] The display may be a virtual display in a heads-up display, see-through display, rather than a physical form or energy as described above. The pointer may be in the form of a bracket inside the edge of the AOI so the presenter knows that the pointer is within the camera's FOV. This display method may be a bracket of any shape, or a single indicator pointer dot, i.e. a "line of sight instruction point" (LOSIP).

[0081] In a further embodiment, the four corners of the FOV may "bracket" the AOI with a geometric shape that covers or surrounds the AOI. Any suitable shape may be used, such as, for example, brackets, a circle, an ellipse, or a complex polygon that highlights the AOI, or the outline of an object.

[0082] FIG. 13 shows the option of visually defining the FOV by four corner brackets 101 that can be projected in the AR headset 12 with the AOI located within the FOV.

[0083] FIG. 14 shows an example where the AOI is centered within the FOV and the LOS instruction point ("LOSIP") is in the center of the FOV, indicating that a person wearing head mounted display 24 has the AOI perfectly centered within the FOV.

[0084] In yet another embodiment, instead of using a fixed physical or virtual marker covering or pointing to the central LOS of the physical location of the AOI, image processing or machine learning methods may be implemented to align the camera's FOV to the center of the detected object, for example in the form of a glove tip, barcode, or other trackable object.

[0085] The pointer, in yet another embodiment, can act as a ruler or other measuring device and can function in the XY or XYZ directions.

[0086] FIG. 15 shows an example where the AOI is off-center in the FOV, in the lower left, and further shows that the LOSIP is in the lower left quadrant of the FOV, indicating that the person wearing the headset 12 is not centering the head mounted display 24 within the FOV.

[0087] Since the center of the FOV can be found with the methods proposed above, the wearer can be instructed to center the FOV on the detected object using directional arrows or by displaying a small virtual rectangular or circular LOSIP that shows the wearer where the center of the FOV is, no matter where the camera is pointed. Other means of presenting direction may also be used, such as visually recognizable arrows that instruct the wearer to move their head left, right, up, or down to look.

[0088] FIG. 16 is a simplified diagram showing how a barcode 103, machine, or other item or device placed on an AOI (e.g., a mannequin) can be used to center the AOI in the FOV using mechanical tracking to direct the trainer's attention to that spot. What if the trainee wants to show the trainer a specific location, such as a button on an ultrasound machine, but doesn't want to manually adjust the optical bench or laser built into the headset to ensure that both the trainee and trainer are looking at the same place? Each location of interest can be labeled with a barcode 103 or other indicator, which will cause the trainee's headset to recognize the location and adjust the headset to ensure the trainer is looking at the same spot. To achieve this, a feedback loop is required where the barcode 103 is recognized by an element in the headset 12, which then mechanically adjusts the optical bench and laser so that the location is inside the center of the view. As an example of this functionality, a laser is aimed at the barcode 103, then a signal (optical, electromagnetic, or other) is sent that activates a receiver in the headset 12, which in turn sends a signal to a motor that activates a gear and moves the headset to center the location in the FOV. The motor can be actuated by any number of mechanisms, including slides, tabs, and more. In addition to mechanical gears, motion can also be initiated by magnets or other electromagnetic action. In one embodiment, the trainee may need to press a button on the controller while holding the laser centered on the location. Once the headset reaches the proper configuration (movement of both the optical bench and the laser), the trainee releases the button. In another embodiment, the trainee can use a software-activated hand gesture that represents the function of pressing a button.

[0089] In yet another embodiment, rather than using optical means to achieve magnification, an enhanced AR headset has selective use of digital zoom provided by a wide-angle lens camera and high megapixel-based zoom to display the AOI. For example, if a 100 megapixel digital camera is used with a 180 degree wide-angle lens, a 2000 pixel by 1000 pixel image will only occupy two percent (2%) of the available viewing area. FIG. 17 illustrates an embodiment in which the entire FOV is defined by a rectangular outline 105 and includes a sub-area of ​​interest 107 having a zoomed-in view of the AOI viewable through the headset 12.

[0090] The image of the head-mounted camera, when displayed on see-through optics, can act as a "digital magnifying glass" allowing the magnification of the AOI. A magnifying glass function can be added to a multi-function headset, thus realizing an AR headset that magnifies the surgical AOI. This functionality can be useful in many other applications, for example in other fields including manufacturing, mechanical assembly / repair, or law enforcement.

[0091] In yet another embodiment, another feature added to the multi-function head mounted display includes the ability to create a hybrid AR / VR system by using a mechanical black background that allows less than ninety-five percent (95%) of light within the FOV to reach the wearer's eyes. The black background cover is inserted within the LOS of the front-of-sight display. This allows the user to switch from the AR system to a VR system. The VR system is useful for tasks such as viewing remote and / or endoscopic cameras or watching instructional videos before performing a task without being distracted by the environment. FIG. 18 is a simplified diagram showing the mechanical shade mechanism 109 retracted (top half of the diagram), meaning the wearer is viewing in AR mode. The bottom half of the diagram shows the shade mechanism 109 deployed, meaning the wearer is utilizing the VR model.

[0092] Multi-function headset 12, in another embodiment, may have control over all of the video feeds associated with the wearer, including imaging associated with the on-site procedure, such as head-worn cameras, e.g., endoscopic, ultrasound, MRI, x-ray, thermal imaging, etc. These video feeds may be presented as remote virtual camera inputs, typically used in telecommunications software, to present remote participants with a multi-angle view of the procedure being performed away from their physical location.

[0093] In one embodiment, an embedded controller is provided that is operable to selectively position the video feeds in preferred locations. Instead of using a laptop to control the number and positioning of the video feeds, a belt-worn controller of the headset 12 may be used, as seen in FIG. 21, which is a simplified diagram of an embedded controller 113, that is operable to perform one or more functions or combinations of functions, including, for example, selective control of the laser pointer 23 (FIG. 3) in VR mode, directing the laser pointer in the XY direction to the AOI by turning the knob clockwise ("CW") or counter-clockwise ("CCW"), and directing it within and out of the Z dimension by holding down the knob and turning CW and CCW, respectively. Further options may be by using hand gestures or XY or XYZ controllers. This function can also be automated by pointing the webcam at a unique physical object, such as a barcode or an encasement enclosure for a simulated organ or organ system.

[0094] In another embodiment, a built-in mechanism for adjusting the convergence of the see-through light guide can be provided. The convergence point is adjusted automatically or manually by a push button on the controller using a webcam, a point at the desired distance, and a distance sensor. The lenses will then angle to converge to that point. Changing the angle of the lenses would require a mechanical feature to physically move the lenses. A mechanism for doing this is described in

[0087] . Diopter correction can be performed electronically by distorting the image to the desired correction value. Either the diopter or convergence correction value can be encoded on a logo embedded in the housing or case of the mannequin. FIG. 19 is a simplified diagram showing the left and right convergence angles of the AR waveguide and the distance to the AOI. FIG. 20 is a simplified diagram showing the convergence point "CP" where the distance to the FOV is sensed by a distance sensor 111 that may be attached to the headset 12.

[0095] When using an endoscope, hand gestures or XY controllers on the belt-worn puck can be used to highlight or point to specific features in the video as explained above. Webcams with laser indicators work well outside the mannequin / body, but it may still be necessary to resort to annotations that are cumbersome and slow. For example, a student trainee performing a cystoscopy may notice a small lesion and want it to be highlighted so that the trainer knows they are looking at the same lesion. By using hand gestures or XY or XYZ controllers on the belt-worn controller, the desired location on the video feed can be annotated.

[0096] FIG. 20 is a simplified diagram showing the convergence points where the distance to the FOV is sensed by the distance sensors shown in the figure.

[0097] FIG. 21 is a simplified diagram showing a belt-worn embedded controller that can control a laser illuminator and, in VR mode, can guide a pointer in the XY direction to an area of ​​interest by turning a knob CW or CCW, and can also guide the pointer within and out of the Z dimension by pressing and turning the knob CW and CCW, respectively.

[0098] Figure 22 is a simplified diagram showing a pointer that can be displayed on images collected from a digital camera attached to an endoscope. The pointer allows the trainer to highlight specific areas on the video image and can be controlled by software or from a dedicated location on the controller.

[0099] While the apparatus, method, and system of the present invention have been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as described. For example, while the invention is described herein in the context of surgery, it will be understood that the invention has application in other fields of technical endeavor (e.g., waste disposal, construction, underwater operations, tactical operations, etc.).

Claims

[Claim 1] A system for communicating between a trainee and a remote trainer of a procedure, comprising: a) first and second headsets wearable by the trainee and a remote trainer, respectively, each of the headsets having a viewing pane operable to simultaneously enable visualization by the wearer of at least two viewing areas, a first viewing area being a see-through lens that allows the wearer to view a live procedure site located adjacent to the wearer, and a second viewing area being an image received as an electronic signal by the headset; b) first and second device connection hubs connected to the first and second headsets, respectively, each of the hubs operable to receive images from at least one imaging device; and c) a software program having telecommunications capabilities, accessible to each of the first and second headsets, and operable to project the image into the second viewing area; and A system comprising:

Images