Systems and methods for augmented reality guided surgical procedures

Abstract

In some embodiments, apparatuses, and methods are provided herein useful to controlling AR surgical systems. In some implementations, a method comprises presenting, via an AR display device of a first user device, a first presentation to a first user including (i) a first AR portion that includes an AR interface providing the first user a first plurality of controls and (ii) a second AR portion that includes a plurality of AR objects, transmitting, to a second user device, data associated with the first presentation, presenting, via the second user device, a second presentation to a second user including a first portion that includes an interface providing the second user a second plurality of controls, receiving, via the second user device, an input associated with the second plurality of controls, and causing the AR display device of the first user device to present a modified version of the first presentation.

Description

SYSTEMS AND METHODS FOR AUGMENTED REALITY GUIDED SURGICAL PROCEDURESCross-Reference to Related Applications

[0001] This application claims the benefit of, and priority to, U.S. Provisional Patent Application No. 63 / 675,086 filed July 24, 2024, which is hereby incorporated by reference herein in its entirety.Technical Field

[0002] This invention relates generally to medical systems and, more specifically, augmented reality medical systems.Background

[0003] Currently, those in the medical field are exploring how augmented reality (AR) can be useful for surgical procedures. Such AR surgical systems typically include a display device (e.g., a head-mounted display device) that is worn by a user, such as a surgeon. The user interacts with the system via gestures, voice commands, button presses, etc. However, it can be difficult for the user to interact with the system for a variety of reasons. For example, if the user is a surgeon, they are likely using their hands during the surgical procedure, making it difficult to provide gesture input and / or button input. Further, systems can have difficulty processing voice commands, making handsfree input cumbersome. Accordingly, a need exists for AR surgical systems that provide for easier interaction between users and the AR guided surgical system.Brief Description of the Drawings

[0004] Disclosed herein are embodiments of systems, apparatuses, and methods pertaining controlling an AR surgical system. This description includes drawings, wherein:

[0005] FIG. 1A is a perspective view of a first user, a second user, and a patient during a surgical procedure using an AR surgical system, according to some implementations;

[0006] FIG. IB is a block diagram of the AR surgical system of FIG. 1A, according to some implementations;

[0007] FIG. 2 is a perspective view of the patient of FIG. 1;

[0008] FIG. 3 is a perspective view of the first user wearing a first user device of the AR surgical system of FIG. 1;

[0009] FIG. 4 is a perspective view of the second user holding a second user device of the AR surgical system of FIG. 1;

[0010] FIG. 5 A depicts a first of a series of images from the first user device and the second user device according to a first implementation as the first user registers landmarks, according to some implementations;

[0011] FIG. 5B depicts a second of the series of images from the first user device and the second user device according to a first implementation as the first user registers landmarks, according to some implementations;

[0012] FIG. 5C depicts a third of the series of images from the first user device and the second user device according to a first implementation as the first user registers landmarks, according to some implementations;

[0013] FIG. 6A depicts a second implementation of a first presentation of the first user device and a second presentation of the second user device, according to some implementations;

[0014] FIG. 6B depicts a third implementation of a first presentation of the first user device and a second presentation of the second user device, according to some implementations;

[0015] FIG. 6C depicts a fourth implementation of a first presentation of the first user device and a second presentation of the second user device, according to some implementations;

[0016] FIG. 6D depicts a fifth implementation of the a presentation of the first user device and a second presentation of the second user device, according to some implementations;

[0017] FIG. 7A depicts a first of a series of images from the first user device and the second user device as an input is entered via the second user device, according to some implementations;

[0018] FIG. 7B depicts a second of a series of images from the first user device and the second user device as an input is entered via the second user device, according to some implementations;

[0019] FIG. 7C depicts a third of a series of images from the first user device and the second user device as an input is entered via the second user device, according to some implementations;

[0020] FIG. 8A depicts a first of a series of images from the first user device and the second user device as an input is entered via the second user device including an indication presented via the first user device, according to some implementations;

[0021] FIG. 8B depicts a second of a series of images from the first user device and the second user device as an input is entered via the second user device including an indication presented via the first user device, according to some implementations;

[0022] FIG. 8C depicts a third of a series of images from the first user device and the second user device as an input is entered via the second user device including an indication presented via the first user device, according to some implementations;

[0023] FIG. 9A depicts a first of a series of images from the first user device and the second user device as an input is entered via the second user device including a prompt dialogue presented via the first user device, according to some implementations;

[0024] FIG. 9B depicts a second of a series of images from the first user device and the second user device as an input is entered via the second user device including a prompt dialogue presented via the first user device, according to some implementations;

[0025] FIG. 9C depicts a third of a series of images from the first user device and the second user device as an input is entered via the second user device including a prompt dialogue presented via the first user device, according to some implementations;

[0026] FIG. 9D depicts a fourth of a series of images from the first user device and the second user device as an input is entered via the second user device including a prompt dialogue presented via the first user device, according to some implementations; and

[0027] FIG. 10 is a flow chart depicting example operations for controlling an AR surgical system, according to some implementations.

[0028] Elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions and / or relative positioning of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of various embodiments of the present disclosure. Also, common but well- understood elements that are useful or necessary in a commercially feasible implementation are often not depicted in order to facilitate a less obstructed view of these various implementations of the present disclosure.Detailed Description

[0029] Generally speaking, pursuant to various embodiments, systems, apparatuses, and methods are provided herein useful to controlling an augmented reality (AR) surgical system. In some implementations, a method for controlling an AR surgical system for use in conducting an AR guided surgical procedure on a patient comprises presenting, via an AR display deviceof a first user device, a first presentation to a first user, the first presentation including (i) a first AR portion that includes an AR interface providing the first user a first plurality of controls for conducting the AR guided surgical procedure and (ii) a second AR portion that includes a plurality of AR objects, transmitting, from the first user device to a second user device, data associated with the first presentation, presenting, via a display device of the second user device based on the data associated with the first presentation, a second presentation to a second user, the second presentation including a first portion that includes an interface providing the second user a second plurality of controls for conducting the AR guided surgical procedure, receiving, via the second user device, an input associated with the second plurality of controls, and directly responsive to the receiving of the input, causing the AR display device of the first user device to present a modified version of the first presentation.

[0030] As previously discussed, AR surgical systems are currently being explored. Many of these systems include a device that is worn by the user, such as a surgeon. The device can be, for example, a head mounted display. Due to the nature of surgery, the surgeon’s hands are occupied for a large portion of the surgical procedure. Because of this, many systems allow the surgeon to provide handsfree user input. Such user input is often in form of voice commands. However, voice-based commands can be difficult for systems to process, leading to errors and / or the surgeon having to repeat themselves, neither of which is desirable. Accordingly, a need exists for AR surgical systems that provide for easier interaction between users and the AR-guided surgical system.

[0031] Described herein are systems, methods, and apparatuses that seek to minimize, if not eliminate, the drawbacks of current systems. In one implementation, a system comprises a first user device and a second user device. The first user device is used by a first user (e.g., a surgeon) and the second user device is used by a second person (e.g., a scrub tech, medical assistant, medical representative, etc.). The first user device can be a wearable device, such as a head mounted display. The second user device is separate from the first user device, and can take any suitable form (e.g., laptop computer, desktop computer, tablet, smartphone, personal digital assistant (PDA), smartwatch, etc.). The first user device presents a first presentation to the first user and the second user device presents a second presentation to the second user. The second presentation includes an interface providing the second user a plurality of controls for conducting an AR guided surgical procedure. The second user can provide user input via the second user device that causes the first presentation presented on the first user device to be modified. As but one example, the second user could select a “next button” that causes the ARguided surgical procedure to proceed to the next step. The discussion of FIGS. 1A and IB provides and overview of such a system.

[0032] FIG. 1A is a perspective view of a first user 102, a second user 104, and a patient 106 during a surgical procedure, and FIG. IB is a block diagram of an AR surgical system 100, according to some implementations. As depicted in FIG. 1A, the first user 102 is a medical professional, such as a surgeon. The first user 102 is performing a surgical procedure on the patient 106. Specifically, the surgical site 108 is adjacent to the patient’s 106 shoulder. The second user 104 is aiding the first user 102 in conducting the surgical procedure.

[0033] The first user 102 is conducting the surgical procedure with the help of a first user device 110. In the example depicted in FIG. 1 A, the first user device 110 is a head mounted display. The first user device 110 is configured to present a first presentation to the first user 102. The first presentation includes a first AR portion including an AR interface providing the first user 102 a first plurality of controls for conducting the surgical procedure. For example, the first portion can include controls such as those depicted in FIG. 6A. The first presentation also includes a second AR portion that includes a plurality of AR objects as depicted, for example, in FIG. 5C.

[0034] In the example depicted in FIG. 1A, the second user device 112 is a tablet computer. The second user device 112 is configured to present a second presentation to the second user 104. The second presentation includes a first portion including an interface providing the second user a second plurality of controls for conducting the surgical procedure. For example, the first portion can include controls such as those depicted in FIGS. 6A - 6D. The second user 104 can provide user input via the second user device 112 to control the AR guided surgical procedure and modify the first presentation that is presented to the first user 102 via the first user device 110.

[0035] As depicted in FIG. IB, the first user device 110 generally includes a control system 118, one or more cameras 120, and an AR display device 114. The cameras 120 are configured to capture image data associated with the patient 106 (e.g., including the surgical site 108). Accordingly, the cameras 120 can take any suitable form, such as still cameras, video cameras, stereoscopic cameras, etc. The AR display device 114 is configured to present the first presentation, including the AR objects, the first user 102. Accordingly, the AR display device 114 can take any suitable form. As one example, the AR display device 114 can present a video feed that is generated by the control system 118 based on the image data captured by the cameras 120 including AR objects superimposed on the video feed. Alternatively, the ARdisplay device 114 can be a “see-through” display device that allows the surgeon to view the physical world (e.g., the patient 106, surgical site 108, etc.) through the AR display device 114 with the AR objects presented via the AR display device 114.

[0036] The second user device 112 generally includes a control system 126, a user input device 128, and a display device 116. The display device 116 is configured to present the second presentation to the second user 104. The user input device 128 is configured to receive user input from the second user 104 and can take any suitable form (e.g., a touchscreen, keyboard, mouse, joystick, touchpad, etc.). The second user 104, via the user input device 128 of the second user device 112, can input commands for the AR guided surgical procedure. For example, the commands can be associated with navigation through a surgical plan, a selection for the AR guided surgical procedure, movement of an object, confirmation of an input, deleting previously recorded data, recording a landmark, loading a case, selecting a patient, confirming a patient, selecting a procedure, confirming a procedure, beginning a step, ending a step, etc. The user input provided by the second user 104 via the second user device 112 causes the first presentation 142 presented by the first user device 110 to be modified.

[0037] The control system 118, 126 can comprise a fixed-purpose hard-wired hardware platform (including but not limited to an application-specific integrated circuit (ASIC) (which is an integrated circuit that is customized by design for a particular use, rather than intended for general-purpose use), a field-programmable gate array (FPGA), and the like) or can comprise a partially or wholly-programmable hardware platform (including but not limited to microcontrollers, microprocessors, and the like). These architectural options for such structures are well known and understood in the art and require no further description here. The control system 118, 126 is configured (for example, by using corresponding programming as will be well understood by those skilled in the art) to carry out one or more of the steps, actions, and / or functions described herein.

[0038] By one optional approach the control system 118, 126 operably couples to a memory. The memory may be integral to the control system 118, 126 or can be physically discrete (in whole or in part) from the control system 118, 126 as desired. This memory can also be local with respect to the control system 118, 126 (where, for example, both share a common circuit board, chassis, power supply, and / or housing) or can be partially or wholly remote with respect to the control system 118, 126 (where, for example, the memory is physically located in another facility, metropolitan area, or even country as compared to the control system 118, 126).

[0039] This memory can serve, for example, to non-transitorily store the computer instructions that, when executed by the control system 118, 126, cause the control system 118, 126 to behave as described herein. As used herein, this reference to “non-transitorily” will be understood to refer to a non-ephemeral state for the stored contents (and hence excludes when the stored contents merely constitute signals or waves) rather than volatility of the storage media itself and hence includes both non-volatile memory (such as read-only memory (ROM) as well as volatile memory (such as an erasable programmable read-only memory (EPROM).

[0040] The first user device 110 and the second user device 112 are communicatively coupled via a network 124. The network 124 can take any suitable form, and include wired and / or wireless links. For example, the network 124 can be a local area network (LAN), wide area network (WAN), wireless wide area network (WWAN), a peer-to-peer network, etc. Additionally, the network 124 can operate over any standard, such as the 802.11 standard, a near field communication (NFC) standard (e.g. Bluetooth, Zigbee, etc.), etc.

[0041] While the discussion of FIGS. 1A and IB provides and overview of an AR surgical system, the discussion of FIGS. 2 - 4 provides additional detail regarding the first user, second user, and patient.

[0042] FIG. 2 is an isolated view of the patient 106 from FIG. 1. The patient 106 is depicted as lying on a surgical table. The first user (i.e., the surgeon) 102 is operating on the patient’s 106 shoulder with an exposed surgical site 108. The first user 102 is holding a medical tool 136.

[0043] FIG. 3 is an isolated view of the first user 102 from FIG. 1. The first user 102 is wearing the first user device 110. The first user device 110 includes cameras 120, the AR display device 114, speakers 132, a microphone 130, and headgear 134. The components (i.e., the cameras 120, the AR display device 114, speakers 132, and the microphone 130) are mounted to the headgear 134. It should be noted that while the first user device 110 depicted in FIG. 3 includes the cameras 120, the AR display device 114, speakers 132, and the microphone 130, implementations are not so limited. That is, in some implementations, the first user device 110 can include greater or fewer components than those depicted in FIG. 3.

[0044] FIG. 4 is an isolated view of the second user 104 holding the second user device 112 from FIG. 1. The second user device 104 is presenting the second presentation 140. The second presentation includes a first portion 160 and a second portion 152. The first portion 160 includes a plurality of controls for conducting the AR guided surgical procedure. As depicted in FIG. 4, the plurality of controls includes a back button, a next button, a brightnesscontrol, an arrow (e.g., controlling the location of the first presentation 142 and / or the second presentation 140 within the AR display device and / or the display device 116, respectively), a map (e.g., to take present a three-dimensional viewer), a gauge (e.g., to present a dashboard), sleep icon (e.g., to put the system in a low power state), and a close button. The second user 104 can control the AR guided surgical procedure via the plurality of controls. That is, user input provided by the second suer 104 via the plurality of controls presented on the second user device 112 can cause the first presentation 142 presented by the first user device 110 to be modified. It should be noted that, in some implementations, the plurality of controls can include greater, fewer, and / or different controls than those depicted in FIG. 4. The second portion 152 includes at least a portion of the view of the patient that is presented via the AR display device 114 of the first device 110. For example, as depicted in FIG. 4, the second portion 152 includes a portion of the patient 106, the surgical site 108, the first user’s 102 hands, and the medical tool 136.

[0045] While the discussion of FIGS. 1 - 4 provides detail regarding an AR guided surgical system, the discussion of FIGS. 5A - 5C provides additional detail regarding the interaction between the first user device and the second user device during a surgical procedure. FIGS. 5 - 9 depict a first presentation as presented by the first user device and a second presentation as presented by the second user device, arranged vertically. In each of FIGS. 5 - 9, the top view is the first presentation as presented by the first user device and the bottom view is the second presentation as presented by the second user device.

[0046] FIGS. 5A - 5C depict a series of images from the first user device 110 and the second user device 112 as the first user 102 registers landmarks, according to some implementations. Each of FIGS. 5 A - 5C depicts the first presentation 142 as presented by the first user device 110 and the second presentation 140 as presented by the second user device 112. In FIG. 5A, the first user 102 has not yet registered any landmarks. Accordingly, neither first presentation 142 nor the second presentation 140 include AR objects associated with landmarks. The first presentation 142 includes the second AR portion 158 and a model 144 of a portion of the patient’s 106 anatomy. The second presentation 140 includes the second portion 152 and the model 144 of the portion of the patient’s 106 anatomy. The second AR portion 158 and the second portion 152 include a view of the patient 106 including the surgical site 108, in addition to the first user’s 102 hands and the medical tool 136.

[0047] In FIG. 5B, the first user 102 has moved the medical tool 136 into position to register a landmark. As can be seen in FIG. 5B, the second AR portion 158 and the second portion 152have been updated based on the new position of the first user’s 102 hands and the medical tool 136 relative to that of FIG. 5 A.

[0048] In FIG. 5C, the first user 102 has registered a landmark 146, as indicated by a dot in the second AR portion 158 and the second portion 152. That is, when the first user 102 registers the landmark 146, an AR object (i.e., the dot) is shown overlaid on the surgical cite 108. Additionally, as depicted in FIG. 5C, the model 144 has been updated to include an indication of the landmark 146.

[0049] While the second presentation mirrors the first presentation in FIGS. 5A - 5C, FIGS. 6A - 6D depict alternative implementations in which the second presentation does not mirror the first presentation.

[0050] FIG. 6 A depicts a second implementation of the first presentation 142 of the first user device 110 and the second presentation 140 of the second user device 112, according to some implementations. As depicted in FIG. 6A, the first user device 110 is presenting the first presentation 142. The first presentation 142 includes a first AR portion 150 and a second AR portion 158. The first AR portion 150 includes an AR interface providing the first user 102 a plurality of controls for conducting the AR guided surgical procedure. The second AR portion 158 includes a plurality of AR objects (e.g., registered landmarks, cut guides, trajectories, etc.). As depicted in FIG. 6A, the second AR portion 158 also includes a portion of the patient 108 including the surgical site 108, as well as he first user’s 102 hands and the medical tool 136. The second AR portion 158 is a video feed captured by the cameras 120 and the AR objects are superimposed over the video feed.

[0051] The second user device 112 is presenting the second presentation 140. In the implementation depicted in FIG. 6A, the second presentation 140 includes only a first portion 160. The first portion 160 includes a second plurality of controls for conducting the AR guided surgical procedure. The second user 112 can provide user input via the second plurality of controls to control the AR guided surgical procedure and cause the first presentation 142 to be modified. It should be noted that in FIG. 6 A, the first portion 160 mirrors the first AR portion 150 in that the second plurality of controls includes the same controls, and similar positioning, as the second plurality of controls.

[0052] While FIG. 6A depicts an implementation in which the second plurality of controls mirrors the first plurality of controls, FIG. 6B depicts an implementation in which the second plurality of controls does not mirror the first plurality of controls.

[0053] FIG. 6B depicts a third implementation of the first presentation 142 of the first user device 110 and the second presentation 140 of the second user device 112, according to some implementations. As with FIG. 6A, the first presentation 142 includes the first AR portion 150 and the second AR portion 158, and the second presentation 140 includes the first portion 140. However, in FIG. 6B, the plurality of controls of the first portion 160 do not mirror the plurality of controls of the first AR portion 150. Specifically, the plurality of controls of the first portion 160 includes fewer controls than the plurality of controls of the first AR portion 150. It should be noted that, in some implementations, the plurality of controls of the first portion 160 can differ from the plurality of controls of the first AR portion 150 by including additional controls and / or different controls with respect to the plurality of controls of the first AR portion 150.

[0054] While FIGS. 6 A and 6B depict implementations in which the second presentation includes on the first portion, FIG. 6C depicts an implementation in which the second presentation includes both the first portion and the second portion.

[0055] FIG. 6C depicts a fourth implementation of the first presentation 142 of the first user device 110 and the second presentation 140 of the second user device 112, according to some implementations. As with FIGS. 6 A and 6B, the first presentation 142 includes the first AR portion 150 and the second AR portion 158, and the second presentation 140 includes the first portion 140. However, in FIG. 6C, the second presentation 140 also includes the second portion 152. The second portion 152 is based, at least in part, on data associated with the first presentation 142. The second portion 152 includes a portion of the patient 106 including the surgical site 108, the second user’s 102 hands, and the medical tool 136. As depicted in FIG. 6C, the second portion 152 is a picture-in-picture view of the second AR portion 158 of the first presentation 142.

[0056] While FIGS. 6A - 6C depict implementations in which the second presentation is generally similar to the first presentation, FIG. 6D depicts an implementation in which the second presentation includes the first portion and the second portion in a different arrangement than that of the first AR portion and the second AR portion of the first presentation.

[0057] FIG. 6D depicts a fifth implementation of the first presentation 142 of the first user device 110 and the second presentation 140 of the second user device 112, according to some implementations. As with FIG. 6C, the first presentation 142 includes the first AR portion 150 and the second AR portion 158, and the second presentation 140 includes the first portion 160 and the second portion 152. However, the first portion 160 and the second portion 152 are arranged in a manner that is different than that of the first AR portion 150 and the second ARportion 158 in the first presentation 142. As depicted in FIG. 6D, the first portion 160 and the second portion 152 are presented as picture-in-picture frames, with the plurality of controls of the first portion 160 being arranged vertically on a first side of the display device 116 and the second portion arranged on a second side of the display device 116.

[0058] While the discussion of FIGS. 6A - 6D provides additional detail regarding various implementations of the first presentation and the second presentation, the discussion of FIGS. 7 - 9 provides additional detail regarding notifications and prompts presented via the first user device when the second user provides user input via the second user device.

[0059] FIGS. 7A - 7C depict a series of images from the first user device 110 and the second user device 112 as an input is entered via the second user device 112, according to some implementations. In FIG. 7A, the first user 102 is preparing to register landmarks via the medical tool 136. To aid the first user 102, the second user 104 will control the navigation through the surgical plan via the second user device 112. As depicted in FIG. 7B, the second user 104 is pressing the “next button” 154 to advance the surgical plan to the next step. The second user 104 is pressing the “next button” 154 via the plurality of controls of the first portion 160 of the second presentation 140 on the second user device 112. As depicted in FIG. 7C, the first presentation 142 of the first user device 110 and the second presentation 140 of the second user device 112 have both advanced to the next screen in which the model 144 of the patient’s anatomy is now visible. That is, responsive to the user input provided by the second user 112 (i.e., selection of the “next button” 154), the first presentation 142 as presented by the first user device 110 has been modified.

[0060] While FIGS. 7A - 7C depict an implementation in which the second user has advanced the surgical plan without a notification to the first user, FIGS 8 A - 8C depict an implementation in which the first user device presents an indication of the advancement of the surgical plan by the second user.

[0061] FIGS. 8 A - 8C depict a series of images from the first user device 110 and the second user device 112 as an input is entered via the second user device 112 including an indication presented via the first user device 110, according to some implementations. As with FIG. 7 A, the first user 102 is preparing to register landmarks via the medical tool 136. To aid the first user 102, the second user 104 will control the navigation through the surgical plan via the second user device 112. As depicted in FIG. 8B, the second user 104 is pressing the “next button” 154 to advance the surgical plan to the next step. The second user 104 is pressing the “next button” 154 via the plurality of controls of the first portion 160 of the second presentation140 on the second user device 112. However, unlike in FIG. 7B, in FIG. 8B, the first presentation 142 is modified to include a dialogue 156 (i.e., an “indication”). The dialogue 156 includes an indication that the second user 104 has provided user input via the “next button” 154 (i.e., the plurality of controls presented via the second user device 112) to advance the surgical plan to the next step. As with FIG. 7C, in FIG. 8C, the first presentation 142 of the first user device 110 and the second presentation 140 of the second user device 112 have both advanced to the next screen in which the model 144 of the patient’s anatomy is now visible. That is, responsive to the user input provided by the second user 112 (i.e., selection of the “next button” 154), the first presentation 142 as presented by the first user device 110 has been modified.

[0062] While FIGS. 8 A - 8C depict an implementation in which a notification that the second user has advanced the surgical plan is presented via the first user device, FIGS. 9A - 9D depict an implementation in which the first user provides a confirmation of the second user’s input before the surgical plan is advanced (i.e., before the first presentation is modified).

[0063] FIGS. 9 A - 9D depict a series of images from the first user device and the second user device as an input is entered via the second user device including a prompt dialogue presented via the first user device, according to some implementations. As with FIG. 7A, the first user 102 is preparing to register landmarks via the medical tool 136. To aid the first user 102, the second user 104 will control the navigation through the surgical plan via the second user device 112. As depicted in FIG. 9B, the second user 104 is pressing the “next button” 154 to advance the surgical plan to the next step. The second user 104 is pressing the “next button” 154 via the plurality of controls of the first portion 160 of the second presentation 140 on the second user device 112. However, unlike in FIG. 7B, in FIG. 9B, the first presentation 142 is modified to include a dialogue 156 requesting that the first user 102 confirm the second user’s 104 user input to advance the surgical plan to the next step. In such implementations, the first user 102 must confirm the second user’s 104 user input before the first presentation 142 (and, in some implementations, the second presentation 140) is modified. As depicted in FIG. 9C, the first user 102 has confirmed the second user’s 104 user input, as indicated by the hashing on the dialogue 156. The first user 102 can confirm the second user’s 104 user input by, for example, selecting the dialogue 156 and / or a button associated with the dialogue 156. For example, the first user 102 can confirm the second user’s 104 user input via a voice command, a gaze (visual) command, a touch input, etc. As with FIG. 7C, in FIG. 9D, the first presentation 142 of the first user device 110 and the second presentation 140 of the second user device 112 have bothadvanced to the next screen in which the model 144 of the patient’s anatomy is now visible. That is, responsive to the first user’s 102 confirmation of the second user’s 104 user input, the first presentation 142 as presented by the first user device 110 has been modified.

[0064] As discussed with respect to FIGS. 7 - 9, in some implementations, the first user device 110 presents a dialogue 156 (e.g., a notification, prompt dialogue, etc.) indicating that the second user 104 has provided user input via the second user device 112 to modify the first presentation 142. Whether a dialogue 156 is presented, and whether a confirmation is required from the first user 102, depends on the type of input associated with the second user device 112 (i.e., the plurality of controls associated with the second user device 112). For example, the input associated with the second user device 112 can be a no confirmation command, a passive confirmation command, an active confirmation command, and / or a restricted command.

[0065] A no confirmation command does not require the first user 102 to approve or acknowledge the command. A no confirmation command may, or may not, include an informative dialogue 156 on the first user device 110. A passive confirmation command requires that the first user 102 not disapprove the command before the first presentation 142 is modified. For example, a passive confirmation command can cause a dialogue 156 to be presented via the first user device 110. So long as the first user 102 does not disapprove of the command (e.g., within a predetermined time period), the first presentation 142 will be modified based on the command. An active confirmation command requires the first user to approve of and / or acknowledge a command before the first presentation 142 is modified, (e.g., as shown in FIGS. 9A - 9D). Finally, a restricted command is a command that the second user device 112 is not able to instruct. For example, certain user input may be restricted such that a command may not be acted upon by the first user device 110 regardless of whether the first user 102 approves of and / or acknowledges the command.

[0066] Additionally, in some implementations, the first user 102 can alter the second user’s 104 ability to provide input via the second user device 112 and / or modify the first presentation 142. For example, the first user 102 can perform limit or otherwise restrict input via the second user device 112 via the first user device. In such implementations, the first user 102 can provide a command (e.g., gaze, voice, touch, etc.) command via the first user device 102. The command 102 alters the functionality of the second user device 112. The functionality of the second user device 112 can be limited such that the second user device 112 does not allow the second user 104 to provide input (e.g., removing controls, disabling controls, etc.), prevents the first user device 102 from acting on commands from the second user device 112, causesthe second user device 114 to stop presenting the second presentation 140 or portions of the second presentation 140, etc.

[0067] While the discussion of FIGS. 1 - 9 describes systems, methods, and apparatuses for controlling an AR surgical system, the discussion of FIG. 10 provides additional details regarding example operations of such a system.

[0068] FIG. 10 is a flow chart depicting example operations for controlling an AR surgical system, according to some implementations. The flow begins at block 1002.

[0069] At block 1002, a first presentation is presented. For example, a first user device can present the first presentation to a first user. The first presentation includes (i) a first AR portion and a (ii) a second AR portion. The first AR portion includes an AR interface providing a first user a first plurality of controls for conducting an AR guided surgical procedure. The second AR portion includes a plurality for AR objects. The flow continues at block 1004.

[0070] At block 1004, data associated with the first presentation is transmitted. For example, the first user device can transmit the data associated with the first presentation to a second user device via a communications network. The data associated with the first presentation can include any suitable data, such as data associated with image data captured by cameras, data associated with a user interface, data associated with the first user’s user input, etc. The flow continues at block 1006.

[0071] At block 1006, a second presentation is presented. For example, a second user device can present the second presentation to a second user. The second presentation includes a first portion. The first portion includes an interface providing the second user a second plurality of controls for conducting the AR guided surgical procedure. The flow continues at block 1008.

[0072] At block 1008, input associated with the second plurality of controls is received. For example, the input associated with the second plurality of controls can be received from the second user via the second user device. In some implementations, the input received via the second user device is mapped to the first user device. That is, the input is associated with the second plurality of controls and, in some implementations, the location of that input is mapped to the first presentation. In such implementations, the system can determine a second presentation coordinate associated with the input associated with the second plurality of controls. The second presentation coordinate is then mapped to a first presentation coordinate associated with the first presentation. Finally, the system determines a selection from the AR interface that corresponds to the first presentation coordinate. The flow continues at block 1010.

[0073] At block 1010, the AR display device is caused to present a modified version of the first presentation. For example, the AR display device can be caused to present a modified version of the first presentation based on the input associated with the second plurality of controls.

[0074] In some implementations, a method for controlling an AR surgical system for use in conducting an AR guided surgical procedure on a patient comprises presenting, via an AR display device of a first user device, a first presentation to a first user, the first presentation including (i) a first AR portion that includes an AR interface providing the first user a first plurality of controls for conducting the AR guided surgical procedure and (ii) a second AR portion that includes a plurality of AR objects, transmitting, from the first user device to a second user device, data associated with the first presentation, presenting, via a display device of the second user device based on the data associated with the first presentation, a second presentation to a second user, the second presentation including a first portion that includes an interface providing the second user a second plurality of controls for conducting the AR guided surgical procedure, receiving, via the second user device, an input associated with the second plurality of controls, and directly responsive to the receiving of the input, causing the AR display device of the first user device to present a modified version of the first presentation.

[0075] In some implementations, a system for use in conducting an AR guided surgical procedure on a patient comprises a a first user device operated by a first user comprising a camera configured to capture image data associated with the patient, a control system configured to generate, based on the image data associated with the patient, a first presentation including (i) a first AR portion that includes an AR interface providing the first user a first plurality of controls for conducting the AR guided surgical procedure and (ii) a second AR portion that includes a plurality of AR objects overlaid on a view of the patient, an AR display device configured to present the first presentation, and a first communications radio configured to transmit data to and receive data from a second user device, and the second user device comprising a second communications radio configured to transmit data to and receive data from the first user device, a display device configured to present a first presentation including a first portion that includes an interface providing the second user a second plurality of controls for conducting the AR guided surgical procedure, and a user input device configured to receive input associated with the second plurality of controls, wherein the AR display device is further configured to present a modified version of the first presentation directly responsive to receipt of the input.

[0076] Those skilled in the art will recognize that a wide variety of other modifications, alterations, and combinations can also be made with respect to the above described embodiments without departing from the scope of the disclosure, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept.

Claims

CLAIMSWhat is claimed is:

1. A method for controlling an augmented reality (AR) surgical system for use in conducting an AR guided surgical procedure on a patient, the method comprising: presenting, via an AR display device of a first user device, a first presentation to a first user, the first presentation including (i) a first AR portion that includes an AR interface providing the first user a first plurality of controls for conducting the AR guided surgical procedure and (ii) a second AR portion that includes a plurality of AR objects; transmitting, from the first user device to a second user device, data associated with the first presentation; presenting, via a display device of the second user device based on the data associated with the first presentation, a second presentation to a second user, the second presentation including a first portion that includes an interface providing the second user a second plurality of controls for conducting the AR guided surgical procedure; receiving, via the second user device, an input associated with the second plurality of controls; and directly responsive to the receiving the input, causing the AR display device of the first user device to present a modified version of the first presentation.

2. The method of claim 1, wherein the modified version of the first presentation includes a modified version of the second AR portion of the first presentation.

3. The method of claim 1, wherein the interface providing the second user a second plurality of controls is different than the AR interface providing the first user a plurality of controls.

4. The method of claim 3, wherein the interface providing the second user a second plurality of controls is different than the AR interface providing the first user a plurality of controls in that the interface providing the second user a second plurality of controls includescontrols (i) in addition to, (ii) less than, (iii) different than, or (iv) any combination of (i), (ii), and (iii) those of the AR interface providing the first user a plurality of controls.

5. The method of claim 1, wherein the input associated with the second plurality of controls is associated with one or more of navigation through a surgical plan, a selection, movement of an object, confirmation of an input, deleting previously recorded data, recording a landmark, loading a case, selecting a patient, confirming a patient, selecting a procedure, confirming a procedure, beginning a step, and ending a step.

6. The method of claim 1, further comprising: determining, based on a type of the input associated with the second plurality of controls prior to the presenting the modified version of the first presentation, whether the input associated with the second plurality of controls requires confirmation; if the input associated with the second plurality of controls requires confirmation: presenting, via the first user device, a prompt dialogue associated with the input associated with the second plurality of controls; and receiving, via the first user device, user input to approve the prompt dialogue associated with the input associated with the second plurality of controls.

7. The method of claim 6, wherein the prompt dialogue is (i) incorporated into the first presentation, (ii) an auditory alert, (iii) a haptic alert, or (iv) any combination of (i) - (iii).

8. The method of claim 7, wherein the user input to approve the prompt dialogue is (i) auditory, (ii) visual, (iii) touch, or (iv) any combination of (i) - (iii).

9. The method of claim 6, wherein the type of the input associated with the second plurality of controls is (i) a no confirmation command, (ii) a passive confirmation command, (iii) an active confirmation command, (iv) a restricted command, or (v) any combination of (i) - (iv).

10. The method of claim 6, wherein the prompt dialogue includes an indication of the input associated with the second plurality of controls.

11. The method of claim 1, further comprising: determining, based on the input associated with the second plurality of controls, a second presentation coordinate associated with the input associated with the second plurality of controls; mapping the second presentation coordinate to a first presentation coordinate associated with the first presentation; determining, based on the first presentation coordinate, a selection from the AR interface that corresponds to the first presentation coordinate, wherein the modified version of the first presentation is modified based on the selection from the AR interface.

12. The method of claim 1, wherein the AR objects are (i) overlaid on an image of the patient, (ii) on the patient, or (iii) a combination of (i) and (ii).

13. The method of claim 12, wherein the image includes (i) a video feed, (ii) a still image, or (iii) any combination of (i) and (ii).

14. The method of claim 1, wherein the second presentation includes a second portion that is based at least in part on the data associated with the first presentation.

15. The method of claim 14, wherein the second portion of the second presentation includes at least a portion of the view of the patient.

16. The method of claim 1, further comprising: receiving, via the first user device, a command to limit control by the second user device; receiving, via the second user device, a second input associated with the second plurality of controls; and responsive to receiving the second input, preventing the AR display device of the first user device from modifying the first presentation.

17. The method of claim 1, further comprising:receiving, via the first user device, a command to limit control by the second user device; and causing at least one of the second plurality of controls to be disabled.

18. A system for use in conducting an augmented reality (AR) guided surgical procedure on a patient, the system comprising: a first user device operated by a first user comprising: a camera configured to capture image data associated with the patient; a control system configured to generate, based on the image data associated with the patient, a first presentation including (i) a first AR portion that includes an AR interface providing the first user a first plurality of controls for conducting the AR guided surgical procedure and (ii) a second AR portion that includes a plurality of AR objects overlaid on a view of the patient; an AR display device configured to present the first presentation; and a first communications radio configured to transmit data to and receive data from a second user device; and the second user device comprising: a second communications radio configured to transmit data to and receive data from the first user device; a display device configured to present a first presentation including a first portion that includes an interface providing the second user a second plurality of controls for conducting the AR guided surgical procedure; and a user input device configured to receive input associated with the second plurality of controls; wherein the AR display device is further configured to present a modified version of the first presentation directly responsive to receipt of the input.

19. The system of claim 18, wherein the modified version of the first presentation includes a modified version of the second AR portion of the first presentation.

20. The system of claim 18, wherein the interface providing the second user a second plurality of controls is different than the AR interface providing the first user a plurality of controls.

21. The system of claim 20, wherein the interface providing the second user a second plurality of controls is different than the AR interface providing the first user a plurality of controls in that the interface providing the second user a second plurality of controls includes controls (i) in addition to, (ii) less than, (iii) different than, or (iv) any combination of (i), (ii), and (iii) those of the AR interface providing the first user a plurality of controls.

22. The system of claim 18, wherein the input associated with the second plurality of controls is associated with one or more of navigation through a surgical plan, a selection, movement of an object, confirmation of an input, deleting previously recorded data, recording a landmark, loading a case, selecting a patient, confirming a patient, selecting a procedure, confirming a procedure, beginning a step, and ending a step.

23. The system of claim 18, wherein the control system of the first user device is further configured to: determine, based on a type of the input associated with the second plurality of controls prior to the presenting the modified version of the first presentation, whether the input associated with the second plurality of controls requires confirmation; if the input associated with the second plurality of controls requires confirmation: cause presentation, via the first user device, of a prompt dialogue associated with the input associated with the second plurality of controls; and receive, via the first user device, user input to approve the prompt dialogue associated with the input associated with the second plurality of controls.

24. The system of claim 23, wherein the prompt dialogue is (i) incorporated into the first presentation, (ii) an auditory alert, (iii) a haptic alert, or (iv) any combination of (i) - (iii).

25. The system of claim 24, wherein the user input to approve the prompt dialogue is (i) auditory, (ii) visual, (iii) touch, or (iv) any combination of (i) - (iii).

26. The system of claim 23, wherein the type of the input associated with the second plurality of controls is (i) a no confirmation command, (ii) a passive confirmation command, (iii) an active confirmation command, (iv) a restricted command, or (v) any combination of (i) - (iv).

27. The system of claim 23, wherein the prompt dialogue includes an indication of the input associated with the second plurality of controls.

28. The system of claim 18, wherein the control system is further configured to: receive an indication of a second presentation coordinate associated with the input associated with the second plurality of controls; map the second presentation coordinate to a first presentation coordinate associated with the first presentation; determine, based on the first presentation coordinate, a selection from the AR interface that corresponds to the first presentation coordinate, wherein the modified version of the first presentation is modified based on the selection from the AR interface.

29. The system of claim 18, wherein the view of the patient is (i) a video feed of the patient, (ii) a view of the patient though the AR display device of the first user device, or (iii) a combination of (i) and (ii).

30. The system of claim 18, wherein the second presentation includes a second portion that is based at least in part on the data associated with the first presentation.

31. The system of claim 30, wherein the second portion of the second presentation includes at least a portion of the view of the patient.

32. The system of claim 18, wherein the control system is further configured to: receive, via the first user device, a command to limit control by the second user device.

33. The system of claim 32, wherein the control system is further configured to:cause transmission, by the first communications radio, an indication of the command to limit control by the second user device34. The system of claim 33, wherein the display device is further configured to: responsive to receipt of the indication of the command to limit control by the second user device, disable at least one of the second plurality of controls.

35. The system of claim 32, wherein the control system is further configured to: responsive to a second user input provided via the user input device, preventing the AR display device of the first user device from modifying the first presentation.

Images