Surgical systems

Through augmented reality technology, AR headsets provide guides during surgery, solving the availability and tracking difficulties of existing systems, achieving more efficient and convenient surgical operations.

CN112932661BActive Publication Date: 2024-11-26GLOBUS MEDICAL INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011451581.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-12-10
Filing Date
2020-12-10
Publication Date
2024-11-26
Estimated Expiration
2040-12-10

AI Technical Summary

Technical Problem

The existing surgical guide system has limitations in availability and ergonomics. Surgeons need to turn their heads to view information and rely on others to operate equipment. The guide may cause intermittent pause and tracking difficulties.

Method used

With augmented reality (AR) headsets, equipped with a perspective display and opacity filter, allowing surgeons to wear and view computer-generated guided information and control operating room equipment in real-world scenarios, through AR headsets The headset controller communicates with the guide controller, generates an AR image and displays it on the perspective display.

Benefits of technology

It improves the usability and efficiency of guided tours in surgical procedures, reduces dependence on other people's operations, avoids navigation pauses, and enhances real-time tracking of patients and surgical instrument postures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112932661B_ABST
    Figure CN112932661B_ABST
Patent Text Reader

Abstract

A surgical system includes an AR headset and an AR headset controller. The AR headset is configured to be worn by a user during a surgical procedure and has a perspective display screen, which is configured to display an AR image and allow at least a portion of a real-world scene to pass through it for viewing by the user. The AR headset also includes an opacity filter, which is positioned between at least one of the user's eyes and the real-world scene when the perspective display screen is viewed by the user. The opacity filter provides opacity to light from the real-world scene. The AR headset controller communicates with a navigation controller to receive navigation information from the navigation controller to provide guidance to the user during the surgical procedure performed on an anatomical structure, and generates the AR image based on the navigation information for display on the perspective display screen.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to medical devices and systems, and more particularly, to computer-assisted navigation in surgical procedures performed using robotic surgical systems. Background Art

[0002] Computer-assisted navigation during surgery provides surgeons with enhanced visualization of surgical instruments relative to radiographic images of the patient's anatomy. However, existing navigation systems have limitations in usability and ergonomics for surgeons, including: 1) the surgeon must turn his / her head away from the patient and surgical instruments to view the navigation information; 2) reliance on other personnel in the operating room to operate the software functions of various devices; and 3) intermittent pauses caused by providing navigation during surgery while personnel and / or objects obstruct the ability of the tracking component to track the posture of the patient, robot, and surgical instruments. Summary of the invention

[0003] Various embodiments disclosed herein relate to improvements in computer-assisted navigation during surgical procedures. An augmented reality (AR) headset is operably connected to a surgical system to provide an interactive environment through which surgeons, assistants, and / or other personnel can view and manipulate patient images, view and manipulate computer-generated navigation information, and / or control surgical equipment in an operating room.

[0004] Some embodiments of the present disclosure relate to a surgical system comprising an AR headset and an AR headset controller. The AR headset is configured to be worn by a user during a surgical procedure and has a perspective display screen, which is configured to display an AR image and allow at least a portion of a real-world scene to pass through it for viewing by the user. The AR headset also includes an opacity filter, which is positioned between at least one of the user's eyes and the real-world scene when the perspective display screen is viewed by the user. The opacity filter is configured to provide opacity to light from the real-world scene. The AR headset controller is configured to communicate with a navigation controller to receive navigation information from the navigation controller to provide guidance to the user during a surgical procedure performed on an anatomical structure, and is configured to generate the AR image based on the navigation information for display on the perspective display screen.

[0005] More specifically, some embodiments of the present disclosure relate to an AR headset comprising a see-through display screen and an opacity filter. The see-through display screen is configured to display an AR image and allow at least a portion of a real-world scene to pass through it for viewing by a user. The opacity filter is configured to be positioned between at least one of the user's eyes and the real-world scene when the user is wearing the AR headset to view the see-through display screen. The opacity filter is also configured to provide opacity to light from the real-world scene.

[0006] After reviewing the following figures and detailed description, other surgical systems, AR headsets, and corresponding methods and computer program products according to the embodiments will be apparent to those skilled in the art. All such surgical systems, AR headsets, and corresponding methods and computer program products are intended to be included in this specification, within the scope of the present disclosure, and protected by the appended claims. In addition, all embodiments disclosed herein are intended to be implemented alone or in combination in any manner and / or combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings illustrate certain non-limiting embodiments of the inventive concept, and are included to provide a further understanding of the present disclosure, and are incorporated in and constitute a part of this application. In the drawings:

[0008] Figure 1 An embodiment of a surgical system according to some embodiments of the present disclosure is presented;

[0009] Figure 2 Some embodiments of the present disclosure are shown Figure 1 A surgical robotic component of a surgical system;

[0010] Figure 3A Some embodiments of the present disclosure are shown Figure 1 A camera tracking system component of a surgical system;

[0011] Figure 3B and 3C It shows that some embodiments of the present disclosure can be used with Figure 1 a front view and an isometric view of another camera tracking system component for use with a surgical system;

[0012] Figure 4 An embodiment of an end effector attachable to a robotic arm and configured in accordance with some embodiments of the present disclosure is presented;

[0013] Figure 5Medical operations were demonstrated in which surgical robots and camera systems were positioned around the patient;

[0014] Figure 6 Demonstrated for medical operations Figure 5 A block diagram view of components of a surgical system;

[0015] Figure 7 Demonstrates the ability to navigate the surgical system Figure 5 and 6 Various display screens displayed on the monitor;

[0016] Figure 8 A block diagram showing some electrical components of a surgical robot according to some embodiments of the present disclosure;

[0017] Fig. 9 A block diagram showing components of a surgical system including an imaging device connected to a computer platform that may be operably connected to a camera tracking system and / or a surgical robot according to some embodiments of the present disclosure is shown;

[0018] Fig.10 An embodiment of a C-arm imaging device that can be used in combination with a surgical robot according to some embodiments of the present disclosure is presented;

[0019] Fig.11 An embodiment of an O-arm imaging device that may be used in combination with a surgical robot according to some embodiments of the present disclosure is presented;

[0020] Fig.12 A block diagram view of components of a surgical system including a pair of AR headsets operating according to some embodiments of the present disclosure is shown;

[0021] Fig.13 An AR headset configured according to some embodiments of the present disclosure is shown;

[0022] Fig.14 An electrical assembly of an AR headset that may be operably connected to a computer platform, one or more imaging devices, and / or a surgical robot according to some embodiments of the present disclosure is shown;

[0023] Figure 15-25 shows an example view through a display screen of an AR headset according to some embodiments of the present disclosure;

[0024] Figure 26-31 showing an example view of a display screen of an AR headset for providing navigation assistance for surgical tools during the medical procedure according to some embodiments of the present disclosure;

[0025] Fig.32 An AR headset having a curved lens and an opacity filter operating as a see-through display extending laterally across and downwardly from a front of the AR headset and having an opacity filter on a surface of the curved lens according to some embodiments of the present disclosure is shown;

[0026] Fig.33 shows a block diagram of another embodiment of an opacity filter configured to have a variable opacity in response to signaling from an AR headset controller according to some embodiments of the present disclosure;

[0027] Fig.34 A block diagram showing components of a surgical system including an AR headset connected to an AR processing circuit system including an AR headset controller configured according to some embodiments of the present disclosure is shown; and

[0028] Figure 35-39 A flowchart is shown of operations that may be performed by an AR headset controller and a computer platform of a surgical system to control how information is displayed on a see-through display screen and / or to control the opacity of an opacity filter in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION

[0029] The inventive concept will now be described more fully below with reference to the accompanying drawings, in which examples of embodiments of the inventive concept are shown. The inventive concept can be implemented in many different forms and should not be construed as being limited to the embodiments listed herein. Rather, these embodiments are provided to make the disclosure more complete and complete, and to fully convey the scope of each inventive concept to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. A component in one embodiment may be defaulted to being present in or used in another embodiment.

[0030] Various embodiments disclosed herein relate to improvements in computer-assisted navigation during surgical procedures. An augmented reality (AR) headset is operably connected to a surgical system and is configured to provide an interactive environment through which a surgeon, assistant, and / or other personnel can view and make selections in patient images, view and make selections in computer-generated surgical navigation information, and / or control surgical equipment in an operating room.

[0031] Figure 1 An embodiment of a surgical system 2 according to some embodiments of the present disclosure is shown. Before performing an orthopedic or other surgical procedure, one may use, for example, Fig.10 C-arm imaging device 104 or Fig.11A three-dimensional ("3D") image scan of the planned surgical area of ​​the patient is taken from the O-arm imaging device 106, or from another medical imaging device such as a computed tomography (CT) image or MRI. Such a scan may be performed preoperatively (e.g., most commonly a few weeks prior to the procedure) or intraoperatively. However, any known 3D or 2D image scan may be used in accordance with various embodiments of the surgical system 2. The image scan is sent to a computer platform in communication with the surgical system 2, such as the computer platform 910 ( Fig. 9 ), the surgical system may include a camera tracking system 6, a surgical robot 4 (e.g., Figure 1 ), imaging devices (e.g., C-arm 104, O-arm 106, etc.), and an image database 950 for storing image scans of the patient. View computer platform 910 ( Fig. 9 ) generates a surgical plan defining a target pose for surgical tools used during a surgical procedure performed on the patient's anatomy. Example surgical tools (also referred to as tools) may include, but are not limited to, drills, screw drivers, retractors, and implants such as screws, spacers, interbody fusion devices, plates, rods, etc. In some embodiments, the surgical plan defining the target plane is planned on the 3D image scan displayed on the display device.

[0032] As used herein, the term "pose" refers to the position and / or rotation angle of an object (e.g., a dynamic reference array, an end effector, a surgical tool, an anatomical structure, etc.) relative to another object and / or a defined coordinate system. Thus, a pose can be defined based solely on the multi-dimensional position of an object relative to another object and / or a defined coordinate system, solely on the multi-dimensional rotation angle of the object relative to another object and / or a defined coordinate system, or in combination with the multi-dimensional position and the multi-dimensional rotation angle. Thus, the term "pose" is used to refer to the position, the rotation angle, or a combination thereof.

[0033] Figure 1 The surgical system 2 can assist the surgeon during a medical procedure by, for example, providing for use of holding tools, alignment tools, application tools, guide tools, and / or positioning tools. In some embodiments, the surgical system 2 includes a surgical robot 4 and a camera tracking system 6. The ability to mechanically couple the surgical robot 4 and the camera tracking system 6 can allow the surgical system 2 to be manipulated and moved as a single unit, and allow the surgical system 2 to have a small footprint in an area, allowing for easier movement through narrow passages and around turns, and allowing for storage within a smaller area.

[0034] The surgical procedure can begin with the surgical system 2 being moved from the medical storage room to the medical procedure room. The surgical system 2 can be manipulated through doorways, halls, and elevators to reach the medical procedure room. Within the medical procedure room, the surgical system 2 can be physically divided into two separate and distinct systems (the surgical robot 4 and the camera tracking system 6). The surgical robot 4 can be positioned adjacent to the patient in any suitable location to appropriately assist the medical staff. The camera tracking system 6 can be positioned at the bottom of the patient, at the patient's shoulder, or at any other location suitable for tracking the current posture and posture movement of the surgical robot 4 and the patient's track portion. The surgical robot 4 and the camera tracking system 6 can be powered by an onboard power supply and / or plugged into an external wall socket.

[0035] The surgical robot 4 may be used to assist the surgeon by holding and / or using tools during a medical procedure. In order to properly utilize and hold the tools, the surgical robot 4 may rely on a number of motors, computers, and / or actuators to function properly. Figure 1 As shown, the robot body 8 can serve as a structure in which multiple motors, computers, and / or actuators can be secured within the surgical robot 4. The robot body 8 can also provide support for the robotic telescoping support 16. The size of the robot body 8 can provide a stable platform to support attached components, and can house, conceal, and protect multiple motors, computers, and / or actuators that can operate the attached components.

[0036] The robot base 10 can act as a lower support for the surgical robot 4. In some embodiments, the robot base 10 can support the robot body 8 and can attach the robot body 8 to a plurality of drive wheels 12. This attachment to the wheels can allow the robot body 8 to move efficiently in space. The robot base 10 can run along the length and width of the robot body 8. The robot base 10 can be about two inches to about 10 inches high. The robot base 10 can cover, protect and support the drive wheels 12.

[0037] In some embodiments, Figure 1As shown, at least one drive wheel 12 can be attached to the robot base 10. The drive wheel 12 can be attached to the robot base 10 at any position. Each individual drive wheel 12 can rotate around a vertical axis in any direction. The motor can be placed above, inside or adjacent to the drive wheel 12. Such a motor can allow the surgical system 2 to be manipulated to any position and stabilize and / or level the surgical system 2. A rod located inside or adjacent to the drive wheel 12 can be pressed into a surface by the motor. The rod not shown can be made of any suitable metal to lift the surgical system 2. The rod can lift the robot base 10, which can lift the surgical system 2 to any height required to level or otherwise fix the surgical system 2 relative to the patient's orientation. The weight of the surgical system 2 is supported by a small contact area of ​​the rod on each wheel, preventing the surgical system 2 from moving during the medical procedure. This rigid positioning can prevent the object and / or person from accidentally moving the surgical system 2.

[0038] The robot track 14 can be used to facilitate moving the surgical system 2. The robot track 14 provides a person with the ability to move the surgical system 2 without grasping the robot body 8. Figure 1 As shown, the length of the robot track 14 can be the same length as the robot body 8, shorter than the robot body 8, and / or longer than the robot body 8. The robot track 14 can further provide protection to the robot body 8, thereby preventing the subject and / or medical personnel from contacting, hitting, or bumping into the robot body 8.

[0039] The robot body 8 may provide support for a selectively compliant articulated robotic arm, hereinafter referred to as a "SCARA". Due to the repeatability and compactness of the robotic arm, the use of the SCARA 24 within the surgical system 2 may be advantageous. The compactness of the SCARA may provide additional space within a medical procedure, which may allow a medical professional to perform a medical procedure without excessive clutter and restricted areas. The SCARA 24 may include a robotic telescopic support 16, a robotic support arm 18, and / or a robotic arm 20. The robotic telescopic support 16 may be positioned along the robot body 8. As shown in FIG. Figure 1 As shown, the robotic telescopic support 16 can provide support for the SCARA 24 and the display 34. In some embodiments, the robotic telescopic support 16 can extend and retract in a vertical direction. The body of the robotic telescopic support 16 can be any width and / or height configured to support the stress and weight placed thereon.

[0040] In some embodiments, the medical staff can move the SCARA 24 via commands submitted by the medical staff. As will be explained in further detail below, the commands can originate from a display 34, a tablet computer, and / or an AR headset (e.g., Fig. 9 The AR headset may eliminate the need for the medical staff to refer to any other display such as the display 34 or tablet computer, which enables the SCARA 24 to be configured without the display 34 and / or tablet computer. As will be explained in further detail below, the command may be generated by pressing a switch and / or pressing multiple switches, and / or may be generated based on gesture commands and / or voice commands sensed by the AR headset.

[0041] like Figure 5 As shown in , the activation assembly 60 may include a switch and / or multiple switches. The activation assembly 60 may be operable to transmit movement commands to the SCARA 24, thereby allowing the operator to manually manipulate the SCARA 24. When the switch or multiple switches are pressed, the medical staff may have the ability to move the SCARA 24 by the applied hand movement. Alternatively or additionally, as will be explained in further detail below, the operator may control the movement of the SCARA 24 by gesture commands and / or voice commands sensed by the AR headset. Additionally, when the SCARA 24 does not receive a command to move, the SCARA 24 may be locked in place to prevent accidental movement by medical staff and / or other objects. By locking in place, the SCARA 24 provides a stable platform through which the end effector 26 can guide surgical tools during a medical procedure.

[0042] The robotic support arm 18 may be connected to the robotic telescopic support 16 by various mechanisms. Figure 1 and 2 , the robotic support arm 18 can rotate in any direction relative to the robotic telescopic support 16. The robotic support arm 18 can rotate three hundred and sixty degrees around the robotic telescopic support 16. The robotic arm 20 can be connected to the robotic support arm 18 in any suitable position and by various mechanisms that enable rotation in any direction relative to the robotic support arm 18. In one embodiment, the robotic arm 20 can rotate three hundred and sixty degrees relative to the robotic support arm 18. This free rotation allows the operator to position the robotic arm 20 according to the surgical plan.

[0043] Figure 4 and 5The end effector 26 in the example is attached to the robotic arm 20 at any suitable position. The end effector 26 is configured to be attached to the end effector coupling 22 of the robotic arm 20 positioned by the surgical robot 4. The example end effector 26 includes a tubular guide that guides the movement of an inserted surgical tool relative to the anatomical structure on which the surgical procedure is to be performed.

[0044] In some embodiments, a dynamic reference array 52 is attached to the end effector 26. The dynamic reference array (also referred to herein as a "DRA") is a rigid body that can be placed on a patient's anatomy (e.g., bone), one or more AR headsets worn by personnel in an operating room, an end effector, a surgical robot, or a surgical tool in a guided surgical procedure. The computer platform 910 is configured to track the pose (e.g., position and rotational orientation) of the DRA in real time in combination with the camera tracking system 6 or other 3D positioning system. The DRA may include a fiducial, such as the illustrated arrangement of a sphere. Such tracking of the 3D coordinates of the DRA may allow the surgical system 2 to determine the DRA's relative position to the patient's anatomy (e.g., bone). Figure 5 The pose of the target anatomical structure of the patient 50 in any multi-dimensional space.

[0045] like Figure 1 As shown, the light indicator 28 can be positioned on top of the SCARA 24. The light indicator 28 can be any type of light that illuminates to indicate the "condition" in which the surgical system 2 is currently operating. In some embodiments, the light can be generated by LED lights that form a ring around the light indicator 28. The light indicator 28 can include a completely permeable material that allows light to pass through the entire light indicator 28. The light indicator 28 can be attached to the lower display support 30. As shown in FIG. Figure 2 As shown, the lower display support 30 can allow an operator to manipulate the display 34 to any suitable position. The lower display support 30 can be attached to the light indicator 28 by any suitable mechanism. In some embodiments, the lower display support 30 can rotate about the light indicator 28 or be rigidly attached to the light indicator. The upper display support 32 can be attached to the lower display support 30 by any suitable mechanism.

[0046] In some embodiments, the tablet computer can be used in conjunction with the display 34 and / or in the absence of the display 34. The tablet computer can be placed on the upper display support 32 in place of the display 34 and can be removed from the upper display support 32 during a medical procedure. In addition, the tablet computer can communicate with the display 34. The tablet computer can be connected to the surgical robot 4 via any suitable wireless and / or wired connection. In some embodiments, the tablet computer can program and / or control the surgical system 2 during a medical procedure. When the surgical system 2 is controlled with the tablet computer, all input and output commands can be replicated on the display 34. Using a tablet computer can allow the operator to manipulate the surgical robot 4 without having to move around the patient 50 and / or the surgical robot 4.

[0047] As will be explained below, in some embodiments, the surgeon and / or other personnel may wear an AR headset that can be used in conjunction with the display 34 and / or tablet computer, or one or more AR headsets may eliminate the need to use the display 34 and / or tablet computer.

[0048] like Figure 3A and 5 As shown, the camera tracking system 6 works in conjunction with the surgical robot 4 via a wired or wireless communication network. Figure 1 , 3 and 5, the camera tracking system 6 can include some components similar to the surgical robot 4. For example, the camera body 36 can provide the functions found in the robot body 8. The robot body 8 can provide a structure on which the camera 46 is mounted. The structure within the robot body 8 can also provide support for the electronic equipment, communication devices and power supply used to operate the camera tracking system 6. The camera body 36 can be made of the same material as the robot body 8. The camera tracking system 6 can communicate directly with the AR headset, tablet computer and / or display 34 via a wireless and / or wired network to enable the AR headset, tablet computer and / or display 34 to control the functions of the camera tracking system 6.

[0049] The camera body 36 is supported by a camera base 38. The camera base 38 may be used as the robot base 10. Figure 1 In the embodiment of the present invention, the camera base 38 can be wider than the robot base 10. The width of the camera base 38 can allow the camera tracking system 6 to be connected to the surgical robot 4. Figure 1 As shown, the width of the camera mount 38 can be large enough to fit outside of the robotic base 10. The additional width of the camera mount 38 can allow the surgical system 2 to provide additional maneuverability and support for the surgical system 2 when the camera tracking system 6 is connected to the surgical robot 4.

[0050] As with the robot base 10, a plurality of drive wheels 12 may be attached to the camera base 38. Similar to the operation of the robot base 10 and the drive wheels 12, the drive wheels 12 may allow the camera tracking system 6 to be stabilized and leveled or set to a fixed orientation relative to the patient 50. Such stabilization may prevent the camera tracking system 6 from moving during the medical procedure and may prevent the camera 46 from losing track of a DRA connected to the AR headset and / or surgical robot 4, losing track of a DRA connected to the AR headset and / or surgical robot 4, etc. Figure 3A and 5 The tracking of one or more DRAs 52 of the anatomical structure 54 and / or tool 58 within the designated area 56 is shown. This stability and maintenance of tracking enhances the ability of the surgical robot 4 to operate effectively with the camera tracking system 6. Additionally, the wide camera base 38 can provide additional support for the camera tracking system 6. Specifically, as Figure 3A and 5 As shown, the wide camera base 38 can prevent the camera tracking system 6 from tipping over when the camera 46 is positioned over the patient.

[0051] The camera telescopic support 40 can support the camera 46. In some embodiments, the telescopic support 40 can move the camera 46 higher or lower in the vertical direction. The camera handle 48 can be attached to the camera telescopic support 40 at any suitable position and is configured to allow the operator to move the camera tracking system 6 to a planned orientation before the medical procedure. In some embodiments, the camera handle 48 can be used to lower and raise the camera telescopic support 40. The camera handle 48 can perform the raising and lowering of the camera telescopic support 40 by pressing a button, a switch, a lever and / or any combination thereof.

[0052] The lower camera support arm 42 may be attached to the camera telescoping support 40 at any suitable location, in embodiments such as Figure 1 As shown, the lower camera support arm 42 can rotate three hundred and sixty degrees around the telescopic support 40. This free rotation can allow the operator to position the camera 46 in any suitable position. The lower camera support arm 42 can be connected to the telescopic support 40 by any suitable mechanism. The lower camera support arm 42 can be used to provide support for the camera 46. The camera 46 can be attached to the lower camera support arm 42 by any suitable mechanism. The camera 46 can pivot in any direction at the attachment area between the camera 46 and the lower camera support arm 42. In an embodiment, the curved track 44 can be disposed on the lower camera support arm 42.

[0053] The curved track 44 may be placed at any suitable location on the lower camera support arm 42. Figure 3AAs shown, the curved track 44 can be attached to the lower camera support arm 42 by any suitable mechanism. The curved track 44 can be of any suitable shape, which can be a crescent, a circle, a flat, an ellipse, and / or any combination thereof. The camera 46 can be movably positioned along the curved track 44. The camera 46 can be attached to the curved track 44 by, for example, rollers, brackets, braces, motors, and / or any combination thereof. Motors and rollers not shown can be used to move the camera 46 along the curved track 44. As shown in FIG. Figure 3A As shown, during a medical procedure, if an object blocks the camera 46 from observing one or more tracked DRAs, the motor can responsively move the camera 46 along the curved track 44. This motorized movement can allow the camera 46 to move to a new position that is no longer blocked by the object without moving the camera tracking system 6. When the camera 46 is blocked from observing one or more DRAs, the camera tracking system 6 can send a stop signal to the surgical robot 4, the AR headset, the display 34, and / or the tablet computer. The stop signal can prevent the SCARA 24 from moving until the camera 46 has reacquired the tracked DRA 52 and / or can warn the operator to wear the AR headset and / or watch the display 34 and / or the tablet computer. This SCARA 24 can be configured to respond to the receipt of the stop signal by stopping further movement of the base and / or the end effector coupling 22 until the camera tracking system can resume tracking of the DRA.

[0054] Figure 3B and 3C Demonstrated that Figure 1 The front view and isometric view of another camera camera tracking system 6' that can be used together with the surgical system of the surgical robot or can be used independently of the surgical robot. For example, the camera tracking system 6' can be used to provide guided surgery without the use of robotic guidance. Figure 3B and 3C The camera tracking system 6' with Figure 3A One of the differences between the camera tracking systems 6 is Figure 3B and 3C The camera tracking system 6' includes a housing for transporting a computer platform 910. The computer platform 910 can be configured to: perform camera tracking operations to track the DRA; perform guided surgical operations to provide surgical guidance information to a display device (e.g., an AR headset and / or other display device); and perform other computing operations disclosed herein.

[0055] Figure 6 Demonstrated for medical operations Figure 5 A block diagram view of the components of a surgical system. Figure 6, the navigation camera 46 has a navigation field of view 600 in which a reference array 602 attached to the patient, a reference array 604 attached to the surgical instrument, and the pose (e.g., position and orientation) of the robotic arm 20 are tracked. The navigation camera 46 may be a computer platform 910 that includes a computer configured to perform the operations described below. Figure 3B and 3C The reference array 602 is part of the camera tracking system 6' of the surgical robot 4. The reference array achieves tracking by reflecting light in a known pattern, which is decoded by the tracking subsystem of the surgical robot 4 to determine the corresponding pose of the reference array. If the line of sight between the patient reference array 602 and the navigation camera 46 is blocked (e.g., by medical staff, instruments, etc.), further navigation of the surgical instruments may not be possible, and responsive notifications can temporarily stop further movement of the robotic arm 20 and the surgical robot 4, display a warning on the display 34, and / or provide an audible warning to the medical staff. The display 34 is available to the surgeon 610 and assistant 612, but viewing requires turning the head away from the patient and changing the eye focus to different distances and positions. The navigation software can be controlled by a technician 614 based on voice commands from the surgeon.

[0056] Figure 7 It is shown that when using the navigation function of the surgical system 2, the surgical robot 4 can be used to Figure 5 and 6 The display screens may include, but are not limited to, a patient radiograph with an overlaid graphical representation of an instrument model positioned in the display screen relative to the anatomy based on the developed surgical plan and / or based on the pose of the tracked reference array, various user-selectable menus for controlling different stages of the surgical procedure, and virtually projected dimensional parameters of the implant (e.g., length, width, and / or diameter).

[0057] For guided surgery, various processing components (e.g., computer platform 910) and associated software described below are provided that enable preoperative planning of a surgical procedure (e.g., implant placement) and electronic delivery of the plan to a computer platform 910 for providing navigation information to one or more users during the planned surgical procedure.

[0058] For robotic navigation, various processing components (e.g., computer platform 910) and associated software described below are provided that enable preoperative planning of a surgical procedure (e.g., implant placement) and electronic transfer of the plan to the surgical robot 4. The surgical robot 4 uses the plan to guide the robotic arm 20 and connected end effector 26 to provide target poses of the surgical tools relative to the patient's anatomy for the steps of the planned surgical procedure.

[0059] The following embodiments relate to the use of one or more AR headsets that can be worn by a surgeon 610, assistant 612 and / or other medical personnel to provide an improved user interface for receiving information from and / or providing control commands to a surgical robot, camera tracking system 6 / 6' and / or other medical equipment in an operating room.

[0060] Figure 8 FIG. 4 is a block diagram showing some electrical components of a surgical robot 4 according to some embodiments of the present disclosure. Figure 8 , a load cell (not shown) can be configured to track the force applied to the end effector coupling 22. In some embodiments, the load cell can communicate with multiple motors 850, 851, 852, 853 and / or 854. When the load cell senses a force, information about the amount of force applied can be distributed from the switch array and / or multiple switch arrays to the controller 846. The controller 846 can obtain the force information from the load cell and process it with a switching algorithm. The controller 846 uses the switching algorithm to control the motor driver 842. The motor driver 842 controls the operation of one or more motors 850, 851, 852, 853 and / or 854. The motor driver 842 can direct a specific motor to generate an equal amount of force, such as measured by the load cell through the motor. In some embodiments, as indicated by the controller 846, the force generated can come from multiple motors, such as 850-854. Additionally, the motor driver 842 can receive input from the controller 846. The controller 846 can receive information about the direction of the force sensed by the force sensor from the force sensor. The controller 846 can use a motion controller algorithm to process this information. The algorithm can be used to provide information to a specific motor driver 842. In order to replicate the direction of the force, the controller 846 can activate and / or deactivate certain motor drivers 842. The controller 846 can control one or more motors, such as one or more of 850-854, to induce the movement of the end effector 26 in the direction of the force sensed by the force sensor. This force-controlled movement can allow the operator to move the SCARA 24 and the end effector 26 effortlessly and / or with very little resistance. The movement of the end effector 26 can be performed to position the end effector 26 in any suitable posture (i.e., relative to the position and angle orientation of the defined three-dimensional (3D) orthogonal reference axis) for use by medical personnel.

[0061] Figure 5As best shown, the activation assembly 60 can be in the form of a bracelet that wraps around the end effector coupling 22. The activation assembly 60 can be positioned on any portion of the SCARA 24, on any portion of the end effector coupling 22, can be worn by a medical professional (and communicate wirelessly), and / or any combination thereof. The activation assembly 60 can include a primary button and a secondary button.

[0062] Pressing the master button may allow the operator to move the SCARA 24 and the end effector coupling 22. According to one embodiment, once set in position, the SCARA 24 and the end effector coupling 22 may not move until the operator programs the surgical robot 4 to move the SCARA 24 and the end effector coupling 22, or uses the master button to move. In some examples, it may be necessary to press at least two non-adjacent master activation switches before the SCARA 24 and the end effector coupling 22 will respond to operator commands. Pressing at least two master activation switches may prevent accidental movement of the SCARA 24 and the end effector coupling 22 during a medical procedure.

[0063] Activated by the main button, the load cell can measure the magnitude and / or direction of the force applied by the operator (i.e., medical personnel) on the end effector coupling 22. This information can be transmitted to one or more motors within the SCARA 24, such as one or more of 850-854, which can be used to move the SCARA 24 and the end effector coupling 22. Information about the magnitude and direction of the force measured by the load cell can cause one or more motors (e.g., one or more of 850-854) to move the SCARA 24 and the end effector coupling 22 in the same direction as sensed by the load cell. This force-controlled movement can allow the operator to easily move the SCARA 24 and the end effector coupling 22, and because the motors move the SCARA 24 and the end effector coupling 22 while the operator moves the SCARA 24 and the end effector coupling 22, a lot of effort is not required.

[0064] In some examples, the operator can use the secondary button as a "select" device. During the medical operation, the surgical robot 4 can notify medical staff of certain situations through one or more AR headsets 920, displays 34, and / or light indicators 28. One or more AR headsets 920 are each configured to display images on a perspective display screen to form an augmented reality image overlaid on a real-world object that can be viewed through the perspective display screen. The surgical robot 4 can prompt medical staff to select functions, modes, and / or evaluate the situation of the surgical system 2. Pressing the secondary button once can activate certain functions, modes, and / or confirm information transmitted to medical staff through one or more AR headsets 920, displays 34, and / or light indicators 28. In addition, pressing the secondary button multiple times in rapid succession can activate additional functions, modes, and / or select information transmitted to medical staff through one or more AR headsets 920, displays 34, and / or light indicators 28.

[0065] Further references Figure 8 , the electrical components of the surgical robot 4 include a platform subsystem 802, a computer subsystem 820, a motion control subsystem 840, and a tracking subsystem 830. The platform subsystem 802 includes a battery 806, a power distribution module 804, a connector panel 808, and a charging station 810. The computer subsystem 820 includes a computer 822, a display 824, and a speaker 826. The motion control subsystem 840 includes a motor drive 842, motors 850, 851, 852, 853, 854, stabilizers 855, 856, 857, 858, an end effector connector 844, and a controller 846. The tracking subsystem 830 includes an orientation sensor 832 and a camera converter 834. The surgical robot 4 may also include a removable foot pedal 880 and a removable tablet computer 890.

[0066] The input power is supplied to the surgical robot 4 through a power supply that can be provided to the power distribution module 804. The power distribution module 804 receives the input power and is configured to generate different power supply voltages provided to other modules, components and subsystems of the surgical robot 4. The power distribution module 804 can be configured to provide different voltage supplies to the connector panel 808, which can be provided to other components (such as a computer 822, a display 824, a speaker 826, a motor driver 842) to, for example, power the motors 850-854 and the end effector connector 844, and to the camera converter 834 and other components for the surgical robot 4. The power distribution module 804 can also be connected to a battery 806, which acts as a temporary power source when the power distribution module 804 does not receive power from the input power source. At other times, the power distribution module 804 can be used to charge the battery 806.

[0067] The connector panel 808 can be used to connect different devices and components to the surgical robot 4 and / or associated components and modules. The connector panel 808 may contain one or more ports for receiving lines or connectors from different components. For example, the connector panel 808 may have a grounding port for grounding the surgical robot 4 to other equipment, a port for connecting a foot pedal 880, and a port for connecting to a tracking subsystem 830, which may include an orientation sensor 832, a camera converter 834, and a DRA tracking camera 870. The connector panel 808 may also include other ports to allow USB, Ethernet, HDMI communication with other components (such as a computer 822). According to some embodiments, the connector panel 808 may include a wired and / or wireless interface for operably connecting one or more AR headsets 920 to the tracking subsystem 830 and / or the computer subsystem 820.

[0068] The control panel 816 may provide various buttons or indicators for controlling the operation of the surgical robot 4 and / or providing information from the surgical robot 4 for the operator to observe. For example, the control panel 816 may include buttons for turning the surgical robot 4 on or off, raising or lowering the robot telescopic support 16, and raising or lowering the stabilizer 855-858, which may be designed to engage the casters 12 to lock the surgical robot 4 without physically moving. Other buttons may stop the surgical robot 4 in the event of an emergency, which may remove all motor power and apply mechanical brakes to stop all movement from occurring. The control panel 816 may also have indicators (such as a line power indicator) that notify the operator of certain system conditions or the charge status of the battery 806. According to some embodiments, one or more AR headsets 920 may communicate, for example, via the connector panel 808 to control the operation of the surgical robot 4 and / or receive and display information generated by the surgical robot 4 for observation by a person wearing the AR headset 920.

[0069] The computer 822 of the computer subsystem 820 contains an operating system and software for operating the designated functions of the surgical robot 4. The computer 822 can receive and process information from other components (e.g., the tracking subsystem 830, the platform subsystem 802, and / or the motion control subsystem 840) to display information to the operator. Further, the computer subsystem 820 can provide output to the operator through a speaker 826. The speaker can be part of the surgical robot, part of the AR headset 920, or in another component of the surgical system 2. The display 824 can correspond to Figure 1 and 2 A display 34 is shown.

[0070] Tracking subsystem 830 may include position sensor 832 and camera transducer 834. Tracking subsystem 830 may correspond to camera tracking system 6 of FIG. 3. DRA tracking camera 870 operates with position sensor 832 to determine the pose of DRA 52. Such tracking may be performed in a manner consistent with the present disclosure, including using infrared or visible light techniques, such as LEDs or reflective markers, to track the position of active or passive elements of DRA 52, respectively.

[0071] The functional operations of the tracking subsystem 830 and the computer subsystem 820 may be included in the Figure 3A and 3B The computer platform 910 may be transported by the camera tracking system 6'. The tracking subsystem 830 may be configured to determine a pose, such as the position and angular orientation of the tracked DRA. The computer platform 910 may also include a navigation controller configured to use the determined pose to provide navigation information to the user to guide them in moving the tracked tool relative to the position-registered patient image and / or the tracked anatomical structure during a planned surgical procedure. The computer platform 910 may be in a Figure 3B and 3C The computer platform 910 can be configured to communicate with the computer subsystem 820 and the computer subsystem 820 when used with a surgical robot. Figure 8 The computer platform 910 may communicate with other subsystems of the computer system 910 to control movement of the end effector 26. For example, as will be explained below, the computer platform 910 may generate a graphical representation of a patient's anatomy, surgical tools, a user's hand, etc., having a displayed size, shape, color, and / or posture that is controlled based on the determined posture of one or more tracked DRAs, and the displayed graphical representation may be dynamically modified to track changes in the determined posture over time.

[0072] The motion control subsystem 840 can be configured to physically move the robot telescopic support 16, the upper arm 18, the lower arm 20, or rotate the end effector coupling 22. The physical movement can be performed by using one or more motors 850-854. For example, the motor 850 can be configured to vertically lift or lower the robot telescopic support 16. Figure 2 As shown, the motor 851 can be configured to move the upper arm 18 laterally about the joint with the robotic telescopic support 16. Figure 2 As shown, motor 852 may be configured to move lower arm 20 laterally about a joint with upper arm 18. Motors 853 and 854 may be configured to move end effector coupling 22 to provide translational movement along and rotational movement about three-dimensional axes. Fig. 9The illustrated computer platform 910 can provide control inputs to the controller 846 that direct the movement of the end effector coupling 22 to position the passive end effector connected thereto in a planned pose (i.e., position and angular orientation relative to defined 3D orthogonal reference axes) relative to the anatomical structure to be operated on during a planned surgical procedure. The motion control subsystem 840 can be configured to measure the orientation of the end effector coupling 22 and / or the end effector 26 using an integrated orientation sensor (e.g., an encoder).

[0073] Fig. 9 A block diagram of components of a surgical system according to some embodiments of the present disclosure is shown, the surgical system including a device connected to a camera tracking system 6 ( Figure 3A ) or 6'( Figure 3B , 3C ) and / or an imaging device (e.g., C-arm 104, O-arm 106, etc.) of a computer platform 910 of the surgical robot 4. Alternatively, at least some of the operations performed by the computer platform 910 disclosed herein may additionally or alternatively be performed by components of the surgical system.

[0074] refer to Fig. 9 , the computer platform 910 includes a display 912, at least one processor circuit 914 (also referred to as a processor for brevity), at least one memory circuit 916 (also referred to as a memory for brevity) containing computer readable program code 918, and at least one network interface 902 (also referred to as a network interface for brevity). The display 912 may be part of an AR headset 920 according to some embodiments of the present disclosure. The network interface 902 may be configured to connect to Fig.10 The C-arm imaging device 104, Fig.11 The O-arm imaging device 106, another medical imaging device, an image database 950 containing medical images of patients, components of the surgical robot 4 and / or other electronic equipment.

[0075] When used with the surgical robot 4, the display 912 may correspond to Figure 2 The display 34 and / or Figure 8 The tablet computer 890 and / or the AR headset 920 operably connected to the surgical robot 4, the network interface 902 may correspond to Figure 8 The platform network interface 812, and the processor 914 may correspond to Figure 8computer 822. The network interface 902 of the AR headset 920 can be configured to communicate according to one or more wireless communication protocols (e.g., WLAN, 3GPP 4G and / or 5G (New Radio) cellular communication standards, etc.) through a wired network (e.g., thin-wire Ethernet) and / or through a wireless RF transceiver link.

[0076] The processor 914 may include one or more data processing circuits, such as general and / or special purpose processors, such as microprocessors and / or digital signal processors. The processor 914 is configured to execute computer readable program code 918 in the memory 916 to perform operations, which may include some or all of the operations described herein as being performed for surgical planning, guided surgery, and / or robotic surgery.

[0077] The computer platform 910 can be configured to provide surgical planning functionality. The processor 914 can be operated to display images of anatomical structures (e.g., vertebrae) received from one of the imaging devices 104 and 106 and / or from the image database 950 via the network interface 902 on the display 912 and / or on the AR headset 920. The processor 914 receives an operator's definition of the location of the anatomical structure shown in one or more images where a surgical procedure (e.g., screw placement) is to be performed, such as by the operator touching the display 912 to select the location for the planned surgical procedure or using a mouse-based cursor to define the location for the planned surgical procedure. As will be explained in further detail below, when the image is displayed in the AR headset 920, the AR headset can be configured to sense gesture-based commands formed by the wearer and / or sense voice-based commands spoken by the wearer, which can be used to control selection between menu items and / or control how objects are displayed on the AR headset 920.

[0078] The computer platform 910 can be configured to be able to perform anatomical measurements that may be particularly useful for knee surgery, similar to the measurement of various angles to determine the center of the hip, the center of the angle, natural landmarks (e.g., the transepicondylar line, the white side line, the posterior condylar line, etc.). Some measurements can be automatic, while some other measurements can involve human input or assistance. The computer platform 910 can be configured to allow the operator to input the selection of the correct implant for the patient, including the selection of size and alignment. The computer platform 910 can be configured to perform automatic or semi-automatic (involving human input) segmentation (image processing) of CT images or other medical images. The patient's surgical plan can be stored in a cloud-based server that can correspond to the database 950 for retrieval by the surgical robot 4.

[0079] For example, during an orthopedic surgery, a surgeon can select a site to be cut (e.g., posterior femur, proximal tibia, etc.) using a computer screen (e.g., touch screen) or augmented reality interaction (e.g., gesture-based commands and / or voice-based commands) through, for example, an AR headset 920. The computer platform 910 can generate navigation information that provides visual guidance to the surgeon to perform the surgical procedure. When used with the surgical robot 4, the computer platform 910 can provide guidance that allows the surgical robot 4 to automatically move the end effector 26 to a target pose so that the surgical tool is aligned with the target position to perform a surgical procedure on the anatomical structure.

[0080] In some embodiments, surgical system 900 can use two DRAs to track patient anatomical orientation, such as a DRA connected to the patient's tibia and a DRA connected to the patient's femur. System 900 can use standard navigation instruments for registration and inspection (e.g., a pointer similar to the pointer used in the Globus Excelsius GPS system for spinal surgery).

[0081] A particularly challenging task in guided surgery is how to plan the position of implants in the spine, knee, and other anatomical structures, where the surgeon strives to perform the task on a computer screen, which is a 2D representation of the 3D anatomical structure. The system 900 can solve this problem by using an AR headset 920 to display three-dimensional (3D) computer-generated representations of anatomical structures and candidate implant devices. Under the guidance of the computer platform 910, the computer-generated representations are scaled and posed relative to each other on the display screen and can be manipulated by the surgeon while viewing through the AR headset 920. The surgeon can, for example, use gesture-based commands and / or voice-based commands sensed by the AR headset 920 to manipulate the displayed computer-generated representations of anatomical structures, implants, surgical tools, etc.

[0082] For example, the surgeon can view the displayed virtual handle on the virtual implant, and can manipulate (e.g., grasp and move) the virtual handle to move the virtual implant to the desired posture and adjust the planned implant placement relative to the graphical representation of the anatomical structure. Afterwards, during the surgical operation, the computer platform 910 can display navigation information through the AR headset 920 that helps the surgeon to more accurately follow the surgical plan to insert the implant and / or perform another surgical procedure on the anatomical structure. When the surgical procedure involves bone removal, the progress of bone removal (e.g., cutting depth) can be displayed in real time through the AR headset 920. Other features that can be displayed by the AR headset 920 may include but are not limited to gaps or ligament balance along the range of joint motion, contact lines along the range of joint motion on the implant, ligament tension and / or laxity rendered by color or other graphics, etc.

[0083] In some embodiments, the computer platform 910 can allow planning for the use of standard surgical tools and / or implants, such as posterior-stabilized implants and cruciate-retaining implants, cemented and cementless implants, and revision systems for surgical procedures associated with, for example, total or partial knee and / or hip replacements and / or trauma.

[0084] The automated imaging system may be used in conjunction with the computer platform 910 to acquire pre-operative, intra-operative, post-operative, and / or real-time image data of anatomical structures. Fig.10 and 11 An example automatic imaging system is shown. In some embodiments, the automatic imaging system is a C-arm 104 ( Fig.10 ) imaging device or 106( Fig.11(Medtronic Navigation, Inc., with a place of business in Louisville, Colo., USA, owns Copyright) may wish to x-ray a patient from many different positions without the need for frequent manual repositioning of the patient, as may be required in an x-ray system. C-arm 104 x-ray diagnostic equipment can solve the problem of frequent manual repositioning and is well known in the medical field for surgical and other interventional procedures. Fig.10 As shown, the C-arm includes an elongated C-shaped member terminating in a "C" shaped opposite distal end 112. The C-shaped member is attached to an x-ray source 114 and an image receptor 116. The space within the C-arm 104 of the arm provides space for a physician to care for a patient substantially uninterrupted by the x-ray support structure.

[0085] The C-arm is mounted so that the arm is capable of rotational movement in two degrees of freedom (i.e., about two perpendicular axes in a spherical motion). The C-arm is slidably mounted to the x-ray support structure, which allows for orbital rotational movement of the C-arm about its center of curvature, which can allow for selective vertical and / or horizontal orientation of the x-ray source 114 and image receptor 116. The C-arm may also be rotatable laterally (i.e., in a perpendicular direction relative to the direction of orbital motion to enable selective adjustment of the positioning of the x-ray source 114 and image receptor 116 relative to the width and length of the patient). The spherical rotational aspect of the C-arm device allows the physician to x-ray the patient at an optimal angle determined relative to the specific anatomy being imaged.

[0086] Fig.11 Shown in 106 includes a gantry housing 124, which can enclose an image capture portion that is not shown. The image capture portion includes an x-ray source portion and / or an x-ray emitting portion and an x-ray receiving portion and / or an image receiving portion, which can be positioned approximately one hundred and eighty degrees from each other and mounted on a rotor (not shown) relative to the track of the image capture portion. The image capture portion can be operatively rotated three hundred and sixty degrees during image acquisition. The image capture portion can rotate about a center point or axis, thereby allowing image data of the patient to be acquired from multiple directions or in multiple planes.

[0087] With stand housing 124 The gantry housing 106 has a central opening for positioning around the object to be imaged, a radiation source that can be rotated around the interior of the gantry housing 124, and the radiation source can be adapted to project radiation from a plurality of different projection angles. The detector system is adapted to detect the radiation at each projection angle, thereby acquiring an image of the object from a plurality of projection planes in a quasi-simultaneous manner. The gantry can be attached to the support structure in a cantilevered manner. A support structure, such as a wheeled mobile cart with wheels. The positioning unit preferably translates and / or tilts the gantry to a planned position and orientation under the control of a computerized motion control system. The gantry may include a source and a detector disposed relative to each other on the gantry. The source and the detector may be fixed to a motorized rotor that can rotate the source and the detector in combination with each other around the interior of the gantry. The source can be pulsed in multiple positions and orientations in a partial and / or full three hundred and sixty degree rotation to perform multi-planar imaging of a target object positioned inside the gantry. The gantry may further include a track and bearing system for guiding the rotor as it rotates, the track and bearing system can carry the source and the detector. Both and / or one of the surgical system 106 and the C-arm 104 may be used as an automated imaging system to scan the patient and send the information to the surgical system 2 .

[0088] The images captured by the imaging system may be displayed on the AR headset 920 and / or another display device of the computer platform 910, the surgical robot 4, and / or another component of the surgical system 900. The AR headset 920 may be connected to one or more of the imaging devices 104 and / or 106 and / or the image database 950, for example, via the computer platform 910, to display images therefrom. The user may provide control input (e.g., gesture-based and / or voice-based commands) through the AR headset 920 to control the operation of one or more of the imaging devices 104 and / or 106 and / or the image database 950.

[0089] Fig.12 A block diagram of components of a surgical system including a pair of AR headsets 1200 and 1210 (head mounted displays HMD1 and HMD2) is shown, which may correspond to Fig.13 The AR headset 920 shown in FIG. 1 and operates according to some embodiments of the present disclosure.

[0090] refer to Fig.12In an example scenario of FIG. 6 , both assistant 612 and surgeon 610 wear AR headsets 1200 and 1210, respectively. It is optional for assistant 612 to wear AR headset 1210. As will be further described below, AR headsets 1200 and 1210 are configured to provide an interactive environment through which the wearer can view and interact with information related to the surgical procedure. This interactive AR-based environment can eliminate the need for the presence of technician 614 in the operating room and can eliminate the need for the use of Figure 6 The display 34 shown is required. Each AR headset 1200 and 1210 may include one or more cameras configured to provide an additional source of tracking DRA or other reference arrays attached to instruments, anatomical structures, end effectors 26, and / or other devices. Fig.12 In the example of , the AR headset 1200 has a field of view (FOV) 1202 for tracking the DRA and other objects, the AR headset 1210 has a FOV 1212 that partially overlaps with the FOV 1202 for tracking the DRA and other objects, and the navigation camera 46 has another FOV 600 that partially overlaps with the FOVs 1202 and 1212 for tracking the DRA and other objects.

[0091] If one or more cameras are obstructed from viewing a DRA attached to a tracked object (e.g., a surgical instrument), but the DRA is in view of one or more other cameras, the tracking subsystem 830 and / or navigation controller 828 can continue to track the object seamlessly without losing navigation. Additionally, if the DRA is partially obscured from the perspective of one camera, but the entire DRA is visible through multiple camera sources, the tracking inputs of the cameras can be merged to continue navigating the DRA. One of the AR headsets and / or the navigation camera 46 can view and track the DRA on another of the AR headsets to enable the computer platform 910 ( Fig. 9 and 14 ), the tracking subsystem 830 and / or another computing component can determine the pose of the AR headset.

[0092] AR headsets 1200 and 1210 may be operably connected to view video, pictures, and / or other information received from a surgical operating room and / or provide commands to control various equipment in the operating room, including but not limited to neuromonitoring, microscopes, cameras, and anesthesia systems. Data from the various devices may be processed and displayed within the headset, such as displaying a patient's vital signs or a microscope feed.

[0093] Example AR headset components and integration with surgical robots and other devices

[0094] Fig.13 An AR headset 920 configured in accordance with some embodiments of the present disclosure is shown. The AR headset includes a headband 1306 configured to secure the AR headset to a wearer's head, and includes a display screen 1302 and an electronics housing 1304 supported by the headband 1306. The display screen 1302 may be a see-through LCD display device or a semi-reflective lens that reflects an image projected by the display device toward the wearer's eyes.

[0095] Fig.14 Electrical components of an AR headset 920 according to some embodiments of the present disclosure are shown, and the AR headset can be operably connected to a computer platform 910, one or more of the imaging devices (such as a C-arm imaging device 104, an O-arm imaging device 106) and / or an image database 950 and / or a surgical robot 800.

[0096] The AR headset 920 provides an improved human-machine interface for performing guided surgical procedures. The AR headset 920 can be configured, for example, through the computer platform 910 to provide functionality including, but not limited to, any one or more of the following: identifying gesture-based commands and / or voice-based commands; displaying AR graphical objects as overlays that can be anchored to specific real-world objects viewed through the display screen 1450 and associated optical devices; displaying images in various defined display areas; displaying video feeds from cameras and other cameras mounted to one or more AR headsets 920; and controlling surgical lights (built-in or otherwise connectable), etc.

[0097] The electrical components of the AR headset 920 may include one or more cameras 1440, a microphone 1442, a gesture sensor 1444, a posture sensor (e.g., an inertial measurement unit (IMU)) 1446, a display module 1448, and a display screen 1450 configured to display an image for viewing by the wearer and allow at least some ambient light to pass through it for viewing by the wearer. The one or more cameras 1440 may be configured to operate as a gesture sensor 1444 by enabling identification of a user gesture performed within the field of view of the one or more cameras 1440. Alternatively, the gesture sensor 1444 may be a proximity sensor and / or a touch sensor that senses a gesture performed in proximity to the gesture sensor 1444 and / or senses physical contact (e.g., tapping the sensor or housing 1304). The posture sensor 1446 (e.g., an IMU) may include a multi-axis accelerometer, a tilt sensor, and / or another sensor that can sense rotation and / or acceleration of the AR headset 920 along one or more defined coordinate axes. Some or all of these electrical components may be contained in component housing 1304 or may be contained in another housing configured to be worn elsewhere, such as on the hip or shoulder.

[0098] As explained above, the surgical system 2 includes a tracking system (e.g., Fig.12 6 / 6' and Figure 8 The tracking subsystem 830 in the embodiment of the present invention and / or the computer platform 910 and the surgical robot 4. The tracking system is configured to determine the pose of the anatomical structure and the pose of the end effector and / or the surgical tool. The surgical robot 4 includes a surgical plan (e.g., based on a surgical plan defined by the surgeon) that defines the position of the anatomical structure to be subjected to the surgical procedure using the surgical tool. Fig. 9 The surgical planning function performed by the computer platform 910 of the present invention) and one or more navigation controllers (e.g., Fig.14 828 in). The one or more navigation controllers are further configured to generate steering information based on the target posture of the surgical tool, the posture of the anatomical structure, and the posture of the surgical tool and / or the end effector, wherein the steering information indicates a position where the surgical tool and / or the end effector needs to be moved under the control of at least one motor.

[0099] The electrical components of the AR headset 920 can be operably connected to the electrical components of the computer platform 910 through the wired / wireless interface 1452. The electrical components of the AR headset 920 can be operably connected, for example, through the computer platform 910 or directly connected through the wired / wireless interface 1452 to various imaging devices (e.g., C-arm imaging device 104, I / O-arm imaging device 106, image database 950) and / or other medical equipment.

[0100] The surgical system 2 further includes at least one AR headset controller 1430 (also referred to as an “AR headset controller” for brevity) that may reside in the AR headset 920 and / or in the computer platform 910. The AR headset controller 1430 is configured to receive navigation information from the navigation controller 828 that provides guidance to the user during a surgical procedure performed on an anatomical structure, and is configured to generate an AR image based on the navigation information for display on the perspective display screen 1450.

[0101] The see-through display screen (also referred to as the "display screen") 1450 is configured to display objects through an optical system in such a way that when the wearer looks through it, the objects appear to be displayed in the real world. The optical system can be positioned by a headband 1306 placed directly in front of the wearer's eyes. Various functions are provided by software executed by the AR headset controller 1430, which can reside in the AR headset 920 and / or in an external component connected by a wired cable and / or a wireless communication link.

[0102] The AR headset controller 1430 may be within a housing configured to be worn on the user's head or other part of the user's body while viewing the display screen 1450, or may be positioned away from the user viewing the display screen 1450 while communicatively connected to the display screen 1450. The AR headset controller 1430 may be configured to operably process signaling from the camera 1440, microphone 142, and / or gesture sensor 1446, and connect, for example, through the display module 1448 to display AR images on the display screen 1450. Thus, the AR headset controller 1430 shown as a box within the AR headset 920 should be understood to be operably connected to the other shown components of the AR headset 920, but not necessarily reside within a common housing or be transportable by the user. For example, the AR headset controller 1430 may reside within the computer platform 910, which in turn may reside within Figure 3B and 3C The computer tracking system 6' is shown within the housing.

[0103] Example wearer viewing through an AR headset

[0104] Figure 15-25 An example view provided to a wearer via the display screen 1450 of the AR headset 920 is shown according to some embodiments of the present disclosure.

[0105] One potential drawback to using an AR headset in an operating room is that bright ambient lighting may "wash out" or reduce the contrast of displayed objects. These potential drawbacks may be reduced or eliminated by some of the embodiments described below in which display screen 1450 is configured with horizontal bands of varying light transmittance (opacity) and / or in which variable-sized attachable components adjust the opacity of a user's view through display screen 1450. These darkened bands of the optical system may serve to increase the contrast of displayed objects, for example enabling a surgeon to more accurately view medical radiographs.

[0106] In one illustrative embodiment, the see-through display screen 1450 is configured with at least two laterally extending bands, wherein an upper band of the at least two laterally extending bands has a lower light transmittance, e.g., is darker (more opaque), than a lower band of the at least two laterally extending bands, the two laterally extending bands limiting incident ambient light passing therethrough for viewing by the user. The at least one AR headset controller 1430 may be configured to display a two-dimensional image in the upper band of the at least two laterally extending bands and to display a three-dimensional model of an anatomical structure provided by the at least one navigation controller 1430.

[0107] The AR headset user interface further enables the prominent display of sterile software controls and information features to address the limitations of current navigation technology. Additionally, as described below, an operating function is provided that enables the wearer to use hands, surgical instruments and / or other objects to virtually interact with navigation elements.

[0108] The AR headset operation can display both 2D images and 3D models in different bands. The 2D images can be preferably displayed in the more opaque band (the upper band of the display screen optics), and the 3D models can be more preferably displayed in the more transparent band otherwise referred to as the ambient area (the bottom band of the display screen optics). Below this band where the display screen optics ends, the wearer can have an unobstructed view of the patient. Note that the separate bands are fluid. The 3D content may move to the opaque band depending on the orientation of the headset relative to the content, and the 2D content can be placed in the transparent band and stabilized relative to the real world. Additionally, the entire display screen optics can be darkened to convert the headset into a virtual reality for surgical planning or completely transparent during operation. Any control can be found in any band depending on the wearer's preference.

[0109] Fig.15 Two laterally extending bands across display screen 1450 are shown divided along dashed line 1500. Although Fig.15, but it should be understood that the upper band of the at least two laterally extending bands has a lower light transmittance (e.g., a darker shade) than the lower band of the at least two laterally extending bands to reduce the amount of incident ambient light that passes through for viewing by the user. The at least one AR headset controller 1430 has displayed multiple two-dimensional images in screen areas 1510 and 1512 within the upper band of the at least two laterally extending bands. The at least one AR headset controller 1430 has also displayed a three-dimensional model of the anatomical structure in another screen area 1520 in the lower band of the at least two laterally extending bands, and displayed implant information 1530 identifying the characteristics of a planned implant to be used in the surgical procedure. The displayed implant information 1530 identifies that a CEO threaded screw with a diameter of 5.5 mm and a length of 43 mm is planned to be implanted in spinal position L2-L.

[0110] Figure 16-19 Example operations of a wearer interacting with various objects displayed on the display screen 1450 of the AR headset 920 are shown.

[0111] The AR headset 920 may be configured to provide hand tracking that enables the wearer to interact naturally and in a sterile manner with the software functions of the surgical system 2. These operations may be provided by processing video streams from one or more cameras 1440 of the AR headset 920. The AR headset controller 1430 may be configured to identify and respond to a number of different hand-based gestures, including but not limited to pinching, pushing, hand signals, predefined movements (hand flips).

[0112] The wearer can access different menus and submenus and perform specific functions, such as menu selection, menu manipulation / repositioning, image manipulation / repositioning, and virtual projection manipulation / repositioning. An example recognizable gesture may be a pinch, which may be formed by connecting the thumb and index finger and lifting the other three fingers (e.g., Fig.16 1440). Specifying specific gestures can reduce the chance of false positives when interacting with gesture recognition software, especially in an operating room where the user occasionally needs to keep their hands in the field of view of one or more cameras 1440. Gestures can be performed with one hand or two hands. The hand tracking function can display different hand models (glove, grid, bat model) on top of the hands to the surgeon.

[0113] In one embodiment, gesture sensor 1444, which may include one or more cameras 1440, is configured to output an indication of a gesture formed by the user to AR headset controller 1430. AR headset controller 1430 is configured to control graphical objects displayed on display screen 1450 in response to recognition of the gesture.

[0114] As explained above, the gesture sensor 1444 may include one or more cameras 1440 mounted facing away from the user to output a video stream of the user's hands when within the field of view of the one or more cameras 1440 when positioned in front of the AR headset 920. The AR headset controller 1430 may be configured to associate different recognizable gestures with different ones of a plurality of operations for controlling graphical objects displayed on the display screen 1450. The plurality of operations may include at least one of: a user command for expanding a menu item displayed on the display screen into a list of sub-menu items that are then displayed on the display screen; a user command for selecting a menu item and / or one of a plurality of sub-menu items displayed on the display screen; and a user command for operable control of the size, position, and / or orientation of one of the graphical objects displayed on the display screen.

[0115] The AR headset controller 1430 may be configured to display a computer-generated hand model on the display screen, with the hand model having a dynamically updated pose to track the current pose of the user's hand identified in the video stream from the camera.

[0116] In other embodiments, the AR headset controller 1430 is configured to associate the recognizable gesture with a command for selecting a surgical implant from a set of surgical implants that can be used for implantation into the anatomical structure, and display a graphical representation of the selected surgical implant on the display screen 1450. In another embodiment, the AR headset controller 1430 is configured to associate different recognizable gestures with different operations of a plurality of operations performed by the surgical system, wherein the plurality of operations include at least one of: a command for controlling imaging of the anatomical structure by the imaging system; a command for controlling which of a plurality of images of the anatomical structure is displayed on the display screen; and a command for controlling movement of a robotic arm of a surgical robot relative to the anatomical structure.

[0117] exist Fig.16 , the AR headset controller 1430 has displayed multiple images in screen areas 1620, 1630, and 1640. In screen area 1620, the AR headset controller 1430 has rendered a computer-generated hand model 1622 of the user's hand forming a pinch gesture, which is displayed as a graphical overlay on a video of a real-world object 1610 (e.g., an anatomical structure residing on a table) and viewed through one or more cameras 1440. The recognized pension gesture can be used to select in a user-selectable marker display along the navigation bar 1600 under the control position, size, and / or other characteristics of the screen areas 1620, 1630, and 1640.

[0118] Typically, the surgeon also wears a helmet with an attached surgical headlight to improve patient visualization. In one variation, the headlight can be part of or connected to the AR headset 920. The AR headset 920 can be configured to control the brightness setting of the light through external control or through AR control that includes gesture-based command recognition and / or voice-based command recognition.

[0119] The AR headset wearer may change characteristics of the surgical tools and / or implanted devices using gesture-based commands and / or voice-based commands that may be recognized by the AR headset controller 1430. For example, the surgeon may say "change screw length to 6.5" to cause the screw length displayed on the display screen 1450 to change to indicate a size, shape, and / or pose corresponding to the requested screw length.

[0120] Enhanced navigation operation

[0121] The AR headset user interface further enables the prominent display of sterile software controls and information features to address the limitations of current navigation technology. Additionally, as described below, an operating function is provided that enables the wearer to virtually interact with the navigation elements using hands and surgical instruments.

[0122] Fig.17 Depicted is what the wearer can see through the display screen 1450. In the upper band (which may be more opaque than the lower band), the software displays images for navigation, the standard being axial and sagittal views for spinal navigation, but any and multiple 2D planes can be shown, such as the coronal plane ( Fig.16 1710). These images can be modified based on the needs of different procedures (such as arthroplasty). The images can be zoomed in and panned using gestures. Additional views can be accessed through a displayed drop-down list 1710 using gestures. The wearer can select a digital loupe view from the list 1710 as well as an image gallery. In this gallery, the wearer can view images and manipulate images associated with the patient. For three-dimensional (3D) images such as MRI / CT scans, the wearer can scroll through slices using gestures. By selecting a two-dimensional (2D) tab, a 2D image appears / disappears. By selecting a 3D tab, a 3D reconstructed model appears / disappears.

[0123] exist Fig.17 In the example of FIG. 1 , controller 1430 has displayed multiple images in screen areas 1700, 1702, and 1704. A user is using a pinch gesture to select a menu item (e.g., a coronal image view) from multiple items 1710 displayed on display screen 1450 (a list of available image views including standard, coronal, loupe, and gallery).

[0124] refer to Fig.18 The AR headset operation also allows the wearer to modify image properties within the display area 1800, such as window / level and slice type (orientation). The wearer can Fig.18 The selectable settings mark 1810 shown in the example view of can be used to perform these modifications.

[0125] refer to Fig.19 , when 3D is enabled, a 3D graphical representation of the spine (which is based on the imaging scan) is obtained and displayed on a display screen 1450 above the patient. This graphical representation can be configured by the AR headset controller 1430 to be the same size and oriented to have the same pose as the patient's actual anatomy, or can be configured to a larger scale (such as 2:1) to enhance visualization. The graphical representation can be anchored to a position determined based on a specific navigation reference array (in this case, the patient reference array). This anchoring or field / environment stabilization can be performed through a combination of both internal tracking and outside-in tracking.

[0126] The IMU 1446 of the AR headset 920 can be configured to measure the relative orientation of the AR headset 920 (inside-out tracking). The navigation camera 46 of the camera tracking system 6 can be used to track a reference array on the AR headset 920 to measure pose, orientation, and angular orientation (outside-in tracking). As the wearer's head moves, this data can be used to update the projection of the display 3D graphical representation, thereby providing the wearer with the perception that the graphical representation has remained stationary. This type of effect can also be achieved through simultaneous localization and mapping (SLAM) technology.

[0127] In one embodiment, the plurality of cameras 1440 are spaced apart and mounted facing away from the user. The tracking subsystem 830 is operably connected to process the video streams from the plurality of cameras 1440 and is configured to determine the pose of the anatomical structure and the pose of the end effector and / or surgical tool. The tracking subsystem 830 may be further configured to process the video streams from the plurality of cameras 1440 to determine the pose of a reference array connected to the anatomical structure and the pose of a reference array connected to the end effector and / or surgical tool.

[0128] In still other embodiments, the tracking subsystem 830 processes data from one or more tracking cameras (e.g., Figure 3A 46 and Figure 8The AR headset controller 1430 may be configured to monitor a video stream of a DRA tracking camera 870 in the AR headset 920 to determine a pose of a reference array connected to the AR headset 920. The AR headset controller 1430 may be configured to control a pose and scale of a three-dimensional model of the anatomical structure displayed on the display screen 1450 based on the pose of the reference array connected to the anatomical structure and the pose of the reference array connected to the AR headset. A pose sensor 1446 of the AR headset 920 may be configured to output a sensed pose of the AR headset 920, and the AR headset controller 1430 may be operably connected to the pose sensor 1446 and may be further configured to control a pose of the three-dimensional model of the anatomical structure displayed on the display screen 1450 based on the pose of the reference array connected to the anatomical structure and the sensed pose of the AR headset 920.

[0129] This allows the wearer's head to move to view from different angles of the anatomy. The position and orientation of the graphical representation relative to the anchor can be moved depending on the wearer's preference and can even be overlaid on the patient's anatomy. Additionally, in this particular example, the graphical representation is displayed as a solid surface, providing enhanced visualization of the surface. In a different variation, the graphical representation can be a grid to allow visualization of the interior of the anatomical structure (e.g., bones). This 3D graphical representation ultimately provides a more intuitive navigation setting for the wearer, rather than burdening the wearer with trying to mentally visualize a 2D image of the 3D anatomical structure.

[0130] refer to Fig. 20 , when the surgical tool 2002 is brought into proximity with the tracked anatomical structure and the camera 1440 ( Fig.14 ) and 46( Figure 6 ), a graphical representation 2000 of the tool may be displayed in 2D and / or 3D images relative to a graphical representation 2010 of the anatomy. The wearer may use the viewed graphical representation to adjust a trajectory 2020, which may be shown extending from the graphical representation 2000 of the tool through the graphical representation 2010 of the anatomy. Such manipulation may be performed by a control command (e.g., a button) displayed or on the instrument. Note that this may be accomplished by an instrument interfaced with the patient or an instrument interfaced with the 3D model itself.

[0131] Once the screw trajectory is set, its size (length, width), orientation and depth can be adjusted using gestures (such as Fig. 20 and 21 to modify the pinch gesture shown). Fig. 22 As shown, the trajectory can also be adjusted in 2D images. Fig. 2222, the wearer has formed a gesture (e.g., pinch) that is interpreted as a command to select a screw of a smaller size type (e.g., smaller diameter and / or length), which is then updated in a graphical representation 2210 of the screw shown as an overlay on the medical image 2200 and / or displayed in another graphical area 2202. Note that in these examples, screws are used as implants, but implants may alternatively include, but are not limited to, spacers, rods, and prostheses.

[0132] refer to Fig.23 Once a trajectory is set, when another instrument 2300 enters the vicinity of the planned trajectory 2020 of the tool 2002 (shown as object 2000), an enhanced trajectory guidance interface is displayed to help the wearer position the instrument 2300 to the same trajectory 2020. Fig.23 In the specific example of , a graphical indicator 2306 indicating the direction to move the instrument 2300 appears until the outer circle 2302 is aligned with the inner circle 2304. Once these circles 2302 and 2304 are aligned, the wearer can then be instructed to push the instrument 2300 downward into the patient's body until the circles 2302 and 2304 are aligned (e.g., become concentric), thereby indicating that the target depth has been reached. There are other possible methods for such trajectory guidance, including but not limited to arrows pointing to the desired trajectory and color gradients. Once the implant is placed, the wearer can select and move to the next step of the operating procedure.

[0133] refer to Fig.24 , the system operation function can enable multiple AR headsets to be used in a defined combined operation mode, the defined combined operation mode comprising any one or more of the following: collaborative mode, teaching mode, and independent mode. In collaborative mode, a secondary wearer wearing a second AR headset has the ability to interface with the display interface of the first AR headset worn by the primary wearer through the second AR headset. Fig.24 As shown, the secondary wearer sees the same 3D model and associated controls stabilized relative to the environment immediately following the primary wearer, with the only difference being that the content is flipped to be recognizable from his / her perspective. The secondary wearer can perform all trajectory modification actions as previously described, which will update the primary wearer's view through the first AR headset. In teaching mode, the primary wearer can change his / her view provided to the second AR headset through the first AR headset for viewing by the secondary user to enable the second AR headset to display images / scans from the secondary wearer's perspective, and vice versa. In independent mode, the views provided by the first and second AR headsets are controlled independently of each other, enabling both wearers to perform independent tours simultaneously.

[0134] For example, in one embodiment, the surgical system 2 includes first and second AR headsets. At least one AR headset controller (one of which may reside in or be connected to the first AR headset) is operably connected to receive a first video stream from a camera of the first AR headset, and is configured to selectively route the first video stream to a display device of the first AR headset and / or a display device of the second AR headset in response to an electronic command obtained from a user wearing the first AR headset. At least one AR headset controller (one of which may reside in or be connected to the second AR headset) is operably connected to receive a second video stream from a camera of the second AR headset, and is configured to selectively route the second video stream to a display device of the first AR headset and / or a display device of the second AR headset in response to an electronic command obtained from a user wearing the second AR headset.

[0135] refer to Fig.25 , at least one AR headset controller can be configured to display notifications and warnings to the wearer within his / her field of view. Such warnings can inform the patient, for example, that a reference array has been moved, that navigation integrity has been lost, that one or more reference arrays cannot be seen, that the possibility of a robot collision is predicted, etc. These notifications can be displayed on the display screen 1450 with or without affecting the operation of the user interface provided by the AR headset 920.

[0136] As explained above, the AR headset 920 and associated operations not only support guided procedures, but can also be performed in conjunction with robotic-assisted procedures. In the latter case, the preoperative plan is loaded from the computer platform 910. The software shows the entire plan to the wearer in 3D with the planned implants. The wearer can select screws or provide adjustments to the surgical plan. Once the plan is finalized, the wearer can move the robot to the planned trajectory through the AR interface, foot pedals, or other controls to guide the placement of additional tools.

[0137] Guided tour of surgical tools using an AR headset

[0138] Reference now Figure 26-31 The example embodiments describe various operations that may be performed by the surgical system 2 to provide navigation assistance for surgical tools via an AR headset during a medical procedure.

[0139] Some embodiments may enable an AR headset wearer to accurately navigate a surgical tool to a target vector location within a reference space using an AR overlaid 3D navigation object (also known as an iron sight). The overlaid 3D navigation object may provide magnified visual feedback to enable the wearer to correct tool depth, angle, and tip orientation relative to the operating navigation trajectory.

[0140] The surgeon can plan a trajectory that defines a specific vector position on the patient's anatomical image, for example, through the surgical procedure planning function of the computer platform 910. The image is localized to a patient static reference array carrying the position of the vector. The patient static reference array is tracked by a camera and compared to another tracked array connected to the surgical tool.

[0141] Figure 26-28 Graphical objects that can be displayed on a display screen of an AR headset to provide a navigation aid for positioning a surgical tool are shown. The graphical objects include elongated objects, lines, circles, text, and other indicia displayed in relative orientation to indicate a planned pose of a surgical tool to a current pose (e.g., orientation and angular orientation). Various embodiments can guide a user to position a surgical tool relative to a planned location on an anatomical structure and guide a user to move a surgical tool along a planned trajectory.

[0142] like Fig.26 As shown, the depiction of the current tool orientation is not properly aligned with the planned orientation in terms of trajectory, causing the distal end of the tool to deviate from the planned orientation on track. The circles on the right side of the image provide the following indications. The small circle 2600 indicates the planned tool tip orientation and is surrounded by a medium-sized circle 2604 indicating the overall tool trajectory and the orientation of the distal end of the tool. The circle 2602 on the right indicates the current orientation of the tool, which shows that the proximal end of the tool is closer to the planned orientation, while the distal end of the tool is off axis or off track.

[0143] Corresponding operations performed by the AR headset controller 1430 may include displaying a graphical representation of a target pose for the surgical tool on the display screen 1450 and displaying a graphical representation of the pose of the surgical tool on the display screen 1450. The AR headset controller 1430 may be configured to determine a trajectory offset distance between a location where a planned trajectory of the target pose for the surgical tool intersects a target location on the anatomical structure and a location where a trajectory of the pose of the surgical tool intersects the target location on the anatomical structure.

[0144] The AR headset controller 1430 may be configured to display each of the graphical representations of the target pose of the surgical tool and the pose of the surgical tool as elongated objects 2610 and 2620 (e.g., rectangular lines) each having a primary axis extending between opposite ends, and to control a spacing between positions at which one end of the elongated objects is displayed on the display screen in response to the trajectory offset distance. For example, Fig.26 As shown, object 2610 may indicate a planned pose of the surgical tool and object 2620 may indicate a current pose of the surgical tool.

[0145] In one embodiment, the AR headset controller 1430 is configured to control a tilt angle between elongated objects 2610 and 2620 displayed on a display screen in response to an angle between a planned trajectory of a target pose of the surgical tool and the trajectory of the pose of the surgical tool.

[0146] In another embodiment, the AR headset controller 1430 can be configured to display each of the graphical representations of the target pose of the surgical tool and the pose of the surgical tool as a circular object 2602-2604, and control the spacing between the centers of the circular objects displayed on the display screen in response to the trajectory offset distance.

[0147] In another embodiment, the AR headset controller 1430 is configured to display a target pose for the surgical tool and a graphical representation of the pose of the surgical tool as objects spaced apart a distance indicating an offset from the tip of the surgical tool to the target location on the anatomical structure.

[0148] Fig. 27 The planned and current positions of the surgical tool are shown with the proximal tip overlapping, while the current posture 2620 of the surgical tool is tilted relative to the planned posture 2610.

[0149] Using the visual guidance information, the wearer moves the tool toward the intended posture, and as the current posture converges toward the intended posture, the convergence causes the displayed circles to converge to become concentric with each other. Fig.28 As shown, as circles overlap, the circles correspondingly become more filled. Fig.28 It is further demonstrated that the planned position and the current position of the proximal tip of the surgical tool overlap, and the current posture of the surgical tool is consistent with the planned posture, so that the current posture of the surgical tool is accurately aligned with the planned posture.

[0150] In some additional or alternative embodiments, the AR headset controller 1430 is configured to display a graphical representation of a target pose of the surgical tool and the pose of the surgical tool relative to three orthogonal axes representing a three-dimensional space. The AR headset controller 1430 controls the interval between the positions displayed along the first of the three orthogonal axes in response to the depth offset distance between the tip of the surgical tool and the target position on the anatomical structure. The AR headset controller 1430 can also control the interval between the positions displayed along the plane formed by the second and third orthogonal axes of the three orthogonal axes in response to the trajectory offset distance between the position where the planned trajectory of the target pose of the surgical tool intersects the target position on the anatomical structure and the position where the trajectory of the pose of the surgical tool intersects the target position on the anatomical structure.

[0151] In addition to general spatial data, this embodiment uses more data from one or more cameras of the tracking reference array to include depth tracking of the surgical tool. This allows for greater navigation accuracy to be provided to the wearer as the surgical tool is moved into a planned pose for a surgical procedure. An example operational flow may include acquiring orientation and angle data from one or more cameras and projecting the data for display via display screen 1450. Fig.29 In the illustrated example of , a planned tool trajectory line 2900 is shown, along with three depth circles 2910, 2912, and 2914. Depth circle 2910 indicates the current depth of a reference array 2930 attached to a surgical tool 2932, relative to depth circle 2912 indicating the current depth of the tip of the surgical tool 2932, and relative to depth circle 2914 indicating the planned maximum insertion depth of the tip of the tool 2932. Offset circles 2920 and 2922 are shown, indicating the offset between the reference array 2930 attached to the surgical tool 2932 and the planned tool trajectory line 2900.

[0152] The goal of the demonstrated navigation operation is to visually magnify the depth, angle, and tool tip orientation corrections so that the wearer can more accurately align the tool to the planned trajectory 2900 and then navigate the tool down to the planned maximum insertion depth within the anatomical structure. The displayed offset circles 2920 and 2922 are dynamically modified to indicate that the offset of the tool tip from the tool relative to the trajectory 2900 is decreasing as the tool is moved closer to the planned trajectory 2900. After the tool becomes sufficiently aligned with the planned trajectory 2900, the wearer then moves the tool down the trajectory 2900 until the depth circles 2912 and 2914 converge at the planned maximum insertion depth.

[0153] Fig.30A graphical representation of the anatomical structure 3000 (eg, a spine) and the current position of the tool 2932 relative to the planned tool trajectory 2900 is shown. Fig.30 Also shown in Fig.29 The offset circles 2920 and 2922 and depth circles 2912 and 2914 are shown to provide navigation guidance to the wearer to move the tool 2932 to align with the planned tool trajectory 2900. The current tool tip orientation is shown to be consistent with the planned tool trajectory 2900, that is, the tip position or lower depth ring can be considered to be in the correct orientation, while the upper ring is shown to be offset.

[0154] Fig.31 The wearer is shown having moved the tool 2932 so that it becomes aligned with the planned tool trajectory 2900 so that Fig.29 A subsequent view of the graphical representation of the anatomical structure 3000 after the offset circles become concentric. The surgeon can then move the tool 2932 downward along the planned tool trajectory 2900 until Fig.29 until the depth circle is aligned.

[0155] The offset circles and / or depth circles may be rendered using color coding, such as changing from red to green when they are aligned with the correct positioning.

[0156] Cross-platform integration of medical devices using one or more AR headsets

[0157] Various other embodiments are directed to providing cross-platform integration of medical devices using one or more AR headsets. These operations can avoid medical personnel feeling the need to remove the AR headset in order to perform certain actions on one or more medical devices and the operating room, such as when navigating an imaging system to capture a desired patient image. Such intermittent removal of an AR headset for interacting with a patient or imaging system interface can cause problems related to sterility and significant undesirable interference between surgical procedure stages, such as when personnel perform operations for recalibrating the AR headset.

[0158] Various embodiments use custom mapped controls executed through an AR headset configured to navigate user interfaces of imaging systems and other medical devices during a surgical procedure. For example, a wearer can interact with the AR headset using gesture commands and / or voice commands to control positioning of an imaging system, plan imaging shots on an imaging system, activate imaging shots, and / or use acquired imaging shots to plan trajectories for one or more medical implants and associated surgical tools.

[0159] It is known that various previously known medical devices, such as various surgical robots and imaging systems, only provide physical user interfaces for manipulation during planning of a surgical procedure and during implementation of the surgical procedure. If a surgeon is wearing an AR headset, the surgeon may feel the need to remove the AR headset in order to travel to the operating room and freely control the physical user interfaces of the various medical devices in the operating room.

[0160] According to various embodiments herein, an AR headset enables a wearer to input information and control the operation of a medical device operably connected (wired / wireless communication connection) to the AR headset through gesture commands, voice commands, and / or other commands that can be sensed by and / or input to the AR headset. The AR headset thereby enables the wearer to focus on the surgical procedure occurring in front of the wearer and maintain sterility and control of the surgical workflow.

[0161] In one embodiment, the AR headset is operably connected to the imaging system and is configured to provide information and / or commands to the imaging system based on identification of gesture commands, voice commands, and / or other commands sensed by and / or input to the AR headset. The navigation controls of the imaging system can be mapped to the virtual space of the AR headset. The AR headset can be used to control the movement of the imaging system relative to the patient into a desired orientation, adjust the imaging lens orientation, select scanning settings (e.g., patient information, scanning type, etc.) of the imaging lens, and initiate imaging lens rotation using virtual controls that may include gestures and / or voice commands executed relative to a virtual menu displayed on the display screen 1450. By using virtual controls, physical contact between the wearer's hands and the imaging system is avoided. To improve safety, a safety stop switch in the form of a foot pedal, such as a fail-safe switch that prevents the imaging system from moving when released, can be included for use by the AR headset wearer.

[0162] The AR headset can be configured to provide an unobstructed view of a typical field of view in front of the wearer, which can be cleared until a menu item is initiated in response to a recognized gesture command, voice command, and / or other command sensed by and / or input to the AR headset. When the C-arm imaging device captures multiple images for a surgical procedure, the images can be stored in an image database 950. Perspective and other views of the procedure that are unique to each imaging system provide the wearer with a variety of ways to ensure the accuracy of planned tool trajectories and the selection of the correct tools and implants for use during the surgical procedure.

[0163] In some other embodiments, the AR headset is operably connected to control the transfer of surgical planning workflow and / or other information (e.g., images) from the computer platform 910 to the surgical robot 4, between two surgical robots 4, and / or between the surgical robot 4 and other medical devices.

[0164] In one embodiment, the AR headset is operably connected to control cross-platform integration between one or more imaging systems (e.g., C-arm imaging device 104 and / or O-arm imaging device 106) and computer platform 910. The wearer can manipulate control menus mapped in the virtual space of the AR headset to allow separate control of the imaging system and computer platform 910. The wearer can navigate through the control menus provided by the AR headset to plan and perform imaging rotations performed by the imaging system, and then quickly perform trajectory planning and tool selection through computer platform 910 using the acquired images.

[0165] In further embodiments, when the line of sight of the AR headset wearer becomes directed toward the surgical robot 4, a graphical indicator may be displayed as an overlay indicating that the wearer may now interface through the AR headset to control various operational functions of the surgical robot 4.

[0166] In another additional embodiment, when the line of sight of the AR headset wearer becomes directed toward, for example, a rod bender apparatus used for a spinal alignment procedure, a graphical indicator may be displayed as an overlay indicating that the wearer may now interface via the AR headset to control various operational functions of the rod bender (such as shaping a rod implant based on current data from the computer platform 910 generated by planning screw placement relative to one or more of the obtained medical images).

[0167] These and other related cross-platform operations that integrate AR headsets with various medical devices can reduce unnecessary downtime between stages of surgical planning and execution of procedures. By allowing regional processing and workflow to transfer information between various devices, multiple devices (more than one of the devices) can work on the same workflow at the same time. These operations can avoid the typical pauses that occur during surgical procedures in the operating room as the surgeon moves sequentially through the various stages to obtain spinal images, use the spinal images to plan screw placement within the spine, and shape the rods used to connect to the spine.

[0168] An example workflow according to this embodiment may include: the wearer uses an AR headset for navigation to guide the C-arm imaging device 104 to the appropriate orientation for imaging the patient. Once in the appropriate orientation, the wearer can use the AR headset display 1450 to find the operating preferences for imaging rotation and perform imaging rotation. The imaging rotation generates a static image, on which the wearer uses the computer platform 910 to plan the screw trajectory. Once the screw trajectory is planned, the wearer passes the surgical plan to the surgical robot 4 to obtain navigation assistance during the placement of the screws into the spine. At the same time, after the trajectory plan is completed, the wearer can send the completed trajectory plan to the rod bending machine device that starts bending the spinal correction rod. When the rod bending machine device uses the completed trajectory plan to bend the spinal correction rod, the wearer can use the AR headset to assist in screw placement through the operation of the surgical robot 4. Once the screws are successfully secured in place in the patient's spine, the wearer can use the virtual model of the spinal correction rod generated by the cross-platform integration between the surgical robot 4 and the rod bender device to ensure the accuracy of securing the rod to the screw trajectory before performing rod placement.

[0169] Thus, in some further embodiments, the AR headset controller is configured to associate a set of defined gestures and / or a set of defined voice commands with a set of control commands for at least one external medical device and identify an occurrence of one of the defined gestures in the camera video stream and / or an occurrence of one of the defined voice commands in the audio stream. The AR headset controller is configured to select a control command from the set of control commands in response to an identified occurrence of one of the defined gestures and / or the defined voice commands and transmit the selected control command to the external medical device.

[0170] In one embodiment, a set of control commands for an external medical device includes commands to control at least one of: movement of the imaging system; adjusting a region in an image to be captured by the imaging system; and initiating capture of an image by the imaging system.

[0171] In one embodiment, a set of control commands for an external medical device includes commands to control at least one of: transmitting a trajectory plan for a spinal correction rod to a rod bender device; initiating bending of the spinal correction rod by the rod bender device; and transmitting operating status information from the rod bender device to an AR headset for display.

[0172] AR headset with opacity filter

[0173] Fig.32An AR headset 920 is shown having a curved lens 3210 extending laterally across and downwardly from a front portion (e.g., a headband) of the AR headset 920. Fig.13 Compared to the AR headset 920 shown, Fig.32 The AR headset 920 further includes an opacity filter 3200 configured as a laterally extending band on the surface of the curved lens 3210 according to some embodiments of the present disclosure.

[0174] exist Fig.32 In an embodiment of the present invention, the curved lens 3210 operates as a see-through display screen (also referred to as a combiner) that reflects light from the display panel toward the user's eyes. The display panel can be positioned between the electronic component housing and the user's head, and is angled to project virtual content toward the curved lens 3210 to reflect toward the user's eyes. The combiner is semi-transparent and semi-reflective, allowing the user to see the reflected virtual content overlaid on the user's view of the real world scene. The opacity of the opacity filter 3200 controls the amount of light from the real world scene that passes through the user's eyes. The high opacity configuration of the opacity filter 3200 causes a high-contrast virtual image to be overlaid on a dim view of the real world scene. The low opacity configuration of the opacity filter 3200 can cause a more blurred virtual image to be overlaid on a clearer view of the real world scene. As will be explained in further detail below, the opacity filter can provide a constant opacity, such as by applying an opaque material on the surface of the lens 3210, or can provide an electrically controllable opacity.

[0175] According to some embodiments, the surgical system includes an AR headset 920 and an AR headset controller, e.g. Fig.14 Controller 1430 or Fig.34 3410 in the AR headset. The AR headset 920 is configured to be worn by a user during a surgical procedure and has a see-through display screen 3210 that is configured to display an AR image and allow at least a portion of a real-world scene to pass through it for viewing by the user. The AR headset 920 also includes an opacity filter that is positioned between at least one of the user's eyes and the real-world scene when the see-through display screen 3210 is viewed by the user. The opacity filter is configured to provide opacity to light from the real-world scene. The AR headset controller is configured to communicate with a navigation controller (e.g., Fig.14One or more controllers 828A, 828B and / or 828C) in the navigation controller communicate to receive navigation information from the navigation controller to provide guidance to the user during a surgical procedure performed on the anatomical structure, and is further configured to generate AR images based on the navigation information for display on the perspective display screen 3210.

[0176] In a further embodiment, the see-through display screen 3210 includes a curved lens extending laterally across and downwardly from a front portion of the AR headset 920, and the opacity filter is located on a surface of the curved lens. In another embodiment, the opacity filter is configured as at least two laterally extending bands, wherein an upper band of the at least two laterally extending bands has a lower light transmittance than a lower band of the at least two laterally extending bands.

[0177] In another embodiment, the opacity filter is configured to have a gradient of opacity that changes more continuously with distance downward from the top portion of the curved lens 3210 (the see-through display screen). The darkest point of the gradient can be located at the top portion of the curved lens 3210 and gradually becomes less transparent further downward on the see-through curved lens 3210 until the opacity filter is transparent or absent on the curved lens 3210. In a further example embodiment, the gradient can change from about 90% opacity to completely transparent at approximately mid-eye level of the curved lens 3210. With the headset properly calibrated and placed, the mid-eye level can correspond to the point at which the user will look straight ahead, and the end of the gradient will be located at the "horizontal" line of the eye. The darker portion of the gradient will allow for clear and unambiguous vision of virtual content and help block the intrusive brightness of overhead operating room lights.

[0178] Using an opacity filter in this manner enables the AR headset 920 to provide virtual reality (VR) functionality along the upper portion of the curved lens 3210 by substantially or completely blocking light from the real world scene and providing AR functionality along the middle or lower portion of the curved lens 3210. This allows the user to have semi-transparent AR where it is needed and allows the use of clear optics for the patient's anatomy during procedures. Configuring the opacity filter 3200 as a gradient rather than a more constant opacity band can allow the wearer to experience a more natural transition between a more VR-type view and a more AR-type view without experiencing abrupt changes in the brightness of the real world scene and depth of field that can otherwise fatigue the eyes, such as during more rapid shifts between upper and lower views.

[0179] The display panel and curved lens 3210 can be configured to provide a wide field of view perspective AR display system. In one example configuration, they provide a user with an 80° diagonal field of view (FOV) and a 55° vertical coverage for viewing virtual content. Other diagonal FOV angles and vertical coverage angles can be provided by display panels of different sizes, lenses of different curvatures, and / or different distances and angular orientations between the display panel and the lens.

[0180] As explained above, the opacity filter is used to change the opacity of the real-world scene viewed through the headset. In one embodiment, the opacity filter and lens are configured to provide three different opacity views, such as: (a) a high opacity upper lens area in which the real-world scene is significantly darkened and the AR image (virtual content) is displayed with high contrast and viewing clarity, such as a VR display; (b) an intermediate lens area with less opacity than the upper lens area and in which more traditional AR is provided; and (c) a lower area in which the lens ends so that the user can look down to get a 100% unobstructed view of the real-world scene.

[0181] Thus, a high opacity (e.g., no less than 80% opacity) upper lens region (e.g., above the horizon) provides high contrast for displaying AR images and where visual clarity of the real world is less important. A lower opacity (e.g., no more than 50% opacity) middle lens region (e.g., below the horizon) provides less contrast for simultaneously viewing AR images overlaid on a real-world scene. The lower region where the lens ends (e.g., 30° below the horizon) provides the user with an unobstructed view of the real-world scene.

[0182] Some additional embodiments relate to the AR headset controller being configured to select among multiple different areas of the see-through display screen to display different types of AR images based on the characteristics of the AR image type and the opacity level of the areas of the opacity filter aligned with different areas of the see-through display screen. For example, more important AR images (virtual content) can be displayed above the horizon, such as in a high-opacity upper lens area to ensure that the AR images can be viewed clearly regardless of bright operating room lighting. Displaying important AR images in higher opacity areas of the opacity filter can be advantageous because the least physical world interaction occurs within the viewed area during surgery. Some examples of AR images (virtual content) that can be configured for display in higher opacity areas of the opacity filter or that can be selected by the AR headset controller for display in the higher opacity areas may include, but are not limited to, any one or more of the following:

[0183] I) 2D axial, sagittal and / or coronal views of the patient’s anatomy;

[0184] 2) an overlay of planned tool and surgical implant positions versus currently tracked tool and surgical implant positions;

[0185] 3) Preoperative image library;

[0186] 4) Video feeds from microscopes and other similar systems or remote video conferencing; and

[0187] 5) Options and configuration settings and buttons.

[0188] A middle lens region with lower opacity than the upper lens region may provide a more traditional AR region for displaying AR content. Some examples of AR images (virtual content) that may be configured for display in the middle lens region of the opacity filter or that may be selected by the AR headset controller for display in the middle lens region may include, but are not limited to, any one or more of the following:

[0189] 1) A floating 3D model of the patient’s anatomy with surgical planning information;

[0190] 2) Real-time tracking of surgical instruments relative to floating patient anatomy;

[0191] 3) Enhanced overlay of patient anatomy with instructions and guidance; and

[0192] 4) Enhanced overlay of surgical equipment.

[0193] The AR headset controller may be configured to determine the location where different types of AR images are displayed. In one embodiment, multiple different types of AR images are displayed sequentially or simultaneously on the perspective display screen. For each of the multiple AR image types, the AR headset controller is configured to select in multiple different areas of the perspective display screen to display the AR image type based on the characteristics of the AR image type and the opacity level of the area of ​​the opacity filter aligned with the different areas of the perspective display screen. For example, the AR headset controller may receive metadata defining the type of AR image to be received in the stream, and may responsively determine the location where it will display the AR image in the stream based on the metadata. The metadata may be encoded by a person to identify a preference for the location where the AR image in the stream should be displayed, or may identify a rule set according to which the AR headset controller may guide the AR headset controller to decide the characteristics of the location where the AR image in the stream should be displayed, such as an indication of the image type (e.g., 2D medical image, 3D graphics model, etc.). The user may enter commands to move the AR image between different areas of the perspective display screen and within the same area.

[0194] like Fig.32 As shown, the opacity filter 3200 can be configured as a laterally extending band. In some embodiments, the AR headset controller is configured to display at least one of the following in an area of ​​the see-through display screen aligned with the laterally extending band of the opacity filter: a 2D axial, sagittal, and / or coronal view image of the patient's anatomy; a planned and / or currently tracked surgical tool pose; a graphical model of the surgical implant location; a video from a medical instrument; and a user-selectable menu item triggering operation to control a medical device. The AR headset controller is further configured to display at least one of the following in another area of ​​the see-through display screen that is not aligned with the laterally extending band of the opacity filter: a 3D graphical model of the anatomical structure and surgical planning information; a 3D graphical model of the surgical instrument; an animated 3D graphical model of the surgical instrument displayed together with the pose of the graphical model relative to the anatomical structure, the animated 3D graphical model being modified to track in real time the measured pose of the surgical instrument relative to the anatomical structure; and a graphical model of the anatomical structure and navigation information from a navigation controller, the navigation information providing visual guidance to the user during a surgical procedure performed on the anatomical structure.

[0195] Note that while an unobstructed view of a prone patient is easily obtained by looking down with both the eyes and the head, an enhanced view of the patient can also be obtained by tilting the head down a little further to look through the intermediate lens area.

[0196] In this way, the AR headset can be configured to provide hybrid functionality and benefits of both VR and AR, thereby crossing the VR-AR continuum in a manner that maximizes the practicality of applications such as real-time interoperable surgery. The AR headset provides options for the degree of contrast to be provided between the displayed AR image (virtual content, such as a medical image) and the real-world scene by selectively displaying the AR image in a high-opacity upper lens area or in a lower-opacity middle lens area. The user can make small head pitch movements to adjust the various areas relative to the real-world scene, for example, to obtain an AR image overlaid on an anatomical structure and alternatively obtain an unobstructed view of the anatomical structure. The AR headset can be configured to identify gestures and / or voice commands that control what type of AR content is displayed at a location on the lens forming a see-through display screen.

[0197] Dynamically control opacity filter and / or display brightness

[0198] As explained above, the opacity of the opacity filter controls the amount of light from the real-world scene that passes through the user's eyes. A high opacity configuration of the opacity filter causes a high-contrast virtual image to be overlaid on a dim view of the real-world scene. A low opacity configuration of the opacity filter can cause a blurrier virtual image to be overlaid on a clearer view of the real-world scene. In a surgical environment, the clarity of the real-world view of the operation or the virtually displayed medical image should not be compromised. Additionally, operating rooms typically contain both very bright lights (i.e., surgical lights for patients) as well as poorly lit areas, resulting in an environment where a fixed opacity cannot take into account different lighting conditions.

[0199] In addition to the overall opacity of the opacity filter, the color of the content in the AR image can significantly affect visibility in different environments. Because the virtual content is overlaid on the real-world scene, any virtual content whose color matches the color of the real-world scene will be more difficult for the user to perceive clearly. For example, a blue virtual menu will be difficult to read in a room with blue walls, but will be easier to read in a room with white walls. In a surgical environment, any issues that reduce clarity or readability should be avoided.

[0200] Some embodiments of the present disclosure are directed to overcoming one or more of these issues by configuring an AR headset controller to electrically control the opacity of an area of ​​an opacity filter based on the brightness and / or color of a real-world scene and / or by configuring the AR headset controller to control the brightness and / or color of an AR image displayed on a see-through display screen based on the brightness and / or color of the real-world scene.

[0201] Fig.33 A block diagram of an opacity filter 3310 configured to have a variable opacity level in response to electrical signaling from an AR headset controller 1430 according to some embodiments of the present disclosure is shown. The opacity filter 3310 can be formed of an electrochromic material configured to have a light transmission property that changes in response to an applied voltage. In another embodiment, the opacity filter 3310 is formed of a liquid crystal device (such as a polymer dispersed liquid crystal device) or other see-through device configured to have a light transmission property that can change in response to an applied voltage and / or current.

[0202] The AR headset controller 1430 may be configured to electronically control the opacity of different regions of the opacity filter 3310 . Fig.33It is shown that the AR headset controller 1430 can be configured to selectively control the opacity of different areas of the opacity filter 3310, such as by providing three laterally extending areas of different opacity 3320, 3322, and 3326. Alternatively or additionally, the AR headset controller 1430 can be compared to control the opacity of a smaller defined area (such as area 3324) that can be positioned to align with a VR object displayed on a see-through display screen to reduce real-world light passing through the object and thereby increase contrast visibility of the object.

[0203] Fig.34 A block diagram of components of a surgical system including an AR headset 920 connected to an AR processing circuit system 3400 that may reside in a computer platform 910, wherein the AR processing circuit system includes an AR headset controller 1430 configured according to some embodiments of the present disclosure. Figure 35-39 A flowchart illustrating operations that may be performed by the AR headset controller 3410 to control how information is displayed on a see-through display screen and / or to control the opacity of an opacity filter in accordance with some embodiments of the present disclosure is shown.

[0204] refer to Fig.34 , the AR headset 920 includes an AR emitter 3430, which is configured to project light for an AR image in response to signaling from an AR headset controller 1430. The AR emitter 3430 transmits light toward a combiner 3440, which is configured to combine the light of an AR image 3432 projected from the AR emitter 3430 with light from a real-world scene 3434 into a combined overlay image 3436 that can be viewed by a user 3450. The combiner 3440 configured in this manner operates as a see-through display screen. An opacity filter 3310 is aligned with the combiner 3440 so that light from the real-world scene 3434 passes through the opacity filter 3310 where the light can be filtered before passing through the combiner 3440 along the line of sight of the user 3450. In the direction of light arriving from the real-world scene, the rear surface of the opacity filter 3310 can be directly on the front surface of the combiner 3440.

[0205] In some embodiments, the opacity filter is directly on the combiner 3440 and provides a fixed amount of shading. The opacity filter may include a shading applied directly to the surface of the combiner 3440 to provide a fixed amount of opacity filtering of the real world scene. For example, one or more laterally extending bands of one or more shading levels may be formed directly on the surface of the combiner 3440. Alternatively, a shading gradient may be formed directly on the surface of the combiner 3440. Shading may be used in place of the electrically controllable opacity filter 3310, or may be used in combination with the electrically controllable opacity filter 3310 to darken the real world scene.

[0206] The surgical system may include a camera configured to output video frames capturing a real-world scene. Fig.34 In an embodiment of the AR headset controller 1430, one or more forward-facing cameras 1440 may be configured to view a real-world scene along the user's line of sight. Alternatively or additionally, camera 3460 may be positioned and configured to sense reflections from one of the user's eyes 3450 in order to capture the real-world scene viewed by the user. Camera 3460 may additionally be used to track where the user is looking, and the AR headset controller 1430 may be configured to adjust the opacity of the opacity filter 3310 and / or the brightness and / or color of the AR image displayed based on signaling indicating where the user is looking.

[0207] Various additional embodiments relate to the AR headset controller 1430 being configured to electrically control the opacity of a region of the opacity filter 3310 based on the brightness and / or color of a real-world scene and / or relate to the AR headset controller 1430 being configured to control the brightness and / or color of an AR image displayed on a see-through display screen (i.e., projected by the AR emitter 3430 toward the combiner 3440) based on the brightness and / or color of the real-world scene.

[0208] refer to Fig.35 In an operational embodiment, the AR headset controller 1430 is configured to 3500 determine the brightness and / or color of a defined area within a video frame, and 3502 control the brightness and / or color of an AR image displayed on the perspective display screen (i.e., projected by the AR emitter 3430 toward the combiner 3440) based on the determined brightness and / or color of the defined area within the video frame.

[0209] The AR headset controller 1430 can be part of an AR processing circuit system 3400, which can be located in a housing configured to be worn on the user's head or other part of the user's body when viewing the display screen 1450, or can be located away from the user viewing the display screen when communicatively connected to the display screen 1450 (such as within a networked server).

[0210] refer to Fig.36 In an operational embodiment of the AR headset controller 1430, the AR headset controller 1430 is configured to 3600 determine the brightness and / or color of a defined area within a video frame, and 3602 control at least one area (e.g., Fig.33 The opacity of band 3320) in.

[0211] refer to Fig.37 In an operational embodiment of the AR headset controller 1430, the AR headset controller 1430 is configured to 3700 determine a pose of the AR headset 920 (e.g., position and / or angular orientation along two or more defined axes), and 3710 control the opacity (e.g., darkness and / or color) of at least one zone of the opacity filter 3310 based on the determined pose of the AR headset 920. The AR headset controller 1430 is configured to 3702 determine a pose of the AR headset relative to a surgical site containing an anatomical structure, and control the opacity of at least one zone of the opacity filter based on the determined pose of the AR headset relative to the surgical site. For example, in Fig.33 In the example embodiment, the AR headset controller 1430 may darken the region 3324 based on the region being determined to be a defined offset distance from the anatomical structure along the user's line of sight in order to increase contrast for viewing the associated AR image.

[0212] The AR headset controller 1430 may be configured to darken the opacity filter 3310 to improve the user's visibility of the AR image when the user is looking up away from the surgical site. For example, the headset controller 1430 may be configured to 3712 increase the opacity of at least one region of the opacity filter 3310 based on detecting that the AR headset is tilted upward away from the surgical site, and then 3714 decrease the opacity of the region of the opacity filter 3310 based on detecting that the AR headset has been tilted downward toward the surgical site.

[0213] The AR headset controller 1430 may be configured to control the opacity of regions of the opacity filter based on the brightness and / or color of objects identified within the video frame from the camera 1440 / 3460. Fig.38In an operational embodiment of the AR headset controller 1430, the AR headset controller 1430 is configured to 3800 identify an object within a video frame, 3802 determine a brightness and / or color of the identified object within the video frame, 3804 determine a posture of the AR headset 920 relative to a real-world object corresponding to the identified object within the video frame, and 3806 control the opacity (e.g., darkness and / or color) of at least one area of ​​the opacity filter 3310 based on the determined brightness and / or color of the identified object within the video frame and based on the determined posture.

[0214] The AR headset controller 1430 may be configured to control the brightness and / or color of information displayed on the see-through display screen (i.e., projected by the AR emitter 3430 toward the combiner 3440) based on the brightness and / or color of objects identified within the video frames from the camera 1440 / 3460. Fig.39 In an operational embodiment, the AR headset controller 1430 is configured to 3900 identify an object within a video frame, 3902 determine the brightness and / or color of the identified object within the video frame, 3904 determine the pose (e.g., position and / or angular orientation) of the AR headset 920 relative to a real-world object corresponding to the identified object within the video frame, and 3906 control the brightness and / or color of an AR image displayed on a perspective display screen based on the determined brightness and / or color of the identified object within the video frame and based on the determined pose.

[0215] The AR headset controller 1430 can be configured to control the opacity of individual pixels or groups of pixels of the opacity filter 3310. For example, the opacity filter 3310 can be configured to have a plurality of pixels arranged in a grid and electrically controllable to adjust the opacity of the pixels according to the light transmittance. The AR headset controller 1430 is configured to electrically control the opacity filter 3310 to provide increased opacity for at least some pixels selected from the grid based on alignment with or offset from at least a portion of the shape of the AR image displayed on the perspective display screen (i.e., projected by the AR transmitter 3430 toward the combiner 3440). For example, a user's view of bright lighting in an operating room can be dimmed by the AR headset controller 1430, thereby providing increased opacity for a group of pixels of the opacity filter 3310 that are aligned with the lighting along the user's line of sight. In a similar manner, a region offset a defined distance from the anatomy can be dimmed during a surgical procedure by the AR headset controller 1430 providing increased opacity to a set of pixels of the opacity filter 3310 aligned with the region along the user's line of sight to provide increased visibility of the AR image overlaid on the region.

[0216] In some other embodiments, the area behind the content of the AR image can be dimmed, and the dimmed area can be moved to track the movement of the AR headset and / or the location where the AR image is being displayed. For example, the AR headset controller 1430 is configured to electrically control the opacity filter 3310 to provide increased opacity for at least some pixels selected from the grid to form a shape defined based on the perimeter of the shape of the AR image displayed on the perspective display screen. The AR headset controller 1430 is configured to electrically control the opacity filter 3310 to provide a relatively high opacity for a pixel area selected from the grid based on alignment with at least a portion of the shape of the AR image displayed on the perspective display screen, and to provide a gradually transitioning opacity for another pixel area selected from the grid based on being located outside the perimeter of the shape of the AR image displayed on the perspective display screen.

[0217] Using visible light camera 1440 / 3460 to detect ambient lighting conditions on combiner 3440 and electronically control the opacity of opacity filter 3310 can overcome various visibility and contrast issues that arise during surgery. This setup allows both opacity and color of the combiner to be dynamically adjusted based on lighting conditions, virtual content, and even user preferences.

[0218] Dynamically controlling combiner opacity in VR headsets will provide better clarity for users in many use cases.

[0219] In one use case, some content needs to be clearly seen in all situations. For example, medical images used for live surgical navigation should be shown with the greatest possible clarity. Dynamic compositor opacity will allow these images to always be displayed on a dark background.

[0220] Fig.32 An example of darkening a specific area (the top sideways extending band) of a see-through display screen (combiner) based on the content shown therein is shown. In this example, a medical image can be displayed in the upper portion of the combiner. Whenever a medical image occupies the space, the area will be darkened; when not, the area will return to its normal transparency. This dynamic adjustment allows high contrast opacity to be achieved exactly where and when it is needed, and not needed elsewhere.

[0221] In the second use case, dynamic opacity adjustment allows responsive changes to the lighting conditions of the environment, which is particularly important in the operating room. The surgical site is often illuminated with a point light source with a high-power surgical lamp, while the rest of the room is less strongly illuminated. Using a visible light camera attached to the headset 920 or any other means to determine the lighting conditions in the environment, the software can determine which parts of the combiner need to be dimmed to address the lighting of the room. For example, the AR headset controller 1430 can respond to a bright light source in the left part of the user's field of view by responsively dimming the left side of each combiner (when a separate combiner is provided for each eye) or providing a dimmed spaced left area on a single combiner (where the area is aligned with the left field of view of the user's corresponding eye). Since the right part of the user's field of view is not so strongly illuminated, the right side of the combiner remains more transparent. If the surgeon is to move to other places in the room or turn on additional lighting equipment, the darkened area can be adjusted to respond to the new lighting conditions.

[0222] Similarly, the lighting conditions of the room can be used to dynamically adjust the color palette of the virtual content to produce better contrast. In the same way that a visible light camera can be used to determine the overall brightness of the environment. The video frames output by the camera can be used to determine the dominant background color of different parts of the room. This information can be used by the AR headset controller 1430 to select appropriate contrasting colors for the rendered virtual content. Between the dynamically controlled combiner opacity and the adjustment of the color palette of the virtual content, the headset 920 can maintain good clarity and high contrast images regardless of the ambient lighting conditions.

[0223] In a third use case, dynamic opacity adjustment is performed to meet the surgeon's personal preferences. Since opacity can be controlled by software executed by the AR headset controller 1430, the user can be directly controlled to control the appearance of the mechanical predefined combiner segments. This direct control style can be used in conjunction with other automatic adjustment processes (such as for specific content backgrounds or for ambient lighting conditions) by using custom settings as a baseline and dynamically adjusting from the starting point. There are many reasons why users may want to customize opacity: some surgeons may prefer a minimally blocked view of the surgical site and only expect blurred virtual objects; some surgeons may want high visibility to make direct eye contact with assistants, technicians, and other doctors in the room; and other surgeons may prefer to navigate primarily with virtual medical images and therefore want some areas to be almost completely opaque. Regardless of the reason, this technology can allow surgeons to customize their experience based on the current situation, the operating room, and most importantly, their own preferences.

[0224] Additional limitations and embodiments:

[0225] In the above description of each embodiment of the inventive concept, it should be understood that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the inventive concept. Unless otherwise specified, all terms used herein (including technical terms and scientific and technological terms) have the same meanings commonly understood by those skilled in the art to which the inventive concept belongs. It should be further understood that the terms as defined in the commonly used dictionary should be interpreted as having the meaning consistent with its meaning in the context of this specification and in the relevant field, and are clearly defined in this article, and will not be interpreted in an idealized or overly formal sense.

[0226] When an element is referred to as being "connected to" or "coupled to" "in response to" another element or its variant, it may be directly connected to, coupled to or responsive to another element, or there may be an intermediate element. In contrast, when an element is referred to as being "directly connected to", "directly coupled to", "directly responsive to" another element or its variant, there is no intermediate element. Similar elements are referred to with similar numbers throughout the text. In addition, "connection", "connection", "response" or its variants as used herein may include wireless connection, connection or response. Unless the context clearly indicates otherwise, as used herein, the singular forms "a, an" and "the" are intended to include plural forms. For the sake of brevity and / or clarity, well-known functions or configurations may not be described in detail. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0227] It should be understood that, although the terms first, second, third, etc. can be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish an element / operation from another element / operation. Therefore, without departing from the teaching of the present invention concept, the first element / operation in some embodiments can be referred to as the second element / operation in other embodiments. Throughout this specification, the same reference numerals or the same figure designators represent the same or similar elements.

[0228] As used herein, the terms "comprise / comprising / comprises", "include / including / includes", "have / has / having" or variations thereof are open ended and include one or more stated features, integers, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or combinations thereof. In addition, as used herein, the general abbreviation "such as", derived from the Latin phrase "exempli gratia", may be used to introduce or specify one or more general examples of previously mentioned items and is not intended to limit such items. The general abbreviation "that is", derived from the Latin phrase "id est", may be used to specify a specific item from a more general statement.

[0229] Example embodiments are described herein with reference to block diagrams and / or flow diagrams of computer-implemented methods, devices (systems and / or apparatuses), and / or computer program products. It should be understood that the blocks of the block diagrams and / or flow diagrams and the combination of blocks in the block diagrams and / or flow diagrams can be implemented by computer program instructions executed by one or more computer circuits. These computer program instructions can be provided to processor circuits of general-purpose computer circuits, special-purpose computer circuits, and / or other programmable data processing circuits to produce a machine so that instructions executed by processors of computers and / or other programmable data processing devices convert and control transistors, values ​​stored in memory locations, and other hardware components within such circuit systems to implement the functions / actions specified in the block diagrams and / or one or more flow diagram blocks, and thereby create a device (function) and / or structure for implementing the functions / actions specified in the block diagrams and / or one or more flow diagram blocks.

[0230] These tangible computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing device to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture containing instructions for implementing the functions / actions specified in the block diagram or one or more flowchart blocks. Thus, embodiments of the inventive concept may be embodied in hardware and / or software (including firmware, resident software, microcode, etc.) that runs on a processor such as a digital signal processor, which may be collectively referred to as a "circuitry system," "module," or variations thereof.

[0231] It should also be noted that, in some alternative implementations, the function / action marked in the frame may not occur in the order marked in the flow chart. For example, according to the function / action involved, the two frames shown continuously can actually be performed substantially simultaneously, or the frame can sometimes be performed in reverse order. In addition, the function of a given frame of a flow chart and / or block diagram can be divided into multiple frames, and / or the function of two or more frames of a flow chart and / or block diagram can be integrated at least in part. Finally, without departing from the scope of the inventive concept, other frames can be added / inserted between the frames shown, and / or frame / operation can be omitted. In addition, although some figures in the figure include arrows on the communication path to illustrate the main direction of communication, it should be understood that communication can occur in the direction opposite to the arrows depicted.

[0232] Without substantially departing from the principle of the inventive concept, many changes and modifications may be made to the embodiments. All such variants and changes are intended to be included in the scope of the inventive concept herein. Therefore, the subject matter disclosed above should be considered illustrative rather than restrictive, and the examples of the attached embodiments are intended to cover all such modifications, enhancements and other embodiments that fall within the spirit and scope of the inventive concept. Therefore, in order to be allowed to the greatest extent by law, the scope of the inventive concept will be determined by the most widely allowed interpretation of the present disclosure, which includes the examples of the following embodiments and their equivalents, and should not be limited to or confined to the aforementioned specific detailed description.

Claims

1. A surgical system, include: An augmented reality headset configured to be worn by a user during a surgical procedure and comprising a see-through display screen configured to display an augmented reality image and allow at least a portion of a real world scene to pass through for viewing by the user, and an opacity filter positioned between at least one of the user's eyes and the real-world scene when the see-through display screen is viewed by the user, wherein the opacity filter is configured to provide opacity to light from the real-world scene; as well as an augmented reality headset controller configured to communicate with a navigation controller to receive navigation information from the navigation controller to provide guidance to the user during the surgical procedure performed on the anatomical structure, and configured to generate the augmented reality image based on the navigation information for display on the see-through display screen; as well as a camera configured to output video frames capturing the real-world scene, The augmented reality headset controller is configured to determine the brightness and / or color of a defined area within the video frame, and to control the brightness and / or color of the augmented reality image displayed on the perspective display screen based on the determined brightness and / or color of the defined area within the video frame.

2. The surgical system of claim 1 , wherein the see-through display screen comprises a curved lens extending laterally across and downwardly from a front portion of the augmented reality headset, and wherein the opacity filter is located on a surface of the curved lens.

3. A surgical system according to claim 1, wherein the opacity filter is configured with at least two laterally extending bands, wherein the light transmittance of an upper band of the at least two laterally extending bands is lower than the light transmittance of a lower band of the at least two laterally extending bands.

4. The surgical system according to claim 1, in: The opacity filter is configured as a laterally extending strip; The augmented reality headset controller is configured to display at least one of the following in an area of ​​the see-through display screen aligned with the laterally extending bands of the opacity filter: a 2D axial, sagittal and / or coronal view image of the patient's anatomy; a planned and / or currently tracked surgical tool pose; a graphical model of a surgical implant location; Videos from medical instruments; and a user-selectable menu item triggering operation for controlling a medical device, and the augmented reality headset controller is configured to display at least one of the following in another area of ​​the see-through display screen that is not aligned with the laterally extending band of the opacity filter: a 3D graphical model of the anatomical structure and surgical planning information; a 3D graphical model of a surgical instrument; an animated 3D graphical model of the surgical instrument displayed together with a pose of the graphical model relative to the anatomical structure, the animated 3D graphical model being modified to track in real time a measured pose of the surgical instrument relative to the anatomical structure; and a graphical model of the anatomical structure and the navigation information from the navigation controller, the navigation information providing visual guidance to the user during the surgical procedure performed on the anatomical structure.

5. The surgical system of claim 1, further comprising: include: an augmented reality transmitter configured to project light of the augmented reality image in response to signaling from the augmented reality headset controller, wherein the opacity filter is configured as a laterally extending strip, wherein the see-through display screen includes a combiner configured to combine the light of the augmented reality image projected from the augmented reality emitter with light from the real world scene into a combined overlay image viewable by the user, and Wherein the laterally extending strips of the opacity filter are located on a surface of the combiner.

6. The surgical system of claim 1, further comprising: include: an augmented reality transmitter configured to project light of the augmented reality image in response to signaling from the augmented reality headset controller, wherein the see-through display screen includes a combiner configured to combine light of the augmented reality image projected from the augmented reality emitter with light from the real world scene into a combined image viewable by the user, and Wherein the opacity filter comprises an opacity gradient extending over a surface of the combiner.

7. A surgical system according to claim 1, wherein for each of a plurality of augmented reality image types, the augmented reality headset controller is configured to select in a plurality of different areas of the see-through display screen to display the augmented reality image type based on characteristics of the augmented reality image type and an opacity level of areas of the opacity filter aligned with the different areas of the see-through display screen.

8. The surgical system of claim 1, further comprising: include: a camera configured to output video frames capturing the real-world scene, The augmented reality headset controller is configured to determine the brightness and / or color of a defined area within the video frame, and to control the opacity of at least one area of ​​the opacity filter based on the determined brightness and / or color of the defined area within the video frame.

9. The surgical system of claim 1 , wherein the augmented reality headset controller is configured to determine a posture of the augmented reality headset and to control the opacity of at least one region of the opacity filter based on the determined posture of the augmented reality headset.