A system for assisting surgical procedures

Abstract

A system for assisting in performing a procedure. The system comprises robot equipment, wearable display equipment, a navigation system and a control system. The robotic device is operable to grip and / or manipulate an instrument to facilitate performing a surgical procedure at a target site of a patient. The wearable display device is operable to superimpose digital content into a user's field of view, the digital content including information related to a surgical procedure. The navigation system is operable to obtain tracking data of a target site and instrument of the patient based at least in part on tracking the position and / or orientation of the target site and instrument of the patient. The control system is operable to control operation of the robotic device and / or the wearable display device based at least in part on the tracking data.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202511260484.8 (22) Application Date 2025.09.04 (30) Priority Data 63 / 702,748 2024.10.03 US (71) Applicant The Chinese University of Hong Kong Address Sha Tin, New Territories, Hong Kong, China (72) Inventors Xu Zhenxing Rong Shuheng Wang Tianxin Huang Wenyang Yu Jia He Yaohua Ye Xin (74) Patent Agency Beijing Weifei Lianchuang Intellectual Property Agency Co., Ltd. 11857 ​​Patent Attorney Jin Libin (51) Int.Cl. A61B 34 / 20 (2016.01) A61B 34 / 30 (2016.01) A61B 34 / 00 (2016.01) G16H 20 / 40 (2018.01) G06T 19 / 00(2011.01) (54) Title of Invention: A System for Assisting Surgery (57) Abstract: A system for assisting surgery. The system includes a robotic device, a wearable display device, a navigation system, and a control system. The robotic device is operable to hold and / or manipulate instruments to facilitate the performance of surgical procedures at a target site on a patient. The wearable display device is operable to overlay digital content onto the user's field of vision, the digital content including information related to the surgical procedure. The navigation system is operable to obtain tracking data of the target site on the patient and the instruments, at least in part based on tracking the position and / or orientation of the target site on the patient. The control system is operable to control the operation of the robotic device and / or the wearable display device, at least in part based on the tracking data. Claims 2 pages, Description 21 pages, Drawings 8 pages, CN 121774637 A 2026.04.03 CN 1 21 77 46 37 A 1. A system for assisting in performing surgery, characterized in that it comprises: a robotic device operable to hold and / or manipulate instruments to perform a surgical operation at a target site on a patient; a wearable display device operable to overlay digital content onto the user's field of vision of the wearable display device, the digital content including information related to the surgical operation; a navigation system operable to obtain tracking data of the target site and the instruments on the patient, at least in part based on tracking the position and / or orientation of the target site and the instruments; and a control system operable to control the operation of the robotic device and / or the wearable display device, at least in part based on the tracking data. 2. The system according to claim 1, characterized in that the control system is operable to control the robot on at least in part based on the tracking data and surgical planning data for the surgical operation.3. The system of claim 2, wherein the surgical planning data is determined at least in part based on the patient's anatomy and / or physiological state. 4. The system of any one of claims 1 to 3, wherein the control system is operable to control the operation of the robotic device at least by influencing the movement of the robotic device. 5. The system of any one of claims 1 to 3, wherein the control system is operable to control the operation of the wearable display device at least by controlling the digital content superimposed on the wearable display device. 6. The system of any one of claims 1 to 3, wherein the control system is at least in part integrated with the robotic device. 7. The system of any one of claims 1 to 3, wherein the control system is at least in part integrated with the wearable display device. 8. The system of any one of claims 1 to 3, wherein the control system is at least in part integrated with the navigation system. 9. The system of any one of claims 1 to 3, wherein the robotic device is operable to manipulate surgical instruments for performing the surgical procedure. 10. The system according to any one of claims 1 to 3, wherein the robotic device is operable to grasp and / or manipulate an imaging device for performing imaging to facilitate the surgical procedure. 11. The system according to any one of claims 1 to 3, wherein the robotic device includes a robotic arm. 12. The system according to claim 11, wherein the robotic arm has six degrees of freedom. 13. The system according to claim 11, wherein the robotic arm includes a haptic feedback mechanism. 14. The system according to claim 11, wherein the robotic arm includes a force feedback sensor. 15. The system according to claim 11, wherein the robotic arm includes a depth measurement mechanism. 16. The system according to any one of claims 1 to 3, wherein the wearable display device includes a head-mounted display device. 17. The system according to claim 16, wherein the head-mounted display device includes an eye-tracking device for tracking the user to influence the digital content. Claims 1 / 2 Page 2 CN 121774637 A 18. The system according to any one of claims 1 to 3, characterized in that, the information related to the surgical operation includes real-time guidance information for guiding the execution of the surgical operation.19. The system according to any one of claims 1 to 3, wherein the information related to the surgical procedure includes real-time anatomical and / or physiological measurement data of the patient. 20. The system according to any one of claims 1 to 3, wherein the information related to the surgical procedure includes a virtual representation of the patient's anatomical structure. 21. The system according to any one of claims 1 to 3, wherein the navigation system includes one or more navigation sensors, each operable to perform tracking. 22. The system according to any one of claims 1 to 3, wherein the system further includes a plurality of markers disposed on the patient and the instrument; and the navigation system is operable to track the target site of the patient and the position and / or orientation of the instrument based at least partially on the plurality of markers. 23. The system according to claim 22, wherein the markers include optical markers attached to the patient and the instrument. 24. The system according to claim 23, wherein the optical markers include an infrared reflective coating. 25. The system according to claim 23, wherein the optical markers are sterilizable. Claims 2 / 2 Page 3 CN 121774637 A A System for Assisting Surgery Technical Field

[0001] This invention relates to a system for assisting surgery. Background Art

[0002] Some surgeries (e.g., traditional orthopedic surgeries) may face challenges such as long operation time, high risk of infection, and high risk of human error. Existing robot-assisted surgical systems and existing augmented reality (AR) applications may alleviate these problems. However, these systems and applications generally operate independently of each other. Therefore, this may result in a fragmented surgical process and poor results. Summary of the Invention

[0003] Currently, there is a need for the design and technology of robot-assisted AR navigation surgical systems.

[0004] The system for assisting surgery of the present invention integrates robotics, AR, and navigation technologies, and is designed to improve the surgical field (e.g., orthopedic surgery). As the healthcare industry continues to pursue higher precision and efficiency, the robot-assisted AR navigation surgical system of the present invention can be a solution to meet these needs.

[0005] In a first aspect, a system for assisting surgery is provided. The system includes a robotic device, a wearable display device, a navigation system, and a control system. The robotic device is operable to hold and / or manipulate instruments to facilitate surgical procedures at target sites on the patient. The wearable display device is operable to overlay digital content, including information relevant to the surgical procedure, onto the user's field of vision. The navigation system is operable to at least partially...The system obtains tracking data of the patient's target site and instruments based on the position and / or orientation of the target site and instruments. The control system is operable to control the operation of the robotic device and / or wearable display device based at least in part on the tracking data.

[0006] Optionally, the control system is operable to control the operation of the robotic device and / or wearable display device based at least in part on the tracking data and surgical planning data for surgical procedures.

[0007] Optionally, the surgical planning data is determined at least in part based on the patient's anatomy and / or physiological state.

[0008] Optionally, the control system is operable to control the operation of the robotic device at least by influencing the movement of the robotic device (e.g., orientation, speed, acceleration, etc.).

[0009] Optionally, the control system is operable to control the operation of the wearable display device at least by controlling (e.g., updating) the digital content superimposed on the wearable display device.

[0010] Optionally, the control system is at least in part integrated with the robotic device.

[0011] Optionally, the control system is at least in part integrated with the wearable display device.

[0012] Optionally, the control system is at least partially integrated with the navigation system.

[0013] Optionally, the robotic device is operable to manipulate surgical instruments, which can be used to perform surgical procedures.

[0014] Optionally, the robotic device is operable to grasp and / or manipulate an imaging device, which can be used to perform imaging to facilitate the performance of surgical procedures.

[0015] Optionally, the robotic device includes a robotic arm.

[0016] Optionally, the robotic arm has six degrees of freedom. Specification 1 / 21 page 4 CN 121774637 A

[0017] Optionally, the robotic arm includes a haptic feedback mechanism.

[0018] Optionally, the robotic arm includes a force feedback sensor.

[0019] Optionally, the robotic arm includes a depth measurement mechanism.

[0020] Optionally, the wearable display device includes a head-mounted display device.

[0021] Optionally, the head-mounted display device includes an eye-tracking device for tracking the user to influence digital content.

[0022] Optionally, the information related to the surgical procedure includes real-time guidance information (e.g., operating instructions) for guiding the execution of the surgical procedure.

[0023] Optionally, the information related to the surgical procedure includes real-time anatomical and / or physiological measurement data of the patient.

[0024] Optionally, the information related to the surgical procedure includes a virtual representation (e.g., 2D or 3D representation) of the patient's anatomy.

[0025] Optionally, the navigation system includes one or more navigation sensors, each operable to perform tracking.

[0026] Optionally, the system also includes a plurality of markers arranged on the patient and instruments.

[0027] Optionally, the navigation system is operable to track target sites on the patient and instruments based at least in part on the plurality of markers.Position and / or orientation of the instrument.

[0028] Optionally, the marking includes optical markings attached to the patient and the instrument.

[0029] Optionally, the optical marking includes an infrared reflective coating.

[0030] Optionally, the optical marking may be sterilizable.

[0031] Other features and aspects of the invention will become apparent from consideration of the specific embodiments and the accompanying drawings. Where appropriate and applicable, any feature described herein with respect to one aspect or embodiment may be combined with any other feature described herein with respect to any other aspect or embodiment. Brief Description of the Drawings

[0032] Embodiments of the invention will now be described by way of example with reference to the accompanying drawings, in which:

[0033] FIG1 shows a schematic diagram of a system for assisting in performing surgery according to one embodiment of the invention.

[0034] FIG2 shows a schematic diagram of a control system according to one embodiment of the invention.

[0035] FIG3 shows a schematic diagram of preoperative planning and intraoperative surgical procedures according to one embodiment of the invention.

[0036] FIG4 shows a schematic diagram of the calibration process of a robotic arm and AR system according to one embodiment of the invention.

[0037] FIG5 shows a schematic diagram of the initialization process of an AR head-mounted device according to one embodiment of the invention.

[0038] Figure 6 shows a schematic diagram of the initialization process of a navigation system according to an embodiment of the present invention.

[0039] Figure 7 shows a schematic diagram of a robot-assisted AR navigation orthopedic surgical system according to an embodiment of the present invention.

[0040] Figure 8 shows a schematic diagram of a tracking element and its orientation according to an embodiment of the present invention.

[0041] Figure 9 shows a schematic diagram of AR overlay and display of an AR head-mounted device according to an embodiment of the present invention.

[0042] Figure 10 shows a schematic diagram of empirically validated postoperative K-wire positioning (for a pre-planned model) according to an embodiment of the present invention. Detailed Description

[0043] The present invention relates to a system for assisting in performing surgery. Specifically, some embodiments of the invention relate to the specification 2 / 21 pages 5 CN 121774637 A A robot-assisted AR navigation surgical system.

[0044] The terminology used herein is only for describing particular embodiments and is not intended to limit the scope of this disclosure. In this document, the term "and / or" includes any and all combinations of one or more of the associated listed items. In this document, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to include both the plural and singular forms. It should also be understood that, in this document, the terms “comprising” and / or “including” specify the presence of the stated feature, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0045] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the meanings commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having meanings consistent with their meanings in the relevant field and the context of this disclosure, and should not be interpreted in an idealized or overly formal manner unless explicitly defined herein.

[0046] In this document, when the term “about” is used in conjunction with a numerical value, it should be understood as meaning that the value can be in the range of 90% to 110% of the value, i.e., the value can be + / - 10% of the value. For example, “about 1 kg” can represent 0.90 kg to 1.1 kg.

[0047] Various techniques and steps are disclosed herein. Each of these techniques and steps may have its own advantages. Furthermore, each of these techniques and steps may be combined or combined with one or more (e.g., all) of the other disclosed techniques and steps. Therefore, for clarity and brevity, all possible combinations of the various techniques and steps will not be described unnecessarily herein.

[0048] Figure 1 illustrates a system 100 for assisting in performing surgery according to one embodiment of the invention. For example, system 100 can be used to assist in orthopedic surgery. In some embodiments, system 100 can be used as a robot-assisted AR navigation surgical system.

[0049] In this embodiment, system 100 includes a robotic device 102, a wearable display device 104, a navigation system 106, and a control system 108. In some embodiments, the control system 108 can communicate with the robotic device 102, the wearable display device 104, and / or the navigation system 106 via wired and / or wireless communication links.

[0050] In this embodiment, the robotic device 102 is operable to hold and / or manipulate instruments to facilitate surgical procedures at target sites on a patient. In some embodiments, the robotic device 102 is operable to manipulate surgical instruments that can be used to perform surgical procedures. In some embodiments, the robotic device 102 is operable to hold and / or manipulate an imaging device that can be used to perform imaging to facilitate surgical procedures. In some embodiments, the robotic device 102 includes a robotic arm. The robotic arm may have multiple (e.g., six) degrees of freedom. The robotic arm may include a haptic feedback mechanism to provide haptic feedback. The robotic arm may include a force feedback sensor to provide force feedback. The robotic arm may include a depth measurement mechanism to measure depths relevant to surgical procedures.

[0051] In this embodiment, the wearable display device 104 is operable to overlay digital content onto the user's field of vision. The digital content includes information relevant to surgical procedures. In some embodiments, the wearable...Display device 104 includes a head-mounted display device (e.g., a head-mounted display). In some embodiments, wearable display device 104 includes an eye-tracking device for tracking a user to influence (change) digital content. In some embodiments, information related to surgical procedures may include one or more of the following: real-time guidance information (e.g., operating instructions) for guiding the execution of surgical procedures, real-time anatomical and / or physiological measurement data of the patient (e.g., in a list or diagram format), virtual representations of the patient's anatomy (e.g., 2D or 3D representations), etc.

[0052] In this embodiment, navigation system 106 is operable to obtain tracking data of the patient's target site and instruments based at least in part on the position and / or orientation of the target site and instruments. In some embodiments, navigation system 106 includes one or more navigation sensors, each operable to perform tracking. In some embodiment specification pages 3 / 21 6 CN 121774637 A, system 100 also includes a plurality of markers arranged on the patient and instruments, and navigation system 106 is operable to track the position and / or orientation of the patient's target site and instruments based at least in part on the plurality of markers. In some embodiments, the marking may include optical markings attached to the patient and instruments. In some embodiments, the optical markings include an infrared reflective coating. In some embodiments, the optical markings may be sterilized for reuse.

[0053] In this embodiment, the control system 108 is operable to control the operation of the robotic device 102 and / or the wearable display device 104 based at least in part on tracking data. Thus, in some embodiments, the system 100 can facilitate the execution of surgical procedures by controlling the operation of the robotic device 102 and / or the wearable display device 104 in real time. In some embodiments, the control system 108 is operable to control the operation of the robotic device 102 and / or the wearable display device 104 based at least in part on tracking data and surgical planning data for the surgical procedure. Thus, the system 100 can control the operation of the robotic device 102 and / or the wearable display device 104 to appropriately execute a predetermined surgical plan according to the real-time situation.

[0054] In some embodiments, the surgical planning data is determined at least in part based on the patient's anatomy and / or physiological state. That is, the surgical planning data is patient-specific surgical planning data. In some embodiments, the control system 108 is operable to control the operation of the robot device 102 at least by influencing the motion (e.g., direction, speed, acceleration, etc.) of the robot device 102. In some embodiments, the control system 108 is operable to control the operation of the wearable display device 104 at least by controlling (e.g., updating) the digital content superimposed on the wearable display device 104. In some embodiments, the control system108 is at least partially integrated with the robotic device 102. For example, the control system 108 may be part of the robotic device 102. In some embodiments, the control system 108 is at least partially integrated with the wearable display device 104. For example, the control system 108 may be part of the wearable display device 104. In some embodiments, the control system 108 is at least partially integrated with the navigation system 106. For example, the control system 108 may be part of the navigation system 106.

[0055] FIG2 illustrates a control system 200 according to one embodiment of the present invention. In some embodiments, the control system 200 may be used to implement the control system 108. In some embodiments, the control system 200 may be part of the robotic device 102, part of the wearable display device 104, and / or part of the navigation system 106. In some embodiments, the control system 200 may be used as a computer system or computing device.

[0056] In this embodiment, the control system 200 generally includes appropriate components required to receive, store, and execute appropriate computer instructions, commands, and / or codes. The main components of the control system 200 are the processor 202 and the memory 204. Processor 202 may include one or more of the following: one or more central processing units (CPUs), one or more microcontrollers (MCUs), one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), one or more neural processing units (NPUs), one or more logic circuits, one or more Raspberry Pi chips, one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or any other digital or analog circuits configured to interpret and / or execute program instructions and / or process signals and / or information and / or data. Processor 202 may be used to perform machine learning-based processing and non-machine learning-based processing. Memory 204 may include: one or more volatile memories (e.g., random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), etc.), one or more non-volatile memories (e.g., read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferroelectric random access memory (FRAM), magnetic random access memory (MRAM), flash memory (FLASH), solid-state drive (SSD), flash memory (NAND), non-volatile dual in-line memory module (NVDIMM), etc.), or any combination thereof. Appropriate computer instructions, commands, codes, information, and / or data (e.g., data used to facilitate the performance of surgical procedures) may...The processor 202 and memory 204 are stored in memory 204. Computer instructions for performing or facilitating the execution of method embodiments of the present invention may be stored in memory 204. The processor 202 and memory 204 may be integrated together or may be disposed separately (but operatively connected). Optionally, the control system 200 may also include one or more input devices 206. Examples of such input devices 206 include: keyboard, mouse, stylus, image scanner, microphone, haptic / touch input device (e.g., touchscreen), image / video input device (e.g., camera), etc. Input devices 206 may be used to receive input from the user of the system. Optionally, the control system 200 may also include one or more output devices 208. Examples of such output devices 208 include: display (e.g., monitor, screen, projector, etc.), speaker, headset, earphone, head-mounted display, printer, additive manufacturing machine (e.g., 3D printer), etc. The display may include an LCD display, an LED / OLED display, or other suitable display (which may or may not be touch-sensitive). The control system 200 may also include one or more disk drives 212, which may include one or more of the following: solid-state drives, hard disk drives, optical disk drives, flash drives, tape drives, etc. A suitable operating system may be installed in the control system 200 (e.g., in the disk drive 212 or in the memory 204). The memory 204 and the disk drive 212 may be operated by the processor 202. Optionally, the control system 200 may also include a communication device 210 for establishing one or more communication links (not shown) with one or more other computing devices (e.g., servers, personal computers, terminals, tablets, mobile phones, watches, IoT devices, or other wired or wireless computing devices). The communication device 210 may include one or more of the following: modems, network interface cards (NICs), integrated network interfaces, NFC transceivers, ZigBee transceivers, Wi-Fi transceivers, transceivers, radio frequency transceivers, cellular (2G, 3G, 4G, 5G, 5G and above, etc.) transceivers, optical ports, infrared ports, USB connections, or other wired or wireless communication interfaces. The transceiver can be implemented by one or more devices (integrated transmitter and receiver, separate transmitter and receiver, etc.). The communication link can be wired or wireless, and is used to transmit commands, instructions, information, and / or data. In one example, processor 202, memory 204 (optionally, input device 206, output device 208, communication device 210, and disk drive 212 (if present)) are directly or indirectly connected via a bus, peripheral component interconnect (PCI) (e.g., peripheral component interconnect high speed), universal serial bus (USB), optical bus, or other similar bus structures.Grounded interconnection. In one embodiment, at least some of these components may be wirelessly connected, for example, via a network (e.g., the Internet or a cloud computing network). Those skilled in the art will understand that the control system 200 shown in FIG2 is merely an example and in other embodiments the control system 200 may have different configurations (e.g., including additional components, omitting one or more components, etc.).

[0057] According to some embodiments of the invention, a system is provided that combines robotic arm assistance with augmented reality (AR) navigation in surgery (e.g., orthopedic surgery). This system can provide a fusion of accuracy, visualization, and data-driven insights. The robotic arm can improve accuracy and / or controllability, thereby reducing human error caused by factors such as hand tremors. Furthermore, AR navigation complements this. Specifically, AR navigation can provide a real-time three-dimensional (3D) visualization of the surgical site to enhance the surgeon's understanding of the relevant anatomy and pathology. The combination of these technologies can provide highly personalized surgical plans, allowing preoperative planning based on the specific needs of each patient.

[0058] The synergy between robotic arm assistance and AR navigation can streamline the surgical procedure. For example, when the robotic arm performs precise movements, AR can display key data directly in the surgeon's field of vision, allowing for real-time adjustments and potentially leading to better surgical outcomes. In some embodiments, this combined approach can reduce radiation exposure because AR navigation can replace the need for fluoroscopy.

[0059] Furthermore, in some embodiments, the system can be used for education. For example, the system can serve as a training tool for medical students and junior surgeons, providing detailed, interactive, and highly accurate demonstrations of surgical procedures. Surgeon fatigue is a problem during long and complex surgeries, and the robotic arm can handle some physically demanding tasks, thereby reducing surgeon fatigue.

[0060] The combination of these technologies can open new avenues for data collection and postoperative analysis, which may be of significant value for future research and quality control. In addition, the precision provided by the robotic arm and AR navigation can reduce the invasiveness of the surgery, thus helping patients recover faster.

[0061] This interdisciplinary collaborative approach may foster closer collaboration among engineers, computer scientists, and medical professionals, thereby driving further innovation in surgical fields, such as orthopedic surgery. While this technology is still in its early stages and requires rigorous clinical trials to fully validate its advantages, combining robotic arm assistance with AR navigation offers several potential benefits.

[0062] Some embodiments of the present invention provide a robot-assisted AR-navigated orthopedic surgical system. This system is expected to...This system can be used in various practical applications and may help improve patient outcomes, increase surgical precision, and / or enhance medical training.

[0063] The system can be configured to improve surgical precision, reduce trauma, and / or improve overall patient outcomes. By combining robot-assisted technology and AR navigation, the system allows surgeons to accurately navigate complex anatomical structures, thereby reducing surgical errors. This precise positioning can reduce or minimize damage to surrounding tissues, thereby alleviating postoperative pain and accelerating patient recovery. In addition, the AR component can provide a real-time visualization of 3D anatomical structures, thereby helping surgeons make informed and timely decisions during surgery. The system can also be used as an educational tool, enabling medical professionals to improve or refine their skills in a simulated environment. In some embodiments, the system can reduce the overall duration and associated costs of surgical procedures through its navigation and visualization capabilities. In some embodiments, the system can be designed to collect and analyze intraoperative data, which can be used to enhance postoperative monitoring, thereby helping to improve long-term patient outcomes.

[0064] For example, the system can serve as a multi-faceted tool that has the potential to transform multiple aspects of orthopedic surgery, such as from spinal fusion and joint replacement to more complex fracture repair and arthroscopic surgery. This system can provide real-time 3D visualization and precise instrument navigation, thereby enhancing surgeons' ability to accurately perform complex surgeries such as limb lengthening, tumor resection, and correction of congenital malformations. In emergency trauma situations, the system can be rapidly deployed to assist in stabilizing fractures and dislocations, thereby ensuring optimal alignment and fixation.

[0065] For example, the system can serve as a valuable educational and postoperative monitoring tool. It can also serve as a training tool, providing a simulated environment that allows medical students and junior surgeons to practice complex surgeries. Furthermore, the system can be designed to collect and analyze surgical data, providing insights into the effectiveness of surgical interventions and assisting in future treatment planning. This data-driven approach can improve long-term patient outcomes and may contribute to research in orthopedic surgery.

[0066] The combination of robotics and AR technology in this system not only opens new avenues for further research and innovation but also has the potential to promote the popularization of advanced surgical techniques. Because the system's AR capabilities allow for convenient remote guidance from experienced professionals, surgeons in remote or resource-poor areas can benefit from its functionality. The practical application of this technology can improve patient care, increase surgical success rates, reduce complications, and / or accelerate patient recovery (e.g., patients undergoing orthopedic surgery).

[0067] Further details of some embodiments of the present invention are provided below.

[0068] Robot-assisted

[0069] In some embodiments, the system has a robotic arm employing sensors and actuators, providing surgeons with a high level of control and precision. The robotic arm can have multiple functions and can be used to perform a range of tasks (e.g., from precise positioning of surgical instruments to more complex operations (e.g., drilling for bone fixation and scalpel control for precise incisions)). The robotic arm can also hold imaging devices, providing real-time imaging data that can facilitate surgical success.

[0070] In addition to instrument positioning and real-time imaging, the robotic arm can also provide specialized functions for specific surgical needs. For example, the robotic arm can assist in inserting cannulas, precisely deploying surgical screws, and / or performing accurate bone cuts in laparoscopic surgery. The system can also help insert screws into bone for secure fixation and can assist in placing clips onto blood vessels or tissue, thereby reducing the risk of complications and improving surgical outcomes. In some embodiments, one feature of the robotic arm is its ability to measure depth, which is particularly useful for surgeries requiring precise incisions or holes. This feature ensures accurate depth of incisions or drill holes, thereby minimizing the risk of damaging underlying structures (e.g., anatomical structures). Whether suturing, cutting bone, or inserting screws, the robotic arm provides additional precision, which not only improves the quality of surgery but also significantly reduces the likelihood of postoperative complications. Whether suturing, cutting bone, or inserting screws, the robotic arm provides additional precision, which not only improves the quality of surgery but also significantly reduces the likelihood of postoperative complications.

[0071] Augmented Reality (AR) Headsets (e.g., Head-Mounted Displays)

[0072] The advent of AR in the medical field has ushered in a new era of surgical precision and patient safety. Surgeons wearing AR headsets (e.g., head-mounted displays) can gain enhanced vision with three-dimensional (3D) visual information, such as virtual images of anatomical structures, surgical plans, and / or real-time feedback from navigation sensors. Some embodiments of the present invention integrate optical navigation trackers into AR headsets, thereby improving the accuracy of superimposed digital information. AR systems can serve as tools to facilitate the precise positioning of instruments, whether for general surgical instruments or more specialized tools such as drills for bone fixation.

[0073] AR head-mounted devices can be particularly useful for tasks requiring high precision and spatial awareness. For example, they can assist in precise drilling by directly overlaying drilling paths onto the patient or surgical area. Similarly, they can assist in precise bone cutting by projecting cutting paths onto the surgical area. This technology is also useful when inserting screws and Kirschner wires into bone for fixation, providing surgeons with digital guidance overlaid onto the surgical area to ensure optimal placement. These features not only guide the surgeon but also reduce the risk of errors that could lead to complications.

[0074] In addition to guiding surgical instruments, the AR system can also highlight key anatomical structures (such as blood vessels, nerves, organs, etc.). This can significantly reduce the risk of accidental injury during surgery. Furthermore, the system can integrate vital signs, instrument readings, and other important data into the surgeon's field of vision. This real-time data integration capability helps the surgeon make immediate adjustments and informed decisions during surgery. Another feature is the measurement of the depth of incisions or orifices, which can provide additional safety and accuracy.

[0075] The AR system may also include features for intraoperative and postoperative monitoring. For example, during surgery, the position of the Kirschner wires can be compared with a virtual AR Kirschner wire image to adjust the bone to the correct fixation position. The same comparison can also be performed after surgery to verify that the bone has been fixed in the correct position. This two-stage application of AR technology can ensure the accuracy of the surgical procedure and can serve as a verification tool for assessing the effectiveness of interventions, thereby helping to improve long-term treatment outcomes for patients.

[0076] Navigation System

[0077] In some embodiments, the core of the surgical system lies in its navigation setup, which employs navigation sensors strategically placed in the operating room to track the real-time position and orientation of the patient and surgical instruments. The navigation sensor can input data into the central processing unit. The central processing unit can use algorithms to generate an accurate 3D spatial map. This ensures accurate instrument positioning (whether for general instrument positioning or more specialized tasks). The navigation system may be particularly well-suited for tracking drill coordinates, angular orientation, and / or penetration depth, making it a useful tool for bone fixation surgery. It can incorporate a range of specially designed methods and instruments to improve the accuracy of drilling operations, thereby enhancing the overall effectiveness of bone fixation techniques.

[0078] In addition to drilling, the navigation system can also be equipped with methods and algorithms to monitor the bone cutting process. This capability may be important for achieving surgical precision in bone cutting, which is a crucial factor in the successful execution of various orthopedic surgical procedures. The system can also provide a comprehensive operational framework for accurately tracking and inserting screws into the bone structure for fixation. This may include a detailed visualization of screw insertion coordinates, angular orientation, and / or penetration depth to ensure optimal bone fixation and contribute to better surgical outcomes. The navigation system may also be able to track data used to measure the depth of incisions or holes, increasing the accuracy and safety of the surgical procedure.

[0079] Workflow and Functions

[0080] The following is a workflow and function of one embodiment of the invention.

[0081] Preoperative Planning: Before surgery, the system is configured to import patient-specific diagnostic data, such as magnetic resonance imaging.(MRI) and computed tomography (CT) scan data. The attending surgeon can use a specialized software interface to develop a comprehensive virtual surgical plan. The plan not only specifies the optimal incision point, path, and instrument movement trajectory, but also includes calculated parameters for the Kirschner wire insertion angle and depth. In addition, the system can be designed to pre-calculate the postoperative Kirschner wire positioning, thereby tailoring a detailed surgical intervention plan to the patient based on the patient's unique anatomical features.

[0082] Intraoperative AR: At the start of the operation, the surgeon can be equipped with an AR head-mounted device. The system overlays the pre-set surgical plan onto the surgeon's real-time field of vision. This enhanced interface can overlay the virtual plan onto the patient's actual anatomical structure, thereby enhancing the surgeon's situational awareness. During the operation, the system allows real-time comparison of the Kirschner wire position and the AR virtual Kirschner wire image, thereby facilitating adjustments to achieve correct bone fixation.

[0083] Real-time navigation: Simultaneously, the system's navigation module is activated to continuously monitor the spatial position and orientation of surgical instruments. This real-time tracking data is processed and integrated into the AR display to update the surgeon's visual interface. The system allows surgeons to observe the real-time position of surgical instruments (including Kirschner wires) relative to a pre-set surgical plan, thereby ensuring accurate placement and alignment of surgical instruments.

[0084] Robot-assisted: The system also includes a robotic arm equipped with a haptic feedback mechanism. The robotic arm can be remotely controlled by the surgeon and can provide a haptic interface for precise instrument manipulation. The haptic feedback function is specifically designed to enhance the surgeon's ability to perform delicate surgical procedures, thereby achieving controlled and accurate movements during the operation.

[0085] Postoperative verification: After the operation, the system can perform a postoperative verification process. The system uses an AR interface to compare the actual postoperative Kirschner wire position with a pre-calculated virtual Kirschner wire image. This comparison can serve as a verification mechanism to confirm the accuracy of bone fixation. Any deviations can be quantified and analyzed to provide key data that can be used for postoperative assessment and future surgical planning. This feature is important for the system's goal of ensuring optimal surgical results through accuracy and verification.

[0086] Materials and methods

[0087] The following is a description of some embodiments of the present invention.

[0088] 1. Preoperative Planning (as shown in Figure 3)

[0089] 1.1 Imaging Data Import

[0090] In some embodiments, the system's proprietary software is capable of importing patient-specific image data in different formats (e.g., DICOM, JPEG, and PNG). The software may employ advanced encryption protocols to ensure data security during the import process.

[0091] 1.2 Preoperative Planning with Two-Dimensional (2D) Medical Images

[0092] In some embodiments, the software allows for manipulation and annotation of 2D medical images, enabling surgeons to plan incisions, entry points, and instrument trajectories on a two-dimensional plane.

[0093] 1.3 Conversion of 3D Medical Images to 2D Medical Images

[0094] In some embodiments, advanced algorithms can convert 3D medical images into 2D medical images to facilitate planning that requires a 2D perspective. This is particularly useful for surgeries that traditionally use 2D images for planning.

[0095] 1.4 3D Medical Image Segmentation Specification 8 / 21 pages 11 CN 121774637 A

[0096] In some embodiments, the software uses machine learning algorithms to segment 3D medical images into different anatomical structures, and then converts the anatomical structures into 3D models for more detailed planning.

[0097] 1.5 3D Printing of Segmented Anatomical Structure Models

[0098] In some embodiments, the segmented 3D anatomical structure models can be exported in a format compatible with 3D printers, thereby facilitating the creation of physical models for preoperative practice and planning.

[0099] 1.6 Preoperative Matching of 3D Printed Model and Medical Image

[0100] In some embodiments, the system has an algorithm that matches the 3D printed model with the original 3D medical image to ensure that the physical model accurately represents the patient's anatomy.

[0101] 1.7 Planning of Bone Cutting Angles

[0102] In some embodiments, the surgeon can specify the angle of bone cutting during the operation. The software can simulate these cuts on the 3D model to provide visual guidance for the operation.

[0103] 1.8 Planning of Screw Insertion Parameters

[0104] In some embodiments, the software allows the surgeon to plan the location, angle, and depth of screw insertion. It can also provide recommendations for screw size based on the patient's bone density and the specific requirements of the operation.

[0105] 1.9 Planning of Kirschner Wire Parameters

[0106] In some embodiments, the surgeon can use the software to plan the diameter, length, and insertion angle of the Kirschner wire. These parameters can then be simulated on the 3D model for visual confirmation.

[0107] 1.10 Simulation of Postoperative Bone Position

[0108] In some embodiments, the software can simulate the expected position of the bones after surgery, enabling the surgeon to predict outcomes and make appropriate adjustments to the surgical plan.

[0109] 1.11 Simulation of Postoperative Kirschner Wire Angles

[0110] In some embodiments, the software can simulate the expected angles of the Kirschner wires after surgery to provide additional data that can be used to refine the surgical plan.

[0111] 2. Calibration and Setup (as shown in Figure 4)

[0112] 2.1 Calibration of Robotic Arms

[0113] In some embodiments, each robotic arm may undergo a rigorous multi-point calibration process. This may involve a series of test movements to calibrate the robotic arm's sensors, followed by a fine-tuning process to adjust the robotic arm's motors for maximum accuracy.

[0114] 2.2 Calibration of Navigation Sensors

[0115] In some implementations, the navigation sensor uses a combination of infrared and laser technology to achieve sub-millimeter accuracy. Calibration may include projecting a series of test patterns and adjusting sensor parameters based on feedback.

[0116] 2.3 Calibration of AR Headset

[0117] In some implementations, an AR headset (e.g., a head-mounted display) is calibrated using a proprietary algorithm that takes into account individual differences in eye distance and focal length. The calibration process may involve displaying a series of test images and adjusting display parameters to ensure optimal visual clarity and alignment with the user's field of vision.

[0118] 2.4 Calibration of AR Headset and Robotic Arm

[0119] In some implementations, a synchronous calibration process may be initiated to align the AR headset (e.g., a head-mounted display) with a robotic arm. This may involve a series of coordinated movements between the robotic arm and visual cues displayed on the AR headset. The algorithm can ensure that the movements of the robotic arm are accurately reflected in the AR display, thus achieving a seamless integration. Instruction manual 9 / 21 pages 12 CN 121774637 A

[0120] 2.5 Comprehensive Calibration: Navigation Sensors, Robotic Arm, and AR Headset

[0121] In some embodiments, a final comprehensive calibration step is performed to ensure that the navigation sensors, robotic arm, and AR headset (e.g., head-mounted display) are accurately aligned. This may involve a series of tests that utilize machine learning algorithms to optimize the interaction between these three components. The system can verify the calibration results by executing a series of simulation programs and adjusting any deviations in real time.

[0122] 3. Patient Positioning

[0123] The surgical procedure can begin with the patient being placed on the operating table with the assistance of a robotic arm, which can be programmed to align the patient according to the preoperative plan. Real-time data confirming alignment can be displayed on an AR headset (e.g., head-mounted display) worn by the surgical team. Subsequently, the incision site can be disinfected and marked, and the area can be scanned using navigation sensors to create a real-time digital map, which can be displayed on the AR headset for surgical guidance. Anesthesia can then be administered, and its effectiveness continuously monitored via sensors that input data into the AR headset, allowing the anesthesiologist to make precise adjustments as needed. Simultaneously, vital signs can be monitored through the AR interface, where the robotic system is programmed to alert the surgical team if any parameters deviate from safe limits.

[0124] The surgeon can utilize a navigation system to precisely mark the surgical site (assisted by a magnified enhanced view provided by the AR headset). Final preoperative scans can be displayed on the AR headset for final adjustments and final calibration of the robotic arm. The robotic arm can position sterilized surgical instruments within the operating room, and their positions can be tracked by the navigation system and displayed on the AR headset. Final communication checks can be performed, including communication with...Any remote consultant observing the surgical procedure through a synchronized AR interface ensures the entire surgical team is fully informed.

[0125] Emergency protocols outlining procedures for failure of the robot or AR system can be reviewed in the final stages of preparation. These protocols can be displayed as a checklist on the AR headset, and the robotic system can be programmed with fail-safe features for rapid response to various emergencies. This comprehensive approach ensures the surgical team is well-prepared and coordinated to leverage robotic and AR technologies to improve the accuracy and safety of the surgical procedure.

[0126] 4. AR Headset Initialization (as shown in Figure 5)

[0127] 4.1 AR Headset Calibration

[0128] In some implementations, the AR headset (e.g., head-mounted display) can be calibrated according to the surgeon's field of vision. This may involve a series of tests to ensure the digital overlay accurately matches the real world.

[0129] 4.2 Software Configuration

[0130] In some embodiments, the software of the AR headset is configured to display specific datasets and visual aids that can be used during surgery, such as vital signs, surgical plans, navigation data, etc.

[0131] 4.3 Synchronization with Robotic Arm

[0132] In some embodiments, the AR headset is synchronized with a robotic arm to ensure that any movements or adjustments made by the robotic system are accurately reflected in the AR display.

[0133] 4.4 Synchronization with Navigation

[0134] In some embodiments, the AR headset is also synchronized with a navigation system. This ensures that real-time tracking data from navigation sensors is accurately displayed on the AR headset.

[0135] 4.5 User Interface Customization

[0136] In some embodiments, the surgeon can customize the user interface of the AR headset, for example, by selecting the data and visual aids to be displayed during surgery. This may include switching between 2D and 3D views, adjusting the transparency level, setting data display preferences, etc.

[0137] 4.6 Voice Command Settings

[0138] In some embodiments, voice recognition software can be activated and calibrated to allow surgeons to control the AR headset and connected systems via voice commands. This enables hands-free operation, which is very useful for maintaining a sterile environment.

[0139] 4.7 Network Security Protocols

[0140] In some embodiments, security protocols are activated to ensure that data displayed by the AR headset is securely transmitted and to prevent unauthorized access. This may include encryption and multi-factor authentication measures, etc.

[0141] 4.8 Battery Life and Power Supply Verification

[0142] In some embodiments, the battery life of the AR headset is checked, and backup power is verified. This ensures that...The AR headset can operate continuously during the surgery.

[0143] 4.9 Emergency Override Procedure

[0144] In some embodiments, the emergency override procedure is reviewed, and the AR headset is programmed to include quick access commands that can disable or reset the system in case of software failure or other emergencies.

[0145] 4.10 Final System Check

[0146] In some embodiments, a final system check is performed to confirm that the AR headset, robotic arm, and navigation system are fully functional and synchronized. Any final adjustments can be made at this stage.

[0147] 4.11 Test Run with Simulated Data

[0148] In some embodiments, a test run is performed using simulated patient data to confirm that all systems are functioning as expected. This allows the surgical team to familiarize themselves with the AR headset's interface and controls.

[0149] 5 Navigation Initialization (as shown in Figure 6)

[0150] 5.1 Sensor Calibration

[0151] In some embodiments, the navigation sensors are calibrated to ensure accurate tracking of surgical instruments and patient anatomy. This involves a series of tests to verify the accuracy and responsiveness of the sensors.

[0152] 5.2 Data Integration with AR Headset

[0153] In some embodiments, navigation data is integrated with an AR headset to ensure that real-time tracking information can be accurately displayed on the surgeon's visual interface.

[0154] 5.3 Synchronization with Robotic Arm

[0155] In some embodiments, the navigation system is synchronized with a robotic arm to allow real-time tracking of robotic instruments. This ensures that the robotic arm moves according to the surgical plan displayed on the AR headset.

[0156] 5.4 Software Configuration

[0157] In some embodiments, the navigation software is configured to display specific tracking markers, areas, and other visual aids that can assist the surgeon during the procedure.

[0158] 5.5 Real-time Error Correction

[0159] In some embodiments, real-time error correction algorithms are activated. These algorithms can immediately correct any discrepancies between the navigation data and the actual position of the instruments or anatomical structures.

[0160] 5.6 Network Security Protocol

[0161] In some embodiments, security measures are taken to ensure that navigation data is securely transmitted and stored. This may include encryption and firewall protection. Specification 11 / 21 pages 14 CN 121774637 A

[0162] 5.7 Backup System Initialization

[0163] In some embodiments, a backup navigation system is initialized and kept in standby mode. This ensures that even if the primary system fails, the operation can continue without interruption.

[0164] 5.8 User Interface Customization for Optical Data

[0165] In some embodiments, the surgeon can customize the user interface to select the navigation data displayed during surgery. This may include options for different tracking markers and visual aids.

[0166] 5.9 Optical Data Recording

[0167] In some embodiments, the system is configured to record all navigation data for postoperative analysis and quality improvement. This data can be used to study and refine future surgical plans.

[0168] 5.10 Emergency Override for Navigation

[0169] In some embodiments, an emergency override procedure specific to the navigation system is reviewed. Quick access commands can be programmed into the system for rapid deactivation or reset in an emergency.

[0170] 5.11 Test Run of Navigation

[0171] In some embodiments, a test run is performed to simulate the surgical procedure to confirm that the navigation system is fully operational and accurately synchronized with the AR headset and robotic arm.

[0172] 5.12 Final System Check of Navigation

[0173] In some embodiments, a final system check can confirm that all components of the navigation system have been calibrated, synchronized, and are ready for surgery. Any final adjustments can be made at this stage.

[0174] Figure 7 illustrates a robot-assisted AR navigation orthopedic surgical system according to one embodiment. Figure 8 illustrates a tracking element and its orientation according to one embodiment. Figure 9 illustrates an overlay and display on an AR head-mounted device according to one embodiment. Figure 10 illustrates empirical postoperative Kirschner wire positioning (for a pre-planned model) according to one embodiment. Further details related to Figures 7 through 10 are provided below.

[0175] 6 Robot-Assisted Engagement

[0176] The initial setup of the robot-assisted system may involve a meticulous calibration process. The robotic arms may be calibrated to ensure their precise movement and consistency with the surgical plan. This may involve a series of tests to verify their range of motion, speed, and accuracy. Surgical instruments may be securely attached to these robotic arms, and their suitability may be verified by real-time data displayed on an AR head-mounted device (e.g., a head-mounted display). The robotic arms may then be synchronized with the AR head-mounted device and navigation system to ensure accurate real-time tracking and adjustment. The haptic feedback mechanism may be calibrated to provide tactile feedback to the surgeon, thereby enhancing the surgical experience. Safety protocols can be activated to set predefined limits and force restrictions, which can be displayed on the AR head-mounted device for surgeons to view.

[0177] The system may also include features to enhance control and safety. Voice recognition software can be calibrated to enable surgeons to operate the robotic arm via voice commands, enabling hands-free operation. Emergency stop buttons and over-the-air commands can be programmed into the system to immediately stop all robotic activity in an emergency. Data logging and monitoring functions can be activated to record all actions and operations for postoperative analysis and quality improvement. Power and backup can be checked.Options and verification of battery life to ensure uninterrupted operation.

[0178] The system may undergo rigorous testing and final inspection. Test runs can simulate the surgical procedure to confirm that the robotic arm is fully operable and accurately synchronized with the AR head-mounted device and navigation system. The user interface can be customized to display selected robot data during the surgery, including various tracking markers and visual aids. Provisions for remote expert consultation can be established to allow experts to view the surgical procedure in real time. The algorithm controlling the movement of the robotic arm can be reviewed to ensure that it is consistent with the surgical plan and safety protocols. A backup robotic arm can be initialized and kept in standby. The final system check can confirm that all components have been calibrated, synchronized and ready for surgery. Any final adjustments can be made at this stage to ensure that the system is ready for surgery.

[0179] 7 AR Visualization Display

[0180] 7.1 Activation of AR Visualization Display

[0181] Navigation: Initial tracking of surgical instruments and patient anatomy.

[0182] Robotic arm: Standby to be activated, consistent with the AR visualization.

[0183] 7.2 Overlay of preoperative patient anatomical images

[0184] Navigation: Ensures the overlay is accurately aligned with the patient's actual anatomical structures.

[0185] Robotic arm: Uses the overlay as a guide for initial positioning.

[0186] 7.3 Overlay of preoperative anatomical sites

[0187] Navigation: Provides real-time tracking to ensure the overlay remains aligned during surgery.

[0188] Robotic arm: Makes real-time adjustments based on the specific anatomical site of interest.

[0189] 7.4 Overlay of preoperative instrument paths

[0190] Navigation: Tracks instruments to ensure they follow the pre-planned path.

[0191] Robotic arm: Navigates instruments along the overlaid path, providing tactile feedback to the surgeon.

[0192] 7.5 Overlay of preoperative cutting paths

[0193] Navigation: Monitors the cutting instruments to ensure they remain on the planned path.

[0194] Robotic arm: Assists in guiding the cutting instrument to move precisely along the superimposed path.

[0195] 7.6 Superimposed preoperative screw path

[0196] Navigation: Tracks the screw and target insertion point for real-time alignment.

[0197] Robotic arm: Precisely inserts the screw according to the superimposed path.

[0198] 7.7 Superimposed preoperative Kirschner wire path

[0199] Navigation: Monitors the Kirschner wire and its target path to ensure accurate placement.

[0200] Robotic arm: Guides the insertion of the Kirschner wire according to the superimposed path.

[0201] 7.8 Real-time instrument tracking

[0202] Navigation: Continuously updates the AR visualization display effect and obtains the real-time position information of the instrument.

[0203] Robotic arm: Adjusts its movement according to the real-time tracking data.

[0204] 7.9 Real-time adaptation

[0205] Navigation: Updates the AR overlay if the anatomical structure changes.

[0206] Robotic Arm: Adapts to new surgical paths or adjustments in real time.

[0207] 7.10 Control of Transparency and Opacity

[0208] Navigation: Does not directly participate, but ensures accurate overlay for efficient use of this function.

[0209] Robotic Arm: Can pause or slow down movement during adjustments to transparency and opacity.

[0210] 7.11 Multi-Layer Visualization

[0211] Navigation: Tracks multiple layers for accurate overlay and visualization.

[0212] Robotic Arm: Can switch tasks based on layers of interest.

[0213] 7.12 Real-Time Feedback and Annotation

[0214] Navigation: Provides data for real-time annotation. Specification 13 / 21 pages 16 CN 121774637 A

[0215] Robotic Arm: Can be adjusted according to annotations, such as changing the grip or angle.

[0216] 7.13 Dynamic Zoom and Zoom

[0217] Navigation: Adjusts tracking parameters to match the zoom level.

[0218] Robotic Arm: The movement ratio can be adjusted to match the zoom level.

[0219] 7.14 User Interface Customization for AR Data

[0220] Navigation: Provides customizable data sources that can be selected for display.

[0221] Robotic Arm: Adapts to the surgeon's customized interface settings, such as changing the displayed data or the sensitivity of controls.

[0222] 8 Real-time Instrument Tracking

[0223] The surgical procedure can begin with the initialization of the instrument tracking system. The navigation system can be activated to track surgical instruments in real time, and this tracking data can be synchronized with the robotic arm. A detailed calibration process can be performed to align the tracking marks on the surgical instruments with the navigation system. This ensures the accuracy of instrument positioning and allows for real-time adjustments. The robotic arm can be recalibrated to further align with the tracking system. Each surgical instrument can be uniquely identified by the navigation system, enabling the robotic arm to distinguish between different instruments, thereby achieving precise manipulation and control. As the surgery progresses, real-time monitoring and alerts become crucial. The navigation system can continuously monitor any deviations from the planned trajectory and issue real-time alerts. These alerts may trigger the robotic arm to automatically adjust its position or pause the surgery to ensure it proceeds as planned. The system can also monitor the speed and direction of instrument movement, enabling the robotic arm to make corresponding adjustments. Surgeons can flexibly set user-defined tracking parameters (e.g., sensitivity levels), and the robotic arm can adjust its movement according to these customized settings. This real-time monitoring and adaptability can significantly improve the accuracy and safety of the surgery.

[0224] Data logging and integration with AR can be some key features. All tracking data (e.g., including deviations, speeds, etc.) can be monitored.The degree, direction, etc. can all be recorded for future analysis and quality improvement. The robotic arm can also use this data to record its movement. Real-time tracking data can be seamlessly integrated into the AR display, providing surgeons with a comprehensive enhanced view of instrument positions, thereby enhancing situational awareness. The system can measure the forces applied by the instruments, and the robotic arm can use this real-time force feedback to adjust the forces applied at different stages of the surgery.

[0225] The system can be designed to be adaptive and dynamically coordinated. It can track multiple instruments simultaneously and coordinate their movements through the robotic arm to prevent collisions and ensure efficient procedures. The tracking system and robotic arm can dynamically adapt to different stages of the surgery (e.g., incision, dissection, closure, etc.) and can optimize their functionality for each stage. Real-time tracking can be continuously compared with the preoperative plan to ensure consistency and can be adjusted accordingly. If any tracking error or system failure occurs, an emergency stop can be initiated, and the navigation system and robotic arm can be quickly recalibrated to resume the surgical procedure. This ensures that the surgical team can immediately understand the position and movement of the instruments, thus contributing to a safer and more efficient surgical procedure.

[0226] 9 Real-time Adjustment

[0227] 9.1 Real-time Anatomical Change Detection

[0228] In some embodiments, the navigation system is equipped with sensors that continuously monitor any changes in the patient's anatomy. These changes may be due to factors such as muscle relaxation or fluid transfer. The robotic arm can be programmed to automatically adjust its position and movement to adapt to these anatomical changes to ensure the surgical plan remains accurate.

[0229] 9.2 Dynamic Replanning

[0230] In some embodiments, the system can allow dynamic replanning if there is a significant deviation from the initial surgical plan (e.g., unexpected bleeding or discovery of unforeseen anatomical structures). The robotic arm can pause its movement, giving the surgical team time to reassess and modify the surgical plan. Once a new surgical plan is determined, the robotic arm can resume operation according to the updated plan.

[0231] 9.3 Real-time Risk Assessment

[0232] In some embodiments, the system employs machine learning algorithms to continuously assess risk based on real-time data. This can include monitoring proximity to critical anatomical structures (such as nerves or blood vessels). The robotic arm can be programmed to slow down or pause its movement and await further instructions from the surgical team if a potential risk is detected.

[0233] 9.4 Surgeon Alert System

[0234] In some embodiments, the system is designed to send real-time alerts to the surgeon via an AR interface with auditory or visual cues. These alerts may be triggered by unexpected anatomical changes, equipment malfunctions, or other events that may occur during the procedure.The challenge triggers. The robotic arm can stop moving and wait for further instructions from the surgeon to ensure patient safety.

[0235] 9.5 Real-time Decision Support

[0236] In some embodiments, the AR interface is capable of displaying decision support information. This may include alternative surgical paths, instrument options, or the probability of success of different surgical methods. The robotic arm can be programmed to adjust its movement according to any changes to the surgical plan through this decision support system.

[0237] 9.6 Adaptive Tactile Feedback

[0238] In some embodiments, the robotic arm is equipped with a tactile feedback mechanism that can adapt to surgical conditions in real time. For example, if the surgeon approaches a sensitive anatomical structure, the tactile feedback can provide greater resistance to alert the surgeon to operate with caution.

[0239] 9.7 Real-time Surgical Annotation

[0240] In some embodiments, the surgeon can make real-time annotations directly on the AR display. These annotations can be used to mark incision points, highlight areas of interest, or indicate areas to be avoided. The navigation system can track these annotations, and the robotic arm can adjust its movement accordingly to be consistent with these surgical annotations.

[0241] 9.8 Real-time Data Recording for Adjustments

[0242] In some embodiments, all adjustments made during the procedure are recorded in real time. This data can be securely stored and used for postoperative analysis, quality improvement, and optimization of future surgical plans. It can also be used for educational and training purposes.

[0243] 9.9 Real-time Communication with the Surgical Team

[0244] In some embodiments, the system is equipped with a communication module that allows real-time dialogue between the surgeon and the surgical team. This is particularly useful for discussing real-time adjustments to the surgical plan. The robotic arm can be designed to be controlled by multiple team members (if necessary) to ensure a collaborative approach.

[0245] 9.10 Real-time Remote Expert Consultation

[0246] In some embodiments, the system is able to connect to remote experts in real time. These experts can view the AR display and provide immediate guidance or supplementary opinions. The robotic arm can be remotely adjusted based on expert advice to ensure a high level of surgical care.

[0247] 9.11 Real-time Overlay Adjustment

[0248] In some embodiments, the surgeon can adjust the AR overlay in real time to better adapt to changing surgical conditions. For example, a surgeon may choose to highlight one or more anatomical structures or pathways. The robotic arm can adjust its movements based on these new overlays to align with the surgical plan.

[0249] 9.12 Real-time Instrument Switching

[0250] In some embodiments, the system is designed to allow for rapid and precise switching of surgical instruments as the surgery progresses. (Surgical manual, pages 15 / 21, 18 CN 121774637 A)Instruments. The robotic arm can achieve this through a variety of readily available instruments and can quickly and accurately change or transform them according to the surgeon's instructions.

[0251] 9.13 Real-time energy source management

[0252] In some embodiments, if the surgery involves the use of energy sources (e.g., electrocautery or laser), the system is able to manage these energy sources in real time. The robotic arm can be configured to hold and manipulate these energy sources and adjust the settings of the energy sources as needed based on real-time data.

[0253] 9.14 Real-time fluid management

[0254] In some embodiments, the system is also able to manage real-time fluid requirements, such as aspiration or flushing. The robotic arm can be equipped with instruments for performing these tasks and can be activated or adjusted in real time according to the surgical situation.

[0255] 10 Instrument navigation and manipulation

[0256] 10.1 Predefined trajectory tracking

[0257] In some embodiments, the robotic arm follows a pre-planned surgical instrument trajectory to ensure that the instruments travel along a predetermined path.

[0258] 10.2 Real-time Trajectory Modification

[0259] In some embodiments, the robotic arm can modify the trajectory of surgical instruments in real time based on feedback from the navigation system and AR display.

[0260] 10.3 Instrument Exchange Mechanism

[0261] In some embodiments, the robotic arm is equipped with an instrument exchange mechanism that allows for rapid and aseptic replacement of surgical instruments during surgery.

[0262] 10.4 Force Adjustment

[0263] In some embodiments, the robotic arm can adjust the force applied by the surgical instruments, enabling the surgeon to perform more precise operations.

[0264] 10.5 Depth Control

[0265] In some embodiments, the robotic arm can control the insertion depth of instruments based on real-time feedback from the navigation system and preoperative planning data.

[0266] 10.6 Angular Accuracy

[0267] In some embodiments, the robotic arm can rotate surgical instruments to precise angles, thereby facilitating the completion of complex surgeries requiring precise angles.

[0268] 10.7 Tactile Feedback Loop

[0269] In some embodiments, the robotic arm provides tactile feedback to the surgeon, enabling them to “feel” the resistance and other tactile sensations of the tissue (biological tissue) during instrument manipulation.

[0270] 10.8 Multi-Instrument Coordination

[0271] In some embodiments, the robotic arm can coordinate the movement of multiple instruments simultaneously to facilitate complex surgeries requiring the simultaneous use of multiple instruments.

[0272] 10.9 Emergency Retraction

[0273] In some embodiments, the robotic arm is programmed to quickly retract surgical instruments to a safe position in case of an unexpected situation.

[0274] 10.10 Fine-tuning Speed ​​Control

[0275] In some embodiments, the robotic arm can provide fine-tuning speed control for instrument movement, enabling the surgeon to...Page 16 / 21 19 CN 121774637 A Adjust the speed according to the specific requirements of the surgical procedure being performed.

[0276] 10.11 User Customization

[0277] In some embodiments, the surgeon can customize the control settings of the robotic arm, for example, adjust the responsiveness and speed according to personal preferences.

[0278] 10.12 Automated Suturing

[0279] In some embodiments, the robotic arm can perform automated suturing under the control of the surgeon to ensure consistent and accurate suturing.

[0280] 10.13 Real-time Sterilization

[0281] In some embodiments, the robotic arm is equipped with a mechanism for real-time sterilization of surgical instruments to reduce the risk of infection.

[0282] 10.14 Energy Source Management

[0283] In some embodiments, the robotic arm can hold and manipulate an energy source (e.g., an electrocautery device or laser device) to adjust the energy in real time according to surgical needs.

[0284] 10.15 Real-time Data Recording

[0285] In some embodiments, all instrument movements and operations performed by the robotic arm are recorded in real time for postoperative analysis and quality improvement.

[0286] 11. Example of Tibial Deformity Osteotomy Application

[0287] 11.1 Incision and Exposure

[0288] A longitudinal incision is made along the medial side of the tibia. The robotic arm provides assistance by holding retractors or other instruments as needed, based on real-time image data displayed on an AR head-mounted device (e.g., a head-mounted display).

[0289] 11.2 Preoperative Fixation Planning in AR

[0290] The AR head-mounted device overlays pre-planned screw and Kirschner wire insertion positions onto the surgeon's field of vision to assist in precise preoperative planning.

[0291] 11.3 Osteotomy Site Recognition in AR

[0292] The AR head-mounted device displays the pre-planned osteotomy site, providing real-time visual guidance to the surgeon.

[0293] 11.4 Kirschner wire insertion for osteotomy guidance

[0294] Kirschner wires are manually inserted into the pre-planned osteotomy site. The navigation system provides real-time tracking data to ensure accurate placement.

[0295] 11.5 Osteotomy execution

[0296] The surgeon performs the osteotomy using a surgical chisel or vibrating saw. The navigation system provides real-time feedback on alignment and incision depth.

[0297] 11.6 Wedge resection and tibial repositioning

[0298] The wedge bone is removed, and the tibia is repositioned. An AR headset provides real-time feedback on alignment, confirming its consistency with the preoperative plan.

[0299] 1.7 Screw fixation

[0300] Guide holes for screw insertion are drilled manually. A robotic arm stably holds the drill bit to provide assistance, while the navigation system confirms the accuracy of the drilling angle and depth.

[0301] 11.8 Final alignment confirmation and verification

[0302] The repositioning was confirmed by fluorescence fluoroscopy and AR overlay. The repositioning was further verified by comparing it with the position of the A needle on page 17 / 21 of the instruction manual 20 CN 121774637 in the preoperative plan.

[0303] 11.9 Wound Closure and Hemostasis

[0304] Hemostasis was achieved by electrocautery, and the wound was then closed in layers. The robotic arm could hold sutures or other instruments as needed.

[0305] 11.10 Postoperative Procedure

[0306] Postoperative X-rays were taken to confirm the repositioning and fixation. The patient was then transferred to the recovery room according to standard procedures.

[0307] 11.11 Data Recording and Quality Assurance

[0308] All surgical steps, instrument manipulation, and deviations from the preoperative plan were recorded in real time for subsequent quality assurance and postoperative review. 12. Postoperative Management and Quality Assurance Using Robotic Arms, AR Headsets, and Navigation

[0309] 12.1 Real-time Postoperative Assessment

[0310] Surgeons can use AR headsets to view the overlay of postoperative anatomy with the preoperative plan in real time to confirm the success of osteotomy and fixation. Robotic arms can assist in keeping the AR headset in the optimal position for observation.

[0311] 12.2 Hardware Verification

[0312] Navigation systems can be used to scan the surgical site to verify the precise placement of screws, Kirschner wires, and other fixation hardware. Data can be displayed on the AR headset for immediate viewing.

[0313] 12.3 Tissue Integrity Check

[0314] AR headsets can overlay preoperative scans of soft tissue with the current state to help surgeons identify any unexpected tissue damage. Robotic arms can assist in operation by holding navigation sensors (which transmit data to the AR headset).

[0315] 12.4 Hemostasis Confirmation

[0316] The navigation system can scan the surgical site to detect any signs of residual bleeding. The AR headset can display this data, and the robotic arm can assist in holding the hemostatic agent for immediate use.

[0317] 12.5 Data Recording for Quality Assurance

[0318] All surgical steps, instrument operations, and deviations from the preoperative plan can be recorded by the system in real time. The robotic arm can assist in the automated recording process by triggering the end of each surgical step.

[0319] 12.6 Postoperative Radiographic Image Alignment

[0320] Final radiographic images can be taken, and the navigation system can assist in aligning the images with preoperative and intraoperative data. The AR headset can display a comprehensive view for the surgeon to perform final verification.

[0321] 12.7 Transition to Recovery Phase

[0322] The patient can be carefully transferred to the post-anesthesia intensive care unit. The robotic arm can assist in holding and transferring any necessary postoperative monitoring equipment, while the AR headset can display the patient's vital signs and other relevant data for the surgeon's reference.View.

[0323] 13 Additional Features

[0324] 13.1 Remote Expert Consultation

[0325] In some embodiments, the system may be equipped with a secure and encrypted real-time video conferencing channel to facilitate consultation by remote experts. These experts can view the real-time feedback of the AR overlay and make adjustments in real time.

[0326] 13.2 Surgical Progress Replay

[0327] In some embodiments, all surgical steps can be recorded (e.g., recorded at 4K resolution) and replayed in a virtual reality environment for training and quality improvement.

[0328] 13.3 Continuous Performance Monitoring

[0329] In some embodiments, all data can be recorded in a database that conforms to the Health Insurance Portability and Accountability Act (HIPAA) standards for future analysis, research, and system optimization.

[0330] The robot-assisted AR navigation surgical system in some embodiments may represent a significant advancement compared to existing products in the field of orthopedic surgery. Specifically, in some implementations, the combination of multiple technologies with precise navigation brings some unique characteristics and advantages, such as: 1. Precise AR navigation: Some implementations include real-time navigation. This means that the virtual information superimposed on the surgeon's field of vision can be continuously updated according to the precise position and orientation of the surgical instruments. This real-time feedback ensures accuracy and enables the surgeon to make informed decisions on the spot.

[0331] 2. Combination of robotics and AR: Some implementations uniquely combine robot-assisted surgical systems with AR applications. The robotic arm can not only assist in holding and manipulating instruments, but also interact seamlessly with AR visualizations. This combination can provide surgeons with a comprehensive and dynamic set of surgical tools.

[0332] 3. Enhanced tactile feedback and remote control: Some implementations emphasize tactile feedback. Surgeons can remotely control the robotic arm with enhanced tactile feedback, enabling fine manipulation and precise tissue interaction. This tactile connection can improve the precision of surgery.

[0333] 4. Reduced surgical trauma: The combination of navigation and AR technology may help reduce or minimize surgical trauma. Surgeons can navigate more accurately, reducing the need for extensive tissue dissection, thereby reducing incision size, blood loss, and / or recovery time.

[0334] 5. Comprehensive training capabilities: Some implementations can serve as advanced educational platforms. They can not only simulate surgical procedures but also integrate real-time data feedback and AR visualization. This comprehensive training approach can provide medical students and experienced surgeons with a more immersive learning experience.

[0335] 6. Real-time adjustments: Some implementations allow for real-time adjustments. Surgeons can adjust based on dynamic conditions and anatomical structures.Surgical approaches can be adapted to changing and unforeseen challenges while maintaining a clear AR overlay to provide guidance.

[0336] 7. Multidimensional data integration: Some implementations combine patient-specific imaging data, surgical plans, real-time instrument tracking, and AR visualizations. This integrated approach provides a comprehensive view, enabling surgeons to make more informed decisions. In summary, some implementations of robot-assisted AR-guided surgical systems may represent significant advancements compared to existing or previous products. This system synergistically combines robotics, AR, and navigation technologies to provide unprecedented precision, real-time feedback, comprehensive training, and promises to revolutionize the outcomes and techniques of orthopedic surgery.

[0337] 8. Example of Kirschner wire osteotomy reduction: In one example, during the preoperative phase, an AR head-mounted device enhances surgical planning by overlaying predetermined locations for screw and Kirschner wire insertion onto the surgeon's field of vision. During the procedure, the Kirschner wire is manually inserted at the designated osteotomy site, while the navigation system provides real-time tracking data to ensure accurate placement. Then, osteotomy is performed using a surgical chisel or vibratory saw, guided by real-time feedback from the navigation system regarding the alignment and depth of the incision. Finally, the alignment after reduction is verified by fluorescence fluoroscopy and AR overlay, and further verified by cross-referencing it with the pre-planned Kirschner wire positions.

[0338] In some examples, embodiments of the invention may be adopted or used by one or more of the following, for example:

[0339] 1. Hospitals and surgical centers: Hospitals and surgical centers specializing in orthopedic surgery may adopt the system to improve the accuracy and success rate of various surgeries, including joint replacement and spinal surgery.

[0340] 2. Orthopedic surgeons: Orthopedic surgeons are generally committed to improving surgical techniques and outcomes. The system's real-time feedback, precise navigation, and AR-assisted capabilities make it an ideal tool for surgeons pursuing excellence in their field. Specification 19 / 21 pages 22 CN 121774637 A

[0341] 3. Medical training institutions: Medical schools, teaching hospitals, and training institutions will recognize the potential of the system as an educational tool. It provides a safe environment for medical students and residents to practice surgery, improve their skills, and gain valuable experience under AR guidance.

[0342] 4. Research institutions: Research institutions that study surgical techniques, patient treatment outcomes, and orthopedic advancements will find the system's data acquisition capabilities invaluable. The detailed information or insights it provides can help develop innovative surgical methods and practices.

[0343] 5. Surgeons in remote or underserved areas: The system allows for remote guidance and assistance, making it ideal for surgeons practicing in remote or underserved areas. They can benefit from expert guidance.(Experts do not need to be physically present), ultimately improving patient care in areas lacking expertise.

[0344] 6. Technology-driven surgeons: Surgeons who want to be at the forefront of cutting-edge technologies and techniques are likely to show great interest in the system. Surgeons committed to staying ahead of the field and embracing innovation will find the system highly aligned with their goals.

[0345] 7. Complex orthopedic surgeries; the potential applications of the system can be extended to complex orthopedic surgeries (e.g., minimally invasive joint replacements, complex spinal surgeries, and revision surgeries). Surgeons performing these surgeries can leverage the technology to improve the accuracy and success rate of the surgery.

[0346] 8. Continuous advancement: as the system develops, its potential applications are expected to expand to other surgical fields beyond orthopedics. The combination of robotics, AR technology, and precision navigation technology is expected to benefit other surgical fields.

[0347] The system of some embodiments of the present invention can provide one or more of the following advantages:

[0348] • Enhanced accuracy: Surgeons can achieve high accuracy and improve surgical outcomes.

[0349] • Reduced trauma: Patients can experience less trauma and pain and recover faster.

[0350] • Comprehensive training: Medical students and surgeons can practice surgery in a risk-free virtual environment.

[0351] • Real-time adjustment: Surgeons can adjust surgical methods based on real-time data to ensure optimal results.

[0352] The system of some embodiments of the present invention can provide one or more other advantages.

[0353] Overall, the potential of robot-assisted AR navigation surgical systems is encouraging. For example, it can improve surgical precision, enhance medical training, and improve patient care. Furthermore, it may be used to reshape the landscape of orthopedic surgery and even the broader medical field.

[0354] In some embodiments, a robot-assisted augmented reality (AR) navigation orthopedic surgical system is provided. The system includes: a robotic arm configured to precisely manipulate surgical instruments based on preoperative planning and real-time navigation data; an AR head-mounted device configured to display an AR overlay to provide the operator with real-time visual guidance and surgical indicators; a navigation device; and a plurality of optical markers configured to serve as reference points for the navigation device. The navigation device is configured with the plurality of optical markers to track the anatomical structures of the test subject and the real-time position of the surgical instruments. Optionally, the robotic arm is configured for force modulation and / or fine-tuning speed control. Optionally, the robotic arm is configured to receive control signals from a computing device and / or position data from a navigation device. Optionally, the robotic arm includes multiple force feedback sensors. Optionally, the robotic arm is configured to have multiple (e.g., six) degrees of freedom. Optionally, the robotic arm is configured to have high (e.g., 0.1 mm) positional accuracy. Optionally, the AR headset wirelessly connects to the computing device and receives tracking data from the navigation device. Optionally, the AR headset...The device is configured with a 110-degree field of view. Optionally, the AR headset is configured with a 2K resolution per eye. Optionally, the AR headset is configured with built-in eye tracking. Optionally, the navigation device is configured to send tracking data to a computer system, which then updates the AR headset and the robotic arm. Optionally, the navigation device is configured with sub-millimeter tracking accuracy. Optionally, the navigation device is configured with a 60Hz update rate. Optionally, the navigation device is configured with multi-marker tracking capability. Optionally, multiple optical marks are attached to surgical instruments and the subject's anatomy and tracked by the navigation device. Optionally, the multiple optical marks have an infrared reflective coating. Optionally, the multiple optical marks are made of a sterilizable material. Optionally, the multiple optical marks are formed into different shapes for different applications. Optionally, the system acquires preoperative or intraoperative images, including two-dimensional (2D) medical images such as X-ray images, MRI scans, or CT scans. Optionally, the system also includes a computing device configured to convert the 2D preoperative or intraoperative images into one or more three-dimensional (3D) images. Optionally, the conversion process includes layer stacking, pixel density analysis, or volumetric rendering to generate the 3D image. Optionally, the generated 3D image is displayed on a high-resolution display for preoperative examination or intraoperative guidance. Optionally, the generated 3D image is overlaid onto the operator's field of vision via an AR head-mounted device. Optionally, the 3D image is synchronized with the navigation device to provide real-time tracking and guidance during the surgery. Optionally, the 3D image is configured to guide the movement of the robotic arm to provide the operator with a comprehensive understanding of the surgical site and enable more precise instrument manipulation. Optionally, the navigation device is integrated into the computing device and configured to track multiple fiducial markers placed on the test subject prior to surgery. Optionally, the computing device is configured to monitor the position of the multiple fiducial markers in real time. Optionally, real-time position data of multiple reference markers are synchronized with a 3D image of the surgical site. Optionally, the 3D image is displayed on a monitor and can be overlaid onto the operator's field of vision via an AR head-mounted device. Optionally, the movement of the robotic arm is guided based on real-time tracking data to allow dynamic adjustments and precise instrument manipulation according to the current position of the test subject. Optionally, the computing device is connected via a secure router for wired and wireless network connections, thereby allowing real-time data sharing and remote consultation. While some of the above embodiments are described with reference to orthopedic surgery, the application of the system of the present invention is not limited to orthopedic surgery.

[0355] Those skilled in the art will understand that variations and / or modifications can be made to the embodiments described and / or shown herein.Modifications have been made to provide other implementations. Therefore, the implementations described and / or illustrated herein should be considered illustrative rather than restrictive in all respects. Example optional features of the invention are provided in the summary and detailed description. Some embodiments of the invention may include one or more of these optional features (some of which are not specifically shown in the figures). Some embodiments of the invention may not include one or more of these optional features (some of which are not specifically shown in the figures). Instruction manual 21 / 21 page 24 CN 121774637 A Figure 1 Figure 2 Instruction manual Appendix 1 / 8 page 25 CN 121774637 A Figure 3 Instruction manual Appendix 2 / 8 page 26 CN 121774637 A Figure 4 Instruction manual Appendix 3 / 8 page 27 CN 121774637 A Figure 5 Instruction manual Appendix 4 / 8 page 28 CN 121774637 A Figure 6 Instruction manual Appendix 5 / 8 page 29 CN 121774637 A Figure 7 Instruction manual Appendix 6 / 8 page 30 CN 121774637 A Figure 8 Instruction manual Appendix 7 / 8 page 31 CN 121774637 A Figure 9 Figure 10 Instruction manual Appendix 8 / 8 page 32 CN 121774637 A Abstract A system for assisting surgical procedures includes a robotic device, a wearable display device, a navigation system and a control system. The robotic device is configured to hold and / or manipulate a surgical instrument to perform a surgical operation at a target site of a patient. The wearable display device is configured to overlay digital content onto a user's field of view, the digital content including information related to the surgical operation. The navigation system isconfigured to obtain tracking data of the target site and the surgical instrument based at least in part on tracking a position and / or orientation of the target site and the instrument. The control system is configured to control operation of the robotic device and / or the wearable display device based at least in part on the tracking data.

Claims

1. A system for assisting in surgical procedures, characterized in that, include: Robotic devices that can be operated to hold and / or manipulate instruments in order to perform surgical procedures at target sites on a patient; A wearable display device operable to overlay digital content onto the user's field of vision, the digital content including information related to the surgical procedure; A navigation system operable to obtain tracking data of the target site of the patient and the device, at least in part, based on tracking the position and / or orientation of the device. as well as A control system operable to control the operation of the robotic device and / or the wearable display device based at least in part on the tracking data.

2. The system according to claim 1, characterized in that, in, The control system is operable to control the operation of the robotic device and / or the wearable display device based at least in part on the tracking data and surgical planning data for the surgical procedure.

3. The system according to claim 2, characterized in that, in, The surgical planning data is determined at least in part based on the patient's anatomy and / or physiological condition.

4. The system according to any one of claims 1 to 3, characterized in that, in, The control system is operable to control the operation of the robotic device at least by influencing the movement of the robotic device.

5. The system according to any one of claims 1 to 3, characterized in that, in, The control system is operable to control the operation of the wearable display device, at least by controlling the digital content superimposed on the wearable display device.

6. The system according to any one of claims 1 to 3, characterized in that, in, The control system is at least partially integrated with the robotic device.

7. The system according to any one of claims 1 to 3, characterized in that, in, The control system is at least partially integrated with the wearable display device.

8. The system according to any one of claims 1 to 3, characterized in that, in, The control system is at least partially integrated with the navigation system.

9. The system according to any one of claims 1 to 3, characterized in that, in, The robotic device is operable to manipulate surgical instruments for performing the surgical procedure.

10. The system according to any one of claims 1 to 3, characterized in that, in, The robotic device is operable to hold and / or manipulate an imaging device for performing imaging to facilitate the surgical procedure.

11. The system according to any one of claims 1 to 3, characterized in that, in, The robotic device includes a robotic arm.

12. The system according to claim 11, characterized in that, in, The robotic arm has six degrees of freedom.

13. The system according to claim 11, characterized in that, in, The robotic arm includes a tactile feedback mechanism.

14. The system according to claim 11, characterized in that, in, The robotic arm includes a force feedback sensor.

15. The system according to claim 11, characterized in that, in, The robotic arm includes a depth measuring mechanism.

16. The system according to any one of claims 1 to 3, characterized in that, in, The wearable display device includes a head-mounted display device.

17. The system according to claim 16, characterized in that, in, The head-mounted display device includes an eye-tracking device for tracking the user to influence the digital content.

18. The system according to any one of claims 1 to 3, characterized in that, in, Information related to the surgical procedure includes real-time guidance information to guide the execution of the surgical procedure.

19. The system according to any one of claims 1 to 3, characterized in that, in, Information related to the surgical procedure includes the patient’s real-time anatomical and / or physiological measurements.

20. The system according to any one of claims 1 to 3, characterized in that, in, Information related to the surgical procedure includes a virtual representation of the patient's anatomical structure.

21. The system according to any one of claims 1 to 3, characterized in that, in, The navigation system includes one or more navigation sensors, each of which is operable to perform tracking.

22. The system according to any one of claims 1 to 3, characterized in that, in, The system also includes a plurality of markers disposed on the patient and the device; and the navigation system is operable to track the target site of the patient and the position and / or orientation of the device based at least in part on the plurality of markers.

23. The system according to claim 22, characterized in that, in, The markings include optical markings attached to the patient and the device.

24. The system according to claim 23, characterized in that, in, The optical markings include an infrared reflective coating.

25. The system according to claim 23, characterized in that, in, The optical markers can be sterilized.

Images