Method and system for proposing spinal rods for orthopedic surgery using augmented reality

Abstract

11. A method for assisting in orthopaedic surgery, comprising: capturing a sequence of images such that a field of view captures images of a plurality of screw extenders, each screw extender holding a pedicle screw, the plurality of screw extenders being arranged in a surgical incision for the orthopaedic surgery; displaying images of the captured images to provide a live video feed; detecting the plurality of screw extenders based on the sequence of captured images; calculating an orientation and position of the detected plurality of screw extenders; calculating a 3D position of a screw head of each pedicle screw based on the orientation and position; and projecting and displaying with a graphical element on a display device each of the calculated 3D positions of the plurality of screw heads at positions corresponding to positions of the screw heads projected onto a currently displayed image of the live video feed.

Description

Technical Field

[0001] [Cross - reference to Related Applications]

[0001] This invention claims the priority of the specification of International Patent Application No. PCT / IB2021 / 051694 filed on March 1, 2021 and the specification of International Patent Application No. PCT / IB2021 / 056242 filed on July 12, 2021, and incorporates them herein by reference in their entirety.

[0002]

[0002] This invention relates to the field of orthopedic surgery using augmented reality or mixed reality. More specifically, it relates to methods, systems, and devices for using augmented reality or mixed reality to assist or facilitate a surgeon in performing orthopedic surgery, specifically for proposing stabilization rods of different types and shapes for spinal fixation and other types of orthopedic surgeries.

Background Art

[0003]

[0003] In the field of orthopedics, and in the field of tools and systems for orthopedic surgery, more specifically for implants for spinal fixation, pedicle screws are used to attach to vertebrae with bone anchors through an incision site on the patient's back. After several pedicle screws are attached to different vertebrae, the heads of these pedicle screws are connected to each other via a rod - type or bar - type device, which is also called a spinal rod and is attached to the heads of the pedicle screws using set screws. As an example, for several adjacent vertebrae for vertebral fixation, for each vertebra, a pedicle screw is screwed onto the vertebra by a bone anchor of the pedicle screw, and then these pedicle screws are mechanically fastened to each other using a spinal rod placed in a groove or U - shaped opening formed by the pedicle screw head, forming a row of pedicle screws along the spine. This makes it possible to provide the mechanical support necessary for spinal stabilization for spinal fixation in a patient or a living body.

[0004]

[0004] To better reach the incision site and screw the pedicle screw into the vertebra, the pedicle screw, specifically the head of the pedicle screw, is usually removably attached to a screw extender or similar device, such as an expansion tap screw head or blade. The purpose of screw extenders and similar devices is to add additional length to the head of the pedicle screw, allowing the operator or surgeon to work outside the surgical incision and not only to keep the surgical incision open but also to help guide different tools and spinal rods to the head of the pedicle screw. A screw extender configured to hold a pedicle screw is usually a tubular longitudinal device that is considerably larger than the head of the pedicle screw, and the screw extender itself has longitudinally shaped slots along its sides. When the pedicle screw head is connected to the screw extender, the longitudinally shaped slots coincide with the U-shaped opening of the screw head of the pedicle screw, thus allowing the spinal rod to be guided through the longitudinally shaped slots to the U-shaped opening. The process of pushing the spinal rod within the longitudinally molded slot of the screw extender toward and into the head of the pedicle screw is also called rod reduction.

[0005]

[0005] For example, U.S. Patent No. 10,058,355, which is incorporated entirely herein by reference, describes an orthopedic implant kit providing a pedicle screw, a corresponding set screw, a rod, and tools for operating them, including a screw extender for holding the pedicle screw and a set screw driver for screwing the set screw onto the screw head of the pedicle screw. U.S. Patent No. 7,160,300, which is incorporated entirely herein by reference, describes a rod reduction method having a tubular shape with a longitudinally shaped channel that can guide the rod from the guide tool to the bone screw attached to the guide tool, and an intermediate guide tool that is attached to a bone screw. As another example, U.S. Patent No. 8,795,283, which is incorporated entirely herein by reference, describes another type of kit orthopedic surgical system for surgical intervention for spinal stabilization, which includes a pedicle screw having a head for receiving a rod and tools necessary for the surgical intervention. The screw extender is fabricated from a tube having two separable half-shells held together by a retaining ring so as to be able to form a tubular shape. In yet another example, U.S. Patent No. 8,262,662, which is incorporated in its entirety herein by reference, provides a system and method for delivering a spinal connector to a spinal anchor site within the spinal column. In one embodiment, a spinal implant and access device is provided, comprising a U-shaped receiving member, a bone engagement member, an extension member, a spinal rod, and a set screw. The extension member is tubular.

[0006]

[0006] Similar orthopedic spinal surgery concepts, tools, and apparatus for attaching rods to pedicle screws via set screws have been proposed as described above, for example, U.S. Patent Nos. 5,129,388, 5,520,689, 5,536,268, 5,720,751, 5,984,923, 6,056,753, 6,183,472, U.S. U.S. Patent No. 6,258,090, U.S. Patent No. 6,454,768, U.S. Patent No. 6,648,888, U.S. Patent No. 6,740,086, U.S. Patent No. 7,618,442, U.S. Patent No. 8,308,782, U.S. Patent No. 8,876,868, U.S. Patent Publication No. 2006 / 0025771, and U.S. Patent Publication No. 2018 / 0289397, all of which are incorporated herein by reference in their entirety.

[0007]

[0007] However, once the pedicle screws are attached to the vertebrae of the spine, the surgeon or operator can only see the screw extender that is removablely attached to the screw head of each pedicle screw, which generally points away from the surgical incision that was necessary to attach the pedicle screw to the vertebra. Generally, unless the surgeon opens the incision, the screw head is embedded in the surrounding tissue of the incision. In this regard, before the rod reduction and rod fixation process, the surgeon or operator usually needs to select a rod of the appropriate length to fit into the U-shaped groove of the pedicle screw head, or pre-bend a spinal rod, or select a pre-bent spinal rod. However, because the exact placement of the pedicle screw and the screw head of the pedicle screw having a groove for accommodating the spinal rod cannot be seen, this is a difficult task that can lead to a trial-and-error procedure to determine the appropriate length, shape and bend of the spinal rod so that it can be percutaneously inserted into each screw head of the pedicle screw. This can lead to significant time loss during surgery, an increased risk of screw loosening or implant failure, and additional costs.

[0008]

[0008] A solution has been proposed in orthopedic surgery to detect observable pedicle screw heads based on machine learning using a convolutional neural network (CNN). For example, see Von Atzigen et al., "HoloYolo: A proof-of-concept study for marker-less surgical navigation of spinal rod implants with augmented reality and on-device machine learning," The International Journal of Medical Robotics and Computer Assisted Surgery, year 2020, e2184. However, this method relies on a direct visual view of the different screw heads of the pedicle screws attached to the vertebra, and therefore directly within the wound. Visible It has many drawbacks, including the need for a fully open surgical site and maximum opening of the incision, the need for relatively long data processing times and slow tracking refresh rates for detection, and substantial uncertainty in detection.

[0009]

[0009] To avoid the shortcomings of camera view-based imaging solutions, several methods can be used to evaluate the placement of pedicle screws using C-arm fluoroscopy with X-ray projection, and to calculate the screw pose estimation based on two-plane X-rays and fluoroscopic images using reflective markers. See Esfandiari et al., "A deep learning framework for segmentation and pose estimation of pedicle screw implants based on C-arm fluoroscopy," International Journal of Computer Assisted Radiology and Surgery, Vol. 13, No. 8, year 2018, pp. 1269-1282, and also Fu et al., "Computer-Assisted Fluoroscopic Navigation of Pedicle Screw Insertion An In Vivo Feasibility Study," Acta Orthopaedica Scandinavica, Vol. 75, No. 6, year 2004, pp. 730-735. However, these methodologies are not suitable for direct use by orthopedic surgeons due to the need for complex and expensive computed tomography equipment and the additional surgical steps that must be performed.

[0010]

[0010] Therefore, there is a need for systems, methods, and apparatus that improve the use of spinal rods during surgery, specifically the placement, implantation, pre-selection, and matching of spinal rods for specific surgical conditions, simplify user use, and substantially reduce the costs required to assist the user. [Overview of the project]

[0011]

[0011] According to one aspect of the present invention, a method for assisting orthopedic surgery is provided. This method can be carried out using a data processing device, the data processing device includes a display device and imagingThe apparatus includes. Preferably, the method includes the steps of: capturing a sequence of images using an imaging apparatus such that the field of view of the imaging apparatus captures images of a plurality of screw extenders, each screw extender holding a pedicle screw, and the plurality of screw extenders being arranged in a surgical incision of a living body undergoing orthopedic surgery; providing a live video feed to a display device by displaying at least a portion of the captured images; detecting the plurality of screw extenders using a data processing device based on the sequence of captured images; a first calculating step of calculating the orientation and position of the detected plurality of screw extenders; a second calculating step of calculating the three-dimensional (3D) position of the screw head of each pedicle screw based on the orientation and position of the first calculating step; and projecting and displaying each calculated 3D position of the plurality of screw heads using graphic elements at positions corresponding to the positions of the screw heads projected onto the currently displayed image of the live video feed on a display device using a graphical user interface.

[0012]

[0012] According to another aspect of the present invention, a non-temporary computer-readable medium on which computer instructions are recorded is provided. The computer instructions, when executed on a computer device, are configured to carry out a method for assisting orthopedic surgery, and the computer device is a display device and imaging The device is connected in an operable manner.

[0013]

[0013] According to yet another aspect of the present invention, a computer system is provided, the computer system is imaging The device includes a display device and a data processing device, and the data processing device is imaging The device and display device are operably connected. Preferably, the data processing device is configured to perform a method for assisting orthopedic surgery using augmented reality.

[0014]

[0014] According to another aspect of the present invention, the spine is based on a bent fixing rod. correction A method is provided to assist in orthopedic surgery in determining [a certain condition]. Preferably, the method is carried out using a data processing device. More preferably, the method is [a certain method]. imaging The device is used to scan the fixed rod, and the spine correction The steps include obtaining scanning data of a fixed rod that has been bent for this purpose, and based on the scanning data, the fixed rod Curvature A first calculation step of calculating data; a step of receiving data on the position of the attachment point of the fixation rod to the spine, wherein the position of the attachment point is determined based on the position data of the screw head of the pedicle screw attached to the vertebra of the spine; and a second calculation step of calculating data on the corrected position of the attachment point, wherein the corrected position of the attachment point is determined based on the position data of the fixation rod by the first calculation step. Curvature By taking the data into consideration, the fixed rod correction A second calculation step based on the correction applied to the position of the mounting point when it is attached to the mounting point of the spine, correction Based on the corrected position data of the attachment points of the spine, correction A third calculation step involves calculating the vertebral parameters of the spine, correction The process includes the step of displaying the spinal parameters of the spine on a display device.

[0015]

[0015] According to yet another aspect of the present invention, a non-temporary computer-readable medium on which computer instructions are recorded is provided. The computer instructions are displayed on a display device and imaging When performed by a computer device operably connected to the device, the spine is bent and fixed by a fixed rod. correction It is configured to perform methods to assist in orthopedic surgery, determining the appropriate course of action.

[0016]

[0016] According to yet another aspect of the present invention, a computer system is provided, the computer system is imaging The device includes a display device and a data processing device, and the data processing device is imaging The device and display device are operably connected. Preferably, the data processing device is based on a bent fixing rod of the spine. correction It is configured to implement methods to support orthopedic surgery in determining the outcome.

[0017]

[0017] According to another aspect of the present invention, the spinal column against A method for assisting orthopedic surgery is provided. Preferably, the method is carried out using a data processing device, the data processing device includes a display device and imaging The apparatus is included. Furthermore, preferably, this method is imaging The device's field of view is such that it captures images of multiple pedicle markers positioned on multiple guidewires, or at least one of the multiple guidewires. imaging A step of acquiring a sequence of images using a device, wherein multiple pedicle markers or multiple guidewires are arranged in a surgical incision in the body of a living person undergoing orthopedic surgery; and a step of displaying at least a portion of the acquired images, or using a transparent display device. Ta directly Visibility The process involves providing a live video feed to a display device by any of the following steps: detecting multiple pedicle markers or multiple guidewires using a data processing device based on a sequence of captured images; a first calculating step of calculating the orientation and position of the detected multiple pedicle markers or detected multiple guidewires; and, based on the orientation and position of at least one of the detected multiple pedicle markers or detected multiple guidewires attached to the vertebrae by the first calculating step, detecting at least two vertebrae bone This includes a second calculation step of calculating posture data information.

[0018]

[0018] By referring to the accompanying drawings illustrating some preferred embodiments of the present invention and studying the following description and the accompanying claims, the above and other objects, features and advantages of the present invention and methods for realizing them will become clearer and the present invention itself will be best understood.

[0019]

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate presently preferred embodiments of the invention and, together with the general description and the detailed description given below, serve to explain the features of the invention.

Brief Description of the Drawings

[0020] [Figure 1A] An exemplary and simplified perspective view of a location or facility where orthopedic surgery is being performed is shown, including a living body or patient L on an operating table with a surgical incision SI, and a user, operator, surgeon, or medical assistant O holding an exemplary data processing device 100 for implementing a method for proposing different types of spinal rods for orthopedic surgery. [Figure 1B] An exemplary and simplified flowchart depicting different steps of a method for proposing different types of spinal rods for orthopedic surgery by using augmented reality, according to one aspect of the invention. [Figure 1C] As non-limiting examples of a screw extender and a pedicle screw for the present method and system of this specification, a side view of an exemplary screw extender SE, and a pedicle screw assembly having a bone anchor BA and a screw head SH is shown, visualizing different elements of this exemplary assembly. [Figure 1D] A simplified schematic perspective view of a surgical incision SI with six exemplary screw extenders SE1 to SE6 protruding is shown, each screw extender SE being equipped with optical markers OM1 to OM6 for detecting and tracking the screw extender. [Figure 1E] A simplified perspective view of an exemplary marker device 50 that can be removably disposed on the distal end 60 of a screw extender SE is shown, the marker device 50 having an optical marker code OM and a mounting device 55 for removably attaching to the screw extender SE. [Figure 2A]An illustrative screenshot of one step of this method is shown, which preferably appears on the graphical user interface of a data processing device and illustrates an embodiment of augmented reality used on a display screen for assisting orthopedic surgery. [Figure 2B] The following is an illustrative screenshot of a different stage of the method from Figure 2A, which is preferably displayed on the graphical user interface of a data processing device and shows a different aspect of augmented reality used on a display screen for orthopedic surgery assistance from Figure 2A. [Figure 2C] The following are illustrative screenshots of a different stage of the method from Figures 2A to 2B, which are preferably displayed on the graphical user interface of a data processing device and show a different aspect of augmented reality used on a display screen for orthopedic surgery assistance from Figures 2A to 2B. [Figure 2D] The following are illustrative screenshots of a different stage of the method than those shown in Figures 2A to 2C. The screenshots are preferably displayed on the graphical user interface of a data processing device and show a different aspect of augmented reality used on a display screen for orthopedic surgery assistance than those shown in Figures 2A to 2C. [Figure 2E] The following are illustrative screenshots of a different stage of the method than those shown in Figures 2A to 2D. The screenshots are preferably displayed on the graphical user interface of a data processing device and show a different aspect of augmented reality used on a display screen for orthopedic surgery assistance than those shown in Figures 2A to 2D. [Figure 2F] The following are illustrative screenshots of a different stage of the method than those shown in Figures 2A to 2E. The screenshots are preferably displayed on the graphical user interface of a data processing device and show a different aspect of augmented reality used on a display screen for orthopedic surgery assistance than those shown in Figures 2A to 2E. [Figure 2G]Figures 2A to 2F illustrate illustrative screenshots of a different stage of this method, which are preferably displayed on the graphical user interface of a data processing device and show a different aspect of augmented reality used on a display screen for orthopedic surgery assistance than Figures 2A to 2F. [Figure 2H] Figures 2A to 2G of this method show illustrative screenshots of a different stage, which are preferably displayed on the graphical user interface of a data processing device and show a different aspect of augmented reality used on a display screen for orthopedic surgery assistance. [Figure 2I] The following are illustrative screenshots of a different stage of the method from Figures 2A to 2H, and the screenshots are preferably displayed on the graphical user interface of a data processing device and show a different aspect of augmented reality used on a display screen for orthopedic surgery assistance from Figures 2A to 2H. [Figure 2J] The following are illustrative screenshots of a different stage of the method from Figures 2A to 2I, which are preferably displayed on the graphical user interface of a data processing device and show a different aspect of augmented reality used on a display screen for orthopedic surgery assistance from Figures 2A to 2I. [Figure 2K] Figures 2A to 2J of this method show illustrative screenshots of a different stage, which are preferably displayed on the graphical user interface of a data processing device and show a different aspect of augmented reality used on a display screen for orthopedic surgery assistance. [Figure 2L] Figures 2A to 2K of this method show illustrative screenshots of a different stage, which are preferably displayed on the graphical user interface of a data processing device and show a different aspect of augmented reality used on a display screen for orthopedic surgery assistance. [Figure 2M]Figures 2A to 2L of this method show illustrative screenshots of stages different from those shown in Figures 2A to 2L. The screenshots are preferably displayed on the graphical user interface of a data processing device and show a different aspect of augmented reality used on a display screen for orthopedic surgery assistance. [Figure 2N] The following are illustrative screenshots of a different stage of the method from Figures 2A to 2M, which are preferably displayed on the graphical user interface of a data processing device and show a different aspect of augmented reality used on a display screen for orthopedic surgery assistance from Figures 2A to 2M. [Figure 3A] This diagram shows a schematic and simplified representation of a spine or vertebral column SC having seven exemplary vertebrae V1-V7 as viewed from the rear, with two attachment points AP for each vertebra V determined by the method described herein, and visualization of different parameters that can be calculated and displayed by several steps of the method, e.g., steps C70, C75, D70, D80, the different parameters include position and posture information PDI_V for each vertebra V1-V7, curvature data SCD for the spine or vertebral column that is not currently corrected, and data on the corrected spinal curve CSC for the spine or vertebral column that has been corrected. [Figure 3B] An exemplary perspective view of a screw extender SE positioned inside a surgical incision SI and fitted with a tool SD is shown, where the tool SD is a screwdriver SD for, for example, attaching a pedicle screw to a vertebra V, and the tool SD has two exemplary optical detection markers OM arranged thereon. [Figure 3C] Based on the fixed geometric relationship between screw extenders SE1-SE3 and screw heads SH1-SH3 and bone anchors BA1-BA3, a side view of three screw extenders SE moved to one predetermined side so as to be in the outermost angular position relative to screw heads SH and bone anchors BA is shown to improve the accuracy of calculations of the positions of vertebrae and spinal column SC. [Figure 4]An exemplary flowchart of a method 500 for scanning, calculating, and displaying rod data RD of an actual spinal stabilization rod R, and for calculating virtual spinal correction based on the rod data RD and visualizing the virtual correction to the spinal column SC, according to another aspect of the present invention, is shown. [Figure 5] An illustrative flowchart of Method 600 is shown for determining different types of information characterizing the spinal column SC before pedicle screw PS placement is performed by detecting a guidewire GW or pedicle marker PM visible from the surgical incision SI. [Figure 6] An exemplary and simplified cross-sectional view of a vertebra according to one aspect of the present invention is shown, having two drill holes DH1 and DH2, two guide wires GW1 and GW2 positioned in the drill holes DH1 and DH2 respectively, and two pedicle markers PM1 and PM2 attached to the guide wires GW1 and GW2 respectively, wherein the optical marker OM comprises an optical marker component 50 that is removable or fixedly attached. [Modes for carrying out the invention]

[0021]

[0032] In this specification, the same reference numeral is used to designate identical elements common to the figures, where possible. Also, images in the figures are simplified for illustrative purposes and may not be depicted to scale.

[0022]

[0033] Figure 1A shows a perspective view of the location where orthopedic surgery is performed, showing a living person or patient L on an operating table having a surgical incision SI, and a user, operator, surgeon, medical assistant O holding an exemplary data processing device 100 for performing a method for proposing different types of spinal rods for orthopedic surgery, and Figure 1B shows an exemplary and simplified flowchart depicting different steps of a method for proposing different types of spinal stabilization rods for stabilizing and fixing the vertebrae of the spinal column SC of a living person or patient L according to one aspect of the present invention, the method is, imagingThe method is implemented by using augmented reality during orthopedic spinal surgery at a surgical incision site SI with a computing device having a device and display screen. As an exemplary embodiment, orthopedic spinal surgery is shown and described, and the method is implemented and used to assist a user, operator, surgeon, medical assistant O in selecting an appropriate stabilizing rod to be attached to two or more pedicle screws. For example, the method can suggest a specific rod having a particular pre-bent shape from among a plurality of rods having different bending shapes, or a specific rod Curvature Alternatively, a bending curve can be proposed, and the rod can then be bent for surgery by surgeon O.

[0023]

[0034] It should be noted that spinal orthopedic surgery is essentially illustrative, and the same methods using augmented reality can be used in other types of orthopedic surgery that require stabilization rods or other types of stabilization devices for attachment to different types of pre-positioned bone screws fitted with detectable screw extenders, such as, for example, fracture repair surgery requiring stabilization by rods, or other types of fracture or reconstructive surgery using external fixators.

[0024]

[0035] Prior to performing Method 200, an orthopedic surgery is performed, in which surgeon O initiates and performs the orthopedic surgery, for example, based on a normal surgical workflow. Thus, a surgical incision SI is made in a living person or patient L, and for illustrative and explanatory purposes, it is assumed that at least two pedicle screws PS1, PS2 are placed in each vertebra of the spinal column SC of living person L, and in this variation, three pedicle screws PS1, PS2, PS3 are shown. This number is illustrative only and selected for illustrative purposes, and it is possible to perform Method 200 with a different number of screw extenders SE and corresponding pedicle screws. Typically, each vertebra requires two pedicle screws on each side. Thus, each one pedicle screw PS1, PS2, PS3 is attached to the respective screw extenders SE1, SE2, SE3 using their respective screw heads SH1, SH2, SH3. At least some examples of such surgeries are shown in U.S. Patent No. 10,058,355 (see Figures 18–38), which is incorporated herein by reference in its entirety.

[0025]

[0036] When multiple pedicle screws PS1, PS2, PS3 are positioned in their final positions for each vertebra V1, V2, V3 by screwing them into individual vertebrae using, for example, the bone fixation elements of the pedicle screws PS1, PS2, PS3, with screw extenders SE1, SE2, SE3 and a screwdriver, as shown in U.S. Patent No. 10,058,355, the surgeon or operator O needs to select or provide a spinal stabilization rod R that is bent or has such a shape so that it can be positioned within the receiving openings SH1, SH2, SH3 of the screw heads of the multiple pedicle screws PS1, PS2, PS3. Preferably, each screw head of the pedicle screws PS1, PS2, PS3 has a U-shaped groove for receiving the rod R and has threads so that the rod R can be attached to the screw head by a set screw. Curvature To determine the shape of the rod R before placement and connection to the screw heads SH1, SH2, SH3, Curvature It is desirable to have information on the relative positions and orientations of the screw heads SH1, SH2, and SH3 so that at least one of their lengths can be determined.

[0026]

[0037] At this stage of surgery, surgeon O can use data processing device 100 to initiate method 200 for proposing different types of spinal stabilization rods for stabilizing and fixing the vertebrae of the spine SC of a living organism or patient L. The steps of method 200 can be carried out by dedicated application software including computer instructions, the computer instructions being executable by the data processor of data processing device 100 to carry out embodiments of the method described herein, and method 200 is configured to operate and display a graphical user interface GUI by user commands, for example, as a graphic overlay on a live video feed which can be displayed on a display device 120 of data processing device 100. The display device 120 may be a display screen which also includes touch-sensitive characteristics for information input, such as a touchscreen. Preferably, data processing device 100 may be a portable device such as a smartphone, mobile phone or tablet, or another type of handheld data processing device, for example, but not limited to these. Furthermore, data processing device 100 may also include a graphics processor that supports image data processing, as well as the generation of the live video feed and GUI, as well as other graphic elements displayed in the GUI.

[0027]

[0038] Method 200 can be started when all pedicle screws PS1, PS2, PS3, each having screw extenders SE1, SE2, SE3, are in place, as illustrated in Figures 1A and 1B. The first step U10 is performed, and Method 200 is started by, for example, a surgeon or operator O launching the application. Next, Method 200 proceeds to step D10, where a live video feed is generated and displayed on the GUI of the display device 120, for example, on the touchscreen of a smartphone. This can be done by touching a GUI button or active graphic element to start the live video feed, which is displayed in one of the applications, or it can be started automatically when the application is launched in step U10. Thus, the data processing device 100, imaging The device 110, for example, a smartphone's built-in camera unit, starts capturing a sequence of images, and displays such images simultaneously in real time on the display screen 120, for example, using a graphical user interface (GUI) window, or on the entire screen of the display device or screen 120. The live video feed is imaging Based on the images captured by the device 110, it is displayed as a real-time video sequence to allow for the overlaying of additional graphic elements, animations, and other objects for augmented reality representation. It should be noted that step D10 is an optional step because, for example, a transparent or translucent display screen or device can be used to directly view the surgical incision SI. Visibility This is because the method can also be implemented using wearable augmented reality (AR) glasses, head-mounted displays with transparent or semi-transparent display screens or devices, or head-up displays (HUDs).

[0028]

[0039] Next, optionally, method 200 proceeds to step D20, where instructions or command CMDs can be displayed or otherwise provided to the surgeon or operator O, for example, by requesting calibration information, orientation information, or other types of information that enable the next scanning step U30, or by requesting basic information to start the method. Furthermore, step D20 may also provide a graphic element that allows the user to input data for method 200, specifically data related to instructions or command CMDs. This can be done, for example, using a text prompt, a graphical prompt, or a graphic element overlaid on a live video feed with one or more selection buttons. It is also possible that step D20 provides audio information in the form of voice commands to assist the surgeon or operator O in providing instructions or command CMDs. Alternatively, step U20 may be performed, where the surgeon or operator O can input data to respond to the instructions or command CMDs, as illustrated in the screenshot of Figure 2A. These steps can be performed at least partially simultaneously while the surgeon or operator O continuously images the surgical incision SI to generate the live video feed of step D10, and enable the display of commands and buttons CMD. For example, in the illustrated modification, step D20 displays a text box for the surgeon or operator O requesting information regarding the orientation of the living body L to the live video feed view, and in step D20, the text box is displayed in the GUI, providing the surgeon or operator O with additional information related to the information request. Step D20 can also display and overlay two symbolized heads with graphical icons on the left and right sides of the GUI, so that the surgeon or operator O can select one of these two graphical icons to indicate which side the living body head L is positioned on relative to the position of the data processing device 100.When a graphic icon representing a head is selected, the head can be highlighted, and the requested information can be confirmed via a confirmation button overlaid on the GUI's live video feed, as shown in Figure 2A.

[0029]

[0040] In general, in the context of this specification, in the step where a surgeon or operator O inputs data into Method 200 or other methods described herein, for example, in step U20, for example, the requested data may be input using the microphone 130 of the data processing device 100 and voice or speech recognition software operating on the data processing device 100, instead of manually inputting the data by touching graphic elements such as buttons on the GUI in touchscreen operation. This allows the surgeon or operator O to provide data or information to respond to an instruction or command CMD by voice command, and such implementation of voice and speed recognition makes it possible to avoid, at least partially, the surgeon or operator O touching the display screen 120 during Method 200. The input of data can then be confirmed by audio, for example, with voice prompts, or with different graphic elements displayed, using one or more speakers that are part of or operably connected to the data processing device 100.

[0030]

[0041] However, this step U20 can also be automated by a computer-based process, for example by an image data processing algorithm using a data processor, and the memory of the data processing device 100 can be used to detect the orientation of the living organism L relative to the captured sequence of images that provide image data for a live video feed. As an example, this can be done by using a trained neural network that can detect the orientation of the living organism L based on training data, for example, or by using an optical marker attached to the living organism L or a medical or operating table or surgical table, as further described below with respect to an optical marker OM attached to a screw extender SE, or by detecting the orientation of a medical or operating table or surgical table, which can be detected by a pattern matching algorithm, and information regarding the orientation of the living organism L can be provided.

[0031]

[0042] As described above in steps U20 and D20, simultaneously with the display of the live video feed in step D10 using the GUI, different user commands and information can be displayed on the GUI at different points in time during the execution of method 200 in order to provide user instructions and receive user information and commands. For example, different text prompts or text boxes with text information can be displayed as overlays on the live video feed to provide the surgeon or operator O with information about the type of procedure being performed or the status information of method 200 as feedback to the surgeon or operator O, or to request user input via icons or buttons. Different information and commands can also be requested by audio, for example, voice prompts. imagingIt is possible to display graphic elements or icons that allow you to open or pull down a menu for configuring Method 200 by locking parameters and features, such as, but not limited to, zoom or image clipping, automatic image correction settings, automatic color and white balance adjustments, and wide-angle settings. It is also possible to provide a graphical icon that can be selected by touch or otherwise by the surgeon or operator O to return to a previous step of Method 200.

[0032]

[0043] Next, in step U30, which scans the screw extender SE, the surgeon or operator O is notified or prompted, for example by a text prompt, to take and capture a sequence of images of the screw extender SE pointing to the surgical incision SI, while live video feedback is displayed on the GUI as a result of the continuous execution of step D10. An exemplary screenshot of this step is shown in Figure 2B, where a text box prompts the user or operator O to move the data processing device 100 to scan all of the screw extender SE. Next, in step D10, the surgeon or operator O can view all of the screw extender SE, for example, three exemplary screw extenders SE1, SE2, and SE3, in the live video feed on the GUI. imaging The field of view and field of view of the device 110 are directed towards the living body L and the surgical incision SI, and an image sequence of the surgical incision SI and its surroundings is acquired. This is illustrated in Figure 2C with a screenshot, and a text box can be displayed to indicate that the screw extender scan is in progress. Figure 2C shows the display using the GUI. imaging The field of view of device 110 also shows five screw extenders SE, three anteriorly and two on the left side, attached to the vertebral bodies on the right side of the pedicle bone at the posterior end of the spine.

[0033]

[0044] Simultaneously with or after the start of scanning step U30, while the surgeon or operator O is still imaging the surgical incision SI and screw extender SE, an image data processing step C10 is performed, which is the step of detecting different screw extender SEs. This can be done by different types of image processing algorithms performed on the acquired images. For example, this can be done in a stepwise manner in which a first screw extender SE is searched for and detected, and its data is stored, for example, by a three-dimensional coordinate data model. Then, the next screw extender SE is searched for and detected, and its data is stored, imaging These substeps are repeated until all screw extenders SE within the field of view of the device 110 have been detected and stored. Preferably, the detection step C10 is performed while the data processing device 100 is moving, which means that the field of view angle and observation window of the screw extenders SE and surgical incision SI are variable and change.

[0034]

[0045] As an example, detection step C10 can be performed using a rigid body model-based three-dimensional (3D) pose, positioning estimation algorithm and tracking algorithm to detect and track the shape of the screw extender SE, and then extract the pose of the screw extender SE to provide a dataset of pose data information PDI. Since the shapes of all screw extender SEs are known and all have the same shape and dimensions, a three-dimensional model, such as a computer-aided design (CAD) data model, can be used in this detection step. After detecting one of several screw extender SEs, the step generates pose data information PDI that can be stored and updated, and the pose data information PDI may include the coordinate reference position, angle and rotation direction of the screw extender SE, and different vectors relative to a real-world coordinate system, such as a coordinate system, such as Euclidean space. For the sake of simplicity of calculation, the pose data information PDI may also include only the coordinate positions of different screw extender SEs. The pose data information PDI can be calculated in various forms and coordinate spaces, but in a preferred embodiment, the coordinate data is three-dimensional data referring to Euclidean coordinate space. Once all pose data information PDIs have been generated for all screw extender SEs, a dataset or table can be generated using the dataset of all collected pose data information PDIs. Although the body shape of the screw extender SE is known, the screw extender SE is only partially visible in the image sequence because its front end is interconnected with the head of the pedicle screw PS inside the surgical incision, as seen in the exemplary screenshot in Figure 2C. Therefore, a robust algorithm for computer vision tracking is needed that can also detect the partial shape of the body and generate pose data information PDIs. Since all surgical incisions are not known a priori and can vary greatly between different incision locations, biological tissue, and arrangements of surgical tools, a vision tracking algorithm that does not require prior knowledge of the scene being tracked is preferably used.

[0035]

[0046] An example of a model-based tracking algorithm that can be used is described in U.S. Patent Application Publication 2019 / 0355150, in which a trained neural network is used for object detection, and this reference is incorporated in its entirety by reference. A trained neural network, for example, a CNN and deep learning based on images of a screw extender with known pose data information PDI, can be used to establish training data that allows a partial view of a screw extender SE to be directly linked to its pose data information PDI. Another example is the robust model-based augmented reality tracking algorithm from Visometry GmbH's visionLib™.

[0036]

[0047] As another example, detection step C10 can be performed using a contour detection algorithm, which first detects each contour of the screw extender SE, then maps each detected contour to a two-dimensional (2D) projection of the three-dimensional (3D) model of the screw extender, and then determines a dataset of pose data information PDI.

[0037]

[0048] In detection step C10, the detected screw extender SE can also be tracked and updated during image acquisition and display on the live video feed. This may be necessary because the surgeon or operator O moves the acquisition position and orientation of the screw extender SE in scanning step U30, thereby collecting more information to further improve the posture data information PDI dataset. However, it is also possible for the screw extender SEs themselves to move slightly relative to each other, thereby changing the coordinate and orientation data of the screw extender SEs. In this regard, the posture data information PDI dataset may change as a function of time, and the data structure or table containing the posture data information PDI dataset can be updated periodically during detection step C10.

[0038]

[0049] In one modification, step C10 is easily performed with the pedicle screw PS attached to the vertebra V, without requiring the placement of the screw extender SE to the pedicle screw PS. For example, the screw head SH of each pedicle screw PS may have an optical marker OM provided by printing, etching, engraving, patterning, or other means for more robust detection of the screw head SH of each pedicle screw PS by a tracking algorithm. For example, the optical marker OM may have some redundant information so that it can still be detected even if the marker OM is covered with flesh, muscle, fat, or other body parts of the surgical incision SI. See, for example, Kohler et al., "Robust Detection and Identification of Partially Occluded Circular Markers," In International Conference on Computer Vision Theory and Applications (VISAPP), Vol.1, pp.387-392, year 2010. See also ARTag reference optical markers. In one modification, before step C10, in which the screw heads SH are scanned instead of the screw extender SE, each screw head SH is equipped with a removable optical marker component 50 with an array of optical markers OM, as shown in Figure 1E, but this time it is not placed on the screw extender SE, but directly on each screw head SH of the pedicle screw PS. The interconnection between the optical marker component 50 and the screw head SH can be achieved by configuring the end of the optical marker component 50 with an interconnection element complementary or corresponding to one of the screw head SHs of the pedicle screw PS, for example, a press-fit engagement as described in U.S. Patent No. 10,058,355, or a snap lock, or other types of geometrically defined locks between the optical marker component 50 and the screw head SH, thereby enabling the optical marker component 50 to be connected to the screw head SH at a precisely defined position, and still being easily removable as it is used only for detection.This defines and maintains a fixed geometric relationship between the optical marker OM, the optical marker component 50, and the screw head SH of the pedicle screw PS in an interconnected state or position. This reduces, or even completely eliminates, the detection reliability problem of the pedicle screw PS. However, at least part of the method may also rely on the detection of the pedicle screw PS using an image processing algorithm without requiring additional optical detection assistance.

[0039]

[0050] As another variation of step C10, each pedicle screw PS may be equipped with one or more radio frequency identification tags (RFIDs), preferably passive RFID tags, enabling the detection of their three-dimensional position in space based on different detection techniques and the use of RFID detection antennas. For example, this can be done by using an RFID tag array having different RFID tags having different orientations from each other, e.g., multiple RFID tags oriented on different axes in three-dimensional coordinate space, attached to the pedicle screw PS, e.g., screw head SH, and using an RFID detection antenna that can move relative to the RFID tags on the pedicle screw PS to improve positional accuracy. See, for example, Zhang et al., "3-Dimensional Localization via RFID Tag Array," In 2017 IEEE 14th International Conference on mobile ad hoc and sensor systems (MASS), pp.353-361. IEEE, year 2017. For example, it is conceivable that multiple reference RFID tags not attached to the pedicle screw PS may be used to provide different known reference positions arranged in a matrix, and that one or more RFID tags may be attached to the pedicle screw PS, e.g., the screw head SH. See, for example, Liu et al., "A Three-Dimensional Localization Algorithm for Passive Radio-Fequency Identification Device Tag," International Journal of Distributed Sensor Networks, Vol.13, No.10, year 2017, ref.1550147717736176.

[0040]

[0051] As another variation, ultrawideband RFID tags can be used, which can be detected using multiple reader antennas and different types of detection algorithms, such as backscatter modulation or UHF and UWB modulation. See, for example, "Ultrawide Bandwidth RFID: The Next Generation?", Proceedings of the IEEE Vol.98, No.9, year 2010, pp.1570-1582. In such a case, different elements such as a reference RFID tag, one or more reader antennas, and a data processing device for implementing a data processing algorithm on the read signals from the RFID tag are part of the system shown in Figure 1A, and are interconnected to the data processing device 100 for reference frame information or further processing by method 200, which can distribute coordinate data of the screw head SH and mounting point AP, and other data that enables the discovery of the screw head SH and mounting point AP within a particular reference frame. However, it is also possible to provide the data processing device 100 with raw data from the reader antennas via the network to determine the coordinate positions of the screw head SH and mounting point AP in the device 100.

[0041]

[0052] As another variation, the screw head SH of a pedicle screw can be detected by thermal imaging, based on the premise that the screw head SH of a pedicle screw becomes cooler than the environment within the surgical incision SI due to different thermal radiation emitted from the metal screw head and the surrounding tissue of the surgical incision SI. For example, infrared thermography can be used to measure the infrared energy emitted from the exposed tissue and bone of the surgical incision SI, as well as from implants such as pedicle screws PS and their screw head SH, and this infrared energy can be converted into a radiant thermal image showing the distribution of surface temperature. Such an image can be subjected to image data processing algorithms for detecting the screw head SH, or even the screw extender SE. An exemplary thermal imaging camera that can be used for this purpose is the FLIR T335 infrared (IR) thermography camera from FLIR Systems Inc. It also operates in the visible light range and can be used by the thermal imaging camera (not shown) to provide a reference position for coordinates, e.g., a ruler, marker, etc. imaging The use of a reference marker or reference frame visible by the device 110 is required. Based on an image processing algorithm, the screw head SH or screw extender SE can thereby be localized and detected from the thermal image, for example by a model-based pattern matching algorithm or by other types of artificial intelligence-based detection algorithms. For this purpose, the system shown in Figure 1A further includes a thermal imaging camera that can provide thermal imaging data to a data processing device, for example, a data processor 100, via a network.

[0042]

[0053] This information regarding either the position of the screw extender SE or the position of the screw head SH can be used in steps D25, D30, and U40, and a graphic base element GP is provided that can be overlaid on the live video feed to highlight different screw extender SEs, or in one variation, a graphic base element GP is provided to highlight different detected screw head SHs of pedicle screws PS, and if no screw extender SE is placed, different pedicle screws PS that need to be considered for the geometric shape and rod template calculations in steps C20 and C30 are selected and deselected.

[0043]

[0054] Furthermore, in one modification, the scanning step U30, the detection step C10, and the step D25 for displaying the basic elements can be performed iteratively, thereby being repeated for each detected screw extender SE. This modification is illustrated in the representations of Figures 2L and 2M, where an exemplary number of four screw extenders SE1-SE4 are tracked and detected. For example, in the scanning step U30, the search and scanning of screw extender SEs can be further assisted by using a graphical locator element GLE that is displayed and overlaid on a live video feed; see, for example, Figure 2M. For example, a graphical locator element GLE that is displayed may represent a graphical representation of a screw extender, e.g., a graphical representation of a rendering or projection of a screw extender SE on the screen, the outline of a screw extender SE, e.g., a translucent graphical representation of a screw extender SE, or other types of graphical locator elements GLE that can be used as locators for scanning and detecting screw extender SEs, e.g., crosshairs, reticles, cursors, arrows, indicators, etc., can be displayed on the screen. For example, this graphical locator element GLE can be presented on the screen in a fixed position relative to the screen, for example, substantially in the center of the displayed field of view. This allows the surgeon or operator O to move the device 100 in step U30, thereby the data processing device 100 imaging The graphical locator elements captured by the device 110 can also be moved to the scene captured or imaged at the surgical incision SI.

[0044]

[0055] In the modified example shown in Figure 2M, the graphical locator element GLE is fixed in an upright position in the center of the GUI screen and rendered as a translucent element as the outline of the screw extender SE on the live video feed, and has a position and orientation that allows operator O to move the device 100 so that the graphical locator element GLE can match one of the screw extenders SE1-SE2 protruding from the surgical incision SI. In the modified example shown, the longitudinal groove of the screw extender is also represented in the graphical locator element GLE, thereby serving as an orientation aid to operator O with respect to the orientation angle for holding and moving the device 100 to detect the screw extenders SE1-SE4.

[0045]

[0056] When a graphical locator element GLE partially or completely visually contacts or touches one of the screw extenders SE1-SE4 captured by the video feed, the contacted screw extender SE can be detected in step C10, and then highlighted by displaying the basic elements of the detected screw extender SE, for example in step D25. For example, detection step C10 can be divided into course detection step C12, which is performed concurrently with scanning step U30, where touch or contact between the graphical locator element GLE and the screw extender SE can be detected. This course detection step C12 can be based on a pattern matching algorithm or other type of detection algorithm that enables the detection of a surface or area in the current image where the screw extender SE is located, and then, if the coordinates or area of ​​the graphical locator element GLE are in contact with, near, or touching an area of ​​the image representing the screw extender SE, a precision detection step C14 can be performed, where the precise position and coordinates of the screw extender SE are detected, and for example, attitude data information PDI is detected. Once the screw extender SE is fully detected in step C14, the augmented reality graphic base elements GP1-GP4 can be displayed on the detected screw extender SE, as shown in Figure 2L. For example, prompts, text boxes, and confirmation buttons can be used to prompt the surgeon or operator to accept the detection of the screw extender SE, thereby also accepting the detected pose data information PDI. In the illustrative diagram of Figure 2M, SE1 and SE2, the two screw extenders arranged on the left, have already been detected and are displayed overlaid with the graphic base elements GP1 and GP2, with the graphical locator element GLE displayed in the center of the image, showing a semi-transparent rendering of the screw extender SE.

[0046]

[0057] In one modified example, for detection, as visualized in Figures 2L and 2M, a graphical locator element GLE having specific orientation information regarding position and orientation can be used to highlight a matching screw extender for selection when there is an approximate match between the GLE and coordinates fixed to the screen 120 or the device 100, using the orientation data information of one of the screw extenders SE1-SE4 of the surgical incision SI. In this regard, when there is an exact or approximate match of the orientation data information PDI of one of the screw extenders SE1-SE4, the detected one can be overlaid with a graphical base element GP, as shown in Figure 2L, and a confirmation prompt can be presented to the operator O. For the operator O to cause a PDI match between one of the screw extenders SE1-SE4 and the PDI of the graphical locator element GLE, the operator must move the device 100, for example by swiveling, tilting, or moving, until the displayed graphical locator element GLE approximately matches one of the screw extenders SE, SE3 in Figure 2M. In this step, for example, a pattern matching algorithm can be used to iteratively calculate the rough PDI information of different screw extender candidates for selection until a match is found with the PDI of the graphical locator element GLE.

[0047]

[0058] Next, the scanning step U30, detection step C10, and step D25, which displays the basic elements, can be repeated for the next screw extender SE, and screw extenders SE1 to SE4 are detected one by one in sequence and highlighted by the basic elements, as shown in Figure 2D, until all desired screw extender SEs are detected. This modification is a stepwise scan that allows for direct visual and intuitive feedback to the surgeon or operator for the detection of each screw extender SE. The detection point or moment in step S14 for each screw extender, during which the PDT of the SE is imaged and tracked, can also be further highlighted by a signal, such as an auditory signal or a vibration signal, or both.

[0048]

[0059] In one modification, the screw extenders SE are not detected by a computer vision algorithm that detects shape, contour, or pattern as described above, but each screw extender SE is equipped with an optical marker OM that can be detected and tracked in the detection step, and the optical marker OM can also function as a reference marker in the viewing landscape. An example of such a viewing landscape with a surgical incision SI is shown in Figure 1D, showing two rows of screw extenders SE1-SE6 attached to pedicle screws PS (not shown). In the illustrated example, each screw extender SE may be equipped with two optical markers OM at different positions to provide more robust detection of the screw extender, with the first marker OM located at the distal end of the body of the screw extender SE and the second marker located in the middle of the body of the screw extender SE. If one of the two markers is obscured from the camera field of view, as shown in screw extenders SE4, SE6, for example, there is redundancy in detection and tracking provided by the other visible optical marker OM.

[0049]

[0060] Each screw extender SE may be equipped with multiple optical markers OM for redundancy purposes, some of which may be hidden inside the surgical incision, covered by other screw extender SEs, or otherwise imaging It may be positioned outside the field of view of the device 110. Alternatively, the surgeon or operator O can visually inspect whether the screw extender SE has been detected and move the camera or imaging device 110's imaging and field of view position so that at least one marker OM is detected and tracked. In the example of Figure 1D, different camera views may be required to detect at least one optical marker OM5 for the detection and tracking of the screw extender by step C10.

[0050]

[0061] Optical markers OM can be fabricated as graphic patterns or designs having a fixed geometric relationship with respect to the screw extender SE, for example, by being positioned in a specific orientation. Examples of patterns that can be used for optical markers include checkerboard patterns, matrix codes or QR codes (registered trademarks), or similar designs, such as those used for robot tracking. Different tracking markers, for example, but not limited to, ARToolKit, ARTag, AprilTag, and ArUco reference tracking markers, are examples of optical markers OM that can be used to label screw extender SEs and are useful for both identification and orientation estimation purposes. For example, each screw extender SE can be fixedly equipped with one or more optical markers OM, for example, by printing, pasting, etching, embossing, lattice, or deposition of layers having such optical markers OM. These optical markers can also be made invisible to the human eye by using, for example, UV-visible inks or NIR-visible inks. For example, optical markers OM can be fabricated as removable or fixedly attached layers or stickers.

[0051]

[0062] Alternatively, as shown in Figure 1E, the optical markers are components 50 separated from each screw extender SE, and can be positioned in a predetermined geometric relationship with respect to the screw extender SE by, for example, placing the optical marker component 50 on the handle mounting component 60 at the distal end of the screw extender SE. The optical marker component 50 also includes an optical marker OM, and can be easily removed from the screw extender 50 by, for example, simple manual operation once method 100 has been carried out and completed. For this purpose, the optical marker component 50 may have a mounting device 55 complementary to the handle mounting component 60 of the screw extender 60. Preferably, the mounting device 55 and the handle mounting component 60 are formed such that the mounting device 55 can only take one position relative to the screw extender SE, and as a result, the correct orientation data information PDI of the screw extender SE can be calculated taking into account the correct orientation of the screw extender SE. Before performing step U30, which involves scanning the screw extender SE for detection and tracking in step C10, the optical marker component 50 can be placed on all screw extender SEs. As described above, it is also possible to use optical marker OMs with redundant information, such as ARTag, TriCode, ARToolkit+, or Kohler circular markers, so that detection of partially shielded marker OMs is possible.

[0052]

[0063] Each screw extender SE may have a different pattern or other graphic element contained in the optical marker OM, which may include information that can be read and identified in an optional identification step C15. This information can be used for verification purposes to confirm whether the correct screw extender SE is being used for the correct surgical procedure. For example, the identification information of each screw extender SE can be read using a database to confirm different aspects of the screw extender SE, such as whether the screw extender SE has exceeded its lifespan or lifecycle, or whether the correct type of screw extender SE is being used for a particular surgery. Furthermore, the identification information contained in each optical marker OM can be used to identify each screw extender SE across several images acquired from the surgical scene, thereby enabling rapid calculation of the correspondence between detected screw extender SEs in a sequence of acquired images. This provides a more robust and faster identification of individual screw extender SEs across several acquired images.

[0053]

[0064] Once at least one of the screw extender SEs is detected, a graphic base element GP can be generated in step D25, which calculates and displays the basic elements of the screw extender, as illustrated in the screenshot of Figure 2D, and can be overlaid on the actually displayed screw extender in the GUI's live video feed. In this figure, a graphic base element is shown that outlines the visible portion of the screw extender SE, and the graphic element highlights or indicates corner points. The graphic base element GP can be displayed to cover or otherwise graphically indicate the position of each screw extender SE in the live video feed in order to provide augmented reality and real-world computer-generated graphic elements for the screw extender SEs. This can be done by calculating and displaying the projection of the screw extender SE as a graphic element of the graphic base element GP on the screw extender SE by calculating data representing the camera position and orientation of the image currently being captured and displayed, as well as based on a dataset of pose data information PDI. However, step D25, which calculates and displays the basic graphic elements GP of the screw extender, can be performed completely separately from the attitude data information PDI collected by detection step C10, and can be based on a contour detection algorithm that detects the outline of the screw extender SE, and then graphically displays the elements, for example, but not limited to, lines, shading, dots, points, and boxes.

[0054]

[0065] Generally, step D25 enables the provision of computer-generated information regarding the screw. This step preferably includes two substeps, the first of which detects fixed points of interest, reference markers, or optical flows on the captured image of the live video feed. This allows the first substep to create an orientation data model of the current camera view. This step can use feature detection methods such as corner detection, blob detection, edge detection or thresholding, and other types of image processing methods. The second substep reconstructs the real-world coordinate system of the environment currently being captured, which is the surgical incision SI and the screw extender SE. Since at least a portion of the viewing scene with the surgical incision SI and body is unknown, simultaneous localization and mapping (SLAM) can map the relative positions of orientation data information SPI to screen position coordinate data SLCD, which can then be calculated to display the graphic base elements GP in the correct positions on the live video feed. In this regard, the graphical representation of the screw extender SE can be a projection of the geometric model of the screw extender projected onto the screen position coordinate data SLCD. In addition, or alternatively, the structure of the viewing scene can be derived from a motion method such that bundle adjustment is used, and the mathematical methods used may include projection (epipolar) geometry, geometric algebra, rotational representation by exponential mapping, Kalman and particle filters, nonlinear optimization, and the use of robust statistics. In this step D25, the graphical representation of the real word object, in this case the graphical base element GP of the screw extender SE, is associated with the real word view or scene of the screw extender SE. Based on three-dimensional information, for example, direct time to flight (dToF) sensor LiDAR sensor, or structure transformationBased on data from distance-measuring sensors, including an optical sensor, and stereo imaging by two image sensors, the viewing landscape can be further analyzed for mapping. The graphic base element GP can be considered a virtual reconstruction projection model of the screw extender SE. An example of such an implementation of step D25 can be found in U.S. Patent No. 10,824,310 and U.S. Patent No. 9,824,495, both of which are incorporated herein by reference in their entirety.

[0055]

[0066] Step D25 provides visual feedback to the surgeon or operator O to confirm whether all screw extenders SE have been detected, and the live video feed of the surgical incision SI is further enhanced by dynamically moving graphic elements to highlight the screw extenders SE, thus providing one aspect of the augmented reality concept. In one modification, all screw extenders SE are first detected, posture data information PDI is extracted and stored in a table or data structure, and then graphic base elements GP are overlaid on the screw extenders. This step can be performed concurrently with scanning step U30 and detection step C10. For example, the graphic base element GP can be opaque, transparent, or translucent shading covering each detected screw extender SE.

[0056]

[0067] Next, Method 200 can perform step D30, where a selector element SF is generated and displayed on the live video feed for each detected screw extender SE, and step U40 is performed to select or deselect the screw extender SE by the selector element SF, allowing the surgeon or operator O to manually select individual screw extenders, preferably by touchscreen operation. Thus, step D30 provides another aspect of the augmented reality concept, enabling easy interaction with the surgeon or operator O to select or deselect the screw extender SE under consideration, and step U40 allows the use of a graphical element SF on the GUI for selecting / deselecting screw extenders placed on the live video feed, for example, by touching the selector element SF with a finger to toggle selection / deselection. Furthermore, even if the viewing angle changes, the graphical representation of the selector element SF can be moved to position itself on or face each screw extender SE. Step D30, which displays the selector element SF, can display dynamically overlaid graphic elements on the live video feed, such as, but not limited to, fields, boxes, arrows, icons, labels, or other types of graphically selectable labels or elements for each of the screw extenders SE, by linking the display coordinates of the selector element SF to the display coordinates of each graphic base element GP generated by step D25, or by calculating the projection from the pose data information PDI of the screw extender SE, as illustrated in the screenshot of Figure 2E. This graphic overlay of the selector element SF on the live video feed allows the surgeon or operator O to select the active screw extender SE in order to suggest a rod template RT or to select a rod template from among several rod templates, as will be further described below.For example, a surgeon or operator O might want to select a front row with three screw extenders SE for rod determination.

[0057]

[0068] The selection made by the surgeon or operator O in step U40 can be confirmed by a graphically displayed confirmation button accessible via touchscreen operation, and can also be guided by a text box containing information about how many screw extender SEs were detected, as illustrated in Figure 2F, and information about how many screw extenders were selected, with the front row of three screw extenders selected. The selection or deselection of screw extenders with SFs can also be highlighted or dehighlighted by graphic elements so as to provide the surgeon or operator O with visual feedback regarding the status of the selected SEs.

[0058]

[0069] After confirming the selection of screw extenders in step U40, method 200 proceeds to step C20, in which the geometric shape of the rod mounting positions of the pedicle screws PS is calculated. For example, in this step, the geometric shape may include coordinate data of all mounting center points AP of the spinal stabilization or fixation rod R, which can be calculated based on the data of the detected and selected screw extender SE from steps C10 and U40. In the modifications described herein, this step determines the mounting center point AP of each pedicle screw PS of each hypothetically positioned or virtual rod R for each selected screw extender SE, taking into account that the spinal stabilization or fixation rod R is reduced and positioned at its final position with the screw head SH for spinal stabilization, as shown in the exemplary embodiment of Figure 1C, and each pedicle screw PS is mounted to the corresponding screw extender SE. This rod is considered hypothetical or virtual because it has not yet been positioned on the pedicle screws PS of the surgical incision. In the illustrated modification, the mounting center point AP of the rod R is defined as the intersection of the central axis CA of the screw head SH and screw extender SE assembly of the pedicle screw PS and the rotational central axis of the rod R, when the rod R is fully positioned within the U-shaped groove UG of the screw head SH. However, with regard to the type of screw head and other considerations, the mounting center point AP can be defined differently. By determining all mounting center points AP in the geometric space in step U40, for example in three-dimensional Euclidean space, it is then possible to propose a rod shape or template RT for the positioning and mounting of the pedicle screw PS onto the screw head SH.

[0059]

[0070] Step C20 can perform geometric calculations based on the dataset of posture data information PDI calculated by step C10, which detects the screw extender SE. Since the screw head SH is usually fully inserted or has a fixed mounting position relative to the screw extender SE, it can be assumed that the screw head SH of the pedicle screw PS has a fixed position relative to the corresponding screw extender SE to which the pedicle screw PS is removablely attached. This allows the three-dimensional coordinate position of the mounting center point AP to be calculated using the coordinate and orientation data from the posture data information PDI for each selected screw head. It should be noted that although the pedicle screw PS is located within the surgical incision SI and is therefore not visible or only partially visible from outside the body L, it is still possible to calculate the mounting center point AP based on the detection of the placed screw extender SE. For example, this can be done by determining the coordinates of different mounting center points AP using Cartesian coordinates, linear equations, distance calculations, and surface equations. For example, this can be done by first determining the linear equation of the central axis CA of the corresponding screw extender SE, using the attitude information of the screw extender SE, and then calculating the position of the mounting center point AP, which is at a fixed distance from the same fixed position as all screw extender SEs.

[0060]

[0071] In addition to the mounting center point AP, additional relevant information can be calculated to determine the proposed rod template RT. For example, in the modifications of Figures 1A and 2F in which the rod template RTs for three screw extenders SE1, SE2, and SE3 are determined, not only can the three coordinate points of AP be used for the geometric shape, but the geometric shape of the rod mounting position can further include the orientation of the screw head SH in coordinate space, represented by the direction DCA or axis of the central axis of the virtual rod R, for example, positioned in a U-shaped groove. Based on the orientation data information PDI of the screw extender SE, data representing the direction DCA of the central axis of the virtual rod R can be calculated for each mounting center point AP. Generally, the screw head SH is firmly mounted to the corresponding screw extender SE so that the central axis of the screw head SH coincides with the central axis CA of the screw extender SE, and this direction corresponds to the direction of groove extension of the U-shaped groove of the screw head SH, and the bone anchor component of the pedicle screw PS may have different orientations due to its multiaxiality.

[0061]

[0072] In one variation, the calculation of the geometric shape may also be part of another step and may be calculated in advance, for example, as part of the detection of the screw extender SE by step C10 after data regarding the positioning or orientation of the screw extender has become available, and the order of the steps of method 200 proposed herein is merely illustrative.

[0062]

[0073] Next, in step C30, a dataset representing the geometric shapes of one or more rod templates RT can be calculated based on the geometric shape of the rod mounting position determined in step C20, the dataset referred herein to as rod template data RTD, and the geometric shape of the rod mounting position includes, for example, the determined mounting center point AP and / or the direction of the central axis DCA. For example, considering the coordinate data of the mounting center point AP and the direction of the central axis DCA, the appropriate geometric shape of the rod template RT can be calculated, for example, by using a curve fitting algorithm, for example, curve fitting that provides a geometric fit to the mounting center point AP, or by also considering the direction of the central axis DCA from step C20, or, for example, the minimum possible or allowable bending radius of the bent rod R, maximum Curvature By using a fitting algorithm that takes into account the bending limits of the actual physical spine stabilization rod R, considering the maximum lateral dimension, and its physical limitations, it is possible to determine what can be considered the best fit for the current position of the mounting center point AP. It is also possible to determine the rod template data RTD as a series of interpolated discrete three-dimensional points located between adjacent mounting center points AP in three-dimensional coordinate space. In this step C30, the total length of the rod template RT can also be calculated and the calculated length can be stored in the rod template data RTD.

[0063]

[0074] In another embodiment, in step D42, a window or other graphic element may be displayed on the graphical user interface of the display to show the selected rod template RT at a one-to-one scale of the actual physical embodiment of the rod. This can be done by double-clicking or otherwise selecting the rod template RT from a list, for example, a list calculated for the best mechanical fit, via a graphical button, context menu item, or other selection operation using the graphical user interface. This allows the surgeon or operator to directly compare the actual physical rod R with the scaled-down rod template RT simply by holding the actual physical rod R on the display screen, and the operator or user can switch between different pre-calculated or determined rod template RTs to graphically verify their fit and suitability. In one variation, the rod shape of the rod template RT may be bent, stretched, or otherwise deformed or reshaped by touchscreen operation, for example, by moving the portion of the graphic element showing the rod template RT laterally by finger operation on the touchscreen. The modified virtual rod template RT can be displayed again with respect to a selected reference or zero point, for example, one of the mounting points AP1, AP2, or AP3, and the offset distance from each mounting point can be recalculated. The display step D42 and the recalculation of parameters related to the rod template RT can be repeated until the operator or surgeon O is satisfied with the rod template RT for use.

[0064]

[0075] A list of coordinates or other descriptive data for multiple different pre-bent rod templates RT may be pre-stored in a dataset or structure, for example, in the memory of the data processing device 100 or on a server accessible by the data processing device 100, and this dataset may then be compared to best fit the geometric shape of the rod mounting position, including, for example, a determined mounting center point AP and / or the direction of the central axis DCA. This allows for the identification of one or more rod templates for presentation to the surgeon or operator O. The performance of step C30 may also be displayed to the surgeon or operator O on the data processing device 100, for example, by a progress bar or circle, an animated waiting symbol, as illustrated in Figure 2G.

[0065]

[0076] Next, Method 200 can proceed to step D40, where different information regarding the rod template RT, mounting center point AP, and central axis direction DCA can be displayed to change and visualize different parameters, and a user interface related to this information can be displayed in the GUI. Exemplary screenshots are provided in Figures 2H to 2I. This can be done while a live video feed is still displayed on the display device 120 of the data processing device 100 to provide augmented reality functionality for the application and thereby provide visual feedback on the accuracy and fit of the rod template to the pedicle screw PS. As exemplified in Figure 2I, different information can be displayed as a graphic overlay on the live video feed, including graphic elements that visualize the calculated mounting center point AP representing the geometric shape of the rod mounting position, which is the line of each selected screw extender SE, which is the projection of the central axis CA of each screw extender SE, and the mounting center point AP located at the projection position of the mounting center point AP. In this augmented reality aspect, the coordinate data of the mounting center point AP can be mapped or projected into the coordinate space of the display. Furthermore, a graphic representation of the rod template RT selected or determined in step C30 can be displayed, including, but not limited to, characteristic data such as thickness, length, bending radius, and bending pattern. In the illustrated variant, the bent rod template RT is displayed in a box in millimeters along with its length.

[0066]

[0077] Furthermore, as shown in Figure 2I, step D40 can also display the same bent rod template RT in the box, but positioned to coincide with at least one of the mounting center points AP to show a graphical representation of the rod template RT installed with three exemplary pedicle screws PS. In the illustrated modification, the rod template RT is displayed so that its central axis coincides with one of the center mounting center points AP2, where AP2 acts as the zero point or reference point and represents the installation of the pedicle screw PS2 onto the screw head SH2. Next, for the other mounting center points AP and pedicle screws PS1, PS3, the distance from the rod template RT to the mounting points PS1, PS3 can be displayed, so that the surgeon or operator O can verify how well or poorly the currently selected rod template RT fits the pedicle screws PS1, PS3 adjacent to the reference point.

[0067]

[0078] For example, assuming the rod template RT is a straight line, the geometric calculation in three-dimensional (3D) space can be performed by locating two geometric surfaces GS1 and GS2 perpendicular to the line, with mounting center point AP1 on one surface and mounting center point AP3 on the other, in order to determine the distance from the straight rod template RT to the mounting center points AP1 and AP3. Then, the distance between the adjacent mounting center points AP1 and AP3 and the point defined by the intersection of each surface and the line provides the definition of these two distances that can be displayed. If the rod template RT is curved, the same method can be used by determining two surfaces GS1 and GS2, each perpendicular to the tangent line located at the intersection point of their respective surfaces GS1 and GS2, and mounting center points AP1 and AP3 also being located within one of surfaces GS1 and GS2. This allows the distance from mounting center points AP1 and AP3 to the rod template RT to be determined. As illustrated in Figures 2I, 1J, and 2K, these distances can be associated with a centerline CA indicating the longitudinal extension of the screw extender SE and displayed in millimeters within a box for highlighting or easy reading. An arrow, pointer, or other directional graphic element can be associated with the distance value to indicate the direction of the offset distance from the rod template RT. For very small distances, an indicator can help identify the direction of the offset distance. These distance values ​​can be displayed in a graphical user interface or as a movable text screen with a graphical association between each of the screw extender SE and pedicle screw PS assemblies, for example, with each centerline CA displayed for the screw extender SE.

[0068]

[0079] In the modified example shown in Figure 2I, the reference point or position for measuring the offset can be changed, for example, in step U55, to determine a different zero point or reference point. For example, in step U55, the operator or surgeon O can select a different reference point or zero point by simply touching, pressing, or otherwise selecting a graphic element representing one of the mounting center points AP1, AP2, AP3 in the GUI, thereby resetting the zero point or reference point and recalculating the offset value for the new reference point. As another example, the user can select one of the screw extenders SE1, SE2, SE3 as one of the zero point or reference points, as illustrated in Figure 2H. Furthermore, the recalculation of all offset values ​​can be obtained automatically or performed by pressing or touching a button, as indicated by the virtual button “Remeasure,” upon confirmation or request by the operator or surgeon O, as illustrated in Figures 2H, 2I, and 2J.

[0069]

[0080] Figure 2N shows one variation of the screen that may result from step D40, where three exemplary different mounting points AP1, AP2, and AP3 are shown, and visual feedback is provided to the operator or surgeon O regarding the offset of mounting points AP1 to AP3 relative to the placed rod template RT as a function of their distance from the selected and placed rod template RT. For example, mounting point AP3 is shown to be the furthest from the rod template RT, with a calculated offset distance of approximately 8 mm, and thereafter mounting point AP3 is highlighted in red, e.g., a red dot, or other types of highlighting indicating that the selected rod template RT is not suitable for the placement and mounting of the corresponding pedicle screw PS3. In contrast, mounting point AP2 is shown to be located on or within an acceptable proximity range to the rod template RT, and therefore can be highlighted in green, e.g., a green dot, or other types of highlighting. The offset is measured as 0 mm. This indicates that the selected rod template RT was suitable for placement at this particular mounting point AP2. Similarly, as shown in Figure 2N, mounting point AP1 is highlighted in orange, indicating a less-than-ideal but somewhat appropriate position with an offset distance of 4 mm. In this regard, an increase in the distance of mounting point AP from the rod template RT, which is positioned to connect to one of mounting points AP1, AP2, or AP3, can be indicated by coloring or other types of visual feedback. In the illustrated modification in Figure 2N, a heatmap coloring scheme is used, where green represents a good match between one of the APs and the rod template RT, and a change from green to orange and then red represents a poor match of the rod template, such as an offset value outside the range in which the rod can be bent.

[0070]

[0081] Furthermore, Method 200 may perform step D50 to display a list LL of rod templates RT that can be selected by the operator or surgeon O, visualized at the surgical incision SI using a live video feed, and enable the operator or surgeon O to perform a visual inspection of the rod placement using augmented reality. For example, this step may display a list of rod templates RT found based on step C30 in which the rod templates RT are calculated, e.g., the one that best matches the geometric shape of the rod mounting position, or a list of rod templates RT from a pre-stored selection. With the displayed list LL, the data processing device 100 is configured to enable the operator or surgeon O to graphically select one of the rod templates RT by step U50, and the selected rod template RT can then be displayed by step D40 as to be virtually connected to or placed at at least one of the mounting center points AP, as shown in Figure 2J. Also, when selecting a rod template RD and virtually placing it on the pedicle screw PS, the offset value can be calculated and displayed.

[0071]

[0082] In this regard, step D55 is performed, and one of the selected rod templates RT can also be displayed as a 1:1 scale graphic element on the display screen or graphical user interface GUI. This can be done with two 1:1 views, for example, a sagittal or longitudinal plan view and a coronal or front plan view, to assist the user or operator O in manufacturing the corresponding rod. If the rod template RT is too long to fit on the screen, for example, longer than a typical tablet screen, it is possible to maintain a 1:1 view scale, but a scroll option is used in the GUI.

[0072]

[0083] In an optional step of Method 200, the surgeon or operator O may be given hints on how to position or adjust another pedicle screw PS, for example, a fourth pedicle screw PS4, as shown in the surgical scene images from Figures 2D to 2K, to match the coordinates of the selected and positioned rod template RT. For example, a graphic element may be displayed that extends from the selected rod template RT and has a graphic element indicating the potential position of the next pedicle screw PS4 mounting location. For example, referring to Figure 2J, a curved rod template RT is shown positioned at the mounting center points AP1 to AP3 of three different pedicle screws PS1 to PS3, and a linear or triangular graphic element may be displayed showing a dot, cross, or other graphic element to indicate the next potential mounting location of the pedicle screw PS4. The triangular graphic element may have corners at the ends of the rod template RT to show various possibilities for the mounting of the pedicle screw PS4.

[0073]

[0084] In another optional step of Method 200, select a specific screw extender SE and pedicle screw PS assembly. correction before, correction During operation and correction Compare the subsequent positions, as explained further below, for example correction before, correction During operation and correction By calculating, displaying, and processing the subsequent different mounting center points AP1 to AP3, it is possible to collect data on the changes in their geometric positions relative to each other.

[0074]

[0085] Next, in step C60, based on the rod template RT selected by the operator or surgeon O in step U50, the rod template data RTD from the selected rod template RT can be processed to generate CAD data or other data that can characterize the rod R resulting from the rod template RT, which can be used to manufacture the physically fixed rod R. The CAD data can be sent to a rod bending machine or another type of rod processing equipment to manufacture the actual physical rod manufactured in step F10. The rod template data RTD can be provided by step U20, or indirectly after being shown in a one-to-one representation in step D55, or simultaneously. In this step, the data for manufacturing the rod R is selected by the operator or surgeon O in step U50 based on the RTD from step C30. Next, the geometric data of the selected rod can be extracted from the RFT data and converted to different data formats, such as CAD data format standards, e.g., STEP, IGES, Parasolid, STL, VRML, X3D, DXF, COLLADA, etc., for example, but not limited to these. For example, at least one dataset for one rod from an RDT can be transmitted to a rod bending or processing machine, such as a rod bending device as described in U.S. Patent No. 6,755,064, U.S. Patent No. 10,405,908, or U.S. Patent Application Publication No. 2005 / 0262911, and these references are incorporated herein by reference in their entirety.

[0075]

[0086] Another optional step of Method 200 is step C70, which calculates an estimate of the postural data information PDI_V for each vertebra V attached to a pedicle screw PS, and an optional display step D70, which displays graphic basic elements on a live video feed or a display image of each vertebra V to show the estimated or calculated position of the actual vertebra V of the spine SC, and the spine CurvatureData SCD or other spinal characterizing parameters or parameterized PARs of the spinal column SC, such as, but not limited to, Cobb angle, sagittal plane angle, and other spinal parameters, can be calculated, thereby enabling the visualization of the living spine during surgery without requiring invasive medical imaging, such as X-ray imaging. Curvature Another optional step C75 is to calculate an estimate of the position. By detecting different screw extender SEs in step C10 and providing posture data information PDI for each detected screw extender SE, it is possible to estimate at least the posture data information PDI_V of vertebra V even if the spinal column SC is not visible in the live video feed image. Since typically two pedicle screw pairs PS are attached to each vertebra V, this step allows for calculating a position estimate based on two different posture data information PDIs of two different screw extender SEs, for example, two adjacently arranged screw extenders SE1, SE4, both attached to the same vertebra V, as shown in Figure 1D.

[0076]

[0087] While the exact geometric relationship between the screw extender SE and the vertebra V may not be known, there is a probability range that can be used for approximate estimation. Using two posture data information PDI datasets for two screw extender SEs attached to one vertebra V, an estimated posture PDI_V for each vertebra V can be provided, for example, by using the average value of the two screw extender posture PDIs. Furthermore, based on historical data of the geometric relationship between the position or posture of the screw extender SE, which has a fixed position relative to the screw extender SE, and the position or posture of the vertebra V, a knowledge database can be generated to use the position most likely to be taken by the vertebra of the spinal column SC, taking into account the detected PDIs to which the two screw extenders are attached. For example, in step C70, for the purpose of calculation and estimation, it can be assumed that each pair of pedicle screws PS has an ideal predetermined arrangement within the vertebra V of a given vertebra V, and that the mounting position of the pedicle screws PS with respect to the perforation central axis with respect to position and orientation is selected to be such an ideal predetermined arrangement position based on the normative size of the vertebra V. Postural data information PDI is detected from step S10 for a pair of screw extenders SE. Assuming that the pair of screw extenders SE are attached via pedicle screws PS to approximate such an ideal position, the position and orientation of the corresponding vertebra V can be approximated. The postural data information PDI_V of the corresponding vertebra V can then be calculated by geometric transformation of the coordinates to obtain the postural data information PDI_V.

[0077]

[0088] Steps C70 and C75 can be based on estimation, calculation, or decision using a knowledge database, which may include historical information regarding correspondence or mapping between attachment points AP of different pedicle screws PS, or posture data information PDI of different screw extenders SE, as well as corresponding vertebral position and orientation information as PDI_V, and spine. CurvatureThe data includes SCD, spinal characterization parameters PAR, or a combination thereof. This allows for the collection of vertebral posture data information PDI_V, and spinal Curvature It is possible to create or establish an artificial intelligence network, such as a convolutional neural network (CNN), decision forest, or other type of network trained on a knowledge database, to determine the data SCD, spinal characterization parameters PAR including the Cobb angle and sagittal plane angle, or a combination thereof, from the detected attachment point AP, posture data information PDI, or a combination thereof. Since the posture data information PDI and attachment point AP have a deterministic and computable geometric relationship, PDI_V, SCD, or PAR can be determined directly from the PDI of the detected screw extender SE. However, in one modification, the posture data information PDI_V of the vertebra, or the spine Curvature The calculation of data SCD is omitted, and the posture of the spine itself or other types of positioning is determined based on either the mounting point AP of different pedicle screws PS, the posture data information PDI of different screw extenders SE, or both. Curvature Without calculating the data PDI_V and SCD, the spinal characterization parameter PAR can also be directly calculated or estimated, and ultimately the user or operator O can then... correction To determine the spine correction I am particularly interested in these parameters PAR for surgery.

[0078]

[0089] Furthermore, based on the estimated posture data information PDI_V obtained in step C70 in this way, the spine Curvature The data SCD is calculated, and thereby the living spine L during surgery is examined. Curvature Step C75 can be performed to calculate the estimated value of the spine. This can also be based on a knowledge database, and patient-specific parameters and values ​​based on the patient's age, weight, height, etc., can be taken into account. CurvaturePossible spinal curves can be calculated as data SCD. Furthermore, step C75 can also calculate spinal characterization parameters PAR based on a geometric transformation based on the vector representation of PDI, PDI_V in Euclidean coordinate space, using a typical algorithm for such determination, e.g., based on the posture data information PDI of the screw extender SE in step C10, or based on both PDI and PDI_V from steps C10 and C70, e.g., but not limited to this. This data can then be used to analyze the spine of a living organism through an automated process. Curvature at least partially correction To do so, automated spinal column or spinal column correction (rectification) or correction (Correction) Can be used in devices or systems. Spinal correction systems can be used, for example, automated spinal correction systems. correction For this purpose, it may take the form of an electric actuator, a robotic device, or an operating table having a pillow with an expandable chamber, and a system such as those described in Chinese Patent Application No. CN108143582 or No. CN110279554, or a similar machine, can be used.

[0079]

[0090] In these embodiments, Method 200 can calculate different postural and positional information of the spine or vertebral column of a living organism L. For example, it is possible to calculate different types of spinal characterization parameters or parameterized PARs, including, but not limited to, sagittal alignment of the lumbar spine or lumbar lordosis, and, but not limited to, lordosis inclination angle, overall lordosis, sacral inclination, lordosis distribution index, lumbar apex position, upper arc angle, relative spine-pelvis alignment, sagittal alignment of the thoracic or cervical spine, kyphosis such as the Cobb angle, parameters of sagittal balance, and other parameters. It is also possible to calculate different geometric parameters associated with the cystic condition of the spine or vertebral column.

[0080]

[0091] For example, Figure 3A shows a schematic and simplified representation of a spine or vertebral column having seven exemplary vertebrae V1-V7, where for each vertebra V, two attachment point pairs AP1.1 and AP1.2 are determined in calculation step C20. Based on the calculated set of attachment point APs, seven pairs of attachment point APn.1 and APn.2 are shown, where n is from 1 to 7 in this example, and different parameters related to the posture and orientation of the spine or vertebral column can be calculated in additional steps of the method.

[0081]

[0092] For example, in step C70, posture data information PDI_V for each vertebra V, which includes three-dimensional (3D) position and orientation information VP1~VP7, can be calculated for each vertebra V1~V7 based on geometric position data from a pair of attachment points APn.1 and APn.2. If two pedicle screws PS with screw extenders SE are attached to each vertebra V, and the exact position of each AP relative to the vertebra is not 100% defined, then since two different geometric points AP are available for each vertebra V, the average or geometric intermediate position of both attachment points APn.1 and APn.2 can be used to calculate the VPn associated with each vertebra at a more accurate position. This calculation can further take into account statistics based on historical data and statistical variability of attachment point positions AP, for example, and a trained artificial network can be used, as described above. The three-dimensional (3D) position and orientation information VP1~VP7 for each vertebra can also be used to display the basic graphic elements of each vertebra in a live video feed, as described above with respect to display step U70.

[0082]

[0093] Next, the spine Curvature In another step C75, which calculates the data SCD, the geometric model or coordinate data of the spinal curve can be calculated based on the posture data information PDI_V for each vertebra V, for example, the 3D position and orientation information VP1~VP7 calculated for each vertebra V1~V7. CurvatureThe data SCD can be a curve determined by curve fitting with geometric points VP1-VP7, or by characterizing it with a series of geometric positions in 3D space. However, other data or parameters related to the spine and vertebrae V1-V7 can be calculated in this step C75. For example, the distance between each vertebra such as D12, D23, D34, D45, D56, and D67 can be calculated, for example, based on the distance between the geometric points of adjacent vertebrae, and the orientation angle β between adjacent vertebrae V can also be calculated, for example, the orientation angle of two adjacent vertebrae V when viewed from different directions, for example, when viewed from the back, when viewed from the front, or when viewed from either side.

[0083]

[0094] Furthermore, in step C75, as described above, spinal characterization parameters or parameterized PARs can also be calculated. Generally, different geometric and orientation parameters PAR of the spine or vertebral column can be calculated based on the posture data information PDI_V of each vertebra V, which includes, for example, calculated 3D position and orientation information VP1~VP7, and can be stored, displayed, archived, and reviewed by the surgeon or operator O. As an example, for different types of spinal surgery, spinal characterization parameters PARs can be calculated, including, but are not limited to, parameters such as lordosis inclination angle, overall lordosis, sacral inclination, lordosis distribution index, lumbar apex position, upper arc angle, relative spine-pelvis alignment, sagittal plane alignment of the thoracic or cervical spine, Cobb angle, kyphosis, sagittal plane balance, and other parameters. As another example, based on the posture data information PDI_V of each vertebra V, which may include data on the orientation of each vertebra V, it is possible to calculate the rotational orientation toward each other of adjacent vertebrae.

[0084]

[0095] The graphical user interface (GUI) can also be configured so that a surgeon or operator can select two vertebrae by, for example, clicking or selecting a basic graphic element on the display, and then display different parameters for these two selected vertebrae, such as distance, rotational direction relative to each other, and posture information, so that they can compare, for example, angular directions.

[0085]

[0096] Display step D70 can be performed, where graphic basic elements representing different vertebrae V can be displayed, such as the spine as a straight or curved graphic element. Curvature For example, the spine calculated by step C75 Curvature Based on the data SCD, a head-up display is used to overlay a live video feed of the surgery or the direct view. Also, in display step D70, correction The calculated spinal curve CSC and all the different calculated parameters characterizing the spine can be displayed. In the modified form of Figure 3A, this can be a straight line because the ideal spinal curve viewed from the back is a straight line.

[0086]

[0097] In another embodiment, method 200 allows for measurements of the spine or vertebral column via the screw extender SE at different points in time during surgery. For example, the surgeon or operator O can first image and detect the screw extender SE in step U30, C10, and then calculate different parameters based on steps C20, C40, C70, C75. After selecting and positioning the rod R, the user or operator O can insert the rod into the open slit of the screw extender SE and then engage in rod reduction of the rod R so that the rod R moves downward into the groove of the screw extender SE, positioning the rod R in the U-shaped groove of the screw head and holding it in place by the set screw of each pedicle screw PS. During the reduction process, the rod R pushes the vertebrae into their new positions. This, once measured, results in a new alignment and new spine. Curvature Spine curves in data SCD correction Or change, for example, coronal, sagittal, and axial correction This results in the following: At this stage, or at any other point during the reduction process, before removing the screw extender SE from the pedicle screw PS, the operator or surgeon O may re-engage in steps U30, C10 to re-detect all screw extender SE and re-determine the mounting points AP, for example, the posture data information PDI_V for each vertebra V, including calculated 3D position and orientation information VP1~VP7.

[0087]

[0098] Subsequently, based on the repetition of steps C10, C70, and C75 correction Before and correction The spine that was determined later Curvature Using the Data SCD, for example, using a graphical user interface (GUI), display different parameters and data in display step D80, showing the rod before and after installation. correction Alternatively, it is possible to show changes. Step D80 can, for example, use rows or columns of two tables, pre- and post-operative, of different spinal parameters PAR or spinal parameters, pre-, post-, or both, for comparison purposes, for example, using rows or columns of two tables of different SCD or PAR. Curvature The data SCD can be displayed. This allows operator O to visually compare the data with other representations on the GUI or screen. correction If the condition is insufficient or outside the preferred range, the rod R connected to the screw head SH can be removed or unlocked, and different Curvature Alternatively, a rod R having a certain shape can be placed on the screw head SH. CurvatureAlternatively, the shape can be modified by an instrument placed on a set screw or screw extender SE, after which the rod R is tightened again onto the screw head SH by the set screw. Different spinal characterization parameters PAR can be displayed and visualized using a graphical user interface GUI, making it possible to compare the most appropriate spinal characterization parameters PAR, including, for example, data before and after rod placement, such as, but not limited to, the Cobb angle, sagittal plane angle, and lordosis inclination angle.

[0088]

[0099] In addition to using the screw extender SE for detection and tracking in steps U30 and C10, a tool SD operably mounted on the screw extender SE SC may also be used, as shown in Figure 3B. For example, the tool SD may be a set screwdriver or rod shortening tool for rod shortening, or a screwdriver for screwing the bone anchor of the pedicle screw PS into the vertebra. This allows the tool SD to fix the axis between the screw extender SE and the bone anchor BA of the pedicle screw PS, thereby having a defined orientation relationship. For this purpose, the shape of the tool SD can be tracked and detected, or the tool SD may be equipped with optical markers OM, in the modified example shown in Figure 3B, where two optical markers OM are positioned on the top and bottom of the handle of the tool SD.

[0089]

[0100] rod correction Before, rod correction Middle, and rod correction By visualizing subsequent changes in the spine or vertebral column, the operator or surgeon can, through the above-mentioned measurements and displays, examine the spine. correction To directly confirm the extent of the impact. A rod template RT that is determined to be appropriate can then be created as an actual physical embodiment to be placed inside the living organism L. Based on the data of the rod template RT, spinal posture information, and the position of the attachment point AP, correctionBefore the procedure is performed, i.e., before the rod is attached to the pedicle screw PS, it is possible to calculate the most likely spinal posture to be obtained from the selected rod template RT.

[0090]

[0101] Furthermore, using a database accessible from different devices 100 that record surgeries, it is possible to create training data for future surgeries and deep learning using different types of artificial intelligence (AI), such as for training convolutional neural networks. For example, for each surgery, video data including screw extender posture information, attachment point AP, rod template, and vertebral position, as well as calculated and detected metadata, can be stored and indexed in the database and used as training data and archives for the database.

[0091]

[0102] Under certain circumstances, the angular orientation between the bone anchor BA and the screw head SH of the pedicle screw PS is not fixed, but is limited to a specific angular range, for example, ±27°, or by the use of multi-axis or multi-axis pedicle screws PS having other angular ranges. In such cases, since the screw extender SE is attached to the screw head SH of the pedicle screw PS, the orientation of the screw extender SE relative to the bone anchor BA may be unknown or invisible. Method 200 can calculate the attachment point AP of the screw head SH depending on the position and orientation of the screw extender SE, and then calculate VP1 to VP7, for example, by calculating the postural data information PDI_V of each vertebra V. Therefore, when the orientation between the screw extender SE or screw head SH and the bone anchor BA is unknown and cannot be seen from outside the surgical incision SI, the calculation of the postural data information PDI_V of the spine can have a relatively high margin of error due to this uncertainty. In such circumstances, the operator or surgeon O can instruct all screw extenders SE to move to the end of the angular range so that the articular joint formed between the screw head SH and the bone anchor BA is at the maximum angular point, thereby fixing and knowing the orientation relationship between the screw extender SE and the screw head SH and bone anchor BA to a certain extent.

[0092]

[0103] For example, as shown in Figure 3C, all three exemplary visualized screw extenders SE are moved in the same direction and tilted by a maximum orientation angle of 27° for the multi-axis or multi-axis pedicle screw PS, and in a modified example, shown along the extension direction of the spine or vertebral column. This step can be commanded to the operator or surgeon O before step U30, in step D25 of Method 200, by a graphical user interface or other type of command, such as by voice command, animation, etc., before the operator or surgeon O engages in step U30, in which the operator or surgeon O scans the surgical incision SI and the screw extender SE. The command may include the display of an arrow or pointer shown on a live video stream to indicate the direction of movement for positioning the screw extender SE at an lateral angular position relative to the screw head SH. In one modified example, it is possible to provide a fixed angular relationship between the screw extender SE and the bone anchor BA using a tool that engages with the bone anchor BA and is inserted into the screw extender SE, thereby providing the temporary uniaxiality of the pedicle screw PS for measurement and calculation in steps C10, C20, and C70. This tool may be a screwdriver SD itself, which can engage with a portion of a bone anchor BA via a screw extender SE, as illustrated in Figure 3B, for example, with the torque drive mechanism of the bone anchor BA or other elements of the bone anchor BA, thereby reorienting the screw head SH to the same extension axis of the bone anchor BA, thereby orienting it as a single-axis screw configuration. Such a tool may also be attached to the bone anchor BA for the purpose of orienting the screw head SH without using the screw extender SE, or when the screw extender SE is removed. Furthermore, such a tool may be equipped with an optical marker OM for detection efficiency, as shown in Figure 3B.

[0093]

[0104] Method 200 is not limited to implementation using the portable data processing device 100, but can also be implemented using a non-portable system, such as a multi-camera system with fixed cameras, a data processing device or server, and an interactive screen. In such modifications, it is possible to use multiple cameras that provide different field of view of the surgical incision SI, thereby providing image data for three-dimensional determination, and a live video feed and GUI are displayed on a display screen located in the operating room. An algorithm can be operated on the data processing device that can switch camera views depending on whether the camera view is obstructed or not. Alternatively, instead of a touchscreen, another type of input device can be used, such as a mouse, a laser pointer with a corresponding screen, or other input device that can read the hand movements or instructions of the operator or surgeon O.

[0094]

[0105] In another embodiment, the data processing device 100 may also include wearable augmented reality (AR) glasses, a head-mounted display having a transparent or translucent display screen, or a head-up display (HUD), the glasses or display also including a camera for capturing image sequences for tracking and detecting the screw extender. For example, a system such as that described in U.S. Patent No. 10,854,098 can be used, which is incorporated herein by reference in its entirety. This makes it possible to provide a see-through augmented reality system in which a live video feed is directly transmitted through a transparent display screen. Visible Therefore, it may not be necessary to display the live video feed for step D10. Graphic elements, such as the basic graphic element GP for the screw extender SE, the selector element SF, text boxes, rod template RT, and other elements of the graphical user interface, can still be displayed on a transparent display screen.

[0095]

[0106] In another embodiment, different radiopaque marker ROMs, or other types of markers that can be detected by X-ray or CT scanning, or by other types of medical imaging, can be placed on the skin of the living body L during surgery. For example, the ROM markers can represent QR codes or other types of optical codes. This allows for intraoperative imaging using marker ROMs positioned to connect intraoperative patient images, for example by X-ray using a C-arm, or CT scanning via an O-arm or 3D C-arm, to determine the position and orientation of bone anchors BA and screw heads SH. These positions can then be matched with the posture information of screw extenders SE by either 3D shape matching with image data from image sensors or QR code matching.

[0096]

[0107] In another aspect of the method presented herein, it is possible to provide an operator or surgeon O with guidance for positioning a screw extender SE to facilitate rod insertion. It can be difficult to percutaneously insert a rod into a long structure with a relatively large number of screw extenders SE and pedicle screws PS, as not all pedicle screws PS may be aligned. For example, one may be positioned more laterally and the other more inward. However, based on the known spatial position of each pedicle screw PS, for example by the mounting center point AP, the surgeon or operator O can tilt different screw extenders SE to the opposite side of where the screw is misaligned or misplaced relative to the other. In the case of a laterally positioned pedicle screw PS, the surgeon or operator O can tilt the screw extender SE inward, and for an inwardly positioned pedicle screw PS, the screw extender SE can be tilted laterally. This repositioning of the screw extender SE, and the resulting reorientation of the screw head SH, can provide better alignment of all slots or openings in the screw extender SE, thereby facilitating the insertion of the rod R.

[0097]

[0108] Another aspect of the present invention includes a method 500 for scanning, displaying, and verifying a spinal rod R bent for attachment to an attachment point AP. In an exemplary flowchart of method 500 illustrated in Figure 4, the actual spinal rod R can be scanned and visualized using a live video of the surgical incision with a live video feed using augmented reality, with the spinal rod R as a template RT in relation to different attachment points AP defined, for example, by pedicle screws PS attached to the spinal column SC.

[0098]

[0109] In method 500, a spinal rod R bent by a surgeon, operator, or user O can be scanned, photographed, or imaged, for example, using a scanning step U100, and subject to a calculation step by a data processing device 100 or 320. It is also possible that the actual rod R is manufactured by step F10 as described above, for example, based on a selected rod template RT. For example, this step can also be assisted or complemented by three-dimensional data or depth data from a time-of-flight sensor, such as a Lidar sensor. Next, step C110 can be performed, and geometric data representing the rod R can be calculated as a rod dataset RD based on the captured image data, for example, a video sequence or image sequence with views of the rod R from different angles, or three-dimensional or depth data. Next, in step D40, a projection or rendering of the actual rod R as a rod template can be performed, as shown in Figures 2H, 2I, 2J, 2K, and 2N, and the readout rod template RRT can be displayed and selected, for example, to be attached to one of the mounting points AP. In step U55 described above, the reference mounting point AP as the zero offset point can be changed, and as a result, different arrangements of the actual rod template RRT can be visually verified by operator O. Next, step C120 can be performed, and based on the actual rod template RRT and the initially proposed or selected reference mounting point AP, the remaining mounting points AP are moved or corrected to match the actual rod template RRT.

[0099]

[0110] Step C120 is derived from steps C10 and C75. correction Based on the original data of the previous state spinal curve SCD, the proposed rod R and rod dataset RD of the actual rod template RRT scanned and calculated from step C110, and from step C120 correctionAn approximation of the vertebral curve SC can be calculated, where the position of the corrected attachment point AP can be calculated based on the placement of the RRT that matches the initially proposed or selected reference position of the actual rod template RRT, for example, one of the selected corrected attachment point APs. This calculates a new dataset of attachment point APs that have been virtually moved so that in step C120 the actual vertebral rod R deviates relative to the attachment point AP when it is attached to the pedicle screw PS. Next, method 500 also calculates the posture data information PDI_V of all involved vertebrae V based on the newly calculated virtual attachment points using step C70 described above, and virtually correction To visualize the resulting spinal column SC, the process may include a step of displaying the basic graphic elements of the vertebrae V, or a step of rendering the spinal column SC using step D70 as described above.

[0100]

[0111] The display in step D70 shows the spine SC as a virtual or augmented reality graphic base element based on the bent rod R, so that the surgeon, operator, or user O can virtually verify the effect of the bent rod R on the spine SC. This allows verification of whether the bent rod R has the desired effect before it becomes necessary to attach the rod R to the mounting point AP of the pedicle screw PS. Steps C75 and D80 can also be performed, where the spinal curve data SCD and spinal parameters are calculated and then displayed in step D80. Step D80 also uses the previously performed steps C75 and D80 for the uncorrected position of the mounting point AP. correction It can include display of spinal curve data (SCD) and vertebral parameters for each individual spine. Curvature Data SCD Pre- correction and virtual spine Curvature The display of data SCD shows that operator O bent the rod R relative to the spine SC as desired. correction This makes it possible to verify whether it has an effect, or at least approximates one.

[0101]

[0112] According to another aspect of the present invention, a method 600 is provided for determining different types of information characterizing the spinal column SC before the placement and fixation of pedicle screws PS to each vertebra is carried out, as illustrated and schematicly shown in the flowchart of Figure 5. Thus, method 600 is provided for determining, for example, spinal parameters PAR or spinal parameters Curvature By determining the data SCD, any vertebra can be fixed by the fixation rod R attached to the pedicle screw PS. correction This allows for the calculation of different spinal data and parameters before implementation, and enables the calculation of different vertebral posture data information PDI_V by first detecting different pedicle markers PM that can be inserted into or otherwise attached to different vertebrae V via a guidewire or another equivalent device. This allows for the calculation of different vertebral posture data information PDI_V even before the pedicle screw PS is attached to vertebrae V which defines the attachment point AP of the fixation rod R. correction This makes it possible to perform the step of defining or suggesting the surgical fixation rod R.

[0102]

[0113] For example, as described in U.S. Patent Application Publication No. 2021 / 0169506, which is incorporated in its entirety herein by reference, a guide wire GW is described which can be inserted and positioned into an initial drill hole DH or bore through a surgical incision SI, for example, but not limited to, Kirschner wires, K wires, guide pins, Schanz pins, Denham pins, Steinmann pins, guide rods, and pedicle markers PM which can be attached to a guide shaft, the drill holes being drilled in different vertebrae V of the spinal column SC, and the pedicle markers can be attached to the guide wire. The guide wire GW can be positioned in each drill hole DH to guide the pedicle screw PS into the drill hole or bore for insertion into the pedicle or vertebra V. The bone anchor BA of the pedicle screw PS typically includes a through bore through which the guide wire passes, enabling the pedicle screw PS to be guided into the drill hole or bore formed in the pedicle. The pedicle marker PM can be used to attach to a guidewire or equivalent thereto, for example, but not limited to, those described in U.S. Patent Application Publication No. 2021 / 0169506, enabling the insertion and placement of the pedicle screw PS onto the guidewire, and can also be used to facilitate surgery for the surgeon or operator O by assisting in the placement of the guidewire GW and the removal of the guidewire from the drilled hole.

[0103]

[0114] Figure 6 shows an illustrative and simplified cross-sectional view of vertebra V, where two drill holes DH1 and DH2 are drilled or otherwise created in vertebra V, two guide wires GW1 and GW2 are placed in drill holes DH1 and DH2 respectively, and two pedicle markers PM1 and PM2 are attached to guide wires GW1 and GW2 respectively, and an optical marker component 50 is provided on the optical marker OM, which can be used for robust detection of the pedicle markers PM1 and PM2, guide wires GW1 and GW2, or both, using computer image data processing with a tracking algorithm. In the illustrated modification, the optical marker component 50 is exemplary fabricated as a removable cap, clip, tube, clamp, flag, tab, or other device, each having two flat surfaces for arranging optical markers OM for redundancy, similar to the device 50 shown in Figure 1E. However, the optical markers OM can also be arranged directly on the pedicle marker PM as, for example, an etched pattern, a printed pattern, a punched or embossed pattern, a machined three-dimensional surface or structure, or other markings on the pedicle marker PM. Alternatively, the optical markers OM can also be arranged directly on the guidewire GW, with or without the optical markers OM on the pedicle marker PM, as, for example, a tab, a flag, a longitudinal code along the shaft forming the guidewire GW, or a three-dimensional structure directly representing the code.

[0104]

[0115] Method 600 has several similar aspects to Method 200 described above, but instead of detecting the screw extender SE with or without using the optical marker OM in step C10, a step is performed in which the pedicle marker PM can be detected to determine information characterizing the spine SC. Steps U10, D10, U20, and D20 may be substantially the same as in Method 200 for providing a live video feed on the display 120, providing a GUI for user operation, and inputting calibration information. Step U230 is for the surgeon or operator O to use the data processing device 100 to acquire images of different pedicle marker PMs. imaging This can be performed when scanning a surgical incision SI using the device 110. Next, the data processing device 100 performs step C210 to detect the pedicle marker PM by image data processing, for example, by using a pedicle marker PM with an optical marker OM, or by detecting the shape of the pedicle marker PM by image shape or pattern recognition without using an optical marker, or by detecting an optical marker OM that is directly attached to the guide wire GW itself or is an integral part of the guide wire GW. In this step, the guide wire GW itself may also be detected. The information obtained in step C210 as a result can be posture data information PDI_PM of the pedicle marker PM, posture data information of the guide wire GW, or other types of coordinate data that can characterize the position and orientation of each guide wire GW. Assuming that two guide wires GW or pedicle marker PMs are attached to one vertebra V, this information can be used to determine the position and orientation of individual vertebra Vs of the spinal column SC.

[0105]

[0116] Subsequently, an optional step D225 can be performed to overlay graphic basic elements onto the live video feed, highlighting the pedicle marker PM, guidewire GW, or both, as well as steps D25, D30, and U40, along with step D230 for showing graphic elements for making a selection, and step U240 for receiving input data from the surgeon or operator O to actually select the different pedicle marker PM or guidewire GW detected. Next, an optional step C220 can be performed by the data processing device 100 to calculate the geometric shape, where a virtual mounting point AP_V can be calculated, which is a specific geometric position where the fixation rod R is most likely to be located relative to the corresponding pedicle screw PS, and the pedicle screw PS is not yet attached to or fixed in the vertebra V. Here, the mounting point AP_V is considered virtual because such a mounting point AP does not currently exist. As a result, in step C220, a specific fixing rod R is positioned and attached to these virtual mounting points AP_V, thereby allowing for the operation of different vertebral column SCs currently being manipulated, when direct information about the actual mounting point AP is not yet available. Curvature or vertebral parameterization, and deviation relative to the spinal column SC CurvatureAlternatively, an estimate of the geometric position of the mounting point AP can be provided as a virtual mounting point AP_V, which can be used to estimate spinal parameterization. This calculation can be performed by using artificial intelligence with a trained network, using historical data on the position of the mounting point AP of a given drill hole and a given guidewire GW placed in the drill hole from image data, or based on past medical imaging data, for example, but not limited to X-ray images, or using tables or other pre-stored information on statistical data of the geometric relationships between the position and orientation of the guidewire GW, the position and orientation of the pedicle marker PM, and the position and orientation of the pedicle screw PS attached to the vertebra V, which define the position of the mounting point AP.

[0106]

[0117] Next, a step C270 similar to step C70 of Method 200 can be performed, which is configured to calculate posture data information PDI_V for each vertebra V associated with the guidewire GW, or the guidewire GW and pedicle marker PM, and an optional display step D70 similar to the same step of Method 200 can be performed to display graphic basic elements on a live video feed or a display image of each vertebra V, providing live video feedback in augmented reality to show the estimated or calculated positioning of the actual vertebra V of the spine SC projected onto the live video feed. Step C270 can use posture data information PDI_PM of the pedicle marker PM or posture data information of the guidewire GW for two or more vertebra V, or it can use data of a pair of virtual attachment points AP_V for two or more vertebra V from step C220, or for both datasets AP_V and PDI_PM.

[0107]

[0118] Also, similar to method 20, another optional step C75 can be performed, where the spine of the spinal column SC Curvature Data SCD or other spinal characterization parameters or parameterized PAR, e.g., geometrically characterizing the approximation of the current spinal curve. CurvatureData SCD can be used to calculate vertebral parameterized data such as the Cobb angle of the spinal column (SC), sagittal plane angle, axial angle, distance between adjacent vertebrae, and other parameters, thereby enabling the calculation of the living spine during surgery without requiring invasive medical imaging such as X-ray imaging, even before pedicle screws (PS) are placed or fixed. Curvature An estimate of the value can be calculated. This data SCD and PAR can then be displayed on the display 120 of the data processing device 100 to provide feedback to the surgeon or operator O.

[0108]

[0119] As described above, in method 600, before inserting the pedicle screw PS, the estimated deviation relative to the spinal column SC is correction This can be verified. For example, after performing method 600 once, the surgeon or operator O has some first estimated information about the spinal column SC, which has data on SCD and PAR that can be displayed in step D80, and in step D270 the spinal column Curvature and may even have visual feedback of position, displaying overlaid basic elements projected onto a live video feed. Thereafter, the surgeon or operator O can select and place a spinal cage, fixation device, or other type of intervertebral implant between two exemplary adjacent vertebrae V1, V2 of the spinal column SC, thereby also selecting the type and configuration of the intervertebral implant by, for example, selecting its thickness, or by selecting and adjusting a specific angle for spinal fusion, for example, the sagittal plane angle. The placement of the intervertebral implant may deviate from specific reorientation and displacement between the position and orientation of the two adjacent vertebrae V1, V2, thereby allowing the operator or surgeon O to perform method 600 again, based on the placement of the intervertebral implant, but without the placement of pedicle screws vs PS1, PS2, and without the attachment of fixation rods R, partially. correction It is possible to determine new values ​​for SCD and PAR of the spinal column SC.

[0109]

[0120] Based on the newly displayed parameter values ​​PAR and SCD, or both, step D80 of method 600 provides a table, curve, or other kind of visualization of PAR and SCD before and after insertion of the intervertebral implant, for the first position of the spine before any rod R is placed. correction Comparative data can be provided. This allows the operator or surgeon O to replace the intervertebral implant with a different intervertebral implant having a different configuration if, for example, the first intervertebral implant is displaced by Method 600 and the operator or surgeon O is not satisfied with the new PAR and / or SCD calculated by Method 600, with a different thickness or angle. In one variation, the operator or surgeon O can also change the parameterization of the intervertebral implant, if the intervertebral implant is of a configurable type, by changing the thickness or distance between the upper and lower bone engagement surfaces, or the angle between the upper and lower bone engagement surfaces. Method 600 can then repeat the step of verifying the results of the changes in the dimensions and properties of the intervertebral implant.

[0110]

[0121] Furthermore, Method 600 allows for the optional calculation of the virtual mounting points AP_V of a pair of pedicle screws PS1, PS2 potentially positioned to perforate holes DH1, DH2 in step C220. For example, in steps C30, D40, D50, U50, U55, D55, and C60 of Method 200, it is possible to determine the potential rod R by calculating one or more rod template RTs that can be proposed to the operator or surgeon O. This is done using one or more rod template RTs in step D50 and the creation of manufacturing data or information in steps D55 and C60 to determine the spine. correction We propose and virtually test different proposed rod templates RT for this purpose.

[0111]

[0122] Furthermore, embodiments of Method 600 and Method 200 can be combined with the steps of Method 500 to virtually test the effect of the rod template RT or rod data RD on the spine SC by calculation before the physical fixation rod R is actually attached to any pedicle screw PS. For example, after performing the step of selecting the rod template RT in step U50, or in step C30, where the rod template RT is calculated and proposed by Method 200, this data can be processed by step C120, in which a new attachment point AP is calculated based on the virtual attachment point AP_V resulting from step C220 of Method 600, thereby virtually testing the effect of the proposed virtual rod template RT or rod data RD. correction Information PDI_V regarding the position and orientation of the vertebrae of the spinal column SC is calculated. This aspect differs from method 500 because, since the pedicle screw PS has not yet been placed, the data regarding the mounting point AP is merely hypothetical and is referred to herein as AP_V. Steps C70, D70, C75, and D80 are also performed to measure the spine. Curvature Calculate data regarding SCD and spinal parameter PAR (step C75), spin Curvature Different data regarding SCD and spinal parameter PAR, for example, correction after and correction The previous data is displayed (step D80), and in step D70, vertebra V can be displayed as a basic element in their new virtual position and orientation, for example, in augmented reality projection onto a live video feed.

[0112]

[0123] Once satisfied with the virtual determination of the rod template RT, the physical fixation rod R can be manufactured, for example with the help of steps D55, C60, or method 500, and the surgeon or operator O can install the pedicle screw PS into the drilled hole DH of the vertebra V, and then the surgeon or operator O can also install the fixation rod R into the pedicle screw PS with the help of method 200, the spine correctionThis can be verified after the placement of the pedicle screw PS and the fixation rod R. Alternatively, the first surgeon or operator O may install the pedicle screw PS into the drilled hole DH of the vertebra V before actually fabricating the rod R, and then perform Method 200 to verify the mounting point AP precisely defined by the installation of the pedicle screw PS and determine an alternative or corrected rod template RT or rod data RD for the physical fixation rod R. At this stage, Method 200 may be performed to verify the vertebra that will be given by the physical fixation rod R during surgery. correction This can be verified.

[0113]

[0124] As described above, the methods 200, 500, and 600 described herein, and combinations of the steps and some parts thereof of these methods, can be implemented in different types of data processing devices 100, but can also be programmed as computer-readable code that can be stored in a non-temporary computer-readable medium, such as any type of data memory device or data storage device, and the computer-readable code is configured to perform methods 200, 500, 600 or its steps when executed on the data processing device 100 or on a data processor of another type of computer system, such as a distributed computer system with network and / or cloud access. For example, image visualization and imaging While tablet devices can be used, the actual computing steps are performed remotely on a server or personal computer that is operablely connected to the tablet via a network, as a variation of distributed computing.

[0114]

[0125] While the present invention is disclosed with reference to certain preferred embodiments, many modifications, alternatives, and changes are possible to the embodiments described without departing from the scope and area of ​​the invention as defined in the appended claims and their equivalents. Therefore, the present invention is not limited to the embodiments described and is intended to have the entire scope as defined by the following claims. The present invention includes embodiments described in the following clauses. [Clause 1] A method for assisting orthopedic surgery on the spine, performed using a data processing device including a display device and an imaging device, A step of using the imaging device to capture a sequence of images such that the field of view of the imaging device captures images of a plurality of screw extenders, wherein each screw extender holds a pedicle screw, and the plurality of screw extenders are arranged in a surgical incision in the body of a living person undergoing orthopedic surgery. The steps include providing a live video feed to the display device by either displaying at least some of the captured images or viewing them directly using a transparent display device, The steps include detecting the plurality of screw extenders using the data processing device based on the sequence of captured images, A first calculation step of calculating the orientation and position of the detected plurality of screw extenders, A second calculation step, based on the orientation and position of the first calculation step, calculates the three-dimensional (3D) position of the screw head of each pedicle screw, The steps include: using a graphical user interface on the display device to project and display the calculated 3D positions of the plurality of screw heads using graphic elements at positions corresponding to the positions of the screw heads projected onto the currently provided image of the live video feed; Methods that include... [Clause 2] A step of displaying a plurality of fixed rod templates as graphic elements on the graphical user interface of the display device, wherein the plurality of fixed rods have different shapes. The method described in Clause 1, further including the method described in Clause 1. [Clause 3] The steps include fitting a curve to points represented by the plurality of 3D positions of the screw head, A step of displaying a template of a fixed rod as a graphic element on the display device, wherein the fixed rod is shaped to at least partially match the adapted curve; The method described in Clause 1, further including the method described in Clause 1. [Clause 4] A step of graphically selecting one of the aforementioned multiple fixed rod templates, The steps include: positioning a graphic element representing one of the selected fixed rod templates at one of the three 3D positions of the screw head; The method described in Clause 2, further including the method described in Clause 2. [Clause 5] The steps include visually highlighting the detected screw extenders, A step of enabling the selection or deselection of at least one of the plurality of screw extenders using the graphical user interface of the data processing device. The method described in Clause 1, further including the method described in Clause 1. [Clause 6] The data processing device further comprises a distance measuring sensor, and the method, Steps to acquire distance information using the distance measuring sensor. It further includes, The method according to Clause 1, wherein the first calculation step is further based on the distance information. [Clause 7] The distance measuring sensor includes a direct time-of-flight (dToF) sensor, a LiDAR sensor, or a structured light sensor such as FaceID. The method described in Article 6. [Clause 8] A third calculating step in which posture data information of at least two vertebrae is calculated based on the orientation and position of at least one of the plurality of screw extenders detected and attached to the vertebrae in the first calculating step, A step of projecting and displaying a graphic element representing the vertebra on the display device using the graphical user interface, wherein the graphic element is displayed at a position corresponding to the position of the vertebra projected onto the currently provided image of the live video feed. The method described in Clause 1, further including the method described in Clause 1. [Clause 9] A fourth calculation step of calculating spinal curvature data of the spine based on the orientation and position of the detected plurality of screw extenders attached to the vertebrae by the first calculation step, A fifth calculation step of calculating the vertebral parameters of the spine based on the orientation and position of the detected plurality of screw extenders attached to the vertebrae as determined by the first calculation step, A step of projecting and displaying a graphic element representing the curvature of the spine on the display device using the graphical user interface, wherein the graphic element is displayed at a position corresponding to the position of the spine projected onto the currently provided image of the live video feed. The method described in Clause 8, further including the method described in Clause 8. [Clause 10] A data processing device comprising a display device, a data processor, and an imaging device, configured to support orthopedic surgery on the spine, wherein the data processor is The method involves commanding the imaging device to capture a sequence of images using the imaging device such that the imaging device's field of view captures images of multiple screw extenders, wherein each screw extender holds a pedicle screw, and the multiple screw extenders are arranged in the surgical incision of a living person undergoing orthopedic surgery. To display at least some of the captured images and instruct the display device to provide a live video feed, Based on the sequence of captured images, a detection algorithm for detecting the plurality of screw extenders is implemented using the data processing device, A first calculation is performed to determine the orientation and position of the detected plurality of screw extenders, Based on the orientation and position of the first calculation, a second calculation is performed to calculate the three-dimensional (3D) position of the screw head of each pedicle screw. On the display device, using a graphical user interface, the calculated 3D positions of each of the multiple screw heads are projected and displayed using graphic elements at positions corresponding to the positions of the screw heads projected onto the currently provided image of the live video feed. A data processing device configured to perform the following actions. [Clause 11] A non-temporary computer-readable medium on which computer instruction code is recorded, wherein the computer instruction code is configured to perform a method for computer-assisted orthopedic surgery on the spine when the computer instruction is executed on a data processing device operably connected to a display device and an imaging device, and the method is A step of using the imaging device to capture a sequence of images such that the field of view of the imaging device captures images of a plurality of screw extenders, wherein each screw extender holds a pedicle screw, and the plurality of screw extenders are arranged in a surgical incision in the body of a living person undergoing orthopedic surgery. The steps include providing a live video feed to the display device by either displaying at least some of the captured images or viewing them directly using a transparent device, The steps include detecting the plurality of screw extenders using the data processing device based on the sequence of captured images, A first calculation step of calculating the orientation and position of the detected plurality of screw extenders, A second calculation step, based on the orientation and position of the first calculation step, calculates the three-dimensional (3D) position of the screw head of each pedicle screw, The steps include: using a graphical user interface on the display device to project and display the calculated 3D positions of the plurality of screw heads using graphic elements at positions corresponding to the positions of the screw heads projected onto the currently provided image of the live video feed; Non-temporary computer-readable media, including [specific examples of such media]. [Article 12] A method for assisting orthopedic surgery, which involves determining spinal correction based on a fixation rod, using a data processing device, A step of scanning a fixed rod using an imaging device to acquire scanning data of the fixed rod, wherein the spinal correction rod is bent for spinal correction. A first calculation step of calculating the curvature data of the fixed rod based on the scanning data, A step of receiving data on the position of the attachment point of the fixation rod to the spine, wherein the position of the attachment point is determined based on position data of the screw heads of pedicle screws attached to at least two vertebrae of the spine, A second calculation step for calculating data for the corrected position of the mounting point, wherein the corrected position of the mounting point is based on a correction applied to the position of the mounting point when the fixing rod is attached to the mounting point of the straightened spine, by taking into account the curvature data of the fixing rod obtained in the first calculation step, A third calculation step of calculating the spinal parameters of the corrected spine based on the data of the corrected position of the attachment point of the corrected spine, The steps include displaying the spinal parameters of the corrected spine on a display device. Methods to assist in orthopedic surgery, including determining spinal correction. [Clause 13] A fourth calculation step: Based on the data of the corrected position of the attachment point of the corrected spine, calculate the posture data information of the vertebrae of the corrected spine. Methods to assist in orthopedic surgery to determine spinal correction as described in Clause 12, further including the following. [Clause 14] A step of displaying a basic graphic element representing the vertebra on the display device using the graphical user interface, wherein the basic graphic element is displayed at a position corresponding to the position of the vertebra. Methods to assist in orthopedic surgery to determine spinal correction as described in Clause 13, further including the above. [Article 15] A method for assisting orthopedic surgery on the spine, performed using a data processing device including a display device and an imaging device, A step of using the imaging device to acquire a sequence of images such that the field of view of the imaging device acquires an image of at least one of a plurality of pedicle markers, each positioned on a plurality of guidewires, or of a plurality of guidewires, wherein the plurality of pedicle markers or the plurality of guidewires are arranged at the surgical incision site of a living person undergoing orthopedic surgery. The steps include providing a live video feed to the display device by either displaying at least some of the captured images or viewing them directly using a transparent display device, The steps include detecting the plurality of pedicle markers or the plurality of guide wires using the data processing device based on the sequence of captured images, A first calculation step of calculating the orientation and position of the detected multiple pedicle markers or the detected multiple guide wires, A second calculation step calculates postural data information of at least two vertebrae based on the orientation and position of at least one of the detected multiple pedicle markers or multiple guide wires attached to the vertebrae in the first calculation step. Methods to support orthopedic surgery on the spine, including those mentioned above. [Clause 16] A third calculation step of calculating parameters characterizing the spine from the posture data information obtained in the second calculation step, The steps of displaying the parameters that characterize the spine on the display device and The method described in Clause 15, further including the method described in Clause 15. [Article 17] A fourth calculation step, based on the orientation and position of the first calculation step, calculates the virtual three-dimensional (3D) position of the screw head of each pedicle screw, The steps include: projecting and displaying the calculated virtual 3D positions of the plurality of screw heads onto the currently provided image of the live video feed using graphic elements, using a graphical user interface on the display device; The method described in Clause 15, further including the method described in Clause 15.

Claims

1. A method for assisting orthopedic surgery on the spine, wherein the method is carried out using a data processing device including a display device and an imaging device, The step of using the imaging device to capture a sequence of images such that the field of view of the imaging device captures images of a plurality of screw extenders arranged at the surgical incision site of a living person undergoing orthopedic surgery, wherein each screw extender holds a pedicle screw. The steps include providing a live video feed on the display device by displaying at least some of the captured images, or by direct viewing using a transparent display device, The steps include detecting the plurality of screw extenders using the data processing device based on the sequence of captured images, A first calculation step of calculating the orientation and position of the detected plurality of screw extenders, A second calculation step calculates the three-dimensional (3D) position of the screw head of each pedicle screw based on the orientation and position determined in the first calculation step, The steps include projecting and displaying the calculated 3D position of the screw head of each pedicle screw as a graphic element using a graphical user interface on the display device, at a position corresponding to the position of the screw head projected onto the currently provided image of the live video feed; Methods that include...

2. The step of displaying a plurality of fixed rod templates having different shapes as graphic elements on the graphical user interface of the display device. The method according to claim 1, further comprising:

3. A step of fitting a curve to a point represented by the calculated 3D position of each of the plurality of screw heads, The steps include displaying a fixed rod template, shaped to at least partially match the adapted curve, as a graphic element on the display device. The method according to claim 1, further comprising:

4. A step of graphically selecting one of the aforementioned multiple fixed rod templates, The steps include: positioning a graphic element representing the selected fixed rod template at one of the three-dimensional positions of the screw head; The method according to claim 2, further comprising:

5. The steps include visually highlighting the detected screw extenders, A step of enabling the selection or deselection of at least one of the plurality of screw extenders using the graphical user interface of the data processing device. The method according to claim 1, further comprising:

6. The data processing device further comprises a distance measuring sensor, and the method, Steps to acquire distance information using the distance measuring sensor. It further includes, The method according to claim 1, wherein the first calculation step is further based on the distance information.

7. The distance measuring sensor includes a direct time-of-flight (dToF) sensor, a LiDAR sensor, or a structured light sensor such as FaceID. The method according to claim 6.

8. A third calculation step of calculating posture data information for at least two vertebrae based on the orientation and position of at least one of the plurality of detected screw extenders attached to the vertebrae, The steps include projecting and displaying the graphic basic elements representing the vertebrae at positions corresponding to the positions of the vertebrae projected onto the currently provided image of the live video feed, using the graphical user interface on the display device; The method according to claim 1, further comprising:

9. A fourth calculation step of calculating spinal curvature data of the spine based on the orientation and position of the detected plurality of screw extenders attached to the vertebrae, as determined by the first calculation step, A fifth calculation step of calculating the vertebral parameters of the spine based on the orientation and position of the detected plurality of screw extenders attached to the vertebrae, as determined by the first calculation step, The steps include projecting and displaying a basic graphic element representing the curvature of the spine at a position corresponding to the position of the spine projected onto the currently provided image of the live video feed, using the graphical user interface on the display device; The method according to claim 8, further comprising:

10. The method according to claim 1, wherein each screw extender includes at least one longitudinally shaped slot along the side surface of the screw extender for guiding a vertebral rod to the pedicle screw held by the screw extender.

11. A data processing device configured to support orthopedic surgery on the spine, wherein the data processing device comprises a display device, a data processor, and an imaging device, and the data processor is The imaging device is commanded to capture a sequence of images so that its field of view captures images of multiple screw extenders arranged at the surgical incision site of a living person undergoing orthopedic surgery, with each screw extender holding a pedicle screw. The system commands the display device to display at least some of the captured images and provide a live video feed. Based on the sequence of captured images, a detection algorithm for detecting the plurality of screw extenders is implemented using the data processing device. A first calculation is performed to determine the orientation and position of the detected plurality of screw extenders. Based on the orientation and position determined by the first calculation, a second calculation is performed to determine the three-dimensional (3D) position of the screw head of each pedicle screw. The calculated 3D position of each screw head of each pedicle screw is projected and displayed as a graphic element on the display device using a graphical user interface, at a position corresponding to the position of the screw head projected onto the currently provided image of the live video feed. A data processing device configured as follows.

12. The data processing apparatus according to claim 11, wherein each screw extender (SE) includes at least one longitudinally shaped slot along the side surface of the screw extender for guiding a spinal rod to the pedicle screw (PS) held by the screw extender (SE).

13. A non-temporary computer-readable medium on which computer instruction code is recorded, wherein the computer instruction code is configured to perform a method for computer-assisted orthopedic surgery on the spine when the computer instruction code is executed on a data processing device operably connected to a display device and an imaging device, and the method is The step of using the imaging device to capture a sequence of images such that the field of view of the imaging device captures images of a plurality of screw extenders arranged at the surgical incision site of a living person undergoing orthopedic surgery, wherein each screw extender holds a pedicle screw. The steps include providing a live video feed on the display device by displaying at least some of the captured images, or by direct viewing using a transparent display device, The steps include detecting the plurality of screw extenders using the data processing device based on the sequence of captured images, A first calculation step of calculating the orientation and position of the detected plurality of screw extenders, A second calculation step calculates the three-dimensional (3D) position of the screw head of each pedicle screw based on the orientation and position determined in the first calculation step, The steps include projecting and displaying the calculated 3D position of the screw head of each pedicle screw as a graphic element using a graphical user interface on the display device, at a position corresponding to the position of the screw head projected onto the currently provided image of the live video feed; Non-temporary computer-readable media, including [specific examples of such media].

Images