Implant Placement Guides and Methods

Abstract

Facilitating proper object placement during surgery includes tracking the position and location of the object as it is moved in the surgical environment to position and position the object in a desired position / position of the object relative to the patient's anatomy, displaying augmented reality (AR) elements on the display device such that the AR elements are overlaid on a view of the surgical environment to assist in proper positioning and placement of the object relative to the patient's anatomy, and visually indicating on the display device that the object has been moved to the desired position and position.

Description

[Background technology]

[0001] The goal of reconstructive surgery is to precisely place the implant into the patient's severed bone for the implant to bond with. Sophisticated instruments, including navigation and robotics, are used to properly cut the bone. Summary of the Invention

[0002] In a first embodiment, the shortcomings of the prior art are overcome and additional advantages are provided by providing a computer-implemented method. The method includes determining a desired position and placement of a physical object to be positioned and placed relative to a patient's anatomy, presenting an augmented reality (AR) element on a display device overlaid on a view of a surgical environment, tracking the position and location of the physical object as it is moved within the surgical environment to determine the position and location of the physical object relative to the patient's anatomy, and visually indicating on the display device that the physical object has been moved to the desired position and location relative to the patient's anatomy based on the position and placement of the physical object matching the desired position and location of the physical object. The desired position and placement is defined by a surgical plan for the patient. The surgical environment includes the patient's anatomy. Placing the AR element at the desired position and placement of the physical object to present the position and placement facilitates proper placement of the physical object relative to the patient's anatomy.

[0003] In an embodiment, the method optionally further includes obtaining a video stream depicting a view of the surgical environment including a depiction of the patient's anatomy, and presenting the video stream on a display device, wherein presenting the AR element on the display device augments the video stream with the AR element in a desired position and arrangement for positioning a physical object relative to the patient's anatomy as depicted in the video stream.

[0004] In an embodiment, the method optionally further includes zooming-in to an area of ​​the surgical environment in which the AR element is presented as part of presenting the video stream and as the physical object is positioned relative to the patient's anatomy.

[0005] In an embodiment, the display optionally includes a transparent display through which the user views the surgical environment, and the AR elements are presented on the transparent display device to augment the user's view of the surgical environment through the transparent display.

[0006] In an embodiment, the transparent display is optionally provided as part of a smart wearable glasses device.

[0007] In an embodiment, the visually indicating optionally includes modifying a visual presentation of the AR element on a display device.

[0008] In embodiments, the AR element is optionally initially at least partially a first color or pattern, and the visually indicating includes changing the first color or pattern to another color or pattern.

[0009] In an embodiment, tracking the position and location of the physical object is optionally performed using at least one of: one or more markers on the physical object, a shape matching algorithm that identifies the position and location of the physical object within the view based on a known shape of the physical object, or tracking the position of at least one other physical object that has a fixed, known location relative to the physical object.

[0010] In an embodiment, the method can optionally further include tracking a location and position of the patient's anatomy in the surgical environment and, based on the patient's anatomy being repositioned, presenting the AR element and repositioning the AR element to maintain the AR element at a desired location and position of the physical object to be placed relative to the patient's anatomy.

[0011] In an embodiment, the surgical plan optionally includes one or more desired cuts made to the patient's anatomy, and the method further comprises determining that the one or more desired cuts have been made to the patient's anatomy, determining a desired location and position of the physical object, and presenting the AR element based on the one or more desired cuts made.

[0012] In an embodiment, the AR element is optionally a first AR element, and the method further includes presenting on the display device a second AR element superimposed on the physical object in a view of the surgical environment.

[0013] In an embodiment, the physical object is optionally an implant.

[0014] In an embodiment, the AR element is optionally a digital model of a physical object.

[0015] In an alternative embodiment, a computer system is provided that includes a memory and a processor in communication with the memory. The computer system is configured to perform any of the methods described above. In another alternative embodiment, a computer program product is provided that includes a computer readable storage medium that stores instructions readable by a processing circuit and executed by the processing circuit to perform any of the methods described above.

[0016] Additionally or alternatively, in a second embodiment, a computer-implemented method is provided to overcome the deficiencies of the prior art and provide additional advantages. The method includes tracking a position and location of a physical object in a surgical environment as the physical object is moved within the surgical environment, presenting on a display device an augmented reality (AR) element overlaying the physical object in a view of the surgical environment, and visually indicating on the display device that the physical object has been moved to a desired position and location relative to a patient's anatomy based on the position and location of the physical object being consistent with the desired position and location of the physical object. The view shows the physical object and the patient's anatomy within which the physical object is positioned and positioned according to the desired position and location of the physical object as dictated by the patient's surgical plan. The presentation maintains an AR element overlaying the physical object in a view as the physical object is moved within the surgical environment based on the tracking.

[0017] In an embodiment, the method optionally further includes obtaining a video stream depicting a view of the surgical environment including a depiction of the patient's anatomical structure and presenting the video stream on a display device, where presenting the AR element on the display device includes an AR element that augments the video stream and is displayed overlaid on a physical object in the view of the surgical environment.

[0018] In an embodiment, the method optionally further includes magnifying a region of the surgical environment to display a desired location and position of the physical object as part of presenting the video stream and as the physical object is positioned relative to the patient's anatomy.

[0019] In an embodiment, the display device optionally comprises a transparent display through which the user views the surgical environment, and the AR elements are presented on the transparent display to augment the user's view of the surgical environment through the transparent display.

[0020] In an embodiment, the transparent display is optionally provided as part of a smart wearable glasses device.

[0021] In an embodiment, the visually indicating optionally includes modifying a visual presentation of the AR element on a display device.

[0022] In embodiments, the AR element is optionally initially at least partially a first color or pattern, and the visually indicating includes changing the first color or pattern to another color or pattern.

[0023] In an embodiment, tracking the position and location of the physical object is optionally performed using at least one of: one or more markers on the physical object, a shape matching algorithm that identifies the position and location of the physical object within the view based on a known shape of the physical object, or tracking the position of at least one other physical object that has a fixed, known location relative to the physical object.

[0024] In an embodiment, the AR element is optionally a first AR element, and the method further includes presenting on the display device a second AR element that is positioned or disposed relative to the patient's anatomy on the physical object at a desired location or arrangement.

[0025] In an embodiment, the AR element is optionally a digital model of a physical object.

[0026] In an embodiment, the physical object is optionally an implant.

[0027] In an alternative embodiment, a computer system is provided that includes a memory and a processor in communication with the memory. The computer system is configured to perform any of the methods described above. In another alternative embodiment, a computer program product is provided that includes a computer readable storage medium that stores instructions readable by a processing circuit and executed by the processing circuit to perform any of the methods described above.

[0028] Additional features and advantages are realized through the concepts described herein.

[0029] The aspects described herein are particularly pointed out and distinctly claimed, and the objects, features, and advantages of the disclosure will be apparent from the detailed description herein when taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0030] [Figure 1A] 1A-1H and 2 illustrate one example of an environment for using or incorporating embodiments described herein. [Figure 1B] 1A-1H and 2 illustrate one example of an environment for using or incorporating embodiments described herein. [Figure 1C] 1A-1H and 2 illustrate one example of an environment for using or incorporating embodiments described herein. [Figure 1D] 1A-1H and 2 illustrate one example of an environment for using or incorporating embodiments described herein. [Figure 1E] 1A-1H and 2 illustrate one example of an environment for using or incorporating embodiments described herein. [Figure 1F] 1A-1H and 2 illustrate one example of an environment for using or incorporating embodiments described herein. [Figure 1G] 1A-1H and 2 illustrate one example of an environment for using or incorporating embodiments described herein. [Figure 1H] 1A-1H and 2 illustrate one example of an environment for using or incorporating embodiments described herein. [Diagram 2] 1A-1H and 2 illustrate one example of an environment for using or incorporating embodiments described herein.

[0031] [Diagram 3] FIG. 3 illustrates an example of a computer system and related devices that incorporate and / or use aspects described herein.

[0032] [Figure 4] FIG. 4 shows an example of a smart eyewear device.

[0033] [Diagram 5] 5 and 6 illustrate one example of a process for tracking, locating, and / or positioning an object relative to a patient's anatomy during a surgical procedure according to embodiments described herein. [Figure 6] 5 and 6 illustrate one example of a process for tracking, locating, and / or positioning an object relative to a patient's anatomy during a surgical procedure according to embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Despite efforts to make precise bone cuts, placement of an implant in the bone after the surgeon and / or robot make the cuts is still highly subjective and dependent on the surgeon's skill. As a non-limiting example, a surgeon may use a guidance application associated with a navigated surgical robot to virtually place an implant based on a defined surgical plan and perform the associated cuts. However, once the surgeon has finished making the cuts, there is no way to guarantee the accuracy of the placement of the implant relative to the planned placement, and the process of placing the implant in the bone is dependent on the surgeon's skill and is imprecise. One problem is that surgeons currently have no way of knowing whether or not they have placed the implant in the intended planned location as dictated by the surgical plan. Although great efforts are made to make precise cuts, currently surgeons manually place implants in those cuts with little or no guidance.

[0035] The embodiments described herein propose novel methods to assist surgeons and / or robotic instruments during surgical procedures to achieve more accurate implant placement, e.g., a placement that is closer to and more consistent with the planned placement of the implant.

[0036] In a first example, the method proposes the use of an augmented overlay (e.g., an Augmented Reality (AR) overlay) to display the location and position of the implant to be placed on / as part of an RGB stream (e.g., a video stream) of the surgical background. Such an AR overlay can present the rendered implant volume with a variable and selectable level of opacity (e.g., solid or semi-transparent). The implant representation can be projected onto the bone at a position dictated by the planned virtual placement. This can be similar to a registration overlay where the surgeon aligns a bone model overlay to the actual bone, but the surgeon, with the assistance of software, may need to align the implant to be placed to a target displayed as an AR overlay on top of the cut bone.

[0037] Referring to FIG. 1A, 100 illustrates an example of a stream / view of a surgical environment in which surgery is in progress. The view is provided by a camera feed / stream, and a computer system can overlay / superimpose AR elements on top of the view and display the view including the AR elements (i.e., an "augmented view") on a display screen. Additionally or alternatively, a user (such as a surgeon) can wear smart glasses or other wearable device and see the environment through a transparent display (such as a transparent lens with an active display embedded), and the AR elements can be displayed on the transparent display to provide the user with an augmented view.

[0038] The view 100 shows a fixed tracking array 102 and a patient's upper leg 104 and lower leg 106, respectively. A registration probe 110, which may be used to identify the exact location of the subject in the registration process, is positioned with a probe tip 112 at the end of the femur 106 in this example. FIG. 1A also shows a guide line 114 (displayed in a predefined color, such as white) extending from the probe 110 (in this example, from the probe tip 112) toward the end of the femur as a central axis line representing the axis of the probe 110. This guide line 114 is presented to the surgeon as an augmented reality (AR) element to help the surgeon properly orient the probe, for example, so that the line extends as close as possible to the central axis of the patient's bone, pointing to the center of the patient's hip joint. This can be utilized to facilitate coaxial alignment between the probe and the actual patient's bone.

[0039] 1A also shows the implant model 120 presented in the view 100 of the environment, in particular as an AR element positioned in a position dictated by the planned virtual placement of the implant, e.g., adjacent to / on the patient's bone. If the patient's anatomy is repositioned, the implant model 120 can be repositioned because the position in space of fixed objects such as the patient's bone is known based on the position of the tracking array 102.

[0040] FIG. 1B shows an example of another view 140 visible to the surgeon. This view can be presented on a display (like display 141 in FIG. 1C). The position of the bone 142 can be tracked using rigid markers / arrays (not shown) attached to the bone so that a computer system receiving / processing the tracking data knows exactly where the bone is in space. Additionally, the computer system can know what cuts are made to the bone and where exactly the cuts are, such as if the cuts are performed by a robot. Referring to FIG. 1D, an AR overlay 150 is added to the view of the display showing the exact location to position the implant and the surgeon can identify this location by looking at the display. Using an ultra-high resolution camera (e.g., 8K resolution or higher), the surgeon can zoom in on the view as needed to more clearly and precisely identify the intended location of the implant, as shown in FIG. 1E. While zooming in on the AR overlay 150, the surgeon can position the physical implant (which may be visible in the view) to match the location of the implant to the location of the AR overlay (which is considered the exact and appropriate location of the implant).

[0041] In additional or alternative embodiments, the location of the physical implant in space is tracked. An AR model of the implant is represented as an overlay of the implant in the view and is displayed in a first color and / or pattern in the view until the implant is in the proper position adjacent / on the patient's bone, at which point the AR overlay of the implant changes to another color and / or pattern to indicate proper placement. More generally, a visual indicator associated with the AR model of the implant can indicate that the actual physical implant has been moved to the proper position relative to the patient's anatomy, compared to when the implant is not in the proper position. The location and position of the implant can be indicated by the AR overlay, and the location and position of the implant can be determined using any desired approach. One approach can be to affix a removable marker to the implant, making it trackable. Additionally or alternatively, a shape matching algorithm can be used to identify and track the implant in the scene (advantages of an RGB camera).

[0042] Regarding the latter point, referring to Figs. 1F-1H, an example of tracking based on object shape is shown. The environment 158 ​​includes a patient's anatomy 165 (e.g., bone) having an object (e.g., implant) 160. Here, the object 160 is fixed to the bone 165, but tracking of the object can be performed regardless of whether the object 160 is actually fixed to the patient's anatomy. Markerless tracking can be provided, in which case the position of the object 160 in space can be tracked based on the shape of the object 160 (e.g., based on a stream from an RGB camera). The object 160 can also be depicted as an AR overlay 170 (drawn in pointillism in this example, but could also be filled in with color to complete its shape and surroundings) on a display showing a virtual representation of the environment, as shown in view 159.

[0043] The exact location of the object in the view can be known based at least in part on the known location of the fixed tracking array 103 and / or by tracking the object using a shape matching algorithm that, in the case of object 160, is expected to be unique and distinguishable from other shapes in the environment. Vertical lines 166 and 167 (which may be displayed in a first color, such as yellow) in FIG. 1F are conveniently displayed to indicate a first location of object 160 in environment 158 ​​and a corresponding first location of AR model 170 in view 159, respectively. FIG. 1G shows the view after object 160 has been moved to a different position (in this example because bone 165 to which object 160 is fixed has moved). Vertical lines 168 and 169 in FIG. 1G are conveniently displayed to indicate a second location of object 160 in environment 158 ​​and a corresponding second location of AR model 170 in view 159, respectively. Vertical lines 168 and 169 after the movement can be displayed in a second color (e.g., green) different from the first color. Thus, the AR model 170 can be used as a virtual overlay to reflect the updated position and movement of the object 160 in real space. In a particular embodiment of implant placement, the implant is placed on the bone and tracked based on its shape and positioned while being displayed as an AR element on the display (e.g., dot / stipple in FIGS. 1F-1H). When the implant is in the desired position (e.g., as determined by a program / calculation based on the exact virtual placement of the implant in the surgical plan), the color or other visual indication of the AR overlay, or a color or other visual indication associated with the AR overlay, can be changed from one color (or indication) to another to indicate to the surgeon that the implant is properly placed and positioned.

[0044] FIG. 2 shows an example in which the AR overlay 220 changes to a different color (such as bright yellow) to indicate to the viewer that the surgeon has placed the implant (here, covered by the AR overlay 220 and not visible) in the proper position.

[0045] In yet another embodiment, the instrument holding the implant can be tracked, for example, reflective markers can be attached to the instrument to allow IR tracking of the instrument and therefore also of the implant, since once the implant is moved into position, its position relative to the instrument remains fixed.

[0046] Some aspects provide increased accuracy and improved user experience. Surgeons no longer need to rely solely on their own judgment to place implants in cut bones, but can accurately place implants based on a surgical plan. This can be useful in either navigated or robotic surgeries where implants must be placed based on a surgical plan. While the examples described herein relate to implant placement on / against bone, the aspects described herein apply more generally to the placement of objects (e.g., implants) on / against a patient's anatomy.

[0047] The processes described herein may be performed by one or more computer systems, such as, for example, one or more systems that are or are in communication with a camera system, a tracking system, and / or an AR system, either singly or collectively. FIG. 3 illustrates an example of such a computer system and related devices incorporating and / or using aspects described herein. The computer system may also be referred to herein as a data processing device / system, a computing device / system / node, or simply a computer. The computer system may be based on one or more of a variety of system architectures and / or instruction set architectures, such as those offered by, for example, Intel Corporation (Santa Clara, Calif., USA) or ARM Holdings plc (Cambridge, England, UK).

[0048] FIG. 3 illustrates a computer system 300 in communication with an external device 312. The computer system 300 includes one or more processors 302 (e.g., a central processing unit (CPU)). The processor may include functional components used to execute instructions (such as functional components to fetch program instructions from locations such as a cache or main memory, functional components to decode program instructions, functional components to execute program instructions, functional components to access memory for instruction execution, functional components to write results of executed instructions, etc.). The processor 302 may also include registers used by one or more of the functional components. The computer system 300 also includes memory 304, input / output (I / O) devices 308, and I / O interfaces 310, which may be connected to the processor 302 and to each other via one or more buses and / or other connections. The bus connections may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a high-speed graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and without limitation, such architectures include Industry Standard Architecture (ISA), Micro Channel Architecture (MCA), Enhanced ISA (EISA), Video Electronics Standards Association (VESA) Local Bus, and Peripheral Component Interconnect (PCI).

[0049] Memory 304 may include, for example, a main or system memory (e.g., random access memory), storage devices such as a hard drive, flash media, or optical media used to execute program instructions, and / or cache memory, etc. Memory 304 may include, for example, a cache, such as a shared cache coupled to a local cache (e.g., an L1 cache, an L2 cache, etc.) of processor 302. Additionally, memory 304 may be or include at least one computer program product that includes a set (e.g., at least one) of program modules, instructions, code, etc. configured to perform functions of embodiments described herein when executed by one or more processors.

[0050] The memory 304 can store an operating system 305 and other computer programs 306, such as one or more computer programs / applications that execute to perform aspects described herein. In particular, the programs / applications can include computer readable program instructions that can be configured to perform the functions of embodiments of the aspects described herein.

[0051] Examples of I / O devices 308 include, but are not limited to, microphones, speakers, global positioning system (GPS) devices, RGB and / or IR cameras, lights, accelerometers, gyroscopes, magnetometers, sensor devices configured to sense light, proximity, heart rate, body and / or ambient temperature, blood pressure, and / or skin resistance, registration probes, and activity monitors. As illustrated, the I / O devices are integrated into the computer system, although in some embodiments the I / O devices may be considered external devices (312) connected to the computer system via one or more I / O interfaces 310.

[0052] The computer system 300 may communicate with one or more external devices 312 through one or more I / O interfaces 310. Examples of external devices include keyboards, pointing devices, displays, and / or other devices that allow a user to interact with the computer system 300. Examples of other external devices include devices that allow the computer system 300 to communicate with one or more other computing systems or peripheral devices, such as printers. Network interfaces / adapters are examples of I / O interfaces that allow the computer system 300 to communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), enabling communication with other computing devices or systems, storage devices, and the like. Ethernet-based (e.g., Wi-Fi) interfaces and Bluetooth® adapters are just a few examples of types of network adapters currently used in computer systems (BLUETOOTH is a registered trademark of Bluetooth SIG, Inc., Kirkland, WA, USA).

[0053] Communications between the I / O interface 310 and the external device 312 occur via a wired and / or wireless communication link 311, such as an Ethernet-based wired or wireless connection. Examples of wireless connections include cellular, Wi-Fi, Bluetooth, proximity-based, short-range, or other types of wireless connections. More generally, the communications link 311 can be any suitable wireless and / or wired communications link that communicates data.

[0054] A particular external device 312 may include one or more data storage devices capable of storing one or more programs, one or more computer readable program instructions, and / or data, etc. Computer system 300 may include and / or be connected to and communicate with removable / non-removable, volatile / non-volatile computer system storage media (e.g., as a computer system external device). For example, it may include and / or be coupled to a non-removable non-volatile magnetic medium (commonly referred to as a "hard drive"), a magnetic disk drive that reads and writes to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and / or an optical disk drive that reads or writes to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media).

[0055] Computer system 300 is operable with numerous other general purpose or special purpose computing system environments or configurations. Computer system 300 can take a variety of forms, familiar examples of which include, but are not limited to, personal computer (PC) systems, server computer systems such as messaging servers, mobile devices / computers such as thin clients, thick clients, workstations, laptops, handheld devices, smartphones, tablets, wearable devices, multiprocessor systems, microprocessor-based systems, telephony devices, network appliances (such as edge appliances), virtualization devices, storage controllers, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0056] FIG. 4 illustrates another example of a computer system incorporating and using aspects described herein. FIG. 4 illustrates an example of an eyewear-based wearable device, such as, for example, a wearable smart glasses device that facilitates presenting AR elements to a wearer of the device. The device 400 can include many of the same types of components that may be included in the computer system 300 described above. In the example of FIG. 4, the device 400 is configured to be worn on the head of a device user. The device includes a display 402 that is positioned within the line of sight of the user's peripheral vision when the device is in an operational position on the user's head. A suitable display can utilize, for example, LCD, CRT, or OLED display technology. The lens 414 can optionally include an active translucent display that can display images and other content on the inner and / or outer surfaces of the lens. This allows this content to be projected directly into the user's line of sight and overlay at least a portion of the user's view of the environment through the lens. In certain embodiments described herein, the content displayed on the lens display is an AR element that overlays a stream from a camera depicting a surgical environment / operating room.

[0057] The device 400 also includes a touch input 404 that allows a user to input touch gestures to control device functions. Such gestures can be interpreted as commands, for example a command to take a picture or a command to launch a particular service. The device 400 also includes buttons 406 that control device functions. Example functions include locking the device, shutting it down, or switching it into a standby or sleep mode.

[0058] Various other input devices are provided that can be used to capture images and video, such as a camera 408. The camera can be used by the device to obtain images / video of the wearer's surroundings, for example to take images / video of a scene. Additionally, the camera can be used to track the direction of the user's gaze to see where the user is looking, and to track other eye activity of the user, such as blinking and movement.

[0059] One or more microphones, proximity sensors, light sensors, accelerometers, speakers, GPS devices, and / or other input devices (not labeled) may further be provided, for example, within the housing 410. The housing 410 may also include other electronic components, such as electronic circuitry including a processor, memory, and / or communications devices (e.g., cellular, short-range wireless (e.g., Bluetooth), or Wi-Fi circuitry for connecting to remote devices). The housing 410 may further include a power source, such as a battery, for powering the components of the device 400. Additionally or alternatively, such circuitry or battery may be included in the extended end 412, which may be extended to accommodate such components. The extended end 412, or other portions of the device 400, may also include physical ports (not shown) used to connect the device 400 to a power source (to recharge the battery) and / or other external devices, such as a computer. Such physical ports may be of any standardized or proprietary type, such as a Universal Serial Bus (USB).

[0060] Aspects of the invention may be systems, methods, and / or computer program products, any of which may be configured to perform or facilitate the aspects described herein. Also provided are computer systems configured to perform these and other methods, and computer program products including a computer-readable storage medium storing executable instructions for performing these and other methods.

[0061] In some embodiments, aspects of the invention may take the form of a computer program product and may be embodied as a computer readable medium. A computer readable medium is a tangible storage device / medium on which computer readable program code / instructions are stored. Examples of computer readable media include, but are not limited to, electronic, magnetic, optical, or semiconductor storage devices or systems, or any combination of the foregoing. Exemplary embodiments of computer readable media include hard drives or other mass storage devices, electrical connections with wires, random access memory (RAM), read only memory (ROM), erasable programmable read only memory such as EPROM or flash memory, fiber optics, portable computer disks / diskettes such as compact disk read only memory (CD-ROM) or digital versatile disk (DVD), optical storage devices, magnetic storage devices, or any combination thereof. A computer readable medium is readable by a processor, processing unit, or the like, which retrieves and executes data (e.g., instructions) from the medium. In certain examples, a computer program product is or includes one or more computer readable media that contain / store computer readable program code that provides and facilitates one or more aspects described herein.

[0062] As discussed above, program instructions contained or stored on a computer-readable medium may be retrieved and executed by any of a variety of suitable components, such as a processor of a computer system, to cause the computer system to operate and function in a particular manner. Such program instructions that perform operations to implement, achieve, or facilitate aspects described herein may be written in any programming language or compiled from code written in any programming language. In some embodiments, such programming languages ​​include object-oriented and / or procedural programming languages, such as C, C++, C#, Java, etc.

[0063] The program code may include one or more program instructions acquired for execution by one or more processors. The computer program instructions may be provided to one or more processors of, for example, one or more computer systems to generate a machine that, when executed by the one or more processors, performs, accomplishes, or facilitates aspects of the invention, such as the operations or functions described in the flowcharts and / or block diagrams set forth herein. Thus, each block, or combination of blocks, of the flowchart illustrations and / or block diagrams illustrated and described herein may, in some embodiments, be implemented by computer program instructions.

[0064] 5 and 6 illustrate an exemplary process for tracking, locating, and / or positioning an object relative to a patient's anatomy during a surgical procedure in accordance with embodiments described herein. The process may be performed by a computer system executing software that performs embodiments discussed herein, such as a computer system as described above with reference to FIGS. 3 and 4.

[0065] 5, a process is presented that focuses on maintaining an augmented reality (AR) at a desired location of an object (e.g., an implant) and visually indicating when the object is correctly placed. The process includes determining (502) a desired location and position of a physical object to be positioned and placed relative to the patient's anatomy. In an embodiment, the desired location and position is determined by the patient's surgical plan. The physical object may be an implant, as an example.

[0066] The process continues by presenting (504) on the display device an augmented reality (AR) element overlaid on a view of the surgical environment. The surgical environment includes the patient's anatomy. The AR element is presented such that the AR element is positioned and positioned at a desired location and position of the physical object to allow proper positioning of the physical object relative to the patient's anatomy. In an embodiment, the AR element is a digital model of the physical object.

[0067] In an embodiment, the process obtains a video stream depicting a view of a surgical environment including a depiction of a patient's anatomy and displays the video stream on a display device. The AR element can be displayed on the display device to augment the video stream to include the AR element in a desired location and position that places a physical object relative to the patient's anatomy depicted in the video stream. In one example, as part of the display of the video stream and once the physical object has been placed relative to the patient's anatomy, the process can zoom-in to an area of ​​the surgical environment where the AR element is presented.

[0068] Additionally or alternatively, in an embodiment, the display device may include a transparent display through which the user sees a view of the surgical environment. The AR elements may be displayed on the transparent display to augment the user's view of the surgical environment through the transparent display. In an embodiment, the transparent display is provided as part of a smart wearable glasses device.

[0069] Continuing in Figure 5, the process tracks the position and location of the physical object as it is moved within the surgical environment (e.g., moved to position and position the physical object relative to the patient's anatomy) (506). In an embodiment, tracking the position and location of the physical object is performed using at least one of one or more markers on the physical object, a shape matching algorithm that identifies the position and location of the physical object within the view based on a known shape of the physical object, or tracking the position of at least one other physical object that has a fixed, known position relative to the physical object.

[0070] Optionally, the process includes tracking the position and placement of the patient's anatomy in the surgical environment 508. In these embodiments, the AR element is repositioned by presenting the AR element based on the patient's anatomy being repositioned to maintain the AR element at a desired position and placement of the physical object to be positioned relative to the patient's anatomy.

[0071] The tracking 506 (and optionally 508) can be performed continuously / substantially continuously, with checking / determining whether the position and location of the physical object matches the desired position and placement of the physical object. Based on the position and location of the physical object matching the desired position and placement of the physical object, the process can visually indicate (510) on a display device that the physical object has been moved to the desired position and placement relative to the patient's anatomy. In an embodiment, the visually indicating includes changing a visual presentation of an AR element on the display device. For example, the AR element is initially at least partially a first color or pattern, and the visually indicating can include changing the first color or pattern to another color or pattern.

[0072] In an embodiment, the surgical plan includes one or more desired incisions made to the patient's anatomy, and the process determines 502 and presents 504 that the one or more desired incisions have been made to the patient's anatomy, i.e., determines a desired location and position of a physical object and presents an AR element based on the one or more desired incisions made.

[0073] In some embodiments, the AR element is a first AR element, and the process may further include displaying on the display device a second AR element that overlays the physical object in a view of the surgical environment.

[0074] 6, a process is presented that focuses on an AR element tracking / overlaying an object (e.g., an implant) as it is moved into position and visually indicating correct placement as it is moved into position. The process includes tracking (602) a position and location of a physical object in the surgical environment as the physical object is moved within the surgical environment. In an embodiment, the physical object is an implant. In an embodiment, tracking the position and location of the physical object is performed using at least one of one or more markers on the physical object, a shape matching algorithm that identifies the position and location of the physical object in the view based on a known shape of the physical object, or tracking the position of at least one other physical object that has a fixed, known position relative to the physical object.

[0075] Continuing with FIG. 6, the process presents, on the display device, an augmented reality (AR) element that overlays the physical object with a view of the surgical environment (604). The view shows the physical object and the patient's anatomical structures where the physical object is to be positioned and placed according to the desired location and placement of the physical object as dictated by the patient's surgical plan. The presentation can be based on tracking to maintain the AR element overlaying the physical object in the view as the physical object is moved within the surgical environment. In an embodiment, the AR element is a digital model of the physical object.

[0076] In an embodiment, the process obtains a video stream depicting a view of the surgical environment including a depiction of the patient's anatomy and presents the video stream on a display device. The presentation of the AR element on the display device can include an AR element that augments the video stream to overlay a physical object with the view of the surgical environment. In one example, as part of the presentation of the video stream, once the physical object is positioned relative to the patient's anatomy, the process can zoom in on an area of ​​the surgical environment that shows the desired location and placement of the physical object.

[0077] Additionally or alternatively, in an embodiment, the display device may include a transparent display through which the user sees a view of the surgical environment. The AR elements may be displayed on the transparent display to augment the user's view of the surgical environment through the transparent display. In an embodiment, the transparent display is provided as part of a smart wearable glasses device.

[0078] The tracking 602 and presenting 604 may be performed continuously / substantially continuously with checking / determining whether the position and location of the physical object matches the desired position and placement of the physical object. Based on the position and location of the physical object matching the desired position and placement of the physical object, the process may visually indicate on a display device that the physical object has been moved to the desired position and placement relative to the patient's anatomy (606). In an embodiment, the visually indicating includes changing a visual presentation of an AR element on the display device. For example, the AR element is initially at least partially a first color or pattern, and the visually indicating may include changing the first color or pattern to another color or pattern.

[0079] In some embodiments, the AR element is a first AR element, and the process can further include presenting on a display device a second AR element positioned and arranged relative to the patient's anatomy to indicate the desired location and arrangement of the physical object.

[0080] Although various embodiments have been described above, these are merely examples.

[0081] As described herein, a small sample of embodiments of the present disclosure is provided.

[0082] A1. A computer-implemented method comprising: determining a desired position and placement of a physical object to be positioned and placed relative to a patient's anatomy; presenting, on a display device, an augmented reality (AR) element overlaid on a view of a surgical environment; tracking the position and location of the physical object as the physical object is moved within the surgical environment to determine and place the position and location of the physical object relative to the patient's anatomy; and visually indicating on the display device that the physical object has been moved to the desired position and location relative to the patient's anatomy based on the position and placement of the physical object matching the desired position and location of the physical object. The desired position and placement is defined by a surgical plan for the patient. The surgical environment includes the patient's anatomy. Placing the AR element at the desired position and placement of the physical object to present the position and placement facilitates proper placement of the physical object relative to the patient's anatomy.

[0083] A2. The method of A1, further comprising obtaining a video stream depicting a view of the surgical environment including a depiction of a patient's anatomy, and presenting the video stream on a display device, wherein presenting the AR element on the display device augments the video stream with the AR element in a desired position and arrangement for positioning a physical object relative to the patient's anatomy as depicted in the video stream.

[0084] A3. The method of A2, further including zooming-in to an area of ​​the surgical environment in which the AR element is presented as part of presenting the video stream and as the physical object is positioned relative to the patient's anatomy.

[0085] A4. The method of A1, wherein the display device includes a transparent display through which a user views the surgical environment, and the AR element is presented on the transparent display to augment the user's view of the surgical environment through the transparent display.

[0086] A5. The method of A4, wherein the transparent display is provided as part of a smart wearable glasses device.

[0087] A6. The method of A1, A2, A3, A4 or A5, wherein visually indicating comprises modifying a visual presentation of the AR element on a display device.

[0088] A7. The method of A6, wherein the AR element is initially at least partially a first color or pattern, and the visually indicating includes changing the first color or pattern to another color or pattern.

[0089] A8. The method of A1, A2, A3, A4 or A5, wherein tracking the position and location of the physical object is performed using at least one of: one or more markers on the physical object, a shape matching algorithm that identifies the position and location of the physical object within the view based on a known shape of the physical object, or tracking the position of at least one other physical object that has a fixed, known location relative to the physical object.

[0090] A9. The method of A1, A2, A3, A4 or A5, wherein the method further includes tracking a location and position of a patient's anatomy in the surgical environment, and presenting and repositioning the AR element based on the patient's anatomy being repositioned to maintain the AR element at a desired location and position of the physical object to be positioned relative to the patient's anatomy.

[0091] A10.A method of A1, A2, A3, A4 or A5, wherein the surgical plan includes one or more desired cuts to the patient's anatomical structure, the method further comprising determining that the one or more desired cuts have been made to the patient's anatomical structure, determining a desired location and position of a physical object, and presenting an AR element based on the one or more desired cuts made.

[0092] A11. The method of A1, A2, A3, A4 or A5, wherein the AR element is a first AR element, and the method further includes presenting on the display device a second AR element that is superimposed on the physical object in a view of the surgical environment.

[0093] A12. The method of A1, A2, A3, A4 or A5, wherein the physical object is an implant.

[0094] A13. The method of A1, A2, A3, A4 or A5, wherein the AR element is a digital model of a physical object.

[0095] A computer system including a memory and a processor in communication with the memory, the computer system executing a method including: determining a desired position and placement of a physical object to be positioned and placed relative to a patient's anatomy; presenting an augmented reality (AR) element on a display device overlaid on a view of a surgical environment; tracking the position and location of the physical object as it is moved within the surgical environment and determining and placing the position and location of the physical object relative to the patient's anatomy; and visually indicating on the display device that the physical object has been moved to the desired position and location relative to the patient's anatomy based on the position and placement of the physical object matching the desired position and location of the physical object. The desired position and placement is defined by a surgical plan for the patient. The surgical environment includes the patient's anatomy. Placing the AR element at the desired position and placement of the physical object to present the position and placement facilitates proper placement of the physical object relative to the patient's anatomy.

[0096] A15. The computer system of A14, wherein the method further includes obtaining a video stream depicting a view of the surgical environment including a depiction of a patient's anatomy, and presenting the video stream on a display device. Presenting the AR element on the display device augments the video stream with the AR element in a desired position and arrangement for positioning a physical object relative to the patient's anatomy as depicted in the video stream.

[0097] A16. The computer system of A15, further including zooming in on an area of ​​the surgical environment in which the AR element is presented as part of presenting the video stream and as the physical object is positioned relative to the patient's anatomy.

[0098] A17. The computer system of A14, wherein the display device includes a transparent display through which a user views the surgical environment, and the AR element is presented on the transparent display to augment the user's view of the surgical environment through the transparent display.

[0099] A18. The computer system of A17, wherein the transparent display is provided as part of a smart wearable glasses device.

[0100] A19. The computer system of A14, A15, A16, A17 or A18, wherein visually indicating includes modifying a visual presentation of the AR element on a display device.

[0101] A20. The computer system of A19, wherein the AR element is initially at least partially a first color or pattern, and the visually indicating includes changing the first color or pattern to another color or pattern.

[0102] A21. The computer system of A14, A15, A16, A17 or A18, wherein tracking the position and location of the physical object is performed using at least one of: one or more markers on the physical object, a shape matching algorithm that identifies the position and location of the physical object within the view based on a known shape of the physical object, or tracking the position of at least one other physical object that has a fixed, known location relative to the physical object.

[0103] A22. The computer system of A14, A15, A16, A17 or A18, wherein the method further includes tracking a location and position of the patient's anatomy in the surgical environment, and presenting the AR element and repositioning the AR element based on the patient's anatomy being repositioned to maintain the AR element at a desired location and position of the physical object to be positioned relative to the patient's anatomy.

[0104] A23. The computer system of A14, A15, A16, A17 or A18, wherein the surgical plan includes one or more desired cuts to the patient's anatomy, and the method further comprises determining that the one or more desired cuts have been made to the patient's anatomy, determining a desired location and position of a physical object, and presenting an AR element based on the one or more desired cuts made.

[0105] A24. The computer system of A14, A15, A16, A17 or A18, wherein the AR element is a first AR element, and the method further includes presenting on the display device a second AR element that is superimposed on the physical object in a view of the surgical environment.

[0106] A24. The computer system of A14, A15, A16, A17 or A18, wherein the physical object is an implant.

[0107] A26. The computer system of A14, A15, A16, A17 or A18, wherein the AR element is a digital model of a physical object.

[0108] A computer program product comprising a computer readable storage medium storing instructions readable by and executed by a processing circuit for performing a method including: determining a desired position and placement of a physical object to be positioned and placed relative to a patient's anatomy; presenting an augmented reality (AR) element on a display device overlaid on a view of a surgical environment; tracking the position and location of the physical object as it is moved within the surgical environment and determining and placing the position and location of the physical object relative to the patient's anatomy; and visually indicating on the display device that the physical object has been moved to the desired position and location relative to the patient's anatomy based on the position and placement of the physical object matching the desired position and location of the physical object. The desired position and placement is defined by a surgical plan for the patient. The surgical environment includes the patient's anatomy. Placing the AR element at the desired position and placement of the physical object to present the position and placement facilitates proper placement of the physical object relative to the patient's anatomy.

[0109] A28. The computer program product of A27, wherein the method further includes obtaining a video stream depicting a view of the surgical environment including a depiction of a patient's anatomy, and presenting the video stream on a display device. Presenting the AR element on the display device augments the video stream with the AR element in a desired position and arrangement for positioning a physical object relative to the patient's anatomy as depicted in the video stream.

[0110] A29. The computer program product of A28, further including zooming-in to an area of ​​the surgical environment in which the AR element is presented as part of presenting the video stream and as the physical object is positioned relative to the patient's anatomy.

[0111] A30.The computer program product of A27, wherein the display device includes a transparent display through which a user views the surgical environment, and the AR element is presented on the transparent display to augment the user's view of the surgical environment through the transparent display.

[0112] A31. The computer program product of A30, wherein the transparent display is provided as part of a smart wearable glasses device.

[0113] A32. The computer program product of A27, A28, A29, A30 or A31, wherein visually indicating includes modifying a visual presentation of the AR element on a display device.

[0114] A33. The computer program product of A32, wherein the AR element is initially at least partially a first color or pattern, and the visually indicating includes changing the first color or pattern to another color or pattern.

[0115] A34. The computer program product of A27, A28, A29, A30 or A31, wherein tracking the position and location of the physical object is performed using at least one of: one or more markers on the physical object, a shape matching algorithm that identifies the position and location of the physical object within the view based on a known shape of the physical object, or tracking the position of at least one other physical object that has a fixed, known location relative to the physical object.

[0116] A35. The computer program product of A27, A28, A29, A30 or A31, wherein the method further includes tracking a location and position of the patient's anatomy in the surgical environment, and presenting the AR element and repositioning the AR element based on the patient's anatomy being repositioned to maintain the AR element at a desired location and arrangement of the physical object to be positioned relative to the patient's anatomy.

[0117] A36. The computer program product of A27, A28, A29, A30 or A31, wherein the surgical plan includes one or more desired cuts to the patient's anatomy, and the method further comprises determining that the one or more desired cuts have been made to the patient's anatomy, determining a desired location and position of a physical object, and presenting the AR element based on the one or more desired cuts made.

[0118] A37. The computer program product of A27, A28, A29, A30 or A31, wherein the AR element is a first AR element, and the method further includes presenting on the display device a second AR element that is superimposed on the physical object in a view of the surgical environment.

[0119] A38. The computer program product of A27, A28, A29, A30 or A31, wherein the physical object is an implant.

[0120] A39. The computer program product of A27, A28, A29, A30 or A31, wherein the AR element is a digital model of a physical object.

[0121] B1. A computer-implemented method including: tracking a position and location of a physical object in a surgical environment as the physical object is moved within the surgical environment; presenting, on a display device, an augmented reality (AR) element overlaying the physical object in a view of the surgical environment; and visually indicating on the display device that the physical object has been moved to a desired position and location relative to a patient's anatomy based on the position and location of the physical object matching a desired position and location of the physical object. The view shows the physical object and the patient's anatomy within which the physical object is positioned and positioned according to a desired position and location of the physical object as dictated by a surgical plan for the patient. The presentation maintains, based on the tracking, an AR element overlaying the physical object in the view as the physical object is moved within the surgical environment.

[0122] B2.The method of B1, further comprising obtaining a video stream depicting a view of the surgical environment including a depiction of the patient's anatomical structure, and presenting the video stream on a display device, wherein presenting the AR element on the display device includes an AR element that augments the video stream and is displayed overlaid on a physical object in the view of the surgical environment.

[0123] B3. The method of B2, further including, as part of presenting the video stream and as the physical object is positioned relative to the patient's anatomy, magnifying a region of the surgical environment to display a desired location and position of the physical object.

[0124] B4.The method of B1, wherein the display device includes a transparent display through which a user views the surgical environment, and the AR element is presented on the transparent display to augment the user's view of the surgical environment through the transparent display.

[0125] B5. The method of B4, wherein the transparent display is provided as part of a smart wearable glasses device.

[0126] B6. The method of B1, B2, B3, B4 or B5, wherein visually indicating includes modifying a visual presentation of an AR element on a display device.

[0127] B7. The method of B6, wherein the AR element is initially at least partially a first color or pattern, and the visually indicating includes changing the first color or pattern to another color or pattern.

[0128] B8. The method of B1, B2, B3, B4 or B5, wherein tracking the position and location of the physical object is performed using at least one of: one or more markers on the physical object, a shape matching algorithm that identifies the position and location of the physical object within the view based on a known shape of the physical object, or tracking the position of at least one other physical object that has a fixed, known location relative to the physical object.

[0129] B9. The method of B1, B2, B3, B4 or B5, wherein the AR element is a first AR element, the method further comprising presenting on the display device a second AR element positioned or disposed relative to the patient's anatomy on the physical object at a desired location or arrangement.

[0130] B10. The method of B1, B2, B3, B4 or B5, wherein the AR element is a digital model of a physical object.

[0131] B10. The method of B1, B2, B3, B4 or B5, wherein the physical object is an implant.

[0132] B12. A computer system including a memory and a processor in communication with the memory, the computer system configured to perform a method including tracking a position and location of a physical object in a surgical environment as the physical object is moved within the surgical environment, presenting on a display device an augmented reality (AR) element overlaying the physical object in a view of the surgical environment, and visually indicating on the display device that the physical object has been moved to a desired position and location relative to a patient's anatomy based on the position and location of the physical object being consistent with a desired position and location of the physical object. The view shows the physical object and the patient's anatomy within which the physical object is positioned and positioned according to a desired position and location of the physical object as dictated by the patient's surgical plan. The presentation maintains an AR element overlaying the physical object in the view as the physical object is moved within the surgical environment based on the tracking.

[0133] B13. The computer system of B12, further comprising: acquiring a video stream depicting a view of the surgical environment including a depiction of the patient's anatomical structure; and presenting the video stream on a display device, wherein presenting the AR element on the display device includes an AR element that augments the video stream and is displayed overlaid on a physical object in the view of the surgical environment.

[0134] B14. The computer system of B13, wherein the method further includes magnifying an area of ​​the surgical environment to display a desired location and position of the physical object as part of presenting the video stream and as the physical object is positioned relative to the patient's anatomy.

[0135] B15. The computer system of B12, wherein the display device includes a transparent display through which a user views the surgical environment, and the AR element is presented on the transparent display to augment the user's view of the surgical environment through the transparent display.

[0136] The computer system of B16.B15, wherein the transparent display is provided as part of a smart wearable glasses device.

[0137] B17. The computer system of B12, B13, B14, B15 or B16, wherein visually indicating includes modifying a visual presentation of the AR element on a display device.

[0138] B18. The computer system of B17, wherein the AR element is initially at least partially a first color or pattern, and the visually indicating includes changing the first color or pattern to another color or pattern.

[0139] B19. The computer system of B12, B13, B14, B15 or B16, wherein tracking the position and location of the physical object is performed using at least one of: one or more markers on the physical object, a shape matching algorithm that identifies the position and location of the physical object within the view based on a known shape of the physical object, or tracking the position of at least one other physical object that has a fixed, known location relative to the physical object.

[0140] B20. The computer system of B12, B13, B14, B15 or B16, wherein the AR element is a first AR element, and the method further includes presenting on the display device a second AR element positioned or disposed relative to the patient's anatomy on the physical object at a desired location or arrangement.

[0141] B21. The computer system of B12, B13, B14, B15 or B16, wherein the AR element is a digital model of a physical object.

[0142] B22. The computer system of B12, B13, B14, B15 or B16, wherein the physical object is an implant.

[0143] A computer program product comprising a computer-readable storage medium storing instructions readable by and executed by a processing circuit for performing a method including: tracking a position and location of a physical object in a surgical environment as the physical object is moved within the surgical environment; presenting, on a display device, an augmented reality (AR) element overlaying the physical object in a view of the surgical environment; and visually indicating on the display device that the physical object has been moved to a desired position and location relative to a patient's anatomy based on the position and location of the physical object being consistent with the desired position and location of the physical object. The view shows the physical object and the patient's anatomy within which the physical object is positioned and positioned according to the desired position and location of the physical object as dictated by the patient's surgical plan. The presentation maintains, based on the tracking, an AR element overlaying the physical object in a view as the physical object is moved within the surgical environment.

[0144] B24. The computer program product of B23, wherein the method further includes obtaining a video stream depicting a view of the surgical environment including a depiction of a patient's anatomical structure, and presenting the video stream on a display device, wherein presenting the AR element on the display device includes an AR element that augments the video stream and is displayed overlaid on a physical object in the view of the surgical environment.

[0145] B25. The computer program product of B25, wherein the method further includes magnifying a region of the surgical environment to display a desired location and position of the physical object as part of presenting the video stream and as the physical object is positioned relative to the patient's anatomy.

[0146] The computer program product of B26.B23, wherein the display comprises a transparent display device through which a user views the surgical environment, and the AR element is presented on the transparent display to augment the user's view of the surgical environment through the transparent display.

[0147] The computer program product of B27.B26, wherein the transparent display is provided as part of a smart wearable glasses device.

[0148] B28. The computer program product of B23, B24, B25, B26 or B27, wherein visually indicating includes modifying a visual presentation of the AR element on a display device.

[0149] B29. The computer program product of B28, wherein the AR element is initially at least partially a first color or pattern, and wherein visually indicating includes changing the first color or pattern to another color or pattern.

[0150] B30.The computer program product of B23, B24, B25, B26 or B27, wherein tracking the position and location of the physical object is optionally performed using at least one of: one or more markers on the physical object, a shape matching algorithm that identifies the position and location of the physical object within the view based on a known shape of the physical object, or tracking the position of at least one other physical object that has a fixed, known location relative to the physical object.

[0151] B31. The computer program product of B23, B24, B25, B26 or B27, wherein the AR element is a first AR element, and the method further includes presenting on the display device a second AR element positioned or disposed relative to the patient's anatomy on the physical object at a desired location or arrangement.

[0152] B32. The computer program product of B23, B24, B25, B26 or B27, wherein the AR element is a digital model of a physical object.

[0153] B33. The computer program product of B23, B24, B25, B26 or B27, wherein the physical object is an implant.

[0154] The terms used herein are for the purpose of describing particular embodiments only and are not limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly indicates otherwise. Furthermore, it is understood that the terms "comprises" and / or "comprising" as used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0155] Corresponding structures, materials, acts, and equivalents (if any) of all means or step-plus-function elements in the following claims are intended to include any structures, materials, or acts for performing the functions in combination with other claimed elements as specifically claimed. The description of one or more embodiments has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art. The present embodiments have been selected and described to best explain various aspects and practical applications and to enable those skilled in the art to appreciate various embodiments with various modifications suited to the particular use envisaged.

Claims

1. A method that is performed on a computer, Determining the desired position and placement of a physical object to be positioned and placed relative to the anatomical structure of a patient, The display device will show augmented reality (AR) elements overlaid on a view of the surgical environment, In order to determine the position and location of the physical object relative to the anatomical structure of the patient, the position and location of the physical object are tracked as it is moved within the surgical environment. Based on the position and placement of the physical object to match the desired position and location of the physical object, the display device visually indicates that the physical object has been moved to the desired position and location relative to the patient's anatomical structure. Includes, The desired position and arrangement are determined by the patient's surgical plan. The surgical environment includes the anatomical structure of the patient, Placing the AR elements at the desired position and arrangement of the physical object and presenting the said position and arrangement facilitates the appropriate placement of the physical object relative to the anatomical structure of the patient. method.

2. Obtaining a video stream depicting a view of the surgical environment, including a depiction of the patient's anatomical structure, Presenting the aforementioned video stream on the display device, It further includes, Presenting the AR elements on the display device extends the video stream and includes the AR elements in a desired position and arrangement to position the physical object in correspondence with the patient's anatomical structure as depicted in the video stream. The method according to claim 1.

3. The presenting of the video stream and further including zooming in on the AR element into the region of the surgical environment where it is presented, as part of the presentation of the video stream, when the physical object is positioned relative to the anatomical structure of the patient. The method according to claim 2.

4. The display device includes a transparent display for the user to view the surgical environment. The AR elements are presented on the transparent display, and the user's view of the surgical environment is extended through the transparent display. The method according to claim 1.

5. The transparent display is provided as part of a smart wearable glasses device. The method according to claim 4.

6. The aforementioned visual representation includes changing the visual display of the AR elements on the display device. The method according to claim 1, 2, 3, 4, or 5.

7. The aforementioned AR element is initially at least partially a first color or pattern. The aforementioned visual representation includes changing the first color or pattern to another color or pattern. The method according to claim 6.

8. Tracking the position and location of the aforementioned physical object is, One or more markers on the aforementioned physical object, A shape matching algorithm that identifies the position and location of the physical object in a view based on the known shape of the physical object, or Tracking the position of at least one other physical object having a known fixed location relative to the aforementioned physical object. It is executed using at least one of the following. The method according to claim 1, 2, 3, 4, or 5.

9. Further includes tracking the location and position of the patient's anatomical structures in the surgical environment, Based on the repositioning of the patient's anatomical structure, the AR element is presented, the AR element is repositioned, and the AR element is maintained in the desired position and arrangement of the physical object to be positioned relative to the patient's anatomical structure. The method according to claim 1, 2, 3, 4, or 5.

10. The surgical plan includes one or more desired cuts into the patient's anatomical structure. The aforementioned method, It is determined that one or more desired cuts have been made to the anatomical structure of the patient. Furthermore, the system performs the following actions: determining the desired position and arrangement of the physical object, and presenting the AR elements based on the one or more desired cuts that have been performed. The method according to claim 1, 2, 3, 4, or 5.

11. The aforementioned AR element is the first AR element, The method further includes presenting a second AR element superimposed on the physical object in the view of the surgical environment to the display device. The method according to claim 1, 2, 3, 4, or 5.

12. The aforementioned physical object is an implant. The method according to claim 1, 2, 3, 4, or 5.

13. The AR element is a digital model of the physical object. The method according to claim 1, 2, 3, 4, or 5.

14. A computer system including memory and a processor that communicates with the memory, Determining the desired position and placement of a physical object to be positioned and placed relative to the anatomical structure of a patient, The display device will show augmented reality (AR) elements overlaid on a view of the surgical environment, In order to determine the position and location of the physical object relative to the anatomical structure of the patient, the position and location of the physical object are tracked as it is moved within the surgical environment. Based on the position and placement of the physical object to match the desired position and location of the physical object, the display device visually indicates that the physical object has been moved to the desired position and location relative to the patient's anatomical structure. A computer system that causes a method including the following to be performed: The desired position and arrangement are determined by the patient's surgical plan. The surgical environment includes the anatomical structure of the patient, Placing the AR elements at the desired position and arrangement of the physical object and presenting the said position and arrangement facilitates the appropriate placement of the physical object relative to the anatomical structure of the patient. Computer system.

15. The aforementioned method, Obtaining a video stream depicting a view of the surgical environment, including a depiction of the patient's anatomical structure, Presenting the aforementioned video stream on the display device, It further includes, Presenting the AR elements on the display device extends the video stream and includes the AR elements in a desired position and arrangement to position the physical object relative to the patient's anatomical structure as depicted in the video stream. The computer system according to claim 14.

16. The presenting of the video stream and further including the AR element expanding into the region of the surgical environment in which it is presented when the physical object is positioned relative to the anatomical structure of the patient, The computer system according to claim 15.

17. The display device includes a transparent display for the user to view the surgical environment. The AR elements are presented on the transparent display, and the user's view of the surgical environment is extended through the transparent display. The computer system according to claim 14.

18. The transparent display is provided as part of a smart wearable glasses device. The computer system according to claim 17.

19. The aforementioned visual representation includes changing the visual display of the AR elements on the display device. The computer system according to claim 14, 15, 16, 17, or 18.

20. The aforementioned AR element is initially at least partially a first color or pattern. The aforementioned visual representation includes changing the first color or pattern to another color or pattern. The computer system according to claim 19.

21. Tracking the position and location of the aforementioned physical object is, One or more markers on the aforementioned physical object, A shape matching algorithm that identifies the position and location of the physical object in a view based on the known shape of the physical object, or Tracking the position of at least one other physical object having a known fixed location relative to the aforementioned physical object. It is executed using at least one of the following. The computer system according to claim 14, 15, 16, 17, or 18.

22. The aforementioned method, Further includes tracking the location and position of the patient's anatomical structures in the surgical environment, Based on the repositioning of the patient's anatomical structure, the AR element is presented, the AR element is repositioned, and the AR element is maintained in the desired position and arrangement of the physical object to be positioned relative to the patient's anatomical structure. The computer system according to claim 14, 15, 16, 17, or 18.

23. The surgical plan includes one or more desired cuts into the patient's anatomical structure. The aforementioned method, It is determined that one or more desired cuts have been made to the anatomical structure of the patient. Furthermore, the system performs the following actions: determining the desired position and arrangement of the physical object, and presenting the AR elements based on the one or more desired cuts that have been performed. The computer system according to claim 14, 15, 16, 17, or 18.

24. The aforementioned AR element is the first AR element, The method further includes presenting a second AR element superimposed on the physical object in the view of the surgical environment to the display device. The computer system according to claim 14, 15, 16, 17, or 18.

25. The aforementioned physical object is an implant. The computer system according to claim 14, 15, 16, 17, or 18.

26. The AR element is a digital model of the physical object. The computer system according to claim 14, 15, 16, 17, or 18.

27. A computer program product comprising a computer-readable storage medium that is readable by a processing circuit and stores instructions for performing a method executed by the processing circuit, The method described above Determining the desired position and placement of a physical object to be positioned and placed relative to the anatomical structure of a patient, The display device will show augmented reality (AR) elements overlaid on a view of the surgical environment, In order to determine the position and location of the physical object relative to the anatomical structure of the patient, the position and location of the physical object are tracked as it is moved within the surgical environment. Based on the position and placement of the physical object to match the desired position and location of the physical object, the display device visually indicates that the physical object has been moved to the desired position and location relative to the patient's anatomical structure. Includes, The desired position and arrangement are determined by the patient's surgical plan. The surgical environment includes the anatomical structure of the patient, Placing the AR elements at the desired position and arrangement of the physical object and presenting the said position and arrangement facilitates the appropriate placement of the physical object relative to the anatomical structure of the patient. Computer program products.

28. The aforementioned method, Obtaining a video stream depicting a view of the surgical environment, including a depiction of the patient's anatomical structure, Presenting the aforementioned video stream on the display device, It further includes, Presenting the AR elements on the display device extends the video stream and includes the AR elements in a desired position and arrangement to position the physical object relative to the patient's anatomical structure as depicted in the video stream. The computer program product according to claim 27.

29. The presenting of the video stream and further including the AR element expanding into the region of the surgical environment in which it is presented when the physical object is positioned relative to the anatomical structure of the patient, The computer program product according to claim 28.

30. The display device includes a transparent display for the user to view the surgical environment. The AR elements are presented on the transparent display, and the user's view of the surgical environment is extended through the transparent display. The computer program product according to claim 27.

31. The transparent display is provided as part of a smart wearable glasses device. The computer program product according to claim 30.

32. The aforementioned visual representation includes changing the visual display of the AR elements on the display device. A computer program product according to claim 27, 28, 29, 30, or 31.

33. The aforementioned AR element is initially at least partially a first color or pattern. The aforementioned visual representation includes changing the first color or pattern to another color or pattern. A computer program product according to claim 27, 28, 29, 30, or 31.

34. Tracking the position and location of the aforementioned physical object is, One or more markers on the aforementioned physical object, A shape matching algorithm that identifies the position and location of the physical object in a view based on the known shape of the physical object, or Tracking the position of at least one other physical object having a known fixed location relative to the aforementioned physical object. It is executed using at least one of the following. A computer program product according to claim 27, 28, 29, 30, or 31.

35. The aforementioned method, Further includes tracking the location and position of the patient's anatomical structures in the surgical environment, Based on the repositioning of the patient's anatomical structure, the AR element is presented, the AR element is repositioned, and the AR element is maintained in the desired position and arrangement of the physical object to be positioned relative to the patient's anatomical structure. A computer program product according to claim 27, 28, 29, 30, or 31.

36. The surgical plan includes one or more desired cuts into the patient's anatomical structure. The aforementioned method, It is determined that one or more desired cuts have been made to the anatomical structure of the patient. Furthermore, the system performs the following actions: determining the desired position and arrangement of the physical object, and presenting the AR elements based on the one or more desired cuts that have been performed. A computer program product according to claim 27, 28, 29, 30, or 31.

37. The aforementioned AR element is the first AR element, The method further includes presenting a second AR element superimposed on the physical object in the view of the surgical environment to the display device. A computer program product according to claim 27, 28, 29, 30, or 31.

38. The aforementioned physical object is an implant. A computer program product according to claim 27, 28, 29, 30, or 31.

39. The AR element is a digital model of the physical object. A computer program product according to claim 27, 28, 29, 30, or 31.

40. A method that is performed on a computer, When a physical object is moved within the surgical environment, the position and location of the physical object within the surgical environment are tracked. The display device presents augmented reality (AR) elements in which the physical object is superimposed on the view of the surgical environment, Based on the fact that the position and location of the physical object coincide with the desired position and arrangement of the physical object, the display device visually indicates that the physical object has been moved to the desired position and arrangement relative to the patient's anatomical structure. Includes, The view shows the physical object and the anatomical structure of the patient in which the physical object is positioned and placed according to the desired position and arrangement of the physical object as indicated by the patient's surgical plan. The aforementioned presentation maintains the AR element that superimposes the physical object in the view as the physical object is moved within the surgical environment, based on the tracking. method.

41. Obtaining a video stream depicting a view of the surgical environment, including a depiction of the patient's anatomical structure, The video stream is presented on the aforementioned display device, It further includes, Presenting the AR elements on the display device includes extending the video stream and displaying the AR elements overlaid on the physical objects in the view of the surgical environment. The method according to claim 40.

42. As part of presenting the video stream, and further including zooming in on the area of ​​the surgical environment to show the desired position and placement of the physical object when the physical object is positioned relative to the patient's anatomical structure. The method according to claim 41.

43. The display device includes a transparent display for the user to view the surgical environment. The AR elements are presented on the transparent display, and the user's view of the surgical environment is extended through the transparent display. The method according to claim 40.

44. The transparent display is provided as part of a smart wearable glasses device. The method according to claim 43.

45. The aforementioned visual representation includes changing the visual display of the AR elements on the display device. The method according to claim 40, 41, 42, 43, or 44.

46. The aforementioned AR element is initially at least partially a first color or pattern. The aforementioned visual representation includes changing the first color or pattern to another color or pattern. The method according to claim 45.

47. Tracking the position and location of the aforementioned physical object is, One or more markers on the aforementioned physical object, A shape matching algorithm that identifies the position and location of the physical object in a view based on the known shape of the physical object, or Tracking the position of at least one other physical object having a known fixed location relative to the aforementioned physical object. It is executed using at least one of the following. The method according to claim 40, 41, 42, 43, or 44.

48. The aforementioned AR element is the first AR element, The method further includes presenting to the display device a second AR element positioned or positioned relative to the anatomical structure of the patient on the physical object at the desired position or arrangement. The method according to claim 40, 41, 42, 43, or 44.

49. The AR element is a digital model of the physical object. The method according to claim 40, 41, 42, 43, or 44.

50. The aforementioned physical object is an implant. The method according to claim 40, 41, 42, 43, or 44.

51. A computer system including memory and a processor that communicates with the memory, When a physical object is moved within the surgical environment, the position and location of the physical object within the surgical environment are tracked. The display device presents augmented reality (AR) elements in which the physical object is superimposed on the view of the surgical environment, Based on the fact that the position and location of the physical object coincide with the desired position and arrangement of the physical object, the display device visually indicates that the physical object has been moved to the desired position and arrangement relative to the patient's anatomical structure. A computer system that causes a method including the following to be performed: The view shows the physical object and the anatomical structure of the patient in which the physical object is positioned and placed according to the desired position and arrangement of the physical object as indicated by the patient's surgical plan. The presentation maintains the AR element that superimposes the physical object in the view when the physical object is moved within the surgical environment, based on the tracking. Computer system.

52. The aforementioned method, Obtaining a video stream depicting a view of the surgical environment, including a depiction of the patient's anatomical structure, The video stream is presented on the aforementioned display device, It further includes, Presenting the AR elements on the display device includes extending the video stream and displaying the AR elements overlaid on the physical objects in the view of the surgical environment. The computer system according to claim 51.

53. As part of presenting the video stream, and further including zooming in on the area of ​​the surgical environment to show the desired position and placement of the physical object when the physical object is positioned relative to the anatomical structure of the patient, The computer system according to claim 52.

54. The display device includes a transparent display for the user to view the surgical environment. The AR elements are presented on the transparent display, and the user's view of the surgical environment is extended through the transparent display. The computer system according to claim 51.

55. The transparent display is provided as part of a smart wearable glasses device. The computer system according to claim 54.

56. The aforementioned visual representation includes changing the visual display of the AR elements on the display device. The computer system according to claim 51, 52, 53, 54, or 55.

57. The aforementioned AR element is initially at least partially a first color or pattern. The aforementioned visual representation includes changing the first color or pattern to another color or pattern. The computer system according to claim 56.

58. Tracking the position and location of the aforementioned physical object is, One or more markers on the aforementioned physical object, A shape matching algorithm that identifies the position and location of the physical object in a view based on the known shape of the physical object, or Tracking the position of at least one other physical object having a known fixed location relative to the aforementioned physical object. It is executed using at least one of the following. The computer system according to claim 51, 52, 53, 54, or 55.

59. The aforementioned AR element is the first AR element, The method further includes presenting to the display device a second AR element positioned or positioned relative to the anatomical structure of the patient on the physical object at the desired position or arrangement. The computer system according to claim 51, 52, 53, 54, or 55.

60. The AR element is a digital model of the physical object. The computer system according to claim 51, 52, 53, 54, or 55.

61. The aforementioned physical object is an implant. The computer system according to claim 51, 52, 53, 54, or 55.

62. A computer program product comprising a computer-readable storage medium that is readable by a processing circuit and stores instructions for performing a method executed by the processing circuit, The method described above When a physical object is moved within the surgical environment, the position and location of the physical object within the surgical environment are tracked. The display device presents augmented reality (AR) elements in which the physical object is superimposed on the view of the surgical environment, Based on the fact that the position and location of the physical object coincide with the desired position and arrangement of the physical object, the display device visually indicates that the physical object has been moved to the desired position and arrangement relative to the patient's anatomical structure. Includes, The view shows the physical object and the anatomical structure of the patient in which the physical object is positioned and placed according to the desired position and arrangement of the physical object as indicated by the patient's surgical plan. The presentation maintains the AR element that superimposes the physical object in the view when the physical object is moved within the surgical environment, based on the tracking. Computer program products.

63. The aforementioned method, Obtaining a video stream depicting a view of the surgical environment, including a depiction of the patient's anatomical structure, The video stream is presented on the aforementioned display device, It further includes, Presenting the AR elements on the display device includes extending the video stream and displaying the AR elements overlaid on the physical objects in the view of the surgical environment. The computer program product according to claim 62.

64. The method further includes, as part of presenting the video stream, and zooming in on the area of ​​the surgical environment to display the desired position and placement of the physical object when the physical object is positioned relative to the patient's anatomical structure. The computer program product according to claim 63.

65. The display device includes a transparent display for the user to view the surgical environment. The AR elements are presented on the transparent display, and the user's view of the surgical environment is extended through the transparent display. The computer program product according to claim 62.

66. The transparent display is provided as part of a smart wearable glasses device. The computer program product according to claim 65.

67. The aforementioned visual representation includes changing the visual display of the AR elements on the display device. The computer program product according to claim 62, 63, 64, 65, or 66.

68. The aforementioned AR element is initially at least partially a first color or pattern. The aforementioned visual representation includes changing the first color or pattern to another color or pattern. The computer program product according to claim 67.

69. Tracking the position and location of the aforementioned physical object is, One or more markers on the aforementioned physical object, A shape matching algorithm that identifies the position and location of the physical object in a view based on the known shape of the physical object, or Tracking the position of at least one other physical object having a known fixed location relative to the aforementioned physical object. It is executed using at least one of the following. The computer program product according to claim 62, 63, 64, 65, or 66.

70. The aforementioned AR element is the first AR element, The method further includes presenting to the display device a second AR element positioned or positioned relative to the anatomical structure of the patient on the physical object at the desired position or arrangement. The computer program product according to claim 62, 63, 64, 65, or 66.

71. The AR element is a digital model of the physical object. The computer program product according to claim 62, 63, 64, 65, or 66.

72. The aforementioned physical object is an implant. The computer program product according to claim 62, 63, 64, 65, or 66.

Images