Ultrasound-based multi-bone registration surgical system and its application in computer-assisted surgery

Abstract

This document discloses a surgical system for surgical registration of patient bones with a surgical plan. As part of its ability to perform surgical registration, the system is configured to process ultrasound images of patient bones, the ultrasound images including the bone surface of each patient bone. The system includes a computing device comprising a processing device and a computer-readable medium storing one or more executable instructions thereon. The processing device is configured to execute the one or more executable instructions. The one or more executable instructions: i) detect the bone surface of each patient bone in the ultrasound image; and ii) separate a first point cloud of ultrasound image pixels associated with the bone surface of each patient bone.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 105,973, filed October 27, 2020, the entire contents of which are incorporated by reference into this application. TECHNICAL FIELD

[0003] The present disclosure relates to medical systems and methods for computer- assisted surgery. More particularly, the present disclosure relates to surgical registration systems and methods in computer-assisted surgery. BACKGROUND

[0004] Modern orthopedic joint replacement surgery often involves at least some degree of surgical pre-planning to improve the effectiveness and efficiency of a particular procedure. In particular, pre-planning can improve the accuracy of bone resection and implant placement while reducing the overall time of the procedure and the time the patient’s joint is open and exposed.

[0005] The use of robotic systems in the execution of orthopedic joint replacement surgery can greatly reduce intra-operative time for a particular procedure. Increasingly, the effectiveness of a procedure can be based on the tools, systems, and methods used in the pre-planning phase.

[0006] Examples of steps involved in pre-planning can involve determining: implant size, position, and orientation; resection planes and depths; access trajectories to the surgical site; and the like. In some cases, pre-planning can involve generating a three-dimensional (“3D”) patient-specific model of the patient’s bone and soft tissue to perform a joint replacement. The 3D patient model can be used as a visual aid to plan various possibilities for implant size, implant orientation, implant position, and corresponding resection planes and depths, among other parameters.

[0007] However, before a robotic system can perform a joint replacement, the robotic system and navigation system must be registered to the patient. Registration involves mapping the virtual boundaries and constraints defined in the pre-planning to the patient in physical space, so that the robotic system can be accurately tracked relative to the patient and constrained relative to the boundaries applied to the patient’s anatomy.

[0008] While frameworks for certain aspects of surgical registration can be known in the art, there is a need for systems and methods to further refine certain aspects of registration to further improve the efficiency and effectiveness of robotic and robot-assisted orthopedic joint replacement surgery. SUMMARY

[0009] Aspects of the present disclosure can include one or a combination of various neural networks trained to detect and optionally classify bone surfaces in ultrasound images.

[0010] This disclosure may also include an algorithm capable of simultaneously co-registering the bone surfaces of N bones (typically forming joints) between an ultrasound modality and a second modality (e.g., CT / MRI, or one or more statistical / general models deformed based on patient anatomical data), by optimizing the following to capture these bones: at least an Nx6DOF transformation from the ultrasound modality to the second modality; and classification information for assigning regions in the ultrasound modality image data to one of the N captured bones.

[0011] In some cases, to stitch individual ultrasound images (capturing only slices / small portions of bone) into a coherent 3D image dataset, an anatomical tracker tracks the ultrasound probe relative to each of the N captured bones. When the scanned bone is fixed, the ultrasound images can be stitched together by tracking only the ultrasound probe.

[0012] In some cases, multiple registration can be based on 3D point clouds / meshes. In this case, N bones can be segmented in the second modality (CT / MRI bone segmentation) to obtain a triangular mesh.

[0013] In some cases, multiple registration can be image-based. In this case, the classified ultrasound data is directly matched to the second modality without needing to detect the bone surface of the ultrasound image.

[0014] Aspects of this disclosure may include a system for surgically registering patient bone with a surgical plan, the registration employing ultrasound images of the patient's bone, the ultrasound images comprising a single ultrasound image including multiple bone surfaces, the single ultrasound image being generated from an ultrasound scan produced by an ultrasound probe passing through a single scan band of the patient's bone. In such a system, the system includes a computing device comprising a processing device and a computer-readable medium thereon storing one or more executable instructions. The processing device is configured to execute the one or more instructions. The one or more executable instructions include one or more neural networks trained to: i) detect bone surfaces in the single ultrasound image; and ii) classify each bone surface in the single ultrasound image according to bone type to obtain classified bone surfaces.

[0015] One or more neural networks may include convolutional networks for detecting bone surfaces in a single ultrasound image. According to embodiments, one or more neural networks may include pixel classification networks and / or likelihood classification networks for classifying each bone surface.

[0016] This system is advantageous because it can receive a single ultrasound image of bone surfaces with multiple bones, and then: i) detect the bone surfaces in the single ultrasound image; ii) classify each bone surface according to the type of bone to obtain classified bone surfaces. In other words, although a single ultrasound image includes bone surfaces of multiple bones, the system can still detect and classify the bone surfaces. This capability advantageously allows the generation of a single ultrasound image from an ultrasound scan produced by a single scan band of an ultrasound probe spanning the patient's bones forming a joint. Therefore, due to this capability, it is not necessary to limit the scan band of the ultrasound probe spanning the patient's joint bones to a single bone; the scan band can simply extend to all the bones of the joint, so that the resulting ultrasound image contains multiple bones, and the system is able to detect and classify the bone surfaces so that the bone surfaces can be sorted by the system.

[0017] In one variation of the system, the processing device executes one or more instructions to calculate the transformation from 2D image pixels to 3D points of a classified bone surface in a single ultrasound image, thereby generating a classified 3D bone surface point cloud. According to an embodiment, when calculating the transformation from 2D image pixels to 3D points of a classified bone surface in a single ultrasound image, the propagation speed of ultrasound waves in a specific medium can be taken into account, the known set of ultrasound probe orientations can be obtained with respect to the probe tracker in the ultrasound probe coordinate system, and the transformation between the probe tracker space and the ultrasound probe coordinate system can be calculated.

[0018] In one variation of the system, the processing device executes one or more instructions to calculate an initial or coarse registration of the patient bone to the computer model of the patient bone.

[0019] In one embodiment of the system, during the initial or coarse registration of patient bone to patient bone computer models, the system generates a first point cloud and a second point cloud. The first point cloud is a point cloud of a first bone in the patient bone relative to a first tracker associated with the first bone, and the second point cloud is a point cloud of a second bone in the patient bone relative to a second tracker associated with the second bone. Therefore, in the context of a patient's knee, the first point cloud is a point cloud of the femur in the patient bone relative to a first tracker fixed to the femur, and the second point cloud is a point cloud of the tibia relative to a second tracker fixed to the tibia. During the initial or coarse registration of patient bone to patient bone computer models, the system matches bone surface points of the first point cloud to the computer model of the first bone and matches bone surface points of the second point cloud to the computer model of the second bone.

[0020] In other embodiments of the system, when calculating the initial or coarse registration of the patient bone to patient bone computer model, the system may employ landmark-based registration and / or anatomical structure tracker pin-based registration.

[0021] In one variation of the system, the processing device executes one or more instructions to compute a final multiple bone registration using an initial or coarse registration and a classified 3D bone surface point cloud, wherein the final multiple bone registration achieves convergence between the classified 3D bone surface point cloud and the patient bone. Depending on the embodiment, when computing the final multiple bone registration in which convergence exists between the classified 3D bone surface point cloud and the patient bone, the system may refer to a first tracker to apply the initial or coarse registration to the classified 3D bone surface point cloud. In the context of a knee joint, the first tracker may be attached to the femur.

[0022] According to an embodiment, when calculating a convergent final multiple bone registration where there is a convergent 3D bone surface point cloud and patient bone, the system iteratively calculates the nearest point from the classified 3D surface point cloud to the computer model of the patient bone.

[0023] This disclosure may include a method for registering multiple bones of a patient's joint to a surgical plan. According to an embodiment, the method may include: receiving ultrasound images of a patient's joint, wherein at least some of the ultrasound images depict multiple bones; detecting bone surfaces of the multiple bones in the ultrasound images using a convolutional network; classifying each bone surface according to bone type using at least one of a likelihood classifier network or a pixel classifier network to obtain classified bone surfaces; transforming the 2D ultrasound image pixels of the classified bone surfaces to 3D to obtain a classified 3D bone surface point cloud; generating an initial coarse registration of the multiple bones of the patient's joint to medical image representations of the multiple bones of the patient's joint; and calculating a final multiple bone registration of the multiple bones of the patient's joint to the surgical plan by applying the initial coarse registration to the classified 3D bone surface point cloud.

[0024] In one embodiment, when transforming 2D ultrasound image pixels of classified bone surfaces into 3D, the propagation speed of ultrasound waves in a particular medium can be taken into account.

[0025] In one embodiment, when transforming the 2D ultrasound image pixels of the classified bone surface into 3D, the 2D ultrasound image pixels of the classified bone surface can be mapped from the 2D pixel space to the 3D metric coordinate system of the ultrasound probe coordinate system.

[0026] In one embodiment, when transforming the 2D ultrasound image pixels of the classified bone surface into 3D, a known set of poses of the probe tracker relative to the ultrasound probe coordinate system can be obtained.

[0027] In one embodiment, when transforming 2D ultrasound image pixels of the classified bone surface into 3D, the transformation between the probe tracker space and the ultrasound probe coordinate system can be calculated.

[0028] In one embodiment, during the initial coarse registration of multiple bone-to-bone medical image representations of a patient joint, a first point cloud and a second point cloud may be generated, wherein the first point cloud is a point cloud of a first bone among the multiple bones relative to a first tracker associated with the first bone, and the second point cloud is a point cloud of a second bone among the multiple bones relative to a second tracker associated with the second bone.

[0029] In one embodiment, when generating an initial coarse registration of multiple bones of a patient's joint with medical image representations of multiple bones of the patient's joint, bone surface points of a first point cloud can be matched to a computer model of a first bone, and bone surface points of a second point cloud can be matched to a computer model of a second bone.

[0030] In one embodiment, landmark-based registration may be employed when generating an initial coarse registration of multiple bone-to-bone medical image representations of a patient's joint.

[0031] In one embodiment, when generating an initial coarse registration of multiple bones of a patient's joint to multiple bone medical image representations of the patient's joint, registration based on anatomical structure tracker pins may be employed.

[0032] In one embodiment, when calculating the final multiple bone registration of multiple bones of a patient's joint with the surgical plan by applying an initial coarse registration to the classified 3D bone surface point cloud, the final multiple bone registration achieves convergence between the classified 3D bone surface point cloud and the patient's bones. In doing so, the initial coarse registration can be applied to the classified 3D bone surface point cloud with reference to a first tracker. Depending on the embodiment, during this final registration, the algorithm employed converges until its results reach a steady state, and the algorithm may also refine the classification of the classified 3D bone surface point cloud itself, such that any initial errors in the classification can be eliminated or at least reduced. In achieving these aspects of the final registration, the classified 3D bone surface point cloud and the initial or coarse registration become well-registered, resulting in the final multiple bone registration.

[0033] In one embodiment, when calculating the final multiple bone registration of multiple bones of a patient's joint with the surgical plan by applying an initial coarse registration to the classified 3D bone surface point cloud, the nearest point from the classified 3D surface point cloud to the computer model of the patient's bone can be calculated iteratively.

[0034] Aspects of this disclosure may include methods for registering a patient's bone surgery to a surgical plan. According to an embodiment, the method may include: receiving an ultrasound image of a patient's bone, the ultrasound image comprising a single ultrasound image including multiple bone surfaces, the single ultrasound image having been generated from an ultrasound scan by a single scan band of an ultrasound probe passing through the patient's bone; and employing one or more neural networks trained to: detect bone surfaces in the single ultrasound image; and classify each bone surface in the single ultrasound image according to bone type, thereby obtaining classified bone surfaces.

[0035] Aspects of this disclosure may include a surgical system configured to process ultrasound images of a patient's bone, the ultrasound images including the bone surface of each patient's bone. In one embodiment, the system includes a computing device including a processing device and a computer-readable medium thereon storing one or more executable instructions. The processing device is configured to execute one or more executable instructions. The one or more executable instructions i) detect the bone surface of each patient's bone in the ultrasound image; ii) separate a first point cloud of ultrasound image pixels associated with the bone surface of each patient's bone.

[0036] In a variation of this embodiment, bone surface detection can be performed using an image processing algorithm that forms at least a portion of one or more executable instructions. The image processing algorithm may include a machine learning model. Separation of the first point cloud can be performed using a pixel classification neural network that forms at least a portion of the one or more executable instructions. Separation of the first point cloud can also be performed using an image-based classification neural network that forms at least a portion of the one or more executable instructions.

[0037] In a variation of the embodiment, the processing device may execute the one or more executable instructions to calculate a transformation from the first point cloud to a separated 3D point cloud, the separated 3D point cloud being separated such that each of the ultrasound image pixels of the separated 3D point cloud is associated with a corresponding bone surface of the patient's bone. When calculating the transformation from the first point cloud to the separated 3D point cloud, the ultrasound image pixels may be calibrated for an ultrasound probe tracker and the ultrasound probe tracker may be calibrated for a tracking camera. When calibrating the ultrasound image pixels for the ultrasound probe tracker, the propagation speed of ultrasound waves in a particular medium may be taken into account. When calculating the transformation from the first point cloud to the separated 3D point cloud, the ultrasound image pixels are calibrated for the ultrasound probe tracker, the ultrasound probe tracker may be calibrated for a tracking camera, and the coordinate system is relative to the bone surface via an anatomical structure tracker located on the bone surface of the patient's bone. Separation of the first point cloud may occur through geometric analysis of the first point cloud.

[0038] In a variation of the embodiment, one or more executable instructions may compute an initial or coarse registration from a second point cloud obtained from the patient's bone to a bone model of the patient's bone. The second point cloud may include multiple point clouds relative to a plurality of trackers on the patient's bone. The multiple point clouds may include one point cloud registered with one bone model in the bone model of the patient's bone and another point cloud registered with another bone model in the bone model of the patient's bone.

[0039] Initial or coarse registration can be based on landmarks. Initial or coarse registration can be calculated based on the position and orientation of the anatomical structure tracker. During the calculation of initial or coarse registration, the system can generate a third point cloud and a fourth point cloud, where the third point cloud is the point cloud of the first bone in the patient's bone relative to the first tracker associated with the first bone, and the fourth point cloud is the point cloud of the second bone in the patient's bone relative to the second tracker associated with the second bone.

[0040] In a variation of the embodiment, during the initial or coarse registration calculation, the system can match bone surface points of the third point cloud to the computer model of the first bone and bone surface points of the fourth point cloud to the computer model of the second bone.

[0041] In a variation of this embodiment, the processing device may execute one or more instructions to compute a final multiple bone registration using initial or coarse registration and separated 3D point clouds, wherein the final multiple bone registration achieves a final registration between the separated 3D point clouds and the patient bone. In computing the final multiple bone registration, in which the final registration exists between the classified 3D bone surface point clouds and the patient bone, the system may iteratively refine the registration of the separated 3D point clouds to the computer model of the patient bone, and iteratively refine the separation of the separated 3D point clouds.

[0042] Aspects of this disclosure may include a method for processing ultrasound images of patient bones, the ultrasound images including the bone surface of each patient bone. One embodiment of this method may include: detecting the bone surface of each of the patient bones in the ultrasound image; and separating a first point cloud of ultrasound image pixels associated with the bone surface of each of the patient bones.

[0043] In a variation of this embodiment, bone surface detection can be performed using an image processing algorithm. The image processing algorithm may include a machine learning model. Separation of the first point cloud can be performed using a pixel classification neural network. Separation of the first point cloud can be performed using an image-based classification neural network.

[0044] In a variation of this embodiment, the method further includes: calculating a transformation from a first point cloud to a separated 3D point cloud, the separated 3D point cloud being separated such that each ultrasound image pixel of the separated 3D point cloud is associated with a corresponding bone surface of the patient's bone. When calculating the transformation from the first point cloud to the separated 3D point cloud, the ultrasound image pixels can be calibrated for an ultrasound probe tracker and the ultrasound probe tracker can be calibrated for a tracking camera. When calibrating the ultrasound image pixels for the ultrasound probe tracker, the propagation speed of ultrasound waves in a specific medium can be taken into account.

[0045] In a variation of this embodiment, when calculating the transformation from the first point cloud to the separated 3D point cloud, the ultrasound image pixels can be calibrated for the ultrasound probe tracker, the ultrasound probe tracker can be calibrated for the tracking camera, and the coordinate system is relative to the bone surface via an anatomical structure tracker located on the bone surface of the patient's bone. Separation of the first point cloud can occur through geometric analysis of the first point cloud.

[0046] In a variation of this embodiment, the method further includes calculating an initial or coarse registration from a second point cloud obtained from the patient's bone to a bone model of the patient's bone. The second point cloud may include multiple point clouds relative to a plurality of trackers on the patient's bone. The plurality of point clouds may include one point cloud registered to one bone model of the bone model of the patient's bone and another point cloud registered to another bone model of the bone model of the patient's bone. The initial or coarse registration may be landmark-based. The initial or coarse registration may be calculated based on the position and orientation of the anatomical structure trackers.

[0047] In a variation of this embodiment, during the initial or coarse registration calculation, a third point cloud and a fourth point cloud can be generated. The third point cloud is a point cloud of the first bone in the patient's bone relative to a first tracker associated with the first bone, and the fourth point cloud is a point cloud of the second bone in the patient's bone relative to a second tracker associated with the second bone. During the initial or coarse registration calculation, bone surface points of the third point cloud can be matched to the computer model of the first bone, and bone surface points of the fourth point cloud can be matched to the computer model of the second bone.

[0048] In a variation of this embodiment, the method further includes: calculating a final multiple bone registration using initial or coarse registration and separated 3D point clouds, wherein the final multiple bone registration achieves a final registration between the separated 3D point clouds and the patient bone. When calculating the final multiple bone registration, in which a final registration exists between the classified 3D bone surface point clouds and the patient bone, the registration of the separated 3D point clouds to the computer model of the patient bone can be iteratively refined, and the separation of the separated 3D point clouds can be iteratively refined. Attached Figure Description

[0049] This patent or application document contains at least one color drawing. A copy of this patent or patent application disclosure with color drawings will be provided by the Office upon request and payment of the necessary fees.

[0050] Figure 1 This is a diagram of a surgical system.

[0051] Figure 2 This is a flowchart illustrating the surgical plan and execution of arthroplasty.

[0052] Figure 3A and 3B The illustration shows tactile guidance during arthroplasty.

[0053] Figure 4A It is a coronal image showing the knee joint with the femur and tibia.

[0054] Figure 4B It is an axial image of the knee joint showing the femur and patella.

[0055] Figure 4C It is a sagittal diagram showing the knee joint, including the femur, patella, and tibia.

[0056] Figure 4D It is a coronal image showing the knee joint with the femur and tibia.

[0057] Figure 4E It is a 3D joint model, including 3D models of the femur, patella, and tibia.

[0058] Figure 5A It is shown Figure 2 The flowcharts shown illustrate the preoperative and intraoperative aspects of the registration process, which is an ultrasound-based multiple bone registration process.

[0059] Figure 5B yes Figure 5A The registration process is part of the graphical depiction of various aspects, namely the process of creating a classified or separated three-dimensional (“3D”) point cloud of the bone surface from ultrasound scans.

[0060] Figure 6A yes Figure 5A The flowchart shows a part of the registration process, namely the process of detecting bone surface in ultrasound images using pixel classification.

[0061] Figure 6B yes Figure 6A A graphical depiction of the process.

[0062] Figure 7A yes Figure 5A The flowchart is part of the registration process, namely the process of detecting bone surface in ultrasound images using likelihood classification.

[0063] Figure 7B yes Figure 7A A graphical depiction of the process.

[0064] Figure 8A yes Figure 5A The flowchart is part of the registration process, which is the process used to calculate the transformation from pixels in a two-dimensional (“2D”) ultrasound image to 3D points.

[0065] Figure 8B yes Figure 8A A graphical depiction of the process.

[0066] Figure 9A yes Figure 5A The flowchart is part of the registration process, namely the process of obtaining general registration data (probe attitude / markers) in CAD model space.

[0067] Figure 9B yes Figure 9A A graphical depiction of the process.

[0068] Figure 10A yes Figure 5A The flowchart shows the registration process, where the initial registration is obtained during the procedure.

[0069] Figure 10B and Figure 10C yes Figure 10A A graphical depiction of alternative aspects of the process.

[0070] Figure 11A yes Figure 5A The flowchart shows a part of the registration process, namely the "one-click / one-pose" registration process.

[0071] Figure 11B and Figure 11C yes Figure 11A Graphical depiction of various aspects of the process.

[0072] Figure 12 yes Figure 5A The flowchart is a part of the registration process, namely the landmark-based registration process.

[0073] Figure 13A yes Figure 5A The flowchart shows a part of the registration process, namely the registration process based on the anatomical structure tracker pin.

[0074] Figure 13B yes Figure 13A A graphical depiction of aspects of the process.

[0075] Figure 14A yes Figure 5A The flowchart shows a part of the registration process, namely the calculation process for the final multiple bone registration.

[0076] Figure 14B and Figure 14C yes Figure 14A Graphical depiction of various aspects of the process.

[0077] Figure 15 It is a diagram of a registration system or a verification of surgical targets.

[0078] Figure 16 It is an example computing system having one or more computing units that can implement the various systems and methods discussed herein. Detailed Implementation

[0079] This application is incorporated herein by reference in its entirety from the following applications: International Application PCT / US2017 / 049466, filed August 30, 2017, entitled "Systems and Methods for Intra-Operative Pelvic Registration"; PCT / US2016 / 034847, filed May 27, 2016, entitled "Preoperative Planning and Associated Intra-Operative Registration for a Surgical System"; and U.S. Patent Application No. 12 / 894,071, filed September 29, 2010, entitled "Surgical System for Positioning Prosthetic Component and / or for Constructing Movement of Surgical Instruments". SURGICAL TOOL); U.S. Patent Application No. 13 / 234,190, filed September 16, 2011, entitled "Systems and methods for measuring parameters in joint replacement surgery"; U.S. Patent Application No. 11 / 357,197, filed February 21, 2006, entitled "Haptic guidance system and method"; U.S. Patent Application No. 12 / 654,519, filed December 22, 2009, entitled "Transmission with first and second transmission elements"; U.S. Patent Application No. 12 / 644,964, filed December 22, 2009, entitled "Device that can be assembled by coupling". "BE ASSEMBLED BY COUPLING"; and U.S. Patent Application No. 11 / 750,807, filed on May 18, 2007, entitled "System and Method for Verifying Calibration of Assured Device".

[0080] This article discloses a surgical registration system and method for use in conjunction with a surgical system 100. Surgical registration requires mapping virtual boundaries, such as those defined in the preoperative plan, to working boundaries in physical space. This allows the surgical robot to perform certain actions within the virtual boundaries, such as drilling or removing bone surfaces. Once the virtual boundaries are mapped to the patient's physical space, the robot can drill or remove bone surfaces according to the planned location and orientation, but may be constrained from performing such operations outside the pre-planned virtual boundaries. Accurate and precise registration of the patient's anatomy allows for accurate navigation of the surgical robot during the surgical procedure. The requirements for accuracy and precision during registration must be balanced with the time required to perform the registration.

[0081] In the case of robot-assisted surgery, virtual boundaries can be defined in the preoperative plan. In the case of fully robotic surgery, virtual toolpaths can be defined in the preoperative plan. In either case, the preoperative plan may include, for example, defining the bone resection depth and identifying whether unacceptable notches in the anterior femoral cortex are associated with the proposed bone resection depth and the proposed candidate implant posture. Assuming that the bone resection depth and implant posture in the preoperative plan do not have unacceptable anterior femoral cortex notches and are approved by the surgeon, the bone resection depth can be updated to take into account cartilage thickness by intraoperative registration of the cartilaginous condylar surface of the actual patient bone with the patient bone model used in the preoperative plan. By taking cartilage thickness into account in this way, the actual implants, when implanted via the surgical system 100, will position their respective condylar surfaces to function as cartilaginous condylar surfaces replacing the resection of the actual patient bone. Further description of the preoperative planning can be found in PCT / US2016 / 034847, filed on May 27, 2016, entitled "Preoperative planning and associated intraoperative registration for a surgical system", which is incorporated herein by reference in its entirety.

[0082] Before we begin a detailed discussion of surgical registration, we will now give an overview of the surgical system and its operation.

[0083] I. Overview of the Surgical System

[0084] To begin a detailed discussion of the surgical system, refer to Figure 1 .from Figure 1As can be seen, the surgical system 100 includes a navigation system 42, a computer 50, and a tactile device 60 (also referred to as a robotic arm 60). The navigation system tracks the patient's bones (i.e., the tibia 10 and femur 11) and the surgical tools used during the procedure (such as pointer devices, probes, and cutting tools), enabling the surgeon to visualize the bones and tools on a monitor 56 during the osteotomy procedure.

[0085] Navigation system 42 can be any type of navigation system configured to track the pose (i.e., position and orientation) of the object. For example, navigation system 42 may include a non-mechanical tracking system, a mechanical tracking system, or any combination of non-mechanical and mechanical tracking systems. Navigation system 42 includes a detection device 44 that obtains the pose of the object relative to a reference coordinate system. The detection device tracks the pose of the object to detect its motion as it moves within the reference coordinate system.

[0086] In one embodiment, navigation system 42 includes, for example, Figure 1The non-mechanical tracking system shown is an optical tracking system having a detection device 44 and trackable elements (such as navigation markers 46, 47) respectively disposed on the tracked object (e.g., the patient's tibia 10 and femur 11) and detectable by the detection device 44. In one embodiment, the detection device 44 includes a visible light-based detector, such as MicronTracker (Claron Technology Inc., Toronto, Canada), which detects patterns (e.g., checkerboard patterns) on the trackable elements. In another embodiment, the detection device 44 includes a pair of stereo cameras sensitive to infrared radiation and locatable in the operating room where the arthroplasty procedure will be performed. The trackable elements are securely and stably attached to the tracked object and include an array of markers having a known geometric relationship with the tracked object. It is known that trackable elements can be active (e.g., light-emitting diodes or LEDs) or passive (e.g., reflective spheres, checkerboard patterns, etc.) and have unique geometries (e.g., unique geometrical arrangements of markers) or, in the case of active, wired, or wireless markers, unique emission patterns. During operation, the detection device 44 detects the position of the trackable element, and the surgical system 100 (e.g., the detection device 44 using embedded electronics) calculates the posture of the tracked object based on the position of the trackable element, its unique geometry, and its known geometric relationship to the tracked object. The tracking system 42 includes trackable elements for each object the user wishes to track, such as navigation markers 46 on the tibia 10 and navigation markers 47 on the femur 11. During haptic-guided robot-assisted surgery, the navigation system may also include haptic device markers 48 (to track the global or overall position of the haptic device 60), end effector markers 54 (to track the distal end of the haptic device 60), and freehand navigation probes 55, 57 for the registration process, which may be in the form of a tracked ultrasound probe 55 and a tracked pen 57 having a tip for touching certain relevant anatomical landmarks on the patient and certain registration positions on various parts of the system 100. Alternatively, the system 100 may employ electromagnetic tracking.

[0087] Although the systems and methods disclosed herein are presented in the context of robot-assisted surgical systems employing the aforementioned navigation systems, for example... of The surgical robot used is a robot-assisted surgical system, but this disclosure is readily applicable to other navigation surgical systems. For example, additionally or alternatively, the systems and methods disclosed herein can be applied to surgical procedures that use navigation arthroplasty jigs to prepare bone, such as in This is within the context of the eNact Knee Navigation software. Similarly, and additionally or alternatively, the systems and methods disclosed herein can be applied to surgical procedures that employ a navigation saw or a handheld robot to prepare bone.

[0088] like Figure 1 As shown, the surgical system 100 also includes processing circuitry, represented in the figure as computer 50. The processing circuitry includes a processor and a memory device. The processor may be implemented as a general-purpose processor, an application-specific integrated circuit (ASIC), one or more field-programmable gate arrays (FPGAs), a set of processing components, a dedicated processor, or other suitable electronic processing components. The memory device (e.g., memory, memory cell, storage device, etc.) is one or more devices (e.g., RAM, ROM, flash memory, hard disk storage, etc.) for storing data and / or computer code to perform or facilitate the various processes, layers, and functions described herein. The memory device may be or include volatile or non-volatile memory. The memory device may include a database component, an object code component, a script component, or any other type of information structure for supporting the various activities and information structures described herein. According to an exemplary embodiment, the memory device is communicatively connected to the processor via the processing circuitry and includes computer code for performing (e.g., via the processing circuitry and / or the processor) one or more processes described herein.

[0089] Computer 50 is configured to communicate with navigation system 42 and haptic device 60. Furthermore, computer 50 can receive information related to orthopedic / arthroplasty procedures and perform various functions related to the execution of osteotomy procedures. For example, computer 50 may have the necessary software to perform functions related to image analysis, surgical planning, registration, navigation, image guidance, and haptic guidance. More specifically, the navigation system can operate in conjunction with an autonomous robot or a surgeon's aid (haptic device) to execute arthroplasty procedures.

[0090] Computer 50 receives images of the patient's anatomical structures to be used in the arthroplasty procedure. (Reference) Figure 2Prior to performing arthroplasty, the patient's anatomy can be scanned using any known imaging technique, such as CT or MRI captured by a medical imaging machine (step 801). Furthermore, while this disclosure refers to medical images captured or generated by medical imaging machines such as CT or MRI machines, other methods for generating medical images are possible and considered herein. For example, images of bone can be generated intraoperatively using a medical imaging machine (e.g., a handheld scanning or imaging device that scans or registers the topography of bone surfaces). As yet another example, medical images can be generated using patient anatomical data obtained preoperatively or intraoperatively from multiple modalities by transforming one or more statistical / general models based on patient anatomical data. Therefore, the term "medical imaging machine" is intended to encompass devices of various sizes located at the imaging center or used intraoperatively (e.g., C-arms, handheld devices), and the term "medical image" is intended to encompass images, models, or other patient anatomically representative data useful for planning and performing arthroplasty procedures.

[0091] Continuing, the scanned data is then segmented to obtain a three-dimensional representation of the patient's anatomy. For example, a three-dimensional representation of the femur and tibia is created before performing knee arthroplasty. Using this three-dimensional representation and as part of the planning process, femoral and tibial landmarks can be selected, and the patient's femoral-tibial alignment, as well as the orientation and location of the proposed femoral and tibial implants, can be calculated, which can be selected for modeling and sizing via computer. Femoral and tibial landmarks may include the femoral head center, distal trochlear groove, center of the intercondylar eminence, tibial-ankle center, and medial tibial spine, etc. Femoral-tibial alignment is the angle between the femoral mechanical axis (i.e., the line from the femoral head center to the distal trochlear groove) and the tibial mechanical axis (i.e., the line from the ankle center to the center of the intercondylar eminence). Based on the patient's current femoral-tibial alignment and the required femoral-tibial alignment to be achieved through the arthroplasty procedure, and further including the size, type, and location of the proposed femoral and tibial implants, including the required extension, varus-valgus angle, and medial-lateral rotation associated with the proposed implantation, computer 50 is programmed to calculate a preoperative plan for the desired implantation of the proposed implant, or at least to assist in the implantation of the proposed implant, including resections to be performed via tactile device 60 during the arthroplasty procedure (step 803). The preoperative plan achieved through step 803 is provided to the surgeon for review, adjustment, and approval, and is updated according to the surgeon's instructions (step 802).

[0092] Since computer 50 is used to develop a surgical plan according to step 803, it should be understood that the user can interact with computer 50 at any stage during surgical planning to input information and modify any part of the surgical plan. The surgical plan may include multiple virtual boundaries (in the case of haptic-based robot-assisted surgery) or toolpath plans (in the case of autonomous robotic surgery). Virtual boundaries or toolpaths may represent holes and / or incisions to be made in bones 10, 11 during an arthroplasty procedure. Once the surgical plan is developed, haptic device 60 is used to assist the user in creating the planned holes and incisions in bones 10, 11. Preoperative planning, particularly regarding bone resection depth planning and prevention of anterior femoral notches, will be explained more fully below.

[0093] The creation of holes and incisions or resections in bones 10 and 11 can be accomplished with the aid of a tactilely guided interactive robotic system, such as the tactile guidance system described in U.S. Patent No. 8,010,180, entitled "Haptic Guidance System and Method," granted August 30, 2011, the entire contents of which are hereby incorporated herein by reference. When a surgeon manipulates a robotic arm to drill holes in the bone or to make cuts using a high-speed drill, sagittal saw, or other suitable tool, the system provides tactile feedback to guide the surgeon in carving the holes and cutting to the appropriate shapes; it is pre-programmed into the robotic arm's control system. Tactile guidance and feedback will be explained more fully below.

[0094] During surgical planning, computer 50 also receives information related to the femoral and tibial implants to be implanted in the arthroplasty procedure. For example, a user can use input device 52 (e.g., keyboard, mouse, etc.) to input parameters of the selected femoral and tibial implants into computer 50. Alternatively, computer 50 may contain a pre-established database of various implants and their parameters, from which the user can select the chosen implant. In yet another embodiment, the implant may be custom-designed according to the patient's specific surgical plan. Implant selection may occur at any stage of surgical planning.

[0095] The surgical plan may also be based on at least one parameter of the implant or a function of the implant parameters. Because the implant can be selected at any stage of the surgical planning process, it can be selected before or after the computer 50 determines the virtual boundaries of the plan. If the implant is selected first, the virtual boundaries of the plan can be based at least in part on the implant's parameters. For example, the distance (or any other relationship) between the virtual boundaries representing the holes or incisions to be created in bones 10, 11 can be planned based on desired varus-valgus femoral-tibial alignment, extension, medial-lateral rotation, or any other factors related to the desired surgical outcome of implanting the arthroplasty implant. In this way, the implementation of the surgical plan will result in proper alignment of the removed bone surfaces and holes to allow the selected implant to achieve the desired surgical outcome. Alternatively, the computer 50 can develop a surgical plan, including the virtual boundaries of the plan, prior to implant selection. In this case, the implant can be selected (e.g., input, selected, or designed) based at least in part on the virtual boundaries of the plan. For example, implants can be selected based on the planned virtual boundaries, so that the execution of the surgical plan will result in the correct alignment of the removed bone surface and the hole, allowing the selected implant to achieve the desired surgical outcome.

[0096] Virtual boundaries or toolpaths exist in virtual space and can represent features that exist or will be created in physical (i.e., real) space. A virtual boundary corresponds to a working boundary in physical space that can interact with objects in physical space. For example, a working boundary can interact with a surgical tool 58 coupled to a haptic device 60. While surgical plans are often described herein as including virtual boundaries representing holes and resections, surgical plans can include virtual boundaries representing other modifications to bones 10, 11. Furthermore, a virtual boundary can correspond to any working boundary in physical space that can interact with objects in physical space.

[0097] It should be noted that while the systems and methods disclosed herein are in the context of arthroplasty, they are readily applicable in surgical settings without implants. Therefore, for example, but not as a limitation, navigation and tactile sensing can be planned preoperatively to allow the systems disclosed herein to remove bone tumors (sarcomas) or to make other types of incisions or resections in bone or soft tissue during routine navigation procedures of any type.

[0098] Refer again Figure 2 Following surgical planning and before performing the arthroplasty procedure, a registration technique is used to register physical anatomical structures (e.g., bones 10, 11) with virtual representations of the anatomical structures (e.g., preoperative 3D representations) (step 804), as detailed below. Registration of the patient's anatomy allows for accurate navigation during the surgical procedure (step 805), enabling each virtual boundary to correspond to a working boundary in physical space. For example, referencing... Figure 3Aand Figure 3B A virtual boundary 62 representing the resection in the tibia 10 is displayed on a computer or other monitor 63, and the virtual boundary 62 corresponds to a working boundary 66 in physical space 69, such as a surgical site in a surgical room. A portion of the working boundary 66, in turn, corresponds to the planned resection location in the tibia 10.

[0099] Virtual boundaries and their corresponding working boundaries can be of any configuration or shape. (Reference) Figure 3A The virtual boundary 62 representing the proximal resection to be created in the tibia 10 can be any configuration suitable for assisting the user during the creation of the proximal resection in the tibia 10. A portion of the virtual boundary 62 shown in the virtual representation of the tibia 10 represents the bone to be removed by surgical tools. A similar virtual boundary can be generated for holes to be drilled or milled in the tibia 10 to facilitate the implantation of a tibia implant onto the resected tibia 10. The virtual boundary (and thus the corresponding working boundary) can include one or more surfaces that completely enclose and surround the three-dimensional volume. In alternative embodiments, the virtual boundary and working boundary do not completely surround the three-dimensional volume but instead include “active” surfaces and “open” portions. For example, the virtual boundary 62 representing the proximal resection in the tibia can have a substantially rectangular box-shaped “active” surface 62a and a collapsed funnel-shaped or triangular box-shaped “active” surface 62b connected to the rectangular box-shaped portion, having an “open” portion 64. In one embodiment, the virtual boundary 62 can be created using a collapsed funnel as described in U.S. Patent Application Serial No. 13 / 340,668, entitled "Systems and Methods for Selectively Activating Haptic Guide Zones," filed December 29, 2011, the entire contents of which are incorporated herein by reference. The working boundary 66 corresponding to the virtual boundary 62 has the same configuration as the virtual boundary 62. In other words, the working boundary 66 guiding proximal resection in the tibia 10 can have a substantially rectangular box-shaped "active" surface 66a and a collapsed funnel or triangular box-shaped "active" surface 66b connected to the rectangular box-shaped portion, having an "open" portion 67.

[0100] In another embodiment, the virtual boundary 62 representing the resection in bone 10 comprises only a generally rectangular box-shaped portion 62a. One end of the virtual boundary having only the rectangular box-shaped portion may have an "opening" top, such that the opening top of the corresponding working boundary coincides with the outer surface of bone 10. Alternatively, as Figure 3A and Figure 3B As shown, the rectangular box-shaped working boundary portion 66a corresponding to the virtual boundary portion 62a can extend beyond the outer surface of the bone 10.

[0101] In some embodiments, the virtual boundary 62 representing a resection through the bone may have a substantially planar shape, with or without thickness. Alternatively, the virtual boundary 62 may be curved or have an irregular shape. When the virtual boundary 62 is depicted as a line or planar shape and also has thickness, the virtual boundary 62 may be slightly thicker than the surgical tool used to create the resection in the bone, such that the tool may be constrained within the active surface of the working boundary 66 while inside the bone. This linear or planar virtual boundary 62 may be designed such that the corresponding working boundary 66 extends through the outer surface of the bone 10 in a funnel or other suitable shape to assist the surgeon as the surgical tool 58 approaches the bone 10. Tactile guidance and feedback (described below) may be provided to the user based on the relationship between the surgical tool 58 and the active surfaces of the working boundary.

[0102] Surgical planning may also include virtual boundaries to facilitate the entry and exit of tactile control, including the automatic alignment of surgical tools, as described in U.S. Patent Serial No. 13 / 725,348, entitled "Systems and Methods for Haptic Control of a Surgical Tool", filed December 21, 2012, the entire contents of which are incorporated herein by reference.

[0103] Surgical planning, including virtual boundaries, can be developed based on information related to the patient's bone mineral density. The patient's bone density is calculated using data obtained from CT, MRI, or other imaging of the patient's anatomy. In one embodiment, a calibrated object representing human bone with a known calcium content is imaged to obtain a correspondence between image intensity values ​​and bone mineral density measurements. This correspondence can then be applied to convert the intensity values ​​of individual images of the patient's anatomy into bone mineral density measurements. The individual images of the patient's anatomy and the corresponding bone mineral density maps are then segmented and used to create a three-dimensional representation (i.e., a model) of the patient's anatomy, including the patient's bone mineral density information. Image analysis, such as finite element analysis (FEA), can then be performed on the model to assess its structural integrity.

[0104] The ability to assess the integrity of a patient's anatomical structure improves the effectiveness of arthroplasty planning. For example, if certain portions of a patient's bone appear to have low density (i.e., osteoporosis), the holes, resections, and implant placement can be planned to minimize the risk of fracture in vulnerable bone areas. Furthermore, the structural integrity of the planned bone and implant combination can be assessed preoperatively to improve the surgical plan after the implementation of the surgical plan (e.g., postoperative bone and implant placement). In this embodiment, holes and / or incisions are planned, and bone and implant models are manipulated to represent the patient's bone and implant placement after the arthroplasty and implantation procedures. Various other factors that may affect the structural integrity of the postoperative bone and implant placement, such as the patient's weight and lifestyle, can be considered. The structural integrity of the postoperative bone and implant placement is analyzed to determine whether the placement is structurally sound and kinematically functional postoperatively. If the analysis reveals structural weaknesses or kinematic problems, the surgical plan can be modified to achieve the desired postoperative structural integrity and function.

[0105] In one embodiment, once the surgical plan has been finalized, the surgeon can perform the arthroplasty procedure with the aid of the haptic device 60 (step 806). In one embodiment, as an alternative to or supplement to the haptic device 60 (step 806), the surgical system 100 employs... Advanced Guidance Technologies Precision Knee navigation software. Precision knee navigation software helps guide the cutting guide into place.

[0106] In the context of the embodiment employing the tactile device 60 according to step 806, the surgical system 100 provides tactile guidance and feedback to the surgeon via the tactile device 60 to help the surgeon accurately execute the surgical plan. Compared to conventional arthroplasty techniques, tactile guidance and feedback during the arthroplasty procedure allows for better control of surgical instruments, resulting in more accurate alignment and placement of implants. Furthermore, tactile guidance and feedback are designed to eliminate the need for the use of Kirschner wires and fluoroscopy for planning purposes. Instead, the surgical plan is created and validated using a three-dimensional representation of the patient's anatomy, and the tactile device provides guidance during the surgical procedure.

[0107] "Haptic feedback" refers to the sensation of touch, and the field of haptics involves human-interactive devices that provide tactile and / or force feedback to the operator. Tactile feedback typically includes tactile sensations such as vibration. Force feedback (also known as a "wrench") refers to feedback in the form of force (e.g., resistance to movement) and / or torque. Wrenches include, for example, feedback in the form of force, torque, or a combination of force and torque. Haptic feedback can also include disabling or altering the amount of power supplied to surgical tools, which can provide tactile and / or force feedback to the user.

[0108] Surgical system 100 provides tactile feedback to the surgeon based on the relationship between surgical tool 58 and at least one of the working boundaries. The relationship between surgical tool 58 and the working boundaries can be any suitable relationship between surgical tool 58 and the working boundaries, which can be obtained by the navigation system and utilized by surgical system 100 to provide tactile feedback. For example, the relationship can be the position, orientation, posture, velocity, or acceleration of surgical tool 58 relative to one or more working boundaries. The relationship can also be any combination of the position, orientation, posture, velocity, and acceleration of surgical tool 58 relative to one or more working boundaries. The “relationship” between surgical tool 58 and the working boundaries can also refer to a quantity or measurement resulting from another relationship between surgical tool 58 and the working boundaries. In other words, one “relationship” can be a function of another relationship. As a specific example, the “relationship” between surgical tool 58 and the working boundaries can be the magnitude of the tactile force resulting from the positional relationship between surgical tool 58 and the working boundaries.

[0109] During surgery, the surgeon manipulates the haptic device 60 to guide the surgical instrument 58 coupled to the device. The surgical system 100 provides haptic feedback to the user via the haptic device 60 to assist the surgeon during planned bone preparation, incisions, or other modifications required to facilitate the implantation of femoral and tibial implants. For example, the surgical system 100 can assist the surgeon by substantially preventing or restraining the surgical instrument 58 from crossing the working boundary. The surgical system 100 can restrain the surgical instrument from crossing the working boundary through any number and combination of haptic feedback mechanisms, including by providing tactile feedback, by providing force feedback, and / or by changing the amount of force supplied to the surgical instrument. As used herein, “restraint” is used to describe a tendency to limit movement. Thus, the surgical system can directly restrain the surgical instrument 58 by applying a reaction force to the haptic device 60, which tends to limit the movement of the surgical instrument 58. The surgical system can also indirectly restrain the surgical instrument 58 by providing haptic feedback to alert the user to change his or her action, as alerting the user to change his or her action often restrains the movement of the surgical instrument 58. In yet another embodiment, the surgical system 100 can constrain the surgical tool 58 by limiting the power supplied to it, which again tends to restrict the movement of the tool.

[0110] In various embodiments, the surgical system 100 provides tactile feedback to the user when the surgical tool 58 approaches the working boundary, when the surgical tool 58 contacts the working boundary, and / or after the surgical tool 58 has penetrated the working boundary to a predetermined depth. The surgeon may experience the tactile feedback as, for example, vibration, resistance or active resistance to further movement of the tactile device, or as a solid “wall” that substantially prevents further movement of the tactile device. Alternatively, the user may experience the tactile feedback as a tactile sensation caused by a change in the power supplied to the surgical tool 58 (e.g., a change in vibration), or a tactile sensation caused by the cessation of power supplied to the tool. If the power supplied to the surgical tool is changed or stopped while the surgical tool 58 is drilling, cutting, or otherwise manipulating directly into bone, the surgeon will feel tactile feedback in the form of resistance to further movement because the tool is no longer able to drill, cut, or otherwise penetrate the bone. In one embodiment, when the surgical tool 58 contacts the working boundary, the power of the surgical tool is changed (e.g., the power of the tool is reduced) or stopped (e.g., the tool is disabled). Alternatively, the power supplied to the surgical tool 58 can be changed (e.g., reduced) as the surgical tool 58 approaches the working boundary.

[0111] In another embodiment, the surgical system 100 can assist the surgeon in creating planned holes, incisions, and other modifications to the bone by providing tactile feedback to guide the surgical tool 58 toward or along the working boundary. As an example, the surgical system 100 can apply forces to the tactile device 60 based on the positional relationship between the tip of the surgical tool 58 and the nearest coordinate of the working boundary. These forces can bring the surgical tool 58 closer to the nearest working boundary. Once the surgical tool 58 is substantially close to or in contact with the working boundary, the surgical system 100 can apply a force that tends to guide the surgical tool 58 to move along a portion of the working boundary. In another embodiment, the force tends to guide the surgical tool 58 from one portion of the working boundary to another portion of the working boundary (e.g., from a funnel-shaped portion of the working boundary to a rectangular box-shaped portion of the working boundary).

[0112] In yet another embodiment, the surgical system 100 is configured to assist the surgeon in creating planned holes, incisions, and modifications to bone by providing haptic feedback to guide surgical tools from one working boundary to another. For example, when the user guides the surgical tool 58 toward the working boundary 66, the surgeon may experience a force that tends to pull the surgical tool 58 toward the working boundary 66. When the user subsequently removes the surgical tool 58 from the space enclosed by the working boundary 66 and manipulates the haptic device 60 to bring the surgical tool 58 closer to a second working boundary (not shown), the surgeon may experience a force that pushes away from the working boundary 66 and toward the second working boundary.

[0113] The haptic feedback described herein can be operated in conjunction with modifications to the working boundary by the surgical system 100. Although modifications to the “working boundary” are discussed herein, it should be understood that the surgical system 100 modifies a virtual boundary, which corresponds to the working boundary. Some examples of modifying the working boundary include: 1) reconfiguration of the working boundary (e.g., changes in shape or size), and 2) activation and deactivation of the entire working boundary or portions of the working boundary (e.g., converting an “open” portion to an “active” surface and vice versa). Similar to haptic feedback, modifications to the working boundary can be performed by the surgical system 100 based on the relationship between the surgical tool 58 and one or more working boundaries. Modifications to the working boundary further assist the user in creating the desired holes and incisions during arthroplasty procedures by facilitating various actions, such as movement of the surgical tool 58 toward bone and cutting bone through the surgical tool 58.

[0114] In one embodiment, modification of the working boundary facilitates the movement of the surgical tool 58 toward the bone 10. During the surgical procedure, because the patient's anatomy is tracked by the navigation system, the surgical system 100 moves the entire working boundary 66 in response to the movement of the patient's anatomy. In addition to this baseline movement, portions of the working boundary 66 can be reshaped and / or reconfigured to facilitate the movement of the surgical tool 58 toward the bone 10. As an example, the surgical system may tilt the funnel-shaped portion 66b of the working boundary 66 relative to the rectangular box-shaped portion 66a during the surgical procedure based on the relationship between the surgical tool 58 and the working boundary 66. Thus, the working boundary 66 can be dynamically modified during the surgical procedure such that the surgical tool 58 remains within the space enclosed by the portion 66b of the working boundary 66 as it approaches the bone 10.

[0115] In another embodiment, a working boundary or a portion thereof is activated and deactivated. Activating and deactivating the entire working boundary can assist the user as the surgical instrument 58 approaches bone 10. For example, a second working boundary (not shown) may be deactivated while the surgeon is approaching the first working boundary 66 or while the surgical instrument 58 is within the space enclosed by the first working boundary 66. Similarly, the first working boundary 66 may be deactivated after the surgeon has completed the creation of the first corresponding resection and is ready to create the second resection. In one embodiment, the working boundary 66 may be deactivated after the surgical instrument 58 has entered the area within the funnel portion leading to the second working boundary but still outside the area of ​​the first funnel portion 66b. Activating a portion of the working boundary converts a previously open portion (e.g., the top 67 of the opening) into an active surface of the working boundary. Conversely, deactivating a portion of the working boundary converts a previously active surface of the working boundary (e.g., the end 66c of the working boundary 66) into an "open" portion.

[0116] Activation and deactivation of the entire working boundary or a portion thereof can be dynamically performed by the surgical system 100 during the surgical procedure. In other words, the surgical system 100 can be programmed to determine the presence of factors and relationships that trigger the activation and deactivation of a virtual boundary or a portion thereof during the surgical procedure. In another embodiment, a user can interact with the surgical system 100 (e.g., by using input device 52) to indicate the start or completion of various stages of the arthroplasty procedure, thereby triggering the activation or deactivation of the working boundary or a portion thereof.

[0117] In view of the operation and function of the surgical system 100 as described above, the discussion will now turn to the method of preoperative planning of the surgery performed by the surgical system 100, followed by a detailed discussion of the method of registering the preoperative plan with the patient’s actual bone and the applicable components of the surgical system 100.

[0118] The tactile device 60 can be described as a surgeon-assistive device or tool because it is manipulated by a surgeon to perform various resections, drilling, etc. In some embodiments, the device 60 can be an autonomous robot, distinct from a surgeon-assistive device. That is, a tool path opposite to the tactile boundary can be defined for bone resection and drilling, as the autonomous robot may operate simply along a predetermined tool path, thus requiring no tactile feedback. In some embodiments, the device 60 can be a cutting device with at least one degree of freedom operating in conjunction with a navigation system 42. For example, the cutting tool may include a rotating burr with a tracker on the tool. The cutting tool can be freely manipulated and held by the surgeon. In this case, tactile feedback may be limited to the burr stopping its rotation when it encounters a virtual boundary. Therefore, the device 60 is broadly considered to include any device described herein, as well as others.

[0119] Postoperative analysis can be performed immediately after the surgical procedure or some time later (step 807). Postoperative analysis determines the accuracy of the actual surgical procedure compared to the planned procedure. That is, the actual implant placement and orientation can be compared to the planned values. Factors such as varus-valgus femoral-tibial alignment, extension, internal-external rotation, or any other factors related to the surgical outcome of the arthroplasty implant can be compared to the planned values.

[0120] II. Preoperative steps in arthroplasty

[0121] Preoperative steps in arthroplasty procedures may include patient imaging and preoperative planning. Preoperative planning may include implant placement, determination of bone resection depth, and assessment of the anterior axial notch, among other evaluations. Determination of bone resection depth involves selecting and positioning candidate femoral and tibial implants in three-dimensional computer models relative to the patient's distal femur and proximal tibia to determine the location and orientation of the implants that will achieve the desired surgical outcome of the arthroplasty procedure. As part of this assessment, the necessary depths of tibial and femoral resection, as well as the orientation of the planes of these resections, are calculated.

[0122] Anterior axial notching assessment involves determining whether the anterior convex portion of the selected femoral implant's 3D model will intersect the anterior axis of the patient's distal femur 3D model when the implant 3D model is positioned and oriented relative to the proposed femoral 3D model during bone resection depth determination. Such intersection of the two models indicates an anterior femoral notch, which must be avoided.

[0123] The determination of bone resection depth and the performance of anterior axial notch assessment are described in PCT / US2016 / 034847, filed May 27, 2016, in whole or in part, which is incorporated herein by reference.

[0124] A. Preoperative imaging

[0125] In preparing for surgical procedures (e.g., knee arthroplasty, hip arthroplasty, ankle arthroplasty, shoulder arthroplasty, elbow arthroplasty, spinal procedures (e.g., fusion, implantation, scoliosis correction, etc.)), patients can undergo preoperative imaging, for example, at an imaging center. Patients may undergo magnetic resonance imaging (“MRI”), computed tomography (“CT”), radiographic scanning (“X-ray”), and other imaging modalities of the surgical joint. Figure 4A As shown, Figure 4A This is an example coronal image scan of a patient's knee joint 102, including the femur 104, patella 105 (shown in other images), and tibia 106, which may have undergone a CT scan. The CT scan may include a helical scan of the knee joint 102 packaged as a Medical Digital Imaging and Communications (“DICOM”) file. From the file, two-dimensional image slices or cross-sections can be viewed in multiple planes, such as coronal, sagittal, and axial. Figure 4B , Figure 4C and Figure 4D It is understood that the segmentation process of the two-dimensional image 108 can be performed by applying spline 110 along the bone contour. Alternatively, the segmentation process of the image 108 can be performed on the whole without applying spline 110 to 2D image slices. Such preoperative imaging and planning steps can be found in PCT / US2019 / 066206, filed December 13, 2019, the entire contents of which are incorporated herein by reference.

[0126] Figure 4B , Figure 4C and Figure 4D Axial images 108 of the femur 104 and patella 105, with splines 110 on the bone surfaces, are shown respectively; sagittal images 108 of joint 102, with splines 110 on the bone surfaces of the femur 104, patella 105, and tibia 106; and coronal images 108 of joint 102, with splines 110 on the femur 104 and tibia 106. In some cases, the segmentation process can be a manual process, with a human identifying the splines 110 on each two-dimensional image slice 108. In some cases, the segmentation process can be automated, where the splines 110 are automatically applied to the bone contours in the image slices 108. And in some cases, the segmentation process can be a combination of manual and automated processes.

[0127] After the segmentation process is completed, the segmented images 108 can be combined to generate a three-dimensional (“3D”) bone model 111 of the joint 102, including a 3D femoral model 112, a 3D patellar model 113, and a 3D tibia model 114.

[0128] like Figure 4E As shown, Figure 4E This is an isometric axial-coronal-sagittal view of a 3D joint model 111, representing joint 102 before any surgical procedure altering the bone is performed, more specifically, its femur 104, patella 105, and tibia 106 in a degenerated state. From this 3D joint model 111, various steps of the preoperative planning process can be performed. Each of these 3D bone models 112-114 can be generated relative to the coordinate system of the medical imaging system used to generate the two-dimensional image slices 108. For example, in the case of generating image slices 108 via CT imaging, the 3D bone model can be generated relative to the CT coordinate system 115.

[0129] In some cases, 3D models 111 of the patient's joints can be generated from statistical or general models of these bones and joints, including 3D models 112, 113, and 114 of each bone 104, 105, and 106 of the patient's joint 102. These models approximate the bones 104, 105, and 106 of the patient's joint 102 by deforming or otherwise modifying the statistical or general models based on certain factors that do not require segmentation of the 2D image slices 108 with splines 110. In some cases, the segmentation process can manually, automatically, or a combination of both to fit the 3D statistical or general bone model to the scanned images 108 of the femur 104, patella 105, and tibia 106. In this case, the segmentation process will not require applying splines 110 to each 2D image slice 108. Instead, the 3D statistical or general bone model will be fitted or deformed to the shape of the femur 104, patella 105, and tibia 106 in the scanned image 108. Therefore, the deformed or fitted 3D bone model will require... Figure 4E The 3D joint model 111 shown is shown.

[0130] In one embodiment, a generic bone model can be the result of an analysis of the size and shape of many (e.g., thousands or tens of thousands) medical images (e.g., CT, MRI, X-ray, etc.) of actual bones, used to generate a generic bone model that is a statistical average of many actual bones. In another embodiment, a statistical model describing demographic distribution is derived, including variations in size, shape, and appearance in the images.

[0131] In some cases, other methods for generating patient models can be employed. For example, a patient bone model or a portion thereof can be generated intraoperatively by registering bone or cartilage surfaces in one or more regions of the bone. Such a process can produce one or more bone surface contours. Therefore, the various methods described herein are intended to cover three-dimensional bone models generated from segmented medical images (e.g., CT, MRI) and intraoperative imaging methods.

[0132] Although the imaging and subsequent steps of the method are described with reference to the knee joint 102, the teachings in this disclosure are equally applicable to other joints such as the hip, ankle, shoulder, wrist, elbow, and spine, etc.

[0133] B. Preoperative planning for implant selection, implant location, and orientation.

[0134] After generating a 3D femoral model 112 of the patient's joint 102, the remainder of the preoperative planning can begin. For example, the surgeon or surgical system 100 can select a suitable implant and determine its location and orientation. These selections can determine the appropriate cutting or removal of the patient's bone to fit the chosen implant. Such preoperative planning steps can be found in PCT / US2016 / 034847, filed May 27, 2016, the entire contents of which are incorporated herein by reference.

[0135] III. Surgical Procedure

[0136] After the preoperative planning steps are completed, the surgery can begin according to the plan. That is, the surgeon can use the tactile device 60 of the surgical system 100 to perform bone removal on the patient, and the surgeon can implant an implant to restore joint function. The steps of the surgical procedure may include the following.

[0137] A. Registration

[0138] Registration is to include bone models 111-114 ( Figure 4E The preoperative plan, including bone models 111-114 and associated virtual boundaries or tool paths, is mapped to the patient's physical bones, thus spatially orienting the robotic arm 60 relative to the patient's physical bones for accurate execution of surgical procedures. The preoperative plan, including bone models 111-114 and associated virtual boundaries or tool paths, can be stored on computer 50 in a first coordinate system (x1, y1, z1). Navigation system 42, which tracks the movement of robotic arm 60 via various tracker arrays (e.g., 48, 54), also communicates with computer 50. Navigation system 42 also tracks the patient's body via various tracker arrays 46, 47 positioned on the tibia 10 and femur 11, respectively. Thus, the position and orientation (i.e., pose) of robotic arm 60 and the surgical bones 10, 11 are known relative to each other in a second coordinate system (x2, y2, z2) in computer 50. The process of mapping, transforming, or registering the first coordinate system (x1, y1, z1) and the second coordinate system (x2, y2, z2) together in a common coordinate system is called registration.

[0139] Once registered, the bone models 111-114 and the virtual boundaries or tool paths can be "locked" to the appropriate positions on the patient's physical bones, such that any movement of the patient's physical bones will cause the bone models 111-114 and the virtual boundaries or tool paths to move accordingly. Therefore, the robotic arm 60 may be constrained to operate using the virtual boundaries or along a tool path defined in the preoperative plan and moving with the patient's bones. In this way, the robotic arm 60 spatially understands the patient's body posture through the registration process.

[0140] i. Creating classified / separated 3D bone surface point clouds from intraoperative ultrasound data

[0141] As discussed in detail below, the computer 50 of the surgical system 100, more specifically, the computer's processor and memory, stores and executes one or more algorithms employing one or a combination of various neural networks trained to: detect bone surfaces in ultrasound images; and classify those bone surfaces in the ultrasound images according to captured anatomical structures.

[0142] As discussed in detail below, the computer 50 of the surgical system 100, more specifically, the computer's processor and memory, stores and executes one or more algorithms that allow simultaneous co-registration of the bone surfaces of N bones (typically forming joints) between an ultrasound modality and a second modality (e.g., CT / MRI) capturing N bones. The co-registration of the bone surfaces of the N bones between the two modalities is achieved through one or more algorithms that optimize: an Nx6DOF transform from the ultrasound modality to the second modality; and classification information that assigns regions in the image data of the ultrasound modality to one of the N captured bones.

[0143] In one embodiment, common registration of the bone surfaces of N bones between two modalities can occur between a 3D point cloud and a triangular mesh to become a 3D point cloud / mesh-based model. In this case, the N bones need to be segmented in the second modality (e.g., CT / MRI bone segmentation) to obtain a triangular mesh, and the 3D point cloud is applied to the triangular mesh.

[0144] In another embodiment, the common registration of the bone surfaces of N bones between the two modalities can be image-based. In other words, the classified ultrasound image data is directly matched to the second modality without the need to detect the bone surfaces.

[0145] To begin discussing one or more algorithms for simultaneously co-registering the bone surfaces of N bones between two modalities capturing N bones, refer to... Figure 5A . Figure 5A It's a flowchart that explains the details about... Figure 2 The diagram shows the composition of the diagram. Figure 1 The entire registration process (step 804) used by the surgical system 100 shown. Figure 5AThe process described herein is a two-step ultrasound-based multiple bone registration process 503, in which the initial or coarse registration generated by steps 610, 612, 776 (step 776) is combined with a 3D bone surface point cloud of classification or segmentation generated from steps 512, 527, 580, 600 and created from an ultrasound scan of the surgical target region (i.e., the patient joint region in the context of patient arthroplasty) (step 600) (step 900). Thus, the ultrasound-based multiple bone registration process 503 can be said to have two main steps or aspects, in which the initial registration establishes a first guess of the registration alignment, and then refines the starting point by calculating a highly accurate alignment.

[0146] The classification or segmentation of 3D bone surface point clouds in the ultrasound-based multiple bone registration process 503 (step 600) begins with acquiring intraoperative ultrasound images of the patient surface region around the joint and most (if not all) of each bone near the joint. For example, in knee arthroplasty, ultrasound scans cover most (if not all) of the knee, scanning up and down once or multiple times to acquire ultrasound image data of the bone surface of each bone (femur, tibia, and patella) of the patient's knee. The intraoperative ultrasound images are then algorithmically analyzed using machine learning to determine which individual points out of millions of individual points in the acquired ultrasound images belong to each bone of the patient's joint, thereby generating a classification or segmentation point cloud associated with each bone. In other words, in the context of knee arthroplasty, the algorithm appropriately assigns each point or pixel of the intraoperative ultrasound images to the corresponding bone of the knee joint, so that each point or pixel can be said to be classified or segmented to correspond to its respective bone, thus producing a classified or segmented 3D bone surface point cloud. In other words, each point or pixel in the intraoperative ultrasound image is transformed into a classified or separated 3D bone surface point cloud, such that the ultrasound image pixels or points of the classified or separated 3D bone surface point cloud are each associated with the corresponding bone surface of the patient's bone.

[0147] from Figure 5AAs can be understood, the initial or coarse registration (step 776) begins with the tracked probe 57 being applied intraoperatively to the patient's knee according to a specific preoperative planned posture or a set of landmarks relative to the patient's anatomy to generate a transformation that registers the physically tracked bone with a 3D CAD bone model 111 generated from segmented medical imaging (CT, MRI, etc.) of the patient's kneecap. Final registration occurs through an algorithmic combination of the initial registration of step 776 and the classification of the 3D bone surface point cloud in step 600, where the various portions of the classification point cloud are algorithmically matched to their respective surfaces in the initially registered point cloud and the 3D CAD bone model (step 900). During this final registration, the algorithm converges until its results reach a steady state, and the algorithm may also refine the classification or separation of the 3D bone surface point cloud itself so that any initial errors in classification / separation can be eliminated or at least reduced. In achieving these aspects of final registration, the classification or separation of the 3D bone surface point cloud and the initial or coarse registration become well-registered, resulting in the final multiple bone registration. Then, the surgical system 100 can employ this final multiple bone registration of step 900 when performing surgery on the patient's joint.

[0148] This ultrasound-based registration process 503 of the surgical system 100 is effective because registration can be achieved simply by a medical professional performing an ultrasound scan of the patient's joint region. Machine learning then takes over to identify which points in the ultrasound scan belong to which bone of the patient's joint, and then assigns / matches these points to the correct bone in the 3D bone model. Therefore, the registration process 503 allows all multiple bones of the patient's joint to be imaged by ultrasound at once. The system then identifies and separates points in the point cloud associated with each bone of the joint, and then assigns / matches these points to the appropriate bone in the 3D model of the joint to complete the final registration process. These points are not only assigned to the appropriate bone, but also located at the corresponding anatomical position on the bone.

[0149] like Figure 5A As shown, the ultrasound-based registration process 503 of the surgical system 100 includes a preoperative aspect 500 and an intraoperative aspect 502, and the preoperative aspect 500 and the intraoperative aspect 502 are each divided into a workflow and a data flow component. For the workflow component, a person operates the machine / tool / equipment / system or physically performs the identified workflow steps. For the data flow component, see the following reference... Figure 16 A more detailed discussion, with Figure 1 One or more hardware processors 1302 of the computer system 1300 associated with the surgical system 100 depicted in the diagram execute the program when performing the identified data stream steps.

[0150] In the preoperative workflow section 500, medical images of the patient's joints are acquired (step 504), as discussed above in the "A. Preoperative Imaging" section of this specific embodiment.Figure 5A As shown, the preoperative aspect 500 continues the data stream section, where medical images are subsequently used to generate a 3D CAD model of the bone forming the patient's joint (step 506), as referenced above. Figure 4A-4E This is discussed in section A. Preoperative Imaging of this specific embodiment. The data stream portion of the preoperative aspect 500 is relative to the patient's bone 104-106 (in the CAD model space) relative to the patient joint image 108. Figure 4A-4D 3D CAD models 111-114 Figure 4E The process ends with the generation of initial registration data (step 508). Specifically, the generation of initial registration data (step 508) includes defining the probe pose and anatomical landmarks in the 3D CAD model space.

[0151] Go to Figure 5A In the intraoperative aspect of the ultrasound-based multiple bone registration process 503, the workflow section 502 involves the medical professional using an anatomical structure tracker (e.g., see...). Figure 1 46 and 47) are installed in the patient's bone (e.g., Figure 1 On the tibia 10 and femur 11 (step 510). Although in a preferred embodiment, trackers 46, 47 are typically located on the tibia 10 and femur 11 as shown in step 510. Figure 5A The intraoperative portion 502 of the entire registration process 503 begins with installation in the patient's bones 10, 11. In an alternative embodiment, if the target limb (e.g., the leg) is adequately secured, the trackers 46, 47 can be used. Figure 1 Before attaching to bones 10 and 11, partial registration is completed. For example, after the initial registration (step 776) is completed, the classification point cloud is created (step 600), as follows... Figure 5A As will be discussed in more detail. The positions of trackers 46 and 47 can then be identified, and the trackers can be mounted on the patient's bones 10 and 11 accordingly. The initial registration 776 is repeated, and then the initial registration of step 776 and the classification point cloud of step 600 can be combined to achieve the final registration (step 900).

[0152] Refer again Figure 5A To continue with the preferred embodiment, wherein at the beginning of the intraoperative portion 502 of the entire registration process 503, trackers 46, 47 are mounted on the patient's bones 10, 11 according to step 510, and the medical professional then uses a trackable ultrasound probe (e.g., see...) Figure 1 (55) to record multiple ultrasound scans of the patient’s bones across the joint region (step 512). These scans may be performed in a general manner targeting the joint region, capturing various bones (e.g., femur, tibia, and patella in the background of the knee joint) in a mosaic of ultrasound image data points where it is not defined which data point belongs to which bone or its location on the bone.

[0153] from Figure 5B This is understandable. Figure 5B It is a graphical depiction of the process used to create a classified three-dimensional (“3D”) point cloud of bone surface from ultrasound scans, by Figure 5A The ultrasound scan 514 generated in step 512 has connections with the femur 11, patella (not shown), and tibia 10 (see, for example, see...). Figure 1 The associated ultrasound image data is undefined in terms of which bone and its location. Although this point in the process is not actually defined, as... Figure 5B As shown, ultrasound scan 514 can be understood as having femoral data points 516, patellar data points 518, and tibial data points 520. This data can be placed in the form of a set of 2D ultrasound images 523, including a femoral ultrasound image 522, a patellar ultrasound image (not shown), and a tibial ultrasound image 524, although these ultrasound images contain data points for which specific points are undefined as part of bone or soft tissue, and undefined which specific bone of the joint the specific data point belongs to, or its location on that specific bone. The surgical system 100 then transmits this set of 2D ultrasound images 523 via… Figure 5A The classification module (step 527) is converted into classified bone surface pixels 526, such as Figure 5B As shown in the image. It should be noted that, although in Figure 5B The 2D ultrasound images described herein and discussed herein can be readily performed using 3D ultrasound images, either in lieu of or in combination with 2D ultrasound images. Therefore, throughout this disclosure, any reference to 2D ultrasound images should be understood to also include 3D ultrasound images.

[0154] like Figure 5AAs shown, in one embodiment, bone surface pixels 526 for classification are generated from a set 523 of 2D ultrasound images. The classification module (step 527) can employ either of two alternative classification processes: ultrasound image bone surface detection via a pixel classification neural network (step 528) or ultrasound image bone surface detection via a likelihood classification neural network (step 530). In other embodiments, bone surface pixels 526 for classification are generated from the set of 2D ultrasound images 523. The classification module (step 527) can employ other processes, such as deriving no classification from the distance between two navigation markers 46, 47 fixed on bones 10, 11, and / or running a classification algorithm on the generated point cloud itself, without viewing the image, but only viewing the 3D arrangement of the different points. In other embodiments, the classification of the point cloud is solved by other non-machine learning processes or networks other than classification and convolution, such as random forests. In a further embodiment, the classification of the point cloud can be solved by non-machine learning processes. In one embodiment, the classification of the point cloud can be solved by geometric analysis of the point cloud. For example, such geometric analysis of point clouds might include principal component analysis (e.g., in the context of knee arthroplasty), principal axis splitting, clustering methods like connectivity component analysis (e.g., in the context of spine / vertebral procedures), and shape attributes like convex / concave / tubular / etc. Finally, without unduly limiting this disclosure, a classification module 527 is provided, which receives an image as input and outputs a bone classification point cloud, and various processes that may be part of the classification module are provided to achieve these purposes.

[0155] Figure 6A This is a flowchart (step 528) of the process of a surgical system 100 that uses pixel classification to detect bone surfaces in ultrasound images. Figure 6B yes Figure 6A A graphical depiction of the process. For example... Figure 6A As shown, the process employs an ultrasound image bone surface detector 532 and an ultrasound image pixel classifier 534, both of which are divided into input, processing, and output sections. Figure 6A and Figure 6B As shown, the ultrasound image bone surface detector 532 receives... Figure 5B The 2D ultrasound images 523 are used as input (step 536) and processed by a convolutional network to detect whether there is a bone surface point at each point in each ultrasound image (step 538), and output a binary image 540, where "zero (0) equals no bone surface" 542 and "one (1) equals bone surface" 544 (step 546).

[0156] Similarly, the ultrasound image pixel classifier 534 receives... Figure 5BThe 2D ultrasound images 523 are used as input (step 548) and processed by a classification network to determine the specific anatomical structure type present at each point in each ultrasound image (step 550). The output is a single number 552 representing the anatomical structure type, where “zero (0) equals tibia,” “one (1) equals femur,” and “two (2) equals patella” (step 554). It should be remembered that although the examples given in this detailed discussion are in the context of the knee joint, the concepts taught herein are equally applicable to any type of joint, such as, but not limited to, the spine, shoulder, elbow, wrist, hip, ankle, etc.

[0157] Figure 7A This is a flowchart of the process (step 530) in which the surgical system 100 uses likelihood classification to detect bone surface in ultrasound images. Figure 7B yes Figure 7A A graphical depiction of the process. This process is divided into an input section, a processing section, and an output section. For example... Figure 7A and Figure 7B As shown, the convolutional network 556 receives... Figure 5B The 2D ultrasound image 523 is used as input (step 558) and processed to detect bone surface points and classify each point by defining the probability that the point belongs to a specific anatomical structure (step 560). For example, against the background of the knee, the convolutional network 556 detects bone surface points in the 2D ultrasound image 523 and then calculates the probability that any particular detected bone surface point belongs to the femur, tibia, or patella (step 560). The convolutional network 556 outputs N binary images 562, where N is the number of bones considered, and each pixel of each binary image is decoded with "zero (0)" or "one (1)" depending on whether the pixel represents a bone surface (step 563).

[0158] from Figure 7B It is understandable that in this illustrative example, the patella is omitted for the sake of this example, and the number N of binary images will be two because the femur and tibia are present. If the patella were also used in this example, the number N of binary images would be three because the femur, patella, and tibia are present.

[0159] Continuing with the example where the number of binary images N is two, the convolutional network 556 analyzes the ultrasound image 523 and classifies the image 523 according to the probability that the image represents either the tibia image 564 or the femur image 566. The classification is performed by algorithmic evaluation of the ultrasound image 523 in the context of machine learning. Figure 7B Step 562). Then, based on the pixels of each classified image 564 classified in step 562, decoding is performed to determine whether each particular pixel represents a bone surface. Figure 7BStep 568). For example, the classified tibial image 564 is evaluated to identify its non-bone surface pixels 570 and its bone surface pixels 572, where the bone surface pixel 572 will be the tibial bone surface pixel 572 (step 568). Similarly, the classified femoral image 566 is evaluated to identify its non-bone surface pixels 574 and its bone surface pixels 576, where the bone surface pixel 576 will be the femoral bone surface pixel 576 (step 568).

[0160] return Figure 5A and Figure 5B Once the classified bone surface pixels 526 have been generated from the set of 2D ultrasound images 523 via the classification module (step 527) to provide bone surface pixels 526F for 2D femur classification and bone surface pixels 526T for 2D tibia classification, the 2D to 3D transformation of these 2D surface pixels 526F and 526T is calculated to convert the 2D surface pixels 526F and 526T into 3D surface pixels 578F and 578T (step 580).

[0161] Figure 8A This is a flowchart for calculating the process of transforming 2D surface pixels 526F and 526T into 3D surface pixels 578F and 578T (step 580), and Figure 8B yes Figure 8A A graphical depiction of the process. For example... Figure 8A and Figure 8B As shown, the process employs ultrasound probe time calibration (step 582), wherein ultrasound waves 583 projected and detected via the distal tip 55A of the ultrasound probe are applied to the bone ( Figure 1 Ultrasound scans were performed on 10 and 11 of the samples. Figure 5B When considering the propagation speed of ultrasound 583 in certain media / tissues, as the ultrasound probe 55 is tracked intraoperatively, the ultrasound probe trackable element 55B is detected by the detection device 44 of the navigation system 42 (step 584). The distal tip 55A includes a sensor array with its own intrinsic coordinate system 586. In doing so, the 2D surface pixels 526F, 526T of the ultrasound images 522, 524 undergo transformation 585 to map from the 2D pixel space (2D pixel coordinate system) 586 to the 3D metric coordinate system 588, transforming the 2D surface pixels 526F, 526T into 3D surface pixels 578F, 578T, and transforming their 2D pixel coordinates into 3D coordinates in the intrinsic ultrasound probe coordinate system 588 (step 590).

[0162] like Figure 8A As shown and from Figure 8BUnderstandably, after completing the ultrasonic probe time calibration (step 582), the process moves to ultrasonic probe-to-probe tracker calibration (step 592). In doing so, the system acquires a known set of attitudes of the ultrasonic probe 55 relative to the probe detection device 44 of the navigation system 42 with respect to the intrinsic ultrasonic probe coordinate system 588 (step 594). The system then performs the transformation between the probe tracker space and the intrinsic ultrasonic probe coordinate system (step 596).

[0163] Additional information regarding supplementary and / or alternative processes associated with calculating the transformation from 2D surface pixels 526F, 526T to 3D surface pixels 578F, 578T according to step 580 or variations thereof, can be found in PCT application number PCT / IB2018 / 056189 (International Publication No. WO2019 / 035049A1), internationally filed on August 16, 2018, entitled “Ultra-sound bone registration with learning-based segmentation and sound speed calibration,” which is hereby incorporated herein by reference in its entirety.

[0164] Although the foregoing discussion has been conducted in the context of a 2D ultrasound probe, it should be understood that a 2D ultrasound probe can be replaced by a 3D ultrasound probe to perform the procedures disclosed in this embodiment. Therefore, the procedures disclosed in this embodiment should not be limited to 2D ultrasound probes and 2D pixels / points, but should be considered to include 3D ultrasound probes in an image coordinate system and any type of ultrasound pixels / points, whether those ultrasound pixels / points are 2D or 3D.

[0165] Each individual ultrasound scan using an ultrasound probe generates a separate ultrasound image capturing a slice or small portion of the patient's bone. When imaging a patient's bone with ultrasound, multiple individual ultrasound scans using an ultrasound probe are typically required. To stitch the individual ultrasound images into a coherent 3D ultrasound image dataset, the ultrasound probe is tracked relative to an anatomical tracker attached to each of the N captured bones.

[0166] When the bone in an ultrasound scan is fixed, the process of stitching together individual ultrasound images can be simplified. Specifically, in this case, each individual ultrasound image can be stitched together with other individual ultrasound images simply by tracking the ultrasound probe.

[0167] from Figure 5A and Figure 5BUnderstandably, once the system completes step 580, it creates a classified 3D bone surface point cloud 598 (step 600). The classified 3D bone surface point cloud 598 will have a femoral point 602 classified as the femur, a patellar point 604 classified as the patella, and a tibial point 606 classified as the tibia. For additional or alternative aspects of the registration process using an ultrasound probe, reference is made to U.S. Patent Application No. 14 / 144,961, filed December 31, 2013, entitled "Systems and Methods of Registration Using an Ultrasound Probe," the disclosure of which is incorporated herein by reference in its entirety.

[0168] ii. Initial coarse registration

[0169] As discussed above and Figure 5A As shown, during the preoperative workflow portion 500, medical images of the patient's joints are acquired (step 504), as discussed above in the "A. Preoperative Imaging" section of this specific embodiment. Figure 5A As shown, the preoperative aspect 500 continues the data stream section, where medical images are subsequently used to generate a 3D CAD model of the bone forming the patient's joint (step 506), as referenced above. Figure 4A-4E As discussed in the "A. Preoperative Imaging" section of this specific embodiment, the data stream portion of the preoperative aspect 500 is relative to the patient's bone 104-106 (in the CAD model space) as shown in the patient joint image 108. Figure 4A-4D 3D CAD models 111-114 Figure 4E The process ends with the generation of initial registration data (step 508). Specifically, the generation of the initial registration data (step 508) includes defining the probe pose and anatomical landmarks in the 3D CAD model space.

[0170] In order to discuss the basis Figure 5A Step 508 defines the preoperative procedures for probe orientation and anatomical landmarks, now refer to Figure 9A , Figure 9A This is a flowchart illustrating step 508. Figure 9A As shown, step 508 begins by defining an arbitrary orientation P of the probe relative to the patient's anatomy to be registered (step 700). From Figure 9A and Figure 9B This is understandable; probe 57 ( Figure 1 The 3D CAD model (i.e., the 3D CAD probe model 57M) was then compared with... Figure 4E The 3D CAD bone model 111 shown is positioned in a defined arbitrary pose P, and the transformation (T) is recorded. Probe-to-3DimageTransformation 701 maps the 3D CAD probe model 57M to the 3D CAD bone model 111 from the probe coordinate system CS. probe To 3D image coordinate space CS 3Dimage (Step 702). Specifically, as a non-limiting example of many possible arbitrary poses P, the 3D CAD probe model 57M is positioned such that it is perpendicular to the center of the anterior femoral cortex of the 3D CAD bone model 111 of the knee region 102, pointing inward and facing medially, as shown. Figure 9B As illustrated in the diagram. Of course, any other arbitrary pose P suitable for a specific surgical application can be defined. Step 702 can be performed by a specialized surgical planner or surgeon. The 3D CAD bone model 111 of the knee region 102 can be a volumetric rendering or other type of CT image or medical image, or a model defined from a CT image or medical image.

[0171] like Figure 5A As shown, once the generation of initial registration data has been completed as discussed above regarding step 508, the initial registration data (step 610) is acquired as part of the workflow of the intraoperative aspect 502 of the ultrasound-based multiple bone registration process 503. For those based on Figure 5A The discussion of step 610, obtaining initial registration data, is now referenced. Figure 10A , Figure 10A A flowchart summarizing step 610. (See attached flowchart.) Figure 10A As shown, step 610 begins with positioning the tracked probe 57 relative to the patient's anatomy according to the pose P defined in step 508 (step 704). In other words, for step 704, in this example, Figure 9B The posture P between the 3D CAD probe model 57M and the 3D CAD bone model 111 and the 3D CAD femur model 112 depicted in the procedure is replicated between the actual physical probe 57 and the patient's actual femur 11 during the operation. This intraoperative posture P is then recorded (step 706).

[0172] from Figure 10B It can be understood that the recording of the intraoperative posture P in step 706 can be achieved using the tracking camera 44 of the tracking system 42, which obtains the transformation (T). Probe-to-NavCamera 705. Alternative locations, such as Figure 10C As shown, the recording of the intraoperative posture P in step 706 can be used to obtain the transformation (T) Probe-to-AnatomyTracker The anatomical structure tracker 47 of the tracking system 42 of )707 is used to achieve this.

[0173] Step 610 concludes with the calculation of the initial registration via 4x4 matrix multiplication, where T NavCamera-to-3Dimage =TProbe-to-3Dimage *inv(T Probe-to-NavCamera ), such as Figure 10B The image shows a tracking camera 44, or T. AnatomyTracker to-3Dimage =T Probe-to-3Dimage *inv(T Probe-to-AnatomyTracker ), such as Figure 10C The anatomical structure tracker 47 is shown (step 710). For example... Figure 5A As shown, the initial registration data then enters the registration module 612 as part of the data stream portion of the intraoperative aspect 502 of the ultrasound-based multiple bone registration process 503.

[0174] like Figure 5A As shown, the registration module (step 612) can employ any of a variety of alternative registration processes, such as, for three non-limiting examples, “one-click / one-pose” registration (step 614), “landmark-based” registration (step 616), or “anatomical tracker pin-based” registration (618). Any of these three applications 614, 616, and 618 can establish an initial registration (i.e., a “rough” guess) of the transformation from the anatomical tracker space to the CAD model (CT / MRI) coordinate system. Other alternative registration processes (step 612) that can be part of the registration model can include, for example, “probe-based” registration, “probe-microscan-based” registration, or even utilizing a calibrated digital camera to generate photographs of the patient's anatomy to be registered and estimate the pose and position accordingly.

[0175] Figure 11A This is a flowchart of the registration process performed by the surgical system 100 using "one-click / one-pose" registration (step 614), and Figure 11B and Figure 11C yes Figure 11A A graphical depiction of the process. For example... Figure 11A and Figure 11B As shown, the process (step 614) begins by using the surgical technique... Figure 1 , Figure 10B and Figure 10A The tracking probe 57 generates two point clouds 620 and 622 to record bone surface points of the femur 11, tibia 10, and optionally the patella (step 624). In this step 624, one point cloud (i.e., the femoral tracker relative point cloud (“FTRPC”) 620) is relative to... Figure 1 (Step 624) Two point clouds 620 and 622 are obtained, one for the femoral tracker 47 and the other for the tibial tracker 46 (i.e., the tibial tracker relative point cloud (“TTRPC”) 622 is obtained relative to the tibial tracker 46). In other words, essentially two point clouds 620 and 622 are obtained, one for the femoral tracker 47 and the other for the tibial tracker 46.

[0176] from Figure 11B It can be understood that FTRPC 620 will have femoral data point 620F, patellar data point 620P, and tibial data point 620T, although none of these data points are so identified and only represent the data points of the entire FTRPC 620. Similarly, TTRPC 622 will have femoral data point 622F, patellar data point 622P, and tibial data point 622T, although none of these data points are so identified and only represent the data points of the entire TTRPC 622.

[0177] like Figure 11A and Figure 11B As shown, the "one-click / one-pose" registration process (step 614) continues by applying a classification algorithm to FTRPC 620 and TTRPC 622 intraoperatively to output a femoral point cloud only relative to the femoral tracker ("FOPCRFT") 626 and a tibial point cloud only relative to the tibial tracker ("TOPCRTT") 628 (step 630). This separation of point clouds is advantageous because the knee pose in the preoperative 3D CAD bone model may differ from the knee pose during surgery.

[0178] from Figure 11A and Figure 11C It is understandable that after step 630, the "one-click / one-pose" registration process (step 614) continues by calculating the registration transformations 632 and 634 of the femur 11 and tibia 10 intraoperatively (step 636). This registration transformation can be calculated using a registration algorithm (e.g., "Iterative Closest Point"). For example, matching bone surface points of FOPCRT 626 to the femoral CAD model 112 yields femoral registration (T...). FemurTracker -to-CAD Femur Model (Step 638). Similarly, the bone surface points of TOPCRTT 628 are matched to the tibial CAD model 114 to obtain tibial registration (T TibiaTracker -to-CAD Tibia Model (Step 640). This “one-click / one-pose” registration in step 614 is beneficial because it at least partially facilitates the intraoperative ultrasound surface capture process in step 512, in which multiple bone surfaces are simultaneously acquired via ultrasound probe 55.

[0179] Figure 12 This is a flowchart (step 616) of the process by which the surgical system 100 performs registration using landmark-based registration. For example... Figure 12As shown, the process (step 616) begins by generating a first set of points (step 750) by defining the XYZ coordinates of three or more anatomical landmarks on each 3D CAD model of the bone in the CAD model space. In this example, the surgery is performed in the context of knee arthroplasty, and the bones are the tibia 10 and femur 11 (see...). Figure 1 Step 750 generates a first set of points by defining the XYZ coordinates of three or more anatomical landmarks on each of the 3D CAD femoral model 112 and the 3D CAD tibia model 114. This step 750 can be performed preoperatively or intraoperatively.

[0180] Following step 750, a second set of points is generated by digitizing intraoperative anatomical landmarks via navigation probe 57 to obtain the XYZ coordinates in anatomical structure tracker space for each anatomical landmark defined in step 750 (step 755). In other words, for step 755, the second points are intraoperatively digitized at landmarks on the actual tibia 11 and femur 12, corresponding to those landmarks defined on the corresponding 3D CAD tibia model 114 and 3D CAD femur model 112, respectively. For the final aspect of step 616, the first and second set of points are matched to each other using a classic point-to-point matching algorithm to achieve initial registration for each bone (step 760). In an alternative embodiment, the point-to-point algorithm of step 760 can be replaced by a point-to-surface algorithm, where the preoperative 3D CAD femur model 112 and 3D CAD tibia model 114 are surface models instead of point clouds.

[0181] In short, it can be said Figure 12 The landmark-based registration 616 described herein includes the digitization of landmarks in both CAD space (preoperative planning) and anatomical tracker space (intraoperative). When using landmark-based registration 616 (see...), Figure 5A When these two landmark sets are sufficient to calculate the initial registration completed by the registration module 612, the initial registration is sufficient.

[0182] Figure 13A This is a flowchart (step 618) of the process by which the surgical system 100 performs registration using registration based on anatomical structure tracker pins. For example... Figure 13A and Figure 13B As shown, the process (step 618) begins with installing pins for rigidly attaching the anatomical trackers 46, 47 to the patient's bones (e.g., the tibia 10 and femur 11 in the context of this example, for knee arthroplasty), with the pins installed in a consistent, repeatable manner for each type of surgical procedure (step 770). Then, the navigation system 42 ( Figure 1Track the anatomical structure trackers 46 and 47 to obtain a rough estimate of the position of the patient's bones 10 and 11 relative to the anatomical structure trackers 46 and 47 (step 772).

[0183] from Figure 13A and Figure 13B Understandably, for the final aspect of step 618, the knee joint 762 is positioned and its degrees of freedom are determined (step 774). In doing so, the patient's femur 11 and tibia 10 ( Figure 1 During the procedure, the points 764 are hinged relative to each other around the knee joint 762, causing the second point cloud 764 (which references trackers 46, 47 mounted on the patient's tibia 10 and femur 11) to deflect such that the tibial portion 766 and the femoral portion 768 of the point cloud 764 are hinged relative to each other at the knee joint 762. This hinge is then transformed into a femoral model 112 and a tibial model 114.

[0184] In conclusion, it can be said that Figure 13A and Figure 13B The anatomical tracker pin-based registration 618 described in [reference 618] employs some assumptions about the typical placement of the anatomical tracker and how the knee is flexed (e.g., the anatomical tracker pin is mounted on the anterior midline with the knee in a mid-flexion position (e.g., 30° to 70°)). Using this knowledge, the location and orientation of the knee joint center relative to the anatomical tracker pin in 3D space can be roughly estimated. Ultimately, this location and orientation are used to define the initial registration transformation when applying the anatomical tracker pin-based registration 618 (see [reference 618]). Figure 5A When this is done, it is sufficient to calculate the initial registration completed by the registration module 612.

[0185] like Figure 5A As shown, once the registration module (step 612) has completed the initial or "coarse guess" registration process through any of the three alternative registration processes (i.e., "one-click / one-pose" registration (step 614), "landmark-based" registration (step 616), or "anatomical tracker pin-based" registration (618)), the registration module outputs the initial or "coarse guess" registration data (e.g., a guess of the transformation from trackers 46, 47 to 3D CAD bone models 111, 112, 113, 114) (step 776).

[0186] iii. Calculation of final multiple bone registration

[0187] from Figure 5A and Figure 5B It is understandable that the initial registration data from step 776 is used together with the classified 3D bone surface point cloud 598 from step 600 to calculate the final multiple bone registration (step 900), as will now be discussed regarding... Figure 14A-14C As discussed, among whichFigure 14A yes Figure 5A The flowchart of step 900 in the middle section. Figure 14B and Figure 14C yes Figure 14A Graphical depictions of various aspects of the process. For example... Figure 14A and Figure 14B As shown, Figure 5A The initial or coarse registration data 902 of step 776 is applied to the reference femoral tracker 47. Figure 5A Step 600 classifies 3D point clouds 598 (step 904). As mentioned above regarding... Figure 5B The discussion and such Figure 14B As shown, the 3D point cloud 598 has a femoral point 602 classified as the femur, a patellar point 604 classified as the patella, and a tibial point 606 classified as the tibia. (As mentioned above...) Figure 5A The discussion in the context of the initial registration module in step 612 and in Figure 14B As depicted, the initial registration data 902 includes a femoral point cloud 912, a patellar point cloud 913 (optional), and a tibial point cloud 914. Each point cloud is generated by any one of the three initial registration processes 614, 616, and 618 of the initial registration module in step 612. Each point cloud 912, 913, and 914 is registered to the applicable 3D CAD bone model 112, 113, and 114 via the initial registration model in step 612 and output from it according to step 776.

[0188] like Figure 14AAs shown, the final registration process (step 900) continues by iteratively calculating the nearest points of the classified 3D point clouds 602, 604, 606 to the points of the point clouds 912, 913, 914 on the initially registered or transformed 3D CAD bone models 112, 113, 114 (step 920). More specifically, the registration transformation is updated from the CAD model space to the anatomical structure tracker coordinate system by matching the points of the classified 3D point clouds 602, 604, 606 with the points of the point clouds 912, 913, 914 on the initially registered or transformed 3D CAD bone models 112, 113, 114 (step 925). After step 925, the classification of the points in the classified 3D point clouds generated in step 925 is updated based on point-to-nearest-point distance analysis, penalizing any bone-to-bone interference (step 930). Then, a check is performed to determine whether convergence has been achieved between the classified 3D point clouds 602, 604, 606 and the point clouds 912, 913, 914 on the 3D CAD bone models 112, 113, 114 (step 935). If convergence has not been achieved, the final multi-bone registration calculation returns to step 920 from the convergence check in step 935. If convergence has been achieved, the final registration is completed (step 940), and the converged initial or coarse registration data (e.g., 3D CAD bone models 112, 113, 114 and the point clouds 912, 913, 914 on them) are finally registered with the classified 3D point clouds 602, 604, 606. The two sets of point clouds 912, 913, 914 and 602, 604, 606 are respectively matched with each other and generally extend together with respect to the 3D CAD bone models 112, 113, 114, as shown in step 935. Figure 14C As shown. Figure 2 As shown, according to Figure 5A Step 900 completes the final registration, and the surgical system 100 and procedure can then proceed from registration (step 805) to navigation (step 805), etc., using the final registration data from step 805 as needed throughout the entire process of performing surgery on the patient through the surgical system 100.

[0189] As mentioned above Figure 14A-14C Discussion Figure 5AIn one embodiment, the final registration process (900) proceeds by iteratively calculating the nearest points of the classified 3D point clouds 602, 604, 606 to the triangular mesh bone surfaces on the initially registered or transformed 3D CAD bone models 112, 113, 114. More specifically, the registration transformation is updated from the CAD model space to the anatomical tracker coordinate system by matching the points of the classified 3D point clouds 602, 604, 606 to the triangular mesh bone surfaces on the initially registered or transformed 3D CAD bone models 112, 113, 114. The final registration process 900 then runs an optimization algorithm where the cost function is minimized. In one embodiment, the cost function depends on the registration matrix that is present at the time and is a weighted sum of the following different terms: (1) for each point in the classified 3D point clouds 602, 604, 606, the minimum distance to the nearest triangular mesh bone surface on the initially registered or transformed 3D CAD bone models 112, 113, 114 after the application of the registration matrix; (2) for each point in the classified 3D point clouds 602, 604, 606, a fixed penalty is assigned to 3D CAD bone models 112, 113, 114 that do not match their initial guess to prevent 3D CAD bone models 112, 113, 114 from being swapped; (3) for each point or location on the triangular mesh surface of each 3D CAD bone model 112, 113, 114, if the point or location is located on another 3D CAD bone model 112, 113, 114. Within the CAD bone models 112, 113, and 114, a fixed penalty is applied to avoid bone conflicts; (4) For each degree of freedom seeking registration, a penalty term of the translation / registration magnitude is applied on top of the initial registration, assuming the initial registration is accurate enough that deviation from it is not necessary. In one embodiment, bone assignment is implicitly computed in the first step, so there is no need to explicitly alternate between point cloud assignment optimization and transformation optimization, which might be necessary when using an iterative nearest-point algorithm.

[0190] The final registration process 900 continues to check whether convergence has been achieved between the classified 3D point clouds 602, 604, 606 and the triangular mesh bone surfaces on the 3D CAD bone models 112, 113, 114. If convergence has not been achieved, the final multi-bone registration calculation returns from the convergence check to iteratively calculate the nearest points of the classified 3D point clouds 602, 604, 606 to the triangular mesh bone surfaces on the initially registered or transformed 3D CAD bone models 112, 113, 114, and continues the remainder of the above process until convergence is checked again.

[0191] If convergence has been achieved, the final registration is complete. The initial or coarse registration data (e.g., the triangular mesh bone surfaces of 3D CAD bone models 112, 113, and 114) are finally registered with the classified 3D point clouds 602, 604, and 606. Point clouds 912, 913, and 914 are matched with the corresponding regions of the triangular mesh bone surfaces of 3D CAD bone models 112, 113, and 114, respectively, and roughly extend together. Similarly, as Figure 2 As shown, according to Figure 5A Step 900 completes the final registration, and the surgical system 100 and procedure can then proceed from registration (step 805) to navigation (step 805), etc., using the final registration data from step 805 as needed throughout the entire process of performing surgery on the patient through the surgical system 100.

[0192] The registration process disclosed in this paper has the advantage that consistent registration of individual bones can be non-mandatory, resulting in no overlapping bones in the obtained registrations. Furthermore, the process is flexible and user-friendly, providing a faster workflow because medical professionals do not need to avoid scanning more than one bone. The process is also not adversely affected by outliers in other bones. Therefore, when registering only one bone, users do not need to avoid accidentally scanning another nearby bone.

[0193] Finally, the registration process disclosed herein is advantageous because it is independent of the incision size of the procedure, unlike registration processes known in the art. This is particularly helpful for hip and shoulder procedures, and even more so for ankle procedures, where the incisions are very small, making it difficult to approach the relevant bone structures using typical digital tools (navigation pointers, sharp probes, etc.). Ultrasound advantageously allows access to virtually all bone structures throughout the entire bone.

[0194] Furthermore, the registration process disclosed herein is advantageous because it is not limited to fully robotic or robot-assisted applications. Specifically, the registration process can also be used in any navigation surgery employing preoperative imaging. For example, the registration process can be used as part of a navigation-guided cutting fixture application, navigation-guided ACL reconstruction, or even a navigation procedure to remove osteosarcoma.

[0195] IV. Registration system for validating surgical targets

[0196] A continued high level of concern remains regarding minimizing the risk of performing surgical procedures on the wrong side of a patient's body, for example, performing arthroplasty on the right knee when surgery should be performed on the left knee. Therefore, a registration system 1500 is needed that can be used to quickly confirm or verify that the surgical team will be performing the procedure on the correct target before the surgical team takes any significant steps during the procedure.

[0197] Figure 15 This is a schematic diagram of the registration system 1500. (Example)Figure 15 As shown, the registration system 1500 includes a navigation or tracking system 42, a computer 50, and registration tools 55, 57. The navigation or tracking system 42 includes a detection device 44 for tracking the registration tools 55, 57, and the computer 50 includes an input device and a display 56. The registration tools may be in the form of a tracked ultrasound probe 55 and / or a tracked pen 57, which can be used to image and / or touch certain anatomical landmarks of the patient near the surgical target 1502 in preoperatively generated images that register the patient's anatomy to a patient-specific model and / or patient's anatomy. All these components of the registration system 1500 are configured to work in accordance with the above description. Figure 1 The same components of the surgical system 100 function.

[0198] The navigation or tracking system 42 tracks the registration tools 55, 57 used in the registration of the patient's surgical target 1502 to verify that the surgical target is correct. Figure 15 In this context, the surgical target is the patient's knee, but it can also be the shoulder, elbow, hip, ankle, spine, etc.

[0199] In operation, the registration system 1500 can be used as a precursor to robotic or robot-assisted surgery, which is... Figure 1 The aforementioned surgical system 100 is used. Similarly, the registration system 1500 can be used as a precursor to conventional non-robotic surgery. In either case, medical personnel can use the registration system 1500 on the patient preoperatively to correctly identify the intended surgical target 1502. For example, in the case of knee arthroplasty or other arthroplasty, the registration system 1500 is used to distinguish the target knee 1502 from another non-target knee by scanning and / or palpating the landmarks of the patient's tibia 10 and / or femur 11 adjacent to the target knee 1502. In the case of spinal procedures, the registration system 1500 can be used to identify the bone boundaries of the vertebrae and identify the appropriate vertebral level as the surgical target. In any case, the registration system is used to determine the correct location of the first and subsequent incisions.

[0200] In one embodiment, preoperative registration for surgical target verification purposes can be performed by keeping the patient's suspected surgical target 1502 stationary and scanning the patient's suspected surgical target 1502 with a tracking ultrasound probe. According to Figure 5AThe method, outlined and described in detail above, involves processing the resulting image using computer 50 and registering it with a preoperative patient-specific image or computer model of the patient's surgical target. If the patient's suspected surgical target 1502 is successfully registered to the preoperative patient-specific image or computer model of the patient's surgical target, it verifies that the patient's suspected surgical target 1502 is indeed the correct surgical target. Robotic, robot-assisted, or conventional surgery can then be performed on the correctly identified surgical target.

[0201] V. Exemplary Computing System

[0202] refer to Figure 16 This document provides a detailed description of an example computing system 1300 having one or more computing units that can implement the various systems and methods discussed herein. The computing system 1300 is applicable to any computer or system used in preoperative planning, registration, and postoperative analysis of arthroplasty procedures, as well as other computing or network devices. It should be appreciated that the specific implementation of these devices can be different possible specific computing architectures, all of which are not specifically discussed herein, but will be understood by those skilled in the art.

[0203] Computer system 1300 may be a computing system capable of executing computer program products to perform computer processes. Data and program files can be input into computer system 1300, which reads the files and executes the programs contained therein. Some components of computer system 1300 are... Figure 16 As shown, it includes one or more hardware processors 1302, one or more data storage devices 1304, one or more memory devices 1308, and / or one or more ports 1308-1310. Furthermore, other elements that those skilled in the art will recognize may be included in the computing system 1300 but are not shown in the diagram. Figure 16 The various components of the computer system 1300 are explicitly described or discussed further herein. These components can be communicated via one or more communication buses, point-to-point communication paths, or... Figure 16 Other communication methods not explicitly described in the text allow for communication between them.

[0204] Processor 1302 may include, for example, a central processing unit (CPU), a microprocessor, a microcontroller, a digital signal processor (DSP), and / or one or more internal cache levels. There may be one or more processors 1302, such that processor 1302 includes a single central processing unit, or multiple processing units capable of executing instructions and performing operations in parallel with each other, which is generally referred to as a parallel processing environment.

[0205] Computer system 1300 can be a conventional computer, a distributed computer, or any other type of computer, such as one or more external computers available through a cloud computing architecture. The techniques described herein are optionally implemented as software stored on data storage device 1304, stored on memory device 1306, and / or communicating via one or more ports 1308-1310, thereby enabling… Figure 16 The computer system 1300 is transformed into a dedicated machine for implementing the operations described herein. Examples of the computer system 1300 include personal computers, terminals, workstations, mobile phones, tablets, laptops, multimedia consoles, game consoles, set-top boxes, etc.

[0206] One or more data storage devices 1304 may include any non-volatile data storage device capable of storing data generated or adopted within the computing system 1300, such as computer-executable instructions for performing computer processes, which may include instructions for both applications and an operating system (OS) for managing various components of the computing system 1300. Data storage device 1304 may include, but is not limited to, disk drives, optical disc drives, solid-state drives (SSDs), flash drives, etc. Data storage device 1304 may include removable data storage media, non-removable data storage media, and / or external storage devices available via wired or wireless network architectures in conjunction with such computer program products, including one or more database management products, network server products, application server products, and / or other additional software components. Examples of removable data storage media include compact disc read-only memory (CD-ROM), digital universal disc read-only memory (DVD-ROM), magneto-optical disk, flash drives, etc. Examples of non-removable data storage media include internal magnetic hard disks, SSDs, etc. One or more memory devices 1306 may include volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and / or non-volatile memory (e.g., read-only memory (ROM), flash memory, etc.).

[0207] A computer program product containing mechanisms for implementing systems and methods according to the currently described techniques may reside in data storage device 1304 and / or memory device 1306, which may be referred to as a machine-readable medium. It should be appreciated that a machine-readable medium may include any tangible, non-transitory medium capable of storing or encoding instructions to perform any one or more operations of this disclosure for machine execution, or capable of storing or encoding data structures and / or modules used by or associated with such instructions. A machine-readable medium may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing one or more executable instructions or data structures.

[0208] In some implementations, computer system 1300 includes one or more ports, such as input / output (I / O) port 1308 and communication port 1310, for communicating with other computing, network, or vehicle devices. It should be understood that ports 1308-1310 can be combined or separated, and computer system 1300 may include more or fewer ports.

[0209] I / O port 1308 can be connected to I / O devices or other devices, through which information is input to or output from computing system 1300. Such I / O devices may include, but are not limited to, one or more input devices, output devices, and / or other devices.

[0210] In one implementation, the input device converts human-generated signals (such as human speech, physical motion, physical touch, or pressure) into electrical signals as input data to the computing system 1300 via I / O port 1308. Similarly, the output device can convert electrical signals received from the computing system 1300 via I / O port 1308 into signals that can be perceived by humans as output, such as sound, light, and / or touch. The input device may be an alphanumeric input device, including alphanumeric and other keys for transmitting information and / or command selection to the processor 1302 via I / O port 1308. The input device may be another type of user input device, including but not limited to: directional and selection control devices, such as a mouse, trackball, cursor arrow keys, joystick, and / or scroll wheel; one or more sensors, such as a camera, microphone, position sensor, orientation sensor, gravity sensor, inertial sensor, and / or accelerometer; and / or a touch-sensitive display (“touchscreen”). The output device may include, but is not limited to, a monitor, touchscreen, speaker, haptic and / or haptic output device, etc. In some implementations, the input device and the output device can be the same device, for example, in the case of a touchscreen.

[0211] In one implementation, communication port 1310 is connected to a network through which computer system 1300 can receive network data useful in performing the methods and systems described herein, as well as transmit information determined therefrom and network configuration changes. In other words, communication port 1310 connects computer system 1300 to one or more communication interface devices configured to send and / or receive information between computing system 1300 and other devices via one or more wired or wireless communication networks or connections. Examples of such networks or connections include, but are not limited to, Universal Serial Bus (USB), Ethernet, Wi-Fi, etc. Near Field Communication (NFC), Long Term Evolution (LTE), etc. One or more such communication interface devices can be used through communication port 1310 to communicate directly with one or more other machines via point-to-point communication paths, via wide area networks (WANs) (e.g., the Internet), via local area networks (LANs), via cellular networks (e.g., 3G or 4G) networks, or other communication methods. Furthermore, communication port 1310 can communicate with antennas or other links used for transmitting and / or receiving electromagnetic signals.

[0212] In the example implementation, patient data, bone models (e.g., generic, patient-specific), transformation software, registration software, implant models, and other software, modules, and services can be embodied by instructions stored on data storage device 1304 and / or memory device 1306 and executed by processor 1302. Computer system 1300 may be integrated with surgical system 100 or otherwise form part of surgical system 100.

[0213] Figure 16 The system described herein is merely one possible example of a computer system that can be adopted or configured according to various aspects of this disclosure. It should be appreciated that other non-transitory tangible computer-readable storage media may be used to store computer-executable instructions for implementing the currently disclosed techniques on a computing system.

[0214] In this disclosure, the methods disclosed herein, for example, Figure 5A-14C The methods illustrated herein can be implemented as a device-readable instruction set or software. Furthermore, it should be understood that the specific order or hierarchy of steps in the disclosed methods is an example of an exemplary method. Based on design preferences, it is understood that the specific order or hierarchy of steps in a method can be rearranged while remaining within the scope of the disclosed subject matter. The appended method claims present the elements of each step in an exemplary order and are not necessarily limited to the specific order or hierarchy presented.

[0215] The disclosures described herein, including any methods described herein, may be provided as computer program products or software, which may include a non-transitory machine-readable medium on which instructions are stored, instructions which can be used to program a computer system (or other electronic device) to perform processes according to this disclosure. Machine-readable media include any mechanism for storing information in a machine-readable form (e.g., software, processing application). Machine-readable media may include, but are not limited to, magnetic storage media, optical storage media; magneto-optical storage media, read-only memory (ROM); random access memory (RAM); erasable programmable memory (e.g., EPROM and EEPROM); flash memory; or other types of media suitable for storing electronic instructions.

[0216] While this disclosure has been described with reference to various implementations, it should be understood that these implementations are illustrative and the scope of this disclosure is not limited thereto. Many variations, modifications, additions, and improvements are possible. More generally, embodiments according to this disclosure have been described in the context of specific implementations. Functions described in the various embodiments of this disclosure or in different terms may be differently separated or combined in blocks. These and other variations, modifications, additions, and improvements may fall within the scope of the disclosure as defined in the appended claims.

[0217] In summary, while the embodiments described herein are described with reference to specific embodiments, modifications may be made thereto without departing from the spirit and scope of this disclosure. It should also be noted that the term "comprising" as used herein is intended to include, i.e., "including but not limited to".

[0218] The construction and arrangement of the systems and methods illustrated in the various exemplary embodiments are merely illustrative. Although only a few embodiments are described in detail in this disclosure, many modifications are possible (e.g., variations in the size, dimensions, structure, shape and scale, parameter values, mounting arrangements, use of materials, color, orientation, etc. of various elements). For example, the positions of elements may be reversed or otherwise varied, and the nature or number of discrete elements or positions may be altered or varied. Therefore, all such modifications are intended to be included within the scope of this disclosure. According to alternative embodiments, the order or sequence of any process or method steps may be changed or reordered. Other substitutions, modifications, alterations, and omissions may be made in the design, operating conditions, and arrangements of the exemplary embodiments without departing from the scope of this disclosure.

Claims

1. A surgical system configured to process an ultrasound image of a patient bone, the ultrasound image comprising a bone surface of each of the patient bone, the system comprising: at least one surgical tool; and a computing device comprising a processing device and a computer readable medium having stored thereon one or more executable instructions, the processing device configured to execute the one or more executable instructions, the one or more executable instructions comprising: i) detecting a bone surface of each of the patient bone in the ultrasound image as an ultrasound image pixel; ii) converting the ultrasound image pixel to a three-dimensional point; iii) generating a classified three-dimensional point cloud of the bone surface of each of the patient bone; iv) acquiring a point cloud by three-dimensional position tracking of a tracker mounted on the patient bone; and v) final registration of a bone model and a point cloud of the patient bone to the classified three-dimensional point cloud of each bone surface, the point cloud acquired by three-dimensional position tracking of a tracker mounted on the patient bone and registered to the bone model, wherein the point cloud acquired by three-dimensional position tracking of a tracker mounted on the patient bone and registered to the bone model matches the classified three-dimensional point cloud of each bone surface of the patient bone, wherein the at least one surgical tool is in communication with the computing device and the final registration result is inputted to navigate the at least one surgical tool relative to each of the patient bone. The detection of the bone surface occurs by an image processing algorithm forming at least a portion of the one or more executable instructions.

2. The system of claim 1, wherein, The image processing algorithm comprises a machine learning model.

3. The system of claim 2, wherein, 4. A method of processing an ultrasound image of a patient bone, the method performed by a computing device comprising a processing device executing one or more executable instructions and a computer readable medium for storing the one or more executable instructions, the ultrasound image comprising a bone surface of each of the patient bone, the method comprising: detecting a bone surface of each of the patient bone in the ultrasound image as an ultrasound image pixel; generating a classified three-dimensional point cloud of the bone surface of each of the patient bone; acquiring a point cloud by three-dimensional position tracking of a tracker mounted on the patient bone; and final registration of a bone model and a point cloud of the patient bone to the classified three-dimensional point cloud of each bone surface, the point cloud acquired by three-dimensional position tracking of a tracker mounted on the patient bone and registered to the bone model, wherein the point cloud acquired by three-dimensional position tracking of a tracker mounted on the patient bone and registered to the bone model matches the classified three-dimensional point cloud of each bone surface of the patient bone, and using the final registration result to navigate at least one surgical tool relative to each of the patient bone. The detection of the bone surface occurs by an image processing algorithm. The image processing algorithm comprises a machine learning model.

5. The method of claim 4, wherein, ​ 6. The method of claim 5, wherein, ​

Images