Ultrasound-based multiple bone registration surgical systems and methods of use in computer-assisted surgery

Abstract

To provide a surgical system for surgical registration of patient bones to a surgical plan, where, as part of the ability of the surgical system to perform the surgical registration, the system is configured to process an ultrasound image of the patient bones, the ultrasound image including a bone surface for each of the patient bones.SOLUTION: The system includes a computing device including a processing device and a computer-readable medium with one or more executable instructions stored thereon. The processing device is configured to execute the one or more executable instructions. The one or more executable instructions are to: i) detect the bone surface of each of the patient bones in the ultrasound image; and ii) segregate a first point cloud of ultrasound image pixels associated with the bone surface of each of the patient bones.SELECTED DRAWING: Figure 5A

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 105,973, filed on October 27, 2020, the entire disclosure of which is hereby incorporated by reference into this application.

[0002] This disclosure relates to medical systems and methods in computer - assisted surgery. More specifically, this disclosure relates to a surgical positioning system and method in computer - assisted surgery.

Background Art

[0003] Recent orthopedic joint replacement surgeries typically involve at least some preoperative planning of the surgery to enhance the effectiveness and efficiency of a particular procedure. In particular, preoperative planning can increase the accuracy of bone resection and implant placement while reducing the overall time of the procedure and the time the patient's joint is open and exposed.

[0004] The use of robotic systems during the execution of orthopedic joint replacement surgery can significantly reduce the intraoperative time of a particular procedure. Moreover, the effectiveness of the procedure can be based on the tools, systems, and methods utilized during the preoperative planning phase.

[0005] Examples of steps involved in preoperative planning can include determining implant size, position, and orientation, resection planes and depths, the trajectory to access the surgical site, and other things. In certain cases, preoperative planning can involve generating a three - dimensional (“3D”) patient - specific model of the bones and soft tissues of a patient undergoing joint replacement. The 3D patient model can be used as a visual aid in planning various possibilities for implant size, implant orientation, implant position, and corresponding resection planes and depths, among other parameters.

[0006] However, before a robotic system can perform an arthroplasty, it is necessary to position the robotic system and the navigation system with respect to the patient. Since positioning involves mapping virtual boundaries and constraints defined in a preoperative plan onto the physical space with respect to the patient, the robotic system can be accurately tracked with respect to the patient and constrained with respect to the boundaries applied to the patient's anatomical structure.

[0007] While frameworks for particular aspects of surgical positioning may be known in the art, there is a need for systems and methods that further improve particular aspects of positioning to further enhance the efficiency and effectiveness of robotic and robot-assisted orthopedic arthroplasty surgeries. SUMMARY OF THE INVENTION

[0008] Aspects of the present disclosure may include one or a combination of various neural networks trained to detect and optionally classify bone surfaces in ultrasound images.

[0009] Aspects of the present disclosure may also simultaneously and commonly position the bone surfaces of N bones (typically forming a joint) between an ultrasound modality and a second modality (e.g., CT / MRI, or surface reconstruction employing one or more statistical / general models deformed according to the patient's anatomical data), and optimize at least Nx6 degrees of freedom of transformation from the ultrasound modality to the second modality, and classification information that assigns a region within the image data of the ultrasound modality to one of the N captured bones, to capture the bone.

[0010] In certain cases, the ultrasound probe is tracked with respect to an anatomical structure tracker attached to each of the N captured bones to assemble individual ultrasound images (capturing only slices / small portions of the bone) into one consistent 3D image data set. If the scanned bone is fixed, the ultrasound images can be assembled by tracking only the ultrasound probe.

[0011] In certain cases, the plurality of positionings can be 3D point cloud / mesh-based. In such a situation, the N bones can have a triangulated mesh obtained and segmented in a second modality (segmentation of the bones of the CT / MRI).

[0012] In certain cases, the plurality of positionings can be image-based. In such a situation, the classified ultrasound data is directly matched with the second modality without the need to detect the bone surface of the ultrasound image.

[0013] Aspects of the present disclosure can include a system for surgical positioning of a patient's bones for a surgical plan, the surgical positioning employing ultrasound images of the patient's bones, the ultrasound images including individual ultrasound images that include bone surfaces of a plurality of bones, and the individual ultrasound images being generated from an ultrasound scan resulting from a single swath of an ultrasound probe across the patient's bones. In such a system, the system includes a computing device including a processing device and a computer-readable medium storing one or more executable instructions. The processing device is configured to execute the one or more instructions. The one or more executable instructions include i) detecting bone surfaces within the individual ultrasound images and ii) classifying each of the bone surfaces within the individual ultrasound images according to bone type, including one or more neural networks trained to reach the classified bone surfaces.

[0014] The one or more neural networks can include a convolutional network that detects bone surfaces within the individual ultrasound images. Depending on the embodiment, the one or more neural networks can include a pixel classification network and / or a likelihood classification network that classifies each of the bone surfaces.

[0015] The system receives individual ultrasonic images having the bone surfaces of a plurality of bones, and then advantageously can: i) detect the bone surfaces within the individual ultrasonic images and ii) classify each of the bone surfaces according to its type of bone such that the classified bone surfaces can be reached. In other words, the system can still detect and classify the bone surfaces even though the individual ultrasonic images contain the bone surfaces of a plurality of bones. Advantageously, this ability enables generating individual ultrasonic images from an ultrasonic scan resulting from a single swath of an ultrasonic probe across the bones of a patient forming a joint. Thus, due to this ability, there is no need to limit the swath of the ultrasonic probe across the bones of a joint patient to a single bone, and the swath can simply span across all the bones of the joint such that the resulting ultrasonic images contain a plurality of bones, and the system can detect and classify the bone surfaces such that the bone surfaces are classified by the system.

[0016] In one version of the system, the processing device executes one or more instructions to calculate the transformation of the 2D image pixels of the classified bone surfaces of the individual ultrasonic images to 3D points, thereby generating a classified 3D bone surface point cloud. Depending on the embodiment, the propagation speed of ultrasonic waves in a particular medium may be considered when calculating the transformation of the 2D image pixels of the classified bone surfaces of the individual ultrasonic images to 3D points, the known set of poses of the ultrasonic probe may be obtained with respect to a probe tracker in relation to the ultrasonic probe coordinate system, and the transformation may be calculated between the probe tracker space and the ultrasonic probe coordinate system.

[0017] In one version of the system, the processing device executes one or more instructions to calculate an initial or rough positioning of the patient's bones into a computer model of the patient's bones.

[0018] In one embodiment of the system, when calculating an initial or approximate positioning of a patient's bone onto a computer model of the patient's bone, a first point cloud and a second point cloud are generated by the system, the first point cloud being related to a first bone of the patient's bone and being for a first tracker associated with the first bone, and the second point cloud being related to a second bone of the patient's bone and being for a second tracker associated with the second bone. Thus, in the context of a patient's knee, the first point cloud is related to the femur of the patient's bone and is for a first tracker fixed to the femur, and the second point cloud is related to the tibia and is for a second tracker fixed to the tibia. When calculating an initial or approximate positioning of a patient's bone onto a computer model of the patient's bone, the system matches the bone surface points of the first point cloud onto the computer model of the first bone and the bone surface points of the second point cloud onto the computer model of the second bone.

[0019] In other embodiments of the system, when calculating an initial or approximate positioning of a patient's bone onto a computer model of the patient's bone, the system may employ landmark-based positioning and / or tracker pin-based positioning of anatomical structures.

[0020] In one version of the system, the processing device executes one or more instructions to calculate a final positioning of a plurality of bones by employing an initial or approximate positioning and a classified 3D bone surface point cloud, the final positioning of the plurality of bones achieving convergence between the classified 3D bone surface point cloud and the patient's bones. Depending on the embodiment, when calculating a final positioning of a plurality of bones where there is convergence between the classified 3D bone surface point cloud and the patient's bones, the system may apply an initial or approximate positioning to the classified 3D bone surface point cloud with reference to the first tracker. In the context of a knee joint, the first tracker may be attached to the femur.

[0021] In accordance with an embodiment, when calculating the classified 3D bone surface point cloud and the final positioning of a plurality of bones where convergence exists between the bones of a patient, the system iteratively calculates the closest points of the classified 3D surface point cloud to the computer model of the patient's bones.

[0022] Aspects of the present disclosure may include a method of positioning a plurality of bones of a patient's joint for a surgical plan. In accordance with an embodiment, the method includes receiving an ultrasonic image of the patient's joint in which at least a portion of the ultrasonic image depicts a plurality of bones, employing a convolutional neural network to detect the bone surfaces of the plurality of bones within the ultrasonic image, employing at least one of a likelihood classifier network or a pixel classifier network to classify each of the bone surfaces according to its type of bone to reach the classified bone surfaces, converting the 2D ultrasonic image pixels of the classified bone surfaces into 3D to result in a classified 3D bone surface point cloud, generating an initial rough positioning of the plurality of bones of the patient's joint for a medical image display of the plurality of bones of the patient's joint, and calculating a final positioning of the plurality of bones of the patient's joint for the surgical plan by applying the initial rough positioning to the classified 3D bone surface point cloud.

[0023] In one embodiment, when converting the 2D ultrasonic image pixels of the classified bone surfaces into 3D, the propagation speed of ultrasonic waves in a specific medium may be considered.

[0024] In one embodiment, when converting the 2D ultrasonic image pixels of the classified bone surfaces into 3D, the 2D ultrasonic image pixels of the classified bone surfaces may be mapped from a 2D pixel space to a 3D metric coordinate system of an ultrasonic probe coordinate system.

[0025] In one embodiment, when converting the 2D ultrasonic image pixels of the classified bone surfaces into 3D, a known set of poses of the ultrasonic probe may be obtained with respect to a probe tracker in relation to the ultrasonic probe coordinate system.

[0026] In one embodiment, when converting the 2D ultrasound image pixels of the classified bone surface into 3D, the conversion can be calculated between the probe tracker space and the ultrasound probe coordinate system.

[0027] In one embodiment, when generating an initial rough positioning of a plurality of bones of a patient's joint for display of a medical image of the plurality of bones of the patient's joint, a first point cloud and a second point cloud can be generated. The first point cloud relates to a first bone of the plurality of bones and is for a first tracker associated with the first bone. The second point cloud relates to a second bone of the plurality of bones and is for a second tracker associated with the second bone.

[0028] In one embodiment, when generating an initial rough positioning of a plurality of bones of a patient's joint for display of a medical image of the plurality of bones of the patient's joint, the bone surface points of the first point cloud can be matched onto the computer model of the first bone, and the bone surface points of the second point cloud are matched onto the computer model of the second bone.

[0029] In one embodiment, when generating an initial rough positioning of a plurality of bones of a patient's joint for display of a medical image of the plurality of bones of the patient's joint, landmark-based positioning can be employed.

[0030] In one embodiment, when generating an initial rough positioning of a plurality of bones of a patient's joint for display of a medical image of the plurality of bones of the patient's joint, tracker pin-based positioning of anatomical structures can be employed.

[0031] In one embodiment, by applying an initial rough positioning to a classified 3D bone surface point cloud, when calculating the final positioning of a plurality of bones of a patient's joint for a surgical plan, the final positioning of the plurality of bones achieves convergence between the classified 3D bone surface point cloud and the patient's bones. In doing so, the initial rough positioning can be applied to the classified 3D bone surface point cloud with reference to a first tracker. Depending on the embodiment, during this final positioning, the algorithm employed converges until its result reaches a steady state, and the algorithm can also improve the classification of the classified 3D bone surface point cloud itself, such that any initial error in the classification can be removed or at least reduced. In achieving these aspects of the final positioning, the classified 3D bone surface point cloud and the initial or rough positioning become fully positioned, resulting in the final positioning of the plurality of bones.

[0032] In one embodiment, by applying an initial rough positioning to a classified 3D bone surface point cloud, when calculating the final positioning of a plurality of bones of a patient's joint for a surgical plan, iterative calculation of the closest points of the classified 3D surface point cloud to the computer model of the patient's bones can be performed.

[0033] Aspects of the present disclosure may include a method for surgical positioning of a patient's bones for a surgical plan. Depending on the embodiment, the method may include receiving an ultrasonic image of the patient's bones, wherein the ultrasonic image includes individual ultrasonic images including the bone surfaces of a plurality of bones, and the individual ultrasonic images are generated from an ultrasonic scan resulting from a single swath of an ultrasonic probe across the patient's bones, and detecting the bone surfaces in the individual ultrasonic images and classifying each of the bone surfaces in the individual ultrasonic images according to the type of bone, employing one or more neural networks trained to reach the classified bone surfaces.

[0034] Aspects of the present disclosure may include a surgical system configured to process ultrasonic images of a patient's bone, the ultrasonic images including the bone surfaces for each of the patient's bones. In one embodiment, the system includes a computing device including a processing device and a computer-readable medium storing one or more executable instructions. The processing device is configured to execute the one or more executable instructions. The one or more executable instructions are to: i) detect the bone surface of each of the patient's bones in the ultrasonic image, and ii) separate a first point cloud of ultrasonic image pixels associated with the bone surface of each of the patient's bones.

[0035] In one version of the embodiment, the detection of the bone surface may occur via an image processing algorithm that forms at least a portion of the one or more executable instructions. The image processing algorithm may include a machine learning model. The separation of the first point cloud may occur via a pixel classification neural network that forms at least a portion of the one or more executable instructions. The separation of the first point cloud may occur via an image-based classification neural network that forms at least a portion of the one or more executable instructions.

[0036] In one version of the embodiment, the processing device may execute one or more executable instructions to calculate a transformation of a first point cloud to a separated 3D point cloud, where the separated 3D point cloud is separated such that each ultrasonic image pixel of the separated 3D point cloud is mutually associated with a corresponding bone surface of a patient's bone. When calculating the transformation of the first point cloud to the separated 3D point cloud, the ultrasonic image pixels may be calibrated to an ultrasonic probe tracker, and the ultrasonic probe tracker is calibrated to a tracking camera. When calibrating the ultrasonic image pixels to the ultrasonic probe tracker, the propagation speed of ultrasonic waves in a particular medium may be considered. When calculating the transformation of the first point cloud to the separated 3D point cloud, the ultrasonic image pixels may be calibrated to an ultrasonic probe tracker, the ultrasonic probe tracker is calibrated to a tracking camera, and the coordinate system is with respect to the bone surface via a tracker of an anatomical structure placed on the bone surface of the patient's bone. The separation of the first point cloud may occur via a geometric analysis of the first point cloud.

[0037] In one version of the embodiment, one or more executable instructions may calculate an initial or approximate positioning of a second point cloud obtained from a patient's bone to a bone model of the patient's bone. The second point cloud may include a plurality of point clouds with respect to a plurality of trackers on the patient's bone. The plurality of point clouds may include one point cloud positioned on a certain bone model of the bone model of the patient's bone and another point cloud positioned on another bone model of the bone model of the patient's bone.

[0038] Initial or coarse positioning may be landmark-based. Initial or coarse positioning may be calculated from the position and orientation of trackers on anatomical structures. When calculating the initial or coarse positioning, a third point cloud and a fourth point cloud may be generated by the system, where the third point cloud relates to a first bone of the patient's bone and is for a first tracker associated with the first bone, and the fourth point cloud relates to a second bone of the patient's bone and is for a second tracker associated with the second bone.

[0039] In one version of the embodiment, when calculating the initial or coarse positioning, the system may align the bone surface points of the third point cloud onto the computer model of the first bone and align the bone surface points of the fourth point cloud onto the computer model of the second bone.

[0040] In one version of the embodiment, the processing device may execute one or more instructions to calculate the final positioning of a plurality of bones using the initial or coarse positioning and the separated 3D point clouds, where the final positioning of the plurality of bones achieves the final positioning between the separated 3D point clouds and the patient's bones. When calculating the final positioning of the plurality of bones where there is a final positioning between the classified 3D bone surface point clouds and the patient's bones, the system may iteratively refine the positioning of the separated 3D point clouds onto the computer model of the patient's bones and iteratively refine the separation of the separated 3D point clouds.

[0041] Aspects of the present disclosure may include a method of processing ultrasonic images of a patient's bones, where the ultrasonic images include the bone surfaces for each of the patient's bones. One embodiment of such a method may include detecting the bone surface of each of the patient's bones in the ultrasonic image and separating a first point cloud of ultrasonic image pixels associated with the bone surface of each of the patient's bones.

[0042] In one version of the embodiment, the detection of the bone surface can occur via an image processing algorithm. The image processing algorithm can include a machine learning model. The separation of the first point cloud can occur via a pixel classification neural network. The separation of the first point cloud can occur via an image-based classification neural network.

[0043] In one version of the embodiment, the method further includes calculating a transformation of the first point cloud to the separated 3D point cloud, where the separated 3D point cloud is separated such that each ultrasonic image pixel of the separated 3D point cloud is mutually associated with a corresponding bone surface of the patient's bone. When calculating the transformation of the first point cloud to the separated 3D point cloud, the ultrasonic image pixels can be calibrated to an ultrasonic probe tracker, and the ultrasonic probe tracker is calibrated to a tracking camera. When calibrating the ultrasonic image pixels to the ultrasonic probe tracker, the propagation speed of ultrasonic waves in a specific medium can be considered.

[0044] In one version of the embodiment, when calculating the transformation of the first point cloud to the separated 3D point cloud, the ultrasonic image pixels can be calibrated to an ultrasonic probe tracker, the ultrasonic probe tracker is calibrated to a tracking camera, and the coordinate system is with respect to the bone surface via a tracker of an anatomical structure disposed on the bone surface of the patient's bone. The separation of the first point cloud can occur via a geometric analysis of the first point cloud.

[0045] In one version of the embodiment, the method further includes calculating an initial or approximate positioning of a second point cloud obtained from a patient's bone onto a bone model of the patient's bone. The second point cloud may include a plurality of point clouds for a plurality of trackers on the patient's bone. The plurality of point clouds may include one point cloud positioned on one bone model of the bone model of the patient's bone and another point cloud positioned on another bone model of the bone model of the patient's bone. The initial or approximate positioning may be landmark-based. The initial or approximate positioning may be calculated from the position and orientation of the trackers of the anatomical structure.

[0046] In one version of the embodiment, when calculating the initial or approximate positioning, a third point cloud and a fourth point cloud may be generated. The third point cloud relates to a first bone of the patient's bone and is for a first tracker associated with the first bone. The fourth point cloud relates to a second bone of the patient's bone and is for a second tracker associated with the second bone. When calculating the initial or approximate positioning, the bone surface points of the third point cloud may be matched onto the computer model of the first bone, and the bone surface points of the fourth point cloud are matched onto the computer model of the second bone.

[0047] In one version of the embodiment, the method further includes calculating a final positioning of a plurality of bones by employing the initial or approximate positioning and the separated 3D point cloud, and the final positioning of the plurality of bones achieves a final positioning between the separated 3D point cloud and the patient's bone. When calculating the final positioning of the plurality of bones where there is a final positioning between the classified 3D bone surface point cloud and the patient's bone, the positioning of the separated 3D point cloud onto the computer model of the patient's bone may be repeatedly improved, and the separation of the separated 3D point cloud is repeatedly improved.

[0048] The patent or application file includes at least one drawing made in color. A copy of the patent or patent application publication having color drawings will be provided by the Office upon payment of the required fee upon request.

Brief Description of the Drawings

[0049]

Figure 1

[0050]

Figure 2

[0051]

Figure 3A

Figure 3B

[0052]

Figure 4A

[0053]

Figure 4B

[0054]

Figure 4C

[0055]

Figure 4D

[0056]

Figure 4E

[0057]

Figure 5A

[0058]

Figure 5B

[0059]

Figure 6A

[0060]

Figure 6B

[0061]

Figure 7A

[0062]

Figure 7B

[0063]

Figure 8A

[0064]

Figure 8B

[0065]

Figure 9A

[0066]

Figure 9B

[0067]

Figure 10A

[0068]

Figure 10B

Figure 10C

[0069]

Figure 11A

[0070]

Figure 11B

Figure 11C

[0071]

Figure 12

[0072]

Figure 13A

[0073]

Figure 13B

[0074]

Figure 14A

[0075]

Figure 14B

Figure 14C

[0076]

Figure 15

[0077]

Figure 16

DETAILED DESCRIPTION OF THE INVENTION

[0078] This application claims the benefit of the following applications: International Application No. PCT / US2017 / 049466, filed August 30, 2017, entitled "System and Method for Intraoperative Pelvic Positioning"; International Application No. PCT / US2016 / 034847, filed May 27, 2016, entitled "Preoperative Planning and Associated Intraoperative Positioning for Surgical Systems"; U.S. Patent Application No. 12 / 894,071, filed September 29, 2010, entitled "Surgical System for Placing Prosthetic Components and / or Constraining Movement of Surgical Tools"; U.S. Patent Application No. 13 / 234,190, filed September 16, 2011, entitled "System and Method for Measuring Parameters in Arthroplasty Surgery"; U.S. Patent Application No. 11 / 357,197, filed February 21, 2006, entitled "Haptic Guidance System and Method"; U.S. Patent Application No. 12 / 654,519, filed December 22, 2009, entitled "Transmission Using First and Second Transmission Elements"; U.S. Patent Application No. 12 / 644,964, filed December 22, 2009, entitled "Devices Assemblable by Connection"; and U.S. Patent Application No. 11 / 750,807, filed May 18, 2007, entitled "System and Method for Verifying Calibration of Surgical Devices."

[0079] A surgical positioning system and method of use are disclosed herein in connection with a surgical system 100. Surgical positioning, for example, requires mapping a virtual boundary determined in a preoperative plan at a working boundary in a physical space. A surgical robot may be permitted to perform certain operations within the virtual boundary, such as drilling a hole or excising a bone surface. Once the virtual boundary is mapped to the patient's physical space, the robot may drill a hole or excise a bone surface at a planned location and orientation, but may be constrained from performing such an operation outside the preplanned virtual boundary. Accurate and precise positioning of the patient's anatomical structure enables accurate navigation of the surgical robot during a surgical procedure. The need for accuracy and precision in the positioning process must be balanced with the time required to perform the positioning.

[0080] In the case of robotic-assisted surgery, the virtual boundary can be defined in the preoperative plan. In the case of a complete robotic surgery, the virtual tool path can be defined in the preoperative plan. In either case, the preoperative plan may include, for example, determining the bone resection depth and identifying whether an unacceptable notch in the anterior femoral cortex is associated with the proposed bone resection depth and proposed orientation of the candidate implant. Assuming that there is no unacceptable notch in the anterior femoral cortex for the preoperatively planned bone resection depth and implant orientation and that it is approved by the surgeon, the bone resection depth can be updated to account for the cartilage thickness by positioning the cartilage surface of the actual patient's bone on the patient's bone model employed in the preoperative plan during the surgery. By taking the cartilage thickness into account in this way, the actual implant, when implanted via the surgical system 100, has respective surfaces arranged to function in place of the resected cartilage surface of the actual patient's bone. Further explanation of the preoperative plan can be found in the specification of international application PCT / US2016 / 034847, filed on May 27, 2016, and titled "Preoperative Planning and Associated Intraoperative Positioning for a Surgical System", the entirety of which is incorporated herein by reference.

[0081] Before starting a detailed discussion of surgical positioning, a schematic of the surgical system and its operation is given here as follows.

[0082] I. Schematic of the Surgical System

[0083] To begin a detailed discussion of the surgical system, reference is made to FIG. 1. As can be understood from FIG. 1, the surgical system 100 includes a navigation system 42, a computer 50, and a haptic device 60 (also referred to as a robotic arm 60). The navigation system tracks the patient's bones (i.e., tibia 10, femur 11) and the surgical tools (e.g., pointer device, probe, cutting tool) utilized during the surgery to enable the surgeon to visualize the bones and tools on the display 56 during the bone cutting procedure.

[0084] Navigation system 42 can be any type of navigation system configured to track the pose (i.e., position and orientation) of a bone. For example, navigation system 42 can include a non-mechanical tracking system, a mechanical tracking system, or any combination of non-mechanical and mechanical tracking systems. Navigation system 42 includes a detection device 44 that obtains the pose of an object relative to a reference coordinate frame of the detection device 44. As the object moves within the reference coordinate frame, the detection device tracks the pose of the object to detect the movement of the object.

[0085] In one embodiment, the navigation system 42 includes a non-mechanical tracking system as shown in FIG. 1. The non-mechanical tracking system is an optical tracking system having a detection device 44 and trackable elements (e.g., navigation markers 46, 47) respectively disposed on objects to be tracked (e.g., the patient's tibia 10 and femur 11) and detectable by the detection device 44. In one embodiment, the detection device 44 includes a visible light-based detector such as a MicronTracker (Claron Technology Inc., Toronto, Canada) that detects a pattern (e.g., a checkerboard pattern) on the trackable element. In another embodiment, the detection device 44 includes a stereo camera pair that responds to infrared radiation and can be disposed in the operating room where the arthroplasty procedure is performed. The trackable elements are attached to the objects to be tracked in a fixed and stable manner and include an array of markers having known geometric relationships to the objects to be tracked. As is known, the trackable elements can be active (e.g., light-emitting diodes, i.e., LEDs) or passive (e.g., reflective spheres, checkerboard patterns, etc.), and can have a unique shape (e.g., the unique geometric arrangement of the markers), or in the case of active wired or wireless markers, a unique emission pattern. During operation, the detection device 44 detects the positions of the trackable elements, and the surgical system 100 (e.g., the detection device 44 using implanted electronics) calculates the posture of the object to be tracked based on the positions, unique shapes, and known geometric relationships of the trackable elements with respect to the object to be tracked. The tracking system 42 includes trackable elements for each object that the user desires to track, such as navigation marker 46 disposed on the tibia 10 and navigation marker 47 disposed on the femur 11.During robot-assisted surgery with a tactile guide, the navigation system may further include a tactile device marker 48 (for tracking the overall or global position of the tactile device 60), an end effector marker 54 (for tracking the distal end of the tactile device 60), and freehand navigation probes 55, 57 for use in a positioning process in the form of a tracked ultrasonic probe 55 and a tracked stylus 57 having a sharp tip for touching specific relevant anatomical landmarks in the patient and specific positioning locations in parts of the system 100. Additionally or alternatively, the system 100 may employ electromagnetic tracking.

[0086] The systems and methods disclosed herein are provided in the context of a robot-assisted surgical system employing the above-described navigation system, such as that employed by Stryker's Mako surgical robot, for example, but the present disclosure is readily applicable to other surgical systems that are navigated. For example, additionally or alternatively, the systems and methods disclosed herein may be applied to surgical procedures for creating bone using a jig for arthroplasty that is navigated, such as in the context of Stryker's eNact Knee Navigation software, for example. Similarly, additionally or alternatively, the systems and methods disclosed herein may be applied to surgical procedures for creating bone using a saw or a hand-held robot that is navigated.

[0087] As shown in FIG. 1, the surgical system 100 further includes a processing circuit represented in the figure as computer 50. The processing circuit includes a processor and a memory device. The processor may be implemented as a general-purpose processor, an application-specific integrated circuit (ASIC), one or more field-programmable gate arrays (FPGAs), a group of processing components, an application-specific processor, or other suitable electronic processing components. The memory device (e.g., memory, memory unit, storage device, etc.) is one or more devices (e.g., RAM, ROM, flash memory, hard disk storage, etc.) that store data and / or computer code for completing or facilitating the various processes, layers, and functions described in this application. The memory device may be volatile memory or non-volatile memory, or may include volatile memory or non-volatile memory. The memory device may include database components, object code components, script components, or any other type of information structure that supports various operations and the information structures described in this application. According to a preferred embodiment, the memory device is communicatively connected to the processor via the processing circuit and includes computer code for performing one or more processes described herein (e.g., by the processing circuit and / or the processor).

[0088] Computer 50 is configured to communicate with navigation system 42 and haptic device 60. Further, computer 50 may receive information related to orthopedic / arthroplasty procedures and perform various functions related to the execution of osteotomy procedures. For example, computer 50 may have software necessary to perform functions related to image analysis, surgical planning, positioning, navigation, image guidance, and haptic guidance. More specifically, the navigation system may operate in conjunction with an autonomous robot or a surgeon-assisting device (haptic device) when performing an arthroplasty procedure.

[0089] Computer 50 receives an image of the anatomical structure of a patient on whom an arthroplasty procedure is to be performed. Referring to FIG. 2, prior to the performance of an arthroplasty, the anatomical structure of the patient can be scanned (step 801) using any known imaging technique such as CT or MRI captured using a medical imaging machine. The present disclosure refers to medical images captured or generated using a medical imaging machine such as a CT or MRI machine, although other methods of generating medical images are possible and contemplated herein. For example, an image of a bone can be generated intraoperatively via a medical imaging machine such as a hand-held scan or imaging device that scans or positions the topography of the bone surface. As yet another example, anatomical data of a patient obtained from a variety of modalities can be used to generate a medical image by deforming one or more statistical / generalized models according to the anatomical data of the patient, preoperatively or intraoperatively. Accordingly, the term medical imaging machine is intended to encompass devices of various sizes (e.g., C-arm, hand-held device) placed at an imaging center or used intraoperatively, and the term medical image is intended to encompass images, models, or other anatomical display data of a patient useful for planning and performing an arthroplasty procedure.

[0090] Subsequently, the scan data is then segmented to obtain a three-dimensional display of the patient's anatomical structure. For example, prior to performing knee arthroplasty, three-dimensional displays of the femur and tibia are generated. Using the three-dimensional displays, as part of the planning process, landmarks on the femur and tibia can be selected, and the patient's femur-tibia alignment can be calculated with respect to the orientation and placement of the proposed femur and tibia implants selected via computer 50 and with respect to the model and size. The landmarks on the femur and tibia can particularly include the center of the femoral head, the distal trochlear groove, the center of the intercondylar eminence, the tibia-ankle center, and the central tibial spine. The femur-tibia alignment is the angle between the femoral functional axis (i.e., the line from the center of the femoral head to the distal trochlear groove) and the tibial functional axis (i.e., the line from the ankle center to the center of the intercondylar eminence). Based on the patient's current femur-tibia alignment and the desired femur-tibia alignment to be achieved by the arthroplasty procedure, and further including the size, model, and placement of the proposed femur and tibia implants, and including the desired extension, varus-valgus angle, and internal-external rotation associated with the implantation of the proposed implant, computer 50 is programmed to calculate the desired implantation of the proposed implant or at least assist in the preoperative planning (step 803) of the implantation of the proposed implant, including the resection to be generated via haptic device 60 in the process of performing the arthroplasty procedure. The preoperative plan achieved via step 803 is provided to the surgeon for review, adjustment, and approval, and the preoperative plan is updated as directed by the surgeon (step 802).

[0091] The computer 50 is used to develop a surgical plan in accordance with step 803. It is to be understood that the user can interact with the computer 50 at any stage during the surgical plan to input information and modify any part of the surgical plan. The surgical plan may include a plurality of virtual boundaries (in the case of tactile-based robotic-assisted surgery) or a tool path plan (in the case of autonomous robotic surgery). The virtual boundaries or tool paths may represent the holes and / or cuts created in the bones 10, 11 during an arthroplasty procedure. Once the surgical plan is developed, the tactile device 60 is used to assist the user in creating the planned holes and cuts in the bones 10, 11. In particular, the preoperative planning regarding the planned bone resection depth and prevention of notches in the anterior femoral shaft will be described more fully below.

[0092] The creation of holes and cuts or resections in the bones 10, 11 can be achieved with the assistance of a haptically guided interactive robotic system, such as the haptic guidance system described in U.S. Patent No. 8,010,180, titled "Haptic Guidance System and Method," which was granted on August 30, 2011, and is hereby incorporated by reference in its entirety. When a surgeon operates a robotic arm to drill a hole or perform a cut in the bone with a high-speed drill, a sagittal saw, or other suitable tool, the system provides haptic feedback to guide the surgeon when sculpting the holes and cuts into the appropriate shape pre-programmed into the control system of the robotic arm. Haptic guidance and feedback will be described more fully below.

[0093] During the surgical planning, the computer 50 further receives information related to the femur and tibia implants to be implanted during the arthroplasty procedure. For example, the user may use an input device 52 (e.g., keyboard, mouse, etc.) to input into the computer 50 the parameters of the selected femur and tibia implants. Alternatively, the computer 50 may include a pre-established database of various implants and their parameters, and the user may select the implant to be used from the database. In yet a further embodiment, the implant may be custom designed based on the patient-specific surgical plan. The selection of the implant may occur at any stage of the surgical planning.

[0094] The surgical plan may further be based on at least one parameter of the implant or a function of the parameters of the implant. Since the implant may be selected at any stage of the surgical planning process, the implant may be selected before or after the determination of the planned virtual boundaries by the computer 50. If the implant is selected first, the planned virtual boundaries may be based at least in part on the parameters of the implant. For example, the distance (or any other relationship) between the planned virtual boundaries representing the holes or cuts to be made in the bones 10, 11 may be planned based on the desired varus-valgus femur-tibia alignment, extension, internal-external rotation, or any other factor associated with the desired surgical outcome of the implantation of the arthroplasty implant. Thus, the implementation of the surgical plan results in an appropriate alignment of the resected bone surfaces and holes, enabling the selected implant to achieve the desired surgical outcome. Alternatively, the computer 50 may develop a surgical plan that includes the planned virtual boundaries prior to the implant selection. In this case, the implant may be selected (e.g., input, selected, or designed) based at least in part on the planned virtual boundaries. For example, the implant may be selected based on the planned virtual boundaries such that the execution of the surgical plan results in an appropriate alignment of the resected bone surfaces and holes, enabling the selected implant to achieve the desired surgical outcome.

[0095] Virtual boundaries or tool paths exist within a virtual space and can represent features that exist within or are generated within a physical (i.e., real) space. A virtual boundary corresponds to a working boundary within the physical space that can interact with an object within the physical space. For example, the working boundary can interact with a surgical tool 58 connected to a haptic device 60. Surgical plans are often described herein as including virtual boundaries that represent holes and resections, but a surgical plan can include virtual boundaries that represent other modifications to the bones 10, 11. Further, a virtual boundary can correspond to any working boundary within the physical space that can interact with an object within the physical space.

[0096] Note that while the systems and methods disclosed herein are in the context of arthroplasty, they can readily be useful in the context of surgeries that do not employ implants. Thus, for example, and without limitation, navigation and haptics can be pre-planned such that the systems disclosed herein enable resection of a bone tumor (sarcoma), or another type of incision or resection in bone or soft tissue, when performing generally any type of navigated surgery.

[0097] Referring again to FIG. 2, after the surgical plan and prior to performing the arthroplasty procedure, the physical anatomical structures (e.g., bones 10, 11) are positioned, using positioning techniques (step 804), to a virtual representation (e.g., a preoperative three-dimensional representation) of the anatomical structure as described in detail below. Positioning of the patient's anatomical structure enables accurate navigation (step 805) during the surgical procedure, whereby each of the virtual boundaries can correspond to a working boundary within the physical space. For example, referring to FIGS. 3A and 3B, a virtual boundary 62 representing a resection in the tibia 10 is displayed on a computer or other display 63, and the virtual boundary 62 corresponds to a working boundary 66 within the physical space 69, such as the surgical site, within the operating room. And a portion of the working boundary 66 corresponds to the location of the planned resection in the tibia 10.

[0098] The virtual boundary, and thus the corresponding working boundary, can be of any configuration or shape. Referring to FIG. 3A, the virtual boundary 62 representing the proximal resection generated in the tibia 10 can be of any configuration suitable for assisting the user during the generation of the proximal resection in the tibia 10. The portion of the virtual boundary 62 shown within the virtual representation of the tibia 10 represents the bone to be removed by the surgical tool. Similar virtual boundaries can be generated for holes drilled or milled within the tibia 10 to facilitate implantation of the tibial implant in the resected tibia. The virtual boundary (and thus the corresponding working boundary) can include a surface that completely surrounds and encloses a three-dimensional volume. In an alternative embodiment, the virtual boundary and the working boundary do not completely surround the three-dimensional volume, but rather include both "active" surfaces and "open" portions. For example, the virtual boundary 62 representing the proximal resection in the tibia can have a substantially rectangular box-shaped "active" surface 62a and a folded funnel or triangular box-shaped "active" surface 62b connected to a rectangular box-shaped portion having an "open" portion 64. In one embodiment, the virtual boundary 62 can be generated by a folded funnel as described in U.S. Application No. 13 / 340,668, filed Dec. 29, 2011, entitled "System and Method for Selectively Activating a Tactile Guidance Zone," which is hereby incorporated by reference in its entirety. The working boundary 66 corresponding to the virtual boundary 62 has the same configuration as the virtual boundary 62. In other words, the working boundary 66 guiding the proximal resection in the tibia 10 can have a substantially rectangular box-shaped "active" surface 66a and a folded funnel or triangular box-shaped "active" surface 66b connected to a rectangular box-shaped portion having an "open" portion 67.

[0099] In a further embodiment, the virtual boundary 62 representing the resection in the bone 10 includes only a substantially rectangular box-shaped portion 62a. The ends of the virtual boundary having only the rectangular box-shaped portion can have an "open" upper portion such that the open upper portion of the corresponding working boundary coincides with the outer surface of the bone 10. Alternatively, as shown in FIGS. 3A and 3B, the rectangular box-shaped working boundary portion 66a corresponding to the virtual boundary portion 62a can extend beyond the outer surface of the bone 10.

[0100] In some embodiments, the virtual boundary 62 representing the resection through the bone portion may have a substantially planar shape, with or without thickness. Alternatively, the virtual boundary 62 may be curved or have an irregular shape. If the virtual boundary 62 is depicted as a line or planar shape and also has thickness, the virtual boundary 62 may be slightly thicker than the surgical tool used to create the resection in the bone, such that the tool may be constrained within the active face of the working boundary 66 while it is within the bone. Such a linear or planar virtual boundary 62 may be planned such that the corresponding working boundary 66 extends beyond the outer surface of the bone in a funnel or other suitable shape to assist the surgeon as the surgical tool 58 approaches the bone 10. Haptic guidance and feedback (as described below) may be provided to the user based on the relationship between the surgical tool 58 and the active face of the working boundary.

[0101] The surgical plan may also include automatic alignment of the surgical tool and may include virtual boundaries to facilitate entry into and exit from haptic control, as described in U.S. Application No. 13 / 725,348, filed December 21, 2012, entitled "Systems and Methods for Haptic Control of Surgical Tools," which is hereby incorporated by reference in its entirety.

[0102] A surgical plan that includes a virtual boundary can be developed based on information related to the patient's bone density. The density of the patient's bone is calculated using data obtained from CT, MRI, or other imaging of the patient's anatomical structure. In one embodiment, a calibration object that represents human bone and has a known calcium content is imaged to obtain a correspondence between the density values of the image and bone density measurements. This correspondence can then be applied to convert the density values of the individual images of the patient's anatomical structure into bone density measurements. The individual images of the patient's anatomical structure are then segmented and used with the corresponding map of bone density measurements to generate a three-dimensional representation (i.e., a model) of the patient's anatomical structure that includes the patient's bone density information. Image analysis, such as finite element analysis (FEA), can then be performed on the model to evaluate its structural integrity.

[0103] By being able to evaluate the structural integrity of the patient's anatomical structure, the effectiveness of the arthroplasty plan is improved. For example, if a particular portion of the patient's bone is thought to be less dense (i.e., osteoporosis), holes, resections, and implant placement can be planned to minimize the risk of fracture of the weakened portion of the bone. Further, the planned structure of the combination of bone and implant after the implementation of the surgical plan (e.g., the postoperative bone and implant placement) can also be evaluated for structural integrity preoperatively to improve the surgical plan. In this embodiment, holes and / or cuts are planned and the bone model and implant model are manipulated to represent the patient's bone and implant placement after the performance of the arthroplasty and implantation procedures. Various other factors that affect the structural integrity of the postoperative bone and implant placement, such as the patient's weight and lifestyle, can be considered. The structural integrity of the postoperative bone and implant placement is analyzed to determine whether the placement is structurally sound and kinematically functional postoperatively. If structural weakness or kinematic concerns are revealed by the analysis, the surgical plan can be modified to achieve the desired postoperative structural integrity and function.

[0104] In one embodiment, once the surgical plan is finalized, the surgeon may perform the arthroplasty procedure with the assistance of the haptic device 60 (step 806). In one embodiment, as an alternative to or in addition to the haptic device 60 (step 806), the surgical system 100 employs the OrthoMap® Precision Knee navigation software of Stryker's Advanced Guidance Technologies. The OrthoMap® Precision Knee navigation software facilitates the navigation of the cutting guide to a predetermined location.

[0105] In the context of an embodiment that employs the haptic device 60 according to step 806, through the haptic device 60, the surgical system 100 provides haptic guidance and feedback to the surgeon to assist the surgeon in accurately performing the surgical plan. The haptic guidance and feedback during the arthroplasty procedure enable better control of the surgical tools and result in more accurate alignment and placement of the implant compared to conventional arthroplasty techniques. Further, the haptic guidance and feedback are intended to eliminate the need to use K-wires and fluoroscopy for planning purposes. Instead, the surgical plan is generated and confirmed using a three-dimensional display of the patient's anatomical structure, and the haptic device provides guidance during the surgical procedure.

[0106] "Haptic" refers to the sense of touch, and the field of haptics relates to human-interactive devices that provide tactile sensation and / or force feedback to an operator. Tactile feedback generally includes the sense of touch such as vibration. Force feedback (also known as "tactile") refers to feedback in the form of force (e.g., resistance to movement) and / or torque. A wrench includes feedback in the form of, for example, force, torque, or a combination of force and torque. Haptic feedback may also include disabling or varying the amount of force provided to a surgical tool that can provide tactile sensation and / or force feedback to the user.

[0107] The surgical system 100 provides haptic feedback to the surgeon based on the relationship between at least one of the surgical tool 58 and the working boundary. The relationship between the surgical tool 58 and the working boundary can be any suitable relationship between the surgical tool 58 and the working boundary that can be obtained by the navigation system and utilized by the surgical system 100 to provide haptic feedback. For example, the relationship can be the position, orientation, posture, velocity, or acceleration of the surgical tool 58 relative to one or more working boundaries. The relationship can further be any combination of the position, orientation, posture, velocity, and acceleration of the surgical tool 58 relative to one or more working boundaries. The "relationship" between the surgical tool 58 and the working boundary can also refer to a quantity or measurement resulting from another relationship between the surgical tool 58 and the working boundary. In other words, the "relationship" can be a function of another relationship. As a specific example, the "relationship" between the surgical tool 58 and the working boundary can be the magnitude of the haptic force generated by the positional relationship between the surgical tool 58 and the working boundary.

[0108] During surgery, the surgeon operates the haptic device 60 to guide the surgical tool 58 connected to the device. The surgical system 100 provides haptic feedback to the user through the haptic device 60 to assist the surgeon during the generation of the planned holes, cuts, or other modifications to the patient's bone that are required to facilitate the implantation of the femur and tibia implants. For example, the surgical system 100 may assist the surgeon by substantially preventing or restricting the surgical tool 58 from crossing the working boundary. The surgical system 100 may restrict the surgical tool 58 from crossing the working boundary by any number of haptic feedback mechanisms and combinations thereof, including providing tactile feedback, providing force feedback, and / or changing the amount of force applied to the surgical tool. As used herein, "restrict" is used to describe a tendency to limit movement. Thus, the surgical system may directly restrict the surgical tool 58 by applying an opposing force to the haptic device 60 that tends to limit the movement of the surgical tool 58. The surgical system may also indirectly restrict the surgical tool 58 by providing tactile feedback to warn the user to modify their actions. This is because warning the user to modify their actions tends to limit the movement of the surgical tool 58. In yet a further embodiment, the surgical system 100 may restrict the surgical tool 58 by limiting the force applied to the surgical tool 58, which further tends to limit the movement of the tool.

[0109] In various embodiments, the surgical system 100 provides haptic feedback to the user when the surgical tool 58 approaches the working boundary, upon contact of the surgical tool 58 with the working boundary, and / or after the surgical tool 58 has entered the working boundary by a predetermined depth. The surgeon may experience the haptic feedback, for example, as vibrations, as a wrench that resists or actively opposes further movement of the haptic device, or as a rigid "wall" that substantially prevents further movement of the haptic device. Alternatively, the user may experience the haptic feedback as a sensation of touch due to a change in the force provided to the surgical tool 58 (e.g., a change in vibration), or as a sensation of touch due to a cessation of the force provided to the tool. When the surgical tool 58 is drilling, cutting, or otherwise operating directly on bone, if the force on the surgical tool is changed or stopped, the surgeon will feel haptic feedback in the form of resistance to further movement due to the tool no longer being able to drill, cut, or otherwise move through the bone. In one embodiment, upon contact between the surgical tool 58 and the working boundary, the force on the surgical tool is changed (e.g., the force on the tool is decreased) or stopped (e.g., the tool is disabled). Alternatively, the force provided to the surgical tool 58 may be changed (e.g., decreased) when the surgical tool 58 approaches the working boundary.

[0110] In another embodiment, the surgical system 100 may assist the surgeon in generating planned holes, cuts, and other modifications to the bone by providing haptic feedback to guide the surgical tool 58 toward or along the working boundary. As an example, the surgical system 100 may provide a force to the haptic device 60 based on the positional relationship between the tip of the surgical tool 58 and the closest coordinates of the working boundary. The force may bring the surgical tool 58 closer to the closest working boundary. Once the surgical tool 58 is substantially close to or in contact with the working boundary, the surgical system 100 may apply a force that tends to guide the surgical tool 58 to move along a portion of the working boundary. In another embodiment, the force tends to guide the surgical tool 58 to move from one portion of the working boundary to another portion of the working boundary (e.g., from a funnel-shaped portion of the working boundary to a rectangular box-shaped portion of the working boundary).

[0111] In yet another embodiment, the surgical system 100 is configured to assist the surgeon in generating planned holes, cuts, and modifications to the bone by providing haptic feedback to guide the surgical tool from one working boundary to another. For example, the surgeon may experience a force that tends to pull the surgical tool 58 toward the working boundary 66 when the user guides the surgical tool 58 toward the working boundary 66. Thereafter, when the user removes the surgical tool 58 from the space enclosed by the working boundary 66 and operates the haptic device 60 so that the surgical tool 58 approaches a second working boundary (not shown), the surgeon may experience a force that pushes out from the working boundary 66 toward the second working boundary.

[0112] Haptic feedback as described herein may operate in relation to a modification to the working boundary by the surgical system 100. Although discussed herein as a modification to the "working boundary", it should be understood that the surgical system 100 modifies a virtual boundary corresponding to the working boundary. Some examples of modifications to the working boundary include: 1) reconfiguration of the working boundary (e.g., changing the shape or size), and 2) making all or a portion of the working boundary active and non-active (e.g., converting an "open" portion to an "active" surface and converting an "active" surface to an "open" portion). Similar to haptic feedback, the modification to the working boundary can be performed by the surgical system 100 based on the relationship between the surgical tool 58 and one or more working boundaries. The modification to the working boundary further aids the user in generating the holes and cuts required during an arthroplasty procedure by facilitating various actions such as the movement of the surgical tool 58 towards the bone and the cutting of the bone by the surgical tool 58.

[0113] In one embodiment, the modification to the working boundary facilitates the movement of the surgical tool 58 towards the bone 10. During a surgical procedure, since the patient's anatomical structure is tracked by the navigation system, the surgical system 100 moves the entire working boundary 66 in response to the movement of the patient's anatomical structure. In addition to this movement of the reference line, portions of the working boundary 66 can be reshaped and / or reconfigured to facilitate the movement of the surgical tool 58 towards the bone 10. As an example, the surgical system can tilt the funnel-shaped portion 66b of the working boundary 66 relative to the rectangular box-shaped portion 66a during the surgical procedure based on the relationship between the surgical tool 58 and the working boundary 66. Thus, the working boundary 66 can be dynamically modified during the surgical procedure such that when the surgical tool 58 approaches the bone 10, the surgical tool 58 remains within the space surrounded by the portion 66b of the working boundary 66.

[0114] In another embodiment, the working boundary or a portion of the working boundary is made active and inactive. Making the entire working boundary active and inactive can assist the user when the surgical tool 58 is approaching the bone 10. For example, while the surgeon is approaching the first working boundary 66, or while the surgical tool 58 is within the space enclosed by the first working boundary 66, a second working boundary (not shown) can be made inactive. Similarly, after the surgeon has completed generating the first corresponding resection and is attempting to generate a second resection, the first working boundary 66 can be made inactive. In one embodiment, the working boundary 66 can be made inactive after the surgical tool 58 has entered an area within the funnel portion leading to the second working boundary but is still outside the first funnel portion 66b. By making a portion of the working boundary active, a previous open portion (e.g., the open upper portion 67) is converted into an active face of the working boundary. In contrast, by making a portion of the working boundary inactive, a previous active face of the working boundary (e.g., the end 66c of the working boundary 66) is converted into an "open" portion.

[0115] Making the entire working boundary or a portion thereof active and inactive can be dynamically achieved by the surgical system 100 during a surgical procedure. In other words, the surgical system 100 can be programmed to determine the presence of factors and relationships that trigger the activation and deactivation of a virtual boundary or a portion of the virtual boundary during a surgical procedure. In another embodiment, the user can interact with the surgical system 100 (e.g., by using the input device 52) to indicate the start or completion of various stages of an arthroplasty procedure, thereby triggering the working boundary or a portion thereof to be made active or inactive.

[0116] Considering the operation and function of the surgical system 100 as described above, the discussion here will now turn to a detailed discussion of how to preoperatively plan a surgery to be performed via the surgical system 100, and how to align the preoperative plan with the patient's actual bone and the applicable components of the surgical system 100.

[0117] Since the device 60 is operated by a surgeon to perform various resections, perforations of holes, etc., the haptic device 60 can be described as a device or tool for surgeon assistance. In certain embodiments, the device 60 can be an autonomous robot, as opposed to surgeon assistance. That is, as opposed to the haptic boundary, the autonomous robot can only operate along a predetermined tool path, and as a result, since haptic feedback is not required, the tool path can be defined to resect bone and perforate holes. In certain embodiments, the device 60 can be a cutting device having at least one degree of freedom that operates in association with the navigation system 42. For example, the cutting tool can include a rotating burr having a tracker on the tool. The cutting tool can be freely operable by a surgeon and can be hand-held. In such a case, the haptic feedback can be limited to the bar stopping rotation when it intersects the virtual boundary. Thus, the device 60 should be considered to broadly encompass any and other of the devices described in this application.

[0118] After the surgical procedure is completed, immediately or after a period of time, a postoperative analysis (step 807) can be performed. The postoperative analysis can determine the accuracy of the actual surgical procedure when compared to the planned procedure. That is, the position and orientation of the actual implant placement can be compared to the planned values. Factors such as varus-valgus femur-tibia alignment, extension, internal-external rotation, or any other factor associated with the desired surgical outcome of the implantation of an arthroplasty implant can be compared to the planned values.

[0119] II. Preoperative Steps of an Arthroplasty Procedure

[0120] The preoperative steps of an arthroplasty procedure can include imaging of the patient and a preoperative planning process that, among other evaluations, can include implant placement, determination of bone resection depth, and notch evaluation of the anterior femoral shaft. Determination of bone resection depth includes selecting and positioning three-dimensional computer models of candidate femoral and tibial implants relative to three-dimensional computer models of the patient's distal femur and proximal tibia to determine the position and orientation of the implant to achieve a desired surgical outcome for the arthroplasty procedure. As part of this evaluation, the depth of the required tibial and femoral resections is calculated along with the orientation of the resection planes.

[0121] Notch evaluation of the anterior femoral shaft includes determining whether the anterior flange of the three-dimensional model of the selected femoral implant intersects the anterior femoral shaft of the three-dimensional model of the patient's distal femur when the implant three-dimensional model is positioned and oriented relative to the femoral three-dimensional model as proposed during determination of bone resection depth. Such intersection of the two models indicates a notch in the anterior femoral shaft that needs to be avoided.

[0122] The determination of bone resection depth and the performance of notch evaluation of the anterior femoral shaft are described in International Application PCT / US2016 / 034847, filed May 27, 2016, which is hereby incorporated by reference in its entirety.

[0123] A. Preoperative Imaging

[0124] In preparation for surgical procedures (e.g., knee arthroplasty, hip arthroplasty, ankle arthroplasty, shoulder arthroplasty, elbow arthroplasty, spinal procedures (e.g., fusion, implantation, correction of scoliosis, etc.)), a patient may undergo pre-operative imaging, e.g., at an imaging center. The patient may undergo imaging modalities, particularly magnetic resonance imaging ("MRI"), computed tomography ("CT") scan, and fluoroscopic scan ("x-ray") at the joint to be operated on. As can be seen in FIG. 4A, an exemplary coronal image scan of a patient's knee joint 102 including the femur 104, patella 105 (shown in other figures), and tibia 106, the patient's knee may undergo a CT scan. The CT scan may include a helical scan of the knee joint 102 packaged as a Digital Imaging and Communications in Medicine ("DICOM") file. From the file, slices or cross-sections of the 2D image can be viewed in multiple planes (e.g., coronal, sagittal, axial). As can be understood from FIGS. 4B, 4C, and 4D, a segmentation process may be performed on the 2D image 108 by applying a spline 110 at the bone contour. Alternatively, the segmentation process may be performed on the image 108 as a whole without the need to apply the spline 110 to the 2D image slices. Such pre-operative imaging and planning steps can be found in International Application PCT / US2019 / 066206, filed Dec. 13, 2019, which is hereby incorporated by reference in its entirety.

[0125] Figures 4B, 4C, and 4D respectively show an axial image 108 of the femur 104 and patella 105 having a spline 110 on the bone surface, a sagittal image 108 of the joint 102 having a spline 110 on the bone surfaces of the femur 104, patella 105, and tibia 106, and a coronal image 108 of the joint 102 having a spline 110 on the femur 104 and tibia 106. In certain cases, the segmentation process can be a manual process in which a person identifies the spline 110 on each two-dimensional image slice 108. In certain cases, the segmentation process may be automated, where the spline 110 is automatically applied to the bone contour lines within the image slice 108. In certain cases, the segmentation process can be a combination of a manual process and an automated process.

[0126] After the segmentation process is completed, the segmented images 108 can be combined to generate a three-dimensional ("3D") bone model 111 of the joint 102 that includes a 3D femur model 112, a 3D patella model 113, and a 3D tibia model 114.

[0127] As can be seen in Figure 4E, an isometric axial-coronal-sagittal view of the 3D joint model 111, the model 111 represents the joint 102 in a deteriorated state prior to the execution of any surgical procedure to correct the bone, more specifically its femur 104, patella 105, and tibia 106. From this 3D joint model 111, various steps of the preoperative planning process can be performed. Each of these 3D bone models 112-114 can be generated relative to the coordinate system of the medical imaging system used to generate the two-dimensional image slices 108. For example, if the image slices 108 are generated via CT imaging, the 3D bone model can be generated relative to the CT coordinate system 115.

[0128] In certain cases, a 3D model 111 of a patient's joint that includes 3D models 112, 113, and 114 of each of the bones 104, 105, and 106 of the patient's joint 102 may be generated from a statistical or generic model of the bones and joints, and the statistical or generic model is deformed or otherwise modified to approximate the bones 104, 105, and 106 of the patient's joint 102 based on certain factors that do not require segmenting the 2D image slices 108 with a spline 110. In certain cases, the segmentation process may adapt a 3D statistical or generic bone model to the scanned images 108 of the femur 104, patella 105, and tibia 106, either manually, automatically, or a combination of manual and automatic. In such cases, the segmentation process does not require applying a spline 110 to each of the 2D image slices 108. Instead, the 3D statistical or generic bone model is adapted or deformed to the shape of the femur 104, patella 105, and tibia 106 in the scanned image 108. Thus, the deformed or adapted 3D bone model is required for the 3D joint model 111 shown in FIG. 4E.

[0129] In one embodiment, the generic bone model may be the result of an analysis of a large number (e.g., thousands or tens of thousands) of medical images (e.g., CT, MRI, X-ray, etc.) of actual bones regarding size and shape, and this analysis is used to generate a generic bone model that is a statistical average of a large number of actual bones. In another embodiment, a statistical model is derived that describes the statistical distribution of a population including variations in size, shape, and appearance within the image.

[0130] In certain cases, other methods of generating a patient model may be employed. For example, a patient's bone model or a portion thereof may be generated during surgery by positioning the bone or cartilage surface within one or more areas of the bone. Such a process may generate one or more bone surface contours. Thus, the various methods described herein are intended to encompass 3D bone models generated from segmented medical images (e.g., CT, MRI) as well as intraoperative imaging methods and other things.

[0131] The imaging of the method and subsequent steps are described with respect to the knee joint 102, but the teachings of the present disclosure are equally applicable to other joints, particularly the hip, ankle, shoulder, wrist, elbow, and spine.

[0132] B. Pre-operative planning of implant selection, implant placement, and orientation

[0133] After the 3D femur model 112 of the patient's joint 102 is generated, the remaining portion of the pre-operative planning can commence. For example, the surgeon or the surgical system 100 can select an appropriate implant, and the position and orientation of the implant can be determined. Such selection can determine the appropriate cutting or resection of the patient's bone to fit the selected implant. Such pre-operative planning steps can be found in International Application PCT / US2016 / 034847, filed on May 27, 2016, which is hereby incorporated by reference in its entirety.

[0134] III. Surgical procedure

[0135] After the pre-operative planning steps are completed, the surgery can commence according to the plan. That is, the surgeon can perform the resection of the patient's bone using the haptic device 60 of the surgical system 100, and the surgeon can implant the implant to repair the function of the joint. The steps of the surgical procedure can include the following.

[0136] A. Positioning

[0137] Positioning is a process of mapping a preoperative plan that includes bone models 111-114 (of FIG. 4E) and associated virtual boundaries or tool paths to the patient's physical bone. Therefore, the robotic arm 60 is spatially oriented with respect to the patient's physical bone to accurately perform the surgical procedure. The preoperative plan including bone models 111-114 and associated virtual boundaries or tool paths can be stored on the computer 50 in a first coordinate system (x1, y1, z1). A navigation system 42 that tracks the movement of the robotic arm 60 via various tracker arrays (e.g., 48, 54) also communicates with the computer 50. The navigation system 42 also tracks the patient's body via various tracker arrays 46, 47 disposed on the tibia 10 and the femur 11, respectively. Thus, the positions and orientations (i.e., postures) of the robotic arm 60 and the bones 10, 11 being operated on are known relative to each other in a second coordinate system (x2, y2, z2) within the computer 50. The process of mapping, transforming, or positioning the first coordinate system (x1, y1, z1) and the second coordinate system (x2, y2, z2) together in a common coordinate system is known as positioning.

[0138] Once positioned, the bone models 111-114 and the virtual boundaries or tool paths can be "locked" in place on the patient's physical bone such that any movement of the patient's physical bone will correspondingly move the bone models 111-114 and the virtual boundaries or tool paths. Thus, the robotic arm 60 can be constrained to operate along the virtual boundaries or tool paths defined in the preoperative plan and that move with the patient's bone as it moves. In this way, the robotic arm 60 spatially recognizes the posture of the patient's physical body via the positioning process.

[0139] i. Generation of a classified / separated 3D bone surface point cloud from intraoperative ultrasound data

[0140] As discussed in detail below, the computer 50 of the surgical system 100, and more specifically, the processor and memory of the computer, store and execute one or more algorithms that employ one or a combination of various neural networks trained to detect bone surfaces in ultrasound images and classify such bone surfaces in the ultrasound images according to the captured anatomical structures.

[0141] As also discussed in detail below, the computer 50 of the surgical system 100, and more specifically, the processor and memory of the computer, store and execute one or more algorithms that enable simultaneous co - registration of the bone surfaces of N bones (usually forming joints) between an ultrasound modality that captures the N bones and a second modality (e.g., CT / MRI). The co - registration of the bone surfaces of the N bones between the two modalities is achieved via one or more algorithms that optimize an Nx6 degree - of - freedom transformation from the ultrasound modality to the second modality and classification information that assigns a region in the image data of the ultrasound modality to one of the N captured bones.

[0142] In one embodiment, the co - registration of the bone surfaces of the N bones between the two modalities can occur between a 3D point cloud and a triangulated mesh and be 3D point cloud / mesh - based. In such a situation, the N bones need to be segmented in the second modality (e.g., segmentation of CT / MRI bones) to obtain a triangulated mesh, and the 3D point cloud is applied to the triangulated mesh.

[0143] In another embodiment, the co - registration of the bone surfaces of the N bones between the two modalities can be image - based. In other words, the classified ultrasound image data is directly matched to the second modality without the need to detect bone surfaces.

[0144] For purposes of initiating a discussion of one or more algorithms for simultaneously co - localizing the bone surfaces of N bones between two modalities that capture the N bones, reference is made to FIG. 5A. FIG. 5A is a flowchart that shows in more detail the overall positioning process (step 804) as employed by the surgical system 100 depicted in FIG. 1 and shown in FIG. 2. The process depicted in FIG. 5A is an ultrasound - based multi - bone positioning process 503 that employs a two - step procedure, where the initial or gross positioning (step 776) resulting from steps 610, 612, 776 is combined (step 900) with a classified or segmented 3D bone surface point cloud (step 600) generated from an ultrasound sweep obtained in the surgical target region (i.e., the patient's joint region in the context of the patient's arthroplasty). Thus, the ultrasound - based multi - bone positioning process 503 has two main steps or aspects, where the initial positioning establishes a first guess of the positioning alignment and then the starting point is refined by calculating a very accurate alignment from there.

[0145] The classification or separation of the 3D bone surface point cloud (step 600) of the ultrasonic-based multiple bone positioning process 503 starts with intraoperative ultrasonic images that are obtained over most, if not all, of the patient's surface area surrounding the patient's joint and each bone in the vicinity of the patient's joint. For example, for knee arthroplasty, the ultrasonic sweep is performed one or more times up and down over most, if not all, of the knee to obtain ultrasonic image data of the bone surfaces of each bone of the patient's knee (femur, tibia, and patella). The intraoperative ultrasonic images are then algorithmically analyzed via machine learning to determine which of the millions of individual points of the acquired ultrasonic image points belong to each bone of the patient's joint, resulting in a classified or separated point cloud belonging to each bone. In other words, in the context of knee arthroplasty, the algorithm can be said to appropriately assign each point or pixel of the intraoperative ultrasonic image to its respective knee joint bone, such that each point or pixel is classified or separated to correspond to its respective bone, thereby resulting in a classified or separated 3D bone surface point cloud. Stated another way, each point or pixel of the intraoperative ultrasonic image is converted into a classified or separated 3D bone surface point cloud such that the ultrasonic image pixels or points of the classified or separated 3D bone surface point cloud are each mutually associated with the corresponding bone surface of the patient's bone.

[0146] As can be understood from FIG. 5A, the initial or rough positioning (step 776) is started using the tracked probe 57 applied to the patient's knee during the operation according to a set of landmarks relative to a particular pre-operative planned posture or the patient's anatomical structure, and a transformation is generated that positions the physical tracked bone relative to the 3D CAD bone model 111 generated from the segmented medical imaging (CT, MRI, etc.) of the patient's knee bone. The final positioning occurs via a combination of the initial positioning algorithm of step 776 with the classified 3D bone surface point cloud of step 600, and each portion of the classified point cloud is algorithmically matched to the surface of the initial positioning point cloud and its respective 3D CAD bone model (step 900). During this final positioning, the algorithm employed converges until its result reaches a steady state, and the algorithm can also improve the classification or separation of the classified or separated 3D bone surface point cloud itself, such that any initial errors in the classification / separation can be removed or at least reduced. In achieving these aspects of the final positioning, the classified or separated 3D bone surface point cloud and the initial or rough positioning become fully positioned, resulting in the final positioning of multiple bones. This final positioning of multiple bones in step 900 can then be employed by the surgical system 100 when performing surgery on the patient's joint.

[0147] This ultrasonic-based positioning process 503 of the surgical system 100 is efficient in that a medical expert simply performs an ultrasonic sweep of the patient's joint region, and then machine learning takes over to identify which points in the ultrasonic sweep belong to which bones of the patient's joint, and then assigns / matches the points to the correct bones in the 3D bone model to achieve positioning. Thus, the positioning process 503 enables all of the multiple bones of the patient's joint to be imaged via ultrasound at once, and then the system identifies and separates the points of the point cloud belonging to each bone of the joint, and then assigns / matches the points to the appropriate bones in the 3D model of the joint to complete the final positioning process, where the points are not only assigned to the appropriate bones but are also placed at the corresponding anatomical locations on the bones.

[0148] As shown in FIG. 5A, the ultrasonic-based positioning process 503 of the surgical system 100 includes a preoperative aspect 500 and an intraoperative aspect 502, and the preoperative aspect 500 and the intraoperative aspect 502 are each divided into a workflow part and a data flow part. For the workflow part, a person operates a machine / tool / device / system or physically performs the execution of the identified workflow steps. For the data flow part, as will be discussed in more detail below with reference to FIG. 16, one or more hardware processors 1302 of the computer system 1300 associated with the surgical system 100 depicted in FIG. 1 execute a program when executing the identified data flow steps.

[0149] During the workflow section of the preoperative aspect 500, medical images are acquired of the patient's joint as described above in the section "A. Preoperative Imaging" of this detailed description (step 504). As shown in FIG. 5A, following the data flow section of the preoperative aspect 500, the medical images are then used to generate a 3D CAD model of the bones forming the patient's joint as described above in the section "A. Preoperative Imaging" of this detailed description with reference to FIGS. 4A-4E (step 506). The data flow section of the preoperative aspect 500 ends with the generation of initial positioning data within the CAD model space for the 3D CAD models 111-114 (FIG. 4E) of the patient's bones 104-106 of the patient's joint image 108 (FIGS. 4A-4D) (step 508). Specifically, this generation of initial positioning data (step 508) includes determining the above-described probe pose and anatomical landmarks within the 3D CAD model space.

[0150] Moving on to the workflow section of the intraoperative aspect 502 of the ultrasonic-based multi-bone positioning process 503 of FIG. 5A, the medical professional places anatomical structure trackers (see, e.g., 46, 47 in FIG. 1) on the patient's bones (e.g., the tibia 10 and femur 11 in FIG. 1) (step 510). In a preferred embodiment, the trackers 46, 47 are typically placed on the patient's bones 10, 11 at the start of the intraoperative portion 502 of the overall positioning process 503 of FIG. 5A as shown in step 510, but in an alternative embodiment, if the target limb (e.g., the leg) is sufficiently immobilized, the positioning portion can be performed before the trackers 46, 47 (FIG. 1) are attached to the bones 10, 11. For example, an initial positioning (step 776) is completed, and then a classified point cloud is generated (step 600), both of which are discussed in more detail below with respect to FIG. 5A and others. The location of the trackers 46, 47 can then be identified, and accordingly, the trackers are placed on the patient's bones 10, 11. The initial positioning 776 is repeated, and then the initial positioning of step 776 and the classified point cloud of step 600 can be combined to achieve a final positioning (step 900).

[0151] Referring again to FIG. 5A, continuing with the preferred embodiment, here, at the start of the intraoperative portion 502 of the overall positioning process 503, trackers 46, 47 are placed on the patient's bones 10, 11 according to step 510, and then the medical professional records a plurality of ultrasonic sweeps of the patient's bones across the joint region using a trackable ultrasonic probe (e.g., see 55 in FIG. 1) (step 512). The sweeps can be of the joint area in a general way and capture various bones (e.g., the femur, tibia, and patella in the context of the knee joint) in a collage of ultrasonic image data points that are not defined as to which bone and which location on the bone each data point belongs to.

[0152] As can be appreciated from FIG. 5B, which is a pictorial representation of the process of generating a classified three-dimensional (“3D”) bone surface point cloud from an ultrasonic sweep, the ultrasonic sweep 514 resulting from step 512 of FIG. 5A has ultrasonic image data associated with the femur 11, patella (not shown), and tibia 10 (see, e.g., FIG. 1) that is not defined as to which bone and which location on the bone. Although not actually defined at this point in the process, as shown in FIG. 5B, the ultrasonic sweep 514 can be understood to have femur data points 516, patella data points 518, and tibia data points 520. This data can be arranged in the form of a set of 2D ultrasonic images 523 that includes an ultrasonic image 522 of the femur, an ultrasonic image of the patella (not shown), and an ultrasonic image 524 of the tibia, but the ultrasonic images are not defined as to whether a particular point is part of bone or soft tissue and have data points that are not defined as to which particular bone of the joint a particular data point belongs to and which location on the particular bone. The surgical system 100 then converts the set of 2D ultrasonic images 523 into classified bone surface pixels 526, as shown in FIG. 5B, via the classification module (step 527) of FIG. 5A. Although 2D ultrasonic images are depicted in FIG. 5B and discussed herein, it should be noted that the processes described herein can be readily achieved by the use of 3D ultrasonic images in lieu of or in combination with 2D ultrasonic images. Accordingly, throughout this disclosure, any reference to 2D ultrasonic images should be understood to include 3D ultrasonic images as well.

[0153] As shown in FIG. 5A, in one embodiment, classified bone surface pixels 526 are generated from a set of 2D ultrasound images 523, and the classification module (step 527) can employ either of two alternative classification processes, namely, by the ultrasound image bone surface via a pixel classification neural network (step 528), or by ultrasound image bone surface detection via a likelihood classification neural network (step 530). In other embodiments, classified bone surface pixels 526 are generated from a set of 2D ultrasound images 523, and the classification module (step 527) can employ other processes such as, for example, deriving a classification of none from the distances of two navigation markers 46, 47 fixed to bones 10, 11 by looking only at the 3D placement of various points without looking at the image, and / or running a classification algorithm on the resulting point cloud itself. In yet other embodiments, the classification of the point cloud is elucidated via other non-machine learning processes or other networks in addition to classification and convolution such as, for example, random forest. In still further embodiments, the classification of the point cloud can be elucidated via a non-machine learning process. In one embodiment, the classification of the point cloud can be elucidated via a geometric analysis of the point cloud. For example, such a geometric analysis of the point cloud can include separation of the principal axis such as principal component analysis (e.g., in the context of knee arthroplasty), clustering methods such as connected component analysis (e.g., in the context of spinal / vertebral procedures), and shape characteristics such as convex / concave / tubular / etc. Finally, for the purpose of not unduly limiting the present disclosure, there is a classification module 527 that receives an image as input, and the classified bone point cloud is output from the classification module, and there are a number of different processes that can be part of the classification module to achieve these purposes.

[0154] FIG. 6A is a flowchart of a process of the surgical system 100 for ultrasonic image bone surface detection (step 528) using pixel classification, and FIG. 6B is an image depiction of the process of FIG. 6A. As shown in FIG. 6A, this process employs an ultrasonic image bone surface detector 532 and an ultrasonic image pixel classifier 534, both of which are divided into an input section, a processing section, and an output section. As shown in FIGS. 6A and 6B, the ultrasonic image bone surface detector 532 receives the 2D ultrasonic image 523 of FIG. 5B as an input (step 536), processes it through a convolutional network to detect the presence or absence of bone surface points at each point across each ultrasonic image (step 538), and outputs a binary image 540 of "zero (0) is equivalent to non-bone surface" 542 and "one (1) is equivalent to bone surface" 544 (step 546).

[0155] Similarly, the ultrasonic image pixel classifier 534 receives the 2D ultrasonic image 523 of FIG. 5B as an input (step 548), processes it through a classification network to determine the specific type of anatomical structure present at each point across each ultrasonic image (step 550), and outputs a single number 552 indicating the type of anatomical structure of "zero (0) is equivalent to the tibia", "one (1) is equivalent to the femur", and "two (2) is equivalent to the patella" (step 554). The examples given in this detailed discussion are in the context of the knee joint, but it should be noted that the concepts taught herein are similarly applicable to any type of joint, such as, for example, but not limited to, the spine, shoulder, elbow, wrist, hip, ankle, etc.

[0156] FIG. 7A is a flowchart of a process of the surgical system 100 for ultrasonic image bone surface detection (step 530) using likelihood classification, and FIG. 7B is an image depiction of the process of FIG. 7A. This process is divided into an input part, a processing part, and an output part. As shown in FIGS. 7A and 7B, the convolutional network 556 receives the 2D ultrasonic image 523 of FIG. 5B as an input (step 558), processes it to detect bone surface points, and classifies each point by determining the likelihood that the point belongs to a specific anatomical structure (step 560). For example, in the context of the knee, the convolutional network 556 detects bone surface points in the 2D ultrasonic image 523 and then calculates the likelihood that any particular detected bone surface point belongs to the femur, tibia, or patella (step 560). The convolutional network 556 outputs N binary images 562, where N is the number of bones being considered, and each pixel of each binary image is decoded as "zero (0)" or "one (1)" depending on whether the pixel represents a bone surface (step 563).

[0157] As can be understood from FIG. 7B, in the exemplary example, for this example only, the patella is ignored and the femur and tibia are present, so the number N of binary images is 2. If the patella is also used in this example, since the femur, patella, and tibia are present, the number N of binary images is 3.

[0158] Following the example where the number N of binary images is 2, the convolutional network 556 analyzes the ultrasonic image 523 and classifies the image 523 according to the likelihood that it represents an image 564 of the tibia or an image 566 of the femur. The classification is through the evaluation of the algorithm of the ultrasonic image 523 in the context of machine learning (step 562 in FIG. 7B). Then, each pixel of each classified image 564 classified according to step 562 is decoded as to whether each particular pixel represents a bone surface (step 568 in FIG. 7B). For example, the classified tibia image 564 is evaluated to identify its non-bone surface pixels 570 and its bone surface pixels 572, and the bone surface pixels 572 are the tibia bone surface pixels 572 (step 568). Similarly, the classified femur image 566 is evaluated to identify its non-bone surface pixels 574 and its bone surface pixels 576, and the bone surface pixels 576 are the femur bone surface pixels 576 (step 568).

[0159] Returning to FIGS. 5A and 5B, once the classified bone surface pixels 526 are generated from the set of 2D ultrasonic images 523 through the classification module (step 527) to provide the classified 2D femur bone surface pixels 526F and the classified 2D tibia bone surface pixels 526T, a 2D to 3D conversion of the 2D surface pixels 526F, 526T is calculated to convert the 2D surface pixels 526F, 526T to 3D surface pixels 578F, 578T (step 580).

[0160] FIG. 8A is a flowchart of a process (step 580) for calculating the conversion of 2D surface pixels 526F, 526T to 3D surface pixels 578F, 578T, and FIG. 8B is an image depiction of the process of FIG. 8A. As shown in FIGS. 8A and 8B, this process employs a temporal calibration of the ultrasonic probe (step 582), and the ultrasonic scan (514 in FIG. 5B) is acquired by the bone (10, 11 in FIG. 1) via ultrasonic waves 583 projected and detected through the distal tip 55A of the ultrasonic probe, and when the ultrasonic probe 55 is being tracked during the operation, the propagation speed of the ultrasonic waves 583 in a specific medium / tissue is considered when the trackable element 55B of the ultrasonic probe is detected by the detection device 44 of the navigation system 42 (step 584). The distal tip 55A includes a sensor array having its own intrinsic coordinate system 586. In so doing, the 2D surface pixels 526F, 526T of the ultrasonic images 522, 524 undergo a transformation 585 that maps them from the 2D pixel space (2D pixel coordinate system) 586 to the 3D metric coordinate system 588, converting the 2D surface pixels 526F, 526T to 3D surface pixels 578F, 578T and converting their 2D pixel coordinates to 3D coordinates within the intrinsic ultrasonic probe coordinate system 588 (step 590).

[0161] As shown in FIG. 8A and understood from FIG. 8B, upon completion of the temporal calibration of the ultrasonic probe (step 582), the process moves on to the calibration of the ultrasonic probe with respect to the probe tracker (step 592). In so doing, the system acquires a known set of poses of the ultrasonic probe 55 with respect to the probe detection device 44 of the navigation system 42 in relation to the intrinsic ultrasonic probe coordinate system 588 (step 594). The system then completes the transformation between the probe tracker space and the intrinsic ultrasonic probe coordinate system (step 596).

[0162] For further information regarding complementary and / or alternative processes associated with the calculation of the conversion of 2D surface pixels 526F, 526T to 3D surface pixels 578F, 578T according to step 580 or a version thereof, reference is made to the specification of PCT application number PCT / IB2018 / 056189, filed on August 16, 2018, entitled "Ultrasound Bone Positioning Using Learning-Based Segmentation and Sonic Calibration" (International Publication No. WO 2019 / 035049 (A1)), which is hereby incorporated by reference in its entirety into this disclosure.

[0163] It should be understood that the previous discussion has been in the context of a 2D ultrasound probe, but the 2D ultrasound probe can be replaced by a 3D ultrasound probe to continue the processes disclosed in this detailed description. Thus, the processes disclosed in this detailed description should not be limited to 2D ultrasound probes and 2D pixels / points, but should be considered to include 3D ultrasound probes and any type of ultrasound pixel / point within the image coordinate system, regardless of whether the ultrasound pixel / point is 2D or 3D.

[0164] Each individual ultrasound sweep using the ultrasound probe generates an individual ultrasound image that captures a slice or small portion of the patient's bone. Multiple individual ultrasound sweeps using the ultrasound probe are typically required when the ultrasound images the patient's bone. The ultrasound probe is tracked with respect to an anatomical structure tracker attached to each of the N captured bones in order to assemble each individual ultrasound image into one consistent 3D ultrasound image dataset.

[0165] When the bone being ultrasound scanned is fixed, the process of assembling the individual ultrasound images can be simplified. Specifically, in such a case, each individual ultrasound image can be assembled with other individual ultrasound images by tracking only the ultrasound probe.

[0166] As can be understood from FIGS. 5A and 5B, once the system completes step 580, it then generates a classified 3D bone surface point cloud 598 (step 600). The classified 3D bone surface point cloud 598 has femur points 602 classified as the femur, patella points 604 classified as the patella, and tibia points 606 classified as the tibia. For further complementary or alternative aspects of the positioning process using the ultrasonic probe, reference is made to U.S. Patent Application No. 14 / 144,961, filed on December 31, 2013, entitled "Positioning System and Method Using an Ultrasonic Probe", the entire disclosure of which is incorporated herein by reference.

[0167] ii. Initial rough positioning

[0168] As described above and as shown in FIG. 5A, during the workflow portion of the preoperative aspect 500, the medical image is acquired at the patient's joint as described above in the section "A. Preoperative Imaging" of this detailed description (step 504). As shown in FIG. 5A, the preoperative aspect 500 follows the data flow portion, and then the medical image is used to generate a 3D CAD model of the bones forming the patient's joint as described above in the section "A. Preoperative Imaging" of this detailed description with reference to FIGS. 4A-4E (step 506). The data flow portion of the preoperative aspect 500 ends with the generation of initial positioning data in the CAD model space for the 3D CAD models 111-114 (FIG. 4E) of the patient's bones 104-106 of the patient's joint image 108 (FIGS. 4A-4D) (step 508). Specifically, this generation of the initial positioning data (step 508) includes determining the probe posture and anatomical landmarks described above in the 3D CAD model space.

[0169] Regarding the discussion of the preoperative process of determining the probe posture and anatomical landmarks related to step 508 in FIG. 5A, reference is now made to FIG. 9A, which is a flowchart outlining the process of step 508. As shown in FIG. 9A, step 508 begins by determining an arbitrary posture P of the probe with respect to the anatomical structure of the patient to be positioned (step 700). Then, as can be understood from FIGS. 9A and 9B, a 3D CAD model of the probe 57 (i.e., 3D CAD probe model 57M) is placed in the arbitrary posture P determined with respect to the 3D CAD bone model 111 shown in FIG. 4E, and a transformation (T プローブ対3D画像 ) 701 is recorded. The transformation 701 maps the 3D CAD probe model 57M from the probe coordinate system CS プローブ to the 3D image coordinate space CS 3D画像 to the 3D CAD bone model 111 (step 702). Specifically, as a non-limiting example of a number of possible arbitrary postures P, as shown in FIG. 9B, the 3D CAD probe model 57M is arranged to point in a direction perpendicular to the center of the femoral anterior cortex of the 3D CAD femoral model 112 of the 3D CAD bone model 111 of the knee region 102 and face the center. Of course, any other arbitrary posture P suitable for a particular surgical application can be determined. Step 702 can be performed by a dedicated surgical planner or a surgeon. The 3D CAD bone model 111 of the knee region 102 can be a volume rendering, or other type of CT image or medical image, or a model determined therefrom.

[0170] As shown in FIG. 5A, once the generation of the initial positioning data is completed as described above with respect to step 508, the initial positioning data is acquired (step 610) as part of the workflow portion of the intraoperative aspect 502 of the ultrasonic-based multiple bone positioning process 503. Regarding the discussion of the intraoperative process of acquiring the initial positioning data according to step 610 in FIG. 5A, reference is now made to FIG. 10A, which is a flowchart outlining the process of step 610. As shown in FIG. 10A, step 610 begins by positioning the tracked probe 57 relative to the patient's anatomical structure according to the defined pose P of step 508 (step 704). In other words, in this example, for step 704, the pose P depicted in FIG. 9B between the 3D CAD probe model 57M and the 3D CAD femur model 112 of the 3D CAD bone model 111 is reproduced intraoperatively between the actual physical probe 57 and the patient's actual femur 11. Next, this intraoperative pose P is recorded (step 706).

[0171] As can be understood from FIG. 10B, the recording of the intraoperative pose P of step 706 can be achieved using the tracking camera 44 of the tracking system 42 to obtain the transformation (T プローブ対ナビゲーションカメラ ). Alternatively, as shown in FIG. 10C, the recording of the intraoperative pose P of step 706 can be achieved using the anatomical structure tracker 47 of the tracking system 42 to obtain the transformation (T プローブ対解剖学的構造のトラッカ ).

[0172] The process of step 610 concludes with the calculation of the initial positioning via 4x4 matrix multiplication, where T ナビゲーションカメラ対3D画像 = T プローブ対3D画像 * inv(T プローブ対ナビゲーションカメラ ) if the tracking camera 44 is employed as shown in FIG. 10B, or T 解剖学的構造のトラッカ対3D画像 = T プローブ対3D画像 * inv(T プローブ対解剖学的構造のトラッカIt is (step 710). As shown in FIG. 5A, this initial positioning data then enters the positioning module 612 as part of the data flow section of the intraoperative aspect 502 of the ultrasonic-based multiple bone positioning process 503.

[0173] As shown in FIG. 5A, the positioning module (step 612) can adopt any of a variety of alternative positioning processes, such as "one click / one pose" positioning (step 614), "landmark-based" positioning (step 616), or "anatomical structure tracker pin-based" positioning (618) in three non-limiting examples. Any of the three applications 614, 616, 618 can establish an initial positioning (i.e., "rough" estimation) of the transformation from the anatomical structure tracker space to the CAD model (CT / MRI) coordinate system. Other alternative positioning processes that can be part of the positioning model (step 612) can include, for example, "probe-based" positioning, "probe mini-sweep-based" positioning, and even using a calibrated digital camera to generate a photo of the patient's anatomical structure to be positioned and estimating the pose and location therefrom.

[0174] FIG. 11A is a flowchart of the process of the positioning surgical system 100 utilizing "one click / one pose" positioning (step 614), and FIGS. 11B and 11C are pictorial depictions of the process of FIG. 11A. As shown in FIGS. 11A and 11B, this process (step 614) starts by using the tracked probe 57 in FIGS. 1, 10B, and 10A during surgery to record the femoral 11, tibial 10, and optionally patellar bone surface points to generate two point clouds 620, 622 (step 624). In this step 624, one point cloud (i.e., the femoral tracker-related point cloud ("FTRPC") 620 is obtained with respect to the femoral tracker 47 in FIG. 1), and the other point cloud (i.e., the tibial tracker-related point cloud ("TTRPC") 622 is obtained with respect to the tibial tracker 46) (step 624). In other words, substantially two point clouds 620, 622 are obtained, where one point cloud 620 is related to the femoral tracker 47 and the other point cloud 622 is related to the tibial tracker 46.

[0175] As can be understood from FIG. 11B, the FTRPC 620 has femoral data points 620F, patellar data points 620P, and tibial data points 620T, but none of these data points have yet been so identified and are simply data points of the overall FTRPC 620. Similarly, the TTRPC 622 has femoral data points 622F, patellar data points 622P, and tibial data points 622T, but none of these data points have yet been so identified and are simply data points of the overall TTRPC 622.

[0176] As shown in FIGS. 11A and 11B, the "one click / one pose" positioning process (step 614) continues, and the classification algorithm is applied to the FTRPC 620 and TTRPC 622 during the operation to output a point cloud of only the femur ( "FOPCRFT") 626 for the femur tracker and a point cloud of only the tibia ( "TOPCRTT") 628 for the tibia tracker (step 630). This separation of the point cloud is advantageous because the knee pose in the preoperative 3D CAD bone model may be different from the intraoperative knee pose.

[0177] As can be understood from FIGS. 11A and 11C, after step 630, the "one click / one pose" positioning process (step 614) continues, and positioning conversions 632, 634 are calculated during the operation for the femur 11 and the tibia 10 respectively (step 636). This calculation of the positioning conversion may employ a positioning algorithm (e.g., "iterative closest point"). For example, to obtain the positioning of the femur (T 大腿骨のトラッカ対CAD大腿骨モデル ), the bone surface points of the FOPCRFT 626 are matched on the femur CAD model 112 (step 638). Similarly, to obtain the positioning of the tibia (T 脛骨のトラッカ対CAD脛骨モデル ), the bone surface points of the TOPCRTT 628 are matched on the tibia CAD model 114 (step 640). This "one click / one pose" positioning of step 614 is beneficial to at least partially facilitate the intraoperative ultrasonic surface capture process of step 512 in which multiple bone surfaces are simultaneously acquired via the ultrasonic probe 55.

[0178] FIG. 12 is a flowchart of a process of a positioning surgical system 100 that utilizes landmark-based positioning (step 616). As shown in FIG. 12, this process (step 616) begins by determining X-Y-Z coordinates for three or more anatomical landmarks on each 3D CAD model of the bone within the CAD model space to generate a first point set (step 750). In this example where the surgery is in the context of knee arthroplasty and the bones are the tibia 10 and the femur 11 (see FIG. 1), this step 750 generates a first point set by determining X-Y-Z coordinates for three or more anatomical landmarks on each of the 3D CAD femur model 112 and the 3D CAD tibia model 114. This step 750 can be achieved preoperatively or intraoperatively.

[0179] After step 750, a second point set is generated by digitizing the anatomical landmarks during the surgery via a navigated probe 57 that acquires the X-Y-Z coordinates in the tracker space of the anatomical structure for each anatomical landmark defined in step 750 (step 755). In other words, for step 755, the second points are digitized during the surgery at the landmarks on the actual tibia 11 and femur 12 that correspond to the landmarks defined in the respective 3D CAD tibia model 114 and 3D CAD femur model 112. For the final aspect of step 616, a typical point-to-point matching algorithm is used to match the first and second point sets to each other, resulting in an initial positioning for each bone (step 760). In an alternative embodiment, the point-to-point algorithm of step 760 may be replaced by a point-to-plane algorithm, and the preoperative 3D CAD femur model 112 and 3D CAD tibia model 114 do not have point clouds and are surface models.

[0180] In summary, the landmark-based positioning 616 described in FIG. 12 can be said to include digitization of landmarks in the CAD space (pre-operative planning) and the tracker space of the anatomical structure (intra-operative). Having these two sets of landmarks is efficient for calculating the initial positioning completed via the positioning module 612 when adopting the landmark-based positioning 616 (see FIG. 5A).

[0181] FIG. 13A is a flowchart of the process of the surgical system 100 for positioning using the tracker pin-based positioning of the anatomical structure (step 618). As shown in FIGS. 13A and 13B, this process (step 618) starts with the attachment of pins used to fixedly attach the anatomical structure trackers 46, 47 to the patient's bones (e.g., the tibia 10 and femur 11 in the context of this example of knee arthroplasty), and the pin attachment is performed in a consistent and repeatable manner for each type of surgical procedure (step 770). Next, the navigation system 42 (FIG. 1) tracks the anatomical structure trackers 46, 47 to obtain a rough estimate of the location of the patient's bones 10, 11 relative to the anatomical structure trackers 46, 47 (step 772).

[0182] As can be understood from FIGS. 13A and 13B, for the final aspect of step 618, the knee joint 762 is placed and its degrees of freedom are determined (step 774). In doing so, the patient's femur 11 and tibia 10 (FIG. 1) are jointed to each other for the knee 762 during the operation, and a second point cloud 764 referring to the trackers 46, 47 attached to the patient's tibia 10 and femur 11 is bent so that the tibia part 766 of the point cloud 764 and the femur part 768 of the point cloud 764 are jointed to each other at the knee joint 762. This joint is converted into the femur model 112 and the tibia model 114.

[0183] To summarize, the tracker pin-based positioning 618 of the anatomical structure described in FIGS. 13A and 13B adopts several assumptions regarding the typical locations where the tracker of the anatomical structure is placed and how the knee is bent (e.g., the tracker pin of the anatomical structure is attached to the anterior midshaft, and the knee is in mid-flexion (e.g., 30° - 70°)). By using this recognition, the location and orientation of the center of the knee joint in 3D space relative to the tracker pin of the anatomical structure can be roughly estimated. Finally, this location and orientation are used to define the initial positioning transformation, which is efficient for calculating the initial positioning completed via the positioning module 612 when adopting the tracker pin-based positioning 618 of the anatomical structure (see FIG. 5A).

[0184] As shown in FIG. 5A, once the positioning module (step 612) completes the initial or "rough guess" positioning process via any of the three alternative positioning processes, namely, the "1 click / 1 pose" positioning (step 614), the "landmark-based positioning" (step 616), or the "tracker pin of anatomical structure-based" positioning (618), the positioning module outputs the initial or "rough guess" positioning data (e.g., the guess of the transformation from trackers 46, 47 to 3D CAD bone models 111, 112, 113, 114) (step 776).

[0185] iii. Calculation of final multiple bone positioning

[0186] As can be understood from FIGS. 5A and 5B, the initial positioning data of step 776 is utilized using the classified 3D bone surface point cloud 598 of step 600 to calculate the final positioning of multiple bones (step 900), as will be discussed herein with respect to FIGS. 14A - 14C. Here, FIG. 14A is a flowchart of the process for step 900 in FIG. 5A, and FIGS. 14B and 14C are image depictions of aspects of the process of FIG. 14A. As shown in FIGS. 14A and 14B, the initial or approximate positioning data 902 of step 776 in FIG. 5A is applied to the classified 3D point cloud 598 of step 600 in FIG. 5A with reference to the femur tracker 47 (step 904). As described above with respect to FIG. 5B and shown in FIG. 14B, the classified 3D point cloud 598 has femur points 602 classified as the femur, patella points 604 classified as the patella, and tibia points 606 classified as the tibia. As described above in the context of the initial positioning module of step 612 in FIG. 5A and depicted in FIG. 14B, the initial positioning data 902 includes a femur point cloud 912, an optional patella point cloud 913, and a tibia point cloud 914, each generated via any one of the three initial positioning processes 614, 616, 618 of the initial positioning module of step 612. Each point cloud 912, 913, 914 is positioned to the applicable 3D CAD bone models 112, 113, 114 via the initial positioning model of step 612 and the output therefrom related to step 776.

[0187] As shown in FIG. 14A, the final positioning process (step 900) continues, and the closest points of the classified 3D point clouds 602, 604, 606 to the points of the point clouds 912, 913, 914 on the 3D CAD bone models 112, 113, 114 of the initial positioning or transformation are repeatedly calculated (step 920). More specifically, the positioning transformation updates from the CAD model space to the tracker coordinate system of the anatomical structure by matching the points of the classified 3D point clouds 602, 604, 606 to the points of the point clouds 912, 913, 914 on the 3D CAD bone models 112, 113, 114 of the initial positioning or transformation (step 925). After step 925, based on the point-to-closest-point distance analysis, due to step 925, the classification of the points of the classified 3D point clouds is updated, and a penalty is given to any bone-to-bone interference (step 930). Then, a check is made to determine whether convergence has been achieved between the classified 3D point clouds 602, 604, 606 and the point clouds 912, 913, 914 on the 3D CAD bone models 112, 113, 114 (step 935). If convergence has not yet been achieved, the calculation of the final positioning of the multiple bones returns from the convergence check in step 935 to step 920. If convergence is achieved, the final positioning is completed (step 940), and the converged initial or approximate positioning data (e.g., the 3D CAD bone models 112, 113, 114 and the point clouds 912, 913, 914 thereon) is finally positioned with the classified 3D point clouds 602, 604, 606, and the two sets of point clouds 912, 913, 914 and 602, 604, 606 respectively coincide with each other and generally have the same spread for the 3D CAD bone models 112, 113, 114 as depicted in FIG. 14C. When the final positioning is achieved according to step 900 of FIG. 5A, as shown in FIG. 2, the surgical system 100 and the procedure then proceed from positioning (step 805) to navigation (step 805), etc., and can utilize the final positioning data from step 805 required throughout the surgery on the patient via the surgical system 100.

[0188] As a refinement in the final positioning process (900) of FIG. 5A described immediately above with respect to FIGS. 14A-14C, in one embodiment, the final positioning process 900 continues and repeatedly calculates the nearest points of the classified 3D point clouds 602, 604, 606 to the triangulated mesh bone surfaces on the 3D CAD bone models 112, 113, 114 of the initial positioning or transformation. More specifically, the positioning transformation updates from the CAD model space to the tracker coordinate system of the anatomical structure by matching the points of the classified 3D point clouds 602, 604, 606 to the triangulated mesh bone surfaces on the 3D CAD bone models 112, 113, 114 of the initial positioning or transformation. The final positioning process 900 continues and executes an optimization algorithm that minimizes the cost function. In one embodiment, this cost function, which depends on the positioning matrix that exists at that time, is a weighted sum of the following various terms. (1) The minimum distance to the nearest triangulated mesh bone surface on the 3D CAD bone models 112, 113, 114 of the initial positioning or transformation after application of the positioning matrix for each point of the classified 3D point clouds 602, 604, 606, (2) A fixed penalty assigned to the 3D CAD bone models 112, 113, 114 that do not match their initial guess to prevent the 3D CAD bone models 112, 113, 114 from being exchanged for each point of the classified 3D point clouds 602, 604, 606, (3) A fixed penalty for each point or location on the triangulated mesh surface of each 3D CAD bone model 112, 113, 114 if this point or location is within another 3D CAD bone model 112, 113, 114 to avoid bone collisions, and (4) A penalty term regarding the magnitude of the transformation / positioning applied in addition to the initial positioning for each degree of freedom of the required positioning, assuming that the initial positioning is accurate enough that there is no need to deviate from it. In one embodiment, since the bone assignment is implicitly calculated within the first step, there is no need to explicitly alternate between the optimization of the point cloud assignment and the optimization of the transformation as may be required when employing the iterative closest point algorithm.

[0189] The final positioning process 900 continues, and a check is performed to determine whether convergence has been achieved between the classified 3D point clouds 602, 604, 606 and the triangulated mesh bone surfaces on the 3D CAD bone models 112, 113, 114. If convergence has not yet been achieved, the calculation of the final positioning of the plurality of bones returns from the convergence check and reiterates the calculation of the nearest points of the classified 3D point clouds 602, 604, 606 with respect to the triangulated mesh bone surfaces on the initial positioning or transformed 3D CAD bone models 112, 113, 114, and continues through the remainder of the process described above until convergence is checked again.

[0190] If convergence is achieved, the final positioning is complete, and the converged initial or approximate positioning data (e.g., the triangulated mesh bone surfaces of the 3D CAD bone models 112, 113, 114) are finally positioned with the classified 3D point clouds 602, 604, 606, and the point clouds 912, 913, 914 each have a consistent and generally the same spread for each area of the triangulated mesh bone surfaces of the 3D CAD bone models 112, 113, 114. When the final positioning is achieved according to step 900 of FIG. 5A, again, as shown in FIG. 2, the surgical system 100 and the procedure then proceed from positioning (step 805) to navigation (step 805), etc., and can utilize the final positioning data from step 805 required throughout the surgery on the patient via the surgical system 100.

[0191] The positioning process disclosed herein is advantageous in that it does not require a single consistent positioning of a bone such that there are no overlapping bones in the resulting positioning. Further, the process is flexible / user-friendly and provides a faster workflow because medical professionals do not need to avoid scanning multiple bones. The process is also not affected by outliers from other bones.

[0192] Thus, when only one bone is positioned, the user need not avoid accidentally scanning another nearby bone.

[0193] Finally, the positioning process disclosed herein is advantageous because it is independent of the incision size of the procedure, which is not the case when using positioning processes known in the art. This is particularly useful for hip and shoulder procedures and even more useful for ankle procedures, as the incision for such procedures is very small and it becomes difficult to access the relevant bone surfaces using typical digitizing tools (navigated pointers, sharp probes, etc.). Advantageously, ultrasound allows access to substantially all of the bone structure of the entire bone.

[0194] Furthermore, the positioning process disclosed herein is advantageous because it is not limited to full robotic use or robot-assisted use. Specifically, the positioning process can also be any navigated surgery that employs preoperative imaging. By way of example, the positioning process can be employed as part of a navigated cutting jig application, a navigated ACL reconstruction, or even a navigated procedure for removing osteosarcoma.

[0195] IV. Positioning System for Confirming Surgical Goals

[0196] For example, there continues to be a high concern for minimizing the risk of completing a surgical procedure on the wrong side of the patient, such as performing arthroplasty on the patient's right knee when the surgery was scheduled for the left knee. Thus, there is a need for a positioning system 1500 that can be used to quickly verify or confirm that the surgical team is operating on the correct target before taking any significant steps during the execution of the surgery.

[0197] FIG. 15 is a diagram of a positioning system 1500. As shown in FIG. 15, the positioning system 1500 includes a navigation or tracking system 42, a computer 50, and positioning tools 55, 57. The navigation or tracking system 42 includes a detection device 44 that tracks the positioning tools 55, 57, and the computer 50 includes an input device and a display 56. The positioning tools, which can be in the form of an ultrasonic probe 55 and / or a stylus 57 to be tracked, can be used to image and / or touch specific anatomical landmarks of the patient in the vicinity of the surgical target 1502 when positioning the patient's anatomical structure with respect to a patient-specific model and / or image generated preoperatively of the patient's anatomical structure. All of these components of the positioning system 1500 are configured to function in substantially the same manner as the elements of the surgical system 100 of FIG. 1 described above.

[0198] The navigation or tracking system 42 tracks the positioning tools 55, 57 utilized in the positioning of the patient's surgical target 1502 to confirm that the surgical target is the correct one. In FIG. 15, the surgical target 1502 is the patient's knee 1502, but it could also be the shoulder, elbow, hip, ankle, spine, etc.

[0199] During operation, the positioning system 1500 can be used prior to a robotic or robot-assisted surgery performed using the above-described surgical system 100 of FIG. 1. Similarly, the positioning system 1500 can be used prior to a conventional non-robotic surgery. In either case, a healthcare provider can utilize the positioning system 1500 preoperatively on a patient to correctly identify the intended surgical target 1502. For example, in the context of a knee arthroplasty or other arthroplasty, the positioning system 1500 is used to distinguish the target knee 1502 from other non-target knees by scanning and / or touching landmarks on the patient's tibia 10 and / or femur 11 adjacent to the target knee 1502. In the context of a spinal procedure, the positioning system 1500 can be used to identify the bony boundaries of the vertebrae and to identify the appropriate vertebral height that is the target of the surgery. In either case, the positioning system is employed to determine the correct location for a first incision and subsequent incisions.

[0200] In one embodiment, preoperative positioning for surgical target confirmation can occur by holding the presumed patient surgical target 1502 stationary and scanning the presumed patient surgical target 1502 using a tracked ultrasound probe. The resulting image is processed and positioned via computer 50 into a preoperative patient-specific image or computer model of the patient's surgical target according to the method outlined in FIG. 5A and described in detail above. If the presumed patient surgical target 1502 is successfully positioned into the preoperative patient-specific image or computer model of the patient's surgical target, this confirms that the presumed patient surgical target 1502 is in fact the correct surgical target. Then, a robotic, robot-assisted, or conventional surgery can be performed at the correctly identified surgical target.

[0201] V. Preferred Computing System

[0202] With reference to FIG. 16, a detailed description of an exemplary computing system 1300 having one or more computing units that may implement the various systems and methods discussed herein is provided. Computing system 1300 may be applicable to any of a computer or system utilized in preoperative planning, positioning, and postoperative analysis of an arthroplasty procedure, as well as other computing or network devices. Specific implementations of the device may be of potentially different specific computing architectures, all of which are not specifically discussed herein but will be recognized as understood by those skilled in the art.

[0203] Computer system 1300 may be a computing system capable of executing a computer program product to execute a computer process. Data and program files may be input into computer system 1300, which internally executes a program to read the files. Some of the elements of computer system 1300 are shown in FIG. 16, including one or more hardware processors 1302, one or more data storage devices 1304, one or more memory devices 1308, and / or one or more ports 1308 - 1310. Additionally, other elements recognized by those skilled in the art may be included in computing system 1300 but are not explicitly depicted in FIG. 16 or further discussed herein. The various elements of computer system 1300 may communicate with each other via one or more communication buses, point-to-point communication paths, or other communication means not explicitly depicted in FIG. 16.

[0204] Processor 1302 may include, for example, a central processing unit (CPU), a microprocessor, a microcontroller, a digital signal processor (DSP), and / or one or more levels of internal cache. There may be one or more processors 1302, and as a result, processor 1302 may comprise a single central processing unit or multiple processing devices that are commonly referred to as a parallel processing environment and that can execute instructions and perform operations in parallel with each other.

[0205] The computer system 1300 can be any other type of computer, such as a conventional computer, a distributed computer, or one or more external computers made available via a cloud computing architecture. The technology described herein can optionally be implemented in software stored on the data storage device 1304, stored on the memory device 1306, and / or communicated via one or more of the ports 1308 - 1310, thereby transforming the computer system 1300 in FIG. 16 into a dedicated machine implementing the operations described herein. Examples of the computer system 1300 include personal computers, terminals, workstations, mobile phones, tablets, laptops, personal computers, multimedia consoles, gaming consoles, set-top boxes, and the like.

[0206] One or more data storage devices 1304 can include any non-volatile data storage device capable of storing data generated or employed within computing system 1300, such as computer-executable instructions that execute a computer process that can include instructions of both an application program and an operating system (OS) that manages various components of computing system 1300. Without limitation, data storage device 1304 can include magnetic disk drives, optical disk drives, solid state drives (SSDs), flash drives, and the like. Data storage device 1304 can include removable data storage media, non-removable data storage media, and / or external storage devices made available via a wired or wireless network architecture using such computer program products that include one or more database management products, web server products, application server products, and / or other further software components. Examples of removable data storage media include compact disc read-only memory (CD-ROM), digital versatile disc read-only memory (DVD-ROM), magneto-optical discs, flash drives, and the like. Examples of non-removable data storage media include internal magnetic hard disks, SSDs, and the like. One or more memory devices 1306 can include volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and / or non-volatile memory (e.g., read-only memory (ROM), flash memory, etc.).

[0207] A computer program product including a mechanism for implementing a system and method in accordance with the present technology described may reside in a data storage device 1304 and / or a memory device 1306, which may be referred to as a machine-readable medium. It will be appreciated that the machine-readable medium can include any tangible non-transitory medium that can store or encode instructions for performing any one or more of the operations of the present disclosure for machine execution, or can store or encode data structures and / or modules utilized by or associated with such instructions. The machine-readable medium can include a single medium or multiple media (e.g., a centralized or distributed database and / or associated cache and server) that store one or more executable instructions or data structures.

[0208] In some implementations, computer system 1300 includes one or more ports such as input / output (I / O) port 1308 and communication port 1310 that communicate with other computing, network, or vehicle devices. It will be appreciated that ports 1308 - 1310 may be combined or separated, and that more or fewer ports may be included in computer system 1300.

[0209] I / O port 1308 may be connected to an I / O device or other device, thereby enabling information to be input into or output from computing system 1300. By way of non-limiting example, such I / O devices can include one or more input devices, output devices, and / or other devices.

[0210] In one implementation, the input device converts signals generated by a human, such as a human voice, physical movement, physical touch or pressure, and / or the like, into electrical signals as input data to the computing system 1300 via the I / O port 1308. Similarly, the output device can convert the electrical signals received from the computing system 1300 via the I / O port 1308 into signals that can be perceived by a human as output, such as sound, light, and / or touch. The input device can be an alphanumeric input device that communicates a selection of information and / or commands to the processor 1302 via the I / O port 1308 and includes alphanumeric and other keys. The input device can be a direction and selection control device, such as a mouse, trackball, cursor direction keys, joystick, and / or wheel, one or more sensors, such as a camera, microphone, position sensor, orientation sensor, gravity sensor, inertial sensor, and / or accelerometer, and / or another type of user input device including, but not limited to, a touch-sensitive display screen (“touch screen”). Without limitation, the output device can include a display, touch screen, speaker, tactile and / or haptic output device, and / or the like. In some implementations, the input device and the output device can be the same device, for example, in the case of a touch screen.

[0211] In one implementation, communication port 1310 is connected to a network, whereby computer system 1300 can receive network data useful for the execution of the methods and systems described herein, as well as the transmission of the information and network configuration changes determined thereby. Stated differently, communication port 1310 connects computer system 1300 to one or more communication interface devices configured to transmit and / or receive information between computing system 1300 and other devices via one or more wired or wireless communication networks or connections. By way of non-limiting example, such networks or connections include Universal Serial Bus (USB), Ethernet, Wi-Fi, Bluetooth®, Near Field Communication (NFC), Long Term Evolution (LTE), and the like. One or more such communication interface devices may be utilized via communication port 1310 to communicate directly with one or more other machines on a point-to-point communication path, on a wide area network (WAN) (e.g., the Internet), on a local area network (LAN), on a cellular (e.g., third generation (3G) or fourth generation (4G)) network, or on another communication means. Additionally, communication port 1310 may communicate with an antenna or other link for the communication and / or reception of electromagnetic signals.

[0212] In an exemplary implementation, patient data, bone models (e.g., generic, patient-specific), conversion software, positioning software, implant models, and other software, as well as other modules and services, may be embodied by instructions stored on data storage device 1304 and / or memory device 1306 and executed by processor 1302. Computer system 1300 may be integrated as part of surgical system 100 or, in other cases, may form part of the same.

[0213] However, the system described in FIG. 16 is an example of a possible computer system that may adopt or be configured in accordance with aspects of the present disclosure. It will be recognized that other non-transitory tangible computer-readable storage media storing computer-executable instructions for implementing the techniques of the present disclosure on a computing system may be utilized.

[0214] In the present disclosure, the methods disclosed herein, for example, those particularly shown in FIGS. 5A - 14C, may be implemented as a set of instructions or software readable by a device. Further, it is understood that the specific order or hierarchy of steps in the disclosed methods are examples of exemplary approaches. Based on design preferences, it is understood that the specific order or hierarchy of steps in a method may be rearranged while remaining within the disclosed subject matter. The appended methods claim elements with various steps in a sample order and are not necessarily meant to be limited to the specific order or hierarchy presented.

[0215] The described disclosure including any of the methods described herein may be provided as a computer program product or software that may include a non-transitory machine-readable medium storing instructions for programming a computer system (or other electronic device) to execute a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer), such as software, a processing application. The machine-readable medium may include, but is not limited to, magnetic storage media, optical storage media, magneto-optical storage media, read-only memory (ROM), random access memory (RAM), erasable programmable memory (e.g., EPROM and EEPROM), flash memory, or other types of media suitable for storing electronic instructions.

[0216] Although the present disclosure has been described with reference to various implementations, it will be understood that such implementations are illustrative and that the scope of the present disclosure is not limited thereto. Numerous changes, modifications, additions, and improvements are possible. More generally, embodiments in accordance with the present disclosure are described in the context of particular implementations. The functions may be separated or combined into blocks differently, or described in different terms, in various embodiments of the present disclosure. Such and other changes, modifications, additions, and improvements may fall within the scope of the present disclosure as defined in the following claims.

[0217] Generally, the embodiments described herein are described with reference to particular embodiments, but modifications thereto can be made without departing from the spirit and scope of the present disclosure. It should also be noted that the term "comprising" as used herein is inclusive, that is, it is intended to mean "including but not limited to".

[0218] The structures and arrangements of the systems and methods as shown in various preferred embodiments are merely illustrative. Although only some embodiments are described in detail in the present disclosure, numerous modifications are possible (for example, changes in the sizes, dimensions, structures, shapes, and ratios of various elements, the values of parameters, mounting arrangements, use of materials, colors, orientations, etc.). For example, the positions of the elements may be reversed or otherwise changed, and the nature or number of individual elements or positions may be modified or changed. Accordingly, all such modifications are intended to be included within the scope of the present disclosure. The order or sequence of any process or method steps may be changed or reordered according to alternative embodiments. Other substitutions, modifications, changes, and omissions may be made in the design, operating conditions, and arrangements of the preferred embodiments without departing from the scope of the present disclosure. The invention disclosed herein includes the following. [Aspect 1] A surgical system configured to process ultrasonic images of a patient's bone, the ultrasonic images including bone surfaces for each of the patient's bones, the system comprising A computing device comprising a processing device and a computer-readable medium storing one or more executable instructions, the processing device being configured to execute the one or more executable instructions, the one or more executable instructions being: i) to detect each bone surface of the patient's bones in the ultrasonic image; ii) to separate a first point cloud of ultrasonic image pixels associated with each bone surface of the patient's bones, a system. [Aspect 2] The system according to aspect 1, wherein the detection of the bone surface occurs via an image processing algorithm forming at least a part of the one or more executable instructions. [Aspect 3] The system according to aspect 2, wherein the image processing algorithm includes a machine learning model. [Aspect 4] The system according to aspect 2, wherein the separation of the first point cloud occurs via a pixel classification neural network forming at least a part of the one or more executable instructions. [Aspect 5] The system according to aspect 2, wherein the separation of the first point cloud occurs via an image-based classification neural network forming at least a part of the one or more executable instructions. [Aspect 6] The processing device executes the one or more executable instructions to calculate a transformation of the first point cloud to a separated 3D point cloud, the separated 3D point cloud being separated such that the ultrasonic image pixels of the separated 3D point cloud are each mutually associated with a corresponding bone surface of the patient's bones, the system according to aspect 1. [Aspect 7] When calculating the transformation of the first point cloud to the separated 3D point cloud, the ultrasonic image pixels are calibrated to an ultrasonic probe tracker, and the ultrasonic probe tracker is calibrated to a tracking camera, the system according to aspect 6. [Aspect 8] The system according to aspect 7, wherein when calibrating the ultrasonic image pixels to the ultrasonic probe tracker, the propagation speed of ultrasonic waves in a specific medium is considered. [Aspect 9] The system according to aspect 6, wherein when calculating the transformation of the first point cloud to the separated 3D point cloud, the ultrasonic image pixels are calibrated to an ultrasonic probe tracker, the ultrasonic probe tracker is calibrated to a tracking camera, and the coordinate system is with respect to the bone surface via a tracker of an anatomical structure disposed on the bone surface of the patient's bone. [Aspect 10] The system according to aspect 6, wherein the separation of the first point cloud occurs via a geometric analysis of the first point cloud. [Aspect 11] The system according to aspect 6, wherein the one or more executable instructions calculate an initial or approximate positioning of a second point cloud obtained from the patient's bone to a bone model of the patient's bone. [Aspect 12] The system according to aspect 11, wherein the second point cloud includes a plurality of point clouds for a plurality of trackers in the patient's bone. [Aspect 13] The system according to aspect 12, wherein the plurality of point clouds includes one point cloud positioned to a certain bone model of the bone model of the patient's bone and another point cloud positioned to another bone model of the bone model of the patient's bone. [Aspect 14] The system according to aspect 11, wherein the initial or approximate positioning is marker-based. [Aspect 15] The system according to aspect 11, wherein the initial or approximate positioning is calculated from the position and orientation of a tracker of an anatomical structure. [Aspect 16] When calculating the initial or approximate positioning, a third point cloud and a fourth point cloud are generated by the system, the third point cloud relates to a first bone of the patient's bone, and is for a first tracker associated with the first bone, and the fourth point cloud relates to a second bone of the patient's bone, and is for a second tracker associated with the second bone, the system according to aspect 11. [Aspect 17] When calculating the initial or approximate positioning, the system matches the bone surface points of the third point cloud onto the computer model of the first bone, and matches the bone surface points of the fourth point cloud onto the computer model of the second bone, the system according to aspect 16. [Aspect 18] The processing device executes the one or more instructions to calculate the final positioning of a plurality of bones by adopting the initial or approximate positioning and the separated 3D point cloud, and the final positioning of the plurality of bones achieves the final positioning between the separated 3D point cloud and the patient's bone, the system according to aspect 11. [Aspect 19] When calculating the final positioning of the plurality of bones where there is the final positioning between the classified 3D bone surface point cloud and the patient's bone, the system repeatedly improves the positioning of the separated 3D point cloud onto the computer model of the patient's bone, and repeatedly improves the separation of the separated 3D point cloud, the system according to aspect 18. [Aspect 20] A method for processing an ultrasonic image of a patient's bone, the ultrasonic image includes a bone surface for each of the patient's bones, and the method includes detecting the bone surface of each of the patient's bones in the ultrasonic image; and separating a first point cloud of ultrasonic image pixels associated with the bone surface of each of the patient's bones. The method includes. [Aspect 21] The detection of the bone surface is the method according to Aspect 20, which occurs via an image processing algorithm. [Aspect 22] The image processing algorithm is the method according to Aspect 21, which includes a machine learning model. [Aspect 23] The separation of the first point cloud occurs via a pixel classification neural network, which is the method according to Aspect 21. [Aspect 24] The separation of the first point cloud occurs via an image-based classification neural network, which is the method according to Aspect 21. [Aspect 25] The method according to Aspect 20 further includes calculating the conversion of the first point cloud to the separated 3D point cloud, and the separated 3D point cloud is separated such that each of the ultrasonic image pixels of the separated 3D point cloud is mutually associated with the corresponding bone surface of the patient's bone. [Aspect 26] When calculating the conversion of the first point cloud to the separated 3D point cloud, the ultrasonic image pixels are calibrated to an ultrasonic probe tracker, and the ultrasonic probe tracker is calibrated to a tracking camera, which is the method according to Aspect 25. [Aspect 27] When calibrating the ultrasonic image pixels to the ultrasonic probe tracker, the propagation speed of ultrasonic waves in a specific medium is considered, which is the method according to Aspect 26. [Aspect 28] When calculating the conversion of the first point cloud to the separated 3D point cloud, the ultrasonic image pixels are calibrated to an ultrasonic probe tracker, and the ultrasonic probe tracker is calibrated to a tracking camera, and the coordinate system is with respect to the bone surface via a tracker of an anatomical structure arranged on the bone surface of the patient's bone, which is the method according to Aspect 25. [Aspect 29] The separation of the first point cloud occurs through geometric analysis of the first point cloud, according to the method described in aspect 25. [Aspect 30] The method according to aspect 25, further comprising calculating an initial or approximate positioning of a second point cloud obtained from the patient's bone onto a bone model of the patient's bone. [Aspect 31] The method according to aspect 30, wherein the second point cloud includes a plurality of point clouds for a plurality of trackers in the patient's bone. [Aspect 32] The method according to aspect 31, wherein the plurality of point clouds includes one point cloud positioned on a certain bone model of the bone model of the patient's bone and another point cloud positioned on another bone model of the bone model of the patient's bone. [Aspect 33] The method according to aspect 30, wherein the initial or approximate positioning is landmark-based. [Aspect 34] The method according to aspect 30, wherein the initial or approximate positioning is calculated from the position and orientation of trackers of anatomical structures. [Aspect 35] When calculating the initial or approximate positioning, a third point cloud and a fourth point cloud are generated. The third point cloud relates to a first bone of the patient's bone and is for a first tracker associated with the first bone. The fourth point cloud relates to a second bone of the patient's bone and is for a second tracker associated with the second bone, according to the method described in aspect 30. [Aspect 36] When calculating the initial or approximate positioning, the bone surface points of the third point cloud are matched onto the computer model of the first bone, and the bone surface points of the fourth point cloud are matched onto the computer model of the second bone, according to the method described in aspect 35. [Aspect 37] Further comprising calculating a final positioning of a plurality of bones by employing the initial or rough positioning and the separated 3D point cloud, the method according to aspect 30, wherein the final positioning of the plurality of bones achieves a final positioning between the separated 3D point cloud and the patient's bones. [Aspect 38] When calculating the final positioning of the plurality of bones where there is the final positioning between the classified 3D bone surface point cloud and the patient's bones, the positioning of the separated 3D point cloud to the computer model of the patient's bones is repeatedly improved, and the separation of the separated 3D point cloud is repeatedly improved, the method according to aspect 37.

Claims

1. A surgical system configured to process ultrasonic images of a patient's bone, wherein the ultrasonic images include bone surfaces for each of the patient's bones, and the surgical system comprises: at least one surgical tool; a processing device; and a computing device including a computer-readable medium storing one or more executable instructions, wherein the processing device is configured to execute the one or more executable instructions, and the one or more executable instructions are: i) detecting each of the bone surfaces of the patient's bones in the ultrasonic image as ultrasonic image pixels; ii) converting the ultrasonic image pixels into 3D points; iii) generating a 3D point cloud classified for each bone surface; iv) obtaining the point cloud by utilizing the tracking of the three-dimensional position of a tracker installed on the patient's bone; v) finally positioning the point cloud obtained by utilizing the tracking of the three-dimensional position of a tracker installed on the patient's bone and positioned with respect to the bone model of the patient's bone with the 3D point cloud classified for each bone surface, and for the bone model of the patient's bone, the point cloud obtained by utilizing the tracking of the three-dimensional position of a tracker installed on the patient's bone and positioned with respect to the bone model of the patient's bone coincides with the 3D point cloud classified for each bone surface; wherein the at least one surgical tool communicates with the computing device, and the final positioning of the plurality of bones serves as an input for navigating the at least one surgical tool associated with each of the patient's bones. A system.

2. The system according to claim 1, wherein the detection of the bone surface occurs via an image processing algorithm forming at least a part of the one or more executable instructions.

3. The system according to claim 2, wherein the image processing algorithm includes a machine learning model.

4. A method for processing ultrasonic images of a patient's bone, wherein the ultrasonic images include bone surfaces for each of the patient's bones, and the method comprises: detecting each of the bone surfaces of the patient's bones in the ultrasonic image as ultrasonic image pixels; converting the ultrasonic image pixels into 3D points; generating a 3D point cloud classified for each bone surface; Obtaining a point cloud by using the tracking of the three-dimensional position of a tracker installed on the bone of the patient; Finally, the point cloud obtained by using the tracking of the three-dimensional position of the tracker installed on the bone of the patient and positioned with respect to the bone model of the bone of the patient is positioned with the 3D point cloud classified into each bone surface, and for the bone model of the bone of the patient, the point cloud obtained by using the tracking of the three-dimensional position of the tracker installed on the bone of the patient and positioned with respect to the bone model of the bone of the patient and the 3D point cloud classified into each bone surface match; comprising; A method of using the final positioning of the plurality of bones to navigate at least one surgical tool associated with each of the bones of the patient.

5. The method according to claim 4, wherein the detection of the bone surface occurs via an image processing algorithm.

6. The method according to claim 5, wherein the image processing algorithm includes a machine learning model.

Images