Method and apparatus for three-dimensional reconstruction

Abstract

A method for generating a three-dimensional model of a skeletal system, reconstructing a three-dimensional bone and soft tissue model, and related apparatus are disclosed. An exemplary method for generating a virtual 3-D patient-specific bone model may include obtaining a preliminary 3-D bone model of a first bone, obtaining an auxiliary image of the first bone, aligning the preliminary 3-D bone model of the first bone with the auxiliary image of the first bone, extracting geometric information about the first bone from the auxiliary image of the first bone, and / or generating a virtual 3-D patient-specific bone model of the first bone by improving the preliminary 3-D bone model of the first bone using geometric information about the first bone from the auxiliary image of the first bone.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 364,656, titled "METHODS AND APPARATUS FOR THREE - DIMENSIONAL RECONSTRUCTION USING MULTIPLE IMAGING MODALITIES," filed on May 13, 2022, which is incorporated herein by reference in its entirety.

[0002] This disclosure generally relates to methods for generating three - dimensional virtual models of the musculoskeletal system, and more particularly, to the reconstruction of three - dimensional bone and soft tissue models, and related apparatuses.

[0003] This disclosure contemplates that three - dimensional ("3 - D") models of tissue structures, such as the musculoskeletal system (e.g., bone, ligament, tendon, and / or cartilage), can be used in connection with the diagnosis and / or treatment related to such musculoskeletal systems. For example, a 3 - D bone model can be used in connection with orthopedic surgery for purposes such as preoperative planning, intraoperative surgical navigation, intraoperative bone formation, and / or postoperative evaluation.

Background Art

[0004] This disclosure contemplates that various imaging modalities that may be used in connection with tissue structures can be associated with several potential advantages and / or potential disadvantages. For example, in the field of orthopedics, ultrasound imaging may assist in extremely accurate 3-D surface mapping and may not expose the patient or nearby people to ionizing radiation. However, ultrasound may generally be limited to imaging external features of bone. More specifically, ultrasound may have limited ability to image some tissue structures, such as internal features of bone and / or external features of bone that are blocked by other bone. As another example, X-ray imaging and / or fluoroscopic imaging may enable visualization of internal features of bone and / or portions of bone that are blocked by other bone. However, these modalities may expose the patient and nearby people to ionizing radiation. Additionally, most common X-ray imaging and fluoroscopic imaging techniques provide only two-dimensional imaging.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0006]

Non-Patent Document 1

Non-Patent Document 2

[0007] Therefore, there is a need for improved methods and apparatuses related to 3-D imaging of skeletal features. [Means for Solving the Problems]

[0008] One aspect of the present disclosure provides a method for generating a virtual 3-D patient-specific bone model, the method comprising obtaining a preliminary virtual 3-D bone model of a first bone, obtaining an auxiliary image of the first bone, aligning the preliminary virtual 3-D bone model of the first bone with the auxiliary image of the first bone, extracting geometric information about the first bone from the auxiliary image of the first bone, and / or improving the preliminary virtual 3-D bone model of the first bone using the geometric information about the first bone from the auxiliary image of the first bone to generate an improved virtual 3-D patient-specific bone model of the first bone.

[0009] In a detailed embodiment, the step of obtaining a preliminary 3-D bone model may include the step of obtaining a point cloud of a first bone and the step of reconstructing a preliminary 3-D bone model by morphing a generalized 3-D bone model using the point cloud of the first bone. The step of obtaining a point cloud of a first bone may utilize a first imaging modality. The step of obtaining an auxiliary image of the first bone may utilize a second imaging modality. The first imaging modality may be different from the second imaging modality. The first imaging modality may include ultrasound. The second imaging modality may include 2-D X-ray.

[0010] In a detailed embodiment, the auxiliary image of the first bone may include at least one portion of the first bone that was not included in the point cloud of the first bone. The step of obtaining a point cloud of a first bone may include the step of performing an ultrasound scan of the first bone. The step of obtaining an auxiliary image of the first bone may include the step of obtaining a 2-D X-ray of the first bone. The 2-D X-ray of the first bone may include at least one portion of the first bone that was not available in the ultrasound scan of the first bone. At least one portion of the first bone that was not available in the ultrasound scan of the first bone may have been at least partially blocked from the ultrasound scan by tissue structure.

[0011] In a detailed embodiment, at least one portion of the first bone that was not available in the ultrasound scan of the first bone may include the internal structure of the first bone. The blocked internal structure of the first bone may include a medullary canal. The first bone may include a femur. The medullary canal may include a femoral medullary canal.

[0012] In a detailed embodiment, at least one portion of the first bone that was not visible in the ultrasound scan of the first bone may include the external structure of the first bone. The external structure of the first bone may have been at least partially blocked from the ultrasound scan by a second bone. One of the first bone and the second bone may include a femoral head, and the other of the first bone and the second bone may include an acetabular cup.

[0013] In a detailed embodiment, the external structure of the first bone that was blocked from the ultrasonic scan by the second bone may include soft tissue. The soft tissue may include cartilage. The cartilage may include hip joint cartilage. The cartilage may include knee joint cartilage. The cartilage may include shoulder joint cartilage.

[0014] In a detailed embodiment, each of the first bone and the second bone may include one or more of the pelvis, femur, tibia, patella, scapula, and humerus.

[0015] In a detailed embodiment, the first bone may include one or more of the pelvis, femur, tibia, patella, scapula, and humerus.

[0016] In a detailed embodiment, the step of aligning a preliminary 3-D bone model of the first bone with an auxiliary image of the first bone may include solving for the pose of the preliminary 3-D bone model that produces a 2-D projection corresponding to the projection of the auxiliary image.

[0017] In a detailed embodiment, the step of obtaining an auxiliary image of the first bone may include obtaining a plurality of auxiliary images of the first bone. The step of aligning a preliminary virtual 3-D bone model of the first bone with an auxiliary image of the first bone may include aligning the preliminary virtual 3-D bone model of the first bone with a plurality of auxiliary images of the first bone. The step of extracting geometric information about the first bone from the auxiliary image of the first bone may include extracting geometric information about the first bone from a plurality of auxiliary images of the first bone. The step of refining the preliminary virtual 3-D bone model of the first bone using the geometric information about the first bone from the auxiliary image of the first bone may include refining the preliminary virtual 3-D bone model of the first bone using the geometric information about the first bone from a plurality of auxiliary images of the first bone.

[0018] In a detailed embodiment, the method may include obtaining a preliminary virtual 3-D bone model of a second bone, obtaining an auxiliary image of the second bone, aligning the preliminary virtual 3-D bone model of the second bone with the auxiliary image of the second bone, extracting geometric information about the second bone from the auxiliary image of the second bone, and / or generating an improved virtual 3-D patient-specific bone model of the second bone by improving the preliminary virtual 3-D bone model of the second bone using the geometric information about the second bone from the auxiliary image of the second bone.

[0019] In a detailed embodiment, the step of obtaining a point cloud of the second bone may include performing an ultrasonic scan of the second bone. The step of obtaining an auxiliary image of the second bone may include obtaining a 2-D X-ray of the second bone. The 2-D X-ray of the second bone may include at least one portion of the second bone that was not visible in the ultrasonic scan of the second bone.

[0020] In a detailed embodiment, the step of extracting geometric information from the auxiliary image of the first bone may include extracting at least one of a length dimension, an angular dimension, or a curvature of the first bone from the auxiliary image of the first bone.

[0021] In a detailed embodiment, a method for determining the size of an orthopedic implant preoperatively may include generating an improved virtual 3-D patient-specific bone model according to the method described above and / or determining the size of the orthopedic implant using the improved virtual 3-D patient-specific bone model.

[0022] In a detailed embodiment, the apparatus may be configured to perform the method described above. In a detailed embodiment, the memory may include instructions that, when executed by the processor, cause the processor to perform the method described above.

[0023] One aspect of the present disclosure provides a method for generating a virtual 3-D patient-specific bone model, the method including obtaining ultrasonic data regarding an external surface of a first bone, obtaining X-ray data regarding at least one of internal features of the first bone and / or occluded features of the first bone, and / or generating a 3-D patient-specific bone model of the first bone using the ultrasonic data and the X-ray data, the 3-D patient-specific bone model representing the external surface of the first bone and at least one of internal features of the first bone and / or occluded features of the first bone.

[0024] In a detailed embodiment, the step of obtaining ultrasonic data regarding an external surface of the first bone may include obtaining an ultrasonic point cloud of the external surface of the first bone and generating a preliminary 3-D bone model of the first bone. The step of generating a 3-D patient-specific bone model of the first bone may include improving the preliminary 3-D bone model of the first bone using the X-ray data.

[0025] In a detailed embodiment, an apparatus may be configured to perform the method described above. In a detailed embodiment, the memory may include instructions that, when executed by a processor, cause the processor to perform the method described above.

[0026] One aspect of the present disclosure provides a method for determining a spinal-pelvic tilt, the method including obtaining a virtual 3-D model of the pelvis, obtaining a first ultrasonic point cloud of the pelvis and a first ultrasonic point cloud of the lumbar spine in a state where the pelvis and the lumbar spine are in a first functional limb position, aligning the virtual 3-D model of the pelvis with the first point cloud of the pelvis, and / or determining a first spinal-pelvic tilt in the first functional limb position using a first relative angle of the first point cloud of the lumbar spine with respect to the 3-D model of the pelvis.

[0027] In a detailed embodiment, the method may include positioning at least one of the pelvis or the lumbar spine into the first functional limb position.

[0028] In a detailed embodiment, the method may include obtaining a second ultrasonic point cloud of the pelvis and a second ultrasonic point cloud of the lumbar spine while the pelvis and the lumbar spine are in a second functional limb position, aligning a virtual 3-D model of the pelvis with the second point cloud of the pelvis, and / or determining a second spinal-pelvic tilt in the second functional limb position using a second relative angle of the second point cloud of the lumbar spine with respect to the 3-D model of the pelvis.

[0029] In a detailed embodiment, the method may include positioning at least one of the pelvis or the lumbar spine into a second functional limb position.

[0030] In a detailed embodiment, the method may include obtaining a third ultrasonic point cloud of the pelvis and a third ultrasonic point cloud of the lumbar spine while the pelvis and the lumbar spine are in a third functional limb position, aligning a virtual 3-D model of the pelvis with the third point cloud of the pelvis, and / or determining a third spinal-pelvic tilt in the third functional limb position using a third relative angle of the third point cloud of the lumbar spine with respect to the 3-D model of the pelvis.

[0031] In a detailed embodiment, the method may include positioning at least one of the pelvis or the lumbar spine into a third functional limb position.

[0032] In a detailed embodiment, each of the first functional limb position, the second functional limb position, and the third functional limb position may include one of sitting position, standing position, and / or supine position.

[0033] In a detailed embodiment, the step of obtaining a virtual 3-D model of the pelvis may include generating a virtual 3-D model of the pelvis using ultrasound.

[0034] In a detailed embodiment, the step of obtaining a first ultrasonic point cloud of the pelvis and a first ultrasonic point cloud of the lumbar spine in the first functional limb position may include obtaining a sparse ultrasonic point cloud of the pelvis and a sparse ultrasonic point cloud of the lumbar spine.

[0035] In a detailed embodiment, at least one of the first ultrasound point cloud of the pelvis and the first ultrasound point cloud of the lumbar spine in a state where the pelvis and the lumbar spine are in the first functional limb position may include additional points regarding the femur. The method may include a step of determining at least one of a femoral tilt, an acetabular tilt, or a combined tilt. The step of determining at least one of a femoral tilt, an acetabular tilt, or a combined tilt may include a step of identifying a superior condylar axis or an inferior condylar axis of the femur to determine a reference axis of the femoral tilt angle.

[0036] In a detailed embodiment, the method may include a step of obtaining information regarding the leg length by obtaining data from at least one X-ray image taken while the subject is in a standing position.

[0037] In a detailed embodiment, the apparatus may be configured to implement the method described above. In a detailed embodiment, the memory may include instructions that, when executed by the processor, cause the processor to implement the method described above.

[0038] One aspect of the present disclosure provides a method for generating a virtual 3-D patient-specific bone model of a ligament, the method including obtaining a virtual 3-D patient-specific bone model of a joint, detecting at least one ligament point on the virtual 3-D patient-specific bone model, obtaining ultrasonic data regarding the ligament associated with at least one ligament point by scanning the ligament using ultrasound, and / or reconstructing a virtual 3-D model of the ligament using the ultrasonic data.

[0039] In a detailed embodiment, the step of obtaining ultrasonic data regarding the ligament may be performed at a plurality of joint angles of the joint over a range of motion of the joint.

[0040] In a detailed embodiment, the step of obtaining a virtual 3-D patient-specific bone model of a joint may include the step of reconstructing the joint using ultrasound. The step of reconstructing the joint using ultrasound may include the step of obtaining at least one point cloud associated with one or more bones of the joint.

[0041] In a detailed embodiment, the step of detecting at least one ligament point on a patient-specific virtual 3-D bone model may include the step of determining at least one insertion position of the ligament.

[0042] In a detailed embodiment, the step of scanning a ligament using ultrasound may include the step of providing automated guidance information. The step of providing automated guidance information may include the step of providing a display comprising the current placement of the ultrasound probe relative to one or more tissue structures. The step of providing automated guidance information may include the step of providing a display comprising an indication of a desired position or direction of the scan. The step of providing automated guidance information may include the step of providing a display comprising an A-mode ultrasound image or a B-mode ultrasound image.

[0043] In a detailed embodiment, the joint may include a knee. The ligament may include a medial collateral ligament.

[0044] In a detailed embodiment, the joint may include a knee. The ligament may include a lateral collateral ligament.

[0045] In a detailed embodiment, the apparatus may be configured to perform the methods described above. In a detailed embodiment, the memory may include instructions that, when executed by the processor, cause the processor to perform the methods described above.

[0046] One aspect of the present disclosure provides a method for generating a virtual 3-D patient-specific tissue model, the method comprising obtaining a preliminary virtual 3-D tissue model of a first tissue, obtaining an auxiliary image of the first tissue, aligning the preliminary virtual 3-D tissue model of the first tissue with the auxiliary image of the first tissue, extracting geometric information about the first tissue from the auxiliary image of the first tissue, and / or generating an improved virtual 3-D patient-specific tissue model of the first tissue by using the geometric information about the first tissue from the auxiliary image of the first tissue to improve the preliminary virtual 3-D tissue model of the first tissue.

[0047] In a detailed embodiment, the step of obtaining a preliminary 3-D tissue model may include obtaining a point cloud of the first tissue and reconstructing the preliminary 3-D tissue model by morphing a generalized 3-D tissue model using the point cloud of the first tissue. The step of obtaining a point cloud of the first tissue may utilize a first imaging modality. The step of obtaining an auxiliary image of the first tissue may utilize a second imaging modality. The first imaging modality may be different from the second imaging modality.

[0048] In a detailed embodiment, the first imaging modality may include ultrasound. The second imaging modality may include 2-D X-ray.

[0049] In a detailed embodiment, the auxiliary image of the first tissue may include at least one portion of the first tissue that was not included in the point cloud of the first tissue. The step of obtaining a point cloud of the first tissue may include performing an ultrasound scan of the first tissue. The step of obtaining an auxiliary image of the first tissue may include obtaining a 2-D X-ray of the first tissue. The 2-D X-ray of the first tissue may include at least one portion of the first tissue that was not available in the ultrasound scan of the first tissue. At least one portion of the first tissue that was not available in the ultrasound scan of the first tissue may have been at least partially blocked from the ultrasound scan by tissue structures.

[0050] In a detailed embodiment, at least one portion of the first tissue that was not available in the ultrasound scan of the first tissue may include the internal structure of the first tissue. The obscured internal structure of the first tissue may include a medullary canal. The first tissue may include a femur. The medullary canal may include a femoral medullary canal.

[0051] In a detailed embodiment, at least one portion of the first tissue that was not visible in the ultrasound scan of the first tissue may include the external structure of the first tissue. The external structure of the first tissue may have been at least partially obscured from the ultrasound scan by the second tissue. One of the first tissue and the second tissue may include a femoral head, and the other of the first tissue and the second tissue may include an acetabular cup.

[0052] In a detailed embodiment, the external structure of the first tissue that was obscured from the ultrasound scan by the second tissue may include soft tissue. The soft tissue may include cartilage. The cartilage may include hip joint cartilage. The cartilage may include knee joint cartilage. The cartilage may include shoulder joint cartilage.

[0053] In a detailed embodiment, each of the first tissue and the second tissue may include one or more of the pelvis, femur, tibia, patella, scapula, and / or humerus.

[0054] In a detailed embodiment, the first tissue may include one or more of the pelvis, femur, tibia, patella, scapula, and / or humerus.

[0055] In a detailed embodiment, the step of aligning a preliminary 3-D tissue model of the first tissue with an auxiliary image of the first tissue may include solving for the pose of the preliminary 3-D tissue model that produces a 2-D projection corresponding to the projection of the auxiliary image.

[0056] In a detailed embodiment, the step of obtaining the auxiliary image of the first tissue may include the step of obtaining a plurality of auxiliary images of the first tissue. The step of aligning the preliminary virtual 3-D tissue model of the first tissue with the auxiliary image of the first tissue may include the step of aligning the preliminary virtual 3-D tissue model of the first tissue with a plurality of auxiliary images of the first tissue. The step of extracting geometric information about the first tissue from the auxiliary image of the first tissue may include the step of extracting geometric information about the first tissue from a plurality of auxiliary images of the first tissue. The step of improving the preliminary virtual 3-D tissue model of the first tissue using the geometric information about the first tissue from the auxiliary image of the first tissue may include the step of improving the preliminary virtual 3-D tissue model of the first tissue using the geometric information about the first tissue from a plurality of auxiliary images of the first tissue.

[0057] In a detailed embodiment, the method may include the step of obtaining a preliminary virtual 3-D tissue model of a second tissue, the step of obtaining an auxiliary image of the second tissue, the step of aligning the preliminary virtual 3-D tissue model of the second tissue with the auxiliary image of the second tissue, the step of extracting geometric information about the second tissue from the auxiliary image of the second tissue, and / or the step of generating an improved virtual 3-D patient-specific tissue model of the second tissue by improving the preliminary virtual 3-D tissue model of the second tissue using the geometric information about the second tissue from the auxiliary image of the second tissue. The step of obtaining the point cloud of the second tissue may include the step of performing an ultrasonic scan of the second tissue. The step of obtaining the auxiliary image of the second tissue may include the step of obtaining a 2-D X-ray of the second tissue. The 2-D X-ray of the second tissue may include at least one part of the second tissue that was not visible in the ultrasonic scan of the second tissue.

[0058] In a detailed embodiment, the step of extracting geometric information from the auxiliary image of the first tissue may include the step of extracting at least one of the length dimension, the angular dimension, or the curvature of the first tissue from the auxiliary image of the first tissue.

[0059] In a detailed embodiment, a method for determining the size of an orthopedic implant preoperatively may include generating an improved virtual 3-D patient-specific tissue model according to the method described above, and / or determining the size of the orthopedic implant using the improved virtual 3-D patient-specific tissue model.

[0060] In a detailed embodiment, the apparatus may be configured to perform the method described above. In a detailed embodiment, the memory may include instructions that, when executed by the processor, cause the processor to perform the method described above.

[0061] In a detailed embodiment, the first tissue may include the first bone.

[0062] One aspect of the present disclosure provides any method, process, device, apparatus, or system related to any aspect or embodiment as described above or herein. One aspect of the present disclosure provides any combination of any elements of any of the preceding aspects or embodiments, or as described herein.

[0063] The accompanying drawings, which are incorporated herein and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the detailed description of the exemplary embodiments given below, serve to explain the principles of the present disclosure.

Brief Description of the Drawings

[0064]

Figure 1

Figure 2

Figure 2A

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 6F

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 10E

Figure 11

Figure 12A

Figure 12B

Figure 12C

Figure 12D

Figure 12E

Figure 13A

Figure 13B

Figure 13C

Figure 13D

Figure 13E

Figure 13F

Figure 14A

Figure 14B

Figure 14C

Figure 14D

Figure 14E

Figure 14F

Figure 15

Figure 16A

Figure 16B

Figure 16C

Figure 16D

Figure 17A

Figure 17B

Figure 17C

Figure 17D

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23A

Figure 23B

Figure 24A

Figure 24B

Figure 24C

Figure 24D

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29A

Figure 29B

Figure 30A

Figure 30B

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

[0065] The present disclosure includes, among other things, methods and apparatuses related to creating virtual models of tissue structures, such as generating 3-D models of musculoskeletal features. Some exemplary embodiments according to at least some aspects of the present disclosure are described and illustrated below as including devices, methods, and techniques related to generating virtual musculoskeletal models using multiple imaging modalities, such as ultrasonic imaging and X-ray imaging. It will be apparent to those skilled in the art that the embodiments discussed below are examples and may be reconfigured and combined without departing from the scope and spirit of the present disclosure. It should also be understood that variations of the exemplary embodiments contemplated by those skilled in the art may simultaneously incorporate portions of the present disclosure. However, for clarity and accuracy, exemplary embodiments as discussed below may include optional steps, methods, and features that those skilled in the art should recognize are not requirements for falling within the scope of the present disclosure.

[0066] Some exemplary embodiments according to at least some aspects of the present disclosure may utilize ultrasonic imaging in relation to reconstructing 3-D models of tissue structures. Accordingly, the following sections provide an explanation of exemplary methods and apparatuses for reconstructing 3-D models of joints (e.g., bones and / or soft tissues) using ultrasonic waves.

[0067] 3-D Reconstruction of Joints Using Ultrasonic Waves The reconstruction of 3-D models of joints, such as the bones of the knee joint, is an important component of computer-assisted joint surgery systems. Having a pre-operative acquired model enables the surgeon to pre-plan the surgery by selecting the appropriate implant size, providing the cutting planes of the femur and tibia in the case of knee surgery, and evaluating the fit of the selected implant. Conventional methods for generating 3-D models are the segmentation of computed tomography ("CT") or magnetic resonance imaging ("MRI") scans, which are the conventional imaging modalities for creating 3-D patient-specific bone models. The segmentation methods used are either manual, semi-automatic, or fully automatic. These methods produce highly accurate models, but CT and MRI have inherent drawbacks. Namely, they are both fairly expensive procedures (especially for MRI), and CT exposes the patient to ionizing radiation.

[0068] One alternative method for forming 3-D patient-specific models is to use previously acquired X-ray images as prior information to guide the morphing of a generalized bone model (whose projection matches the X-ray images). Several X-ray image-based model reconstruction methods have been developed for the femur (specifically, including the proximal and distal parts), pelvis, spine, and thorax.

[0069] Conventional ultrasonic imaging utilizes B-mode images. B-mode images are constructed by extracting the envelope of the received and scanned lines of radio frequency ("RF") signals using the Hilbert transform. These envelopes are then decimated (causing a reduction in resolution), converted to grayscale (where the intensity of each pixel is represented by 8 bits), and the final B-mode image is formed. The conversion to grayscale results in a reduction in the dynamic range of the ultrasonic data.

[0070] The use of ultrasound in computer-assisted orthopedic surgery has gained interest in recent years due to its relatively low cost and radiation-free nature. More specifically, A-mode ultrasound intraoperative alignment is used in computer-assisted orthopedic surgery and, in limited cases, in neurosurgery. Ultrasound-MRI alignment using B-mode ultrasound images has been developed. However, it has been found difficult to generate 3-D bone models with sufficient quality using conventional ultrasound techniques due to limitations in image quality.

[0071] Therefore, there is a need to develop improved devices and methods for constructing 3-D patient-specific bone and cartilage models using ultrasound techniques.

[0072] The present invention overcomes the above-described problems and other disadvantages, drawbacks, and challenges of high-cost or radiation-exposed imaging modalities and generates patient-specific models by ultrasound techniques. Although the present invention is described in connection with several embodiments, it will be understood that the present invention is not limited to these embodiments. On the contrary, the present invention includes all alternatives, modifications, and equivalents as may be included within the spirit and scope of the present invention.

[0073] According to one embodiment of the present invention, a method for generating a 3-D patient-specific bone model is described. The method includes obtaining a plurality of raw radio frequency (「RF」) signals from an A-mode ultrasound scan of the bone, the scan being spatially tracked in 3-D space. The bone contour is separated in each of the plurality of RF signals and converted into a point cloud. A 3-D patient-specific bone model of the bone is then optimized with respect to the point cloud.

[0074] According to another embodiment of the present disclosure, a method for 3-D construction of a bone surface includes imaging the bone using A-mode ultrasound. A plurality of RF signals are acquired during imaging. The imaging of the bone is also tracked. A bone contour is extracted from each of the plurality of RF signals. Then, using the tracked data and the extracted bone contour, a point cloud representing the surface of the bone is generated. A generalized model of the bone is morphed to match the surface of the bone as represented by the point cloud.

[0075] In yet another embodiment of the present invention, a computer method for simulating the surface of a bone is described. The computer method includes executing a computer program according to a process. This process includes extracting a bone contour from each of a plurality of A-mode RF signals. The extracted bone contour is transformed from a reference local frame to a point cloud in a reference world frame. A generalized model of the bone is compared to the point cloud, and the generalized model is deformed to match the point cloud as determined from this comparison.

[0076] Another embodiment of the present disclosure is directed to a computer program product including a non-transitory computer-readable medium and program instructions stored on the computer-readable medium. When executed by a process, these program instructions cause the computer program product to separate a bone contour from a plurality of RF signals. The plurality of RF signals were previously acquired from a reflected A-mode ultrasound beam. The bone contour is then transformed into a point cloud and used to optimize a 3-D model of the bone.

[0077] Yet another embodiment of the present invention is directed to a computing device having a processor and a memory. The memory includes instructions that, when executed by the processor, cause the processor to separate a bone contour from a plurality of RF signals. The plurality of RF signals were previously acquired from a reflected A-mode ultrasound beam. The bone contour is then transformed into a point cloud and used to optimize a 3-D model of the bone.

[0078] Various embodiments of the present invention are directed to a method of generating a 3-D patient-specific bone model. To generate the 3-D patient-specific model, a plurality of raw RF signals are acquired using an A-mode ultrasound acquisition method. Next, the bone contour is separated in each of the plurality of RF signals and converted into a point cloud. Then, the point cloud can be used to optimize the 3-D model of the bone so that a patient-specific model can be generated. Although various embodiments of the present invention are shown with respect to human patients herein, those skilled in the art will understand that the embodiments of the present invention can also be used to generate 3-D patient-specific bone models of animals (e.g., dogs, horses, etc.) for veterinary applications and the like.

[0079] Referring now to the figures, and specifically to FIG. 1, one embodiment of an ultrasound device 50 for use with one or more embodiments of the present invention is shown. The ultrasound device 50 should be configurable so that a user can access the acquired RF ultrasound data. One suitable device can be, for example, the ultrasound diagnostic model SonixRP by Ultrasonix Inc. (Richmond, British Columbia, Canada). The ultrasound device 50 includes a housing 52 that houses a controller (e.g., computer 54), an energy source or power supply (not shown), a user input device 56, an output device (e.g., monitor 58), and at least one ultrasound probe 60. The housing 52 can include caster wheels 62 for transporting the ultrasound device 50 within a medical facility.

[0080] At least one ultrasonic probe 60 is configured to acquire raw radio frequency (「RF」) signals of ultrasonic waves, and is shown in more detail in FIG. 2. The ultrasonic probe 60, such as the specific embodiment shown, can be a high-resolution linear transducer with a center frequency of 7.5 MHz, as conventionally used in musculoskeletal procedures. The sampling frequency used when digitizing ultrasonic echoes may be, for example, 20 MHz, and must be at least twice the maximum ultrasonic frequency. Generally, the ultrasonic probe 60 includes a body 64 coupled to the ultrasonic device housing 52 by a cable 66. The body 64 further includes a transducer array 68 configured to transmit ultrasonic pulses and receive the reflected ultrasonic RF energy. The received RF echoes are transmitted along the cable 66 to the computer 54 of the ultrasonic device 50 for processing according to certain embodiments of the present invention.

[0081] As shown in FIG. 3, the computer 54 of the ultrasonic device 50 can be regarded as representing any type of computer, computer system, computing system, server, disk array, or programmable device such as a multi-user computer, single-user computer, handheld device, network-connected device, or embedded device. The computer 54 can be implemented, for example, in a cluster or other distributed computing system through a network interface 76 (shown as 「network I / F」) using one or more networks 74 together with one or more network-connected computers 70 or network-connected storage devices 72. For the sake of brevity, the computer 54 is simply referred to as 「computer」, but it should be understood that the term 「computing system」 may also include other suitable programmable electronic devices that do not conflict with the embodiments of the present invention.

[0082] Computer 54 typically includes at least one processing unit 78 (shown as "CPU") coupled to memory 80, along with several different types of peripheral devices, such as mass storage device 82, user interface 84 (shown as user I / F, which may include input device 56 and monitor 58), network I / F 76, and input / output (IO) interface 85 for coupling computer 54 to additional devices such as the aforementioned ultrasonic device 50. Memory 80 may include dynamic random access memory ("DRAM"), static random access memory ("SRAM"), non-volatile random access memory ("NVRAM"), persistent memory, flash memory, at least one hard disk drive, and / or other digital storage media. Mass storage device 82 is typically at least one hard disk drive and may be located external to computer 54, such as in a separate enclosure, or in one or more of network-connected computers 70, or in one or more of network-connected storage devices 72 (e.g., servers).

[0083] In various embodiments, CPU 78 can be a single-threaded, multi-threaded, multi-core, and / or multi-element processing unit (not shown). In alternative embodiments, computer 54 can include multiple processing units that can include a single-threaded processing unit, multi-threaded processing unit, multi-core processing unit, multi-element processing unit, and / or combinations thereof. Similarly, memory 80 may include one or more levels of data, instruction, and / or combined cache, and the cache may service individual processing units or multiple processing units (not shown).

[0084] The memory 80 of the computer 54 may include an operating system 81 (shown as "OS") for controlling the main operations of the computer 54 in a manner known in the art. The memory 80 may also include at least one application, component, algorithm, program, object, module, or sequence of instructions, and even subsets thereof, which are referred to herein as "computer program code" or simply "program code" 83. The program code 83 is typically present in the memory 80 and / or the mass storage device 82 of the computer 54 at various times, and when read and executed by the CPU 78, causes the computer 54 to perform steps necessary to execute steps or elements that embody various aspects of the present invention, and includes one or more instructions.

[0085] The I / O interface 85 is configured to operably couple the CPU 78 to other devices and systems, including the ultrasonic device 50 and an optional electromagnetic tracking system 87 (FIG. 20). The I / O interface 85 may include signal processing circuitry that conditions incoming and outgoing signals so that the signals are compatible with both the CPU 78 and the components to which the CPU 78 is coupled. For this purpose, the I / O interface 85 may include conductors, analog-to-digital (A / D) and / or digital-to-analog (D / A) converters, voltage level and / or frequency shift circuits, optical isolation and / or driver circuits, and / or any other analog or digital circuitry suitable for coupling the CPU 78 to other devices and systems. For example, the I / O interface 85 may include one or more amplifier circuits for amplifying signals received from the ultrasonic device 50 prior to analysis in the CPU 78.

[0086] Those skilled in the art will recognize that the environment shown in FIG. 3 does not limit the present invention. In fact, those skilled in the art will recognize that other alternative hardware and / or software environments may be used without departing from the scope of the present invention.

[0087] Returning again to FIG. 2, the ultrasonic probe 60 is equipped with a tracking marker 86 shown as an optical marker for illustrative purposes only that is configured to spatially align the movement of the ultrasonic probe 60 during signal acquisition. The tracking marker 86 may consist of a plurality of reflective portions 90, which will be described in more detail below. The probe to be tracked is a component of the hybrid probe 94. In other embodiments, the tracking marker and associated system can be an electromagnetic system, an RF system, or any other known 3-D tracking system.

[0088] The optical tracking marker 86 is operably coupled to a position sensor 88, one embodiment of which is shown in FIG. 2A. During use, the position sensor 88 emits energy (e.g., infrared light) in a direction toward the optical tracking marker 86. The reflective portions 90 of the optical tracking marker 86 reflect the energy back to the position sensor 88, which then triangulates the 3-D placement and orientation of the optical tracking marker 86. An example of a suitable optical tracking system is the Polaris model manufactured by Northern Digital Inc. (Waterloo, Ontario, Canada).

[0089] The optical tracking marker 86 is rigidly attached to the ultrasonic probe 60, providing a reference local coordinate frame ("local frame" 92). In addition, the ultrasonic probe 60 is provided with another reference local coordinate frame ("ultrasonic frame"). For convenience, the combination of the optical tracking marker 86 and the ultrasonic probe 60 is referred to as a "hybrid probe" 94. The position sensor 88 located remotely from the hybrid probe 94 determines a fixed world coordinate frame ("world frame"). The operation of the optical tracking system (optical tracking marker 86 with position sensor 88) with the ultrasonic probe 60, when calibrated, is configured to determine the transformation between the local coordinate frame and the ultrasonic coordinate frame.

[0090] Referring now to FIG. 4 while continuing to refer to FIG. 2, a method 100 for calibrating an optical tracking system according to an embodiment of the present invention is described. To calibrate the optical tracking marker 86 with the position sensor 88, a homogeneous transformation between the local frame OP and the world frame W is required for real-time tracking of the hybrid probe 94.

[0091]

Number

[0092] The calibration method 100 begins by determining a plurality of calibration parameters (block 102). In the example for a particular illustration, four parameters are used, which are P trans-origin , i.e., the origin of the transducer array 68, L trans , i.e., the length of the transducer array 68,

[0093]

Number

[0094] , i.e., the unit vector along the length direction of the transducer array 68, 4)

[0095]

Number

[0096] , i.e., including the unit vector in the direction orthogonal to the length of the transducer array 68. These calibration points and vectors are relative to the local frame 92 ("OP").

[0097] The optical camera of the position sensor 88 is, for example, P trans1 , i.e., the first end of the transducer array 68, P trans2 , i.e., the second end of the transducer array 68, and P plane , i.e., P trans1 and Ptrans2 While acquiring several points including points on the transducer array 68 that are not on the same line, the hybrid probe is kept in a fixed position (block 104). Then, the homography

[0098]

Number

[0099] between OP and W is recorded (block 106). Then, a plurality of calibration parameters are calculated as follows from the measured number of points and the transformation

[0100]

Number

[0101] (block 108).

[0102]

Number

[0103]

Number

[0104] L trans = |P trans2 - P trans1 | (3)

[0105]

Number

[0106]

Number

[0107] Once a plurality of calibration parameters are determined, the hybrid probe 94 can be used to scan a portion of the patient's musculoskeletal system while the position sensor 88 tracks the physical movement of the hybrid probe 94.

[0108] Since the reflection and attenuation of bone to ultrasound are large, ultrasonic energy generally does not penetrate bone tissue significantly. Therefore, it is not possible to image the soft tissue behind the bone, which poses a problem for ultrasonic imaging of joints. For example, as shown in FIGS. 5A to 5C, the knee joint 114 is formed from three articulating bones, the femur 116, the tibia 118, and the patella 120, with the fibula 122 shown as the environment. These bones 116, 118, 120 articulate together at two joints, namely (1) the tibiofemoral joint 136 is formed by the articulation of the femur 116 and the tibia 118 at their respective condyles 124, 126, 128, 130, and (2) the patellofemoral joint 138 is formed by the articulation of the patella 120 and the femur 116 at the patellar surface 132 of the femur 116 and the articular surface 134 of the patella 120. During flexion and extension of the knee joint 114, multiple portions of one or more articulating surfaces of the bones 116, 118, 120 are visible to the ultrasonic beam, while other articulating surfaces are blocked. FIGS. 6A to 6F include various fluoroscopic images of a patient's knee joint 114, showing the articulating surfaces at multiple degrees of flexion.

[0109] To obtain ultrasonic images of most of the joint articular surfaces, for example, at least two degrees of flexion are required, including a fully extended state (Figure 6A) and a deeply bent knee state (Figure 6F) (or 90° flexion (Figure 6E) if it is difficult for the patient to bend the knee deeply). That is, when the knee joint 114 is in a fully extended state (Figure 6A), the ultrasonic beam can reach the posterior parts of the distal femur 116 and the proximal tibia 118. When the knee joint 114 is in a deeply bent knee state (Figure 6F), the ultrasonic beam can reach the front surface of the distal femur 116, the trochlear groove 140, most of the lower surfaces of the femoral condyles 124, 126, the anterior upper surface of the tibia 118, and the front surface of the tibia 118. Both the medial and lateral portions of the femur 116 and the tibia 118 are visible at all flexion angles of the knee joint 114.

[0110] Referring now to Figure 7, one method 150 for obtaining data for constructing a 3-D patient-specific bone model according to an aspect of the present invention is described. The method begins by obtaining a plurality of RF signals from an A-mode ultrasonic beam scan of the bone. To obtain RF signals for creating a 3-D patient-specific model of the knee joint 114, the patient's knee joint 114 is positioned and held at one of two or more degrees of flexion (block 152). The hybrid probe 94 is placed at two or more positions on the patient's epidermis 144 adjacent to the knee joint 114 for obtaining the A-mode RF signal 142, and an example is as shown in Figure 8. The acquired signals include a plurality of RF signals, but for convenience, the RF signals may be referred to in the singular form herein.

[0111] As shown in FIG. 8, continuing to refer to FIG. 7, the placement of the patient's knee joint 114 is kept stationary to avoid motion artifacts during image acquisition. If movement occurs, the scan can be automatically registered to the statistically most likely placement based on the acquired data. Further, by keeping the knee stationary and compensating for movement, there is no need for invasive fiducial markers or error-prone skin markers. In some embodiments, as will be described in detail below, B-mode images similar to those shown in FIG. 9 can also be processed (block 154) from the data collected for later visualization and overlaid on the bone contour.

[0112] When the acquisition of the RF signal 142 and, if desired, the B-mode image is completed for a first degree of flexion, the patient's knee 114 is moved to another degree of flexion and the reflected RF signal 142 is acquired (block 156). Again, if desired, a B-mode image can also be acquired (block 158). The user then determines whether the acquisition is complete or whether additional data will be acquired (block 160). That is, if the visualization of the desired surfaces of one or more bones 116, 118, 120 is not obstructed (the "NO" branch of decision block 160), the method returns to acquire additional data at another degree of flexion (block 156). If the desired bone surfaces are sufficiently visible (the "YES" branch of decision block 160), method 150 continues.

[0113] FIG. 8 shows the acquisition of the RF signal 142 in yet another manner. That is, while the patient's leg is in a fully extended state (shown by the phantom), the hybrid probe 94 is placed at two or more locations on the patient's epidermis 144 adjacent to the knee joint 114. The patient's leg is then moved to a second degree of flexion (a 90° flexion is shown by the solid line) and the hybrid probe 94 is again placed at two or more locations on the patient's epidermis 144. Throughout this, the position sensor 88 tracks the position of the hybrid probe 94 in 3-D space. The resulting RF signal profiles, bone models, bone contours, etc. can be displayed on monitor 58 during model reconstruction and on monitor 58' after model reconstruction.

[0114] After all data and RF signal acquisition is complete, computer 54 operates to automatically separate that portion of the RF signal, i.e., the bone contour, from each of the plurality of RF signals. In that regard, computer 54 may sample the echoes comprising the RF signal to extract the bone contour for generating the 3-D point cloud 165 (FIG. 16B) (block 164). More specifically, referring now to FIGS. 10A-10E and 11 while continuing to refer to FIGS. 7-9, one method of extracting the bone contour from each of the RF signals 142 is shown. FIG. 10A shows one exemplary raw RF signal 142 as acquired by one transducer comprising the transducer array 68 of the ultrasonic probe portion of the hybrid probe 94. Each acquired raw RF signal includes several echoes 162, which may be separated, partially overlapping, or fully overlapping. As will be described in more detail below, each of the plurality of echoes is derived from the reflection of at least a portion of the ultrasonic energy at the interface between two tissues having different reflection and / or attenuation coefficients.

[0115] FIGS. 10B and 10C show an ultrasonic frame 146 having a selected one of the raw RF signals 142 in a state where several echoes 162 have been identified. FIGS. 10D and 10E are 3-D renderings of 2D images taken from the ultrasonic frame 146 in a state where a selected one of the RF signals 142 is identified in FIG. 10E.

[0116] Referring particularly to FIG. 11 here, the method of extracting the bone contour 162a starts with a model-based signal processing approach that incorporates empirical knowledge of the underlying physical problem into the signal processing scheme. In this way, the computer 54 can process the RF signal 142 and remove some preliminary noise based on the estimated or expected results. For example, for the acquisition of ultrasonic signals, the physical problem is represented by a wave equation that complies with, among others, "Computer Simulation of Forward Wave Propagation in Soft Tissue", IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 1473-1482:52(9), September 2005, the entire disclosure of which is incorporated herein by reference. This wave equation describes the behavior of ultrasonic propagation in heterogeneous media. The solution of the wave equation can be represented as a state space model-based processing scheme such as that described in CHEN Z et al., "Bayesian Filtering: From Kalman Filters to Particle Filters, and Beyond", Statistics, 1-69. According to an embodiment of the present invention, the general solution to the problem for a model-based ultrasonic estimator is developed using a Bayesian estimator (e.g., maximum a posteriori probability), which leads to a non-linear model-based design.

[0117] Model-based signal processing of the RF signal 142 begins by improving the RF signal by applying a model-based signal processor (here a Bayesian estimator) (block 167). To apply the Bayesian estimator, first, off-line measurement results are collected from phantoms, cadavers, and / or simulated tissue to estimate several unknown parameters, such as attenuation coefficients (i.e., absorption and scattering) and acoustic impedances (i.e., density, porosity, compressibility), in the manner generally described in VARSLOT T (referenced above), the entire disclosure of which is incorporated herein by reference. The off-line measurement results (block 169) are input into the Bayesian estimator, and the unknown parameters are then estimated as follows. z = h(x) + v (6)

[0118]

Number

[0119] Here, h is a measurement function that models the system, and v is noise and modeling error. When modeling the system, the parameter x that best fits the measurement result z is determined. For example, the data fitting process finds the estimate that best fits the measurement result of z by minimizing some error norm |∈| of the residual, where

[0120]

Number

[0121] of, and where

[0122]

Number

[0123] is.

[0124] For ultrasonic modeling, the input signal z is the raw RF signal from the off-line measurement results, and the estimate

[0125]

Number

[0126] is based on a state - space model with known parameters of the offline measurement results (i.e., density, etc.). To reduce the influence of v by identifying unknown parameters from iterative measurements by minimizing the residuals, the error v can include noise, unknown parameters, and modeling errors. Assigning approximately 99% weight to the last echo in the scan line as bone is an example of using likelihood in a Bayesian framework. Alternatively, a Kalman filter may be used, which is a special case of recursive Bayesian estimation where the signal is assumed to be linear and have a Gaussian distribution.

[0127] It will be readily understood that the exemplary use of the Bayesian model here is not limiting. Rather, within the spirit of the present invention, other model - based processing algorithms or probabilistic signal processing methods may be used.

[0128] Once the model - based signal processing is complete, the RF signal 142 is then converted into a plurality of envelopes in order to extract the individual echoes 162 within the RF signal 142. Each envelope is determined by applying a moving - power filter to each RF signal 142 (block 168) or by other suitable envelope - detection algorithms. The moving - power filter can consist of a moving kernel of a length equal to the average length of the individual ultrasonic echoes 162. At each iteration of the moving kernel, the power of the RF signal 142 at the location of the instantaneous kernel is calculated. One exemplary kernel length can be 20 samples. However, other lengths can also be used. The value of the RF signal 142 represents the value of the signal envelope at that location of the RF signal 142. Given a discrete - time signal X of length N, each envelope V is calculated using a moving - power filter of length L as follows:

[0129] [Number]

[0130] is defined by. In some embodiments, this and subsequent equations are for samples

[0131] [Number]

[0132] of the sample before (left filter) and sample

[0133] [Number]

[0134] after (right filter). For the special case of using a variable-length one-sided filter.

[0135] Each envelope produced by the moving power filter, shown in FIG. 10B, includes a plurality of local peaks (identified in FIG. 10B as the enlarged points at the intersections of each envelope with echo 162), each of which is a distinct representation of an individual echo 162 present in the RF signal 142 acquired for various tissue interfaces. As an example of such processing, FIGS. 12A - 12D more clearly show the RF signal 142 (top in each figure) and the corresponding envelopes (bottom in each figure) for four iterations of the kernel of the moving power filter. The individual echoes 162 within each envelope are also identified by the enlarged points.

[0136] Among the plurality of echoes 162 in the RF signal 142, one echo 162, for example the echo corresponding to the bone-soft tissue interface, is of particular interest. This bone echo (hereinafter referred to as 162a) is generated by the reflection of ultrasonic energy at the surface of the scanned bone. More specifically, the soft tissue-bone interface is characterized by a high reflection coefficient of 43%, which means that 43% of the ultrasonic energy reaching the surface of the bone is reflected back to the transducer array 68 of the ultrasonic probe 60 (FIG. 2). This high reflectivity gives the bone a characteristic echo-rich appearance in the ultrasonic image.

[0137] Bone is also characterized by a high attenuation coefficient of the applied RF signal (6.9 db / cm / mHz for trabecular bone and 9.94 db / cm / mHz for cortical bone). At high frequencies such as those used in musculoskeletal imaging (i.e., in the range of 7-14 MHz), the attenuation of bone becomes very large and the ultrasonic energy ends at the surface of the bone. Therefore, the echo 162a corresponding to the bone-tissue-bone interface is the last echo 162a in the RF signal 142. The bone echo 162a is identified by selecting the last echo with a normalized envelope amplitude that exceeds a preset threshold (with respect to the maximum value present in the envelope) (block 170).

[0138] Next, the bone echo 162a is extracted from each frame 146 (block 172) and used to generate the bone contour present in its RF signal 142 as shown in FIG. 10C (block 174). When extracting the bone echo, a probability model (block 171) can be input and applied to the RF signal of each frame 146. The probability model (block 171) can further be used to detect cartilage within the envelope of the RF signal 142 (block 173). The probability signal processing method may include the Bayesian estimator described previously, but in other embodiments, the signal processing may be a maximum likelihood ratio, a neural network, or a support vector machine ("SVM"), for example, the SVM among them will be further described below.

[0139] Before implementing the SVM, the SVM can be trained to detect cartilage in RF signals. One such method of training the SVM involves training the SVM to distinguish echoes related to cartilage from the RF signal 142 from echoes within noise or ambiguous soft tissue echoes using information obtained from a database comprising MRI images and / or RF ultrasound images. When constructing the database according to one embodiment, knee joints from a plurality of patients are imaged using both MRI and ultrasound. The volumetric MRI images of each knee joint are reconstructed and processed to identify and segment cartilage and bone tissue. The segmented volumetric MRI images are then aligned with the corresponding segmented ultrasound images (where bone tissue is identified). This alignment results in a transformation matrix that can then be used to align the raw RF signal 142 to the reconstructed MRI surface model.

[0140] After the raw RF signal 142 is aligned to the reconstructed MRI surface model, spatial information from the volumetric MRI images regarding the cartilage tissue can be used to determine the location of the cartilage interface in the raw RF signal 142 above the articular surface of the knee joint.

[0141] The database of all knee joint image pairs (MRI and ultrasound) is then used to train the SVM. Generally, training involves loading all of the raw RF signals, as well as the location of the bone-cartilage interface for each respective RF signal. The SVM can then determine the location of the cartilage interface in an unknown input raw RF signal. If desired, the user can select from one or more kernels to maximize the classification rate of the SVM.

[0142] During use, the SVM being trained receives the reconstructed knee joint images of the new patient as well as the raw RF signals. The SVM returns the position of the cartilage on the RF signal data, which can be used together with the tracking information from a tracking system (e.g., optical tracking markers 86 and position sensors 88) to generate the 3-D coordinates of each point on the cartilage interface. The 3-D coordinates can be triangulated and interpolated to form the complete cartilage surface.

[0143] Referring further to FIG. 11, the resulting bone contour may be noisy and may require filtering to remove echoes 162 that may be misdetected as bone echoes 162a. The misdetected echoes 162 may originate from at least one of two sources: (1) isolated abnormal echoes and (2) false bone echoes. Further, since some images may not include bone echoes 162a, any detected echo 162 may be noise and should be removed. Thus, an appropriate determination of a preset threshold or filtering algorithm can prevent the incorrect selection of misdetected echoes 162.

[0144] Isolated outliers are echoes 162 in the RF signal 142 corresponding to tissue interfaces that are not the soft tissue-bone interface. The selection of isolated outliers can occur when the reference setting is too high. If necessary, isolated outliers can be removed by applying a median filter to the bone contour (block 176). That is, given a specific bone contour X of length N and the length L of the median filter, the median filter contour Y k is

[0145] [Number]

[0146] is.

[0147] A false bone echo is an echo 162 caused by noise or scattered echoes, which results in the detection of a bone contour where there is no bone contour. False bone echoes can occur when an area without bone is scanned, when the ultrasonic probe 60 is not oriented substantially orthogonally with respect to the bone surface, when the bone is deeper than the selected scan depth, when the bone is within the selected scan depth but its echo is highly attenuated by soft tissue overlapping the bone, or by a combination of these. Selection of false bone echoes can occur when a preset threshold is too low.

[0148] Frames 146 containing false bone echoes should be removed. One such method (block 178) of removing false bone echoes can include the step of applying a continuity criterion. That is, since the surface of the bone is rectangular, the contour of the bone in two dimensions of the ultrasonic image should be continuous and smooth. False bone echoes create discontinuities and exhibit a high degree of irregularity with respect to the bone contour.

[0149] One way to remove false bone echoes is to apply a moving standard deviation filter. However, other filtering methods can also be used. For example, given a bone contour X of length N and a median filter length L, the standard deviation filter contour is

[0150]

Equation

[0151] where. Here, Y k is the local standard deviation of the bone contour, which is a measure of the regularity and continuity of the bone contour. Segments of the bone contour containing false bone echoes are characterized by a high degree of irregularity and have high Y k values. On the other hand, segments of the bone contour containing only echoes due to the bone surface are characterized by a high degree of regularity and have low Y k values.

[0152] The resulting bone contour 180, due to the application of the moving median filter and the moving standard deviation filter, may include the contour of the overall length of the entire bone surface, one or more partial contours of the entire surface, or no bone contour segments.

[0153] Figures 12A - 12F and 13A - 13F show the resulting bone contour 180 selected from segments of the extracted bone contour that meet two conditions: (1) a continuity criterion that the local standard deviation value is less than a selected standard deviation threshold, and (2) a minimum length criterion that prevents piecewise smooth noise contour segments from being misdetected as bone contours. In some exemplary embodiments, the length of the standard deviation filter may be set to 3 and the threshold may be set to 1.16 mm, which may correspond to 30 signal samples. Thus, FIGS. 13A and 13D show two exemplary RF signals 142 along with the resulting bone contour 180 (including isolated outliers and false body echoes) extracted and filtered from the noise 182 shown in FIGS. 13B and 13E, respectively. FIGS. 13C and 13F each show the standard deviation Y calculated as given in Equation 11 above k is shown. FIGS. 14A - 14F are similar to FIGS. 13A - 13F but include two exemplary RF signals 142 when bone tissue was not scanned.

[0154] Once the bone contour is separated from each of the RF signals, the bone contour can be converted into a point cloud. For example, referring again to FIG. 7 here, the obtained bone contour 180 may then undergo alignment with the optical system to construct a bone point cloud 194 representing the surface of at least a portion of each scanned bone (block 186), which is described herein as a multi-step alignment process. In one embodiment, the process is a two-step alignment process. The alignment step (block 186) begins by converting the bone contour 180 obtained from the 2D contour in the ultrasound frame to a 3-D contour in the world frame (block 188). This conversion is applied to all of the obtained bone contours 180 extracted from all of the acquired RF signals 142.

[0155] To convert the obtained bone contour 180 into a 3-D contour, each detected bone echo 162a undergoes a conversion to a 3-D point as follows. d echo =n echo T s C us (12)

[0156]

Number

[0157]

Number

[0158]

Number

[0159] Here, the variables are defined as follows.

[0160]

Table 1

[0161] If so desired, an intermediate alignment process (block 190) can be performed between the obtained bone contour and, if acquired, the B-mode image. This alignment step is performed to visualize the obtained bone contour 180 using the B-mode image (FIG. 9), which provides visual verification and feedback of the detection process of the obtained bone contour 180 in real time while the user is performing the scan. As previously explained, this visual verification can assist the user in determining that the acquisition is complete (block 160). More specifically, the obtained bone contour 180 is

[0162]

Number

[0163] aligned with the B-mode image by. Here, l x and I y represent the B-mode image resolutions (pixels / cm) of the x-axis and y-axis, respectively.

[0164]

Number

[0165] indicate the coordinates of the bone contour points relative to the ultrasound frame.

[0166] After the obtained bone contour 180 is transformed and, if desired, aligned (block 190) (FIG. 15), a plurality of point clouds 165 (FIG. 16B) representing the bone surface are generated. During the second alignment process, the plurality of point clouds 165 are integrated into a bone point cloud 194 representing the entire surface of the scanned bone. The entire surface of the bone is represented by the bone point cloud 194.

[0167] To initiate the second alignment process, as shown in FIGS. 16A - 17D, a plurality of point clouds 194 are first registered to a standardized model of the scanned bone, here the model femur 200, by using, for example, 4 - 6 previously specified landmarks 196. More specifically, the user may identify a plurality of landmarks 196 on the model femur 200, which need not be identified with high precision. After this initial registration, iterative closest point (“ICP”) registration is performed to more precisely register the standardized model with respect to the plurality of point clouds. If necessary, noise can be removed by setting a threshold on the distance between each point of the plurality of point clouds and the closest vertex in the model femur 200. However, alternatively, other filtering methods may be used. For example, one standard deviation added to the average distance may be used as the threshold. This process is repeated for each point cloud 165 of the plurality of point clouds on the surface of the bone being scanned. The point clouds 165, now registered, are then integrated into a single uniform point cloud 194 representing the surface of the scanned bone (block 202).

[0168] After the point cloud 194 is formed, the bone model can be optimized according to the point cloud 194. That is, then, the bone point cloud 194 is used to reconstruct the 3-D patient-specific model of the surface of the scanned bone. This reconstruction begins with the determination of the bone model from which the 3-D patient-specific model is derived (block 210). The bone model may be a generalized model based on a plurality of patient bone models and may be selected from a statistical bone atlas based on principal component analysis ("PCA"). One such prior bone atlas formed according to method 212 of FIG. 18 includes a data set of 400 pairs of dry femurs and tibias scanned by CT (block 214) and is segmented to create a model of each bone (block 216). One method of constructing and using such a statistical atlas is described in MAHFOUZ M et al., "Automatic Methods for Characterization of Sexual Dimorphism of Adult Femora: Distal Femur", Computer Methods in Biomechanics and Biomedical Engineering, 10(6) 2007, the entire disclosure of which is incorporated herein by reference. Each bone model M i (where I ∈ [1, N], and N is the number of models in the data set) has the same number of vertices, and the vertex V in one selected model j corresponds to the vertex V in another model within the statistical atlas j (being in the same histological location of the bone)

[0169] Next, to extract the modes of variation of the bone surface, PCA is performed on each model in the dataset (block 218). Each mode of variation is represented by a plurality of eigenvectors resulting from the PCA. These eigenvectors, sometimes called eigenbones, define the vector space of the variation of the bone shape extracted from the dataset. PCA can include any one model from the dataset, represented as a linear combination of eigenbones. All average models of the 3-D models comprising the dataset are extracted (block 220),

[0170]

Number

[0171]

Number

[0172] and can be defined as follows. Here, the variables are defined as follows

[0173]

Table 2

[0174] Furthermore, any new model M new (i.e., a model not yet present in the dataset) can be approximately represented as follows by the new values of the shape descriptors (eigenvector coefficients).

[0175]

Number

[0176] Here, the variables are defined as follows.

[0177]

Table 3

[0178] M new The accuracy is directly proportional to the number of principal components (W) used when approximating the new model and the number L of models in the dataset used for PCA. The residual or root mean square error (“RMS”) for using the PCA shape descriptor is

[0179]

Number

[0180] defined by

[0181] Thus, when comparing any two different models A and B having the same number of vertices, the RMS is

[0182]

Number

[0183] defined by. Here, V Aj is the j-th vertex in model A, and similarly, V Bj is the j-th vertex in model B.

[0184] Referring again to FIG. 7, the average model (the "average" branch of block 210) is loaded (block 230), or a subset model is selected from a statistical atlas based on attributes similar to the patient (the "selection" branch of block 210) and loaded for optimization (block 232). Then, the bone point set 194 is applied to the loaded model so that the shape descriptors of the loaded model can be modified to create a 3-D patient-specific model (block 234). If desired, one or more shape descriptors can be constrained so that the 3-D patient-specific model has the same tissue characteristics as the loaded model (the "YES" branch of block 254). Thus, one or more shape descriptors are set (block 238). Using a set of constraints, the loaded model can be deformed (or optimized) into a model that resembles an appropriate bone rather than a model with an irregular random shape (block 240). If constraints are not desired (the "NO" branch of block 240), the loaded model is optimized (block 240).

[0185] Modifying the shape descriptors to optimize the loaded model (block 240) may be performed by one or more optimization algorithms derived by a scoring function to find the values of the principal component coefficients to create a new 3-D patient-specific model, as described with reference to FIG. 19. The optimization algorithms shown include a two-step optimization method of algorithms that are applied sequentially to obtain a 3-D patient-specific model that best fits the bone point set 194, as discussed below. Although a two-step method is described, the present invention is not limited to only a two-step optimization method.

[0186] The first algorithm may use a numerical method to search for an eigen space for the optimal shape descriptor. More specifically, the first algorithm can be an iterative method that searches for the shape descriptors of the loaded model to find the points that most closely match the bone point set 194 (block 250). One such iterative method can include, for example, Powell's conjugate gradient descent method using RMS as the scoring function. The new model M defined by Equation 19new To form, the shape descriptor of the loaded model is modified by a first algorithm (block 252). Then, the new model M new is compared with the bone point set 194, and a residual E is calculated to determine whether further iterative search is necessary (block 254). More specifically, given a bone point set Q with n points and an average model M with I vertices avg is given, there may be a set V of vertices in the average model M avg that are closest to the bone point set Q within M.

[0187]

Number

[0188] Here, v i is the point in set V that is closest to q i in the bone point set Q. To efficiently find the closest point in M new , an octree can be used. And the residual E between the new model M new and the bone point set Q is E = ||V - Q|| 2 (23) is defined as.

[0189] If the residual is large enough (the "YES" branch of block 254), the method returns to further search for the shape descriptor (block 250). If the residual is small (the "NO" branch of block 254), the method proceeds.

[0190] The second algorithm of the two-step method improves the new model derived from the first algorithm by converting the new model into a system of linear equations of shape descriptors. The linear system is easily solved by a conventional technique for obtaining a solution, which gives a 3-D patient-specific shape descriptor.

[0191] Continuing with FIG. 19, the roots of the linear system must be determined (block 256) in order to transform the new model into a linear system. More specifically, the first partial derivative coefficients of the residual E with respect to the shape descriptor α k are equal to 0. The error function (Equation 23) can be expressed as follows with respect to the vertices v i of the set V and the points p i of the point cloud Q.

[0192]

Equation

[0193] And the shape descriptor of the new model can also be expressed as follows.

[0194]

Equation

[0195] Here, V avg is the set of vertices from the vertices of the loaded model, which corresponds to the set of vertices V that includes the closest vertices in the new model M new that has been morphed to fit the bone point cloud Q. U k ' is a reduced version of the eigenbone U k that includes only the set of vertices corresponding to the set of vertices V.

[0196] Combining Equations 24 and 25, E can be

[0197]

Equation

[0198] expressed as. Here, v avg,I is the i-th vertex of V avg . Similarly, u k ' ,I is the i-th vertex of the reduced eigenbone U k '.

[0199] The error function can be expanded as

[0200]

Number

[0201] and so on. Here, x avg,I is the x-coordinate of the i-th vertex of the average model, and x k,I is the x-coordinate of the i-th vertex of the k-th eigenbone, and x Q,I is the x-coordinate of the i-th point of the point cloud Q. Similar discussions apply to the y-coordinate and z-coordinate. For each shape descriptor α k calculating the partial derivative coefficients of E, the following is obtained.

[0202]

Number

[0203]

Number

[0204] Re-combining the coordinate values into a vector, the following is obtained.

[0205]

Number

[0206] And arranging them,

[0207]

Number

[0208] is obtained.

[0209] Rearranging Equation 31 into matrix form, a system of linear equations in the form of Ax = B is obtained as follows.

[0210]

Number

[0211] The system of linear equations can be solved using any number of known methods, such as singular value decomposition (block 258).

[0212] In one embodiment, the bone point set is dense so the Mahalanobis distance is omitted, thereby providing a constraining force on model deformation. Thus, the constraining function of the Mahalanobis distance may not be necessary and, rather, was avoided to give more freedom to model deformation to generate a new model that best fits the bone point set.

[0213] The ultrasonic treatment according to an embodiment of the present invention can generate, for example, about 5000 ultrasonic images. The generated 3-D patient-specific model (block 260, FIG. 7) produced an average error of about 2 mm as compared to the CT-based segmented model.

[0214] The solution to the set of linear equations provides a description of the patient-specific 3-D model optimized according to point clouds converted from bone contours separated from a plurality of RF signals from a statistical atlas, derived from an average or a selected model. This solution can be applied to the average model to assist in preoperative planning, precisely plan the injection site, and display the patient-specific 3-D bone model for planning a physical therapy regimen or other diagnosis- and / or treatment-based procedures involving a part of the musculoskeletal system.

[0215] The cartilage 3-D model can be reconstructed using a method similar to that outlined above for bone. During contour extraction, the cartilage contour is more difficult to detect than bone. Probability modeling (block 171) is used to process the raw RF signal to more easily identify the cartilage, and an SVM helps detect the cartilage boundary based on the MRI training set (block 173). The cartilage statistical atlas can be formed by a method similar to that described for bone. However, as shown previously, MRI is used instead of CT (as was the case for bone). Segmentation (block 216), variation extraction (block 218), and morphing of the base model (block 240) (FIG. 19) are processed to produce the reconstructed cartilage model in the same manner as the bone model is reconstructed. The cartilage model can be displayed alone or together with the 3D patient-specific bone model.

[0216] Referring now to FIGS. 20-27, according to another embodiment of the present invention, an additional method for extracting the bone contour and generating a point cloud from the raw RF ultrasound signal is described. Referring now to FIG. 20, the ultrasound device 50 is shown in more detail together with the electromagnetic tracking system 87 and the computer 54. The ultrasound device 50 may include an ultrasound transceiver 356 operably coupled to the ultrasound probe 60 by a cable 66 and a controller 360. The ultrasound transceiver 356 generates a drive signal that excites the ultrasound probe 60 such that the ultrasound probe 60 generates an ultrasound signal 362 that can be transmitted to the patient. In one embodiment of the present invention, the ultrasound signal 362 comprises a burst or pulse of ultrasound energy suitable for generating an ultrasound image. The ultrasound probe 60 may also include a tracking marker 86, shown here as an electromagnetic tracking marker 86.

[0217] The reflected ultrasonic signal or echo 364 is received by the ultrasonic probe 60 and converted into an RF signal that is transmitted to the transceiver 356. Each RF signal may be generated by a plurality of echoes 364, which may be isolated, partially overlapping, or completely overlapping. Each of the plurality of echoes 364 results from the reflection of at least a portion of the ultrasonic energy at the interface between two tissues of different densities and represents a pulse echo mode ultrasonic signal. One type of pulse echo mode ultrasonic signal is known as an "A-mode" scan signal. The controller 360 converts the RF signal into a form suitable for transmission to the computer 54, such as by digitizing, amplifying, or otherwise processing the signal, and transmits the processed RF signal to the computer 54 via the I / O interface 85. In certain embodiments of the present invention, the signal transmitted to the computer 54 can be the raw RF signal representing the echo 364 received by the ultrasonic probe 60.

[0218] The electromagnetic tracking system 87 includes an electromagnetic transceiver unit 328 and an electromagnetic tracking system controller 366. The transceiver unit 328 may include one or more antennas 368 and transmits a first electromagnetic signal 370. The first electromagnetic signal 370 excites the tracking marker 86, and the tracking marker 86 responds by transmitting a second electromagnetic signal 372 that is received by the transceiver unit 328. The tracking system controller 366 can then determine the relative position of the tracking marker 86 based on the received second electromagnetic signal 372. The tracking system controller 366 can then transmit the tracking element position data to the computer 54 via the I / O interface 85.

[0219] Referring now to FIG. 21, flowchart 380 illustrates an alternative embodiment of the present invention where the acquired scan data is used to reconstruct a patient-specific bone model. The patient-specific bone model can be generated from raw RF signals that are used directly to automatically extract bone contours from ultrasonic scans. Specifically, these embodiments of the present invention include additional methods for bone / cartilage contour detection, point cloud, and 3-D model reconstruction from ultrasonic RF signal data. The ultrasonic signal processing of these alternative embodiments optimizes scan reconstruction through a multi-layer signal processing model. The processing algorithm is decomposed into multiple models, which are separated into different layers. Each layer performs a specific optimization or estimation on the data. The main functions of the layers include, but are not limited to, feature detection and estimation, detection of scan line features, estimation, update, and smoothing of global features for optimization of the raw signal data. These layers operate within the framework of a Bayesian estimation model. The characteristics and properties of the algorithm inputs are determined by the mathematical and physical models within the layer. An example of an implementation form of this processing model is the three-layer processing system described below.

[0220] The first layer of the three-layer system optimizes the raw signal data and estimates the envelope of the feature vector. The second layer estimates the features detected from each of the scan lines from the first layer and constructs a parametric model for Bayesian smoothing. The third layer estimates the features extracted from the second layer in order to further estimate three-dimensional features in real time using the Bayesian estimation method.

[0221] In block 382, raw RF signal data representing the ultrasonic echo 364 detected by the ultrasonic probe 60 is received by the program code 83 and processed by a first layer of filtering for feature detection. The detected feature vectors include bone, adipose tissue, soft tissue, and muscle. The optimal output is the envelope of these features detected from the filter. There are two basic aspects to this design. The first aspect relates to the ultrasonic probe 60 and ultrasonic controller firmware. In conventional ultrasonic machines, the transmitted ultrasonic signal 362 is generated at a fixed frequency during a scan. However, different frequencies of ultrasonic signals have been found to reveal different soft tissue characteristics when used to scan a patient. Thus, in certain embodiments of the present invention, the frequency of the transmitted ultrasonic signal 362 varies with time using a predetermined excitation function. One exemplary excitation function is the linear ramp sweep function 383, which is shown in FIG. 22.

[0222] The second aspect is to utilize data collected from multiple scans to assist a Bayesian model for estimation, correction, and optimization. Two exemplary filter classes are shown in FIG. 21, and either of them can be used to assist this algorithm. In decision block 384, the program code 83 selects a feature detection model that determines the class of filter through which the RF signal data should be processed. If the data is to be processed by a linear filter, this application example proceeds to block 386. In block 386, the imaging program code 83 selects a linear class of filters, such as a linear Gaussian model or a non-linear Gaussian model with a linearization method, based on the Kalman filter family. The operation of this linear class of filters is shown in more detail by FIGS. 23A and 23B, which outline the basic operation of the Kalman filter and other extensions of the filter are built on top of these.

[0223] In block 388, a Kalman class filter is used to estimate the optimal time delay to identify peaks in the amplitude or envelope of the RF signal. Referring now to FIG. 23A, at time k = 1, the filter is initialized by setting the ultrasonic frequency f k = f 1 . The received echo or RF signal (s obs ) is represented by plot line 390a, while the signal envelope is represented by plot line 392a. A peak data matrix (p k,fk ) containing the positions of the RF signal peaks may be calculated by p k,fk = E(s obs ) (33) where E is an envelope detection and extraction function. Thereby, the peak data matrix (p k,fk ) comprises a plurality of points representing the signal envelope 392 and may be used to predict the positions of the envelope peaks 394, 396, 398 generated by the frequency f k+1 using the following equation p est,fk+1 = H(p k,fk+1 ) (34) where H is an estimation function.

[0224] Referring now to FIG. 23B, at time k = 2, the filter enters the recursive part of the algorithm. The frequency of the transmitted ultrasonic signal 362 is raised to be f k = f 2 , and a new RF signal is received (s obs,fk ), as represented by plot line 390b. The new RF signal 390b also generates a new signal envelope 392b. A peak data matrix is calculated for the new signal envelope 392b (p k,fk ), which identifies another set of peaks 404, 406, 408. The prediction error is ε = p est,fk-1 - p k,fk (35) and the error correction (Kalman) gain (K k ) is

[0225] [Number]

[0226] is calculated by, where P k- is the error covariance matrix and R is the covariance matrix of the measurement noise. The equation for estimating the peak data matrix for the next cycle is p est,k+1 = p k,fk + K k (ε) (37) and the error covariance is

[0227] [Number]

[0228] is updated by.

[0229] If the second class of filters is to be used, the program code 83 proceeds to block 410 instead of block 386 of the flowchart 380 and selects a non-linear non-Gaussian model that follows the recursive Bayesian filter approach. In the illustrated embodiment, the sequential Monte Carlo method or particle filter is shown as an exemplary implementation of the recursive Bayesian filter. At block 412, the program code 83 uses the particle filter to estimate the optimal time delay in order to identify the signal envelope peak. An example of a particle filter is shown in FIGS. 24A and 24B. In principle, the particle filter generates a set of N equally weighted particles (p k,fk ) 412, 414, 416 around each envelope peak 418, 420, 422 of the peak data matrix detected during initialization. The set of equally weighted particles is based on an arbitrary statistical density (p) which is

[0230] [Number]

[0231] is approximated by. These particles 412, 414, 416 are f via the following equation k+1 to predict the peak position in,

[0232]

Number

[0233] where H is the estimation function.

[0234] Referring now to FIGS. 24C and 24D, at time k = 2, when the RF signal 906(s obs ) becomes available, a new peak data matrix (p k,fk ) is calculated and a new set of estimated particles 424, 426, 428 is created around each peak 430, 432, 434 for (f k = f 2 ). The set of estimated particles 412, 414, 416 from time k = 1 is compared with the observed data obtained at time k = 2 and the error is determined using the following equation.

[0235]

Number

[0236] The normalized importance weights of the particles in the set of particles 424, 426, 428 are

[0237]

Number

[0238] is evaluated and this produces a set 436, 438, 440 of weighted particles. This step is known as importance sampling, where the algorithm approximates the true probability density of the system. An example of importance sampling is shown in FIG. 25, which shows a series of signal envelopes 392a - 392f for times k = 1 to 6. Each signal envelope 392a - 392f includes peaks 442a - 442f and projections 444a - 444f of the peaks 442a - 442f onto a sweep - line time scale 446 indicating the echo return time. And these projections 444a - 444f can be plotted as contours 448 representing the transition of tissue density or the estimated position of the surface. In any case, the prediction of the peak - data matrix can then be calculated as follows, based on the importance weights and the estimation of the particles.

[0239]

Number

[0240] In addition, it may be necessary to maintain the particles to avoid particle degeneracy, which refers to the situation where the weights concentrate on a small number of particles over time. As follows, particle resampling can be used by replacing the degenerate particles with new particles sampled from the posterior density.

[0241]

Number

[0242] Referring now to FIG. 26, when an envelope peak is identified, program code 83 proceeds to block 450, applies Bayesian smoothing to the envelope peaks 442 in the temporally adjacent scan lines 452, and then proceeds to block 454 to extract 2-D features from the resulting smoothed contour line 456. Thus, this second layer of the filter applies Bayesian techniques to smooth the detected features at the two-dimensional level. Conventional peak detection methods have the limitation that envelope peaks 442 spanning different scan lines are not statistically weighted. Thus, only the peak 442 with the highest power is detected for reconstruction. This can result in an error-prone contour, such as that shown by the contour line 458 connecting the envelope peaks 442 with the highest amplitudes. Thus, signal artifacts or inappropriate amplitude compensation by the gain control circuit in the RF signal path can obscure the signal envelope containing the features of interest by distorting the amplitudes of the envelope peaks. Thus, the goal of filtering in the second layer is to correlate signals from different scan lines to form a matrix that determines or identifies two-dimensional features.

[0243] This is achieved in embodiments of the present invention by Bayesian model smoothing, which produces an exemplary smoothed contour line 456. The principle is to retrospectively examine the signal envelope data and attempt to reconstruct the previous state. The main difference between a Bayesian estimator and a smoother is that the estimator propagates the state forward in each recursive scan, whereas the smoother operates in the reverse direction. The initial state of the smoother begins with the last measurement and propagates backward. A common implementation of a smoother is the Rauch-Tung-Striebel (RTS) smoother. Features embedded in the ultrasonic signal are initialized based on prior knowledge of the scan, which may include ultrasonic transducer placement data received from the electromagnetic tracking system 87. Then, using the RTS smoother, sequential features are estimated and updated in the ultrasonic scan lines.

[0244] In certain embodiments of the present invention, the ultrasonic probe 60 is equipped with electromagnetic or optical tracking markers 86 so that the movement of the ultrasonic probe 60 is accurately known. This tracking data 460 is provided to the program code 83 at block 462 and is necessary to determine the placement of the ultrasonic probe 60, since the movement of the ultrasonic probe 60 is arbitrary with respect to the patient's joint. As scans are acquired by the ultrasonic probe 60, the system estimates 3-D features of the joint, such as the shape of bone and soft tissue. This type of tracking problem can be viewed as a probabilistic estimation problem, the goal of which is to compute the most likely value of the state vector Xi given a series of measurements yi that are the acquired scans. In certain embodiments of the present invention, the state vector Xi is the placement of the ultrasonic probe 60 with respect to some fixed known coordinate system or “world frame” (such as the ultrasound machine at time k = 0), as well as the mode of bone deformation. The two main steps in tracking are as follows. (1) Prediction - The state of the system at k = i can be predicted based on all measurements up to time k = i - 1. To do this, a conditional probability P(X i |y 0 ,y 1 ,…,y i-1 ), called the prior distribution, must be computed. Assuming this process is a first-order Markov process, this can be computed by integrating P(X i-1 |X i )P(X i-1 |y i ,y 0 ,…y 1 ,…y i-1 ) over all X i-1 . (2) Correction - Correct the state estimate given the new measurement y i . To do this, a probability P(X i |y 0 ,y 1 ,…,y i ), called the posterior distribution, must be computed.

[0245] The system dynamics model is the transition distribution P(Xi |X i-1 ) via the previous state X i-1 is associated with the new state X, which is a model of how the state is expected to evolve over time. In certain embodiments of the present invention, X i is a 3-D feature estimate calculated from Bayesian contour estimation performed between layer 2 filterings, and the transformation information includes the translation and rotation of data obtained from the tracking system 87. In joint imaging, it is not expected that the optimal density or features change over time, because the bone arrangement is spatially fixed and the shape of the bone being scanned does not change. Therefore, the translational distribution does not change the model state.

[0246] The measurement model associates the state with the predicted measurement results, y = f(X). Since there is uncertainty in the measurement results, this relationship is generally represented by a conditional probability P(y i |X i ), also called the likelihood function. In certain embodiments of the present invention, the placement and shape of the RF signal and the prior features are associated by the Anisotropic Iterative Closest Point (AICP) method.

[0247] To estimate the placement and shape of the features, the program code 83 proceeds to block 464. In block 464, the program code 83 implements the AICP method that iteratively searches for the closest points between two datasets and establishes a correspondence based on the anisotropic weighted distance calculated from the local error covariance of both datasets. This correspondence is then used to calculate a rigid body transformation that is iteratively determined by minimizing the error until convergence. Then, based on the received RF signal and the placement and shape of the prior features, 3-D features can be predicted. By calculating the residual between the predicted 3-D features and the RF signal data, the placement and shape of the prior features are updated and corrected in each recursion. Using Bayes' theorem, the posterior distribution can be calculated based on the measurement results from the raw RF signal.

[0248] When both the dynamic model and the measurement model are additive Gaussian noise and linear, the conditional probability distribution is a normal distribution. Specifically, P(X i |y 0 ,y 1 ,…,y i ) is unimodal and Gaussian distributed, so it can be expressed using the mean and covariance of the predicted measurement results. Unfortunately, the measurement model is not linear, and the likelihood function P(y i |X i ) is not Gaussian distributed. One way to address this is to linearize the model for local estimation and assume that the distribution is locally Gaussian.

[0249] Referring to FIG. 27, a surface 466 representing an exemplary probability distribution associated with a point cloud 468 of a bone 469 to be scanned shows that the probability distribution of the measurement model is not Gaussian and has many peaks. This suggests that there are multiple hidden states in the model. The posterior probability P(X i |y 0 ,y 1 ,…,y i ) will also have multiple peaks. The problem becomes worse if the state includes shape parameters as well as placement. Linear tracking filters such as the Kalman filter (or its non-linear extension, the extended Kalman filter) cannot handle non-linear and non-Gaussian systems with multi-peak distributions, and this can sometimes converge to incorrect solutions.

[0250] Instead of treating the probability distribution as a Gaussian distribution, statistical estimation can be performed using Monte Carlo sampling of the state. Thereby, the optimal placement and shape of the features are estimated through the posterior density, which is determined from the sequential data obtained from the RF signals. For recursive Bayesian estimation, one exemplary implementation is particle filtering, which has been found useful in applications where the state vector is complex and the data contains a large amount of noise, such as tracking an object in an image sequence. The basic idea is to represent the posterior probability by a weighted set of independent uniform distributions of states or particles. Given a sufficient number of samples, it can represent even very complex probability distributions. As measurements are made, the importance weights of the particles are adjusted using the likelihood model according to the formula w j '=P(y i |X i )w j , where w j is the weight of the j-th particle. This is known as importance sampling.

[0251] The main advantage of this method is that it can approximate the true probability distribution of the system, which cannot be directly determined by approximating a finite set of particles from the distribution from which the samples can be drawn. As measurement results are obtained, the algorithm adjusts the weights of the particles to minimize the error between the predicted state and the observed state. With sufficient particles and iterations, the posterior probability approaches the true density of the system. Thereby, surface contours of multiple bones or other tissue features are generated, which can be used to generate 3-D images and models of joint or tissue features. And these models can be used to assist medical procedures such as joint injections by enabling the visualization of joint or other tissue features in real time during treatment using an ultrasound scan.

[0252] International Publication No. WO2014 / 121244, published on August 7, 2014, which claims priority to International Application No. PCT / US2014 / 014526, filed on February 4, 2014, describes exemplary 3-D constructs of joints using ultrasound, and is hereby incorporated by reference in its entirety.

[0253] 3-D Reconstruction Using Multiple Imaging Modalities FIG. 28 is a flowchart of an exemplary method 500 for generating a virtual 3-D model of a tissue structure using multiple imaging modalities, according to at least some aspects of the present disclosure. Method 500 is described in connection with creating a 3-D virtual model of the hip joint including the pelvis and femur, but various aspects of this method can be utilized with respect to modeling individual bones (e.g., tibia only, patella only, scapula only, humerus only, femur only, and / or pelvis only, etc.), joints including multiple bones (hip joint, knee joint, shoulder joint, ankle joint, etc.), as well as various other tissue structures including soft tissues (cartilage, ligaments, tendons, etc.), separately or together with one or more other tissue structures. Further, it will be understood that the operations associated with this method can generally be performed in relation to one or more tissue structures simultaneously or sequentially. For example, as described below, the pelvis and femur can be modeled together in a coordinated process. In alternative sequentially configured embodiments, one or more tissue structures may be modeled according to this method, and then the method may be performed with respect to one or more tissue structures.

[0254] Method 500 may include operation 502, which includes obtaining a preliminary virtual 3-D bone model 504 of one or more bones. For example, the ultrasonic scan and 3-D reconstruction processes described above in the "3-D Reconstruction of Joints Using Ultrasound" section may be utilized to generate the preliminary virtual 3-D bone model 504. The 3-D bone model 504 may comprise the final output of the 3-D reconstruction process described above, but in the context of this method 500, the bone model 504 may be "preliminary" because it may be refined in later operations.

[0255] FIG. 29A and FIG. 29B are isometric views of exemplary ultrasonic point clouds 506, 508 of the femur 510 and pelvis 512, respectively, according to at least some aspects of the present disclosure. Referring to FIGS. 28, 29A, and 29B, in the illustrated embodiment, obtaining the preliminary 3-D bone model 504 may include obtaining one or more ultrasonic point clouds, such as point clouds 506, 508 of one or more bones 510, 512. The ultrasonic point cloud 506 of the femur 510 may include the femoral neck. The ultrasonic point cloud 508 of the pelvis 512 may include at least a portion (e.g., an edge) of the acetabular fossa. Generally, in some exemplary embodiments, the ultrasonic data may include one or more bones and / or one or more tissues other than bone, such as ligaments, muscles, adipose tissue, tendons, and / or cartilage (collectively "soft tissue"). For example, embodiments related to the hip joint may include ultrasonic imaging of ligaments such as the ischiofemoral ligament, the iliofemoral ligament, and / or the transverse ligament. Generally, the ultrasonic data may be used to reconstruct a virtual 3-D model of bone and / or soft tissue.

[0256] In accordance with at least some aspects of the present disclosure, FIG. 30A is an isometric view of point clouds 506, 508 arranged as obtained by ultrasonic scanning of a patient's hip joint, and FIG. 30B is an isometric view of point clouds 506, 508 superimposed on a preliminary 3-D model 504. Specifically, in this embodiment, the point cloud 506 of the femur 510 may be associated with the preliminary 3-D model of the femur 504A, and / or the point cloud 508 of the pelvis 512 may be associated with the preliminary 3-D model of the pelvis 504B. In this embodiment, the preliminary 3-D model 504 of FIG. 28 includes both the preliminary 3-D model of the femur 504A and the preliminary 3-D model of the pelvis 504B.

[0257] Referring to FIGS. 29A, 29B, 30A, and 30B, the point clouds 506, 508 may not include at least some portions of the respective bones 510, 512. For example, for the femur 510, a plurality of portions of the femoral head 510A may not be included in the point cloud 506, which may be because, for example, but not limited to, the femoral head 510A may be at least partially blocked from ultrasonic imaging by a plurality of portions of the pelvis 512. As another example, for the pelvis 512, a plurality of portions of the acetabular cup 512A may not be included in the point cloud 508, which may be because, for example, but not limited to, the acetabular cup 512A may be at least partially blocked from ultrasonic imaging by a plurality of portions of the femur 510. In some tissue contexts, the bone and / or joint can be repositioned to enhance tissue exposure to ultrasonic imaging. For example, as described above, the knee joint may be positioned at various degrees of flexion to assist in ultrasonic imaging of portions that may be blocked at other degrees of flexion. Similarly, in the context of the hip joint, the femoral head and acetabular cup or ring can be scanned in multiple postures, such as by dynamic scanning. In addition to increasing the portion of the surface of the tissue structure included in the point cloud, scanning in multiple postures can assist in determining the preoperative range of motion of the joint.

[0258] The above example relates to the outer surface of a bone where ultrasonic imaging is blocked by other bones. However, in some embodiments, some important internal features of some bones may not be included in point clouds 506, 508. For example, in the context of a total hip replacement surgery, the intermedullary canal of the femur can be an important tissue feature for selecting, sizing, and / or placing a femoral implant, particularly a femoral stem. Since ultrasonic signals generally may not penetrate the first bone they encounter (i.e., the bone surface), internal features of the bone such as internal bone canals (such as the medullary cavity of the femur) may not be included as part of the point clouds 506, 508 obtained using ultrasonic imaging. Similarly, some other tissues such as cartilage, tendons, and / or ligaments may be blocked by bones, etc., and thus may not be visible using ultrasound.

[0259] Referring to FIG. 30B, in the illustrated embodiment, the preliminary 3-D models 504A, 504B may include portions of the bones 510, 512 that are not included in the point clouds 506, 508. As discussed in detail above, the preliminary 3-D models 504A, 504B may be created by customizing a generalized bone model using the point clouds 506, 508. The generalized bone model may be obtained from a database or statistical atlas containing bone information. In the case of portions of bone or soft tissue that are missing in the point clouds 506, 508, in some embodiments, the corresponding portions of the preliminary 3-D models 504A, 504B may be generated using a generalized bone model such that they are customized by the available points in the point clouds 506, 508. Thus, the portions of the preliminary 3-D models corresponding to the portions of the bones 510, 512 not included in the point cloud 506 may be customized by the available points in the point clouds 506, 508, although those portions of the preliminary 3-D models 504A, 504B may not reflect all patient-specific tissue deviations from the generalized 3-D model. Specifically, in the illustrated embodiment, the geometric shape of the femoral head and / or the geometric shape of the acetabular cup may be primarily based on the generalized 3-D model, as patient-specific information about these portions may not be available using ultrasound imaging.

[0260] In other embodiments, some portions of important tissues may not be included in the generalized 3-D model. For example, some generalized 3-D bone models may not include some features such as the medullary cavity.

[0261] Returning to FIG. 28, method 500 may include operation 514, which includes obtaining one or more auxiliary images 516 of one or more critical tissues. In the illustrated embodiment, the auxiliary image 516 comprises a digital two-dimensional still X-ray image of one or more tissue aspects of interest, such as bone and soft tissue. In other embodiments, the auxiliary image 516 may include one or more scanned X-ray images, or one or more 2-D still images obtained from fluoroscopy, as well as one or more 2-D and / or 3-D images obtained via other imaging modalities. For example, the auxiliary image 516 may comprise a partial X-ray image and / or multiple X-ray images. In various exemplary embodiments, the auxiliary image 516 may be obtained in connection with routine diagnostic imaging using routine diagnostic imaging equipment and / or obtained specifically for use in connection with 3-D model generation as described herein. Generally, in some exemplary embodiments, the point clouds 506, 508 may be obtained using a first imaging modality (e.g., ultrasound), and the auxiliary image 516 may be obtained using a second different imaging modality (e.g., 2-D X-ray, CT, MRI, or fluoroscopy).

[0262] FIG. 31 is an exemplary auxiliary image 516 comprising a 2-D X-ray, according to at least some aspects of the present disclosure.

[0263] Referring to FIGS. 29A, 29B, and 31, in the illustrated embodiment, the auxiliary image 516 shows at least a portion of the femur 510 and at least a portion of the pelvis 512. Thus, in the illustrated embodiment, this auxiliary image 516 can be utilized (e.g., as an auxiliary image) in connection with generating a virtual 3-D patient-specific tissue bone model for each of the femur 510 and the pelvis 512. That is, one auxiliary image can be utilized in connection with models of more than one tissue structure of interest. In other embodiments, separate auxiliary images can be acquired for the various tissue structures being modeled. Also, in some exemplary embodiments, two or more auxiliary images can be acquired and / or utilized in connection with modeling a particular tissue structure. For example, more than one 2-D x-ray view can be utilized to gather information that can be obtained from one or more views but not from other views, or information that can be confirmed or verified using additional views.

[0264] In the illustrated embodiment, the auxiliary image 516 includes at least some portions of the relevant tissue not included in the respective point clouds 506, 508. For example, in the illustrated embodiment, the auxiliary image 516 shows the size and / or shape of the femoral head 510A, the size and / or shape of the acetabular cup 512A, and / or the size and / or shape of the medullary canal 510B of the femur 510. More generally, in the illustrated embodiment, the auxiliary image 516 includes data regarding at least a portion of the tissue not included in the imaging (e.g., ultrasound imaging) used to generate the preliminary 3-D model 504. For example, an auxiliary image 516 comprising an x-ray image may show the internal structure of the bone clearly.

[0265] Returning to FIG. 28, method 500 may include operation 518, which includes aligning preliminary virtual 3-D model 504 and auxiliary image 516. More specifically, in the illustrated embodiment, the preliminary virtual 3-D model of femur 504A may be aligned with a portion of auxiliary image 516 showing femur 510, and / or the preliminary virtual 3-D model of pelvis 504B may be aligned with a portion of auxiliary image 516 showing pelvis 512.

[0266] FIG. 32 shows preliminary 3-D model 504 (shown as surface points) aligned with auxiliary image 516, according to at least some aspects of the present disclosure.

[0267] Prior to or simultaneously with performing 2-D-3-D alignment 518, exemplary process 500 may include image distortion correction. Specifically, using the 3D preliminary model output from the initial ultrasound imaging, 2-D images from additional imaging modalities (e.g., X-ray images) may be processed by an image distortion correction algorithm that calculates the relative image magnification with respect to the images of that imaging modality and the orientation of tissue features in 3-D space. Specifically, the image distortion correction algorithm aligns the 3-D bone model with the 2-D image to determine the optimal magnification and tissue placement in the 3-D image relative to the 2-D image. The comparison of the 3-D model and the 2-D image is performed for all or a plurality of 2-D images, which enables the algorithm to align the 2-D images in space and extract surface contours in regions where ultrasound data cannot be collected. Examples where ultrasound data may not be available include, but are not limited to, the femoral head, the intramedullary canal of the femur, and the acetabular cup.

[0268] Referring to FIGS. 28 and 32, in the illustrated embodiment, the alignment operation 518 may include solving for the pose (e.g., relative placement and / or orientation) and / or relative scale / enlargement of the preliminary virtual 3-D model 504 that produces a 2-D projection corresponding to the projection of the auxiliary image 516. In some exemplary embodiments, tissue features visible in both the preliminary virtual 3-D model 504 and the auxiliary image 516 may be utilized in the alignment operation. Various methods such as optimization may be utilized in the alignment. In some exemplary embodiments, the alignment may be performed individually for tissue structures of interest such as the femur 510 and / or the pelvis 512.

[0269] In some exemplary embodiments, it may not be necessary to separately determine the scale and / or enlargement of the auxiliary image 516 (e.g., X-ray image). Specifically, since the size of the tissue structure can be determined from the ultrasonic point clouds 506, 508, the scale / enlargement of the auxiliary image 516 can be determined in relation to the alignment operation 518.

[0270] Returning to FIG. 28, the method 500 may include an operation 520 that includes extracting geometric information from the auxiliary image 516. The geometric information extracted from the auxiliary image 516 may include, for example, but not limited to, one or more length dimensions, angular dimensions, curvatures, etc. related to important bones or soft tissues.

[0271] Referring to FIGS. 28 and 31, in the illustrated embodiment, the geometric information to be extracted can relate to, for example, but not limited to, the size and / or shape of the femoral head 510A, the size and / or shape of the intramedullary canal of the femur 510B, the size and / or shape of the acetabular cup 512A, the thickness of the cartilage in the acetabular fossa, and / or the position of the ligamentum teres femoris. Thus, in this embodiment, at least a portion of the extracted geometric information can relate to portions of the bones 510, 512 that were not included in the respective point clouds 506, 508. More generally, in some exemplary embodiments, the auxiliary image 516 can provide patient-specific data (e.g., the point clouds 506, 508 obtained using ultrasonic imaging) regarding at least a portion of the tissue that was not included in the imaging used to generate the preliminary virtual 3-D model. Similarly, in some exemplary embodiments, the auxiliary image 516 can provide patient-specific information regarding at least a portion of the tissue on which the preliminary virtual 3-D model was primarily based on a generalized 3-D model. In some embodiments, the extracted geometric information can include one or more parameters that can be directly measured from, or otherwise directly obtained from, the auxiliary image 516, such as the size and / or shape of the bone feature. In some embodiments, the extracted geometric information can be utilized to predict and / or estimate parameters that were not directly measured from, or otherwise directly obtained from, the auxiliary image 516. For example, the thickness of the cartilage (e.g., the hip joint cartilage) in the acetabular fossa can be predicted and / or estimated using data from ultrasonic imaging and / or geometric information extracted from the auxiliary image 516 (e.g., an X-ray image) by using statistical techniques for estimating cartilage based on bone information. In other embodiments, for example, the articular cartilage in other joints can be estimated (e.g., knee joint cartilage, shoulder joint cartilage).

[0272] Returning to FIG. 28, method 500 may include operation 522 of generating an improved virtual 3-D patient-specific bone model 524 by improving the preliminary virtual 3-D model 504 using at least a portion of the geometric information extracted from the auxiliary image 516. FIG. 33 shows an improved virtual 3-D model 524 produced by operation 522, superimposed on the auxiliary image 516, according to at least some aspects of the present disclosure. A fusion step is then performed to fuse the data extracted from the 2-D image with the 3D ultrasound point cloud (or 3D model), followed by a morphing step to create a new 3D model that more accurately captures information from both the preliminary 3-D model and the 2-D image.

[0273] Referring to FIGS. 28 and 33, since the preliminary 3-D model 504 and the auxiliary image 516 were aligned in operation 518, the geometric information extracted from the auxiliary image 516 in operation 520 can be correlated with the preliminary 3-D model 504. In the illustrated embodiment, the extracted geometric information regarding the size and / or shape of the femoral head 510A and / or the acetabular cup 512A can be utilized to improve the respective portions of the preliminary 3-D model 504. Specifically, since the point clouds 506, 508 do not include these portions and at least a portion of the geometric information extracted from the auxiliary image 516 relates to these portions, in the illustrated embodiment where the preliminary 3-D model 504 is not primarily based on a generalized 3-D bone model, the improved 3-D model 524 produced by the improvement operation 522 can provide a patient-specific representation of these portions of the tissue that is more accurate than the preliminary 3-D model.

[0274] In addition, in some exemplary embodiments where the 3-D bone model generalized by the statistical atlas may lack some features, geometric data extracted from the auxiliary image 516 may be used to add such features to the preliminary 3-D model 504. For example, in the illustrated embodiment, the generalized 3-D bone model may not include the medullary canal of the femur 510B. Thus, the preliminary 3-D model may not include the medullary canal 510B. The medullary canal 510B may be visible on the auxiliary image 516, and important geometric information regarding the medullary canal 510B may be extracted from the auxiliary image 516 in operation 520. In the illustrated embodiment, the geometric information regarding the medullary canal 510B extracted from the auxiliary image 516 may be used in the refinement operation 522 to add features of the medullary canal 510B, resulting in a refined 3-D model 524 that includes a patient-specific representation of the medullary canal 510B.

[0275] In the illustrated embodiment, the refined 3-D model 524 may include the external topography of the femur and / or pelvis in the vicinity of the hip joint. In some exemplary embodiments, the refined 3-D model 524 may be used to obtain tissue measurements of the relevant tissue features. For example, FIG. 34 shows an exemplary display 526 that aids in tissue measurement using a refined 3-D model 528 of the pelvis. Various positions, dimensions, angles, curvatures, etc. may be determined and / or shown on the model 528, such as in the form of annotations 530, 532, 534. This information may be used, for example, for preoperative planning, implant design and / or selection (e.g., size determination), etc.

[0276] For example, referring to FIG. 34, an improved 3-D model 524 can be used to determine leg length and / or offset, as well as femoral inclination and / or acetabular inclination, including the combined inclination. As used herein, "femoral inclination" can refer to the relationship of the axis of the femoral neck to the transepicondylar axis of the distal end of the femur. As used herein, "acetabular inclination" can refer to the angle between a line connecting the anterior and posterior acetabular rims and a line orthogonal to a transverse reference line passing through any of the femoral head center, the posterior acetabular wall, or the posterior side of the ischium. As used herein, "combined inclination" can refer to the sum of the femoral inclination and the acetabular inclination. In some exemplary embodiments, imaging (e.g., ultrasound) can be performed (e.g., imaging of the femoral neck) to provide data regarding neck inclination. In some exemplary embodiments, various images and / or other data regarding the various inclinations may be provided to a preoperative planner and / or used to create a jig. The calculated inclinations can be used preoperatively and / or intraoperatively for planning, such as to reproduce the angle of the neck. Similarly, in some exemplary embodiments, imaging (e.g., ultrasound) can be performed (e.g., imaging of the acetabular rim) to provide data regarding acetabular inclination and / or cup inclination angle. The images can be used in preoperative planning to reproduce the angle with an implanted cup. More generally, various 3-D bone models generated in accordance with at least some aspects of the present disclosure can be used preoperatively, such as to ensure appropriate inclinations and cup inclination angles.

[0277] In some exemplary embodiments, the various methods described herein (e.g., method 500) may be performed preoperatively, and / or the generated models (e.g., improved 3-D model 524) may be used preoperatively (e.g., for surgical planning such as femoral stem sizing, cup placement determination, etc.), intraoperatively (e.g., for surgical navigation), and / or postoperatively (e.g., for postoperative evaluation). In some exemplary embodiments, the 3-D models generated by the exemplary methods described herein may be aligned intraoperatively using ultrasound. Unlike intraoperative fluoroscopy, using a 3-D model with ultrasound alignment during surgery does not expose the patient or nearby person (e.g., the surgeon) to ionizing radiation. In some exemplary embodiments, the preliminary 3-D model 504 may be aligned and utilized intraoperatively using ultrasound without being improved by the auxiliary image 516.

[0278] In addition, the present disclosure takes into account that in the context of total hip arthroplasty, many hip arthroplasties can be performed using a posterior lateral approach and / or an anterior lateral approach. The use of these approaches typically involves placing the patient in the lateral decubitus position. When the patient is positioned laterally in this way, radiation imaging generally cannot accurately evaluate the femoral inclination and / or the acetabular inclination. The intraoperative ultrasound alignment of the 3-D models generated according to at least some aspects of the present disclosure may be capable of accurately determining the femoral inclination, the acetabular inclination, and / or the combined inclination during surgery, such as in real-time or near real-time. More generally, the intraoperative ultrasound alignment of the 3-D model may be useful when the intraoperative patient placement is not suitable for alignment using other imaging modalities.

[0279] The examples described herein relate to the hip joint and may describe 3-D models of the femur and pelvis, but it should be understood that, to achieve similar results, it is within the scope of the present disclosure for the femur and pelvis to be replaced by any one or more tissue structures (e.g., one or more bones or soft tissues). As a more detailed example, using ultrasonic imaging taken proximal to the scapula and humerus, the scapular joint may be the subject of this practice. After a preliminary patient-specific virtual 3-D bone model is created, these bone models may be further refined using one or more 2-D X-ray images.

[0280] Various exemplary embodiments according to at least some aspects of the present disclosure may include an apparatus (e.g., ultrasonic device 50 (FIG. 1)) configured to implement method 500. Some exemplary embodiments may include a memory (e.g., memory 80 (FIG. 3) or a non-transitory computer-readable medium) comprising instructions that, when executed by a processor (e.g., CPU 78 (FIG. 3)), cause the processor to implement method 500.

[0281] Determination of Spinal-Pelvic Tilt The present disclosure contemplates that preoperative planning for some surgeries may include assessment and planning of function. For example, in preoperative planning for hip replacement surgery, it may be useful to consider a patient's spinal-pelvic tilt in one or more functional limb positions.

[0282] FIG. 35 is a flowchart illustrating an exemplary method 600 for determining spinal-pelvic tilt according to at least some aspects of the present disclosure. Method 600 may include an operation 602 that may include obtaining a virtual 3-D model 604 of a patient's pelvis. In the illustrated embodiment, this operation 602 may include generating a 3-D model of the patient's pelvis using ultrasound, as described elsewhere herein.

[0283] Method 600 may include operation 606A, which may include obtaining one or more ultrasonic point clouds 608 of a patient's pelvis and / or spine in a first functional limb position among a series of functional limb positions. In the illustrated embodiment, operation 606A may include obtaining a 3-D ultrasonic point cloud of pelvis 608A and / or an ultrasonic point cloud of at least a portion of spine 608 (e.g., lumbar spine and / or sacrum) in the first functional limb position (e.g., sitting position). In some exemplary embodiments, the point cloud 608 may generally be sparse. In some exemplary embodiments, the point cloud 608 may also include additional points regarding the patient's femur, which may assist in determining femoral inclination, acetabular inclination, and / or combined inclination. For example, the point cloud 608 may include sufficient data to identify the superior and / or posterior condylar axes of the femur to determine the reference axis of the femoral inclination angle. Further, in some exemplary embodiments, information regarding leg length may be obtained. For example, data from at least one X-ray image taken of a patient in a standing position may be obtained.

[0284] Method 600 may include operation 612A, which may include aligning at least a portion of the point cloud 608 with a 3-D model 604 of the pelvis. In the illustrated embodiment, the ultrasonic point cloud of pelvis 608A may be aligned with the 3-D model 604 of the pelvis.

[0285] Method 600 may include operation 614A, which may include determining the spine-pelvis inclination in the first functional limb position using the relative angle of the point cloud of spine 610A with respect to the 3-D model 604 of the pelvis.

[0286] Method 600 may include operations 606B, 612B, and 614B, which may be substantially similar to operations 606A, 612A, and 614A, respectively, except that they are performed on important tissues in a second functional limb position (e.g., standing position). Similarly, method 600 may include operations 606C, 612C, and 614C, which may generally correspond to operations 606A, 612A, and 614A, respectively, except that they are performed on important tissues in a third functional limb position (e.g., lateral lying position).

[0287] Some exemplary embodiments may enable 3-D visualization of spinal-pelvic interactions and / or may provide information about spinal-pelvic interactions in each functional limb position.

[0288] Various exemplary embodiments according to at least some aspects of the present disclosure may include an apparatus (e.g., ultrasonic device 50 (FIG. 1)) configured to implement method 600. Some exemplary embodiments may include a memory (e.g., memory 80 (FIG. 3) or a non-transitory computer-readable medium) that, when executed by a processor (e.g., CPU 78 (FIG. 3)), causes the processor to implement method 600.

[0289] 3-D Soft Tissue Reconstruction Some exemplary embodiments described herein may focus on generating a virtual 3-D model of bone. Some other embodiments according to the present disclosure may include generating a virtual 3-D model that focuses on and / or includes soft tissues such as ligaments, tendons, cartilage, etc. For example, FIG. 36 is a flowchart of an exemplary method 700 for generating a virtual 3-D patient-specific model 702 of a tissue structure (e.g., knee joint 704 including femur 706 and tibia 708) including at least one ligament (e.g., medial collateral ligament 710 and lateral collateral ligament 712) and / or other soft tissues (e.g., cartilage 714) according to at least some aspects of the present disclosure. Various aspects of the method may be utilized in connection with modeling various other tissue structures, including individual tissue structures (e.g., individual soft tissues and / or bones), joints (such as hip joint, knee joint, shoulder joint, ankle joint, etc.) including multiple tissue structures.

[0290] Method 700 may include operation 716, which includes reconstructing a joint (e.g., knee 704) using ultrasound. This operation 716 may be performed in a manner generally described elsewhere in this specification and / or may include obtaining one or more point clouds 718, 720 associated with bones 706, 708. As described in detail elsewhere in this specification, operation 716 may generate one or more patient-specific virtual 3-D images and / or models of one or more tissue structures associated with joint 702. For example, the output of operation 716 may comprise one or more patient-specific virtual 3-D bone models.

[0291] Method 700 may include operation 720, which includes automatically detecting one or more ligament sites (e.g., where the ligament attaches to the bone) on a patient-specific virtual 3-D bone model. For example, operation 720 may include determining insertion positions 722, 724 of the medial collateral ligament 710. All ligament sites are predefined in a template bone model, which is stored in an.IV format (a special 3D surface representation that maintains the correspondence of the model during processing). Ligament sites are specified by a set of indices stored in text format. To detect ligament sites in a patient-specific bone model, the system reconstructs a patient-specific virtual 3D model of the bone that maintains the correspondence to the template bone model. The virtual 3D patient-specific bone model and the template bone model may be different, but they share some common characteristic of the same bone. By specifying ligament sites in the template bone model, the system can automatically detect ligament sites in the patient-specific bone model. This innovative approach enables more accurate and efficient detection of ligament sites, which is essential in various medical applications.

[0292] Method 700 may include operation 726, which includes an ultrasound scan of at least a portion of at least one ligament associated with at least one of the detected ligament sites 722, 724. For example, in the illustrated embodiment, the medial collateral ligament 710 may be scanned using an ultrasound probe 728.

[0293] In some exemplary embodiments, an ultrasound operator may be provided with automated guidance information for performing a ligament scan. For example, a display 730, which may be shown on an output device such as monitor 58 (FIG. 1) and / or monitor 58' (FIG. 8), may indicate the current relative placement of the ultrasound probe 728 with respect to one or more tissue structures (such as bones 706, 708, ligament points 722, 724, etc.). In some exemplary embodiments, the display 730 may provide specific guidance for performing a ligament scan, such as an indication (e.g., arrow 732) indicating the desired position and / or direction of the scan. The ultrasound operator may perform a ligament scan using the displayed information. In some exemplary embodiments, the display 730 may include an A-mode or B-mode ultrasound image 734.

[0294] Method 700 may include an operation 736 that includes reconstructing a virtual 3-D model of a soft tissue (such as ligament 710). One exemplary method includes using ultrasound. Ultrasound is a dynamic imaging modality, which means that when a transducer is fixed in place and the object being imaged is moved, changes in the geometric shape and spatial position of the object can be captured. With this in mind, an exemplary method may guide a user of the ultrasound transducer to move the ultrasound transducer to specific positions where soft tissues, such as tendon attachment positions, muscle attachment positions, and ligament attachment positions of bones, can be imaged, using the reconstructed bones and points thereon, similar to GPS coordinates. At each predetermined position, the user keeps the transducer relatively stationary while repositioning the tissue joint, enabling the acquisition of ultrasound data regarding changes in the position and cross-sectional area of the soft tissue. This information is utilized along with changes in the structural joint tissue and joint flexion angle to construct a 3-D soft tissue.

[0295] Additionally or alternatively, the method may utilize machine learning to generate a 3D model of the soft tissue. By way of example, the machine learning may include a 2-D and / or 3-D data training set that is specified and associated with a given set of features specific to a particular soft tissue. As mentioned herein, in an exemplary form, dynamic ultrasound imaging may be utilized to image the movement of the soft tissue in real time. By combining machine learning with dynamic ultrasound imaging, the accuracy and efficiency of the constructed 3-D soft tissue is improved.

[0296] In some exemplary embodiments, one or more of operations 716, 720, 726, 736 may be repeated one or more times at one or more joint angles, for example, over the range of motion of a joint. Thus, in some exemplary embodiments, a virtual 3-D model of the ligament over the entire range of motion may be generated.

[0297] Various exemplary embodiments according to at least some aspects of the present disclosure may include an apparatus (e.g., ultrasound device 50 (FIG. 1)) configured to perform method 700. Some exemplary embodiments may include a memory (e.g., memory 80 (FIG. 3) or a non-transitory computer-readable medium) that, when executed by a processor (e.g., CPU 78 (FIG. 3)), causes the processor to perform method 700.

[0298] Guided diagnostic scan Some exemplary embodiments may be configured to provide guidance to an ultrasound operator, which may facilitate obtaining improved, more accurate, and / or more reproducible ultrasound scan results than relying solely on the skills and / or experience of the ultrasound operator. In some exemplary embodiments, the guidance may be provided automatically. For example, arrow 732 in FIG. 36 shows exemplary guidance information provided to the ultrasound operator.

[0299] Some exemplary embodiments may be configured to provide one or more displays, including information about the relative current placement (e.g., periodically and / or constantly updated) of an ultrasonic probe with respect to one or more tissue structures, on a monitor 58 (FIG. 1) and / or a monitor 58' (FIG. 8), etc. For example, FIG. 37 is an exemplary display shown during an ultrasonic scan of the femur, FIG. 38 is an exemplary display shown during an ultrasonic scan of the outer surface of the knee, FIG. 39 is an exemplary display shown during an ultrasonic scan of the outer surface of the knee, FIG. 40 is an exemplary display shown during an ultrasonic scan of the inner surface of the knee, all in accordance with at least some aspects of the present disclosure.

[0300] Referring to FIG. 37, an exemplary display 800 may be shown in relation to an ultrasonic scan of the femur. In the embodiment shown, the display 800 may include a relative placement representation 802, which may include a representation of the femur 804 and / or a representation of the ultrasonic probe 806. In the relative placement representation 802, the representation of the femur 804 and the representation of the ultrasonic probe 806 may be arranged on the display 800 in a manner that indicates the current relative placement of the corresponding physical objects. Additionally, in the embodiment shown, the display 800 may include an A-mode or B-mode ultrasonic image 808 corresponding to the current ultrasonic data being acquired by the ultrasonic probe.

[0301] Referring to FIG. 38, an exemplary display 820 can be shown in connection with an ultrasonic scan of the outer surface of the knee. In the illustrated embodiment, the display 820 may include a relative arrangement representation 822, which may include a representation of the knee 824 (e.g., a representation of the femur 804 and / or a representation of the tibia 826) and / or a representation of the ultrasonic probe 806. In the relative arrangement representation 822, the representation of the knee 824 and the representation of the ultrasonic probe 806 can be arranged on the display 820 in a manner that indicates the current relative arrangement of the corresponding physical objects. In some exemplary embodiments regarding a tissue structure having a plurality of components such as a knee with a femur and a tibia, the relative arrangement of the components of the tissue structure (e.g., the representation of the femur 804 and the representation of the tibia 826) with respect to each other and with respect to the representation of the ultrasonic probe 806 can be shown. Additionally, in the illustrated embodiment, the display 820 may include an A-mode or B-mode ultrasonic image 808 corresponding to the current ultrasonic data acquired by the ultrasonic probe.

[0302] Referring to FIG. 39, an exemplary display 840 can be shown in connection with an ultrasonic scan of the outer surface of the knee. In the illustrated embodiment, the display 840 may be shown on the monitor 58, which may be located near the imaged tissue structure (e.g., the patient's knee 842) and the ultrasonic probe, such as within the field of view of the ultrasonic operator. During the imaging / scan procedure, the relative arrangement of the representation of the ultrasonic probe 806 and the representation of the imaged tissue structure (e.g., the representation of the knee 824) can be shown.

[0303] Referring to FIG. 40, an exemplary display 900 can be shown during an ultrasonic scan of the inner surface of the knee. In the illustrated embodiment, the display 900 may be shown on the monitor 58, which may be located near the imaged tissue structure (e.g., the patient's knee 842) and the ultrasonic probe, such as within the field of view of the ultrasonic operator. During the imaging / scan procedure, the relative arrangement of the representation of the ultrasonic probe 806 and the representation of the imaged tissue structure (e.g., the representation of the knee 824) is shown.

[0304] Bone alignment system for intraoperative procedures Intraoperative surgical procedures involving bone manipulation benefit from the accurate alignment of the patient's bone or tissue model to the patient's actual tissue. Accurate alignment of the preoperative 3D patient-specific tissue model with the intraoperative patient tissue can help reduce surgical complications and improve the overall surgical outcome. However, current alignment techniques have limitations in achieving accurate alignment, especially when there are deformed or changed areas in the patient's tissue (including bone and soft tissue). Most existing tissue alignment methods are performed after an incision, leading to blood loss and other surgical complications. Therefore, there is a need for a reliable, accurate, non-invasive, and bloodless tissue alignment system for intraoperative procedures that can overcome the limitations of existing techniques.

[0305] The present disclosure provides a tissue alignment system that enables accurate alignment of a preoperative 3D patient-specific tissue model to the intraoperative patient tissue. The tissue alignment system includes an ultrasonic probe, a computer algorithm, and a point cloud alignment module. The system uses a combination of tissue landmarks and ultrasonic scans to achieve accurate alignment of the preoperative 3D patient-specific tissue model with the intraoperative patient tissue.

[0306] In an exemplary embodiment, the intraoperative tissue alignment system may include an initial alignment step, which is preferably performed prior to any incision made during the operation. This initial alignment step may include identifying one or more (e.g., two, three, four, or more) predefined tissue landmarks on the preoperative 3D patient-specific tissue model. By way of example, these tissue landmarks can be readily recognizable features such as the tip of a bone or the attachment point of a muscle to a bone. After identifying the one or more tissue landmarks, an ultrasound probe (including the ultrasound transducer) can be utilized to scan the corresponding tissue of the patient corresponding to each tissue landmark such that ultrasound data is recorded while tracking the 3-D placement of the ultrasound transducer. Using the ultrasound data and the placement tracking data, an algorithm uses a feature-based method to estimate the placement and orientation (pose) of the preoperative 3D patient-specific tissue model with respect to the intraoperative patient model generated using ultrasound in the operating room, thereby initially aligning the preoperative 3D patient-specific tissue model to the patient's intraoperative tissue.

[0307] After the initial alignment, the exemplary methods disclosed herein may utilize a refined alignment. The refined alignment may be performed after the preoperative 3D patient-specific tissue model is aligned with the patient's intraoperative bone, at which time the ultrasound transducer is repositioned to scan the patient's tissue to generate a 3-D point cloud corresponding to points on one or more surfaces of the patient's tissue (e.g., bone surface points). The 3-D point cloud can then be aligned to the preoperative 3D patient-specific tissue model to fine-tune the placement of the preoperative 3D patient-specific tissue model. In an exemplary embodiment, the alignment of the 3-D point cloud to the 3-D patient model can be performed by aligning the point cloud to the preoperative 3D patient-specific tissue model using an iterative closest point (ICP) algorithm. The ICP algorithm minimizes the distance between corresponding points of the preoperative 3D patient-specific tissue model and the point cloud, thereby achieving an accurate alignment.

[0308] Exemplary bone alignment systems and methods can provide one or more advantages. By way of example, one advantage is the accurate alignment of a preoperative 3D patient-specific tissue model with the patient's tissue during surgery. Another advantage is improved surgical outcomes and reduced complications. And a further advantage is the ability to align a preoperative model with the patient's intraoperative tissue when the tissue has undergone significant changes or exhibits significant deformities. The exemplary systems and methods can be used in any surgery where it is advantageous to correlate the virtual world with the real world.

[0309] Non-invasive pinless bone and spine tracking systems and methods using ultrasound and localization techniques The present disclosure provides systems and related non-invasive methods for tracking a patient's tissue (including bones such as the pelvis and vertebrae) during surgery using ultrasound and localization techniques.

[0310] In orthopedic and spinal surgeries, computer-assisted surgery is used to track and localize bones and the spine using bone arrays or optical trackers. These trackers are typically attached to the patient's bone, which can lengthen the postoperative recovery period and, in some cases, cause complications. Accordingly, there is a need for non-invasive methods for tracking a patient's tissue (including bones such as the pelvis and vertebrae) during surgery.

[0311] The present disclosure provides a pinless bone and spine tracking system and method using a custom-made ultrasonic probe. An exemplary ultrasonic probe may include a tissue shape that conforms to the soft tissue around the target bone or spine. By way of example, the ultrasonic probe may be shaped to interact with the outside of the patient's tissue in only a single placement and orientation such that signals received by the ultrasonic probe during surgery have a fixed reference frame. Specifically, the surface of the bone or spine is detected by the ultrasonic probe and used to generate a 3-D point cloud representing surface points on the bone or spine. These surface points are constantly tracked in real time (using one or more electromagnetic (EM) sensors, optical arrays, inertial measurement units, etc.) to provide the surgeon with information regarding the current placement and orientation of the tissue.

[0312] In an exemplary form, an exemplary pinless bone and spine tracking system and method may utilize a customized ultrasonic probe having a tissue shape that conforms to the soft tissue around the target bone or spine. The ultrasonic probe includes an ultrasonic transducer that detects the surface of the bone or spine by generating data representing echoes used by a computer system and associated algorithms to generate a 3-D point cloud representation of the bone or spine in a static placement by receiving ultrasonic echoes. The ultrasonic probe may be equipped with a tracker to track the 3-D placement and orientation of the ultrasonic probe. Exemplary trackers include, but are not limited to, electromagnetic (EM) sensors, optical arrays, and inertial measurement units. As discussed above, after the 3-D point cloud is generated, the 3-D point cloud is registered to the patient's tissue. After registration, the ultrasonic probe may be utilized to track the bone or vertebra by combining a motion signal from the ultrasonic probe with a tissue depth measurement from echoes received by the ultrasonic transducer, thereby enabling real-time non-invasive tracking of the tissue.

[0313] As an example, an exemplary system that uses a pinless bone and spine tracking system can be used by placing an ultrasonic probe on the patient's skin near the target bone or vertebra. The ultrasonic probe can then be repositioned in 3-D, and its 3-D placement and orientation are tracked to perform a scan of the patient's tissue (preferably stationary). Since those skilled in the art are familiar with ultrasonic transducers and scanning of patient tissue, a detailed description of this aspect of the method is omitted for the sake of brevity. Information regarding the placement and orientation of the ultrasonic probe is supplied to a computer during the scan, and the ultrasonic probe also generates signal data indicative of the echoes detected by the transducer, which enables the computer system to generate 3-D points corresponding to surface points of the tissue of interest, such as one or more bones such as vertebrae. When combined, these 3-D points are operable to form a 3-D point cloud, and those skilled in the art will appreciate that such point clouds represent the patient's tissue or real-world tissue. The system then operates to align the point cloud to a patient-specific tissue model generated preoperatively (and optionally reinforced preoperatively as discussed herein), and optionally displays the patient-specific tissue model on a graphical display available to the surgeon. After alignment, the placement and orientation data from the ultrasonic probe are combined by the computer system with the signal data from the transducer to generate one or more 3-D points, which are correlated with the patient-specific tissue model. In this way, the 3-D points are associated with the patient-specific tissue model to update the placement and orientation of the patient-specific tissue model displayed on the display in real time or near real time.

[0314] Of course, the exemplary systems and methods provide numerous advantages over conventional surgical tracking systems. By way of example, the exemplary surgical tracking system is non-invasive, which reduces possible complications and shortens the patient's recovery time compared to invasive surgical trackers. When using a custom-molded ultrasound probe, the probe is configured to interact with the patient's tissue in a single placement and orientation, which simplifies the tracking of the target tissue, thereby providing a fixed reference frame for changes in the 3-D placement of the probe, as well as changes in the 3-D placement of the target tissue. Moreover, the exemplary systems and methods are not specifically tied to tracking techniques in any placement and orientation, and can be used with any placement and tracking technique, including but not limited to EM, IMU, and optical trackers.

[0315] In general, an apparatus related to the methods described herein may include a computer and / or processor configured to perform such methods, and software and / or a storage device comprising or storing instructions configured to cause the computer or processor to perform such methods. In some exemplary embodiments, some operations related to some methods may be performed by two or more computers and / or processors, which may or may not be in the same location. For example, some operations and / or methods may be performed by a remote computer or server, and the resulting output may be provided to other devices for preoperative, intraoperative, and / or postoperative use.

[0316] Although various exemplary embodiments have been described herein in relation to specific tissues, it will be understood that the same methods and devices may be utilized in relation to other tissues, such as any joint. For example, various embodiments according to at least some aspects of the present disclosure may be used in relation to tissue structures associated with the shoulder joint, hip joint, knee joint, ankle joint, and the like.

[0317] From the above description and the summary of the present invention, the methods and apparatuses described herein are components of exemplary embodiments in accordance with the present disclosure. However, it should be apparent to those skilled in the art that the scope of the present disclosure included herein is not limited to the above exact embodiments, and modifications can be made without departing from the scope as defined by the appended claims. Similarly, although not explicitly discussed herein, there may be inherent and / or unexpected advantages, so it should be understood that it is not necessary to meet any or all of the specific advantages or objectives disclosed herein in order to fall within the scope of the claims.

Explanation of Reference Numerals

[0318] 50 Ultrasonic device 52 Housing 54 Computer 56 User input device 58 Monitor 60 Ultrasonic probe 62 Caster wheel 64 Main body 66 Cable 68 Transducer array 70 Computer 72 Storage device 74 Network 76 Network interface 78 CPU 80 Memory 81 OS 82 Mass storage device 83 Program code 84 User interface 85 I / O interface 86 Optical tracking marker 88 Position sensor 90 Reflective part 92 Local frame 94 Hybrid probe 114 Knee joint 116 Femur 118 Tibia 120 Patella 122 Fibula 124 pieces 126 pieces 128 pieces 130 pieces 132 Knee surface 134 Articular surface 136 Tibiofemoral joint 138 Patellofemoral joint 140 Trochlear groove 142 A-mode RF signal 144 Epidermis 146 Ultrasonic frame 162 Echo 165 3-D point cloud 180 Bone contour 182 Noise 194 Bone point cloud 196 Landmark 200 Model femur 328 Electromagnetic transceiver unit 356 Ultrasonic transceiver 360 Controller 362 Ultrasonic signal 364 Echo 366 Electromagnetic tracking system controller 368 Antenna 370 First electromagnetic signal 372 Second electromagnetic signal 390a, 392a, 390b Plot lines 392a~392f Signal envelope 394 Envelope peak 396 Envelope peak 398 Envelope peak 412 Particles 414 Particles 416 Particles 418 Envelope peak 420 Envelope peak 422 Envelope peak 424 Estimated particles 426 Estimated particles 428 Presumed particle 430 Peak 432 Peak 434 Peak 436 Set of weighted particles 438 Set of weighted particles 440 Set of weighted particles 442 Peak 444 Projection 448 Contour 452 Scanning line 456 Contour line 458 Contour line 466 Surface 468 Point group 469 Scanned bone 504 Preliminary virtual 3-D bone model 506 Ultrasonic point group 508 Ultrasonic point group 510 Femur 512 Pelvis 516 Auxiliary image 524 Virtual 3-D patient-specific bone model 528 Improved 3-D model 530 Annotation 532 Annotation 534 Annotation 604 Virtual 3-D model 608 Ultrasonic point group 706 Femur 708 Tibia 710 Medial collateral ligament 712 Lateral collateral ligament 714 Cartilage 722 Insertion position 724 Insertion position 728 Ultrasonic probe 730 Display 734 Ultrasonic image 800 Display 802 Relative arrangement representation 804 Femur 806 Ultrasonic probe 808 Ultrasonic image 820 Display 822 Relative arrangement expression 824 Knee 826 Tibia 840 Display 842 Knee 900 Display

Claims

1. A method for generating a virtual 3-D patient-specific bone model, The steps include obtaining a preliminary virtual 3-D bone model of the patient's first bone, The steps include obtaining an auxiliary image of the first bone of the patient, The steps include aligning the preliminary virtual 3D bone model of the patient's first bone with the auxiliary image of the patient's first bone, A step of extracting geometric information about the patient's first bone from the auxiliary image of the patient's first bone, The steps of generating an improved virtual 3-D bone model of the patient's first bone by improving the preliminary virtual 3-D bone model of the patient's first bone using the geometric information of the patient's first bone from the auxiliary image of the patient's first bone, and A method that includes [a certain feature].

2. The method according to claim 1, wherein the step of obtaining the preliminary 3-D bone model comprises the steps of obtaining a point cloud of the patient's first bone and reconstructing the preliminary 3-D bone model by morphing a generalized 3-D bone model using the point cloud of the patient's first bone.

3. The step of acquiring the point cloud of the first bone of the patient utilizes a first imaging modality. The step of acquiring the auxiliary image of the first bone of the patient utilizes a second imaging modality. The first imaging modality is different from the second imaging modality. The method according to claim 2.

4. The method according to claim 3, wherein the first imaging modality comprises ultrasound.

5. The method according to claim 4, wherein the second imaging modality comprises 2-DX lines.

6. The method according to claim 3, wherein the second imaging modality comprises 2-DX lines.

7. The method according to claim 2, wherein the auxiliary image of the patient's first bone includes at least one portion of the patient's first bone that was not included in the point cloud of the patient's first bone.

8. The step of acquiring the point cloud of the patient's first bone comprises the step of performing an ultrasound scan of the patient's first bone, The step of acquiring the auxiliary image of the patient's first bone comprises the step of acquiring the 2-DX line of the patient's first bone, The 2-DX rays of the patient's first bone include at least one portion of the patient's first bone that was not available in the ultrasound scan of the patient's first bone, The method according to claim 7.

9. The method according to claim 8, wherein the at least one portion of the patient's first bone that was not available in the ultrasound scan of the patient's first bone was at least partially obscured from the ultrasound scan by tissue structure.

10. The method according to claim 8, wherein at least one portion of the patient's first bone that was not available in the ultrasound scan of the patient's first bone comprises the internal structure of the patient's first bone.

11. The method according to claim 10, wherein the obstructed internal structure of the first bone of the patient comprises a pulp canal.

12. The first bone of the aforementioned patient comprises a femur, The medulla includes the femoral medullary duct. The method according to claim 11.

13. The method according to claim 8, wherein at least one portion of the patient's first bone that was not visible in the ultrasound scan of the patient's first bone comprises the external structure of the patient's first bone.

14. The method according to claim 13, wherein the external structure of the patient's first bone was at least partially obscured from the ultrasound scan by the patient's second bone.

15. The method according to claim 14, wherein one of the patient's first bone and the patient's second bone comprises a femoral head, and the other of the patient's first bone and the patient's second bone comprises an acetabular cup.

16. The method according to claim 14, wherein the external structure of the patient's first bone, which was obscured from the ultrasound scan by the patient's second bone, comprises soft tissue.

17. The method according to claim 16, wherein the soft tissue comprises cartilage.

18. The method according to claim 17, wherein the cartilage comprises hip joint cartilage.

19. The method according to claim 17, wherein the cartilage comprises knee joint cartilage.

20. The method according to claim 17, wherein the cartilage comprises shoulder joint cartilage.

21. The method according to claim 14, wherein each of the first bone of the patient and the second bone of the patient comprises one or more of the pelvis, femur, tibia, patella, scapula, and humerus.

22. The method according to claim 1, wherein the first bone of the patient comprises one or more of the pelvis, femur, tibia, patella, scapula, and humerus.

23. The method according to claim 1, wherein the step of aligning the preliminary 3-D bone model of the patient's first bone with the auxiliary image of the patient's first bone comprises the step of solving for the orientation of the preliminary 3-D bone model, which produces a 2-D projection corresponding to the projection of the auxiliary image.

24. The step of acquiring an auxiliary image of the patient's first bone comprises the step of acquiring a plurality of auxiliary images of the patient's first bone, The step of aligning the preliminary virtual 3-D bone model of the patient's first bone with the auxiliary images of the patient's first bone comprises the step of aligning the preliminary virtual 3-D bone model of the patient's first bone with the plurality of auxiliary images of the patient's first bone, The step of extracting geometric information about the patient's first bone from the auxiliary image of the patient's first bone comprises the step of extracting geometric information about the patient's first bone from a plurality of auxiliary images of the patient's first bone, The step of improving the preliminary virtual 3-D bone model of the patient's first bone using the geometric information of the patient's first bone from the auxiliary images of the patient's first bone comprises the step of improving the preliminary virtual 3-D bone model of the patient's first bone using the geometric information of the patient's first bone from a plurality of auxiliary images of the patient's first bone. The method according to claim 1.

25. The steps of obtaining a preliminary virtual 3-D bone model of a patient's second bone, The steps include obtaining an auxiliary image of the second bone of the patient, The steps include aligning the preliminary virtual 3D bone model of the patient's second bone with the auxiliary image of the patient's second bone, A step of extracting geometric information about the patient's second bone from the auxiliary image of the patient's second bone, The steps of generating an improved virtual 3D patient-specific bone model of the patient's second bone by improving the preliminary virtual 3D bone model of the patient's second bone using the geometric information of the patient's second bone from the auxiliary image of the patient's second bone, and The method according to claim 1, further comprising:

26. The step of acquiring the point cloud of the patient's second bone comprises the step of performing an ultrasound scan of the patient's second bone, The step of acquiring the auxiliary image of the patient's second bone comprises the step of acquiring the 2-DX line of the patient's second bone, The 2-DX line of the patient's second bone includes at least one portion of the patient's second bone that was not visible in the ultrasound scan of the patient's second bone, The method according to claim 25.

27. The method according to claim 1, wherein the step of extracting geometric information from the auxiliary image of the patient's first bone comprises the step of extracting at least one of the length dimension, angle dimension, or curvature of the patient's first bone from the auxiliary image of the patient's first bone.

28. This is a method for determining the size of orthopedic implants before surgery. The steps of generating the improved virtual 3-D patient-specific bone model according to the method of claim 1, The steps include determining the size of the orthopedic implant using the improved virtual 3D patient-specific bone model, and A method that includes [a certain feature].

29. An apparatus configured to carry out the method described in claim 1.

30. A memory comprising, when executed by a processor, an instruction causing the processor to perform the method according to claim 1.

31. A method for generating a virtual 3-D patient-specific bone model, The steps include obtaining ultrasound data on the external surface of the patient's first bone, A step of acquiring X-ray data relating to at least one of the internal features of the first bone of the patient and / or the occluded features of the first bone of the patient, A step of generating a 3-D patient-specific bone model of the patient's first bone using the ultrasound data and the X-ray data. The 3-D patient-specific bone model comprises the external surface of the patient's first bone, the internal features of the patient's first bone, and / or the obscured features of the patient's first bone, method.

32. The step of acquiring the ultrasound data relating to the external surface of the patient's first bone comprises the steps of acquiring an ultrasound point cloud of the external surface of the patient's first bone and generating a preliminary 3-D bone model of the patient's first bone, The method according to claim 31, wherein the step of generating the 3-D patient-specific bone model of the patient's first bone comprises the step of improving the preliminary 3-D bone model of the patient's first bone using the X-ray data.

33. An apparatus configured to carry out the method described in claim 31.

34. A memory comprising, when executed by a processor, an instruction causing the processor to perform the method according to claim 31.

35. A method for determining spinal-pelvic tilt, Steps include obtaining a virtual 3-D model of the patient's pelvis, The steps include acquiring a first ultrasound point cloud of the patient's pelvis and a first ultrasound point cloud of the patient's lumbar spine while the pelvis and lumbar spine are in a first functional position, The steps include aligning the virtual 3-D model of the patient's pelvis with the first point cloud of the patient's pelvis, A step of determining a first spinal-pelvic tilt in a first functional position using a first relative angle of the first point cloud of the patient's lumbar spine with respect to the 3-D model of the patient's pelvis. A method that includes [a certain feature].

36. The method according to claim 35, further comprising the step of positioning at least one of the patient's pelvis or lumbar vertebrae into the first functional position.

37. The steps include acquiring a second ultrasound point cloud of the patient's pelvis and a second ultrasound point cloud of the patient's lumbar spine while the patient's pelvis and lumbar spine are in a second functional position, The steps include aligning the virtual 3-D model of the patient's pelvis with the second point cloud of the patient's pelvis, A step of determining a second spinal-pelvic tilt in the second functional position using the second relative angle of the second point cloud of the patient's lumbar spine with respect to the 3-D model of the patient's pelvis. The method according to claim 35, further comprising:

38. The method according to claim 37, further comprising the step of positioning at least one of the patient's pelvis or lumbar vertebrae of the patient in the second functional position.

39. The steps include acquiring a third ultrasound point cloud of the patient's pelvis and a third ultrasound point cloud of the patient's lumbar spine while the patient's pelvis and lumbar spine are in a third functional position, The steps include aligning the virtual 3-D model of the patient's pelvis with the third point cloud of the patient's pelvis, A step of determining a third spinal-pelvic tilt in the third functional position using the third relative angle of the third point cloud of the patient's lumbar spine with respect to the 3-D model of the patient's pelvis; The method according to claim 37, further comprising:

40. The method according to claim 39, further comprising the step of positioning at least one of the patient's pelvis or lumbar vertebrae into the third functional position.

41. The method according to claim 39, wherein each of the first functional position, the second functional position, and the third functional position comprises one of sitting, standing, and supine positions.

42. The method according to claim 35, wherein the step of obtaining the virtual 3-D model of the patient's pelvis comprises the step of generating the virtual 3-D model of the patient's pelvis using ultrasound.

43. The method according to claim 35, wherein the step of acquiring the first ultrasound point cloud of the patient's pelvis and the first ultrasound point cloud of the patient's lumbar spine in the first functional position comprises the step of acquiring a sparse ultrasound point cloud of the patient's pelvis and a sparse ultrasound point cloud of the patient's lumbar spine.

44. With the patient's pelvis and lumbar spine in the first functional position, at least one of the first ultrasound point cluster of the patient's pelvis and the first ultrasound point cluster of the patient's lumbar spine comprises additional points relating to the patient's femur, The method further comprises the step of determining at least one of femoral inclination, acetabular inclination, or combined inclination. The method according to claim 35.

45. The method according to claim 44, wherein the step of determining at least one of the femoral inclination, the acetabular inclination, or the combined inclination comprises the step of identifying the epicondylar axis or posterior condylar axis of the patient's femur in order to determine the reference axis of the femoral inclination angle.

46. The method of claim 35, further comprising the step of obtaining information regarding the length of the patient's legs by obtaining data from at least one X-ray image taken while the patient is standing.

47. An apparatus configured to carry out the method described in claim 35.

48. A memory comprising, when executed by a processor, an instruction causing the processor to perform the method according to claim 35.

49. A method for generating a virtual 3-D patient-specific model of a patient's ligaments, The steps include obtaining a virtual 3-D patient-specific bone model of the patient's joints, The steps include detecting the location of at least one ligament on the virtual 3D patient-specific bone model, The steps include: obtaining ultrasound data relating to the patient's ligaments, associated with at least one ligament location, by scanning the patient's ligaments using ultrasound; The steps include: reconstructing a virtual 3-D model of the patient's ligaments using the ultrasound data; A method that includes [a certain feature].

50. A method for generating a virtual 3-D patient-specific tissue model, The steps include obtaining a preliminary virtual 3-D tissue model of the patient's first tissue, The steps include obtaining an auxiliary image of the first tissue of the patient, The steps include aligning the preliminary virtual 3-D tissue model of the patient's first tissue with the auxiliary image of the patient's first tissue, A step of extracting geometric information about the patient's first tissue from the auxiliary image of the patient's first tissue, The steps of generating an improved virtual 3D patient-specific tissue model of the patient's first tissue by improving the preliminary virtual 3D tissue model of the patient's first tissue using the geometric information of the patient's first tissue from the auxiliary images of the patient's first tissue, and A method that includes [a certain feature].

Images