Method for determining accuracy of three-dimensional models for use in orthopaedic surgery - Patents.com
The system addresses inefficiencies in three-dimensional modeling for orthopedic surgery by using edge detection algorithms to automatically assess model accuracy, reducing manual calibration needs and improving precision through iterative alignment and scale determination.
Patent Information
- Application Number
- JP2020153563
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-09-27
- Filing Date
- 2020-09-14
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-09-14
AI Technical Summary
Existing three-dimensional modeling systems for orthopedic surgery face inaccuracies due to complex and time-consuming calibration processes, requiring manual verification of models against the original object, which is inefficient and prone to human error.
A system and method that uses edge detection algorithms to compare two-dimensional silhouettes of candidate three-dimensional models with edge-detected images, automatically determining the accuracy of the models without the need for complex calibration markers, employing iterative processes like gradient ascent or machine learning to optimize alignment and scale.
Automated accuracy assessment of three-dimensional models in orthopedic surgery, reducing human intervention and improving model precision by comparing two-dimensional silhouettes with edge images, thus enhancing the reliability of surgical planning.
Smart Images

Figure 0007580987000001 
Figure 0007580987000002 
Figure 0007580987000003
Abstract
Description
[Technical field]
[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application is related to U.S. patent application Ser. No. 16 / 586,887, filed Sep. 27, 2019, entitled "TECHNOLOGIES FOR DETERMINING THE SPATIAL ORIENTATION OF INPUT IMAGES FOR USE IN AN ORTHOPAEDIC SURGICAL PROCEDURE."
[0002] FIELD OF THEINVENTION The present disclosure relates to orthopaedic surgery, and more particularly, to techniques for determining the accuracy of three-dimensional models for use in orthopaedic surgery. [Background technology]
[0003] Some three-dimensional modeling systems, such as X-ray-based systems, generate a three-dimensional model of an object based on a set of two-dimensional images (e.g., X-ray images) of the object from different viewpoints. For example, to reduce the likelihood that the resulting model will be inaccurate due to imprecise orientation in three-dimensional space, an operator can place calibration markers (e.g., physical objects of known size and / or location) on or near the object being imaged and modeled. However, such calibration objects are complex and time-consuming to place. Furthermore, in such systems, the process of verifying the accuracy of the three-dimensional model requires an operator to visually inspect the model and compare it to the original object to determine whether it is properly aligned and sized. Summary of the Invention [Means for solving the problem]
[0004] In one aspect, the present disclosure describes a device having circuitry for obtaining a two-dimensional image of an anatomical object (e.g., a bone of a human joint), obtaining a candidate three-dimensional model of the anatomical object, and generating a two-dimensional silhouette of the candidate three-dimensional model. The circuitry is also for applying an edge detection algorithm to the two-dimensional image to generate a corresponding edge image, and comparing the two-dimensional silhouette to the edge image to generate a score indicative of the accuracy of the candidate three-dimensional model.
[0005] In another aspect, the disclosure describes one or more machine-readable storage media having instructions stored thereon that, when executed, cause a device to obtain a two-dimensional image of an anatomical object, obtain a candidate three-dimensional model of the anatomical object, and generate a two-dimensional silhouette of the candidate three-dimensional model. The instructions additionally cause the device to apply an edge detection algorithm to the two-dimensional image to generate a corresponding edge image, and compare the two-dimensional silhouette to the edge image to generate a score indicative of accuracy of the candidate three-dimensional model.
[0006] In yet another aspect, the present disclosure describes a method that includes acquiring, by a device, a two-dimensional image of an anatomical object, acquiring, by the device, a candidate three-dimensional model of the anatomical object, and generating, by the device, a two-dimensional silhouette of the candidate three-dimensional model. The method also includes applying, by the device, an edge detection algorithm to the two-dimensional image to generate a corresponding edge image, and comparing, by the device, the two-dimensional silhouette to the edge image to generate a score indicative of accuracy of the candidate three-dimensional model. [Brief description of the drawings]
[0007] The concepts described herein are illustrated by way of example, and not by way of limitation, in the accompanying drawings in which: For simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. Further, where considered appropriate, reference characters have been repeated among the figures to indicate corresponding or analogous elements. The detailed description makes specific reference to the following drawings: [Figure 1] 1 is a simplified diagram of an embodiment of a system for determining the accuracy of a three-dimensional model generated from a two-dimensional image of an object. [Diagram 2] FIG. 2 is a simplified block diagram of one embodiment of a model generation device that may be included in the system of FIG. [Diagram 3] FIG. 3 is a simplified block diagram of one embodiment of a method that may be performed by the model generation device of FIGS. 1 and 2 for determining the accuracy of a three-dimensional model generated from a two-dimensional image of an object. [Figure 4] FIG. 3 is a simplified block diagram of one embodiment of a method that may be performed by the model generation device of FIGS. 1 and 2 for determining the accuracy of a three-dimensional model generated from a two-dimensional image of an object. [Diagram 5] FIG. 3 is a simplified block diagram of one embodiment of a method that may be performed by the model generation device of FIGS. 1 and 2 for determining the accuracy of a three-dimensional model generated from a two-dimensional image of an object. [Figure 6] FIG. 3 is a simplified block diagram of one embodiment of a method that may be performed by the model generation device of FIGS. 1 and 2 for determining the accuracy of a three-dimensional model generated from a two-dimensional image of an object. [Figure 7] FIG. 3 is a simplified block diagram of one embodiment of a method that may be performed by the model generation device of FIGS. 1 and 2 for determining the accuracy of a three-dimensional model generated from a two-dimensional image of an object. [Figure 8] FIG. 3 is a simplified block diagram of one embodiment of a method that may be performed by the model generation device of FIGS. 1 and 2 for determining the accuracy of a three-dimensional model generated from a two-dimensional image of an object. [Figure 9] 2A-2C are diagrams illustrating a 3D model of a knee joint, an input 2D image, and a spatial orientation of a silhouette that may be generated by the system of FIG. 1. [Figure 10] FIG. 2 illustrates the alignment of a three-dimensional model of a femur and the alignment of this model with a two-dimensional image of the femur that can be processed by the system of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are herein described in detail. It should be understood, however, that it is not the intention of this disclosure to limit the concepts of the present disclosure to the particular forms disclosed, but rather, the intention is to cover all modifications, equivalents, and alternatives falling within the scope of the present disclosure and the appended claims.
[0009] Anatomical reference terms such as anterior, posterior, medial, lateral, superior, inferior, etc. may be used throughout this specification with respect to the orthopedic implants or prostheses and orthopedic surgical devices described herein, as well as with reference to the patient's biological anatomy. Such terms have well-understood meanings both in the study of anatomy and in the field of orthopedic surgery. Use of such anatomical reference terms in the written description and claims is intended to be consistent with their well-understood meanings, unless otherwise specified.
[0010] References herein to "one embodiment," "embodiment," "exemplary example," and the like indicate that the embodiment being described may include a particular element, structure, or feature, but not all embodiments necessarily include that particular element, structure, or feature. Moreover, such phrases do not necessarily refer to the same embodiment. Moreover, when a particular element, structure, or feature is described in connection with an embodiment, it is believed to be within the knowledge of one of ordinary skill in the art to implement such element, structure, or feature in connection with other embodiments, whether or not expressly stated. Additionally, it will be understood that elements in a list of "at least one of A, B, and C" may mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). Moreover, it will be understood that elements in a list of "at least one of A, B, and C" may mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).
[0011] Each disclosed embodiment may be implemented as hardware, firmware, software, or any combination thereof, as the case may be. The disclosed embodiments may also be implemented as instructions held or stored on a transitory or non-transitory machine-readable (e.g., computer-readable) storage medium that can be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., a volatile or non-volatile memory, media disk, or other media device).
[0012] In the figures, certain structural or method elements may be shown in a particular arrangement and / or order. However, it should be recognized that such a particular arrangement and / or order may not be required. Rather, in some embodiments, such elements may be arranged in different forms and / or orders than those shown in the illustrative figures. Additionally, the inclusion of a structural or method element in a particular figure does not imply that such element is required in all embodiments, and may not be included in some embodiments or may be combined with other elements.
[0013] 1, a system 100 for determining the accuracy of a three-dimensional model generated from a two-dimensional image of an object includes a model generation device 110 in communication with an image generation device 112 and a client computing device 114 via a network 116. Unlike other systems, the system 100 determines the accuracy of the three-dimensional model without relying on significant human assistance and complex calibration markers, as described in more detail herein. In an exemplary embodiment, the model generation device 110 in operation can obtain a set of two-dimensional images (e.g., X-ray images) of an anatomical object 140 (e.g., a portion of a human body, such as one or more bones of a human joint) from multiple different perspectives (e.g., orthogonal perspectives). Furthermore, the model generation device 110 in the exemplary embodiment generates a three-dimensional model of the anatomical object 140 from the obtained two-dimensional images. In doing so, the model generation device 110 determines the scale of the model based on known dimensions of a reference object (e.g., a steel ball with a diameter of 25 millimeters) in the image, and further based on candidate values that define possible orientations (e.g., rotations and translations) of the anatomical object 140. Additionally, as described in more detail herein, the model generation device 110 determines the accuracy of the generated model based on a scoring process in which the model generation device 110 compares a two-dimensional silhouette of the three-dimensional model to an edge-detected version of the acquired two-dimensional image. Additionally, the model generation device 110 may iteratively adjust the candidate values (e.g., according to a gradient ascent process, a particle swarm process, a genetic algorithm, machine learning, etc.) to generate further versions of the model until an accuracy score threshold (e.g., an accuracy score equal to or greater than a target accuracy, the highest accuracy score out of a set of a predetermined number of generated accuracy scores, an accuracy score that exhibits a global value, etc.) is obtained.
[0014] In an exemplary embodiment, the model generation device 110 includes an orientation determination logic unit 120, which may be embodied as software or any device or circuitry (e.g., a coprocessor, a reconfigurable circuit, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.) configured to perform the model generation and accuracy scoring operations described above (e.g., to offload those operations from the general-purpose processor of the model generation device 110). Further, in an exemplary embodiment, the orientation determination logic unit 120 includes an accuracy determination logic unit 122, which may be embodied as software or any device or circuitry (e.g., a coprocessor, a reconfigurable circuit, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.) configured to perform operations that constitute the accuracy scoring process (e.g., generating a three-dimensional silhouette of the model, projecting the three-dimensional silhouette onto the imaging surface to generate a two-dimensional silhouette, applying an edge detection operation to the resulting two-dimensional image of the anatomical object 140 to generate an edge image (e.g., an edge-detected version of the two-dimensional image), and comparing edges of the two-dimensional silhouette to edge image edges to determine a score indicative of the accuracy of the model, etc.).
[0015] The image generating device 112 in an exemplary embodiment may be embodied as any device (e.g., a computer, a computing device, etc.) capable of generating a set of two-dimensional images of an object (e.g., an anatomical object 140) from a number of different viewpoints (e.g., angles). In an exemplary embodiment, the image generating device 112 includes one or more radiation sources (e.g., x-ray sources), each of which may be embodied as any device capable of irradiating an object with radiation (e.g., x-ray radiation). The image generating device 112 includes one or more detection devices 132, each of which may be embodied as any device capable of generating an image of the corresponding anatomical object 140 from the interaction of radiation with the anatomical object 140. As described above, the image generating device 112 in an exemplary embodiment generates two-dimensional images of the anatomical object 140 from multiple different viewpoints (e.g., angles) by using multiple fixed radiation sources 130 and detection devices 132 positioned at different orientations relative to the anatomical object 140 and / or by repeatedly generating images of the anatomical object 140 from different viewpoints using one or more movable radiation sources 130 and detection devices 132.
[0016] Additionally, system 100 includes a client computing device 114, which may be embodied as any device (e.g., a computer, computing device, etc.) that can communicate with image generating device 112 and / or model generating device 110 to send requests to one or more of devices 110, 112(s) (e.g., to generate two-dimensional images of anatomical objects, to generate models from these images, to determine an accuracy score for the model, etc.) and receive data from the devices (e.g., the generated two-dimensional images, the generated model, one or more accuracy scores, etc.).
[0017] 2, the exemplary model generation device 110 may be embodied as a computational device (e.g., a computer) including a computational engine (also referred to herein as a "computational engine circuit") 210, an input / output (I / O) subsystem 216, a communication circuit 218, and one or more data storage devices 222. Of course, in other embodiments, the model generation device 110 may include other or additional components, such as those typically found in a computer (e.g., a display, peripherals, etc.). Additionally, in some embodiments, one or more of the exemplary components may be incorporated into or form part of another component. The computational engine 210 may be embodied as any type of device or collection of devices capable of performing various computational functions described below. In some embodiments, the computational engine 210 may be embodied as a single device, such as an integrated circuit, an embedded system, a field programmable gate array (FPGA), a system-on-chip (SOC), or other integrated system or device. In an exemplary embodiment, the computation engine 210 includes or is embodied as a processor 212, a memory 214, an orientation determination logic unit 120, and an accuracy determination logic unit 122, described above with reference to FIG. 1. The processor 212 can be embodied as any type of processor capable of performing the functions described herein. For example, the processor 212 can be embodied as a multi-core processor(s), a digital signal processor, a microcontroller, or other processor or processing / control circuitry. In some embodiments, the processor 212 can be embodied as, include, or be coupled to an FPGA, an application specific integrated circuit (ASIC), a reconfigurable hardware or hardware circuit, or other dedicated hardware to facilitate performance of the functions described herein.
[0018] Similarly, memory 214 may be implemented as any type of volatile (e.g., dynamic random access memory (DRAM) or non-volatile memory or data storage capable of performing the functions described herein. Volatile memory may be a storage medium that requires power to maintain the state of data stored on the medium. In some embodiments, all or a portion of main memory 214 may be integrated with processor 212. During operation, main memory 214 may store various software and data used during operation, such as one or more applications, data manipulated by the application(s) (e.g., 2D images, 3D models, candidate orientations, silhouettes, edge images, accuracy scores, etc.), libraries, and drivers.
[0019] The computational engine 210 is communicatively coupled to other components of the model generation device 110 via an I / O subsystem 216, which may be embodied as circuits and / or components for facilitating input / output operations by the computational engine 210 (e.g., by the processor 212 and / or the main memory 214) and other components of the model generation device 110. For example, the I / O subsystem 216 may be embodied as or include a memory controller hub, an input / output control hub, an integrated sensor hub, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) for facilitating input / output operations, and / or other components and subsystems for facilitating input / output operations. In some embodiments, the I / O subsystem 216 may form part of a system-on-chip (SoC) or may be incorporated into the computational engine 210 along with one or more of the processor 212, the main memory 214, and other components of the model generation device 110.
[0020] The communications circuitry 218 may be embodied as any communications circuitry, device, or collection thereof capable of enabling communications over a network between the model generation device 110 and another computing device (e.g., the image generation device 112, the client computing device 114, etc.). The communications circuitry 218 may be configured to use any one or more communications technologies (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, WiFi, WiMAX, Bluetooth, cellular, etc.) to effect such communications.
[0021] Exemplary communications circuitry 218 includes a network interface controller (NIC) 220. NIC 220 may be embodied as one or more add-in boards, daughter cards, network interface cards, controller chips, chipsets, or other devices that may be used by model generation device 110 to connect with another computing device (e.g., image generation device 112, client computing device 114, etc.). In some embodiments, NIC 220 may be embodied as part of a system-on-chip (SoC) that includes one or more processors, or may be included in a multi-chip package that also includes one or more processors. In some embodiments, NIC 220 may include a local processor (not shown) and / or local memory (not shown), both of which are local to NIC 220. In such embodiments, the local processor of NIC 220 may be capable of performing one or more of the functions of computational engine 210 described herein. Additionally or alternatively, in such embodiments, the local memory of the NIC 220 may be integrated into one or more components of the model generation device 110 at the board level, socket level, chip level, and / or other level.
[0022] The one or more exemplary data storage devices 222 may be embodied as any type of device configured for short-term or long-term storage of data, such as, for example, memory devices and circuits, memory cards, hard disk drives, solid state drives, or other data storage devices. Each data storage device 222 may include a system partition that stores data and firmware code for the data storage device 222. Each data storage device 222 may also include one or more operating system partitions that store data files and executable files for the operating system.
[0023] The image generating device 112 and the client computing device 114 may have similar components as those described in FIG. 2 with reference to the model generating device 110. The description of these components of the model generating device 110 applies equally to the description of the components of the image generating device 112 and the client computing device 114, except that in some embodiments the orientation determination logic unit 120 and the accuracy determination logic unit 122 are not included in a device other than the model generating device 110. Furthermore, it should be understood that any of the model generating device 110, the image generating device 112, and the client computing device 114 may include other components, sub-components, and devices typically found in a computing device that are not described above with reference to the model generating device 110 and are not described herein for the sake of brevity. Furthermore, it should be understood that one or more components of a computing device may be distributed over any distance and are not necessarily housed within the same physical unit.
[0024] 1, the model generation device 110, the image generation device 112, and the client computing device 114 communicate over a network 116, which may illustratively be embodied as any type of data communications network including a global network (e.g., the Internet), one or more wide area networks (WANs), local area networks (LANs), digital subscriber line (DSL) networks, cable networks (e.g., coaxial networks, fiber networks, etc.), cellular networks (e.g., Global System for Mobile Communications (GSM), 3G, Long Term Evolution (LTE), Worldwide Interoperability for Microwave Access (WiMAX), etc.), radio access networks (RANs), or any combination thereof. Additionally, although shown as separate devices in FIG. 1, it should be appreciated that in some embodiments, one or more of the model generation device 110, the image generation device 112, and the client computing device 114 may be combined into a single unit.
[0025] 3, during operation, the model generation device 110 may execute a method 300 for determining accuracy of a three-dimensional model generated from a two-dimensional image of an object (e.g., an anatomical object 140). The method 300 begins at block 302, where the model generation device 110 determines whether to perform an accuracy determination. The model generation device 110 may determine to perform an accuracy determination in response to a determination that the model generation device 110 comprises an orientation determination logic unit 120 and an accuracy determination logic unit 122, in response to a determination that a configuration setting (e.g., in memory 214) indicates to perform an accuracy determination, in response to a request from another device (e.g., client computing device 114) to perform an accuracy determination, and / or based on other factors. In response to a determination to perform an accuracy determination, the method 300 proceeds to block 304, where the model generation device 110 obtains a two-dimensional image of the anatomical object (e.g., from the image generation device 112). In doing so, in an exemplary embodiment, the model generation device 110 obtains a two-dimensional image of a portion of a human body, as shown in block 306. As shown in block 308, the model generation device 110 may obtain a two-dimensional image of one or more bones of the human body. For example, as shown in block 310, the model generation device 110 may obtain a two-dimensional image of a hip joint or a knee joint. In an exemplary embodiment, the model generation device 110 obtains a two-dimensional image by obtaining an X-ray image (e.g., an image generated by passing X-ray radiation (e.g., from the radiation source 130) through the patient's body to a detector (e.g., the detection device 132)), as shown in block 312. In other embodiments, the two-dimensional image may be formed from other types of electromagnetic radiation (e.g., visible light, infrared light, etc.). As shown in block 314, the model generation device 110 in an exemplary embodiment obtains two-dimensional images of the anatomical object 140 from a plurality of different viewpoints (e.g., angles). In doing so, in an exemplary embodiment, the model generation device 110 obtains orthogonal images (e.g., images from viewpoints perpendicular to each other), as shown in block 316.In other embodiments, the viewpoints may be at other angles relative to each other.
[0026] Subsequently, in block 318, the model generation device 110 determines a scale factor (data indicating the amount by which the size of an object represented in the two-dimensional image differs from its actual size) for the acquired two-dimensional image. In doing so, the model generation device 110 determines a ratio between a distance represented in the two-dimensional image (e.g., diameter, length, etc. of the object) and an actual distance (e.g., actual diameter, length, etc. of the object), as shown in block 320. As an example, the model generation device 110 can determine the scale factor from a representation in the two-dimensional image of a reference object having a predetermined (e.g., known) size, as shown in block 322. In an exemplary embodiment, the model generation device 110 can determine a ratio between the size of the reference object seen in the acquired two-dimensional image and the predetermined size of the reference object, as shown in block 324. As shown in block 326, the model generation device 110 can determine the scale factor from a sphere having a predetermined diameter. In an exemplary embodiment, as shown in block 328, the model generation device 110 may determine a scale factor from a metal sphere having a predetermined diameter of 25 millimeters (e.g., by determining the ratio of the diameter of the sphere represented in the two-dimensional image to the known diameter of 25 millimeters). That is, the metal sphere may be physically present with the patient (e.g., adjacent to the patient, attached to the patient, etc.) and imaged by the image generation device 112. The method 300 then proceeds to block 330 of FIG. 4, where the model generation device 110 determines candidate values indicative of the translation and rotation of the anatomical object 140 represented in the acquired two-dimensional image.
[0027] 4, when determining the candidate values, the model generation device 110 determines candidate values for translations of the anatomical object in three dimensions (e.g., along the x-axis, y-axis, and z-axis), as shown in block 332. Similarly as shown in block 332, the model generation device 110 determines candidate values for rotations of the anatomical object 140 in three dimensions (e.g., along the x-axis, y-axis, and z-axis), as shown in block 334. The candidate values can be determined in various ways, depending on the embodiment. For example, as shown in block 336, the model generation device 110 can determine a current candidate value as a function of (e.g., based on) previous candidate values and corresponding accuracy scores of models generated using these values (e.g., from previous iterations of method 300). As shown in block 338, the model generation device 110 can determine the candidate values based on a gradient ascent process (e.g., a linear iterative optimization process for finding a maximum of a function by taking steps proportional to the positive value of the gradient or approximate gradient of the current function) or a gradient descent process (e.g., for finding a minimum of a function). In some embodiments, the model generation device 110 can determine the candidate values based on a genetic algorithm (e.g., a metaheuristic using biologically based operators such as mutation, crossover, and selection that stimulate the process of natural selection to find an optimal solution to a problem), as shown in block 340.
[0028] Additionally or alternatively, the model generation device 110 may determine the candidate values based on a particle swarm process (e.g., a process that optimizes a problem by using a population of candidate solutions (referred to as "particles") and iteratively improving the candidate solutions with respect to a predetermined measure of quality (e.g., an accuracy score) by moving the particles in a search space according to mathematical operations that affect the position and velocity of each particle), for example, as shown in block 342. In some embodiments, the model generation device 110 may determine the current candidate values using a machine learning process (e.g., a process that uses training data to identify patterns that indicate mathematical relationships between input variables and outputs), as shown in block 344. In other embodiments, the model generation device 110 may determine the candidate values based on a scan of all values in the available parameter space (e.g., iteratively trying all possible degrees of rotation about each axis, etc.), as shown in block 346. The method 300 then proceeds to block 348 of FIG. 5, where the model generation device 110 obtains a candidate three-dimensional model of the anatomical object 140 as a function of the acquired two-dimensional image (e.g., from block 304), the scale factor (e.g., from block 318), and the candidate value (e.g., from block 330).
[0029] 5, when acquiring the model, the model generation device 110 applies triangulation (e.g., between the anatomical object 140 and each viewpoint from which the two-dimensional images were generated) to determine the location in three-dimensional space of points along the surface of the anatomical object (e.g., each point has x, y, and z coordinates), as shown in block 350. As shown in block 352, the model generation device 110 can use a data set (e.g., in memory 214 or data storage device 222) of a reference model of the anatomical object to provide data indicative of portions of the anatomical object that are not represented in the acquired two-dimensional image (e.g., a portion of a bone is not shown in the acquired two-dimensional image). In an exemplary embodiment, the model generation device 110 applies scaling to the model based on a scale factor (e.g., sets the size of the model), as shown in block 354. In doing so, in an exemplary embodiment, the model generation device 110 applies the scale factor across three spatial dimensions (e.g., sets the size of the model along the x-axis, y-axis, and z-axis using the same scale factor), as shown in block 356. The model generation device 110 may also apply a translation to the model (e.g., change the position of the model) based on the candidate values (e.g., the candidate translation values determined in block 332), as shown in block 358. In doing so, in an exemplary embodiment, the model generation device 110 applies a translation in three dimensions (e.g., in the x-, y-, and z-dimensions), as shown in block 360. Similarly, the model generation device 110 may apply a rotation based on the candidate values, as shown in block 362. In doing so, the model generation device 110 may apply a rotation across three dimensions (e.g., rotate the object around the x-, y-, and z-axes by amounts defined in the candidate values), as shown in block 364.
[0030] Subsequently, as shown in block 366, the model generation device 110 determines a score indicating the accuracy of the generated three-dimensional model (e.g., the model generated in block 348). In doing so, in an exemplary embodiment, the model generation device 110 compares the two-dimensional silhouette of the three-dimensional model with an edge-detected version of the acquired two-dimensional image (e.g., a version of the two-dimensional image in which edges are represented by pixel values (e.g., non-zero pixel values) and regions not representing edges are represented by different pixel values (e.g., zero pixel values)), as shown in block 368. In doing so, as shown in block 370, the model generation device 110 compares a two-dimensional silhouette generated from a projection of the three-dimensional silhouette of the three-dimensional model (e.g., the model generated in block 348) based on (e.g., projected from) the determined position of the X-ray source (e.g., radiation source 130) used to generate the acquired two-dimensional image. As shown in block 372, the model generation device 110 may sum the values of pixels along edges shared between the two-dimensional silhouette (e.g., from block 370) and the corresponding edge-detected version of the acquired two-dimensional image and define a score as the resulting sum.
[0031] 6, when generating a score indicative of the accuracy of the candidate three-dimensional model, the model generation device 110 may position the acquired two-dimensional image and the candidate three-dimensional model in three-dimensional space, as shown in block 374. In doing so, in an exemplary embodiment, the model generation device 110 positions the acquired two-dimensional image and the candidate three-dimensional model relative to one another based on the candidate values (e.g., the candidate values from block 330) and the determined scale factor (e.g., from block 318), as shown in block 376. Additionally, in an exemplary embodiment, the model generation device 110 determines the position in three dimensions of the X-ray source (e.g., radiation source 130) used to generate the acquired two-dimensional image. In block 380, the model generation device 110 traces (e.g., performs ray tracing) rays from the determined position of each X-ray source to the corresponding two-dimensional image. In doing so, the model generation device 110 identifies, for each ray, polygons (e.g., triangles) on the surface of the candidate three-dimensional model that intersect, as shown in block 382. Additionally, and as shown in block 384, the model generation device 110 identifies, for each ray, intersecting polygons (e.g., from the set of intersecting polygons identified in block 382) that are adjacent to each other and that are arranged in opposing orientations (e.g., pointing in opposite directions). Further, as shown in block 386, the model generation device 110, in an exemplary embodiment, determines, for each ray, one or more common edges between the intersecting, adjacent, opposing polygons (e.g., polygons identified in block 384). The method 300 then proceeds to block 388 of FIG. 7, where the model generation device 110 generates a three-dimensional silhouette (e.g., a curve that defines an outer surface of the candidate three-dimensional model) from the one or more common edges (e.g., edges determined in block 386).
[0032] 7, when generating the three-dimensional silhouettes, the model generation device 110, in an exemplary embodiment, generates the three-dimensional silhouettes as polycurves (e.g., multiple curves connected together), as shown in block 390. As shown in block 392, the model generation device 110 can apply a smoothing operation to one or more of the three-dimensional silhouettes, and as shown in block 394, the model generation device 110 can remove all line segments having a length that meets a predetermined length (e.g., remove all line segments that are shorter than a predetermined length) from one or more of the silhouettes. Additionally, and as shown in block 396, the model generation device 110 generates two-dimensional silhouettes by projecting the three-dimensional silhouettes (e.g., from block 388) from the position of the X-ray source (e.g., from the position of the X-ray source determined in block 378) onto one or more imaging planes (e.g., planes located where the acquired two-dimensional images are located in three-dimensional space in block 374). The model generation device 110 may also remove one or more objects (e.g., the reference object from block 322) from the acquired two-dimensional image by applying one or more masks to the acquired two-dimensional image, as shown in block 398. Furthermore, in an exemplary embodiment, the model generation device 110 may apply an edge detection operation to the acquired two-dimensional image to generate a corresponding edge image (e.g., an edge-detected version of the acquired two-dimensional image), as shown in block 400. In doing so, the model generation device 110 may apply a Canny edge detection algorithm to the acquired two-dimensional image, as shown in block 402 (e.g., apply a Gaussian filter to smooth the image to remove noise, locate intensity gradients in the image, apply non-maximum suppression to eliminate spurious responses to edge detection, apply a dual threshold to determine potential edges, and use hysteresis to track edges). As shown in block 404, the model generation device 110 may additionally apply a distance filter to the edge image.At block 406, in an exemplary embodiment, the model generation device 110 cycles through each edge (e.g., from block 396) present in the two-dimensional silhouette. In doing so, the model generation device 110 locates pixels in the edge image that correspond to the current edge from the silhouette and determines values of these corresponding pixels, as shown in block 408. Additionally, as shown in block 410, the model generation device 110 generates a score based on a comparison of the edge from the silhouette to corresponding pixel values in the corresponding edge image. In doing so, and as shown in block 412, the model generation device 110 may sum the pixel values (e.g., determine a running sum) and designate a score as the sum.
[0033] Illustrative examples of spatial orientations 900 of a candidate three-dimensional model 902 (e.g., the candidate model generated in block 348) of a knee joint (e.g., anatomical object 140), input two-dimensional images 910, 912 (e.g., the two-dimensional images acquired in block 304), and a silhouette 920 (e.g., the silhouette used to determine a score indicative of the accuracy of the candidate model in block 368) are shown in FIG. 9. An illustrative example of a typical alignment 1000 (e.g., spatial orientation) of a candidate three-dimensional model of a femur 1002 to an X-ray image 1010 (e.g., the two-dimensional image acquired from block 304) is shown in FIG. 10. After the model generation device 110 determines a score indicative of the accuracy of the candidate three-dimensional model, the method 300 proceeds to block 414 of FIG. 8, where the model generation device 110 determines a subsequent course of action based on whether a threshold accuracy score has been met (e.g., due to the score determined in block 366).
[0034] 8, in determining whether the accuracy score threshold is met, the model generation device 110 may determine whether the score determined in block 366 is equal to or greater than a predetermined score, whether the score determined in block 366 is the highest score among a set of scores generated in successive iterations of the method 300, and / or based on other factors. Regardless, in response to a determination that the accuracy score threshold is not met, the method 300 returns to block 330 of FIG. 4, where the model generation device 110 determines a subsequent set (e.g., a different set) of candidate values and generates a subsequent three-dimensional model having a different spatial orientation based on the subsequent set of candidate values. Otherwise (e.g., if the accuracy score threshold is met), the method 300 proceeds to block 416, where the model generation device 110 generates output data (e.g., for display, for writing to memory, and / or for transmission to another computing device, such as the client computing device 114) indicating that the candidate three-dimensional model is an accurate representation of the anatomical object 140. In doing so, and as indicated at block 418, the model generation device 110 may generate output data including the candidate values used to generate the candidate three-dimensional models. The output data may also include scale factors used in generating the candidate three-dimensional models.
[0035] As an illustrative example of the method 300, the model generation device 110 may obtain a set of two-dimensional x-ray images of a patient's knee joint from an image generation device. Within the images, a metal ball is present. The model generation device 110 is configured to detect the presence of a metal ball having a known (e.g., to the model generation device 110) diameter of 25 millimeters. In this example, the metal ball is 100 pixels wide within the two-dimensional x-ray images obtained by the model generation device 110. Thus, the model generation device 110 determines that every four pixels represents one millimeter (e.g., a scale factor of four to one). Given that the metal ball is symmetric, the model generation device 110 may fix the scale factor across the x, y, and z dimensions (e.g., four pixels in any direction represents one millimeter in that direction).
[0036] Next, in an illustrative example, the model generation device 110 determines candidate translations and rotations of the bones of the knee joint in the X-ray image. In doing so, the model generation device 110 selects a possible translation of 20 millimeters along the x-axis, y-axis, and z-axis. Additionally, the model generation device 110 selects a possible translation of 20° clockwise along each of the x-axis, y-axis, and z-axis. The model generation device 110 then obtains a candidate three-dimensional model of the bones of the knee joint from the X-ray image using any known 2D / 3D conversion method (e.g., TruMatch® 2D3D) and fills in missing bone details using a reference model of the bones of a human knee joint. In this process, the model generation device 110 applies a translation of 20 millimeters along each of the x-axis, y-axis, and z-axis (e.g., assuming that the patient has moved 20 millimeters in each dimension from the position on which the reference knee joint model is based) and a rotation of 20° along the x-axis, y-axis, and z-axis (e.g., assuming that the X-ray detector generating the X-ray image is rotated 20° clockwise or the patient has rotated one or more bones of the knee joint 20° clockwise along each axis relative to the orientation on which the reference knee joint model is based (e.g., by flexing the knee joint)), and scales the model using a scale factor of 4 pixels per millimeter. The model generation device 110 then determines the accuracy of the resulting model by comparing a 2D silhouette of the model with an edge-detected version of the X-ray image to generate an accuracy score, as described with reference to Figs. 3-7 (e.g., importing the model into a workspace, reconstructing the position of the X-ray source, ray tracing from the position of the X-ray source to generate a 3D silhouette, generating a 2D silhouette from a projection of the 3D silhouette, applying edge detection to the 2D X-ray image from the image generation device 112 to generate an edge-detected version of the X-ray image, and comparing the 2D silhouette to the edge-detected version of the X-ray image). Any differences (e.g., non-overlapping lines) between the 2D silhouette and the corresponding edge-detected version of the X-ray image decrease the accuracy score, whereas similarities (e.g., overlapping lines) increase the accuracy score. For illustrative purposes, the accuracy score may be 7.
[0037] The model generation device 110 then repeats the process of obtaining a candidate model, this time using different translation and rotation values. For example, translation values of x=30, y=30, z=30 and rotation values of x=30, y=30, z=30 are used. The resulting accuracy score is 8. The model generation device then repeats the process with translation values of x=40, y=40, z=40, which reduces the accuracy score to 6. Based on the reduction in accuracy score, the model generation device 110 tests different values for translation and rotation in the range of 20-30 until the model generation device 110 determines that a maximum score of 9 has been reached with translation values of x=28, y=28, z=28 and rotation values of x=24, y=20, z=26. The model generation device 110 then indicates to the user or other device that the model generated using translation values of x=28, y=28, z=28 and rotation values of x=24, y=20, z=26 is the most accurate model of the bone(s) of the knee joint.
[0038] While certain exemplary embodiments have been described in detail in the drawings and foregoing description, it is understood that such illustration and description are merely illustrative in nature and should not be regarded as restrictive, and that merely exemplary embodiments have been shown and described, and that all changes and modifications that come within the spirit of the disclosure are desired to be protected.
[0039] The present disclosure has multiple advantages based on various features of the methods, devices, and systems described herein. It is noted that alternative embodiments of the methods, devices, and systems of the present disclosure do not include all of the features described, but still benefit from at least some of the advantages of such features. Those skilled in the art can easily implement their own methods, devices, and systems that incorporate one or more of the features of the present invention and are within the spirit and scope of the present disclosure as defined in the appended claims.
[0040] [Embodiment] (1) A device comprising: Obtaining a two-dimensional image of a bone of a human joint; obtaining a candidate three-dimensional model of the bone; generating a two-dimensional silhouette of the candidate three-dimensional model; applying an edge detection algorithm to the two-dimensional image to generate a corresponding edge image; comparing the 2D silhouette to the edge image to generate a score indicative of accuracy of the candidate 3D model; 4. A device comprising: (2) The circuit comprises: positioning the acquired two-dimensional image and the candidate three-dimensional model in three-dimensional space; 2. The device of claim 1, further comprising: determining a three-dimensional position of an X-ray source used to generate the acquired two-dimensional image. (3) The device of claim 2, wherein the circuitry is further for tracing a ray from a determined position of an X-ray source utilized to generate one of the acquired two-dimensional images to a corresponding one of the acquired two-dimensional images. (4) The device of embodiment 3, wherein the circuit is further for identifying, for each ray, a polygon on a surface of the candidate three-dimensional model that intersects with the ray. (5) The device of embodiment 4, wherein the polygons identified for each ray are adjacent and arranged in opposing orientations.
[0041] (6) The device of embodiment 5, wherein the circuitry is further for determining one or more common edges between the identified polygons for each ray. (7) The device of embodiment 1, wherein the circuitry is further for generating one or more three-dimensional silhouettes from common edges between polygons on a surface of the candidate three-dimensional model. (8) The device of embodiment 7, wherein generating one or more three-dimensional silhouettes includes generating one or more polycurves. (9) The device of claim 7, wherein the circuitry is further for applying a smoothing operation to the one or more three-dimensional silhouettes. (10) The device of claim 7, wherein the circuit is further for removing the line segment from the one or more three-dimensional silhouettes in response to a determination that the line segment has a length that meets a predetermined length.
[0042] (11) The device of embodiment 7, wherein generating a two-dimensional silhouette of the candidate three-dimensional model includes projecting the one or more three-dimensional silhouettes onto an imaging plane. (12) The device of embodiment 11, wherein projecting the one or more three-dimensional silhouettes onto the imaging surface includes projecting the one or more three-dimensional silhouettes from a position of an X-ray source utilized to generate one of the acquired two-dimensional images. (13) The device of embodiment 1, wherein the circuitry is further for applying a mask to the acquired two-dimensional image to remove objects from the acquired two-dimensional image. (14) The device of embodiment 1, wherein applying an edge detection algorithm includes applying a Canny edge detection algorithm. (15) The device of embodiment 1, wherein the circuitry is further for applying a distance filter to the edge image.
[0043] (16) Comparing the two-dimensional silhouette to the edge image to generate a score indicative of accuracy of the candidate three-dimensional model, locating within the edge image pixel values corresponding to edges of the two-dimensional silhouette; and summing the located pixel values. (17) One or more machine-readable storage media containing a plurality of instructions stored thereon, the plurality of instructions, in response to execution, causing a device to: Obtaining a two-dimensional image of an anatomical object; obtaining a candidate three-dimensional model of the anatomical object; generating a two-dimensional silhouette of the candidate three-dimensional model; applying an edge detection algorithm to the two-dimensional image to generate a corresponding edge image; comparing the 2D silhouette to the edge image to generate a score indicative of accuracy of the candidate 3D model. (18) The instructions are transmitted to the device, positioning the acquired two-dimensional image and the candidate three-dimensional model in three-dimensional space; The one or more machine-readable storage media of embodiment 17, further comprising determining a three-dimensional position of an X-ray source used to generate the acquired two-dimensional image. (19) A method comprising the steps of: acquiring, by the device, a two-dimensional image of an anatomical object; obtaining, by said device, a candidate three-dimensional model of said anatomical object; generating, by said device, a two-dimensional silhouette of said candidate three-dimensional model; applying, by said device, an edge detection algorithm to said two-dimensional image to generate a corresponding edge image; comparing, by the device, the two-dimensional silhouette to the edge image to generate a score indicative of accuracy of the candidate three-dimensional model. (20) The anatomical object is a bone of a human joint, and the method further comprises: positioning, by the device, the acquired two-dimensional image of the bone and the candidate three-dimensional model of the bone in three-dimensional space; 20. The method of claim 19, further comprising determining, by the device, a three-dimensional position of an X-ray source utilized to generate the acquired two-dimensional image of the bone.
Claims
1. A device, comprising: Obtaining a two-dimensional image of a bone of a human joint; obtaining a candidate three-dimensional model of the bone based on the two-dimensional image; generating a two-dimensional silhouette of a candidate three-dimensional model; applying an edge detection algorithm to the two-dimensional image to generate a corresponding edge image; comparing the two-dimensional silhouette to the edge image to generate a score indicative of accuracy of the candidate three-dimensional model; determining whether the score meets a threshold, and if the threshold is not met, generating a subsequent three-dimensional model having a different spatial orientation than the candidate three-dimensional model; 4. A device comprising:
2. The circuit, positioning the acquired two-dimensional image and the candidate three-dimensional model in three-dimensional space; 2. The device of claim 1, further for determining a position in three dimensions of an x-ray source used to generate the acquired two-dimensional image.
3. 3. The device of claim 2, wherein the circuitry is further for tracing rays from a determined position of an x-ray source utilized to generate one of the acquired two-dimensional images to a corresponding acquired two-dimensional image.
4. The device of claim 3 , wherein the circuitry is further for identifying, for each ray, a polygon on a surface of the candidate three-dimensional model that intersects the ray.
5. The device of claim 4 , wherein the polygons identified for each ray are adjacent and disposed in opposing orientations.
6. The device of claim 5 , wherein the circuitry is further for determining one or more common edges between the identified polygons for each ray.
7. 10. The device of claim 1, wherein the circuitry is further for generating one or more three-dimensional silhouettes from common edges between polygons on a surface of the candidate three-dimensional model.
8. The device of claim 7 , wherein generating one or more three-dimensional silhouettes comprises generating one or more polycurves.
9. 8. The device of claim 7, wherein the circuitry is further for applying a smoothing operation to the one or more three-dimensional silhouettes.
10. 8. The device of claim 7, wherein the circuitry is further for removing the line segment from the one or more three-dimensional silhouettes in response to determining that the line segment has a length that meets a predetermined length.
11. The device of claim 7 , wherein generating two-dimensional silhouettes of the candidate three-dimensional models comprises projecting the one or more three-dimensional silhouettes onto an imaging plane.
12. 12. The device of claim 11, wherein projecting the one or more three-dimensional silhouettes onto the imaging plane comprises projecting the one or more three-dimensional silhouettes from a position of an x-ray source utilized to generate one of the acquired two-dimensional images.
13. The device of claim 1 , wherein the circuitry is further for applying a mask to the acquired two-dimensional image to remove objects from the acquired two-dimensional image.
14. The device of claim 1 , wherein applying an edge detection algorithm comprises applying a Canny edge detection algorithm.
15. The device of claim 1 , wherein the circuitry is further for applying a distance filter to the edge image.
16. comparing the two-dimensional silhouette to the edge image to generate a score indicative of accuracy of the candidate three-dimensional model; locating within the edge image pixel values corresponding to edges of the two-dimensional silhouette; and summing the located pixel values.
Citation Information
Patent Citations
Computer-assisted bone lengthening
JP2003530177A
Medical test device having means to perform setting correction
JP2005514148A
Image processing apparatus and its program
JP2006212294A
System and method for image segmentation when generating computer models of joints undergoing arthroplasty.
JP2011519605A
Systems and methods for measuring and assessing spinal instability
JP2017532165A
