Techniques for determining the accuracy of three-dimensional models for use in plastic surgery

Abstract

The present invention is entitled "Techniques for Determining the Accuracy of a Three-Dimensional Model for Use in Orthopedic Surgery." The present disclosure relates to techniques for determining the accuracy of a three-dimensional model, the three-dimensional model comprising a device having circuitry for obtaining a two-dimensional image of an anatomical object (e.g., bones of a human joint), obtaining a candidate three-dimensional model of the anatomical object, and generating a two-dimensional outline of the candidate three-dimensional model. The circuitry further applies an edge detection algorithm to the two-dimensional image to generate a corresponding edge image, and compares the two-dimensional outline to the edge image to generate a score indicating the accuracy of the candidate three-dimensional model.

Description

[0001] Cross-references to Related Patent Applications

[0002] This application is related to U.S. patent application Ser. No. 16 / 586,887, filed on September 27, 2019, and entitled “TECHNOLOGIES FOR DETERMINING THE SPATIAL ORIENTATION OF INPUT IMAGES FOR USE IN AN ORTHOPAEDIC SURGICAL PROCEDURE.” Technical Field

[0003] The present disclosure relates to plastic surgery and, more particularly, to techniques for determining the accuracy of three-dimensional models for use in plastic surgery. Background Art

[0004] Some three-dimensional modeling systems, such as x-ray-based systems, generate a three-dimensional model of an object based on a set of two-dimensional images (e.g., x-ray images) of the object from different viewpoints. To reduce the likelihood of inaccuracies in the resulting model, for example due to incorrect orientation in three-dimensional space, an operator may place calibration markers (e.g., physical objects of known size and / or position) on or near the object being imaged and modeled. However, the placement of such calibration markers is complex and time-consuming. Furthermore, in such systems, the process of verifying the accuracy of the three-dimensional model must be performed by an operator, who visually inspects the model and compares it to the original object to determine whether the model is correctly aligned and sized. Summary of the Invention

[0005] In one aspect, the present disclosure describes a device having circuitry for obtaining a two-dimensional image of an anatomical object (e.g., bones of a human joint), obtaining a candidate three-dimensional model of the anatomical object, and generating a two-dimensional outline of the candidate three-dimensional model. The circuitry further applies an edge detection algorithm to the two-dimensional image to generate a corresponding edge image, and compares the two-dimensional outline to the edge image to generate a score indicating the accuracy of the candidate three-dimensional model.

[0006] In another aspect, the present disclosure describes one or more machine-readable storage media comprising a plurality of instructions stored thereon that, in response to being executed, cause an apparatus to obtain a two-dimensional image of an anatomical object, obtain a candidate three-dimensional model of the anatomical object, and generate a two-dimensional outline of the candidate three-dimensional model. The plurality of instructions further cause the apparatus to apply an edge detection algorithm to the two-dimensional image to generate a corresponding edge image, and compare the two-dimensional outline to the edge image to generate a score indicating an accuracy of the candidate three-dimensional model.

[0007] In yet another aspect, the present disclosure describes a method that includes obtaining, by a device, a two-dimensional image of an anatomical object, obtaining, by the device, a candidate three-dimensional model of the anatomical object, and generating, by the device, a two-dimensional outline of the candidate three-dimensional model. The method also includes applying, by the device, an edge detection algorithm to the two-dimensional image to generate a corresponding edge image, and comparing, by the device, the two-dimensional outline to the edge image to generate a score indicating an accuracy of the candidate three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The concepts described herein are illustrated in the accompanying drawings by way of example and not limitation. For simplicity and clarity of illustration, elements shown in the drawings are not necessarily drawn to scale. Where deemed appropriate, reference labels have been repeated among the figures to indicate corresponding or similar elements. The detailed description refers specifically to the accompanying drawings, in which:

[0009] Figure 1 is a simplified diagram of one embodiment of a system for determining the accuracy of a three-dimensional model generated from a two-dimensional image of an object;

[0010] Figure 2 To be included in Figure 1 A simplified block diagram of one embodiment of a model generating device in a system;

[0011] Figure 3-8 For Figure 1 and Figure 2 A simplified block diagram of one embodiment of a method performed by a model generation device for determining the accuracy of a three-dimensional model generated from a two-dimensional image of an object;

[0012] Figure 9 For Figure 1 a system-generated three-dimensional model of the knee joint, an input two-dimensional image, and a spatial orientation map of the contours; and

[0013] Figure 10 For Figure 1 Figure 3 shows the alignment of a three-dimensional model of a femur and the alignment of the model with a two-dimensional image of the femur. DETAILED DESCRIPTION

[0014] While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described in detail herein. However, it should be understood that there is no intention to limit the concepts of the present disclosure to the specific forms disclosed, but on the contrary, the invention is intended to cover all modifications, equivalents, and alternatives consistent with the present disclosure and the appended claims.

[0015] Throughout the specification, when referring to orthopedic implants or prostheses and surgical instruments described herein and the patient's natural anatomy, terms indicating anatomical references, such as anterior, posterior, medial, lateral, superior, inferior, etc., may be used. These terms have well-known meanings in the fields of anatomy and orthopedics. Unless otherwise specified, these anatomical reference terms used in the written detailed description and claims are intended to be consistent with their well-known meanings.

[0016] References in this specification to "one embodiment," "embodiment," "exemplary embodiment," etc. mean that the embodiment may include a particular feature, structure, or characteristic, but each embodiment may or may not include the particular feature, structure, or characteristic. In addition, these terms do not necessarily refer to the same embodiment. In addition, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it should be understood that it is within the knowledge of a person skilled in the art to implement such a particular feature, structure, or characteristic in conjunction with other embodiments, regardless of whether such description is explicit. In addition, it should be understood that items included in a list in the form of "at least one of A, B, and C" may mean (A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C). Similarly, items listed in the form of "at least one of A, B, or C" may mean (A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C).

[0017] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on a transient or non-transient machine-readable (e.g., computer-readable) storage medium, which instructions may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure (e.g., volatile or non-volatile memory, a media disk, or other media device) for storing or transmitting information in a machine-readable form.

[0018] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or ordering may not be required. On the contrary, in some embodiments, such features may be arranged in a manner and / or ordering different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular drawing does not imply that such features are required in all embodiments, and in some embodiments, such features may not be included or may be combined with other features.

[0019] Now see Figure 1, a system 100 for determining the accuracy of a three-dimensional model generated from two-dimensional images of an object includes a model generation device 110 in communication with an image generation device 112 and a client computing device 114 via a network 116. Unlike other systems, the system 100 determines the accuracy of the three-dimensional model without relying on extensive human assistance and complex calibration markers, as described in more detail herein. In an exemplary embodiment, in operation, the model generation device 110 may obtain a set of two-dimensional images (e.g., x-ray images) of an object, such as an anatomical object 140 (e.g., a part of the human body, such as one or more bones of a human joint), from multiple different viewpoints (e.g., orthogonal viewpoints). Furthermore, in an exemplary embodiment, the model generation device 110 generates a three-dimensional model of the anatomical object 140 from the obtained two-dimensional images. In doing so, the model generation device 110 determines the scale of the model based on the known dimensions of a reference object (e.g., a steel ball with a diameter of 25 mm) in the image and based on candidate values ​​that define possible orientations (e.g., rotations and translations) of the anatomical object 140. In addition, the model generation device 110 determines the accuracy of the generated model based on a scoring process, wherein the model generation device 110 compares the two-dimensional outline of the three-dimensional model with an edge-detected version of the obtained two-dimensional image, as described in more detail herein. In addition, the model generation device 110 can iteratively adjust the candidate values ​​(e.g., according to a gradient ascent process, a particle swarm process, a genetic algorithm, machine learning, etc.) to generate additional versions of the model until a threshold accuracy score is obtained (e.g., an accuracy score greater than or equal to a target accuracy, a highest accuracy score among a set of predefined number of generated accuracy scores, an accuracy score indicating a local maximum, etc.).

[0020] In an exemplary embodiment, the model generation device 110 includes an orientation determination logic unit 120, which may be embodied as software or any device or circuit (e.g., a coprocessor, a reconfigurable circuit, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.) configured to perform the above-described model generation and accuracy scoring operations (e.g., offloading those operations from a general-purpose processor of the model generation device 110). In addition, in an exemplary embodiment, the orientation determination logic unit 120 includes an accuracy determination logic unit 122, which can be embodied as software or any device or circuit (e.g., a coprocessor, a reconfigurable circuit, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.) that is configured to perform operations that constitute an accuracy scoring process (e.g., generating a three-dimensional contour of the model, projecting the three-dimensional contour onto an imaging surface to generate a two-dimensional contour, applying an edge detection operation to the obtained two-dimensional image of the anatomical object 140 to generate an edge image (e.g., an edge-detected version of the two-dimensional image), and comparing edges in the two-dimensional contour to edges in the edge image to determine a score indicating the accuracy of the model).

[0021] In an exemplary embodiment, the image generation device 112 may be embodied as any device (e.g., a computer, a computing device, etc.) capable of generating a set of two-dimensional images of an object (e.g., an anatomical object 140) from a plurality of different viewpoints (e.g., angles). In an exemplary embodiment, the image generation device 112 includes one or more radiation sources (e.g., x-ray sources), each of which may be embodied as any device capable of directing radiation (e.g., x-ray radiation) toward the object. The image generation device 112 also includes one or more detector devices 132, each of which may be embodied as any device capable of generating a corresponding image of the anatomical object 140 from the interaction of radiation with the anatomical object 140. As described above, in an exemplary embodiment, the image generating device 112 generates two-dimensional images of the anatomical object 140 from multiple different viewpoints (e.g., angles) by utilizing multiple fixed radiation sources 130 and detector devices 132 arranged in different orientations relative to the anatomical object 140, and / or by iteratively generating images of the anatomical object 140 from different viewpoints by using one or more movable radiation sources 130 and detector devices 132.

[0022] In addition, the system 100 may include a client computing device 114, which may be embodied as being capable of communicating with the image generating device 112 and / or the model generating device 110 to send requests to one or more of the devices 110, 112 (e.g., to generate a two-dimensional image of an anatomical object, to generate a model from the image, to determine an accuracy score of the model, etc.) and receive data from one or more of the devices 110, 112 (e.g., the generated two-dimensional image, the generated model, one or more accuracy scores, etc.).

[0023] Now see Figure 2 , the exemplary model generation device 110 may be embodied as a computing device (e.g., a computer) that includes a computing engine (also referred to herein as a "computing engine circuit") 210, an input / output (I / O) subsystem 216, a communication circuit 218, and one or more data storage devices 222. Of course, in other embodiments, the model generation device 110 may include other components or additional components, such as those commonly found in computers (e.g., a display, peripheral devices, etc.). In addition, in some embodiments, one or more of the exemplary components may be combined in another component or otherwise form part of another component. The computing engine 210 may be embodied as any type of device or collection of devices capable of performing the various computing functions described below. In some embodiments, the computing engine 210 may be embodied as a single device, such as an integrated circuit, an embedded system, a field programmable gate array (FPGA), a system on a chip (SOC), or other integrated system or device. In an exemplary embodiment, the computing engine 210 includes or is embodied as a processor 212, a memory 214, an orientation determination logic unit 120, and an accuracy determination logic unit 122, as described above with reference to FIG. Figure 1 The processor 212 may be embodied as any type of processor capable of performing the functions described herein. For example, the processor 212 may be embodied as one or more multi-core processors, microcontrollers, or other processors or processing / control circuits. In some embodiments, the processor 212 may be embodied as, include, or be coupled to an FPGA, an application specific integrated circuit (ASIC), reconfigurable hardware or hardware circuitry, or other specialized hardware to facilitate performing the functions described herein.

[0024] The main memory 214 may be embodied as any type of volatile memory (e.g., dynamic random access memory (DRAM), etc.) or non-volatile memory or data storage device capable of performing the functions described herein. Volatile memory may be a storage medium that requires power to maintain the state of the data stored by the medium. In some embodiments, all or a portion of the main memory 214 may be integrated into the processor 212. In operation, the main memory 214 may store various software and data used during operation, such as one or more applications, data operated on by one or more applications (e.g., two-dimensional images, three-dimensional models, candidate values ​​for orientation, contours, edge images, accuracy scores, etc.), libraries, and drivers.

[0025] The computing engine 210 is communicatively coupled to the other components of the model generation device 110 via an I / O subsystem 216, which may be embodied as circuitry and / or components to facilitate input / output operations with the computing engine 210 (e.g., with the processor 212 and / or main memory 214) and the other components of the model generation device 110. For example, the I / O subsystem 216 may be embodied as or otherwise include a memory controller hub, an input / output control hub, an integrated sensor hub, a firmware device, communication links (e.g., point-to-point links, bus links, wires, cables, optical guides, printed circuit board traces, etc.), and / or other components and subsystems that facilitate input / output operations. In some embodiments, the I / O subsystem 216 may form part of a system on a chip (SoC) and be incorporated into the computing engine 210 along with one or more of the processor 212, main memory 214, and the other components of the model generation device 110.

[0026] The communication circuitry 218 may be embodied as any communication circuitry, device, or collection thereof capable of enabling communication between the model generation device 110 and another computing device (e.g., the image generation device 112, the client computing device 114, etc.) over a network. The communication circuitry 218 may be configured to utilize any one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., Ethernet, WiMAX, Cellular, etc.) to achieve such communication.

[0027] The exemplary communication circuitry 218 includes a network interface controller (NIC) 220. NIC 220 may be embodied as one or more onboard add-ins, daughter cards, network interface cards, controller chips, chipsets, or other devices that can be used by model generation device 110 to connect to another computing device (e.g., image generation device 112, client computing device 114, etc.). In some embodiments, NIC 220 may be embodied as part of a system-on-chip (SoC) that includes one or more processors, or included on a multi-chip package that also includes one or more processors. In some embodiments, NIC 220 may include a local processor (not shown) and / or local memory (not shown), both local to NIC 220. In such embodiments, the local processor of NIC 220 may be capable of performing one or more of the functions of compute engine 210 described herein. Additionally or alternatively, in such embodiments, the local memory of NIC 220 may be integrated into one or more components of model generation device 110 at the board level, socket level, chip level, and / or other levels.

[0028] One or more exemplary data storage devices 222 may be embodied as any type of device configured for short-term or long-term storage of data, such as, for example, memory devices and circuits, memory cards, hard drives, solid-state drives, or other data storage devices. Each data storage device 222 may include a system partition that stores data and firmware code for the data storage device 222. Each data storage device 222 may also include one or more operating system partitions that store data files and executable files for the operating system.

[0029] The image generation device 112 and the client computing device 114 may have a connection to the reference model generation device 110. Figure 2 10. The description of those components of model generation device 110 applies equally to the description of the components of image generation device 112 and client computing device 114, except that, in some embodiments, orientation determination logic 120 and accuracy determination logic 122 are not included in devices other than model generation device 110. Furthermore, it should be understood that any of model generation device 110, image generation device 112, and client computing device 114 may include other components, subcomponents, and devices typically present in computing devices, which components, subcomponents, and devices are not discussed above with reference to model generation device 110 and, for the sake of clarity of this specification, are not discussed herein. Furthermore, it should be understood that one or more components of a computing device may be distributed over any distance and are not necessarily housed in the same physical unit.

[0030] Re-reference Figure 1, the model generation device 110, the image generation device 112, and the client computing device 114 illustratively communicate via a network 116, which can be embodied as any type of data communication network, including a global network (e.g., the Internet), one or more wide area networks (WANs), local area networks (LANs), digital subscriber lines (DSL) networks, cable networks (e.g., coaxial networks, fiber optic networks, etc.), cellular networks (e.g., Global System for Mobile Communications (GSM), 3G, Long Term Evolution (LTE), Worldwide Interoperability for Microwave Access (WiMAX), etc.), radio access networks (RANs), or any combination thereof. In addition, although Figure 1 1 as separate devices, it should be understood that in some embodiments, one or more of the model generation device 110, the image generation device 112, and the client computing device 114 may be combined into a single unit.

[0031] Now see Figure 3In operation, the model generation device 110 may execute a method 300 for determining the accuracy of a three-dimensional model generated from a two-dimensional image of an object (e.g., the anatomical object 140). The method 300 begins at block 302, where the model generation device 110 determines whether to enable accuracy determination. The model generation device 110 may determine to enable accuracy determination in response to determining that the model generation device 110 is equipped with the orientation determination logic 120 and the accuracy determination logic 122, in response to determining that a configuration setting (e.g., in the memory 214) indicates that accuracy determination is enabled, in response to a request from another device (e.g., the client computing device 114) to enable accuracy determination, and / or based on other factors. In response to determining that accuracy determination is enabled, the method 300 proceeds to block 304, where the model generation device 110 obtains a two-dimensional image of the anatomical object (e.g., from the image generation device 112). In doing so, in the exemplary embodiment, the model generation device 110 obtains a two-dimensional image of a portion of a human body, as indicated in block 306. As indicated in block 308, the model generation device 110 may obtain a two-dimensional image of one or more bones of the human body. For example, and as indicated in block 310, the model generation device 110 may obtain a two-dimensional image of a hip joint or a knee joint. In an exemplary embodiment, when obtaining the two-dimensional image, the model generation device 110 obtains an x-ray image (e.g., an image generated by passing x-ray radiation (e.g., from the radiation source 130) through the patient's body onto a detector (e.g., the detector device 132)), as indicated in block 312. In other embodiments, the two-dimensional image may be formed from other types of electromagnetic radiation (e.g., visible light, infrared light, etc.). In an exemplary embodiment, the model generation device 110 obtains two-dimensional images of the anatomical object 140 from multiple different viewpoints (e.g., angles), as indicated in block 314. In doing so, in the exemplary embodiment, the model generation device 110 obtains orthogonal images (e.g., images from viewpoints that are perpendicular to each other), as indicated in block 316. In other embodiments, the viewpoints may be at other angles relative to each other.

[0032] Subsequently, in block 318, the model generation device 110 determines a scale factor (e.g., data indicating the amount by which the dimensions of the objects represented in the two-dimensional image differ from their actual dimensions) for the obtained two-dimensional image. In doing so, and as indicated in block 320, the model generation device 110 determines a ratio of a distance represented in the two-dimensional image (e.g., the diameter, length, etc. of the object) to an actual distance (e.g., the actual diameter, length, etc. of the object). For example, and as indicated in block 322, the model generation device 110 may determine the scale factor based on a representation of a reference object having a predefined (e.g., known) dimension in the two-dimensional image. In an exemplary embodiment, the model generation device 110 may determine a ratio of the dimensions of the reference object as it appears in the obtained two-dimensional image to the predefined dimensions of the reference object, as indicated in block 324. As indicated in block 326, the model generation device 110 may determine the scale factor based on a sphere having a predefined diameter. In an exemplary embodiment, and as indicated in block 328, the model generation device 110 may determine a scaling factor based on a metal sphere having a predefined diameter of 25 millimeters (e.g., by determining a ratio of the diameter of the sphere represented in the two-dimensional image to the known diameter of 25 millimeters). That is, the metal sphere may be physically present with the patient (e.g., next to the patient, attached to the patient, etc.) and imaged by the image generation device 112. Subsequently, the method 300 proceeds to Figure 4 The process proceeds to block 330 where the model generation device 110 determines candidate values ​​indicative of the translation and rotation of the anatomical object 140 represented in the obtained two-dimensional image.

[0033] Now see Figure 4When determining candidate values, the model generation device 110 determines candidate values ​​for the translation of the anatomical object in three dimensions (e.g., along the x-axis, y-axis, and z-axis), as indicated in block 332. Similarly, and as indicated in block 332, in an exemplary embodiment, the model generation device 110 determines candidate values ​​for the rotation of the anatomical object 140 in three dimensions (e.g., along the x-axis, y-axis, and z-axis), as indicated in block 334. Candidate values ​​can be determined in a variety of ways, depending on the embodiment. For example, and as indicated in block 336, the model generation device 110 can determine the current candidate value based on (e.g., based on) previous candidate values ​​and corresponding accuracy scores of models generated using those values ​​(e.g., from previous iterations of method 300). As indicated in block 338, the model generation device 110 can determine the candidate value based on a gradient ascent process (e.g., a first-order iterative optimization process for finding the maximum value of a function by taking steps proportional to the positive value of the gradient or approximate gradient of the function at the current point) or a gradient descent process (e.g., finding the minimum value of a function). In some embodiments, the model generation device 110 may determine the candidate values ​​based on a genetic algorithm (e.g., a metaheuristic algorithm that utilizes biologically based operators such as mutation, crossover, and selection to simulate the process of natural selection to find the best solution to a problem), as indicated in box 340.

[0034] Additionally or alternatively, the model generation device 110 may be based on a particle swarm process (e.g., a process that optimizes a problem by iteratively improving candidate solutions related to a given quality measure (e.g., an accuracy score) by utilizing a set of candidate solutions (called "particles") and moving the particles around in a search space according to mathematical operations that affect the position and velocity of each particle), as indicated in block 342. In some embodiments, the model generation device 110 may utilize a machine learning process (e.g., a process that utilizes training data to identify patterns that indicate mathematical relationships between input variables and outputs) to determine the current candidate value, as indicated in block 344. In other embodiments, the model generation device 110 may determine the candidate value based on a scan of every value in the available parameter space (e.g., iteratively trying every possible rotation angle about each axis, etc.), as indicated in block 346. Subsequently, the method 300 proceeds to Figure 5 Box 348, in which the model generation device 110 obtains (e.g., generates) a candidate three-dimensional model of the anatomical object 140 based on the obtained two-dimensional image (e.g., from box 304), the scale coefficient (e.g., from box 318) and the candidate value (e.g., from box 330).

[0035] Now see Figure 5Upon obtaining the model, and as indicated in block 350, the model generation device 110 applies triangulation (e.g., between the anatomical object 140 and the viewpoint from which the two-dimensional image is generated) to determine the locations of points along the surface of the anatomical object in three-dimensional space (e.g., each point having x, y, and z coordinates). As indicated in block 352, the model generation device 110 may utilize a data set of a reference model of the anatomical object (e.g., in memory 214 or in data storage device 222) to provide data indicating portions of the anatomical object that are not represented in the obtained two-dimensional image (e.g., a portion of a bone is not shown in the obtained two-dimensional image). In an exemplary embodiment, the model generation device 110 applies scaling to the model (e.g., sets the size of the model) based on a scaling factor, as indicated in block 354. In doing so, in an exemplary embodiment, the model generation device 110 applies the scaling factor in three spatial dimensions (e.g., using the same scaling factor to set the size of the model along the x-axis, y-axis, and z-axis), as indicated in block 356. The model generation device 110 may also apply a translation to the model (e.g., change the position of the model) based on the candidate value (e.g., the candidate value for the translation determined in block 332), as indicated in block 358. In doing so, in the exemplary embodiment, the model generation device 110 applies the translation in three dimensions (e.g., in the x-, y-, and z-dimensions), as indicated in block 360. Similarly, and as indicated in block 362, the model generation device 110 may apply a rotation based on the candidate value. In doing so, and as indicated in block 364, the model generation device 110 may apply the rotation in three dimensions (e.g., rotating the object around the x-, y-, and z-axes by the amounts defined in the candidate value).

[0036] Subsequently, and as indicated in block 366, the model generation device 110 determines a score indicating the accuracy of the generated three-dimensional model (e.g., the model generated in block 348). In doing so, in the exemplary embodiment, the model generation device 110 compares the two-dimensional contour of the three-dimensional model with an edge-detected version of the obtained two-dimensional image (e.g., a version of the two-dimensional image in which edges are indicated by pixel values ​​(e.g., having non-zero pixel values) and areas not representing edges are represented by different pixel values ​​(e.g., zero pixel values)), as indicated in block 368. In doing so, and as indicated in block 370, the model generation device 110 compares the two-dimensional contour generated by the protrusion of the three-dimensional contour of the three-dimensional model (e.g., the model generated in block 348) based on the determined position of the x-ray source (e.g., radiation source 130) used to generate the obtained two-dimensional image (e.g., projected from the determined position). As indicated in block 372 , the model generation device 110 may sum the pixel values ​​along the edges shared between the two-dimensional contour (eg, from block 370 ) and the corresponding edge-detected version of the obtained two-dimensional image, defining a score as the resulting sum.

[0037] Now see Figure 6 When generating a score indicating the accuracy of the candidate three-dimensional model, the model generation device 110 may position the obtained two-dimensional image and the candidate three-dimensional model in three-dimensional space, as indicated in block 374. In doing so, and as indicated in block 376, in the exemplary embodiment, the model generation device 110 positions the obtained two-dimensional image and the candidate three-dimensional model relative to each other based on the candidate value (e.g., the candidate value from block 330) and the determined scale factor (e.g., from block 318). Additionally, in block 378, in the exemplary embodiment, the model generation device 110 determines the position of the x-ray source (e.g., radiation source 130) used to generate the obtained two-dimensional image in three dimensions. In block 380, the model generation device 110 traces (e.g., performs ray tracing) a ray from the determined position of each x-ray source to the corresponding two-dimensional image. In doing so, the model generation device 110 identifies, for each ray, an intersecting polygon (e.g., triangle) on the surface of the candidate three-dimensional model, as indicated in block 382. Additionally, and as indicated in block 384, the model generation device 110 identifies, for each ray, intersecting polygons (e.g., from the set of intersecting polygons identified in block 382) that are adjacent to one another and arranged in opposite orientations (e.g., pointing in opposite directions). Furthermore, and as indicated in block 386, in an exemplary embodiment, the model generation device 110 determines, for each ray, one or more common edges between the intersecting, adjacent, opposing polygons (e.g., the polygons identified in block 384). Subsequently, the method 300 proceeds to Figure 7 The process proceeds to block 388 where the model generator 110 generates a three-dimensional contour (eg, a curve defining an outer surface of the candidate three-dimensional model) from one or more common edges (eg, the edges determined in block 386).

[0038] Now see Figure 7When generating the three-dimensional contour, in the exemplary embodiment, the model generation device 110 generates the three-dimensional contour as multiple curves (e.g., multiple curves joined together), as indicated in block 390. As indicated in block 392, the model generation device 110 may apply a smoothing operation to one or more of the three-dimensional contours, and as indicated in block 394, the model generation device 110 may remove any segments having a length that satisfies a predefined length from one or more of the contours (e.g., remove any segments shorter than the predefined length). Additionally, and as indicated in block 396, the model generation device 110 generates a two-dimensional contour by projecting the three-dimensional contour (e.g., from block 388) from the position of the x-ray source (from the position of the x-ray source determined in block 378) onto one or more imaging surfaces (e.g., a plane located at the position in the three-dimensional space where the acquired two-dimensional image is located in block 374). The model generation device 110 may also apply one or more masks to the acquired two-dimensional image to remove one or more objects (e.g., the reference object from block 322) from the acquired two-dimensional image, as indicated in block 398. Furthermore, in the exemplary embodiment, as indicated in block 400, the model generation device 110 applies an edge detection operation to the obtained two-dimensional image to generate a corresponding edge image (e.g., an edge-detected version of the obtained two-dimensional image). In doing so, and as indicated in block 402, the model generation device 110 applies a Canny edge detection algorithm to the obtained two-dimensional image (e.g., applying a Gaussian filter to smooth the image and remove noise, locating intensity gradients in the image, applying non-maximum suppression to eliminate spurious responses to edge detection, applying a double threshold to determine potential edges, and tracking edges with hysteresis). As indicated in block 404, the model generation device 110 may additionally apply a distance filter to the edge image. In block 406, in the exemplary embodiment, the model generation device 110 loops over all edges present in the two-dimensional contour (e.g., from block 396). In doing so, the model generation device 110 locates pixels in the edge image corresponding to the current edge from the contour and determines the values ​​of those corresponding pixels, as indicated in block 408. Furthermore, and as indicated in block 410, the model generation device 110 generates a score based on a comparison of the edges from the contour with corresponding pixel values ​​in the corresponding edge image. In doing so, and as indicated in block 412, the model generation device 110 may sum the pixel values ​​(e.g., determine a running total) and assign the score as the sum.

[0039] An illustrative example of a spatial orientation 900 of a candidate three-dimensional model 902 (e.g., the candidate model generated in block 348), input two-dimensional images 910, 912 (e.g., the two-dimensional images obtained in block 304), and a contour 920 (e.g., the contour used in block 368 to determine a score indicating the accuracy of the candidate model) of a knee joint (e.g., the anatomical object 140) is shown in FIG. Figure 9 An illustrative example of a prototype alignment 1000 (eg, spatial orientation) of a candidate three-dimensional model of a femur 1002 with an x-ray image 1010 (eg, a two-dimensional image obtained from block 304) is shown in FIG. Figure 10 After the model generation device 110 has determined the score indicating the accuracy of the candidate three-dimensional model, the method 300 proceeds to Figure 8

[0066] The process proceeds to block 414, where the model generation device 110 determines a subsequent course of action based on whether the threshold accuracy score has been met (eg, by the score determined in block 366).

[0040] Now see Figure 8 In determining whether the threshold accuracy score has been met, the model generation device 110 may determine whether the score determined in block 366 is equal to or greater than a predefined score, whether the score determined in block 366 is the highest score in a set of scores generated in a series of iterations of the method 300, and / or based on other factors. In any case, in response to determining that the threshold accuracy score has not been met, the method 300 loops back to Figure 4 330, in which the model generation device 110 determines a subsequent candidate value group (e.g., a different group) and generates a subsequent three-dimensional model with a different spatial orientation based on the subsequent candidate value group. Otherwise (e.g., if the threshold accuracy score has been met), the method 300 proceeds to block 416, in which the model generation device 110 generates output data indicating that the candidate three-dimensional model is an accurate representation of the anatomical object 140 (e.g., to be displayed, written to a memory, and / or sent to another computing device, such as the client computing device 114). In doing so, and as indicated in block 418, the model generation device 110 may generate output data including candidate values ​​used to generate the candidate three-dimensional model. The output data may also include a scaling factor used to generate the candidate three-dimensional model.

[0041] As an illustrative example of method 300, the model generator 110 may obtain a set of two-dimensional X-ray images of a patient's knee joint from an image generator. In the images, there is a metal sphere. The model generator 110 is configured to detect the presence of the metal sphere, which has a known (e.g., to the model generator 110) diameter of 25 mm. In this example, the metal sphere is 100 pixels wide in the two-dimensional X-ray image obtained by the model generator 110. Therefore, the model generator 110 determines that every four pixels represents 1 mm (e.g., a scaling factor of 4 to 1). Given that the metal sphere is symmetrical, the model generator 110 may fix the scaling factor in the x, y, and z dimensions (e.g., 4 pixels in any direction represents 1 mm in that direction).

[0042] Next, in the illustrative example, the model generation device 110 determines candidate values ​​for the translation and rotation of the bones of the knee joint in the X-ray image. In doing so, the model generation device 110 selects a possible translation of twenty millimeters along the x-axis, the y-axis, and the z-axis. In addition, the model generation device 110 selects a possible translation of 20 degrees clockwise along each of the x-axis, the y-axis, and the z-axis. The model generation device 110 then uses any known 2D / 3D conversion method (e.g., 2D3D) obtains a candidate three-dimensional model of the bones of the knee joint from the X-ray image, thereby filling in the missing bone details using a reference model of the bones of the human knee joint. In this process, the model generation device 110 applies a translation of twenty millimeters along each of the x-axis, y-axis, and z-axis (for example, assuming that the patient moves twenty millimeters in each dimension from the position on which the reference knee joint model is based), and rotates twenty degrees clockwise along the x-axis, y-axis, and z-axis (for example, assuming that the X-ray detector that generates the X-ray image is rotated twenty degrees clockwise, or the patient rotates one or more bones of the knee joint clockwise (for example, due to flexion of the knee joint) twenty degrees along each axis relative to the orientation on which the reference knee joint model is based), and draws the model to scale using a scaling factor of 4 pixels per millimeter. The model generation device 110 then determines the accuracy of the resulting model by comparing the two-dimensional outline of the model with an edge-detected version of the X-ray image to generate an accuracy score, such as the reference Figures 3 to 7 The method of claim 110 includes the steps of: importing a model into a workspace, reconstructing a position of an X-ray source, performing ray tracing from the position of the X-ray source to generate a three-dimensional contour, generating a two-dimensional contour from a projection of the three-dimensional contour, applying edge detection to the two-dimensional X-ray image from the image generating device 112 to generate an edge-detected version of the X-ray image, and comparing the two-dimensional contour to the edge-detected version of the X-ray image. Any differences between the two-dimensional contour and the corresponding edge-detected version of the X-ray image (e.g., non-overlapping lines) reduce the accuracy score, while similarities (e.g., overlapping lines) increase the accuracy score. For purposes of example, the accuracy score may be 7.

[0043] The model generator 110 then repeats the process of obtaining candidate models, this time using different translation and rotation values. For example, a translation value of x=30, y=30, z=30 and a rotation value of x=30, y=30, z=30 are used. The resulting accuracy score is 8. The model generator then repeats the process with a translation value of x=40, y=40, z=40, and the accuracy score drops to 6. Based on the drop in accuracy score, the model generator 110 tests various values ​​for translation and rotation within the range of 20 to 30 until the model generator 110 determines that the highest score of 9 is achieved with a translation value of x=28, y=28, z=28 and a rotation value of x=24, y=20, z=26. The model generation device 110 then indicates to the user or other device that the model generated using the translation values ​​of x=28, y=28, z=28 and the rotation values ​​of x=24, y=20, z=26 is the most accurate model of one or more bones in the knee joint.

[0044] While certain exemplary embodiments have been described in detail in the drawings and foregoing description, such illustration and description should be considered illustrative rather than restrictive in nature, and it should be understood that only exemplary embodiments have been shown and described and that all changes and modifications that come within the spirit of this disclosure are intended to be protected.

[0045] The various features of the methods, devices, and systems described herein provide the present disclosure with a number of advantages. It should be noted that alternative embodiments of the methods, devices, and systems of the present disclosure may not include all of the features described, but may still benefit from at least some of the advantages of such features. For the above-described methods, devices, and systems, one of ordinary skill in the art can readily devise their own implementations that incorporate one or more of the features of the present invention and fall within the spirit and scope of the present disclosure as defined by the appended claims.

Claims

1. An apparatus for determining the accuracy of a three-dimensional model for use in plastic surgery, comprising: A circuit for: Obtaining two-dimensional images of bones in human joints; obtaining a candidate three-dimensional model of the bone from the two-dimensional image; generating one or more three-dimensional contours from common edges between polygons on the surface of the candidate three-dimensional model; removing the line segment from the one or more three-dimensional contours in response to determining that the line segment has a length that satisfies a predefined length; generating a two-dimensional contour of the candidate three-dimensional model from the one or more three-dimensional contours; applying an edge detection algorithm to the two-dimensional image to generate a corresponding edge image; as well as The two-dimensional contour is compared to the edge image to generate a score indicative of the accuracy of the candidate three-dimensional model.

2. The apparatus of claim 1 , wherein the circuit is further configured to: Positioning the obtained two-dimensional image and the candidate three-dimensional model in a three-dimensional space; and The position in three dimensions of an x-ray source used to generate the obtained two-dimensional image is determined.

3. The apparatus of claim 2, wherein the circuit is further configured to trace a ray from a determined position of an x-ray source used to generate one of the obtained two-dimensional images to the corresponding obtained two-dimensional image. 4 . The apparatus of claim 3 , wherein the circuit is further configured to identify, for each ray, the polygon on the surface of the candidate three-dimensional model that intersects the ray. The apparatus of claim 4 , wherein the polygons identified for each ray are adjacent and arranged in relative orientations. 6 . The apparatus of claim 5 , wherein the circuitry is further configured to determine one or more common edges between the polygons identified for each ray.

7. The apparatus of claim 1, wherein generating one or more three-dimensional contours comprises generating one or more polycurves.

8. The apparatus of claim 1, wherein the circuitry is further configured to apply a smoothing operation to the one or more three-dimensional contours. 9 . The apparatus of claim 1 , wherein generating two-dimensional contours of the candidate three-dimensional models comprises projecting the one or more three-dimensional contours onto an imaging surface.

10. The apparatus of claim 9, wherein projecting the one or more three-dimensional contours onto the imaging surface comprises projecting the one or more three-dimensional contours from a position of an x-ray source used to generate one of the obtained two-dimensional images.

11. The apparatus of claim 1, wherein the circuitry further applies a mask to the obtained two-dimensional image to remove an object from the obtained two-dimensional image.

12. The apparatus of claim 1, wherein applying an edge detection algorithm comprises applying a Canny edge detection algorithm.

13. The apparatus of claim 1, wherein the circuit further applies a distance filter to the edge image.

14. The apparatus of claim 1 , wherein comparing the two-dimensional contour to the edge image to generate a score indicating an accuracy of the candidate three-dimensional model comprises: locating pixel values ​​corresponding to edges in the two-dimensional contour in the edge image; and The located pixel values ​​are summed.

Images