3D model generation and surgical planning based thereon

Abstract

Surgical planning based on a three-dimensional (3D) model includes obtaining a 3D model of an anatomical region, wherein the anatomical region includes the patient's anatomical features; identifying at least one deformity, wherein the at least one deformity is identified with respect to a target anatomical value of the patient's anatomical features, wherein the target anatomical value includes a desired range into which the anatomical measurement should fall; and using the anatomical measurement, determining at least one planar correction to be made to at least one anatomical structure of the patient. Generating a 3D model using two-dimensional (2D) imaging data may include performing anatomical context processing, which includes annotating the model, contours, edges, or surfaces of the anatomical structure, as well as performing 2D-to-3D reconstruction to optimize the orientation, placement, and scale of the 3D digital volume modeling the anatomical structure, resulting in a 3D anatomical representation of the anatomical region as a 3D model.

Description

[Background technology]

[0001] Software is increasingly being used in diagnosing patients' medical conditions and in planning surgical procedures to address those conditions. The foundation of this work is the accurate reflection and understanding of the patient's anatomical characteristics, which are often collected from patient imaging data. [Overview of the Initiative]

[0002] The shortcomings of conventional techniques are overcome, and further advantages are provided. As an example, a method is provided which includes obtaining a three-dimensional (3D) model of a patient's anatomical region, the anatomical region including the patient's anatomical features. The method further includes identifying at least one deformation of the patient's anatomical features, the at least one deformation being identified with respect to a target anatomical value of the patient's anatomical features, the target anatomical value including a desired range into which the anatomical measurement should fall, and the at least one deformation including at least one of rotational deformation and angular deformation. The method further includes using the anatomical measurements to determine at least one plane correction to be made to at least one anatomical structure of the patient.

[0003] As another example, a method is provided which includes acquiring two-dimensional (2D) imaging data of an anatomical region of a patient, where the anatomical region includes the patient's anatomical features. The method further includes using the 2D imaging data to generate a three-dimensional (3D) model of the patient's anatomical region, where the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features. The method further includes identifying one or more deformities of the patient's anatomical features, where the one or more deformities are shown in the 3D model and, based on the 3D model, are identified in relation to a target anatomical value of the patient's anatomical feature. Identification includes acquiring anatomical measurements based on the patient's anatomical landmarks, such as those shown in the 3D model; comparing the anatomical measurements to a target anatomical value; and determining one or more deformities based on the comparison. The method also includes determining at least one correction to be made to at least one anatomical structure of the patient based on the relationship between the anatomical measurements and the target anatomical value.

[0004] As yet another example, a method is provided which includes acquiring two-dimensional (2D) imaging data of a patient's anatomical region, where the anatomical region includes the patient's anatomical features. The method also further includes using the 2D imaging data to generate a three-dimensional (3D) model of the patient's anatomical region, where the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features. Generating includes performing anatomical context processing, which includes using an artificial intelligence (AI) model to annotate the contours, edges, or surfaces of the anatomical structures of the anatomical region shown in the 2D imaging data. Generating also includes performing a 2D-to-3D reconstruction that models the anatomical structures, optimizing the orientation, placement, and scale of the 3D digital volume, and resulting in a 3D anatomical representation of the anatomical region as a 3D model.

[0005] As a further example, a method is provided which includes acquiring two-dimensional (2D) imaging data of an anatomical region of a patient, where the anatomical region includes the patient's anatomical features. The method also further includes using the 2D imaging data to generate a three-dimensional (3D) model of the patient's anatomical region, where the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features. The method further includes identifying one or more deformities of the patient's anatomical features, where the one or more deformities are shown in the 3D model and, based on the 3D model, are identified in relation to a target anatomical value of the patient's anatomical feature. Identification includes acquiring anatomical measurements based on the patient's anatomical landmarks, such as those shown in the 3D model; comparing the anatomical measurements to a target anatomical value; and determining one or more deformities based on the comparison. The method also includes determining at least one correction to be made to at least one anatomical structure of the patient based on the relationship between the anatomical measurements and the target anatomical value. The method further includes engaging a surgical instrument with at least one anatomical structure of a patient and generating a report containing instructions for manipulating the surgical instrument to perform at least one correction to at least one anatomical structure of the patient.

[0006] Further aspects of this disclosure relate to systems and computer program products configured to perform the methods described above and herein. This summary is not intended to describe each aspect of this disclosure, all implementations, and / or all embodiments. Further features and benefits are realized by the concepts described herein. [Brief explanation of the drawing]

[0007] The embodiments described herein are specifically cited as examples in the last claim herein and are expressly claimed. The above and other purposes, features and advantages of this disclosure will become apparent from the following detailed description in conjunction with the accompanying drawings.

[0008] [Figure 1] This specification illustrates an exemplary process for surgical planning according to the embodiments described herein. [Figure 2] This specification illustrates an exemplary workflow of a method used for surgical planning according to the embodiments described herein. [Figure 3] This specification shows exemplary inputs and outputs of an AI model that performs anatomical context processing according to the embodiments described herein. [Figure 4] An exemplary distance transformation Di for a binary edge map according to the embodiments described herein is shown. [Figure 5] This specification provides an example of a planar transformation that arrives at the optimal solution for a 3D modeled anatomical structure of a patient. [Figure 6] This specification provides an example of a planar transformation that arrives at the optimal solution for a 3D modeled anatomical structure of a patient. [Figure 7] This specification illustrates an exemplary conceptual process for a 2D to 3D reconstruction operation according to the embodiments described herein. [Figure 8] This shows illustrative representations of the patient's body before and after the Rapidus procedure. [Figure 9] This shows illustrative representations of the patient's body before and after the Rapidus procedure. [Figure 10A] An exemplary example of the Rapidus treatment orthodontic procedure is shown. [Figure 10B] An exemplary example of the Rapidus treatment orthodontic procedure is shown. [Figure 10C] An exemplary example of the Rapidus treatment orthodontic procedure is shown. [Figure 11] Exemplary normative measurements for subjects of Rapidus and hallux grafting treatment are shown. [Figure 12A] This shows an exemplary osteotomy site for rapid joint fusion. [Figure 12B] This shows an exemplary osteotomy site for rapid joint fusion. [Figure 12C] This shows an exemplary osteotomy site for rapid joint fusion. [Figure 13A] An exemplary visualization of a Rapidus treatment plan according to the embodiments described herein is shown. [Figure 13B] An exemplary visualization of a Rapidus treatment plan according to the embodiments described herein is shown. [Figure 14] Shows an exemplary process of anatomical correction according to the aspects described herein. [Figure 15] Shows an exemplary process of rotational correction according to the aspects described herein. [Figure 16A] Provides an exemplary depiction of a barrel shape applied to the most distal portion of the first midfoot bone according to the aspects described herein. [Figure 16B] Provides an exemplary depiction of a barrel shape applied to the most distal portion of the first midfoot bone according to the aspects described herein. [Figure 16C] Provides an exemplary depiction of a barrel shape applied to the most distal portion of the first midfoot bone according to the aspects described herein. [Figure 17] Shows exemplary test results based on the intermidfoot angle. [Figure 18] Shows exemplary test results based on the intermidfoot angle. [Figure 19] Shows exemplary test results based on the intermidfoot angle. [Figure 20] Shows exemplary test results based on the intermidfoot angle. [Figure 21] Shows an exemplary virtual correction output related to the Lapidus procedure and adductus midfoot correction according to the aspects described herein. [Figure 22] Shows an exemplary virtual correction output related to the Lapidus procedure and adductus midfoot correction according to the aspects described herein. [Figure 23] Shows an exemplary process of generating a simulated WBCT model according to the aspects described herein. [Figure 24] Shows an exemplary process of generating a simulated WBCT model according to the aspects described herein. [Figure 25] Shows an exemplary overall workflow of surgical planning according to the aspects described herein. [Figure 26] Presents an exemplary interface for user authentication according to the aspects described herein. [Figure 27]This specification shows an exemplary interface for initiating a new case of surgical planning according to the embodiments described herein. [Figure 28] This specification shows an exemplary interface for initiating a new case of surgical planning according to the embodiments described herein. [Figure 29] This specification shows an exemplary interface for initiating a new case of surgical planning according to the embodiments described herein. [Figure 30] An exemplary interface for image type and lateral selection according to the embodiments described herein is shown. [Figure 31] An exemplary interface is shown below, according to the embodiments described herein, for the user to confirm (or reject) the preoperative anatomical position and one or more measurement axes. [Figure 32] An exemplary interface is shown below that allows for selecting whether to proceed with the Rapidus treatment plan when an adduction metatarsal condition is identified, according to the embodiments described herein. [Figure 33] An exemplary interface is shown below that allows for the selection of whether or not to correct the adductor metatarsal condition as part of a Rapidus treatment plan, according to embodiments described herein. [Figure 34] An exemplary interface for presenting a corrective preview, according to the embodiments described herein, is shown. [Figure 35] An exemplary interface for updating target values ​​according to embodiments described herein is shown. [Figure 36] An exemplary report view / save interface according to the embodiments described herein is shown. [Figure 37] An example summary page of a report, according to the embodiments described herein, is shown. [Figure 38A] This section shows example error messages that may be presented to software users. [Figure 38B] This section shows example error messages that may be presented to software users. [Figure 38C] This section shows example error messages that may be presented to software users. [Figure 38D]This section shows example error messages that may be presented to software users. [Figure 39A] This section shows example warning messages that may be presented to software users. [Figure 39B] This section shows example warning messages that may be presented to software users. [Figure 39C] This section shows example warning messages that may be presented to software users. [Figure 40] An exemplary bone model of the bones of the foot and ankle region according to the embodiments described herein is shown. [Figure 41A] An exemplary axis defined for the bones of the foot and ankle region according to the embodiments described herein is shown. [Figure 41B] An exemplary axis defined for the bones of the foot and ankle region according to the embodiments described herein is shown. [Figure 41C] An exemplary axis defined for the bones of the foot and ankle region according to the embodiments described herein is shown. [Figure 41D] An exemplary axis defined for the bones of the foot and ankle region according to the embodiments described herein is shown. [Figure 42A] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42B] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42C] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42D] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42E] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42F]This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42G] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42H] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42I] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42J] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42K] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42L] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42M] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 42N] This specification shows exemplary geometric measurements of the foot and ankle region determined by software, according to the embodiments described herein. [Figure 43] This specification shows a conceptual diagram of exemplary secure communication (over HTTPS) between a user / client and a cloud service hosted on a cloud computing platform, according to the embodiments described herein. [Figure 44] Examples of computer systems and related devices that incorporate and / or use embodiments described herein are shown. [Figure 45] An exemplary diagnostic report generated in accordance with the embodiments described herein is shown. [Modes for carrying out the invention]

[0009] Three-dimensional (3D) digital models are often invaluable representations of a patient's body for diagnostic and therapeutic purposes. The process involves creating a 3D digital model from patient imaging data and then identifying various pieces of information from the model. For example, the process can identify various anatomical landmarks, such as bones and their properties (e.g., axes), and use these landmarks to identify possible deformities. The process can then identify corrective actions for anatomical components to address these deformities, such as corrective positioning, adjustment, or excision.

[0010] According to several embodiments described herein, patient imaging data is used to construct a 3D digital model of the patient's human body. According to several embodiments, patient imaging data is two-dimensional (2D) imaging data ("images") such as radiographs generated from ionizing radiation (e.g., X-rays, gamma rays). From 2D images, embodiments can generate a 3D digital model. In certain embodiments, the 2D images consist of multiple images, possibly as few as two X-rays, providing different views of anatomical regions of the patient. More specifically, in some embodiments, a top X-ray and an internal or external sagittal view of the human body are acquired as two 2D images, and the process uses these two 2D images to construct a 3D model, which can be used to determine anatomical measurements, identify deformities that result in correction, and determine such corrections. Embodiments may be carried out in whole or in part by surgical planning tools implemented as software running on a computer system. Thus, one or more computer systems in a computing environment can incorporate, execute, and / or use the embodiments described herein.

[0011] According to several embodiments described herein, methods for facilitating surgical planning are provided. Such methods may incorporate the generation and use of 3D models. First, embodiments will be described with reference to Figure 1, which shows an exemplary process of surgical planning according to the embodiments described herein, and Figure 2, which shows an exemplary workflow used for surgical planning according to the embodiments described herein.

[0012] Referring first to Figure 2, the workflow can proceed according to one of two paths, depending on the specific patient imaging data available / input. Following the first path 201a, the workflow receives 3D patient imaging data input 202 and proceeds to 3D analysis 204 to generate 3D output 212. The 3D output refers to a 3D model of the patient's human body. Following the second path 201b, the method receives 2D patient imaging data input 206 and proceeds to anatomical context processing 208, followed by 2D-to-3D reconstruction processing 210 to generate 3D output 212. In either scenario, the 3D output 212 is used in the next stage by the surgical planning and procedure module 214 to perform surgical planning and procedure decisions and then generate a report. Further details of the workflow in Figure 2 are explained with reference to Figure 1.

[0013] The aspects of the process in Figure 1 can be performed by one or more computer systems, e.g., a computer, which execute program code to perform the features described herein. Optionally, some aspects of Figure 1 (e.g., 110) may be performed manually by a user, although such manual actions by a user may have corresponding actions performed by a computer system (e.g., in example 110, the computer system receives input patient data).

[0014] Process 100 in Figure 1 includes inputting patient data (110). The input patient data may include one or more of the following, but are not limited to: - Imaging data of the patient's body from multiple fields of view, such as X-ray, magnetic resonance imaging (MRI), computed tomography (CT), and weight-bearing CT images. In this example, the imaging data may be in the form of digital images of the patient's body and communications and medical (DICOM) images. - Data from optical scanners (e.g., to capture skin and other body properties / features), - Data from pressure sensors (e.g., force plate), - Tracking data (e.g., from optical or magnetic trackers), and - Other patient / related data such as age, weight, height, shoe size, foot or other body measurements, and DICOM data.

[0015] More generally, any information relating to / describing a patient's characteristics is useful and can therefore be obtained as part of the patient data. Patient characteristics may be features that can be compared to determined "normal" or other reference features, such as the characteristics of a general or specific population of other individuals.

[0016] The process then proceeds to generate a patient-specific model (120) for at least one anatomical region of the patient, for example, as the 3D output 212 in Figure 2, based on the input patient data. Generating (120) incorporates / uses at least a portion of the patient data provided as input in 110.

[0017] In an example where the input patient data includes imaging data, generating(120) can proceed along one of two paths, more specifically paths 201a and 201b in Figure 2. Referring first to path 201a, the input patient data in 110 includes 3D image data, and generating(120) receives the 3D image data input and performs 3D analysis 204 (e.g., segmentation, alignment, fitting, etc.) to produce a 3D output 212. The 3D analysis 204 may, in this example, include the use of artificial intelligence (AI) based models such as trained neural networks, such as artificial neural networks (ANNs), and more specifically, convolutional neural networks (CNNs), as an example. The AI ​​model can be configured to take in 3D image data / images of a patient that reflect at least an anatomical region (images showing a region the same size as the entire patient's body or the same size as any part of the patient's body), generate a rough segmentation of the image, identify the location and characteristics of anatomical features that may include anatomical structures, e.g., bones, present in the anatomical region, and use that information to generate an initial approximation of a 3D model of the anatomical region with the correct location and orientation of the anatomical structures. The 3D analysis can process the model at various stages to provide anatomical structures that have the correct size, pose, and shape (e.g., with the correct contours on the anatomical tissue of bones) so that the 3D model accurately and accurately reflects the patient's human body in detail. In some embodiments, the 3D analysis 204 includes repeatedly measuring the vertical (i.e., 90-degree) distance between bone models as the model shape evolves toward a patient-specific shape, which serves a dual purpose: to prevent overlaps between bone models (and thus physically impossible bone shapes) and to remove scanned extracortical surfaces that would result in overlaps. The generated 3D model can be output as 3D output 212.

[0018] In some embodiments, when the input patient data includes 2D imaging data and does not include 3D image data, the generation process (Figure 1#120) proceeds according to path 201b (Figure 2), where 2D image data is input (206), followed by anatomical context processing 208 and 2D-to-3D reconstruction processing 210 to generate the 3D output 212.

[0019] Anatomical context processing 208 uses a 2D input that includes, for example, image / image data of two or more fields of view (i.e., different angles) of an anatomical region of a patient. In the context of planning the Rapidus procedure as described elsewhere in this specification, the 2D input may, for example, be (i) a superior (dorsal plantar) field of view and (ii) a medial or lateral (e.g., sagittal) field of view X-ray of the patient's foot. Anatomical context processing 208 processes the image data and performs annotation embodiments as described herein, which may include a trained ANN, e.g., a trained CNN, to identify anatomical structures, e.g., contours, edges, surfaces, etc. of bones depicted whole or partially by the image data. The CNN may be similar to the CNN described above with reference to path 201a, except that the CNN performs image segmentation in two dimensions to identify, for example, the location of contours of the patient's body. In this regard, the identified body parts often overlap at least partially, and therefore such identification is impossible for the user to perform manually / mentally. A trained AI model (e.g., a CNN) can assist in this, and analytical context processing / annotation may be performed automatically, i.e., without manual / user input. User involvement in this may be unnecessary. However, it should be noted that user input can contribute to the training of the AI ​​model before its use / application in anatomical context processing. For example, during the development phase of an AI model, a user can provide annotations in generating the set of training data that the AI ​​model (e.g., a network) learns from. The goal of proper AI model training may be for the AI ​​model to learn the annotation process and, without user input, provide correct output, i.e., annotations, to image data that characterize anatomical structures.

[0020] Therefore, one aspect of the process (208 in Figure 2) is to use a pre-trained ANN, e.g., an ANN trained using the U-Net architecture or a modified version thereof, available at https: / / arxiv.org / abs / 1505.04597. In a particular example of the training process, a set of several hundred manually annotated 2D segmented images (e.g., dorsal, plantar, lateral, and oblique X-rays) is acquired. Some such acquired images ("samples") may be bilateral images that can be split into one-sided versions to increase the number of samples in the set. More broadly, it may be desirable to generate a sufficiently large set of sample structures that include planes of symmetry. Thus, the training process can expand the set and further increase the total number of samples by duplicating the samples by adding, for example, rotation, inversion, translation, and / or other image manipulations. Following this further expansion, the set can then be divided into subsets, for example, (i) a set of training samples used to train the model to an initial fit, (ii) a set of validation samples on which the fitted model is used to predict responses, for example, to avoid overfitting, and (iii) a set of test samples as a separate set used to evaluate network performance.

[0021] The AI ​​model is then trained to take 2D images (2D image data) as input, process the input, and determine / generate / provide an output by neural network inference. The output may, for example, be or include a binary segmentation mask for each of the bones in an anatomical region depicted by the image, such as multiple anatomical structures. For example, in the case of the foot and ankle region, the AI ​​model may examine to identify approximately 30 bones. The 2D segmentation can then be processed into a set of 2D points by computing (i) the center of gravity of each detected anatomical structure (e.g., bone), and (ii) certain other definable markers, such as the proximal and distal ends of the metatarsals and proximal phalanges in the case of a patient's foot. Neural network inference can be invoked separately for each input 2D image.

[0022] Figure 3 shows exemplary inputs and outputs of an AI model for anatomical context processing according to the embodiments described herein. Input X-ray images 302 and 304 are top and sagittal views of a patient's foot, respectively, and are inputs to AI model 306. The output of model 306 is annotated versions of images 302 and 304. Thus, image 302' is an annotated version of image 302, and image 304' is an annotated version of image 304. The annotations in these examples include contour lines (surroundings) of identified bones in the patient's foot. All visible bones are shown with contour lines on the 2D image plane, and different bone contour lines are provided in different colors to aid identification and understanding. It can be seen that there is significant overlap in many of the identified bones. In the examples, this annotated image data is displayed to the user on a display device of the computer system or in communication with the computer system. Other annotations and methods for identifying / distinguishing the human body are also possible.

[0023] Anatomical context processing 208 provides annotated image data, and the workflow then proceeds to 2D-to-3D reconstruction 210 in Figure 2 (part of the generation of the 3D model in 120 in Figure 1) using the annotated image data. Reconstruction 210 constructs the 3D image data / image using the annotated image data. Figure 7 shows an exemplary conceptual process of the 2D-to-3D reconstruction operation according to the embodiments described herein. In the example in Figure 3, there are two annotated X-ray images corresponding to two different fields of view.

[0024] In some embodiments, the centroid of the contour is used in reconstruction 210, but other features of the annotation—in this case, bones with color-coded contour lines—can also be used as needed. For example, reconstruction involves finding the centroid / center point of the depiction area for each annotated anatomical structure, i.e., each bone in these examples. The centroid of the anatomical structure can then be used in an optimization loop to roughly align the plane with a reference shape (e.g., normal or average, etc.) of the anatomical structure from, for example, a set of entire feet in a 3D model. In this embodiment, the process can optimize for each plane its orientation, its in-plane translation (ignoring any depth components of the projection), and its uniform scaling. The goal may be to define equivalent points ("pairs of points") between specific planar features on the image data (e.g., X-rays) and 3D features on the reference shape (e.g., 3D anatomical structure). The centroid (center point) is one option, as described above. Then, based on the definition of these pairs of points, the process can perform optimization, for example, by measuring the correspondence of the pairs of points as a fitness function.

[0025] In various planes where points representing bone center points exist, a “golden model” can be used in conjunction with statistical methods to provide an accurate anatomical representation of an anatomical region. This model fits the planes together into three-dimensional space, optimizing (rotating and scaling—though this is a “transformation”) how the planes position relative to each other and how they position themselves relative to the current best estimate of the entire 3D human body being processed. The “golden model” is, for example, a model with its anatomical region, in this example, the average or other “normal” structure (size, shape, orientation, etc.) of a human foot. The golden model can also encode certain information within it about how the human body might be expected to vary across a patient population—e.g., pose, variations in the shape of individual bones, etc.

[0026] Planar transformations, model shapes, and anatomical poses can be iteratively optimized by referencing the shape arrangement relative to each other. The plane centroid (from image data / X-rays) can be compared, for example, to the 3D centroid (the center point of the actual 3D model), and more broadly, to any defined pair of equivalent points (2D features correlated to 3D features). Furthermore, the "edges" of the 3D model can be compared to the edges of the imaging data (e.g., gradient boundaries) (taking into account the fact that the X-ray image data lies on a plane and its transformation is iteratively estimated). In this way, parameters are found for each plane while changing the anatomical pose. While the planes are generally expected to converge toward a golden model, the 2D image data provides information to adjust the plane transformation to match what the actual 2D image data represents, through image processing techniques such as edge detection and gradient spikes in the image. For example, with respect to bone surfaces, the technique might be to look for cortical bone surfaces that are nearly perpendicular to the projection (X-ray field of view) direction. At such features, it is expected that a drop / spike in the gradient of the X-ray signal will be observed. Using edge detection methods at various sensitivity levels, it is also possible to encourage optimization loops to capture both relatively faint edges on low-signal / low-contrast images while simultaneously finding relatively sharper / stronger edges on images where more "false positives" occur. The equivalent of an X-ray contour on a 3D model is defined as a subset of model vertices (mesh points) whose surface normals are sufficiently perpendicular to the projection (field of view) vector.

[0027] Examples of such shape modeling are provided using statistical shape modeling. In certain embodiments, a predefined statistical pose and shape model ("SSM," which is also used herein as an abbreviation to refer to the technique of statistical shape modeling using such a model) is obtained as input to the model metadata 708, model mean shape 710, and model shape variation 758 in Figure 7. The SSM can be generated based on a variety of segmented human body populations corresponding to the human body to be modeled from a given input patient image. Since the model can be tailored to a particular human body / anatomical region, an SSM can be selected from a variety of available sets of such SSMs configured for various human body / anatomical regions.

[0028] Once a suitable SSM is obtained, reconstruction can proceed for each annotated input image (read from DICOM data 702 as DICOM reference 704 input). For each such 2D image i (i=1, ..., number of input 2D images), the process is as follows: -a) Target point p (as plane XY coordinates on a 2D image) i It has a set (716 in Figure 7), -b) Perform the following image processing steps for each image (720 in Figure 7). -(i) For example, using gradient-size-based methods such as Sobel filtering or the Canny method, image edges of individual anatomical features that are at least partially shown in the image may be detected (Figure 722) (the image processing method may examine raw image data without anatomical context but is expected to correlate with specific anatomical features of interest), -(ii) Distance transformation of binary edge map (Figure 7, 724) D i Calculate (such distance transformation D of binary edge map according to the embodiments described herein) i (See Figure 4 for an example.) -c) A transformation T (initially arbitrary) that represents the following: i Initialize the 3D plane that will hold the data. -(i) 3 degrees of freedom for the face orientation, -(ii) Two degrees of freedom for the plane translation component (i.e., not three-dimensional because the "depth" component of the projected field of view is arbitrary / undefined), -(iii) One degree of freedom of the overall scaling of the plane.

[0029] Initialization can be the initialization of the plane-wise transformation (718), and refers to a set of transformation parameters that can be adjusted (in 738, 754) and improved during optimization. An example of such initialization is the assignment of arbitrary values ​​to these parameters.

[0030] The 716 target points per plane provide an anatomical context for optimization, which can be based on the determination of anatomical structures (such as bones) contained within the field of view of a given image. Such determinations can be made manually, automatically, or semi-automatically using any desired method. In the example, this is done as a subprocess. In the embodiments described, an AI-based model, such as an artificial neural network (ANN), is used. More specifically, the model can be trained to identify specific bones (or other anatomical structures) and segment them on a 2D X-ray image (e.g., trace their boundaries). This can identify structures that are present and absent in the image, provide a rough estimate of where the present structures are located, and can be used for "point-to-point" optimization. The advantage of AI model-based methods is that this embodiment can be fully automated.

[0031] Next, for each anatomical structure (e.g., bone) to be analyzed (j=1, ..., number of anatomical structures), the process is to create a 3D model M based on laterality (712 in Figure 7) by, for example, searching from the model metadata 708 in Figure 7 and the model average shape 710 in Figure 7 from the SSM, average pose and anatomical structure (e.g., bone) shape combinations, for example. jProceed to mesh initialization (706 in FIG. 7), which initializes (714 in FIG. 7) what is sometimes called the patient mesh (732 in FIG. 7). The SSM provides a baseline or “normal” human body for each anatomical structure (e.g., bone), and the appropriate model of that structure as shown in the patient's body is ultimately derived by making modifications to the baseline until it matches the patient's body as shown by the image / scan such that the baseline is reflected. Starting with the mean shape (710 in FIG. 7) from the SSM, the process can make any number of modifications (740, 756 in FIG. 7) to the mean shape. The SSM can also include data regarding the variation of the shape, e.g., how it changes and how large the variation is from normal (shown by variation input 758). In this way, the statistical shape model can provide information on (a) the mean shape (710 in FIG. 7) and (b) how the mean shape varies across a particular population (758 in FIG. 7), and the process can repeat them until a sufficiently good match with the patient's body as represented by the input image is found.

[0032] As a specific example, the process searches for and tests meaningful poses and shape variations from the SSM to transform each plane T i (718 in FIG. 7 - per - plane transformation), each plane correlated with the input image, and each component of each structure model M j (732 in FIG. 7) by performing (executing) an optimization loop that iteratively optimizes them. An example of the optimization loop is the point - to - point optimization loop (726 in FIG. 7), where (i) for each plane, a 3D target point set P i (728 in FIG. 7) is defined by combining each p i (e.g., 716) with the current best estimate of each T i , and then (ii) using Procrustes analysis (734 in FIG. 7), P j (i.e., from 732 in FIG. 7) on the current best estimate of M iA rigid body transformation is found that minimizes the sum of the squared differences between the plane and the corresponding 3D reference point (730 in Figure 7). In this embodiment, each plane is optimized individually (e.g., using the Procrustes technique). After these optimizations, the aggregated results, for example, the sum of the squared errors for each plane, are taken to check whether the improvement yields a global solution. If so, updates are made to the plane and mesh (and optionally, there may be product-specific constraints that prohibit certain impossible relative spatial configurations of the plane). Thus, in each iteration, the solution with the smallest Procrustes error (736 in Figure 7) can be selected and iteratively refined according to the above through (i) plane transformation adjustments (738 in Figure 7) that generate plane-specific transformations (718 in Figure 7), and (ii) mesh pose (i.e., pose) and shape adjustments (740 in Figure 7) that are informed by model shape variation data (758 in Figure 7) that generate an updated patient mesh (732 in Figure 7). In each iteration, there may be some choice to discard certain updates if they appear to increase the recorded error metric. At the end of each iteration, a determination is made as to whether the error metric is low enough (i.e., whether convergence has occurred). If not, the iteration continues; otherwise, the process proceeds to contour vs. edge optimization.

[0033] In this embodiment, the optimization uses a multiscale iterative technique, but other options, such as using the Levenberg-Marquardt algorithm, may be available.

[0034] Another exemplary optimization loop is the contour vs. edge optimization loop (742 in Figure 7), where each M j (From 732 in Figure 7) and T i Given the current best estimate (from Figure 7, 718), the process threshold is calculated based on the vertex normal direction for each patient mesh (3D model, M jThe contour set (744 in Figure 7) is projected onto each plane and then evaluated as a contour distance metric (752 in Figure 7) according to Di (750 in Figure 7), i.e., the closer it is to the optimal global solution, the closer each projected contour point is to the nearest detected edge (from 746 in Figure 7) on the 2D input image. Contour vs. edge optimization 742 can be iterated as described above to generate a plane-by-plane transformation (718 in Figure 7) through plane transformation adjustment (754 in Figure 7), and to generate an updated patient mesh (732 in Figure 7) through mesh pose (i.e.) and shape adjustment (756 in Figure 7) informed by model shape variation data (758 in Figure 7).

[0035] Figures 5 and 6 show examples of planar transformations that arrive at the optimal solution for a 3D model of a patient's anatomical structure. In particular, Figure 5 shows intermediate or final combinations of transformations of multiple (in this case, two) planar images onto the optimized shape model. The green outlines show the planar projections from the 3D model, visible in blue, projected onto both 2D input images. Figure 6 is similar to Figure 5, but with an opaque mesh surface and more realistic lighting. These examples can be rendered in any desired way, for example, as a 3D model using a triangular mesh, or as a point cloud or other non-mesh representation. Thus, the above-described aspects of reconstruction estimate both the spatial configuration of all input planes (how they are positioned relative to the 3D model), as well as the shape, pose, etc., of the 3D model depicting the human body.

[0036] Therefore, according to the above embodiment, statistical shape modeling techniques can be used when generating a 3D model from 2D image data input. This can be useful in situations where the available input image data is limited, for example, when only X-ray (or other 2D image) data from a limited field of view is input, and in some cases, when only two fields of view are available. While the process can identify certain features on these projection planes / images, a considerable amount of information conveying a true 3D representation of the human body is unavailable, and it is desirable to estimate that information. In an example, the human bodies of a particular population are used to define a Standard Scheme Model (SSM) that describes what can be expected three-dimensionally on average in that given population, and thus the SSM can reflect a "normal" human body and / or a human body representing a state of varying severity. Thus, the 2D image data provides a subset of the 3D information necessary to generate a 3D representation / model of a particular patient's human body. The 2D image data (e.g., input projection images containing a subset of information) is analyzed to identify features that serve as anchors to determine the remaining information necessary to construct a 3D model of the patient's human body, i.e., to indicate variations in the SSM model to be attempted. Specifically, the process uses a SSM (model) as a source from which information is sampled to determine missing 3D features not visible in the projection image. The SSM can encode the base shape / pose of a particular human body (for example, average or normal) such as bones within a target population, and can also quantify how the base changes within that population. In embodiments, the base SSM can be modified toward features observed from the patient's projection plane, and then missing information can be interpolated from the SSM. In this way, features shown from the projection image facilitate the extraction of relevant information from the SSM model to nearly meaningful locations.

[0037] As described above, SSM models may include / show variations from the baseline, for example, variations corresponding to a patient's symptoms. Examples of such symptoms specifically discussed herein include hallux valgus and adductor metatarsus. In either case, parameters of the statistical shape modeling process can be used to weight the importance of symptom-specific variations / poses used in the statistical shape modeling process. That is, the SSM process can be guided by emphasizing variations / deviations from the baseline, normal, or mean in order to explore and accurately model the human body of a particular patient. These variations may correspond to specific symptoms. Thus, the process utilizes a combination of parameters that approximates the human body of a baseline or average patient, which may or may not have a given symptom (such as a hallux valgus foot).

[0038] Additionally or alternatively, it may be desirable to investigate what is represented by 2D image data of a particular patient, identify characteristics of the patient or patient population to which that particular patient best fits, and use this information to reflect in specific model parameters to facilitate the generation of 3D models. For example, using what is found in the input data about the patient's body, a particular population and / or related combination or model parameter—e.g., a population with hallux valgus and / or metatarsal adduction, as an example—can be identified and used to reflect in combinations of model parameters, which can then be applied to the mean population SSM to help generate 3D model data of the patient's body. Alternatively, various base SSM models corresponding to different patient populations can be maintained, and the characteristics identified from the input data can indicate which of those base SSMs will serve as a starting point for generating patient-specific 3D models. A population can be defined at least partially by a specific disease, such as hallux valgus or metatarsal adduction, but does not need to be associated with any disease in itself. It can be defined by any other characteristic, trait, etc.

[0039] Continuing to refer to Figure 7, the process assigns planes (760 in Figure 7), that is, determines which of the submitted fields of view in the input data corresponds to which plane in the 3D representation (distal plane field of view, lateral field of view, etc.), normalizes the plane direction to align with a given coordinate system, scales the structural model, and performs output validation. For scaling, any one or more options are possible. Examples include scaling based on input (from 110) indicating the patient's anatomical size (for feet, such as shoe size, foot length, etc.), using image metadata to reconstruct the scale, introducing objects of known dimensions in the scanning / imaging process, i.e., using scale markers, and / or prompting the user to input scale-related information and associate it directly with the image.

[0040] Regarding output validation, it is possible to know how the plane being viewed should be positioned relative to the human body and to each other. For example, there may be a reference point used for this. Thus, the validation effort can identify any scan errors. The rigor of validation can be set to any desired level and can be adjusted depending on the human body, the type of scan involved, and / or other factors. The content of the scan (the human body shown) and the resulting model can also be normalized—in the case of a foot—so that the ground / floor can be aligned with the long side (foot length) of the lateral scan. Additionally or alternatively, the process collects distal first metatarsal rotations relative to a virtual floor plane obtained indirectly from estimates, where the floor plane is a plane perpendicular to the short (vertical) side of the lateral view. When measurements are taken, measurements can be taken against a floor plane (in this example) that serves as a “horizontal” reference. In the case of a foot, this process can consider both dorsal-plantar and lateral views, regardless of whether it is the left or right foot (in the case of dorsal-plantar views, the view is approximately 15 degrees “forward”).

[0041] The process can further perform axis alignment / improvement and measurement steps (762 in Figure 7) as can be performed in workflow 201a, and send the data out as a 3D output 212 (764 in Figure 7) for use by a surgical planning module that performs surgical planning and procedures 214.

[0042] Generating a patient-specific model (120) may include a segmentation process that is performed automatically and / or manually. In some embodiments, there may be manual pre-segmentation option adjustments, but in other examples, segmentation is fully automated. Automated segmentation may include processing image data through a neural network to perform “coarse segmentation,” such as identifying which structures are which or identifying landmarks on the human body. The processing then builds a patient-specific model of the patient’s anatomical regions based on AI model work from the anatomical context 208 and knowledge of what the anatomical regions of a “normal” patient look like.

[0043] Regarding the validation of surgical plans, the goal may be to have a single field of view in the report that includes the projection field and several principal axes used for analysis / correction of structures within the anatomical region. Users can use these to partially validate the input data used.

[0044] 3D models may be generated within a coordinate system and / or assigned to a coordinate system. Furthermore, models can be annotated. Annotation was discussed as part of 3D analysis204 and anatomical context processing208, in which a neural network creates coarse 2D or 3D segmentation (labels) of the input image. In 3D context (204), the process can segment a 3D structural (e.g., bone) model from a 3D image, and in 2D context (208), the process can reconstruct a 3D structural model from multiple 2D images.

[0045] In the example, as part of the 3D output 212, multiple axes of an anatomical structure are fitted to a 3D model, various properties are measured, and shown as preparation for the surgical planning module 214.

[0046] In this example, distance mapping is used to examine how far anatomical components in a 3D model are from other components in the 3D model, and to display this to the user as needed. This can be useful, for example, when demonstrating orthodontic simulations or when performing necessary measurements. In a 3D anatomical model, distance mapping samples the distance from a surface point, such as a bone surface point, along the surface normal of the encountered anatomical component. The surface normal points in the direction perpendicular to the surface on which the surface point is located. This sampling can be done for each of several surface points in the 3D model. For each sample, the distance can be measured by starting from the surface point and tracking / crossing along the surface normal away from the surface until another anatomical component in the model, such as another bone, is encountered. This gives, for example, the distance between a surface point (e.g., a bone) and an adjacent component (e.g., another bone).

[0047] This information can be analyzed for each of several surface points and then represented graphically, for example, as a color map. This process can notify the software user if the 3D model represents problematic anatomical arrangements, such as overlapping bones or other things that may be considered anatomically impossible, and thus can indicate errors that may occur in the segmentation process. In this regard, distance mapping provides a "health check" of the 3D model.

[0048] This distance-based distance mapping and color coding can be used with a current (pre-operative) 3D model of the patient's body to indicate joint abnormalities, the positioning of anatomical components within a joint relative to other anatomical components, and so on. This can be useful to physicians or other users for abnormalities beyond what a user might identify simply by viewing the model without applying distance-based color coding. Additionally or alternatively, distance mapping and color coding can be used with a proposed post-operative 3D model to evaluate the post-operative patient's body and check or verify, for example, whether the post-operative position represented by the post-operative model has returned to the "normal" and / or the patient's "original" range (i.e., before the onset of symptoms). In this regard, distance mapping to the post-operative 3D model can be used to check (automatically and / or by the user) whether the surgical procedure has achieved the desired adjustment, for example, whether the normative corrected position has been achieved. In the case of bunion correction, for example, it can indicate whether the distance between selected anatomical features of the foot is within the desired range.

[0049] Continuing the process in Figure 1, after generating a patient-specific model of the anatomical region, the process identifies one or more anatomical landmarks within the patient's anatomical region within the model (130). Examples of landmarks, in addition to the surfaces and contours of anatomical structures, include axes, e.g., bone axes. In the context of the Rapidus procedure described herein, two major deformities can be measured, identified, and corrected in the software, at least partially simultaneously, as will be discussed later with reference to Figures 14 and 15. For example, a bunion is anatomically corrected by correcting the angle between the long axes of the first and second metatarsals (intermetatarsal angle - "IMA") and eliminating rotation of the first metatarsal. Correcting the deformity of a bunion is not so simple, as it often involves rotating the first metatarsal to address the rotational deformity and also reducing the IMA (however, in some cases, it may not be enough to reduce the angle to the desired angle to correct the deformity).

[0050] Continuing with Figure 1, the process collects one or more measurements within an anatomical region based on landmarks (140) and calculates the difference between the collected measurements and a set of normal values ​​for the measurements (150). Next, based on the calculated difference, the process identifies the deformity within the anatomical region (160). Based on this, the process identifies the corrected position of one or more anatomical structures (170) and identifies one or more corrections that, when performed, manipulate one or more anatomical structures within the anatomical region from their respective deformed positions to their corrected positions (180). In this embodiment 180, these corrections may be a combination of operations that result in the target position. As mentioned above with respect to the correction of bunion deformities, the operations may not be independent of each other, meaning that the performance of one operation, such as angle adjustment of the angle between two bones, may cause another operation, such as a rotational change in one of those bones. Therefore, if a net angular change of 5 degrees is desired between two bones, and a net rotational change of -3 degrees is desired in one of those bones, performing this in two operations—a 5-degree angular change followed by a -3-degree rotation of one of the bones—will result in a net angular change and / or net rotational change other than the desired 5 degrees / -3 degrees. The 5-degree angular operation may itself give a -1.5-degree rotation, and in that case, for example, a further -3-degree rotation will undesirably result in a net rotational change of -4.5 degrees.

[0051] Therefore, deformation can be corrected by a more holistic or complex approach to determine the appropriate correction. Conventionally, when dealing with bunions, the correction of angles and rotations performed has not been considered a complex decision. According to the embodiments described herein, the correction can be determined holistically, for example, to produce precise hardware such as cutting guides and implants.

[0052] Based on the measured deformity and calculated corrected position, the process in Figure 1 continues by generating one or more hardware components (190). Part of the hardware generation may be the generation of specifications for such hardware, e.g., specifications for 3D printable hardware, where the specifications are data / files that guide the printing or other formation of the hardware by the hardware generation equipment. Hardware components may, for example, guides for surgical operations, cuttings, corrections, or other procedures. For example, hardware may include, for example, cutting guides that precisely control cuttings made on anatomical features, correction guides for anatomical corrections, fixation devices for bone fixation, and / or implants.

[0053] The embodiments shown in Figures 1 to 7 can be performed or facilitated by a computer system usable by an end user, such as a healthcare professional. The computer system can run software to perform the embodiments described herein and facilitate interaction with the user, such as input and output. In the example, a front-end is provided as user-facing software. The front-end may be, for example, a web application having analyzed cases and downloadable reports of those cases. The downloadable reports may be reports of surgical planning and procedures, including the presentation, decisions, etc., of any data described herein.

[0054] The front-end can be contrasted with the back-end of software that can run on one or more computer systems, which may or may not be the computer system running the front-end, or which may include it. In an example, the back-end component includes components that implement the aspects shown in Figures 1 and 2, such as a database of image data (X-ray, CT, etc.), analyzers, and processors, surgical planning and procedure modules, and reporting components. In a particular example, the back-end component operates in a public, private, or hybrid cloud environment with an application programming interface and its management. For example, one or more databases may be provided to hold biometric and / or other supplementary information about a patient, such as demographics, height / weight and other physical characteristics, diagnoses, surgeries performed, etc. An analytical database may also be provided. In some examples, the database information is combined with results from actual image analysis, and in some cases, recurring patterns can be detected / learned from this big data. The cloud platform can run containers and / or other processing environments (such as virtual machines) to implement components and perform functions such as receiving image data input, performing the analyses and other operations described herein on 3D and 2D images, implementing surgical planning and treatment modules, and generating reports and images.

[0055] The disclosed embodiments are further illustrated and described with reference to the Rapidus procedure as an example. Figures 8 and 9 show illustrative representations of the patient's body before and after the Rapidus procedure. First, Figure 8 shows the preoperative patient's body showing the location of the bones exhibiting a deformity called a bunion ("hallux valgus"). Lines drawn on the first and second metatarsals, as well as the first proximal and distal phalanges (i.e., of the big toe), indicate the axes of the respective bones. Examples of preoperative measurements are as follows:

[0056] [Table 1.1]

[0057] Positive rotation of the first metatarsal bone indicates pronation, while negative rotation indicates supination.

[0058] Figure 9 shows an exemplary postoperative (corrected) patient's body, again with lines indicating the axes of each bone, including the first and second metatarsals, and the first proximal and distal phalanges (i.e., the big toe). Fixation devices are also shown. Examples of postoperative measurements are as follows:

[0059] [Table 1.2]

[0060] A positive correction amount indicates an increase in the measured value, while a negative amount indicates a decrease.

[0061] The anatomical corrections involved in the Rapidus procedure can be generalized to intermetatarsal angle correction, first metatarsal rotation angle correction, and big toe correction (which do not necessarily have to be separate steps). Further details are provided with reference to Figures 10–16.

[0062] Figure 10A shows an exemplary intermetatarsal angle (IMA) correction. The correction represents the angle adjustment (e.g., reduction) between the first and second metatarsals of the foot, when considered separately from any other corrections (e.g., rotation) that may be performed as part of a standard Rapidus procedure. Lines 1002, 1004, and 1006 represent the positions of the first metatarsal, proximal, and distal phalanx, respectively, before IMA correction. Lines 1012, 1014, and 1016 represent the positions of the first metatarsal, proximal, and distal phalanx, respectively, after IMA correction.

[0063] Figure 10B shows an exemplary first metatarsal rotation correction with lines 1020 and 1022 representing the rotation applied to the first metatarsal bone. Line 1020 represents the axis before rotation, and line 1022 represents the axis after rotation. A fixation device 1024 is also shown.

[0064] Figure 10C shows a hallux valgus correction with lines 1032 and 1034 representing the pre-correction positions of the first proximal and distal phalanges, respectively, and lines 1036 and 1038 representing the post-correction positions of the first proximal and distal phalanges, respectively. Hallux valgus correction rotates the first proximal and distal phalanges around the metatarsophalangeal (MTP) joint to bring the hallux valgus angle to the desired / normative value. As mentioned above, the hallux valgus correction in Figure 10C may be given as part of IMA and rotational correction and may not be performed as a separate correction / treatment.

[0065] The surgical planning and procedure module (e.g., 214 in Figure 2) can provide an expandable framework for performing a series of surgeries in the context of the Rapidus procedure. Normative reference values ​​for all desired measurements can be determined from images, e.g., a set of computed tomography images. Figure 11 shows exemplary normative measurements for subjects of Rapidus and hallux grafting procedures.

[0066] Figure 11 shows three distribution graphs for (i) the correction of the intermetatarsal (first and second) angle in degrees along the x-axis when considering the axial (top to bottom) field of view, (ii) the rotation of the first metatarsal in degrees along the x-axis, and (iii) the hallux valgus angle (from proximal to distal phalanx) in degrees along the x-axis when considering the axial (top to bottom) field of view.

[0067] The goal of Rapidus and hallux correction treatments is to return selected measurements of interest to normative reference values, or at least values ​​appropriate for the patient under the given circumstances.

[0068] Rapidus arthrodesis refers to a procedure that involves cutting and fusing two bones of the foot—the first metatarsal and the medial cuneiform. Figures 12A–12C show exemplary osteotomy locations for rapidus arthrodesis, as shown in the supranutorial view (Figure 12A) and sagittal view (Figure 12B) of the patient's foot. Figure 12C shows a more detailed view of the area shown in Figure 12B. To correct a bunion, rapidus arthrodesis can be used to provide multi-plane correction incorporating IMA adjustment and derotation. However, as mentioned earlier when discussing the corrective locations of anatomical structures and the interdependence that exists between the corrections of different structures, the manipulations of structures do not necessarily have to be independent of each other. Here, the IMA can be contracted by moving the distal portion of the first metatarsal laterally. In this example, the manipulation is performed, for example, as a rotation or modified rotation centered on or near a point on the head (proximal end) of the first metatarsal. Lateral movement / rotation can cause the first metatarsal bone to rotate around its long axis. When viewed along the long axis of the metatarsal bone from proximal to distal, the first metatarsal bone rotates counterclockwise (for the left first metatarsal bone) and clockwise (for the right first metatarsal bone) around the long axis. Similarly, rotation of the first metatarsal bone can essentially reduce the immobility margin (IMA). As a result, the desired IMA and amount of rotation are combined into an overall correction, which itself can be performed in one or more steps. In one example, the overall correction is performed by rapidus arthrodesis and precise osteotomy locations for fusion. The osteotomy locations here are cutting planes, and their location and other properties (including the angle of the cutting plane to a given plane reference, such as one perpendicular to the ground) are important. One of such planes (1210) is located at the distal end of the medial cuneiform bone, and the other (1212) is located at the proximal end of the first metatarsal bone. These two bones are positioned on the same plane as each other, and fixation is applied to fuse them together.

[0069] In performing arthrodesis, once the cutting planes are aligned and rotation is corrected, the saw of the first metatarsal bone may be aligned with the interarticular space, and the saw of the medial cuneiform bone may be positioned, so that the IMA and rotation are at a desired reference value, e.g., the patient's reference value (or within its threshold). Both planes can be moved along their plane normals to cut the entire joint region.

[0070] One method for estimating rotation is to measure the MT1 rotation angle from the articular surface of the distal head of the first metatarsal bone (MT1), and further details of this are discussed below with reference to Figure 15.

[0071] The Rapidus procedure may include phantom intramedullary nailing and nail selection. The nail and mounting screws, as well as the placement, are shown in Figure 10C, where the first metatarsal bone fuses with the cuneiform bone.

[0072] Figures 13A and 13B show the human body before and after correction in the Rapidus treatment plan, and optionally illustrative visualizations of intraoperative procedures, for example, showing IMA correction before and after rotational correction, which may be provided by software performed to carry out the embodiments described herein. In the example, a visual simulation can be automatically generated and presented to the user on a display device to visualize the transition of anatomical structures performed as part of the correction. The visual simulation can show the transition of a structure from a first position to one or more other positions using a 3D model and a representation of the structure as provided by the 3D model. The second position, which is one of such other positions, may be a target position that coincides with a target anatomical value of the patient's anatomical features showing the deformity / multiple deformities. In some examples, the visual simulation can show the progression of a structure through additional positions as the structure transitions from the first position to the second position. In this way, the visual simulation can be an animation showing the movement performed by the correction being performed. Orthodontics can be performed by at least one surgical procedure, such as cutting or connecting / separating instruments, and visual simulations may further include simulations of such surgical procedures, showing the work on anatomical structures as represented in a 3D model, in order to perform the orthodontics to be made.

[0073] The generated 3D model or any part thereof can be communicated visually to a physician, patient, or other user on a display device as described above, and / or provided in the form of a 3D printed model. Examples of such models / parts include models of the first metatarsal and medial cuneiform bones separated from each other, models of the first metatarsal and medial cuneiform bones in a unified configuration (in either the deformed position or the proposed corrected position or both), and / or models of the entire foot of a patient with the first metatarsal and medial cuneiform bones (in either the deformed position or the proposed corrected position or both).

[0074] In relation to the aforementioned Rapidus orthodontic treatment, please note that depending on the situation and as needed, additional corrections to the patient's thumb may be necessary, such as further deformity correction (e.g., by Akin osteotomy to correct the angle of the thumb).

[0075] Figure 14 shows an exemplary process 1400 of anatomical correction according to the embodiments described herein. The process in Figure 14 may represent an example of an embodiment of Figure 1 adjusted to the case of the Rapidus procedure described above, in which case the process in Figure 14 in the context of the Rapidus procedure may provide intermetatarsal angle correction. The process includes identifying at least one first landmark, geometric feature, region, or volume on or adjacent to a first anatomical subregion of a patient-specific model (e.g., a 3D model) (1410). An exemplary first landmark, geometric feature, region, or volume in the context of the Rapidus procedure is the long axis of the first metatarsal bone. An exemplary first anatomical subregion of a patient-specific model in the context of the Rapidus procedure is the first metatarsal bone (or medial column). The process also identifies at least one second landmark, geometric feature, region, or volume on or adjacent to a second anatomical subregion of a patient-specific model (1420). An exemplary second landmark, geometric feature, region, or volume in the context of the Rapidus procedure is the long axis of the second metatarsal bone, and an exemplary second anatomical subregion in the patient-specific model in the context of the Rapidus procedure is the second metatarsal bone. 1410 and 1420 can correspond to 130 in Figure 1.

[0076] The process determines the relationship ("first relationship") between at least one first landmark or geometric feature and at least one second landmark or geometric feature (1430). In the context of the rapidus procedure, this determination may be a measurement of the angle ("first angle") formed between at least one first landmark or geometric feature (the long axis of the first metatarsal) and at least one second landmark or geometric feature (the long axis of the second metatarsal). This first angle may be a measurement of the initial angle, which is the IMA associated with the deformed (e.g., bunion) state of the human body.

[0077] The process also determines the relationship between at least one first landmark or geometric feature and at least one second landmark or geometric feature ("second relationship") (1440). In the context of the Rapidus procedure, this determination may be a desired angle ("second angle") between at least one first landmark or geometric feature (the long axis of the first metatarsal) and at least one second landmark or geometric feature (the long axis of the second metatarsal). This second angle may be a desired corrected angle, i.e., a desired IMA that the human body should represent in the postoperative state to correct a deformity (e.g., a bunion). For example, the desired angle may be 11 degrees if it is desired to correct the patient to an IMA of 11 degrees. The desired angle, or more generally the desired relationship, can be determined in any desired way. In one example, it may be a population-based mean or other reference value (e.g., a "normal" value, or a range considered normal). Alternatively, it may be calculated and / or recommended by software based on the patient's human body, or it may be manually entered by the user. In the context of the Rapidus procedure, the desired angle may be (i) a known normal angle, (ii) a desired angle calculated and recommended by the software based on the patient's body, and / or (iii) a desired angle provided by a physician that can be entered directly, as well as / or a desired angle determined / selected as or based on a physically adjusted recommended or normal value.

[0078] The process in Figure 14 further includes determining the desired correction (1450) based on the first and second relationships. In the context of the Rapidus procedure, the first and second relationships can be the first and second angles described above, and therefore the correction is determined based on these angles. Thus, the correction can include IMA correction provided by a surgical procedure, e.g., Rapidus arthrodesis. In this regard, as previously mentioned, the IMA correction and rotational correction in the Rapidus procedure may be interdependent. For this reason, it may not be appropriate to simply determine the corrective measure (1450) as an angle adjustment equal to the difference between the first and second angles, since the required rotational correction may affect the IMA. Consequently, the determination of the desired correction (1450) may be determined in conjunction with the process in Figure 15, which illustrates an exemplary process of rotational correction related to the Rapidus procedure, for example.

[0079] Figure 15 includes applying a geometric shape to a landmark to assist in the determination of correction. In the context of the Rapidus treatment, the correction is rotational, and the geometric shape is cylindrical, more specifically, a vertically elongated ellipsoid with a truncated tip having circles of equal size at the top and bottom, which is geometrically called a "barrel" shape. The landmark in this case is the distal part of the first metatarsal bone. A cross-section of this part can be viewed against a 3D model, and the barrel can be fitted to it. The central axis of the geometric shape can then be used to illustrate the rotational deformation by comparing it to some reference (such as the true horizontal / ground or some other reference plane).

[0080] Referring to Figure 15, the process identifies at least one third landmark, geometric feature, region, or volume on or adjacent to a third anatomical subregion of the patient-specific model (1510). In the context of the Rapidus procedure, the exemplary third landmark, geometric feature, region, or volume may be the distal portion of the first metatarsal bone, and the third anatomical subregion of the patient-specific model may be the first metatarsal bone.

[0081] The process then applies the first geometric shape to at least one third landmark, geometric feature, region, or volume on or adjacent to a third anatomical subregion of a patient-specific model (1520), identifying one or more axes thereof. In the context of the Rapidus procedure, the geometric shape may be a three-dimensional geometric shape of a barrel / modified cylinder, and one or more axes may include the central axis of the barrel. In the geometric shape or fitting of geometric shapes of this application, the geometric shape may be fitted to a human body which may be a third anatomical subregion / bone surface, e.g., the distal articular surface of the first metatarsal bone. The axis of the barrel can serve as an estimate of the distal medial-lateral axis of the first metatarsal bone, which can then be used together with a fourth landmark (such as a floor surface, as described below with reference to 1530) to measure corrections (such as rotation of the metatarsal bone about its longitudinal axis). Figures 16A–16C provide exemplary depictions of barrel shapes applied to the most distal portion of the first metatarsal bone. Figure 16A shows the superposition of the barrel shape and the 3D triangular mesh representation of the first metatarsal bone. Figures 16B and 16C show the contour lines of the barrel shape on the contour lines of the 3D triangular mesh in a plane and cross-section perpendicular to the longitudinal axis of the first metatarsal bone, respectively.

[0082] Continuing to refer to Figure 15, the process identifies at least one fourth landmark, geometric feature, region, or volume on or adjacent to a fourth anatomical subregion of the patient-specific model (1530). In the context of the Rapidus procedure, the fourth landmark, geometric feature, region, or volume may be a region beneath the foot, and the fourth anatomical subregion of the patient-specific model may be the ground or horizontal plane to which rotational deformation can be measured. Depending on how the image data (e.g., X-rays) were acquired, it may be necessary to define such ground. The process may use, for example, the cross-sections provided / defined by the imaging device used to acquire the initial images and / or the cross-sections provided / defined by the patient-specific model.

[0083] The process measures a third relationship between at least one third landmark, geometric feature, region, or volume and an axis of geometric shape that fits that axis, and at least one fourth landmark, geometric feature, region, or volume (1540). In the context of the Rapidus procedure, the third relationship may be the angle between the axis of geometric shape that fits at least one third landmark and the region beneath the foot ("the third angle"), and the relationship is the angle which is the measurement of the initial rotational deformation. In other words, in the context of the Rapidus procedure, the rotational deformation of the first metatarsal is measured from the distal medial-lateral axis of the first metatarsal (e.g., as indicated by the axis of a barrel) and from the floor plane in a direction perpendicular to the longitudinal axis of the first metatarsal.

[0084] Using the measured third relationship, the process determines a fourth relationship between at least one axis of geometric shape fitting a third landmark, geometric feature, region, or volume and at least one fourth landmark, geometric feature, region, or volume (1550). In the context of the Rapidus procedure, the fourth relationship may be a desired angle ("fourth angle") between the axis of geometric shape fitting at least one third landmark and the region under the foot, i.e., a desired angle of the corrected (rotational) position. This fourth angle may be determined / selected, for example, as (i) a known "normal" angle, (ii) a desired angle calculated and recommended by the software based on the patient's physiology, and / or (iii) a desired angle provided by the physician.

[0085] Next, the process in Figure 15 continues by determining the desired correction (1560) based on the third and fourth relationships, for example, the third and fourth angles in the context of the Rapidus treatment. Thus, the correction may include rotational correction imparted by surgical intervention. In this regard, as previously mentioned, the IMA correction and rotational correction in the Rapidus treatment may be interdependent. Since the required IMA correction itself may result in some rotation, it may not be appropriate to simply determine the corrective measure (1560) as a rotational adjustment equal to the difference between the third and fourth angles. Therefore, the determination of the desired correction (1560) can be determined in conjunction with the procedure in Figure 14.

[0086] Therefore, while some aspects of Figures 14 and 15 can be performed before, simultaneously, or after each other, both can relate to the determination of the correction (e.g., in 1450 in Figure 14 and 1560 in Figure 15), which can be implemented in software as a composite decision based on the same decision, e.g., the processes performed in Figures 14 and 15. Alternatively, they can be separate decisions considering each other, for example, in Figure 14, if the difference between the first angle and the second angle is 11 degrees, this can be determined as part of 1450, and then adjusted based on the rotational correction determined according to Figure 15 (i.e., as part of 1560) to produce a desired correction that is, as an example, actually -10.5 degrees. Similarly, the amount of rotation can be determined as part of Figure 15 (1560), but then adjusted as part of 1560 based on the desired angle correction (from Figure 14). As yet another example, each decision (1450, 1560) can be performed independently of the other (for example, determining apparent corrections as apparent angle adjustment of IMA (e.g., -11 degree IMA adjustment based on the required apparent angle adjustment of 11 degrees) and apparent rotation (e.g., -4.2 degree rotation based on the apparent difference of 4.2 degrees), these are respectively input into another decision (not shown) that determines a composite / actual adjustment based on these apparent corrections, the composite being, for example, a -10.5 degree actual IMA adjustment with a 3.8 degree rotation to achieve the desired postoperative anatomical outcome).

[0087] While the exemplary algorithms above used normative reference values ​​for IMA correction and bone rotation, it should be noted that the embodiments described herein allow for the creation of solutions specific to each patient. For example, one may encounter a patient who has a deformity with respect to IMA (e.g., IMA greater than the reference value) but no rotational deformity. In such cases, it is desirable to reduce and correct the IMA, but it is also desirable to identify that the patient does not have a rotational deformity and that the surgical plan does not require rotational correction (or vice versa in the case of rotational deformity without IMA deformity). Otherwise, there is a risk of removing metatarsal rotation when rotational deformity is not present, thus creating a new problem for the patient. With respect to IMA or rotation, in determining whether a deformity is present, the process can determine whether the patient's IMA or rotation value (in these examples) is some threshold amount different from the “normal” value. For example, if the “normal” value for IMA is understood to be 11 degrees, the process can classify the patient’s IMA deformity if the patient’s IMA is measured to be more than 3 degrees different from that normal value. In this regard, as more data is collected, deformities may be classified in terms of how many standard deviations the patient’s IMA is from the “normal” value. While the aforementioned points are made in relation to IMA, it should be emphasized that they are also applicable to rotational deformities. More broadly, different procedures require consideration of different measurements of different anatomical sites, and such measurements have their own applicable normal / reference values ​​to consider.

[0088] The exemplary features described herein include methods relating to generating 3D models from 2D imaging data. The methods may include obtaining 2D imaging data of an anatomical region of a patient. The anatomical region may include the patient's anatomical features. Anatomical features may be anything that can be compared to a determined “normal” or other reference feature, such as the features of a general or specific population of other individuals. They may be, or at least related to, the characteristics of physical anatomical structures (such as dimensions, shape, orientation, and position), presented and confirmed by anatomical structures present in the anatomical region of interest. The methods may also use the 2D imaging data to generate a 3D model of the patient's anatomical region. The 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features. One or more deformations of anatomical features may be displayed in the 3D model (however they may or may not be visually recognizable to a user viewing the 3D model on a display), and the deformation refers to a deviation from a reference feature. exemplified deformations may be anatomical components that deviate from their conventional or reference position, for example, due to trauma, such as a fracture. Another example of deformation may be anatomical components that lack conventional features / characteristics (e.g., bones with abnormal curvature).

[0089] The method can proceed by identifying one or more such deformities of the patient's anatomical features. The deformities are identified based on a 3D model and determined in relation to target anatomical values ​​(e.g., “normal” values ​​or other reference values) of the patient's anatomical features. Identifying deformities may include, for example, identifying the patient's anatomical landmarks (e.g., angles, distances, orientations, or any other measurements characterizing the patient's body) as shown in the 3D model, obtaining / acquiring anatomical measurements based on those anatomical landmarks, then comparing the anatomical measurements to target anatomical values, and determining one or more deformities based on that comparison. Target anatomical values ​​may be, or include, a certain value relative to a desired range into which the anatomical measurements and / or anatomical measurements should fall.

[0090] Once one or more deformities are determined, the method can determine at least one correction to be made to at least one anatomical structure of the patient, based on the relationship between the anatomical measurement and the target anatomical value. An example of such a relationship is the difference between the patient's anatomical measurement and the reference value. The difference can provide information about the amount of correction or correction based thereon. The correction can indicate at least one corrective position for an anatomical structure or set of anatomical structures. The correction made to an anatomical structure ultimately addresses the deformity by providing a correction that, when made, produces updated anatomical measurements. In some cases, the correction does not need to conform the patient's body to the reference value, but rather to bring the values ​​related to the patient's body closer to the reference value, for example, closer to the reference value (e.g., to fall within a desired range).

[0091] In situations involving patient-specific hardware to facilitate at least one correction performed on at least one anatomical structure, the method can optionally generate hardware specifications. These specifications can be generated based on (i) a representation of the patient's anatomical features provided by a 3D model, and (ii) the determined at least one correction. The specifications may include patient-adjusted measurements / parameters based on the representation of the patient's anatomical features provided by the 3D model and the determined correction. In an example, the patient-specific hardware includes at least one hardware guide that guides the surgical procedure to provide the correction performed on at least one anatomical structure. The hardware guide may include, for example, a cutting guide for a cutting procedure.

[0092] Optionally, the method can generate a visual simulation that graphically presents the transition of the patient's anatomical structure from a first position to a second position, as represented in a 3D model. The first position can be the existing preoperative position, and the second position can be a position that matches the target anatomical value of the patient's anatomical features. Correction may be performed by surgical / multiple operations (cutting, attachment / detachment of instruments, etc.), and the visual simulation can include simulations of surgical / multiple operations on the patient's anatomical structure as represented in the 3D model.

[0093] Additionally or alternatively, a method for generating a 3D model may include acquiring 2D imaging data of a patient's anatomical region, wherein the anatomical region includes the patient's anatomical features, and using the 2D imaging data to generate a 3D model of the patient's anatomical region, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features. The generation of the 3D model may include performing anatomical context processing, which includes, for example, using an artificial intelligence (AI) model to annotate the contours / edges / surfaces of the anatomical structures of the anatomical region presented in the 2D imaging data, and performing a 2D-to-3D reconstruction that optimizes the orientation, position, and scale of a 3D digital volume modeling the anatomical structures (e.g., a digital volume representing anatomical structures such as bones), thereby generating a 3D anatomical representation of the anatomical region as a 3D model. In embodiments, the 2D imaging data includes two or more images / different fields of view of the anatomical region. The images / different fields of view may be, for example, radiographs.

[0094] When used, AI models can perform 2D image segmentation to identify the contours / edges / surface locations of anatomical structures. Image segmentation can generate 2D segmentation. The method can process 2D segmentation into a set of 2D points, which can be based on determining anatomical landmarks for each anatomical structure, such as the centroid of the anatomical structure.

[0095] Annotation of anatomical structures can provide annotations that include contour lines around the anatomical structure. In an example, the anatomical structure includes bone, and the annotation shows the contoured bone on one or more 2D image planes. The method can display the contour lines around the anatomical structure on a display, and the contour lines are presented with various graphic characteristics, such as different colors for different bones, to facilitate identification and distinction between anatomical structures.

[0096] In the example, 2D-to-3D reconstruction involves fitting together planes of 2D imaging data depicting anatomical structures in 3D space. The fitting may include iteratively transforming the planes to change how they position themselves relative to each other. The transformations may be based on comparative anatomical models ("golden models," or SSMs). The fitting provides a 3D anatomical representation of the anatomical region.

[0097] In this example, 2D image data is informed of a planar transformation by one or more image processing techniques, such as edge detection and / or gradient spikes, to ensure consistency between the planar transformation and what is shown by the 2D image data. In this way, the image processing techniques applied to the source 2D data help keep the transformation reasonable / realistic and provide a boundary for how extreme the transformation may be. Without these, one or more transformations could be excessive compared to what is shown by the source 2D data.

[0098] A comparative anatomical model can be a 3D model having structures (e.g., models of individual bones within an anatomical region) that have comparative characteristics (size / shape / orientation) relative to an anatomical region, or may include 3D models. A comparative anatomical model can provide an index of how the comparative characteristics may vary based on a selected population of anatomical samples. This means an index of acceptable or reasonable deviation in how the characteristics may vary. These can be based, for example, on a statistical mean or range. For example, such an index can be encoded in the model itself, but additionally or alternatively, the index can be provided as part of metadata and / or other data files that have information about how reasonably the comparative model may vary.

[0099] The method may include building an AI model to process incoming 2D imaging data and provide annotations to it. Building such a model may include training the AI ​​model with 2D segmented images that have been pre-annotated with respect to human body landmarks. As mentioned above, landmarks may include bone features, such as the center of gravity and the proximal and distal ends of bones. In the example, a training dataset of samples is built, and the samples contain 2D segmented images. To increase the number of samples, building the training dataset may include taking a bilateral image and splitting the bilateral image into unilateral images that form part of the sample. Additionally or alternatively, one or more rotations, one or more flips, one or more translations, and / or other image manipulations may be applied to existing samples in the training dataset to generate additional samples for training.

[0100] Additionally or alternatively, a method is provided for obtaining a 3D model of the patient's anatomical region having the patient's anatomical features, identifying at least one deformity of the patient's anatomical feature, the at least one deformity being identified with respect to a target anatomical value of the patient's anatomical feature, the target anatomical value including a desired range into which the anatomical measurement should fall, the at least one deformity including at least one of rotational and angular deformities, and using the anatomical measurement, determining at least one plane correction to be made to at least one anatomical structure of the patient. The at least one deformity may include multiple deformities. These deformities may include angular deformities, e.g., an anatomical angle between a first and second anatomical feature of the patient's anatomical feature. The first and second anatomical features could be the patient's first and second metatarsals, for example, in the case of a bunion.

[0101] Additionally or alternatively, multiple deformities may include rotational deformities, such as a rotational deformity of the patient's first metatarsal bone (in the case of a bunion).

[0102] In the example, identifying at least one deformity involves selecting a geometric volume and applying the geometric volume to one anatomical feature among the patient's anatomical features presented in a 3D model, determining the measurements of the anatomical measurement using the properties of the geometric volume, which are interpreted as properties of the anatomical feature, and identifying one of the deformities among at least one based on the determined measurements.

[0103] Anatomical features to which geometric features are applied may include, for example, a cross-section of a bone or an articular surface of a bone.

[0104] In the example, the method involves acquiring imaging data of the patient's anatomical regions, and the acquisition of the 3D model involves generating a 3D model (as described above) from the acquired imaging data.

[0105] In situations involving patient-specific hardware to facilitate at least one correction performed on at least one anatomical structure, the method can optionally generate hardware specifications. These specifications can be generated based on (i) a representation of the patient's anatomical features provided by a 3D model, and (ii) the determined at least one correction. The specifications may include patient-adjusted measurements / parameters based on the representation of the patient's anatomical features provided by the 3D model and the determined correction. In an example, the patient-specific hardware includes at least one hardware guide that guides the surgical procedure to provide the correction performed on at least one anatomical structure. The hardware guide may include, for example, a cutting guide for a cutting procedure.

[0106] The modes of processing performed by the software described herein can help correct a variety of symptoms. Another exemplary symptom is adduction metatarsal, or a deformity in which the forefoot bends inward. This deformity can be optionally considered and addressed according to the modes described herein when dealing with other symptoms such as hallux valgus. Adduction metatarsal may appear in cases of bunions.

[0107] The software can identify adduction metatarsal bones based on the 3D model generated as described herein. In certain embodiments, this is detected by measuring a single angle, specifically the second tarsometatarsal angle (e.g., the angle between the longitudinal axes of the second metatarsal bone and the intermediate cuneiform bone) (in a cross-section). If this angle exceeds a threshold, e.g., -24 degrees, adduction metatarsal bones are detected. If adduction metatarsal bones are detected, there are various options.

[0108] One option is to (virtually) correct adductor metatarsal deformity by rotating the second and third metatarsals (in transverse plane) until the second tarsometatarsal angle falls within the normative reference range. This correction is expected to increase the intermetatarsal angle (IMA) between the first and second metatarsals, which is the target of the Rapidus treatment, and therefore influence the degree of IMA adjustment required as part of bunion correction. Conventional bunion treatments do not give such consideration to correction of adductor metatarsal deformity as part of the Rapidus treatment. In this regard, the embodiment provides a surgical planning method for the Rapidus treatment in which the correction of adductor metatarsal deformity is planned first, and then the hallux valgus correction is planned based on the (corrected) adductor metatarsal deformity.

[0109] Another option is to take the adducted metatarsal condition into account when determining the amount of correction to be performed as part of the Rapidus treatment to correct hallux valgus. Under this approach, instead of rotating the second metatarsal bone first, as in the above option, the plan first rotates the first metatarsal bone to an overcorrective degree (relative to the normative value) as part of addressing the hallux valgus, so that the final result addresses not only the bunion but also the adducted metatarsal deformity. Furthermore, the amount of correction is limited by the distance between the heads of the first and second metatarsal bones, and software can prevent their overlap, thereby limiting the amount of correction.

[0110] In certain embodiments, the surgical planning software identifies and notifies the user whether adduction metatarsal pressure has been detected, for example, based on a second tarsometatarsal angle (cross-section) exceeding a threshold. In a particular example, the threshold is -24 degrees. The user can be given options for continuing the Rapidus treatment plan, for example, continuing without correction addressing the adduction metatarsal pressure, and continuing with correction addressing the adduction metatarsal pressure.

[0111] In this example, choosing not to address / correct the adduction metatarsal is an overcorrection of the first throne of the foot (i.e., the segment consisting of the first metatarsal and first cuneiform bones), meaning that the first-to-second metatarsal angle (cross-section) is corrected to a normative standard as if the second metatarsal were in its normal position defined by the normative standard value of the second tarsometatarsal angle (cross-section). Figure 21 shows an example of the Rapidus virtual correction output based on choosing not to address the adduction metatarsal. In this example, the planned postoperative first-to-second metatarsal angle (cross-section) is 5.7 degrees, the second metatarsal remains in its preoperative position, and the overcorrection is limited so that the bones of the first throne do not overlap with the bones on the second throne (i.e., the segment consisting of the second metatarsal and second cuneiform bones).

[0112] In this example, if the option to address / correct adduction metatarsal is chosen, the second and third thrombotarsals (i.e., the segment consisting of the third metatarsal and third cuneiform bones) are rotated so that the second and third tarsometatarsal angles (transverse plane) match their normative values. The Rapidus arthrodesis can then be performed successfully, but as mentioned above, the IMA correction performed as part of the Rapidus procedure may differ from that in cases where adduction metatarsal is not addressed. Figure 22 shows an example of the Rapidus virtual correction output in the option to address adduction metatarsal. The correction of the second and third thrombotarsals is shown for visualization. In this exemplary case, the planned postoperative first-to-second metatarsal angle (transverse plane) is the default reference value of 11.4 degrees.

[0113] When correcting the anatomy of the foot, plantar flexion, particularly of the first metatarsal, can be further considered. Resection at the tarsometatarsal (TMT) joint results in shortening of the first retina, as the first metatarsal is moved posteriorly toward the medial cuneiform. The metatarsal head is lifted off the floor because the first metatarsal is angled relative to the floor.

[0114] It may be desirable to maintain a constant height of the first metatarsal head, to make it substantially the same as its height before surgical intervention, or to avoid this by making it a different height. For this purpose, the software can simulate the position of the first metatarsal head on the patient if the deformity subject to surgical intervention did not exist, based on a generated 3D model of the patient's human body. In other words, the software can determine where the first metatarsal head would be located for this patient if the angle (or other marker of the deformity) were within a healthy range. In a specific example, the software simulates sagittal IMA and rotational correction without resecting the TMT joint and obtains the height of the head in this "corrected" position. The software can then proceed to the next step of the planning procedure—for example, planning a resection—and use the height of the first metatarsal head as a constraint / parameter for the resection plan.

[0115] Embodiments described herein can further be used to facilitate the generation of simulated weight-bearing computed tomography (WBCT) models. CT imaging data is typically collected while the patient is in a non-weight-bearing position, for example, while the patient is lying down. In contrast, embodiments can be used to create a simulated WBCT model by pairing a 3D model of the patient's human body, for example, one generated according to the embodiments described above, with a representation of the position of the patient's anatomical components when in a weight-bearing position (e.g., generated based on one or more X-rays of the human body, such as the feet, feet in a weight-bearing position).

[0116] An exemplary process for generating a simulated WBCT model is shown in Figures 23 and 24 and is described below. Referring first to Figure 23, the process takes at least one first patient data input and at least one second patient data input (2310). In embodiments, the first patient data input is weight-bearing X-ray image data (WBXR), e.g., two-plane X-ray images / image data, but other forms of radiographs, e.g., multiple digitally reconstructed radiographs, may be used. In embodiments, the second patient data input is conventional (non-weight-bearing) CT image data. However, if, for example, the weight-bearing aspect of the WBCT is defective or inaccurate, WBCT data can also be input into this process.

[0117] From at least one first patient data input, the process determines the load-bearing position of at least one anatomical component (2320). For example, the process determines the load-bearing position of each bone (each of the anatomical components) indicated in the first patient data input when in a three-dimensional weight-bearing ("load") position. In some examples, there may be a display of the angle taken by the WBXR in the upper field of view to account for, for example, not being perfectly positioned relative to the human body in question, but there are no strict requirements for this. A wide range of input fields of view can be supported, and in situations involving unsupported fields of view, for example, if the angular deviation from the expected field of view angle is too large, this can be handled in various semi-automatic or fully automatic ways. For example, an AI model can be trained with such fields of view to learn how to use them.

[0118] The process also generates a location map containing each of at least one anatomical component (2330). In a particular example, this location map is a top view showing the location of each of the anatomical components found in the WBXR. In this embodiment, the anatomical components can be identified / segmented manually or automatically.

[0119] From at least one second patient data input, the process determines the three-dimensional geometric shape of at least one anatomical component (2340). For example, for each anatomical component (e.g., bone) shown in the second patient data input, the 3D geometric shape is determined, including volume, shape, surface topology, etc. From the 3D representations available from the second patient data input, the process generates a 3D model containing the 3D representation of each of the at least one anatomical component (2350). The 3D model is generated by software and can provide the form of a 3D object that the software uses to manipulate, measure, etc., the 3D geometric shapes represented in the second patient data input. In the example, generation (2350) is performed as described above, with reference to path 201a in Figure 2, for example, and the second patient data (e.g., CT data) is segmented manually or automatically to create the model. Note that the precise positioning of the anatomical components (e.g., bone) is not important at this point. The process involves overlaying the 3D model onto a location map (2360) and positioning each of the 3D representations of at least one anatomical component according to the location shown on the location map, for example, by aligning each of the 3D representations of individual anatomical components (e.g., bones) to their corresponding locations based on WBXR (2370). In this embodiment, each 3D anatomical model (e.g., bone model) is positioned relative to the location shown for that bone on the location map, and the ID of the 3D component is matched to the location having a complementary ID.

[0120] Next, the process plans one or more surgical procedures using the positioned 3D model (2380). For example, the plan includes identifying axes, measuring angles, and comparing measurements to normal values. Finally, the process generates one or more patient-specific components (2390) based on the surgical plan (one or more surgical procedures planned from 2380) and the positioned 3D model. Examples of such patient-specific components include guides, implants, and instruments based on the surgical plan.

[0121] Figure 24 illustrates an exemplary process for generating a simulated WBCT model in which a second patient data input, such as conventional (non-weight-bearing) CT image data, is not required. Instead, a first patient data input, such as weight-bearing X-ray image data, more specifically two-plane X-ray / image data as an example, is acquired (2410), although other forms of radiographs may be used. From at least one first patient data input, the process determines the weight-bearing position of at least one anatomical component (2420). For example, the process determines the weight-bearing position of each bone (each of the anatomical components) shown in the first patient data input when in a three-dimensional weight-bearing ("weight-bearing") position. The process also generates a position map in this example that includes each of at least one anatomical component (2430). The generation of the position map is used in this example as in the example above, but is not strictly necessary. More generally, the positions on WBXR are identified and the 3D model components are assigned to the corresponding positions.

[0122] From at least one first patient data input, the process determines the 3D geometric shape of at least one anatomical component (2440). For example, the 3D geometric shape, including volume, shape, surface topology, etc., is determined for each of the anatomical components (e.g., bones) indicated in the first patient data input, and the process then generates a model containing a 3D representation of each of the at least one anatomical component (2450), overlays the 3D model onto a location map (2460), and positions the 3D model by aligning each of the 3D representations of each of the at least one anatomical component to its corresponding position, for example, based on WBXR. In this embodiment, each 3D anatomical model (e.g., bone model) is positioned relative to the position indicated for that bone on the location map, and the ID of the 3D component is matched with the position having a complementary ID.

[0123] Next, the process plans one or more surgical procedures using the positioned 3D model (2480). For example, the plan includes identifying axes, measuring angles, and comparing measurements to normal values. Finally, the process generates one or more patient-specific components based on the surgical plan (one or more surgical procedures planned from 2380) and the positioned 3D model (2490). Examples of such patient-specific components include guides, implants, and instruments based on the surgical plan.

[0124] Further details of the embodiments described herein and additional embodiments are presented within the context of the description of the exemplary surgical planning software ("Software") and its functions. The surgical planning software may be intended for use by healthcare professionals, such as orthopedic surgeons, to assist in characterizing various anatomical structures of the human body, such as the foot and ankle, using three-dimensional mathematical modeling and radiometric measurements. The combined information from the structural model and radiometric measurements can be used for diagnostic and treatment planning purposes. For example, various different medical imaging types, such as X-ray, CT, weight-bearing plain film X-ray, and weight-bearing CT (WBCT), can be used as input to the software.

[0125] The software can be used by orthopedic specialists (and other) healthcare professionals for diagnosis and surgical planning in hospital or clinical settings. The software can provide users with the following and other functions: Visualization reports of measurements of 3D mathematical models and anatomical structures (e.g., feet and ankles). Measurement templates including radiation measurements of anatomical structures in the human body, as Surgical planning involving visualization of anatomical 3D structures, radiographic measurements, and 3D models of orthopedic fixation devices and surgical guides.

[0126] For example, a visualization report containing anatomical structural measurements can be used to diagnose orthopedic healthcare conditions. A surgical planning application, including visualizations of 3D structural models, orthopedic fixation device models, and surgical guide models combined with measurements, can be used to plan treatments and surgeries to correct orthopedic healthcare conditions. The surgical planning application / software output can also be used to design patient-specific orthopedic instruments.

[0127] In some embodiments, the software is provided as a web application without specific hardware requirements, and the software is presented and used via a web browser.

[0128] As a non-limiting example, the software can accept computed tomography (CT) and cone-beam computed tomography (CBCT) three-dimensional (3D) imaging data in DICOM format. The quality of visualization and output from the software may be determined by the quality and resolution of the original DICOM image series. Exemplary CBCT imaging parameters are shown in Table 1.3.

[0129] [Table 1.3]

[0130] The software can accept raw X-ray images. Using dorsal-plantar and lateral views, a three-dimensional (3D) digital model can be constructed from planar X-ray images. Exemplary raw X-ray imaging parameters are shown in Table 2.

[0131] [Table 2]

[0132] Specific DICOM tags conforming to NEMA PS 3.1-3.20 2021e must be present on the images used for analysis. If tags are absent, the DICOM image series is considered invalid, and the software may be unable to use the image series. Exemplary tags are listed in Table 3.

[0133] [Table 3] * ) Required when the image is calibrated

[0134] Examples of software measurement range and accuracy (for the foot / ankle imaging area) are as follows: - Range: ±180°, ±500mm (foot and ankle imaging area) -Accuracy: 0°, 0mm (deterministic automatic image analysis) In this example, the software includes a web user interface component for submitting images for analysis and reviewing output reports, as well as a cloud-based service that provides / outputs analysis, measurements, and surgical planning.

[0135] The following are example specifications for the connection requirements to connect to a cloud service: 1. Protocol: Hypertext Transfer Protocol Secure (HTTPS), 2. Encryption: Transport Layer Security (TLS), 3. API Domain: [Domain specified for the service], 4. Port: 443 (TCP).

[0136] The software may have data management capabilities to download existing case reports, as will be further described below.

[0137] Table 4.1 shows exemplary cybersecurity controls for software.

[0138] [Table 4.1]

[0139] Figure 25 shows an exemplary overall workflow of a surgical plan according to the embodiments described herein, including uploading / providing image data, e.g., DICOM data in the form of X-ray or CT images, automated analysis with case-specific review and adjustment, and patient-specific case reports to the software user. The software workflow can be fully automated from the point of image upload through analysis reporting and surgical plan delivery. The workflow may include, for example, the following steps: 1) User authentication (sign in with email and password), 2) Case management (selection of past cases and download of reports), 3) Initiation of a new case, 4) Image type (postoperative / preoperative) and lateral selection, 5) Start analysis, 6) User verification for analysis, 7) Adjustment of target value (optional) 8) Download the report, 9) Save and send the results.

[0140] Further details of the nine steps mentioned above are as follows:

[0141] Figure 26 presents an exemplary interface for user authentication after opening / loading software (e.g., a web application) in a browser. The user is prompted to enter their email address and password.

[0142] Figure 27 presents an exemplary interface for case management according to the embodiments described herein. Users can start a new case by selecting the New Case button, or select and view / download an existing case under "Your Previous Cases." When an existing case is selected, details are displayed in the case information interface area (in this example), allowing the user to open reports and update analyses.

[0143] Additionally, a "Case in Progress" dialogue area is displayed, indicating when the case was initiated and its status (for example, whether the case analysis is being updated).

[0144] Figures 27-29 show an exemplary interface for starting a new case. When selecting to start a new case, the user selects the medical imaging modality (e.g., between CT and X-ray) and the associated DICOM imaging folder / file (see Figure 27). For two-dimensional (2D) imaging using raw planar X-ray modalities, lateral and dorsal-plantar views are provided by the user. For three-dimensional (3D) imaging, at least one image series is required. The user uploads the DICOM files from the selected folder. For folders containing multiple DICOM image series, all series can be listed in the software with their respective metadata and image previews displayed (see Figures 28 and 29). The user selects the DICOM image series (for 3D workflows) or DICOM images (for 2D workflows) for analysis from the generated list. Loading may take time if the folder contains a large number of DICOM series.

[0145] The workflow then proceeds to image type and lateral selection (see Figure 30), where the user selects the image type (pre-operative or post-operative) and lateral orientation based on a pictogram of a human body (e.g., left or right foot).

[0146] In connection with downloading / viewing the report, the analysis report shown below can be obtained for pre- and post-operative cases and may include the following data (in this example) based on the selected image type.

[0147] [Table 4.2]

[0148] Once the workflow begins the analysis, the software can prompt the user to confirm the preoperative foot position and measurement axis as the analysis progresses, as shown in Figure 31.

[0149] If a possible adduction metatarsal is detected, for example, if the software detects a second tarsometatarsal angle (transverse) greater than a predefined criterion (such as -24°), additional confirmation (shown in Figure 32) may be presented to proceed with (confirm) the Rapidus treatment plan (by pressing the confirm button), or the plan may be rejected if the user does not wish to proceed with the Rapidus treatment plan (by pressing the reject and exit buttons).

[0150] If the user wishes to continue with the Rapidus treatment plan (by selecting the confirmation button in the interface shown in Figure 32), the user is presented with an interface (an example shown in Figure 33) to choose between (i) correcting the adductor metatarsal condition and continuing the analysis, and (ii) continuing the analysis without correcting the adductor metatarsal condition. These options are further explained below.

[0151] Based on user selection, a correction preview can be presented. An exemplary interface for providing a correction preview is shown in Figure 34. Users can view and open the report by selecting the appropriate button, or they can choose to adjust the target values ​​by selecting the appropriate button.

[0152] If the user chooses to adjust the target value, they can enter the new value into an input box (see Figure 35) that may be outlined (in green in this example) or otherwise highlighted to provide the updated target and select the Update Analysis button. As a result, the software updates its analysis and presents a new correction preview (Figure 34).

[0153] If the user confirms that they want to open the report (Figure 34), the workflow proceeds to the report view / save interface, as (partially) shown in Figure 36, where the user can download / save the report by clicking the Save as PDF button, or simply view the report in the current interface.

[0154] An example summary page of the report is shown in Figure 37. The software also allows users to save and send results. For example, analysis from the software can be sent for review outside the software, and can also be saved in a (proprietary) format for later review within the software. Analysis reports can also be saved within the software.

[0155] Figures 38A to 38D show exemplary error messages that may be presented to the software user. Figures 39A to 39C show exemplary warning messages that may be presented to the software user.

[0156] As part of its analysis, the software can perform automated anatomical measurements to ensure a reliable three-dimensional (3D) assessment of the human body, for example, by automatically calculating the distance and angle between bones and specific landmarks. Several principles and processes behind different measurement types, as well as several measurements available for each anatomical region, are presented here.

[0157] The software's automated and accurate bone tissue segmentation and shape analysis allows, for example, the calculation of interosseous angles and distances between clinically relevant landmarks in a patient-specific coordinate system, and to quantify displacements and malformations such as hallux valgus. Effective measurements can be calculated for the foot and ankle region to reveal or identify, for example, forefoot deformities and hallux valgus. The following describes such measurements and the axes, landmarks, and surfaces used in their calculation. Various model types are shown in Figure 40, illustrating examples of color-coded solid bone and transparent overlapping bone (with bone axis representation).

[0158] For foot and ankle measurements, angle measurements can be calculated based on a 2D projection of the 3D axes. The 2D projection plane can be estimated from the patient coordinate system of the imaging device, and the measurements can be indicated with a + or - sign. Meary angles (sagittal direction) use a negative sign for flat feet and a positive sign for pes cavus.

[0159] Referring to Figures 41A-41D, the initial analysis from the analyzer allows for the definition of different axes for each bone category (a-c). Referring to Figure 41B, elongated bones (category a), such as metatarsals and proximal phalanges, use a longitudinal axis. The software can scan the bone and determine its cross-sections at various locations. A weighted center point is calculated for each cross-section, and then robust line fitting is used to find a straight line representing the center point of the cross-section. Referring to Figure 41C and category (b), an example of which is the medial cuneiform bone, the software determines the anterior-posterior axis of the cuneiform bone extending through the navicular-cuneiform articular surface weighted center point and the cuneiform-metatarsal articular surface weighted center point. Referring to Figure 41D, which shows the first metatarsal bone (a), the software determines (i) the longitudinal axis for the elongated bone, and (ii) the distal medial-lateral axis, which is the axis of a virtual cylinder aligned with the distal articular surface of the first metatarsal bone.

[0160] Figures 42A to 42N show exemplary geometric measurements for the foot and ankle region determined by the software. Regarding the intermetatarsal angle, the first-to-second intermetatarsal angle (transverse) refers to the angle between the longitudinal axis of the first metatarsal and the longitudinal axis of the second metatarsal measured in the transverse, as shown in Figure 42A.

[0161] Regarding measurements related to hallux valgus, the hallux valgus angle (transverse) refers to the angle between the longitudinal axis of the first metatarsal bone and the longitudinal axis of the first proximal phalanx, as measured in the transverse plane, as shown in Figure 42B.

[0162] Referring to Figure 42C, the interphalangeal angle (transverse) refers to the angle between the longitudinal axis of the first proximal phalanx and the longitudinal axis of the first distal phalanx, as measured in the transverse plane.

[0163] Referring to Figure 42D, the first metatarsal rotation refers to the angle between the distal medial-lateral axis of the first metatarsal and its projection onto the virtual floor plane, measured in a plane perpendicular to the longitudinal axis of the first metatarsal.

[0164] Referring to Figure 42E, the relative length of the first-second metatarsal bone refers to the distance between the distal points of the longitudinal axis of the first metatarsal bone and the longitudinal axis of the second metatarsal bone, measured along the longitudinal axis of the second metatarsal bone.

[0165] Referring to Figure 42F, the first metatarsal deviation refers to the angle between the longitudinal axis of the first metatarsal bone and the floor surface, measured in the sagittal direction.

[0166] Referring to Figure 42G, the height of the first metatarsal bone refers to the distance between the lowest point of the first metatarsal bone and the floor surface.

[0167] Referring to Figure 42H, the plantar space angle refers to the angle between the distal articular surface of the medial cuneiform bone and the proximal articular surface of the first metatarsal bone, measured in the direction of the proximal surface of the first metatarsal bone.

[0168] Referring to Figure 42I, Meary's angle (sagittal direction) refers to the angle between the longitudinal axis of the talus and the longitudinal axis of the first metatarsal.

[0169] Referring to Figure 42J, the second tarsometatarsal angle (transverse) refers to the angle between the second metatarsal bone and the longitudinal axis of the intermediate cuneiform bone.

[0170] Referring to Figure 42K, the distal metatarsal joint angle refers to the angle between a modified cylinder fitted to the distal articular surface of the first metatarsal and its projection onto a plane perpendicular to the first metatarsal.

[0171] Referring to Figure 42L, the hindfoot moment arm (posterior side) refers to the medial-lateral distance between the longitudinal axis of the tibia and the lowest point of the calcaneus.

[0172] Referring to Figure 42M, the trigonometric ratio (cross-section) refers to a visualization of how the first metatarsal, second metatarsal, and calcaneus are positioned relative to each other. The triangle is composed of three points: the lowest point of the calcaneus, the center of gravity of the distal head of the first metatarsal, and the center of gravity of the distal head of the second metatarsal.

[0173] Referring to Figure 42N, the trigonometric ratio (sagittal direction) also refers to a visualization of how the first metatarsal, second metatarsal, and calcaneus are positioned relative to each other. In this sagittal direction, a triangle is formed by the lowest point of the calcaneus, the lowest point of the distal first metatarsal, and the lowest point of the second metatarsal.

[0174] The software can assist in the automated surgical planning of Lapidus procedures. For example, the software performs a virtual Lapidus arthrodesis (against a digital anatomical model) to correct relevant measurements of interest to their normative values. Measurements used may include the first-to-second metatarsal angle (transverse) and the first metatarsal rotation. Examples of these normative values ​​are shown in Table 5, which outlines the measurements and normative values ​​used in Smart Lapidus correction and the associated procedures. In some examples, these values ​​were calculated from any number of patient WBCT images.

[0175] [Table 5]

[0176] It should be noted that any desired baseline value can be used as the “normative” baseline value used by the software. The normative baseline values ​​shown in Table 5 above are similar to the corresponding measurements shown in various literatures, as shown in Table 6.

[0177] [Table 6]

[0178] Based on the normative reference values ​​used, if the measured first-to-second metatarsal angle (transverse) is smaller than the normative reference value, no correction will be attempted for this angle. Furthermore, the software can perform first metatarsal plantar flexion adjustment to account for first retina shortening that occurs as part of the Rapidus arthrodesis. First metatarsal plantar flexion can be adjusted before performing the Rapidus arthrodesis by setting the height of the first metatarsal head relative to the second metatarsal head to the level indicated by the normative reference value of the sagittal first-to-second metatarsal angle (see Table 5). At this time, the height of the first metatarsal head can be kept constant during the procedure. Since the first retina shortens in the Rapidus procedure, the software may include the following note in the case report: First MT shortening can be compensated with a flat wedge. Consider adjusting the plantar flexion value with the wedge. Thus, the user can restore the original first retina length by adding an implant wedge. The software can create a plan without wedges, and the user can adjust the plan accordingly if they want to add something to compensate for shortening, for example, to reduce plantar flexion of the first metatarsal bone. Thus, the user can adjust the target values ​​for the first-to-second metatarsal angle (transverse), first metatarsal rotation, and plantar flexion correction as needed, and have the software update the plan accordingly.

[0179] The software notifies the user if adduction metatarsal is detected, i.e., if the second tarsometatarsal angle (transverse) is greater than a defined threshold. As a non-restrictive example, the defined threshold is -24°. If the user confirms that they wish to continue the plan despite the possibility of adduction metatarsal, the user chooses whether to continue addressing the adduction metatarsal condition as part of the plan. If the option to address adduction metatarsal is selected, the second and third columns can be rotated so that the second and third tarsometatarsal angles (transverse) match their normative reference values ​​(see Table 5). The rapid arthrodesis is then successfully performed. Note that the correction of the second and third columns is shown for visualization purposes only, and for that purpose, surgical intervention may not be defined. If the option of not addressing adduction of the metatarsal bones is chosen, the first thrombotarsal bones will be overcorrected, meaning that the first-to-second metatarsal angle (transverse) will be corrected to a normative value calculated as if the second metatarsal bone were in its normal position defined by the normative value of the second tarsometatarsal angle (transverse). However, the overcorrection may be limited so that the bones of the first thrombotarsal bones cannot overlap with the bones of the second thrombotarsal bones.

[0180] Furthermore, the software performs hallux valgus correction to show how the first proximal and distal phalanges would look if their relative alignment were corrected to normative reference values ​​for hallux valgus angles (transverse and sagittal) (see Table 5). The hallux valgus correction in the software is for visualization purposes only, and no surgical procedure is defined for this purpose.

[0181] When the interphalangeal angle is large, additional Akin treatment may be required to properly correct the entire first rib. Case reports from the software may include the following note: a large interphalangeal angle may indicate the need for Akin treatment; however, the software may not specify further details of the Akin treatment.

[0182] As mentioned above, the normative reference values ​​used (see Table 5) may be any desired normative reference values. In the example, they are determined using an analyzer included in software that measures a set of weight-bearing cone-beam computed tomography (WBCT) images (e.g., 167) of a normal foot (some left foot, some right foot). The corresponding reference values ​​determined by the software for other reported measurements can be presented in the case report.

[0183] Regarding data management and software architecture, Figure 43 shows a conceptual diagram of secure communication (via HTTPS) between the user / client and the cloud service hosted on the cloud computing platform. As previously mentioned, the software may be provided as a cloud service with a web-based user interface. The user uses the web-based user interface to load DICOM data (as an example) into the software components / modules running on the cloud platform. As part of preprocessing or otherwise, a 2D visualization of the input imaging data can be displayed on the web interface before subsequent calculations begin. Furthermore, the user can provide the software with user-defined parameters for software processing via the interface. Communication between the client's web browser / application and the cloud environment can be conducted via an HTTPS connection protected by a TLS certificate.

[0184] In a cloud environment, the software can decrypt DICOM data (e.g., remove personally identifiable information from it), calculate analytical models and measurements (by an analyzer), store the results as numerical data, and delete the original (uploaded) DICOM data, but retain the decrypted DICOM data and use the measurement results for diagnostic purposes, and perform calculations to present the results to the user on a user interface, for example by providing an analytical report that can be viewed / downloaded via an HTTPS connection.

[0185] Regarding the data flow and process to be performed, one embodiment first includes DICOM image processing provided by a client workstation. For example, DICOM data (or other image data files) are loaded into the client computer system, for example, read from a DICOM reader as an image series. The following information confirmed from the image data can then be displayed: patient name, examination identifier, AC number, comments such as examination date and time and description, series number, date, time, modality, and description. Then, when the user initiates the analysis (starts the analysis), the user is prompted for patient assessment information, such as non-DICOM data about the patient, for example, height, weight, and deformity. After the user enters this patient assessment information, the patient assessment information and image series are sent to a cloud environment for analysis by software running in the cloud.

[0186] In the cloud, based on the client initiating a file transfer via a secure HTTPS connection, the cloud environment's servers / systems receive the image series and deidentify the data (e.g., anonymize it with respect to any patient identification information or other sensitive data that may have been included) before analyzing / processing the image series according to the aspects described herein. The client can monitor this cloud analysis on their web interface. Once the analyzer is ready and the analysis is successful, one or more result files are sent to the client via the secure HTTPS connection. The client receives the result files. At that point, the cloud deletes the original image series files and stores the deidentified image series. The user can save the analysis to their cloud account or other storage device. Meanwhile, the cloud can delete / deallocate the software instance used to process the analysis. The analyzed cases can remain in the cloud, and existing case reports can be downloaded by the user at any time.

[0187] The Rapidus procedure, used as an example to illustrate the software's features, is just one of several different surgical procedures that may potentially be used to repair bunions. As a result, the software may provide features such as software toggles or other settings that allow the user to switch between (i) exploring bunion correction procedures recommended by the software based on human body measurements (e.g., IM angle, rotational deformity, adduction metatarsus, etc.) determined / identified by the software from user-provided image data, and (ii) being presented with an interface that provides the desired quantifiable correction that the user wants to provide to the patient, for example, by the user inputting quantifiable correction values ​​for one or more measured deformities, at which point the software will suggest a “best fit” procedure that achieves the user’s desired quantifiable correction.

[0188] Regarding option (i), the software can classify various ranges of deformity measurements into specific procedures and use the process that best addresses those deformities. The “best” can be identified based on any desired parameters, for example, reoperation rate (lowest), invasiveness (as minimal as possible), and / or any other desired factor. The ranges can be initially generated based on clinical data and then further refined with AI based on patient outcomes. Patient outcomes (i.e., feedback) can be used to train the AI ​​model to appropriately classify deformity measurements into the best procedures to address them. For example, suppose a patient has a 22-degree rotational deformity, an 18-degree IM angle deformity, and no signs of adductor metatarsal malformation. The software can use these measurements to identify a surgical approach to address the deformity.

[0189] As an example, rather than limited to clinical practice, the software can construct the following classifications of deformity ranges for rotational deformities, IM angle deformities, and adduction metatarsal deformities, and associate them with recommended treatments.

[0190] [Table 7.1] TIFF2026516733000011.tif31143

[0191] [Table 7.2]

[0192] For example, the procedures described above may include, but are not limited to, proximal rotational osteotomy of the first metatarsal bone ("PROMO" procedure), diaphyseal metatarsal osteotomy and first metatarsal shift with minimally invasive chamfer screw fixation ("MIS chamfer" procedure), and the Rapidus procedure (described herein).

[0193] Based on the patient's exemplary deformity measurements (rotational deformity 22 degrees, IM angle deformity 18 degrees, no adduction metatarsal sign), the recommended treatment based solely on rotational deformity is Treatment 3, the recommended treatment based solely on IM angle deformity is Treatment 3, and the recommended treatment based solely on adduction metatarsal sign (absence) is one of the two. Note that in such a situation, where there are various (in this case, three) deformity measurements related to the identification of the recommended treatment and various recommendations are generated, any desired prioritization or weighting method can be applied. In one example, rotational deformity takes precedence over IM angle deformity and adduction metatarsal deformity. In other examples, priority can be based on the measurements themselves; for example, if the adduction metatarsal measurement exceeds a certain threshold degree, e.g., 5, the recommendation of Treatment 3 may take precedence over recommendations attributable to either of the other two measurements.

[0194] In this example, we assume the software prioritizes recommendations based solely on rotational deformation. Therefore, the software proposes action 3. The software can note that, optionally, action 3 may also be appropriate as a recommendation based on IM angular deformation.

[0195] The measurement ranges, priorities, and treatments described above are provided purely as examples to illustrate aspects of this specification. Actual ranges, variations considered, and the use of prioritization schemes may vary as needed. It should also be noted that there may be fields or other inputs in which the user (e.g., a physician) provides other information to be considered by the algorithm. Examples include the patient's age, weight, diabetes status (yes / no), previous or concurrent treatments, arthritis status, and / or other conditions, such as those related to appropriate / complex fusion / healing.

[0196] Regarding option (ii) in which an interface is presented to the user that provides a desired quantifiable correction, and the software suggests a “best fit” treatment to achieve the correction, this may be particularly useful in situations where the user, for example, a physician or other medical professional, desires an overcorrection or undercorrection to facilitate another treatment to be performed on an adjacent anatomical site, for example. In option (ii), the user is presented with an interface that identifies one or more desired corrections to the patient’s body for one or more measured deformities. The user can input the corrections themselves, and / or the user can provide the desired outcome (measurements) of such corrections, in which case the software uses this to determine the correction by calculating the difference between the starting / current patient measurements and the desired outcome provided by the user. There may also be fields in which the user inputs or indicates other information that the algorithm considers, such as the patient’s age, weight, diabetes status (yes / no), previous or concurrent treatments, arthritis status, and / or other conditions, such as those related to proper / complex fusion / healing.

[0197] As an example, let's assume the user has identified the following desired orthodontic treatment. - Desired position for correcting rotational deformation: 4 degrees -IM angle deformation desired correction position: 14 degrees - Desired corrective position for adduction of the metatarsal bone: 0 degrees

[0198] Furthermore, let's assume the patient's preoperative / current measurements for the above deformities are 22 degrees (rotational deformity), 18 degrees (IM angle deformity), and 0 degrees (adducted metatarsal deformity). In this case, the software calculates the desired, or "total," correction based on the current measurements and the entered desired measurements, as follows: (i) Rotational deformation: 1. Current measured deformation: 22 degrees 2. Desired correction position: 4 degrees (provided by user) 3. Total correction: 18 degrees (calculated by software) (ii) IM angle deformation 1. Current measured deformation: 18 degrees 2. Desired correction position: 14 degrees (provided by user) 3. Total correction: 4 degrees (calculated by software) (iii) Adductor metatarsal 1. Current measured deformation: 0 2. Desired correction position: 0 degrees (provided by user) 3. Total correction: 0 degrees (calculated by software)

[0199] Once the desired correction is identified, the software can, for example, use the method described above with reference to option (i) to recommend the “best” treatment, based on the range confirmed and defined from the clinical data and / or based on AI, to identify the best treatment for a particular deformity. For example, in the above scenario, the software may recommend the Rapidus treatment as the best (or only) treatment that can provide such a large (18-degree) desired rotational correction, and also addresses the desired 4-degree angle correction. In another example, the software may recommend the PROMO treatment instead of Rapidus. The PROMO treatment can also address the 18-degree rotational correction and the 4-degree angle correction. Also, the PROMO treatment cannot address adduction of the metatarsal bone, but there is no such deformity in this example, and therefore the PROMO treatment is still feasible. In this example, the PROMO treatment may be preferable to the Rapidus treatment for the patient, given that the PROMO treatment is less invasive and is expected to lead to faster progress to postoperative weight-bearing.

[0200] As stated above, the measurement ranges, priorities, and procedures described herein are provided purely as examples to illustrate aspects of this specification. Actual ranges, modifications considered, and the use of prioritization schemes may vary as needed.

[0201] In some embodiments, with reference to, for example, Figures 1 and 23-24, one or more of the processes shown and described herein may include an embodiment of generating a diagnostic report (in addition to or in conjunction with another embodiment), an exemplary excerpt of which is shown in Figure 45. The report 5000 may be generated at least partially by a computer system (e.g., 4400 shown in Figure 44) and may be provided to the physician in paper or hard copy and / or electronic format (for example, for review / approval or for use in treatment).

[0202] Report 5000 may include written or visual instructions, configured to be performed, carried out, achieved, executed, etc., by a physician to facilitate the correction of one or more deformities of an affected human body, for example, deformities identified according to the embodiments described herein, corresponding to one or more devices. For example, as shown in Report 5000, instructional image 5002 is shown adjacent to the corresponding anatomical diagram 5004. Instructional image 5002 shows a part of an affected human body, for example, the first metatarsal bone of a patient, engaged with a part of an device configured to facilitate the manipulation of the first and second metatarsals from a deformed position to a corrected position (in anatomical diagram 5004, the deformed position and the corrected position are indicated by dashed and solid lines, respectively, and the long axis of the first metatarsal bone is shown). The instructional image 5002 is provided to provide visual instructions for the physician to reposition the first metatarsal bone from a deformed position to a corrected position (as shown in anatomical figure 5004) so ​​as to manipulate a portion of the instrument into which the first metatarsal bone is engaged from a first position to a second position, thereby adjusting the intermetatarsal angle between the long axes of the first and second metatarsals.

[0203] Report 5000 also includes an instructional image 5006 and an anatomical diagram 5008 placed adjacent to each other. Instructional image 5006 depicts a part of the affected human body, for example, the patient's first metatarsal bone, engaged with a part of an instrument configured to facilitate the manipulation of the first metatarsal bone from a deformed position to a corrected position (in anatomical diagram 5008, the deformed and corrected positions are indicated by dashed and solid lines, respectively, showing the horizontal / short / medial-lateral axis of the first metatarsal bone). Instructional image 5006 is provided to provide visual instructions for a physician to reposition the first metatarsal bone from a deformed position to a corrected position (as shown in anatomical diagram 5008) by manipulating the part of the instrument with which the first metatarsal bone is engaged from a first position to a second position, and thus rotating / unrotating the first metatarsal bone in the coronal / frontal plane.

[0204] Report 5000 may also include recommended parameters for any operations shown in instruction images 5002 and 5006. For example, Report 5000 may suggest that the operation shown in instruction image 5002 involves closing the intermetatarsal angle by a specific angle measurement based on markings (numbered markings, notches, etc.) indicated on the instrument corresponding to a set angle operation. In another example, Report 5000 may suggest that the operation shown in instruction image 5006 involves unrotating the first metatarsal bone by a specific angle measurement based on markings (numbered markings, notches, etc.) indicated on different parts of the instrument shown therein, corresponding to a set rotation operation.

[0205] In some embodiments, the report 5000 may include additional instructional images and / or corresponding anatomical diagrams illustrating one or more deformities and one or more corrective operations that reposition the human body from a deformed position to a corrected position. Furthermore, the instructional images may include various different instruments configured to facilitate such repositioning of the human body, or may illustrate operations that can be performed manually by a physician.

[0206] A computer system having memory and a processor / processing circuit that communicates with the memory can also be provided, and the computer system is configured to perform any one or more aspects of any of the methods described above.

[0207] Furthermore, a computer program product having a computer-readable storage medium that can be read by a processing circuit and storing instructions for execution by the processing circuit can be provided to perform any one or more of the above-described methods.

[0208] The computer systems described herein may implement surgical planning and procedure modules for incorporating and / or using the embodiments described herein. One or more embodiments of a module, for example, include various submodules. Submodules may be, or include, computer-readable program code (e.g., instructions) in a computer-readable medium such as a persistent storage device (e.g., a disk) and / or a cache (e.g., a cache). The computer-readable medium may be part of a computer program product and may be executed by and / or using one or more computers or devices, such as the computer systems described herein, and / or a processor or its processing circuits.

[0209] The processes described herein may be performed individually or collectively by one or more computer systems, such as one or more computer systems running surgical planning software. Figure 44 shows an example of such a computer system and associated devices incorporating and / or using the embodiments described herein. A computer system may also be referred to herein as a data processing device / system, a computing device / system / node, or simply a computer. A computer system may be based on one or more of a variety of system architectures and / or instruction set architectures, such as those offered by Intel Corporation (Santa Clara, California, USA) or ARM Holdings plc (Cambridge, England, UK).

[0210] FIG. 44 shows a computer system 4400 that communicates with an external device 4412. The computer system 4400 includes one or more processors 4402, such as a central processing unit (CPU) for example. The processor can include functional components used for executing instructions, such as obtaining program instructions from a place like a cache or main memory, decoding the program instructions, executing the program instructions, accessing memory for instruction execution, and writing the results of the executed instructions. The processor 4402 can also include registers used by one or more of the functional components. The computer system 4400 also includes a memory 4404, an input / output (I / O) device 4408, and an I / O interface 4410, which can be coupled to the processor 4402 and to each other via one or more buses and / or other connections. The bus connection represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of various bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA), Micro Channel Architecture (MCA), Enhanced ISA (EISA), Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI).

[0211] Memory 4404 can be, or can include, for example, main memory or system memory (e.g., random access memory) used for executing program instructions, a storage device such as a hard drive, flash media, or optical media, and / or cache memory. Memory 4404 can include a cache, such as a shared cache that can be coupled to a local cache (e.g., L1 cache, L2 cache, etc.) of processor 4402. Further, memory 4404 can be, or can include, at least one computer program product having a set (e.g., at least one) of program modules, instructions, code, etc. configured to perform the functions of the embodiments described herein when executed by one or more processors.

[0212] Memory 4404 can store operating system 4405 and other computer programs 4406, such as one or more computer programs / applications configured to perform the aspects described herein. Specifically, the program / application can include computer-readable program instructions configured to perform the functions of embodiments of the aspects described herein.

[0213] Examples of I / O devices 4408 include, but are not limited to, microphones, speakers, global positioning system (GPS) devices, cameras, lights, accelerometers, gyroscopes, magnetometers, sensors configured to sense light, proximity, heart rate, body temperature and / or ambient temperature, blood pressure, and / or skin resistance, and activity monitors. As shown, the I / O devices can be incorporated into the computer system, but in some embodiments, the I / O devices can be considered external devices (4412) coupled to the computer system via one or more I / O interfaces 4410.

[0214] The computer system 4400 may communicate with one or more external devices 4412 via one or more I / O interfaces 4410. Exemplary external devices include keyboards, pointing devices, displays, and / or any other devices that enable a user to interact with the computer system 4400. Other exemplary external devices include any devices that enable the computer system 4400 to communicate with one or more other computing systems or peripheral devices such as printers. Network interfaces / adapters are exemplary I / O interfaces that enable the computer system 4400 to communicate with one or more networks, such as local area networks (LANs), general-purpose wide area networks (WANs), and / or public networks (e.g., the Internet), and provide communication with other computing devices or systems, storage devices, etc. Ethernet-based interfaces (such as Wi-Fi) and Bluetooth® adapters are just examples of currently available types of network adapters used in computer systems (Bluetooth is a registered trademark of Bluetooth SIG, Inc., Kirkland, Washington, USA).

[0215] Communication between the I / O interface 4410 and the external device 4412 can be conducted via a wired and / or wireless communication link 4411, such as an Ethernet-based wired or wireless connection. Illustrative wireless connections include cellular, Wi-Fi, Bluetooth®, proximity-based, near-field, or other types of wireless connections. More generally, the communication link 4411 may be any suitable wireless and / or wired communication link for transmitting data.

[0216] Certain external devices 4412 may include one or more data storage devices capable of storing one or more programs, one or more computer-readable program instructions, and / or data. The computer system 4400 may include removable / non-removable, volatile / non-volatile computer system storage media and / or be coupled to them and communicate (e.g., as an external device of the computer system). For example, it may include, and / or be coupled to, a magnetic disk drive that reads from and writes to a non-removable non-volatile magnetic medium (typically called a “hard drive”), a magnetic disk drive that reads from and writes to a removable non-volatile optical disk such as a CD-ROM, DVD-ROM, or other optical medium.

[0217] Computer System 4400 can operate in a number of other general-purpose or dedicated computing system environments or configurations. Computer System 4400 can take any of the following forms, well known examples of which include, but are not limited to, personal computer (PC) systems, server computer systems such as messaging servers, mobile devices / computers such as thin clients, thick clients, workstations, laptops, handheld devices, smartphones, tablets, and wearable devices, multiprocessor systems, microprocessor-based systems, telephony devices, network appliances (such as edge appliances), virtualization devices, storage controllers, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0218] Aspects of the present invention may be systems, methods, and / or computer program products, any of which may be configured to perform or facilitate the aspects described herein.

[0219] In some embodiments, aspects of the present invention may take the form of a computer program product that can be embodied as a computer-readable medium. The computer-readable medium may be a tangible storage device / medium on which computer-readable program code / instructions are stored. Exemplary computer-readable mediums include, but are not limited to, electronic, magnetic, optical, or semiconductor storage devices or systems, or any combination thereof. Exemplary embodiments of a computer-readable medium include hard drives or other mass storage devices, electrical connections with wiring, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory such as EPROM or flash memory, optical fibers, portable computer disks / diskettes such as compact disk read-only memory (CD-ROM) or digital versatile disk (DVD), optical storage devices, magnetic storage devices, or any combination thereof. The computer-readable medium may be readable by a processor, processing unit, etc., for retrieving data (e.g., instructions) from the medium for execution. In certain examples, a computer program product is or includes one or more computer-readable media containing / storing computer-readable program code that provides and facilitates one or more embodiments described herein.

[0220] As described above, program instructions contained in or stored in a computer-readable medium can be retrieved and executed by any of the various suitable components of a computer system, such as the processor, to enable the computer system to operate and function in a particular manner. Such program instructions that perform actions to perform, achieve, or facilitate the embodiments described herein may be written in any desired programming language or compiled from written code. In some embodiments, such programming languages ​​include object-oriented and / or procedural programming languages ​​such as C, C++, C#, and Java.

[0221] The program code may include one or more program instructions obtained for execution by one or more processors. Computer program instructions may be provided to one or more processors of one or more computer systems, for example, to manufacture an apparatus that, when the program instructions are executed by one or more processors, performs, achieves, or facilitates aspects of the present invention, such as operations or functions described in the flowcharts and / or block diagrams described herein. Therefore, each block or combination of blocks in the flowcharts and / or block diagrams depicted and described herein can, in some embodiments, be implemented by computer program instructions.

[0222] Accordingly, a method according to the embodiments described herein includes obtaining a three-dimensional (3D) model of an anatomical region of a patient, wherein the anatomical region includes the patient's anatomical features; identifying at least one deformation of the patient's anatomical features, wherein the at least one deformation is identified with respect to a target anatomical value of the patient's anatomical feature, the target anatomical value includes a desired range into which the anatomical measurements should fall; the at least one deformation includes at least one of rotational deformation and angular deformation; and determining at least one plane correction to be made to at least one anatomical structure of the patient using the anatomical measurements.

[0223] In one or more embodiments, at least one deformation includes a plurality of deformations. In one or more embodiments, the plurality of deformations includes angular deformations. In one or more embodiments, the angular deformation includes an anatomical angle between a first anatomical feature and a second anatomical feature of the patient. In one or more embodiments, the first and second anatomical features are the first and second metatarsals of the patient. In one or more embodiments, the plurality of deformations includes rotational deformations.

[0224] In one or more embodiments, rotational deformity includes rotational deformity of the patient's first metatarsal bone. In one or more embodiments, identification of at least one deformity includes selecting a geometric volume and applying the geometric volume to one anatomical feature among the patient's anatomical features presented in a 3D model, determining one measurement among the anatomical measurements using the properties of the geometric volume, which are interpreted as properties of the anatomical feature, and identifying one of the at least one deformity based on the determined measurement. In one or more embodiments, the anatomical feature to which the geometric volume is applied is at least one of (i) a cross-section of a bone, or (ii) an articular surface of a bone.

[0225] In one or more embodiments, the method further includes acquiring imaging data of an anatomical region of a patient, and the acquisition of a 3D model includes generating a 3D model from the acquired imaging data.

[0226] In one or more embodiments, the method further comprises generating specifications for patient-specific hardware that facilitate at least one correction to at least one anatomical structure, based on (i) a representation of the patient's anatomical features provided by a 3D model and (ii) at least one correction determined, the specifications including measurements tailored to the patient based on the representation of the patient's anatomical features provided by the 3D model and at least one correction determined. In one or more embodiments, the patient-specific hardware comprises at least one hardware guide that guides one or more surgical procedures to result in at least one correction to at least one anatomical structure. In one or more embodiments, the at least one hardware guide comprises at least one cutting guide for cutting procedures.

[0227] Methods according to embodiments described herein include, additionally or alternatively, acquiring two-dimensional (2D) imaging data of an anatomical region of a patient, wherein the anatomical region includes the patient's anatomical features; generating a three-dimensional (3D) model of the patient's anatomical region using the 2D imaging data, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features; identifying one or more deformities of the patient's anatomical features, wherein the one or more deformities are shown in the 3D model and identified based on the 3D model with respect to a target anatomical value of the patient's anatomical features, and identifying includes acquiring anatomical measurements based on the patient's anatomical landmarks, such as those shown in the 3D model; comparing the anatomical measurements to a target anatomical value; and determining one or more deformities based on the comparison; and determining at least one correction to be made to at least one anatomical structure of the patient based on the relationship between the anatomical measurements and the target anatomical value.

[0228] In one or more embodiments, the method further includes identifying anatomical landmarks and obtaining anatomical measurements based on the identification of anatomical landmarks. In one or more embodiments, the target anatomical value includes a desired range into which the anatomical measurements should fall, and at least one correction includes one or more corrections made to at least one anatomical structure to produce updated anatomical measurements that fall within the desired range. In one or more embodiments, at least one correction indicates at least one correction location on at least one anatomical structure.

[0229] In one or more embodiments, the method further comprises generating specifications for patient-specific hardware that facilitate at least one correction to at least one anatomical structure, based on (i) a representation of the patient's anatomical features provided by a 3D model and (ii) at least one correction determined, the specifications including measurements tailored to the patient based on the representation of the patient's anatomical features provided by the 3D model and at least one correction determined. In one or more embodiments, the patient-specific hardware comprises at least one hardware guide that guides one or more surgical procedures to result in at least one correction to at least one anatomical structure. In one or more embodiments, the at least one hardware guide comprises at least one cutting guide for cutting procedures.

[0230] In one or more embodiments, the method further comprises generating a visual simulation which graphically presents a transition from a first to a second position of at least one patient's anatomical structure as represented in a 3D model, the second position being a position corresponding to a target anatomical value of the patient's anatomical features. In one or more embodiments, at least one correction is performed by at least one surgical operation, and the visual simulation further comprises one or more simulations of at least one surgical operation on at least one patient's anatomical structure represented in a 3D model, the at least one surgical operation comprising (i) at least one surgical cut, or (ii) at least one of joining or separating one or more instruments.

[0231] Methods according to embodiments described herein include, additionally or alternatively, acquiring two-dimensional (2D) imaging data of a patient's anatomical region, wherein the anatomical region includes the patient's anatomical features, and generating a three-dimensional (3D) model of the patient's anatomical region using the 2D imaging data, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features, and generating the 3D model includes performing anatomical context processing, which includes using an artificial intelligence (AI) model to annotate the contours, edges, or surfaces of the anatomical structures of the anatomical region presented in the 2D imaging data, and performing 2D-to-3D reconstruction to optimize the orientation, placement, and scale of the 3D digital volume modeling the anatomical structures, thereby resulting in a 3D anatomical representation of the anatomical region as a 3D model.

[0232] In one or more embodiments, the 2D imaging data includes two or more images or different fields of view of an anatomical region. In one or more embodiments, the images or different fields of view are radiographs. In one or more embodiments, the AI ​​model performs 2D image segmentation to identify the location of contours, edges, or surfaces of anatomical structures. In one or more embodiments, the image segmentation generates a 2D segmentation, and the method further includes processing the 2D segmentation into a set of 2D points based on determining an anatomical structure landmark for each of the anatomical structures, the anatomical structure landmark including the centroid of the anatomical structure.

[0233] In one or more embodiments, the annotation provides annotations that include contour lines around anatomical structures. In one or more embodiments, the anatomical structure includes bone, and the annotation shows contoured bone on one or more 2D image planes. In one or more embodiments, the method further includes displaying contour lines around anatomical structures on a display, where the contour lines are presented with various graphic characteristics to facilitate identification and distinction between anatomical structures.

[0234] In one or more embodiments, the 2D-to-3D reconstruction includes fitting planes of 2D imaging data depicting an anatomical structure within 3D space to each other, the fitting including iteratively transforming the planes based on a comparative anatomy model to change how the planes are positioned relative to each other, and the fitting providing a 3D anatomical representation of the anatomical region. In one or more embodiments, the 2D imaging data informs a plane transformation that guarantees consistency between the plane transformation and what is shown by the 2D imaging data, by way of an image processing technique that includes one or more of edge detection and gradient spikes. In one or more embodiments, the comparative anatomy model includes a 3D model having a structure with comparative characteristics for the anatomical region, the comparative characteristics including at least one of size, shape, or orientation. In one or more embodiments, the comparative anatomy model is provided with an indication of how the comparative characteristics may vary based on a selected population of anatomical samples. In one or more embodiments, the indication is encoded in the comparative anatomy model.

[0235] In one or more embodiments, the method further includes processing incoming 2D imaging data and building an AI model to provide an annotation thereto, the building including training the AI model with pre-annotated and 2D-segmented images with respect to landmarks of the human body. In one or more embodiments, the landmarks include bone centroids as well as proximal and distal ends of bones. In one or more embodiments, the method further includes building a training data set of samples, the samples including 2D-segmented images. In one or more embodiments, building the training data set includes obtaining images with bilateral symmetry and dividing the images with bilateral symmetry into unilateral images that form part of the samples. In one or more embodiments, building the training data set includes generating additional samples of the training data set by applying at least one of rotation, inversion, translation, and other image operations to existing samples of the training data set.

[0236] Methods according to embodiments described herein include, additionally or alternatively, acquiring two-dimensional (2D) imaging data of an anatomical region of a patient, wherein the anatomical region includes the patient's anatomical features; generating a three-dimensional (3D) model of the patient's anatomical region using the 2D imaging data, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features; identifying one or more deformities of the patient's anatomical features, wherein the one or more deformities are shown in the 3D model and identified based on the 3D model with respect to a target anatomical value of the patient's anatomical feature; acquiring anatomical measurements based on the patient's anatomical landmarks, such that the identification is shown in the 3D model; comparing the anatomical measurements to a target anatomical value; determining the one or more deformities based on the comparison; determining at least one correction to be performed on at least one anatomical structure of the patient based on the relationship between the anatomical measurements and the target anatomical value; and generating a report including instructions for engaging a surgical instrument with at least one anatomical structure of the patient and operating the surgical instrument to perform at least one correction on at least one anatomical structure of the patient.

[0237] A computer system having memory and a processing circuit that communicates with the memory, configured to perform one of the methods described above, and a computer program product having a computer-readable storage medium readable by the processing circuit, which stores instructions to be executed by the processing circuit to perform one of the methods described above.

[0238] Although various embodiments have been described above, these are merely examples.

[0239] The terms used herein are intended to describe only specific embodiments and not to limit them. Where used herein, the singular forms “a,” “an,” and “the” are intended to include the plural form unless the context clearly indicates otherwise. Where used herein, the terms “comprises” and / or “comprising” specify the presence of the described features, integers, processes, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, processes, operations, elements, components, and / or groups thereof.

[0240] All means-plus-function or step-plus-function elements in the following claims are intended to include, if present, any structures, materials, or actions that perform a function in combination with other specifically claimed elements. Descriptions of one or more embodiments are presented for illustrative and explanatory purposes, but are not intended to be exhaustive or limitful to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments have been selected and described to best illustrate various aspects and practical applications, enabling those skilled in the art to understand various embodiments with various modifications suitable for specific applications to be considered.

Claims

1. It is a method, To obtain a three-dimensional (3D) model of a patient's anatomical region, wherein the anatomical region includes the patient's anatomical features. Identifying at least one deformation of the anatomical features of the patient, wherein the at least one deformation is identified with respect to a target anatomical value of the patient's anatomical features, the target anatomical value includes a desired range into which the anatomical measurement should fall, and the at least one deformation includes at least one of rotational deformation and angular deformation, and A method comprising determining, using the aforementioned anatomical measurements, at least one planar correction to be performed on at least one anatomical structure of the patient.

2. The method of claim 1, wherein the at least one modification includes a plurality of modifications.

3. The method of claim 2, wherein the plurality of deformations include the angular deformation.

4. The method of claim 3, wherein the angular deformation includes an anatomical angle between a first anatomical feature and a second anatomical feature of the patient's anatomical features.

5. The method of claim 4, wherein the first and second anatomical features are the first and second metatarsals of the patient.

6. The method of claim 2, wherein the plurality of deformations include the rotational deformation.

7. The method of claim 6, wherein the rotational deformation includes rotational deformation of the patient's first metatarsal bone.

8. The identification of the at least one variation is, Select a geometric volume and apply the geometric volume to one of the anatomical features of the patient presented in the 3D model. Determining one of the anatomical measurements using the characteristics of the geometric volume, which are interpreted as characteristics of the anatomical features, and Identifying one of the at least one deformations based on the determined measurement values, The method of claim 1, including the method of claim 1.

9. The method of claim 8, wherein the anatomical feature to which the geometric feature is applied is at least one of (i) a cross-section of a bone or (ii) an articular surface of a bone.

10. The method of claim 1, further comprising acquiring imaging data of the anatomical region of the patient, wherein the acquisition of the 3D model comprises generating the 3D model from the acquired imaging data.

11. The method of claim 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10, further comprising (i) the representation of the patient's anatomical features provided by the 3D model and (ii) the specification of patient-specific hardware that facilitates the at least one correction performed on the at least one anatomical structure, wherein the specification includes measurements tailored to the patient based on the representation of the patient's anatomical features provided by the 3D model and the at least one correction determined.

12. The method of claim 11, wherein the patient-specific hardware comprises at least one hardware guide that guides one or more surgical procedures to result in the at least one correction performed on the at least one anatomical structure.

13. The method of claim 12, wherein the at least one hardware guide comprises at least one cutting guide for a cutting procedure.

14. A computer system, memory, and The computer system is configured to perform a method, comprising a processing circuit that communicates with the memory, and the method is To obtain a three-dimensional (3D) model of a patient's anatomical region, wherein the anatomical region includes the patient's anatomical features. Identifying at least one deformation of the anatomical features of the patient, wherein the at least one deformation is identified with respect to a target anatomical value of the patient's anatomical features, the target anatomical value includes a desired range into which the anatomical measurement should fall, and the at least one deformation includes at least one of rotational deformation and angular deformation, and A computer system comprising determining, using the aforementioned anatomical measurements, at least one planar correction to be performed on at least one anatomical structure of the patient.

15. The computer system of claim 14, wherein the at least one modification includes a plurality of modifications.

16. The computer system of claim 15, wherein the plurality of deformations include the angular deformation.

17. The computer system of claim 16, wherein the angular deformation includes an anatomical angle between a first anatomical feature and a second anatomical feature of the patient's anatomical features.

18. The computer system of claim 17, wherein the first and second anatomical features are the first and second metatarsals of the patient.

19. The computer system of claim 15, wherein the plurality of deformations include the rotational deformation.

20. The computer system of claim 19, wherein the rotational deformation includes rotational deformation of the patient's first metatarsal bone.

21. The identification of the at least one variation is, Select a geometric volume and apply the geometric volume to one of the anatomical features of the patient presented in the 3D model. Determining one of the anatomical measurements using the characteristics of the geometric volume, which are interpreted as characteristics of the anatomical features, and Identifying one of the at least one deformations based on the determined measurement values, A computer system according to claim 20, including the above.

22. The computer system of claim 21, wherein the anatomical feature to which the geometric feature is applied is at least one of (i) a cross-section of a bone or (ii) an articular surface of a bone.

23. The computer system of claim 14, wherein the method further comprises acquiring imaging data of the anatomical region of the patient, and the acquisition of the 3D model comprises generating the 3D model from the acquired imaging data.

24. The computer system of claim 14, 15, 16, 17, 18, 19, 20, 21, 22, or 23, wherein the method further comprises (i) the representation of the patient's anatomical features provided by the 3D model and (ii) the determined at least one correction, the computer system of

25. The computer system of claim 24, wherein the patient-specific hardware comprises at least one hardware guide that guides one or more surgical procedures to result in the at least one correction performed on the at least one anatomical structure.

26. The computer system of claim 25, wherein the at least one hardware guide comprises at least one cutting guide for a cutting procedure.

27. A computer program product, The method comprises a computer-readable storage medium that is readable by a processing circuit and stores instructions executed by the processing circuit to perform the method, and the method is To obtain a three-dimensional (3D) model of a patient's anatomical region, wherein the anatomical region includes the patient's anatomical features. Identifying at least one deformation of the anatomical features of the patient, wherein the at least one deformation is identified with respect to a target anatomical value of the patient's anatomical features, the target anatomical value includes a desired range into which the anatomical measurement should fall, and the at least one deformation includes at least one of rotational deformation and angular deformation, and A computer program product comprising determining, using the aforementioned anatomical measurements, at least one planar correction to be performed on at least one anatomical structure of the patient.

28. The computer program product of claim 27, wherein the at least one variation includes a plurality of variations.

29. The computer program product of claim 28, wherein the plurality of deformations include the angular deformation.

30. The computer program product of claim 29, wherein the angular deformation includes an anatomical angle between a first anatomical feature and a second anatomical feature of the patient's anatomical features.

31. The computer program product of claim 30, wherein the first and second anatomical features are the first and second metatarsals of the patient.

32. The computer program product of claim 28, wherein the plurality of deformations include the rotational deformation.

33. The computer program product of claim 32, wherein the rotational deformation includes rotational deformation of the patient's first metatarsal bone.

34. The identification of the at least one variation is, Select a geometric volume and apply the geometric volume to one of the anatomical features of the patient presented in the 3D model. Determining one of the anatomical measurements using the characteristics of the geometric volume, which are interpreted as characteristics of the anatomical features, and Identifying one of the at least one deformations based on the determined measurement values, A computer program product according to claim 27, including the computer program product of claim 27.

35. The computer program product of claim 34, wherein the anatomical feature to which the geometric feature is applied is at least one of (i) a cross-section of a bone or (ii) an articular surface of a bone.

36. The computer program product of claim 27, wherein the method further comprises acquiring imaging data of the anatomical region of the patient, and the acquisition of the 3D model comprises generating the 3D model from the acquired imaging data.

37. The computer program product of claim 27, 28, 29, 30, 31, 32, 33, 34, 35, or 36, wherein the method further comprises (i) the representation of the patient's anatomical features provided by the 3D model and (ii) the determined at least one correction, the specification comprising measurements adjusted to the patient based on the representation of the patient's anatomical features provided by the 3D model and the determined at least one correction.

38. The computer program product of claim 37, wherein the patient-specific hardware comprises at least one hardware guide that guides one or more surgical procedures to result in the at least one correction performed on the at least one anatomical structure.

39. The computer program product of claim 38, wherein the at least one hardware guide comprises at least one cutting guide for a cutting procedure.

40. It is a method, Acquiring two-dimensional (2D) imaging data of a patient's anatomical region, wherein the acquired anatomical region includes the patient's anatomical features. Using the aforementioned 2D imaging data, generate a three-dimensional (3D) model of the anatomical region of the patient, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features. Identifying one or more deformities of the anatomical features of the patient, wherein the one or more deformities are shown in the 3D model, and are identified based on the 3D model with respect to target anatomical values ​​of the patient's anatomical features, and the identification is Obtaining anatomical measurements based on the patient's anatomical landmarks, as shown in the 3D model; Comparing the aforementioned anatomical measurements with the aforementioned target anatomical values, Identifying, including determining one or more of the above-mentioned modifications based on the above-mentioned comparison, A method comprising determining at least one correction to be performed on at least one anatomical structure of the patient, based on the relationship between the anatomical measurement and the target anatomical value.

41. The method of claim 40, further comprising identifying the anatomical landmark and obtaining the anatomical measurement based on the identification of the anatomical landmark.

42. The method of claim 40, wherein the target anatomical value includes a desired range into which the anatomical measurement should fall, and the at least one correction includes one or more corrections made to the at least one anatomical structure to produce an updated anatomical measurement that falls within the desired range.

43. The method of claim 40, wherein the at least one correction indicates at least one corrective position of the at least one anatomical structure.

44. The method of claim 40, 41, 42, or 43, further comprising (i) the representation of the patient's anatomical features provided by the 3D model and (ii) the specification of patient-specific hardware that facilitates the at least one correction performed on the at least one anatomical structure, wherein the specification includes measurements tailored to the patient based on the representation of the patient's anatomical features provided by the 3D model and the at least one correction determined.

45. The method of claim 44, wherein the patient-specific hardware comprises at least one hardware guide for guiding one or more surgical procedures to result in the at least one correction performed on the at least one anatomical structure.

46. The method of claim 45, wherein the at least one hardware guide comprises at least one cutting guide for a cutting procedure.

47. The method of claim 40, 41, 42, or 43, further comprising generating a visual simulation, the visual simulation graphically presenting a transition from a first to a second position of the anatomical structure of the at least one patient, as represented in the 3D model, the second position being a position that coincides with the target anatomical value of the patient's anatomical feature.

48. The method of claim 47, wherein the at least one correction is performed by at least one surgical procedure, and the visual simulation further comprises one or more simulations of the at least one surgical procedure on the anatomical structure of the at least one patient represented in the 3D model, and the at least one surgical procedure comprises (i) at least one surgical cut, or (ii) joining or separating one or more instruments.

49. A computer system, memory, and The computer system is configured to perform a method, comprising a processing circuit that communicates with the memory, and the method is Acquiring two-dimensional (2D) imaging data of a patient's anatomical region, wherein the acquired anatomical region includes the patient's anatomical features. Using the aforementioned 2D imaging data, generate a three-dimensional (3D) model of the anatomical region of the patient, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features. Identifying one or more deformities of the anatomical features of the patient, wherein the one or more deformities are shown in the 3D model, and are identified based on the 3D model with respect to target anatomical values ​​of the patient's anatomical features, and the identification is Obtaining anatomical measurements based on the patient's anatomical landmarks, as shown in the 3D model; Comparing the aforementioned anatomical measurements with the aforementioned target anatomical values, Identifying, including determining one or more of the above-mentioned modifications based on the above-mentioned comparison, A computer system comprising determining at least one correction to be performed on at least one anatomical structure of the patient, based on the relationship between the anatomical measurement and the target anatomical value.

50. The computer system of claim 49, wherein the method further comprises identifying the anatomical landmark and obtaining the anatomical measurement based on the identification of the anatomical landmark.

51. The computer system of claim 49, wherein the target anatomical value includes a desired range into which the anatomical measurement should fall, and the at least one correction includes one or more corrections made to the at least one anatomical structure to produce an updated anatomical measurement that falls within the desired range.

52. The computer system of claim 49, wherein the at least one correction indicates at least one corrected position of the at least one anatomical structure.

53. The computer system of claim 49, 50, 51, or 52, wherein the method further comprises (i) the representation of the patient's anatomical features provided by the 3D model and (ii) the determined at least one correction, the computer system of

54. The computer system of claim 53, wherein the patient-specific hardware comprises at least one hardware guide that guides one or more surgical procedures to result in the at least one correction performed on the at least one anatomical structure.

55. The computer system of claim 54, wherein the at least one hardware guide comprises at least one cutting guide for a cutting procedure.

56. The computer system of claim 49, 50, 51, or 52, further comprising generating a visual simulation, the visual simulation graphically presenting a transition from a first to a second position of the anatomical structure of at least one patient, as represented in the 3D model, the second position being a position that coincides with the target anatomical value of the patient's anatomical feature.

57. The computer system of claim 56, wherein the at least one correction is performed by at least one surgical procedure, and the visual simulation further includes one or more simulations of the at least one surgical procedure on the anatomical structure of the at least one patient represented in the 3D model, and the at least one surgical procedure includes (i) at least one surgical cut, or (ii) joining or separating one or more instruments.

58. A computer program product, The method comprises a computer-readable storage medium that is readable by a processing circuit and stores instructions executed by the processing circuit to perform the method, and the method is Acquiring two-dimensional (2D) imaging data of a patient's anatomical region, wherein the acquired anatomical region includes the patient's anatomical features. Using the aforementioned 2D imaging data, generate a three-dimensional (3D) model of the anatomical region of the patient, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features. Identifying one or more deformities of the anatomical features of the patient, wherein the one or more deformities are shown in the 3D model, and are identified based on the 3D model with respect to target anatomical values ​​of the patient's anatomical features, and the identification is Obtaining anatomical measurements based on the patient's anatomical landmarks, as shown in the 3D model; Comparing the aforementioned anatomical measurements with the aforementioned target anatomical values, Identifying, including determining one or more of the above-mentioned modifications based on the above-mentioned comparison, A computer program product comprising determining at least one correction to be performed on at least one anatomical structure of the patient, based on the relationship between the anatomical measurement and the target anatomical value.

59. The computer program product of claim 58, wherein the method further comprises identifying the anatomical landmark and obtaining the anatomical measurement based on the identification of the anatomical landmark.

60. The computer program product of claim 58, wherein the target anatomical value includes a desired range into which the anatomical measurement should fall, and the at least one correction includes one or more corrections made to the at least one anatomical structure to produce an updated anatomical measurement that falls within the desired range.

61. The computer program product of claim 58, wherein the at least one correction indicates at least one corrected position of the at least one anatomical structure.

62. The computer program product of claim 58, 59, 60, or 61, wherein the method further comprises (i) the representation of the patient's anatomical features provided by the 3D model and (ii) the at least one determined correction, for generating patient-specific hardware specifications that facilitate the at least one correction performed on the at least one anatomical structure, the specifications including measurements adjusted to the patient based on the representation of the patient's anatomical features provided by the 3D model and the at least one determined correction.

63. The computer program product of claim 62, wherein the patient-specific hardware comprises at least one hardware guide that guides one or more surgical procedures to result in the at least one correction performed on the at least one anatomical structure.

64. The computer program product of claim 63, wherein the at least one hardware guide comprises at least one cutting guide for a cutting procedure.

65. The computer program product of claim 58, 59, 60, or 61, further comprising generating a visual simulation, the visual simulation graphically presenting a transition from a first to a second position of the anatomical structure of at least one patient, as represented in the 3D model, the second position being a position that coincides with the target anatomical value of the patient's anatomical feature.

66. The computer program product of claim 65, wherein the at least one correction is performed by at least one surgical procedure, and the visual simulation further includes one or more simulations of the at least one surgical procedure on the anatomical structure of the at least one patient represented in the 3D model, and the at least one surgical procedure includes (i) at least one surgical cut, or (ii) joining or separating one or more instruments.

67. It is a method, To acquire two-dimensional (2D) imaging data of a patient's anatomical region, wherein the acquired anatomical region includes the patient's anatomical features, and The process involves generating a three-dimensional (3D) model of the anatomical region of the patient using the aforementioned 2D imaging data, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features. The above generation is Performing anatomical context processing, wherein the anatomical context processing includes using an artificial intelligence (AI) model to annotate the contours, edges, or surfaces of anatomical structures in the anatomical region presented in the 2D imaging data, and A method comprising performing a 2D-to-3D reconstruction to optimize the orientation, position, and scale of a 3D digital volume that models the anatomical structure, thereby yielding a 3D anatomical representation of the anatomical region as a 3D model.

68. The method of claim 67, wherein the 2D imaging data includes two or more images or different fields of view of the anatomical region.

69. The method of claim 68, wherein the aforementioned image or different field of view is an X-ray.

70. The method of claim 68, wherein the AI ​​model performs two-dimensional image segmentation to identify the location of the contour, edge, or surface of the anatomical structure.

71. The method of claim 70, wherein the image segmentation generates a 2D segmentation, and the method further comprises processing the 2D segmentation into a set of 2D points based on determining a landmark of the anatomical structure for each of the anatomical structures, wherein the landmark of the anatomical structure includes the centroid of the anatomical structure.

72. The method of claim 67, wherein the annotation provides annotation including contour lines around the anatomical structure.

73. The method of claim 72, wherein the anatomical structure includes bone, and the annotation shows the bone outlined on one or more 2D image planes.

74. The method of claim 72, further comprising displaying the contour lines surrounding the anatomical structures on a display, wherein the contour lines are presented with various graphic characteristics to facilitate identification and distinction between the anatomical structures.

75. The method of claim 67, wherein the 2D-to-3D reconstruction comprises fitting planes of 2D imaging data depicting the anatomical structure to each other in 3D space, the fitting comprises iteratively transforming the planes based on a comparative anatomical model to change how the planes are positioned relative to each other, and the fitting provides the 3D anatomical representation of the anatomical region.

76. The method of claim 75, wherein the 2D imaging data is indicated by an image processing technique including edge detection and gradient spikes to ensure consistency between the plane transformation and what is shown by the 2D imaging data.

77. The method of claim 75, wherein the comparative anatomical model includes a 3D model having a structure having comparative characteristics for the anatomical region, the comparative characteristics including at least one of size, shape, or orientation.

78. The method of claim 77, wherein the comparative anatomical model is given an index of how the comparative characteristics may change based on a selected population of anatomical samples.

79. The method of claim 78, wherein the index is encoded in the comparative anatomical model.

80. The method of claim 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78 or 79, further comprising processing incoming 2D imaging data and constructing the AI ​​model to provide annotations thereto, wherein the construction includes training the AI ​​model with 2D segmented images that have been pre-annotated with respect to human body landmarks.

81. The method of claim 80, wherein the markers include the bone's center of gravity and the proximal and distal ends of the bone.

82. The method of claim 80, further comprising constructing a training dataset of samples, wherein the samples include the 2D segmented images.

83. The method of claim 82, wherein the construction of the training dataset comprises acquiring a bilateral image and splitting the bilateral image into one-sided images that form part of the sample.

84. The method of claim 82, wherein the construction of the training dataset includes generating additional samples of the training dataset by applying at least one of rotation, inversion, translation, and other image manipulations to an existing sample of the training dataset.

85. A computer system, memory, and The computer system comprises a processing circuit that communicates with the memory, and is configured to perform a method, and the method is To acquire two-dimensional (2D) imaging data of a patient's anatomical region, wherein the acquired anatomical region includes the patient's anatomical features, and The process includes generating a three-dimensional (3D) model of the anatomical region of the patient using the aforementioned 2D imaging data, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features, The above generation is Performing anatomical context processing, which includes using an artificial intelligence (AI) model to annotate the contours, edges, or surfaces of the anatomical structures of the anatomical region presented in the 2D imaging data, and A computer system that includes performing a 2D-to-3D reconstruction to optimize the orientation, position, and scale of a 3D digital volume that models the anatomical structure, thereby resulting in a 3D anatomical representation of the anatomical region as a 3D model.

86. The computer system of claim 85, wherein the 2D imaging data includes two or more images or different fields of view of the anatomical region.

87. The computer system of claim 86, wherein the aforementioned image or different field of view is an X-ray photograph.

88. The computer system of claim 86, wherein the AI ​​model performs two-dimensional image segmentation to identify the location of the contour, edge, or surface of the anatomical structure.

89. The computer system of claim 88, wherein the image segmentation generates a 2D segmentation, and the method further comprises processing the 2D segmentation into a set of 2D points based on determining a marker of the anatomical structure for each of the anatomical structures, the marker of the anatomical structure includes the centroid of the anatomical structure.

90. The computer system of claim 85, wherein the annotation provides annotations including contour lines around the anatomical structure.

91. The computer system of claim 90, wherein the anatomical structure includes bone, and the annotation shows the bone outlined on one or more 2D image planes.

92. The computer system of claim 90, wherein the method further includes displaying the contour lines surrounding the anatomical structures on a display, the contour lines being presented with various graphic characteristics to facilitate identification and distinction between the anatomical structures.

93. The computer system of claim 85, wherein the 2D-to-3D reconstruction includes fitting planes of 2D imaging data depicting the anatomical structure to each other in 3D space, the fitting includes iteratively transforming the planes based on a comparative anatomical model to change how the planes are positioned relative to each other, and the fitting provides the 3D anatomical representation of the anatomical region.

94. The computer system of claim 93, wherein the 2D imaging data is indicated by an image processing technique including edge detection and gradient spikes to ensure consistency between the plane transformation and what is shown by the 2D imaging data.

95. The computer system of claim 93, wherein the comparative anatomical model includes a 3D model having a structure having comparative characteristics for the anatomical region, the comparative characteristics including at least one of size, shape, or orientation.

96. The computer system of claim 95, wherein the comparative anatomical model is given an index of how the comparative characteristics may change based on a selected population of anatomical samples.

97. The computer system of claim 96, wherein the index is encoded in the comparative anatomical model.

98. The computer system of claim 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 or 97, wherein the method further comprises processing incoming 2D imaging data and constructing the AI ​​model to provide annotations thereto, the construction comprising training the AI ​​model with 2D segmented images that have been pre-annotated with respect to human body landmarks.

99. The computer system of claim 98, wherein the markers include the bone's center of gravity and the proximal and distal ends of the bone.

100. The computer system of claim 98, wherein the method further comprises constructing a training dataset of samples, the samples comprising the 2D segmented images.

101. The computer system of claim 100, wherein the construction of the training dataset includes acquiring a bilateral image and dividing the bilateral image into one-sided images that form a portion of the sample.

102. The computer system of claim 100, wherein the construction of the training dataset includes generating additional samples of the training dataset by applying at least one of rotation, inversion, translation, and other image manipulations to existing samples of the training dataset.

103. A computer program product, The method comprises a computer-readable storage medium that is readable by a processing circuit and stores instructions executed by the processing circuit to perform the method, and the method is To acquire two-dimensional (2D) imaging data of a patient's anatomical region, wherein the acquired anatomical region includes the patient's anatomical features, and The process includes generating a three-dimensional (3D) model of the anatomical region of the patient using the aforementioned 2D imaging data, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features, The above generation is Performing anatomical context processing, wherein the anatomical context processing includes using an artificial intelligence (AI) model to annotate the contours, edges, or surfaces of the anatomical structures of the anatomical region presented in the 2D imaging data, and A computer program product that includes performing a 2D-to-3D reconstruction to optimize the orientation, position, and scale of a 3D digital volume that models the anatomical structure, thereby resulting in a 3D anatomical representation of the anatomical region as a 3D model.

104. The computer program product of claim 103, wherein the 2D imaging data includes two or more images or different fields of view of the anatomical region.

105. The computer program product of claim 104, wherein the aforementioned image or different field of view is an X-ray photograph.

106. The computer program product of claim 104, wherein the AI ​​model performs two-dimensional image segmentation to identify the location of the contour, edge, or surface of the anatomical structure.

107. The computer program product of claim 106, wherein the image segmentation generates a 2D segmentation, and the method further comprises processing the 2D segmentation into a set of 2D points based on determining a marker of the anatomical structure for each of the anatomical structures, the marker of the anatomical structure includes the centroid of the anatomical structure.

108. The computer program product of claim 103, wherein the annotation provides annotation including contour lines around the anatomical structure.

109. The computer program product of claim 108, wherein the anatomical structure includes bone, and the annotation shows the bone outlined on one or more 2D image planes.

110. The computer program product of claim 108, wherein the method further includes displaying the contour lines surrounding the anatomical structures on a display, the contour lines being presented with various graphic characteristics to facilitate identification and distinction between the anatomical structures.

111. The computer program product of claim 103, wherein the 2D-to-3D reconstruction includes fitting planes of 2D imaging data depicting the anatomical structure to each other in 3D space, the fitting includes iteratively transforming the planes based on a comparative anatomical model to change how the planes are positioned relative to each other, and the fitting provides the 3D anatomical representation of the anatomical region.

112. The computer program product of claim 111, wherein the 2D imaging data is indicated by an image processing technique including edge detection and gradient spikes, which ensures consistency between the plane transformation and what is shown by the 2D imaging data.

113. The computer program product of claim 111, wherein the comparative anatomical model includes a 3D model having a structure having comparative characteristics for the anatomical region, the comparative characteristics including at least one of size, shape, or orientation.

114. The computer program product of claim 113, wherein the comparative anatomical model is given an index of how the comparative characteristics may change based on a selected population of anatomical samples.

115. The computer program product of claim 114, wherein the index is encoded in the comparative anatomical model.

116. The computer program product of claim 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, or 115, wherein the method further comprises processing incoming 2D imaging data and constructing the AI ​​model to provide annotations thereto, the construction comprising training the AI ​​model with 2D segmented images that have been pre-annotated with respect to human body landmarks.

117. The computer program product of claim 116, wherein the aforementioned markers include the bone's center of gravity and the proximal and distal ends of the bone.

118. The computer program product of claim 116, wherein the method further comprises constructing a training dataset of samples, the samples comprising the 2D segmented images.

119. The computer program product of claim 118, wherein the construction of the training dataset includes acquiring a bilateral image and splitting the bilateral image into one-sided images that form part of the sample.

120. The computer program product of claim 118, wherein the construction of the training dataset includes generating additional samples of the training dataset by applying at least one of rotation, inversion, translation, and other image manipulations to existing samples of the training dataset.

121. It is a method, Acquiring two-dimensional (2D) imaging data of a patient's anatomical region, wherein the acquired anatomical region includes the patient's anatomical features. Using the aforementioned 2D imaging data, generate a three-dimensional (3D) model of the anatomical region of the patient, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features. Identifying one or more deformities of the anatomical features of the patient, wherein the one or more deformities are shown in the 3D model and are identified based on the 3D model with respect to target anatomical values ​​of the patient's anatomical features, Obtaining anatomical measurements based on the patient's anatomical landmarks, as shown in the 3D model; Comparing the aforementioned anatomical measurements with the aforementioned target anatomical values, Identifying, including determining one or more of the above-mentioned modifications based on the above-mentioned comparison, Based on the relationship between the anatomical measurements and the target anatomical values, determine at least one correction to be performed on at least one anatomical structure of the patient, and A method comprising: engaging a surgical instrument with the at least one anatomical structure of the patient; and generating a report containing instructions for operating the surgical instrument to perform the at least one correction on the at least one anatomical structure of the patient.

122. A computer system, memory, and The computer system is configured to perform a method, comprising a processing circuit that communicates with the memory, and the method is Acquiring two-dimensional (2D) imaging data of a patient's anatomical region, wherein the acquired anatomical region includes the patient's anatomical features. To generate a three-dimensional (3D) model of the anatomical region of a patient using 2D imaging data, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features. Identifying one or more deformities of the anatomical features of the patient, wherein the one or more deformities are shown in the 3D model and are identified based on the 3D model with respect to target anatomical values ​​of the patient's anatomical features, and the identification is Obtaining anatomical measurements based on the patient's anatomical landmarks, as shown in the 3D model; Comparing the aforementioned anatomical measurements with the aforementioned target anatomical values, Identifying, including determining one or more of the above-mentioned modifications based on the above-mentioned comparison, Based on the relationship between the anatomical measurements and the target anatomical values, determine at least one correction to be performed on at least one anatomical structure of the patient, and A computer system comprising generating a report containing instructions for operating a surgical instrument to engage with the at least one anatomical structure of the patient and to perform the at least one correction on the at least one anatomical structure of the patient.

123. A computer program product, The method comprises a computer-readable storage medium that is readable by a processing circuit and stores instructions executed by the processing circuit to perform the method, and the method is Acquiring two-dimensional (2D) imaging data of a patient's anatomical region, wherein the acquired anatomical region includes the patient's anatomical features. Using the aforementioned 2D imaging data, generate a three-dimensional (3D) model of the anatomical region of the patient, wherein the 3D model is unique to the patient and provides a 3D representation of the patient's anatomical features. Identifying one or more deformities of the anatomical features of the patient, wherein the one or more deformities are shown in the 3D model and are identified based on the 3D model with respect to target anatomical values ​​of the patient's anatomical features, and the identification is Obtaining anatomical measurements based on the patient's anatomical landmarks, as shown in the 3D model; Comparing the aforementioned anatomical measurements with the aforementioned target anatomical values, Identifying, including determining one or more of the above-mentioned modifications based on the above-mentioned comparison, Based on the relationship between the anatomical measurements and the target anatomical values, determine at least one correction to be performed on at least one anatomical structure of the patient, and A computer program product that includes generating a report containing instructions for operating a surgical instrument to engage the surgical instrument with the at least one anatomical structure of the patient and to perform the at least one correction on the at least one anatomical structure of the patient.

Images