Radiographic image generation using generative artificial intelligence models

Abstract

Generative artificial intelligence (AI) models are used to generate synthetic images from one or more radiographic images of a subject. As one example, a radiographic image of a subject in a first view can be converted to a view-corrected radiographic image of the subject in a second view. As another example, two or more radiographic images of a subject can be used to generate a three-dimensional (3D) anatomical model, such as a 3D model of bony structures in the subject. To ensure consistency of the synthetic images with the input radiographic images, a radiographic fingerprint is generated from the input images and used to condition the generative models. The radiographic fingerprint may be an embedding vector that encodes unique anatomical features depicting in the input images.

Description

RADIOGRAPHIC IMAGE GENERATION USING GENERATIVE ARTIFICIAL INTELLIGENCE MODELSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 583,044, filed on September 15, 2023, and entitled “RADIOGRAPHIC IMAGE GENERATION USING GENERATIVE ARTIFICIAL INTELLIGENCE MODELS ’ which is herein incorporated by reference in its entirety.BACKGROUND

[0002] Deep learning (DL) is a rapidly growing field of artificial intelligence research in medicine that has the potential to improve patient care. DL models are created by training neural networks on large datasets of imaging, text, or tabular data. These models can be used to perform various tasks with medical imaging data, including classifying patient diagnoses or prognoses, identifying and segmenting objects of interest, and generating synthetic data. While generative adversarial networks (GANs) have long been considered advantageous DL models for generating synthetic images, they have recently been surpassed by another group of DL models called denoising diffusion probabilistic models (DDPMs). Although both GANs and DDPMs have a wide range of potential applications, DDPMs have been shown to produce images of higher quality and greater diversity, albeit at the cost of slower processing speed.

[0003] Like other DL models, DDPMs are trained by being exposed to a prepared set of images known as training data. The simplest versions of DDPMs are trained without supervision, meaning that no labels are required beyond the imaging data itself. However, DDPMs can also be used for supervised or conditional training. For instance a generated image can be conditioned based on demographic information (age, sex, BMI), image characteristics (view, presence of prosthesis, ty pe of prosthesis), and images (pre-operative image to create a post-operative radiograph).SUMMARY OF THE DISCLOSURE

[0004] The present disclosure addresses the aforementioned drawbacks by providing a method for generating a radiographic fingerprint from a medical image of a subject. The method includes accessing a medical image of a subject with a computer system; accessing a machine learning model with the computer system, where the machine learning model has beentrained on training data to encode anatomical features in medical image data as an embedding vector; and generating a radiographic fingerprint for the medical image of the subject by inputting the medical image to the machine learning model using the computer system, thereby generating an embedding that encodes anatomical features of the subject in the medical image as an output, where the embedding is stored by the computer system as the radiographic fingerprint.

[0005] It is another aspect of the present disclosure to provide a method for generating a view-corrected radiographic image of a subject. The method includes accessing a radiographic image with a computer system, where the radiographic image depicts the subject in a first view; accessing a generative model with the computer system, where the generative model has been trained on training data to convert a medical image from a first projection plane to a second projection plane; and applying the radiographic image to the generative model using the computer system, generating as an output a view-corrected radiographic image that depicts the subject in a second view that is different from the first view.

[0006] It is another aspect of the present disclosure to provide a method for generating a three-dimensional (3D) model of a portion of a subject’s anatomy from at least two radiographic images of the subject. The method includes accessing radiographic image data with a computer system, where the radiographic image data may include at least two radiographic images of the subject; accessing a generative model with the computer system, where the generative model has been trained on training data to generate maximum intensity projections (MIPs) from a radiographic image; inputting the radiographic image data to the machine learning model, generating a plurality7of MIPs corresponding to a plurality of spatial positions; and generating the 3d model with the computer system by constructing a neural radiance field (NeRF) model from the plurality of MIPs, where the nerf model represents the 3D model of the portion of the subject’s anatomy

[0007] The foregoing and other aspects and advantages of the present disclosure will appear from the following description. In the description, reference is made to the accompanying drawings that form a part hereof, and in which there is shown by way of illustration one or more embodiments. These embodiments do not necessarily represent the full scope of the invention, however, and reference is therefore made to the claims and herein for interpreting the scope of the invention.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 illustrates an example workflow for generating radiographic fingerprints, view-correction radiographic images, and 3D anatomical models in accordance with examples described in the present disclosure.

[0009] FIG. 2 is a flowchart setting forth the steps of an example method for generating radiographic fingerprints from input medical image data.

[0010] FIG. 3 is a flowchart setting forth the steps of an example method for training a machine learning model to generate radiographic fingerprints from input medical image data.

[0011] FIGS. 4A and 4B illustrate t-distributed stochastic neighbor embedding (t-SNE) diagrams for radiographic fingerprint vectors extracted from 96 subjects in a validation set. The t-SNE plot in FIG. 4A was captured at the beginning of training, and the t-SNE plot in FIG. 4B was captured at the conclusion of training for the same subject and image cohort.

[0012] FIG. 5 is a flowchart setting forth the steps of an example method for generating view-corrected radiographic image data from input radiographic image data.

[0013] FIG. 6 is a flowchart setting forth the steps of an example method for training a machine learning model to generate view-corrected image data.

[0014] FIG. 7 is a flowchart setting forth the steps of an example method for generating a 3D anatomical model from two or more input radiographic images.

[0015] FIG. 8 is a flowchart setting forth the steps of an example method for training a machine learning model to generate a plurality of maximum intensity proj ections (MIPs) from a radiographic image.

[0016] FIG. 9 illustrates example Python code for a random histogram shift (RandHistogramShift) transformation (right) and examples of its application to random DRRs (left).

[0017] FIG. 10 illustrates example training and inference pipelines implemented in some embodiments described in the present disclosure. (*) Both DRRs underwent the RandHistogramShift transformation (with the same random noise) prior to input into the DDPM; this transformation is omitted in the figure for brevity7. (**) For actual training, “v- prediction” was employed. Abbreviations: DRR, Digitally Reconstructed Radiograph; DDPM, Denoising Diffusion Probabilistic Model; DDIM. Denoising Diffusion Implicit Model. (***) The histogram of the generated radiograph will also be matched to that of the input radiograph (not shown in the figure).

[0018] FIG. 11 is a block diagram of an example radiographic image generative Al system that can implement the methods described in the present disclosure.

[0019] FIG. 12 is a block diagram of example components that can implement the system of FIG. 11.DETAILED DESCRIPTION

[0020] Described here are systems and methods for generating medical images using one or more machine learning models. In some aspects, a radiographic image having a nonstandard projection view can be converted into an image with a standard projection view. Similarly, a single-view radiograph can be received as an input and used to generate radiographs from any point in three-dimensional (3D) space. Subsequently, these radiographs can be used to a virtual video that visualizes the patient’s anatomy in three dimensions. Advantageously, the disclosed systems and methods enable the consistent generation of such radiographs and virtual videos.

[0021] In some other aspects, two or more radiographic images can be used to generate a 3D image or model of the subject, or a portion of the subject. For instance, a 3D bony reconstruction of a subject’s bones can be generated from two or more radiographic images of the subject. A radiographic fingerprint can be generated from radiographic images of the subject and used to ensure that the generative images (e.g., view-corrected images, bony reconstructions) are anatomically accurate based on the subject’s unique anatomical features. The radiographic fingerprint may be generated as a numerical embedding of an input medical image, such as a radiograph and / or a digitally reconstructed radiograph (DRR) of the subject. DRRs are a form of medical imaging that serves as a virtual simulation of traditional x-ray images, generated from three-dimensional CT scans. This means that unlike conventional radiographs, which capture a single 2D view of x-ray absorption through the body, DRRs are generated by simulating the radiograph projection through the 3D CT volume. To create each pixel in a DRR. an algorithm estimates and sums the linear attenuation coefficients of the tissues that a simulated x-ray should pass through when traveling from a virtual x-ray source to the simulated film or receptor. The resulting 2D image resembles a conventional radiograph, albeit sometimes with notable differences in brightness and contrast.

[0022] In some aspects of the present disclosure, generative models are used for creating novel radiographic projections while preserving individual anatomical characteristics through the use of a radiographic fingerprint. The radiographic view correction technique canadvantageously reduce costs and radiation exposure. The 3D anatomical model construction technique can additionally provide valuable insights into patients’ anatomical structures. These systems and methods can improve synthetic image generation in medical applications while reducing costs and improving accuracy and consistency. Advantageously, the disclosed systems and methods can find use in various medical and clinical applications. For example, low-cost radiographs generated using the disclosed systems and methods can be used to create 3D templates and inputs for robotic surgical devices. Additionally or alternatively, hospitals and clinics can use the systems and methods described in the present disclosure to reduce the cost and radiation exposure associated with traditional radiographs while still obtaining accurate and consistent images for diagnosis and treatment planning.

[0023] An overview of these potential workflows is illustrated in FIG. 1. In this example, a contrastive model (e g., an FPNet model) receives an input radiograph and the convolutional neural network (CNN) encoder embeds the spatial features of that input radiograph into a one-dimensional (ID) numerical vector. This embedding may be referred to as a radiographic fingerprint for the subject.

[0024] To generate a 3D bony reconstruction from an input radiograph image, an x-ray (XR) to MIP DDPM module receives an input radiograph and a radiographic fingerprint extracted from that radiograph and, based on these inputs, generates a plurality of maximum intensity projections (MIPs) for that radiograph in different spatial positions. In the nonlimiting illustrated example. 100 MIPs may be generated. The plurality of MIPs may then be input to a neural radiance field (NeRF) model to generate a bony reconstruction of the subj ect’ s bones. For instance, the plurality of MIPs that represent the same bony anatomical structure in different spatial positions may be input to the NeRF model and used to generate a 3D reconstruction of the underlying bony structure. Because the radiographic fingerprint is passed as an input and used to condition the generative models (e g., using an attention mechanism, such as a cross-attention between images and corresponding radiographic fingerprints), the resulting MIPs and bony reconstruction will be anatomically consistent with the subject’s anatomical features.

[0025] Additionally or alternatively, to transform a radiograph from a non-standard view to a standard view (or more generally, from a first view to a second view that is different than the first view), an XR to DRR DDPM module receives an input radiograph and transfers its style to that of a DRR without changing its content. In the illustrated example this synthetic DRR is identified as DRR1. A DRR to DRR DDPM module then receives this DDR1 as wellas a set of user-specified transformations to generate a second DDR (DDR2) that is transformed in space. The DDR to XR DDPM module then receives the transformed DDR (i.e., DDR2) and will transfer its style to that of a radiograph without changing its content. Because the radiographic fingerprint is passed to the generative model in the DRR to DDR DDPM module, the resulting transformed radiography will be anatomically consistent with the subject’s anatomical features in the initial radiograph.

[0026] Diffusion models are a family of deep learning models that learn to gradually convert random noise into a meaningful image that mimics the distribution of their training data. This approach to generating images is fundamentally different from the direct generation methods used by other types of models, such as GANs. Diffusion models work by first "‘destroying” a copy of an image over a series of steps to turn it into random noise, and then learning to reverse this process to create new images from noise. This is achieved in tw o main phases: the forward pass and the backward pass, as illustrated in FIG. 1.

[0027] In the forward pass, the diffusion algorithm incrementally adds Gaussian noise to an input image over a predetermined number of steps, until the image is completely transformed into random Gaussian noise. The added Gaussian noise at each step is sampled from distributions with pre-determined means and standard deviations, so it does not require training a network. The backward pass, which is where the learning happens, involves training a neural network to approximate the parameters of the distributions used for sampling random noise during the forward pass and reverse that pass step-by-step (i.e.. “denoising”). At each step, the model makes a prediction about what the less noisy image looked like one step before, starting from pure noise and gradually removing noise until it arrives at a meaningful image. This reverse process is guided by minimizing the difference between the ground truth images, which are less noisy, and the generated ones, or by using other loss functions that similarly optimize the denoising capabilities of the model.

[0028] The DDPM is a specific ty pe of diffusion model that formalizes the diffusion process as a Markov chain and uses variational inference for the denoising process. DDPMs, distinguished by their ability to produce images of high quality with impressive mode coverage, are better alternatives to GANs and variational autoencoders (VAEs). Their primary drawback, however, is the slower inference speed. In the context of medical imaging, where the accuracy and detail of images are paramount, the slower speed of DDPMs is often a reasonable tradeoff.

[0029] When using a generative model (e.g., a GAN, a DDPM) to generate images of a subject from input medical image data, it is desirable to ensure that the underlying anatomical features of the subject are accurately represented in the generative image data. Currently available methods for generative artificial intelligence (Al) can be conditioned on specific criteria, but have no way to ensure that the individual characteristics of a subject are not being altered in the generative image data. In other words, currently available generative methods cannot tell the model to transform an image of a subject in such a way that the resultant image still belongs to the same subject (i.e., consistent with the same unique anatomical features of the subject). This necessitates having an individual signature for each subject based on their anatomic characteristics so that the generative model would be conditioned on this signature to ensure consistency between input and output images. For example, a generative model can be conditioned on radiographic fingerprints using an attention mechanism, such as a crossattention between images and corresponding radiographic fingerprints (e.g., fingerprint-to- image).

[0030] It is an aspect of the present disclosure to overcome this limitation by creating a radiographic fingerprint for a subject, which can then be used to condition generative models for generating images from medical image data acquired from that same subject. For example, an embedding can be generated from an image of a given subject. The embedding may be a numerical embedding, such as a ID numerical vector. The embedding may be stored as a radiographic fingerprint for the subject.

[0031] The radiographic fingerprint may be compared with embeddings generated from other images of the subject to evaluate the anatomical accuracy of those images. For instance, images of the subject may be created using a generative Al model, or other machine learning model.

[0032] To create a radiographic fingerprint, a contrastive learning approach can be used to train a deep learning model to identify and extract unique anatomical features from each subject’s imaging data. The contrastive learning approach enables the model to unfold the data manifold so that images from the same subject have a similar fingerprint (even if they have different projections). This will allow the creation of a radiographic fingerprint for each subject, which can then be used to condition the generation of synthetic images, ensuring that the individual characteristics of the subject are preserved. This approach has the potential to greatly improve the accuracy and consistency of synthetic image generation in medical applications.

[0033] Additionally or alternatively, a radiographic fingerprint can be used for deidentification and / or reidentification of a patient. As an example, the patient medical image data can be anonymized and the radiographic fingerprint (which conveys no protected health information (PHI) or identifiable information about the patient) can be used to identify medical images of the patient without storing any information about the patient. For instance, the embedding vector of the radiographic fingerprint contains no PHI or other information that can identify the patient, yet the embedding vector can be used to identify other medical images of the patient based on comparing the similarity of embedding vectors extracted from those images with the radiographic fingerprint. In this way, the radiographic fingerprint can advantageously be used for deidentification and / or reidentification of patient medical images.

[0034] Referring now to FIG. 2. a flowchart is illustrated as setting forth the steps of an example method for generating a radiographic fingerprint using a suitably trained neural network or other machine learning algorithm. As will be described, the neural network or other machine learning algorithm takes medical image data as input data and generates a radiographic fingerprint as output data. As an example, the radiographic fingerprint may be an embedding, such as a ID numerical vector, that encodes the unique anatomy of the subject.

[0035] The method includes accessing medical image data with a computer system, as indicated at step 202. Accessing the medical image data may include retrieving such data from a memory or other suitable data storage device or medium. Additionally or alternatively, accessing the medical image data may include acquiring such data with a medical imaging system and transferring or otherwise communicating the data to the computer system, which may be a part of the medical imaging system.

[0036] The medical image data may include one or more radiographic images. For example, the medical image data may include one or more digital radiographic images acquired with an x-ray imaging system. The medical image data may also include one or more x-ray fluoroscopy images acquired with an x-ray fluoroscopy imaging system. Additionally or alternatively, the medical imaging data may include digitally reconstructed radiographs (DRRs) generated from one or more CT images or CT projection data acquired ith a CT imaging system.

[0037] A trained neural network, or other suitable machine learning model, is then accessed with the computer system, as indicated at step 204. In general, the neural network or other machine learning model is trained, or has been trained, on training data in order to generate an embedding from a medical image contained in the medical imaging data.

[0038] Accessing a trained neural network may include accessing network parameters (e.g.. weights, biases, or both) that have been optimized or otherwise estimated by training the neural network on training data. In some instances, retrieving the neural network can also include retrieving, constructing, or otherwise accessing the particular neural network architecture to be implemented. For instance, data pertaining to the layers in the neural network architecture (e.g., number of layers, type of layers, ordering of layers, connections between layers, hyperparameters for layers) may be retrieved, selected, constructed, or otherwise accessed.

[0039] An artificial neural network generally includes an input layer, one or more hidden layers (or nodes), and an output layer. Typically, the input layer includes as many nodes as inputs provided to the artificial neural network. The number (and the type) of inputs provided to the artificial neural network may vary based on the particular task for the artificial neural network.

[0040] The input layer connects to one or more hidden layers. The number of hidden layers varies and may depend on the particular task for the artificial neural network. Additionally, each hidden layer may have a different number of nodes and may be connected to the next layer differently. For example, each node of the input layer may be connected to each node of the first hidden layer. The connection between each node of the input layer and each node of the first hidden layer may be assigned a weight parameter. Additionally, each node of the neural network may also be assigned a bias value. In some configurations, each node of the first hidden layer may not be connected to each node of the second hidden layer. That is, there may be some nodes of the first hidden layer that are not connected to all of the nodes of the second hidden layer. The connections between the nodes of the first hidden layers and the second hidden layers are each assigned different weight parameters. Each node of the hidden layer is generally associated with an activation function. The activation function defines how the hidden layer is to process the input received from the input layer or from a previous input or hidden layer. These activation functions may vary and be based on the ty pe of task associated with the artificial neural network and also on the specific type of hidden layer implemented.

[0041] Each hidden layer may perform a different function. For example, some hidden layers can be convolutional hidden layers which can, in some instances, reduce the dimensionality of the inputs. Other hidden layers can perform statistical functions such as max pooling, which may reduce a group of inputs to the maximum value; an averaging layer; batchnormalization; and other such functions. In some of the hidden layers each node is connected to each node of the next hidden layer, which may be referred to then as dense layers. Some neural networks including more than, for example, three hidden layers may be considered deep neural networks.

[0042] The last hidden layer in the artificial neural network is connected to the output layer. Similar to the input layer, the output layer typically has the same number of nodes as the possible outputs. For example, the output layer may include, for example, anode corresponding to a numerical embedding of the input medical image data.

[0043] The medical image data are then input to the neural network or other machine learning model, generating output as a radiographic fingerprint, as indicated at step 206. As described above, the radiographic fingerprint include a numerical embedding (e.g., a ID numerical vector) that encodes the unique anatomy of the subject as depicted in the input medical image data. For example, when the input medical image data include a radiographic image of the subject, the radiographic fingerprint encodes the unique anatomy of the subject as depicted in a radiographic image. Alternatively, when the input medical image data include a DRR of the subject, the radiographic fingerprint encodes the unique anatomy of the subject as depicted in a DRR. More generally, the radiographic fingerprint encodes the unique anatomy of the subject as depicted in the imaging modality and / or image type of the input medical image data (e g., a CT image, a T1 -weighted magnetic resonance image, a T2-weighted magnetic resonance image, etc.).

[0044] The radiographic fingerprint generated by inputting the medical image data to the trained neural network or other machine learning model can then be displayed to a user, stored for later use or further processing, or both, as indicated at step 208. For example, as described below in more detail, the radiographic fingerprint may be used when training and implementing other machine learning models, including generative models.

[0045] Referring now to FIG. 3, a flowchart is illustrated as setting forth the steps of an example method for training one or more neural networks (or other suitable machine learning models) on training data, such that the one or more neural networks (or other machine learning models) are trained to receive medical image data as input data in order to generate a numerical embedding as output data, where the numerical embedding is indicative of anatomical features in the medical image data being encoded in the lower-dimensional space of the embedding.

[0046] In general, a neural network can implement any number of different neural network architectures. For instance, the neural network(s) could implement a convolutional neural network, a residual neural network, or the like. Alternatively, the neural network(s) could be replaced with other suitable machine learning or artificial intelligence algorithms, such as those based on supervised learning, unsupervised learning, deep learning, ensemble learning, dimensionality reduction, and so on.

[0047] The method includes accessing training data with a computer system, as indicated at step 302. Accessing the training data may include retrieving such data from a memory or other suitable data storage device or medium. Alternatively, accessing the training data may include acquiring such data with a medical imaging system and transferring or otherwise communicating the data to the computer system.

[0048] In general, the training data can include medical images acquired from the one or more subjects for whom a radiographic fingerprint will be generated. More generally, the training data will include medical images of the same imaging modality and / or image type that will be used as an input for the trained model.

[0049] The method can include assembling training data from medical image data using a computer system. This step may include assembling the medical image data into an appropriate data structure on which the neural netw ork or other machine learning model can be trained. Assembling the training data may include assembling medical image data and other relevant data. For instance, assembling the training data may include generating augmented medical image data and including the augmented medical image data in the training data. Augmented medical image data may include medical image data that have been augmented, such as by applying random transformations, random rotations, random noise insertions, or combinations thereof.

[0050] One or more neural networks (or other suitable machine learning algorithms) are trained on the training data, as indicated at step 304. In general, the neural network can be trained by optimizing network parameters (e.g., weights, biases, or both) based on minimizing a loss function. As one non-limiting example, the loss function may be a mean squared error loss function.

[0051] Training a neural network may include initializing the neural network, such as by computing, estimating, or otherwise selecting initial network parameters (e.g., weights, biases, or both). During training, an artificial neural network receives the inputs for a training example and generates an output using the bias for each node, and the connections betweeneach node and the corresponding weights. For instance, training data can be input to the initialized neural network, generating output as a numerical embedding of an input medical image. The artificial neural network then compares the generated output with the actual output of the training example in order to evaluate the quality of the numerical embedding. For instance, the numerical embedding can be passed to a loss function to compute an error. The current neural network can then be updated based on the calculated error (e.g., using backpropagation methods based on the calculated error). For instance, the current neural network can be updated by updating the network parameters (e.g., weights, biases, or both) in order to minimize the loss according to the loss function. The training continues until a training condition is met. The training condition may correspond to, for example, a predetermined number of training examples being used, a minimum accuracy threshold being reached during training and validation, a predetermined number of validation iterations being completed, and the like. When the training condition has been met (e.g., by determining whether an error threshold or other stopping criterion has been satisfied), the current neural network and its associated network parameters represent the trained neural network. Different types of training processes can be used to adjust the bias values and the weights of the node connections based on the training examples. The training processes may include, for example, gradient descent, Newton's method, conjugate gradient, quasi-Newton, Levenberg-Marquardt, among others.

[0052] The artificial neural network can be constructed or otherwise trained based on training data using one or more different learning techniques, such as supervised learning, unsupervised learning, reinforcement learning, ensemble learning, active learning, transfer learning, or other suitable learning techniques for neural networks. As an example, supervised learning involves presenting a computer system with example inputs and their actual outputs (e.g.. categorizations). In these instances, the artificial neural network is configured to learn a general rule or model that maps the inputs to the outputs based on the provided example inputoutput pairs.

[0053] As mentioned above, the neural netw ork or other machine learning model can be trained using contrastive learning. In contrastive learning, a contrastive loss can be defined and used when training the machine learning model. For example, given a list of input samples (e.g., the medical images of the subject that constitute the training samples in the training data), contrastive learning seeks to learn a function that encodes a new sample into an embedding vector such that the samples from the same class have similar embeddings and the samples from different classes have different embeddings. Within the context of generating aradiographic fingerprint, images that share similar unique anatomical features of the subject can be encoded into embedding vectors that are similar to embedding vectors encoded from other images of the subject that also depict the unique anatomical features. Images that do not depict the unique anatomical features are encoded into embedding vectors that are dissimilar and, therefore, representative of a different class. Contrastive loss takes a pair of inputs and minimizes the embedding distance when they are from the same class, but maximizes the distance otherwise. In some instances, the embedding distance can be computed based on a cosine similarity.

[0054] FIGS. 4A and 4B illustrate t-distributed stochastic neighbor embedding (t-SNE) diagrams for fingerprint vectors extracted from 96 subjects in a validation set. Each subject had two radiographs: either from AP, cross-table lateral, or oblique views. In the t-SNE plots, the vector for each radiograph is represented by a dot. Dots representing the same subject share the same color and are connected by a line. The t-SNE plot in FIG. 4A, captured at the beginning of training, shows that the model initially struggled to distinguish between subjects’ fingerprints. This is evidenced by the longer lines indicating lower cosine similarity and greater distance between embeddings from the same subject’s radiographs. The t-SNE plot in FIG. 4B, captured at the conclusion of training for the same subject and image cohort, shows dots from the same subjects in close proximity7with much shorter connecting lines. This indicates the model's successful learning in extracting distinct vectors from the radiographs.

[0055] The one or more trained neural networks or other machine learning models are then stored for later use, as indicated at step 306. Storing a neural network may include storing network parameters (e.g., weights, biases, or both), which have been computed or otherwise estimated by training the neural network(s) on the training data. Storing the trained neural network(s) may also include storing the particular neural network architecture to be implemented. For instance, data pertaining to the layers in the neural network architecture (e.g., number of layers, type of layers, ordering of layers, connections between layers, hyperparameters for layers) may be stored.

[0056] In orthopedic surgery, many preoperative, intraoperative, and postoperative measurements take place on anteroposterior radiographs. However, there are certain criteria that must be met in order for a radiograph to be characterized as standard and suitable for precise measurements. If a radiograph is non-standard, then the patient must take another radiograph, which incurs cost and results in the patient receiving additional doses of radiation.

[0057] It is an aspect of the present disclosure to use a generative model (e.g., a GAN, aDDPM) to bring non-standard radiographic images into the standard view. Furthermore, there are often a variety of standard views that are desirable of a given anatomic structure. Using the systems and methods described in the present disclosure, multiple view projections can also be obtained from a single radiograph. For example, a “Judet View’' of the hip is 45 degrees rotated in the coronal plane from a neutral anteroposterior pelvis radiograph. This view can be synthetically generated based on an initial anteroposterior pelvis radiographic image. Thus, more generally, the disclosed systems and methods can receive arbitrary rotation coordinates and bring an input radiograph into a specific projection plane. For example, automatic view correction of an x-ray or fluoroscopic image in the x, y. and z planes can be provided. The radiographic fingerprint described above can be used to ensure anatomical consistency between the images.

[0058] Referring now to FIG. 5, a flowchart is illustrated as setting forth the steps of an example method for generating view-corrected radiographic images using a suitably trained neural network or other machine learning model. As will be described, the neural network or other machine learning model takes a radiographic image and image transformation data as input data and generates a view-correct radiographic image as output data.

[0059] The method includes accessing radiographic image data with a computer system, as indicated at step 502. Accessing the radiographic image data may include retrieving such data from a memory or other suitable data storage device or medium. Additionally or alternatively, accessing the radiographic image data may include acquiring such data with an x-ray imaging system and / or x-ray fluoroscopy system and transferring or otherwise communicating the data to the computer system, which may be a part of the imaging system. In general, the radiographic image data may include a radiographic image of a subject in a first view. The first view may be a non-standard view, or may otherwise be a view that is desirable to convert into a second view that is different from the first view.

[0060] Image transformation data are also accessed with the computer system, as indicated at step 504. The image transformation data may include one or more image translations, one or more image rotations, or other such image transformations for converting an image from a first view to a second view. The image transformation data may be input by a user (e g., via a user interface), or may be selected from a database of preselected image transformations. In general, the image transformation data may define a transformation from the first view to the second view. As one example, the first view may be a non-standard viewand the second view may be a standard view. More generally, the first view and the second view may be different views associated with different projection planes.

[0061] The radiographic image data may then be converted to DRR image data, as indicated at step 506. As a non-limiting example, the radiographic image data may be converted to DRR image data using a suitably trained machine learning model. For example, a deep learning model may be trained on a large set of DRRs and radiographs to leam a mapping between the radiographs and DRRs. The trained machine learning model may, therefore, be trained to an input radiographic image to a DRR corresponding to the same view as the input radiographic image.

[0062] A radiographic fingerprint for the subject may also be accessed with the computer system, as indicated at step 508. Accessing the radiographic fingerprint can include retrieving a previously generated radiographic fingerprint for the subject, or may include generating the radiographic fingerprint from the radiographic image data. In some other examples, the radiographic fingerprint may be generated from the DRR image data generated in step 506.

[0063] A trained machine learning model is then accessed with the computer system, as indicated at step 510. In general, the machine learning model may be a generative model. A generative model includes a model that is capable of describing how a dataset is generated in terms of a statistical and / or probabilistic model. The generative model can be sampled (e.g., by inputting a new data set to the generative model) in order to generate new data that is consistent with, or plausibly derivable from, the modeled data set. Generative models may be trained or otherwise constructed using machine learning and / or deep learning. As such, the generative models may be referred to as machine learning models, deep learning models, or the like.

[0064] As a non-limiting example, the generative models implemented in the present disclosure can include a GAN (e.g., a ProGAN model, a StyleGAN model, a BigGAN model, and so on), a variational autoencoder (VAE), a DDPM, or the like. GANs have high inference speed and can generate high quality output samples, but may be limited in the diversity' of outputs that they can generate. VAEs have high inference speed and output diversity, but may be limited by the quality of samples generated. DDPMs can produce high quality and diversity outputs, but may be limited by longer inference times. Thus, while many different types of generative models may be used when implementing the systems and methods described in the present disclosure, in some preferred embodiments a DDPM-based generative model may be used.

[0065] In general, a DDPM is a parameterized Markov chain trained using variational inference to produce samples matching the data after finite time. Transitions of this chain can be learned to reverse a diffusion process, which is a Markov chain that gradually adds noise to the data in the opposite direction of sampling until signal is destroyed. When the diffusion includes small amounts of Gaussian noise, it is sufficient to set the sampling chain transitions to conditional Gaussians, too, thereby allowing for a particularly simple neural network parameterization.

[0066] As is the case with other deep learning models, DDPMs are trained by gradually encountering a prepared set of images, known as training data. In some instances, a DDPM can be trained using unsupervised learning techniques. Additionally or alternatively. DDPMs can also be utilized for supervised or conditional training (e.g.. conditioned based at least in part on a radiographic fingerprint to ensure anatomical consistency and accuracy of synthetic images generated by the generative model with input images).

[0067] Accessing the trained neural network may include accessing network parameters (e.g., weights, biases, or both) that have been optimized or otherwise estimated by training the neural network on training data. In some instances, retrieving the neural network can also include retrieving, constructing, or otherwise accessing the particular neural network architecture to be implemented. For instance, data pertaining to the layers in the neural network architecture (e.g., number of layers, type of layers, ordering of layers, connections between layers, hyperparameters for layers) may be retrieved, selected, constructed, or otherwise accessed.

[0068] The DRR image data, image transformation data, and radiographic fingerprint are then input to the generative model, as indicated at step 512, to generate view-corrected DRR image data as an output. The view-corrected DRR image data corresponds to DRR image data that have been converted from the first view to the second view. The view-corrected DRR image data are then converted to view-corrected radiographic image data, as indicated at step 514. For instance, a suitably trained machine learning model can be used to convert the view- corrected DRR image data to the view-corrected radiographic image data.

[0069] The view-corrected radiographic image data can then be displayed to a user, stored for later use or further processing, or both, as indicated at step 516.

[0070] Referring now to FIG. 6, a flowchart is illustrated as setting forth the steps of an example method for training a machine learning model (e.g., a generative model) on training data, such that the machine learning model is trained to receive DRR image data, imagetransformation data, and / or a radiographic fingerprint as input data in order to generate view- corrected DRR image data as output data.

[0071] The method includes accessing training data with a computer system, as indicated at step 602. Accessing the training data may include retrieving such data from a memory' or other suitable data storage device or medium. Alternatively, accessing the training data may include acquiring such data with an imaging system and transferring or otherwise communicating the data to the computer system.

[0072] In general, the training data can include CT image data, from which a plurality of DRRs can be extracted. For example, CT images can be accessed and programmatically “flattened” by moving the projection axis in different directions. The resultant 2D images can be used to train a generative to convert non-standard radiographs into standard ones. Additionally or alternatively, the training data can include a plurality of already available radiographs and / or DRRs.

[0073] Additionally, the training data may include other data, such as image transformation data (e.g., the view angles associated with each DRR), radiographic fingerprints generated from the plurality of DDRs, etc.

[0074] The method can include assembling training data from the CT image data and / orDRR image data using a computer system. This step may include assembling the CT image data and / or DRR image data into an appropriate data structure on which the neural network or other machine learning algorithm can be trained. Assembling the training data may include assembling DRRs and other relevant data, such as associated image transformation data and / or radiographic fingerprints.

[0075] One or more machine learning models are trained on the training data, as indicated at step 604. In general, the machine learning model can be trained by optimizing network parameters (e.g., weights, biases, or both) based on minimizing a loss function. As one non-limiting example, the loss function may be a mean squared error loss function. The training of the machine learning model may be conditioned on radiographic fingerprints generated from the DRRs.

[0076] The one or more machine learning models are then stored for later use, as indicated at step 606. Storing the machine learning model may include storing network parameters (e.g., weights, biases, or both), which have been computed or otherwise estimated by training the machine learning model on the training data. Storing the trained machine learning model may also include storing the particular model architecture to be implemented.For instance, data pertaining to the layers in a neural network architecture (e.g., number of layers, type of layers, ordering of layers, connections between layers, hyperparameters for layers) may be stored.

[0077] It is another aspect of the present disclosure that by using two or more images of the subject, a bony reconstruction, or other 3D reconstruction, can be generated using a suitable trained and conditioned generative model. For example, a 3D bony model of a patient can be generated using two different view radiographs from the same patient. As a non-limiting example, a neural radiance field (NeRF) model coupled with a DDPM can be used to create 3D meshes representing bony structures.

[0078] Referring now to FIG. 7. a flowchart is illustrated as setting forth the steps of an example method for generating a 3D model of patient anatomy from two or more input 2D images using a suitably trained machine learning model. As will be described, the machine learning model takes at least two orthogonal radiographic images and radiographic fingerprints extracted from those images as input data and generates a 3D anatomical model as output data. As an example, the 3D anatomical model can depict bony structures within the patient. In these instances, the 3D anatomical model may be referred to as a bony model or a bony reconstruction.

[0079] The method includes accessing radiographic image data with a computer system, as indicated at step 702. Accessing the radiographic image data may include retrieving such data from a memory or other suitable data storage device or medium. Additionally or alternatively, accessing the radiographic image data may include acquiring such data with an x-ray imaging system or an x-ray fluoroscopy system and transferring or otherwise communicating the data to the computer system, which may be a part of the imaging system.

[0080] A trained machine learning model is then accessed with the computer system, as indicated at step 704. In general, the machine learning model is trained, or has been trained, on training data in order to generate a plurality of MIPs from input radiographic image data and system-specified spatial positions. Accessing the trained machine learning model may include accessing model parameters (e.g.. weights, biases, or both) that have been optimized or otherwise estimated by training the neural network on training data. In some instances, retrieving the machine learning model can also include retrieving, constructing, or otherwise accessing the particular model architecture to be implemented. For instance, data pertaining to the layers in a neural network architecture (e.g., number of layers, type of layers, ordering oflayers, connections between layers, hyperparameters for layers) may be retrieved, selected, constructed, or otherwise accessed.

[0081] The radiographic image data are then input to the machine learning model, generating a plurality of MIPs as output data, as indicated at step 706. As indicated above, system-specified spatial position data may also be input to the machine learning model, where the spatial position data correspond to the different spatial positions for which the MIPs will be generated. Radiographic fingerprints generated from the radiographic image data may also be input to the machine learning model.

[0082] Using the MIPs generated in step 706, a NeRF model is then trained, as indicated at step 708. The NeRF model generates a 3D volume and / or reconstruction generated from the plurality of MIPs. In this context, the ‘"scene” includes the anatomical features to be modeled in the 3D anatomical model. For example, the scene generated by the NeRF model can include foreground data corresponding to the bony structures represented in the MIPs. As such, the NeRF model can represent the 3D anatomical model, which in some instances may correspond to a bony reconstruction.

[0083] As another example, the 3D volume and / or reconstruction may correspond to soft tissue structures, such as nerves, blood vessels, and the like. In these instances, the medical image data used to generate the 3D model may not be radiographic image data, but may instead be CT image data, magnetic resonance image data, and / or ultrasound image data. A 3D model of soft tissue structures can be advantageous for intraoperative guidance with robotic surgical systems and / or augmented reality (AR) or virtual reality (VR) platforms following registration against the bone for calibration.

[0084] The 3D anatomical model generated in step 708 can then be displayed to a user, stored for later use or further processing, or both, as indicated at step 710.

[0085] Referring now to FIG. 8, a flowchart is illustrated as setting forth the steps of an example method for training one or more machine learning models (e.g., a generative model such as a GAN or DDPM) on training data, such that the one or more machine learning models are trained to receive radiographic images (e.g.. at least two orthogonal radiographic images), spatial position data corresponding to spatial positions of MIPs, and / or radiographic fingerprints corresponding to the radiographic images as input data in order to generate a plurality of MIPs as output data.

[0086] In general, the neural network(s) can implement any number of different neural network architectures. For instance, the neural network(s) could implement a convolutionalneural network, a residual neural network, or the like. Alternatively, the neural network(s) could be replaced with other suitable machine learning or artificial intelligence algorithms, such as those based on supervised learning, unsupervised learning, deep learning, ensemble learning, dimensionality reduction, and so on.

[0087] The method includes accessing training data with a computer system, as indicated at step 802. Accessing the training data may include retrieving such data from a memory or other suitable data storage device or medium. Alternatively, accessing the training data may include acquiring such data with an imaging system and transferring or otherwise communicating the data to the computer system.

[0088] In general, the training data can include radiographic images, CT images, MIPs, and / or DRRs. Additionally, the training data may include other data, such as system-specified spatial positions for the MIPs, radiographic fingerprints, and the like. The method can include assembling training data from the images. In general, the training data can include a plurality of MIPs in different spatial positions. The MIPs may be generated from CT image data of the subject, or may be previously generated MIPs.

[0089] One or more machine learning models (e.g., a generative model such as a GAN or DDPM) are trained on the training data, as indicated at step 804. In general, the machine learning model can be trained by optimizing model parameters (e.g., weights, biases, or both) based on minimizing a loss function. As one non-limiting example, the loss function may be a mean squared error loss function.

[0090] The one or more machine learning models are then stored for later use, as indicated at step 806. Storing the machine learning model may include storing model parameters (e.g., weights, biases, or both), which have been computed or otherwise estimated by training the machine learning model on the training data. Storing the trained machine learning model may also include storing the particular model architecture to be implemented. For instance, data pertaining to the layers in a neural network architecture (e.g., number of layers, type of layers, ordering of layers, connections between layers, hyperparameters for layers) may be stored.

[0091] In an example study, the systems and methods disclosed in the present application were evaluated for rotating radiographs in 3D by using a deep learning pipeline that received a single AP pelvis radiograph (with or without hip arthroplasty prosthesis present in the image) and rotated it up to 15 degrees in the x, y, and / or z planes.

[0092] A total of 9.044 non-contrast-enhanced hip and pelvis CT studies were acquired and used in this study. This cohort included imaging studies of patients with and without a history of hip arthroplasty. Digital Imaging and Communications in Medicine (DICOM) images were retrieved for the aforementioned studies and anonymized by converting all studies into Neuroimaging Informatics Technology Initiative (NlfTI) files. Through a combination of programming pipelines and manual review, studies that met the following conditions were selected: (1) captured the entire region of the pelvis (the area between the iliac crest superiorly and the lesser trochanter of the femurs inferiorly); (2) captured both hip joints; (3) had at least 50 axial slices; (4) had a slice thickness of 1.5mm or less; and (5) were axial images. This filtering process yielded 5,052 eligible studies that were then resampled to a pixel spacing of (1mm x 1mm x 1mm) and saved as the data pool for the training and validation of the pipeline. In cases where more than one series was available from a patient, all eligible series were included.

[0093] Following the collection of eligible CT studies, the CT data were converted into DRRs. As a non-limiting example, a machine learning-based approach was used to convert the CT data to DRRs. The machine learning approach used a deep learning-based pipeline that included material decomposition of the CT data, ray-tracing forward projection, Rayleigh scatter estimation, and random noise injection to build the DRRs. To extract DRRs, the CT data were placed in the isocentric (standard) position where the patient is head-first and supine (AP imaging). A baseline DRR (DRRo) was then obtained by AP projection. The CT was then rotated using the Pi = [xi, yi, zi] vector with the center of the volume as the anchor, where {xi, yi, zi] are randomly sampled integer values from [-15°, +15°] and DRRi was obtained. A transformation vector V = [a. b, c] was created, where {a, b, c} were sampled from [-15°, +15°], and the absolute values of the angles in P2 = Pi + V in each axis (x. y, z) were less than 15°, i.e., |P2x| < 15° & |P2y| < 15° & |P2z| < 15°. The CT was then rotated to the new position P2, and DRR2 was obtained. All DRRs were then standardized to a uniform size of 256x256 pixels and the DRRs were saved to disk after rescaling them to a pixel intensity range of 0 to 1. The coordinates of vectors that could convert could convert each DRR into other rotations of that DRR (e.g.„ DRRi— >DRR2, DRR2— >DRRI, DRRI— ►DRRO,, DRR2— ►DRRO., DRRO^DRRI, and DRRo^DRR2) were also calculated and saved. Overall, 512 DRRs were created for each of the 5,052 available CT volumes, resulting in over 2.5 million DRRs saved on disk. This produced more than 15 million pairs of input and output images to serve as the ground truth for training the deep learning model.

[0094] The data were divided into seven random folds, grouping the DRRs at the patient level and stratifying the splits based on the acquisition date, number of slices, and slice thickness of the original CT data. Folds 1 through 5 were designated for training the DL models, Fold 6 was used for validation and hyperparameter tuning, and Fold 7 was reserved as an internal test set.

[0095] A random histogram shift (RandHistogramShift) transformation was developed and used in the data preprocessing pipeline during the training phase. The logic behind the RandHistogramShift transfer was as follows. It was hypothesized that the primary distinction between a DRR and a radiograph lies in their pixel intensity distributions, affecting their overall brightness and contrast. Therefore, by making the DDPM agnostic to the pixel intensity distribution of the input images, it becomes capable of generalizing to any type of radiography data (e g., x-rays, DRRs, or fluoroscopy shots), despite being solely trained on DRRs. An example implementation of this is shown in FIG. 9. During training, a random image filled with stochastic noise is generated and then image processing techniques are applied to align the histogram of each pair of input and output DRRs with that of the noisy image. Given that the noisy image is randomly generated at each training step, the DDPM is pushed to disregard the stochastic pixel intensity distributions of the ground truth DRR pairs, focusing instead on their content to leam the 3D rotations.

[0096] Following the collection of DRRs, a conditional DDPM that received an input DRR as well as three integer values representing the desired amount of transformations on the x, y, and z axes, respectively, was trained. A transformed DRR was generated as an output of this model, as illustrated in FIG. 10. To train the deep learning model, a forward diffusion process that included 1,000 steps was established, in which Gaussian noise was added to the original images at each step. The model was encouraged to leam the reverse diffusion process. This reverse diffusion process aimed to approximate the noise added between two successive steps, thus reconstructing the less noisy image from the noisier version at each step. The deep learning model used for the reverse diffusion process featured a U-Net-like architecture with several advantages over a standard U-Net, including multiple residual blocks in each layer, self-attention modules for improved image quality, and an embedded timestep number that dictated the intensity of the added noise.

[0097] DDPMs receive an input image containing random Gaussian noise that is gradually denoised it into a meaningful image through the reverse diffusion process. Conditional DDPMs require additional signals, either in the form of imaging data or linearvectors, to generate the target image. In this example study, both strategies were implemented to provide the DDPM with data used to generate a target DRR. First, the original DRR was concatenated with the noisy image as a second channel and then a linear vector containing the three desired transformation values was created to guide the DDPM on the required changes to the original DRR. Conditioning the DDPM on this vector was accomplished using classifier- free guidance, a technique that obviated the need for loading an additional network to steer the DDPM generations toward certain classes. The detailed configurations of the DDPM model used in this example study are shown in Table 1.Table 1. Configurations used for the training and validation of the example DDPMParameter ValueDiffusion timesteps 1000 schedule name cosine enforce_zero_terminal_snr true schedule_params:- beta_start 0.0001- beta end 0.02- cosine_s 0.008 timestep_respacing null mean type VELOCITY var type LEARNED RANGE loss_type Mean Squared ErrorOptimizerLearning Rate le-4 Type LionValidation classifier_cond_scale 4 protocol DDPM log_original true log_concat true cls_log_indices [0, 1, 2, 3]Model input_size 256 dims 2 attention_resolutions [8, 16, 32] channel_mult [1, 1, 2, 2, 4, 4] dropout 0.0 in_channels 2 out_channels 2 model channels 128num head channels -1 num_heads 4 num heads upsample -1 num_res_blocks [2, 2, 2, 2, 2, 2] resblock updown false use_checkpoint false use_new_attention_order false use_scale_shift_norm true scale skip connection falseConditions num_classes 4 concat_channels 1 guidance_drop_prob 0. 1 missing class value null

[0098] The DDPM was trained with a constant learning rate of 0.0001, and utilized v- prediction as the training objective for the diffusion model to ensure correct image intensity during inference. The training was continued until an identical network, utilizing the exponential moving average of the weights, with a decay factor of 0.9999, from the actively trained network, overfit the validation set. All images were scaled to a pixel value range of [- 1, 1] before feeding them into the DDPM model. In addition, three specific transformations were applied to the training set. First, two rows of zero-valued pixels were added to the top and bottom margins of the input and output images. Extracted DRRs from the CT data do not have margins of zero pixels; however, the addition of such margins (i.e., padding) helped with creating square-shaped radiographs during inference. Including this transformation during training helped the DDPM not to regard the padded radiographs as outliers during inference. The number of images subjected to this transformation and the length of the added bars were determined randomly. Second, rectangles of stochastic size were randomly added to the input DRRs. This transformation aimed to mimic the appearance of radiographs with covered protected health information using black rectangles using an already established model. Third, the RandHistogramShift transformation described above was leveraged during training to instruct the DDPM on transforming both DRRs and conventional radiographs during inference time.

[0099] Two inference modes for the model were evaluated. In the first (mode A), the model was used in a single-shot or few-shot manner, where the user inputs a set of transformation angles in addition to a baseline radiograph and expects the model to rotate itaccording to the provided angles. In the second (mode B), the model was used to generate an entire range of rotations in one or more axes, allowing the output frames to create a 30 frame- per-second (FPS) video clip of the anatomical volume being rotated in 3D. For both modes, a DDPM protocol with 200 inference steps and a conditioning scale of 4 was used for classifier- free guidance. The noise input to the model was randomly generated for each baseline image, but was kept constant when generating multiple unseen views from a single baseline image (e.g., in mode B or in few-shot mode A). Keeping the noise vector constant and using a deterministic approach for running inference with DDIM ensured that multiple generations from a single baseline radiograph were more consistent with each other.

[0100] Mode A is cost-effective and fast. On a single Al 00 NVIDIA Tensor Core Graphics Processing Unit (GPU), it took 10 to 12 seconds to generate a novel view. This mode was therefore suitable for instances where the user needs to visualize a more desirable view of a baseline radiograph, estimating how much and in which axes the radiograph should be rotated. Conversely. Mode B was more expensive and time-consuming, requiring 300 generations to cover a range of -15° to +15° in each axis: 30 images generated at each angle, multiplied by ten interpolated generations for each 1° of angle shift. Using batch inference, generating all the frames for a single axis took, on average, 7 minutes. Therefore, Mode B was more suited for scenarios where the user prefers to visualize an entire range of rotations for observing certain anatomical (or pathological) features or to choose specific views from the many available.

[0101] Upon initial visual inspection, the DDPM displayed acceptable and consistent overall performance in both modes. The generated radiographs matched the anatomy of the baseline input radiograph, and the 3D rotation of various anatomical structures appeared reasonable and consistent to both reviewing orthopedic surgeons and radiologists. It was observed that the model learned all possible rotations up to ±15° in the x, y, or z axes. However, the model did not learn to extrapolate to any rotations beyond ±15°, such as 45° or 90°. This limitation would be overcome by training the model on a diverse set of input DRRs that include a more comprehensive range of ground truth rotations.

[0102] FIG. 11 illustrates an example of a system 1100 for generating synthetic radiographic images and / or 3D model reconstructions in accordance with some embodiments of the systems and methods described in the present disclosure. As shown in FIG. 11, a computing device 1150 can receive one or more types of data (e g., radiographic image data, CT image data, x-ray fluoroscopy image data, DRRs. MIPs, other medical image data, imagetransformation data, spatial position data, radiographic fingerprints, etc.) from data source 1102. In some embodiments, computing device 1150 can execute at least a portion of a radiographic image generative Al system 1104 to generate radiographic fingerprints, view- corrected radiographic image data, and / or 3D anatomical models from data received from the data source 1102.

[0103] Additionally or alternatively, in some embodiments, the computing device 1150 can communicate information about data received from the data source 1102 to a server 1152 over a communication network 1154, which can execute at least a portion of the radiographic image generative Al system 1104. In such embodiments, the server 1152 can return information to the computing device 1150 (and / or any other suitable computing device) indicative of an output of the radiographic image generative Al system 1104.

[0104] In some embodiments, computing device 1150 and / or server 1152 can be any suitable computing device or combination of devices, such as a desktop computer, a laptop computer, a smartphone, a tablet computer, a wearable computer, a server computer, a virtual machine being executed by a physical computing device, and so on. The computing device 1 150 and / or server 1 152 can also reconstruct images from the data.

[0105] In some embodiments, data source 1102 can be any suitable source of data (e.g., measurement data, images reconstructed from measurement data, processed image data), such as an x-ray imaging system, an x-ray fluoroscopy system, a CT imaging system, another computing device (e.g., a server storing measurement data, images reconstructed from measurement data, processed image data), and so on. In some embodiments, data source 1102 can be local to computing device 1150. For example, data source 1102 can be incorporated with computing device 1150 (e.g., computing device 1150 can be configured as part of a device for measuring, recording, estimating, acquiring, or otherwise collecting or storing data). As another example, data source 1102 can be connected to computing device 1150 by a cable, a direct wireless link, and so on. Additionally or alternatively, in some embodiments, data source 1102 can be located locally and / or remotely from computing device 1150, and can communicate data to computing device 1150 (and / or server 1152) via a communication network (e.g., communication network 1154).

[0106] In some embodiments, communication network 1154 can be any suitable communication network or combination of communication networks. For example, communication network 1154 can include a Wi-Fi network (which can include one or more wireless routers, one or more switches, etc.), a peer-to-peer network (e.g., a Bluetoothnetwork), a cellular network (e.g., a 3G network, a 4G network, etc., complying with any suitable standard, such as CDMA. GSM, LTE, LTE Advanced. WiMAX, etc.), other types of wireless network, a wired network, and so on. In some embodiments, communication network 1154 can be a local area network, a wide area network, a public network (e.g., the Internet), a private or semi -private network (e.g., a corporate or university intranet), any other suitable ty pe of network, or any suitable combination of networks. Communications links shown in FIG. 11 can each be any suitable communications link or combination of communications links, such as wired links, fiber optic links, Wi-Fi links, Bluetooth links, cellular links, and so on.

[0107] Referring now to FIG. 12, an example of hardware 1200 that can be used to implement data source 1102, computing device 1150, and server 1152 in accordance with some embodiments of the systems and methods described in the present disclosure is shown.

[0108] As shown in FIG. 12, in some embodiments, computing device 1150 can include a processor 1202, a display 1204, one or more inputs 1206, one or more communication systems 1208, and / or memory 1210. In some embodiments, processor 1202 can be any suitable hardware processor or combination of processors, such as a central processing unit ('’CPU"), a graphics processing unit (“GPU’’), and so on. In some embodiments, display 1204 can include any suitable display devices, such as a liquid crystal display (“LCD”) screen, a light-emitting diode (“LED”) display , an organic LED (“OLED”) display, an electrophoretic display (e.g., an “e-ink” display), a computer monitor, a touchscreen, a television, and so on. In some embodiments, inputs 1206 can include any suitable input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, and so on.

[0109] In some embodiments, communications systems 1208 can include any suitable hardware, firmware, and / or software for communicating information over communication network 1154 and / or any other suitable communication networks. For example, communications systems 1208 can include one or more transceivers, one or more communication chips and / or chip sets, and so on. In a more particular example, communications systems 1208 can include hardware, firmware, and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, and so on.

[0110] In some embodiments, memory' 1210 can include any suitable storage device or devices that can be used to store instructions, values, data, or the like, that can be used, for example, by processor 1202 to present content using display 1204, to communicate with server1152 via communications system(s) 1208, and so on. Memory 1210 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 1210 can include random-access memory (“RAM”), read-only memory (“ROM”), electrically programmable ROM (“EPROM”), electrically erasable ROM (“EEPROM”), other forms of volatile memory, other forms of non-volatile memory, one or more forms of semi-volatile memory’, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, and so on. In some embodiments, memory 1210 can have encoded thereon, or otherwise stored therein, a computer program for controlling operation of computing device 1150. In such embodiments, processor 1202 can execute at least a portion of the computer program to present content (e g., images, user interfaces, graphics, tables), receive content from server 1152, transmit information to server 1152, and so on. For example, the processor 1202 and the memory 1210 can be configured to perform the methods described herein (e.g., the method of FIG. 2, the method of FIG. 3, the method of FIG. 5, the method of FIG. 6, the method of FIG. 7, the method of FIG. 8).

[0111] In some embodiments, server 1152 can include a processor 1212, a display 1214, one or more inputs 1216, one or more communications systems 1218, and / or memory 1220. In some embodiments, processor 1212 can be any suitable hardware processor or combination of processors, such as a CPU, a GPU, and so on. In some embodiments, display- 1214 can include any suitable display devices, such as an LCD screen, LED display, OLED display, electrophoretic display, a computer monitor, a touchscreen, a television, and so on. In some embodiments, inputs 1216 can include any suitable input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, and so on.

[0112] In some embodiments, communications systems 1218 can include any suitable hardware, firmware, and / or software for communicating information over communication network 1154 and / or any other suitable communication networks. For example, communications systems 1218 can include one or more transceivers, one or more communication chips and / or chip sets, and so on. In a more particular example, communications systems 1218 can include hardware, firmware, and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, and so on.

[0113] In some embodiments, memory- 1220 can include any suitable storage device or devices that can be used to store instructions, values, data, or the like, that can be used, forexample, by processor 1212 to present content using display 1214. to communicate with one or more computing devices 1150, and so on. Memory 1220 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 1220 can include RAM, ROM, EPROM, EEPROM, other types of volatile memory, other types of non-volatile memory, one or more types of semi-volatile memory, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, and so on. In some embodiments, memory 1220 can have encoded thereon a server program for controlling operation of server 1 152. In such embodiments, processor 1212 can execute at least a portion of the server program to transmit information and / or content (e.g., data, images, a user interface) to one or more computing devices 1150, receive information and / or content from one or more computing devices 1150, receive instructions from one or more devices (e.g., a personal computer, a laptop computer, a tablet computer, a smartphone), and so on.

[0114] In some embodiments, the server 1152 is configured to perform the methods described in the present disclosure. For example, the processor 1212 and memory 1220 can be configured to perform the methods described herein (e.g., the method of FIG. 2, the method of FIG. 3, the method of FIG. 5, the method of FIG. 6, the method of FIG. 7, the method of FIG. 8).

[0115] In some embodiments, data source 1102 can include a processor 1222, one or more data acquisition systems 1224, one or more communications systems 1226, and / or memory 1228. In some embodiments, processor 1222 can be any suitable hardware processor or combination of processors, such as a CPU, a GPU, and so on. In some embodiments, the one or more data acquisition systems 1224 are generally configured to acquire data, images, or both, and can include an x-ray imaging system, an x-ray fluoroscopy system, a CT imaging system, and so on. Additionally or alternatively, in some embodiments, the one or more data acquisition systems 1224 can include any suitable hardware, firmware, and / or software for coupling to and / or controlling operations of an x-ray imaging system, an x-ray fluoroscopy system, a CT imaging system, and so on. In some embodiments, one or more portions of the data acquisition system(s) 1224 can be removable and / or replaceable.

[0116] Note that, although not shown, data source 1102 can include any suitable inputs and / or outputs. For example, data source 1102 can include input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, a trackpad, a trackball, and so on. As another example, data source 1102 can include any suitabledisplay devices, such as an LCD screen, an LED display, an OLED display, an electrophoretic display, a computer monitor, a touchscreen, a television, etc., one or more speakers, and so on.

[0117] In some embodiments, communications systems 1226 can include any suitable hardware, firmware, and / or software for communicating information to computing device 1150 (and, in some embodiments, over communication network 1154 and / or any other suitable communication networks). For example, communications systems 1226 can include one or more transceivers, one or more communication chips and / or chip sets, and so on. In a more particular example, communications systems 1226 can include hardware, firmware, and / or software that can be used to establish a wired connection using any suitable port and / or communication standard (e.g., VGA, DVI video, USB, RS-232, etc.), Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, and so on.

[0118] In some embodiments, memory 1228 can include any suitable storage device or devices that can be used to store instructions, values, data, or the like, that can be used, for example, by processor 1222 to control the one or more data acquisition systems 1224, and / or receive data from the one or more data acquisition systems 1224; to generate images from data; present content (e.g., data, images, a user interface) using a display; communicate with one or more computing devices 1150; and so on. Memory 1228 can include any suitable volatile memory7, non-volatile memory7, storage, or any suitable combination thereof. For example, memory 1228 can include RAM, ROM. EPROM, EEPROM, other types of volatile memory, other ty pes of non-volatile memory, one or more types of semi-volatile memory, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, and so on. In some embodiments, memory71228 can have encoded thereon, or otherwise stored therein, a program for controlling operation of data source 1102. In such embodiments, processor 1222 can execute at least a portion of the program to generate images, transmit information and / or content (e.g., data, images, a user interface) to one or more computing devices 1150, receive information and / or content from one or more computing devices 1150, receive instructions from one or more devices (e.g., a personal computer, a laptop computer, a tablet computer, a smartphone, etc.), and so on.

[0119] In some embodiments, any suitable computer-readable media can be used for storing instructions for performing the functions and / or processes described herein. For example, in some embodiments, computer-readable media can be transitory7or non-transitory. For example, non-transitory computer-readable media can include media such as magnetic media (e.g., hard disks, floppy disks), optical media (e.g., compact discs, digital video discs.Blu-ray discs), semiconductor media (e.g., RAM, flash memory', EPROM, EEPROM), any suitable media that is not fleeting or devoid of any semblance of permanence during transmission, and / or any suitable tangible media. As another example, transitory computer- readable media can include signals on networks, in wires, conductors, optical fibers, circuits, or any suitable media that is fleeting and devoid of any semblance of permanence during transmission, and / or any suitable intangible media.

[0120] As used herein in the context of computer implementation, unless otherwise specified or limited, the terms “component,” “system,” “module,” “framework,” and the like are intended to encompass part or all of computer-related systems that include hardware, software, a combination of hardware and software, or software in execution. For example, a component may be, but is not limited to being, a processor device, a process being executed (or executable) by a processor device, an object, an executable, a thread of execution, a computer program, or a computer. By way of illustration, both an application running on a computer and the computer can be a component. One or more components (or system, module, and so on) may reside within a process or thread of execution, may be localized on one computer, may be distributed between two or more computers or other processor devices, or may be included within another component (or system, module, and so on).

[0121] In some implementations, devices or sy stems disclosed herein can be utilized or installed using methods embodying aspects of the disclosure. Correspondingly, description herein of particular features, capabilities, or intended purposes of a device or system is generally intended to inherently include disclosure of a method of using such features for the intended purposes, a method of implementing such capabilities, and a method of installing disclosed (or otherwise known) components to support these purposes or capabilities. Similarly, unless otherwise indicated or limited, discussion herein of any method of manufacturing or using a particular device or system, including installing the device or system, is intended to inherently include disclosure, as embodiments of the disclosure, of the utilized features and implemented capabilities of such device or system.

[0122] The present disclosure has described one or more preferred embodiments, and it should be appreciated that many equivalents, alternatives, variations, and modifications, aside from those expressly stated, are possible and within the scope of the invention.

Claims

CLAIMS1. A method for generating a radiographic fingerprint from a medical image of a subject, the method comprising:(a) accessing a medical image of a subject with a computer system;(b) accessing a machine learning model with the computer system, wherein the machine learning model has been trained on training data to encode anatomical features in medical image data as an embedding vector; and(c) generating a radiographic fingerprint for the medical image of the subject by inputting the medical image to the machine learning model using the computer system, generating an embedding that encodes anatomical features of the subject in the medical image, wherein the embedding is stored by the computer system as the radiographic fingerprint.

2. The method of claim 1 , wherein the machine learning model comprises a contrastive model.

3. The method of claim 1, wherein the machine learning model has been trained on the training data using contrastive learning.

4. The method of claim 1 , wherein the medical image comprises a radiographic image.

5. The method of claim 4. further comprising: accessing a generative model with the computer system, wherein the generative model has been trained on training data to convert a radiographic image from a first projection plane to a second projection plane; and applying the radiographic image and the radiographic fingerprint to the generative model using the computer system, generating as an output a view-corrected radiographic image that depicts the subject in the second projection plane.

6. The method of claim 1 , wherein the medical image comprises a digitally reconstructed radiograph (DRR).

7. The method of claim 6. further comprising: accessing a generative model with the computer system, wherein the generative model has been trained on training data to convert a DRR from a first projection plane to a second projection plane; and applying the DRR and the radiographic fingerprint to the generative model using the computer system, generating as an output a view-corrected DRR that depicts the subject in the second projection plane.

8. A method for generating a view-corrected radiographic image of a subject, the method comprising:(a) accessing a radiographic image with a computer system, wherein the radiographic image depicts the subject in a first view;(b) accessing a generative model with the computer system, wherein the generative model has been trained on training data to convert a medical image from a first projection plane to a second projection plane; and(c) applying the radiographic image to the generative model using the computer system, generating as an output a view-corrected radiographic image that depicts the subject in a second view that is different from the first view.

9. The method of claim 8, wherein applying the radiographic image to the generative model comprises converting the radiographic image to a digitally reconstructed radiograph (DRR) and inputting the DRR to the generative model.

10. The method of claim 9, wherein generating the view-corrected radiographic image as the output comprises generating a view-corrected DRR by inputting the DRR to the generative model and converting the view-corrected DRR to the view-corrected radiographic image using the computer system.

11. The method of claim 8. comprising accessing image transformation data that defines a transformation from the first view to the second view and inputting the image transformation data to the generative model.

12. The method of claim 8, comprising accessing a radiographic fingerprint associated with the radiographic image and inputting the radiographic fingerprint to thegenerative model, wherein the radiographic fingerprint comprises an embedding vector that encodes anatomical features in the radiographic image.

13. The method of claim 8, wherein the generative model comprises a denoising diffusion probabilistic model (DDPM).

14. A method for generating a three-dimensional (3D) model of a portion of a subject’s anatomy from at least two radiographic images of the subject, the method comprising:(a) accessing radiographic image data with a computer system, wherein the radiographic image data comprises at least two radiographic images of the subject;(b) accessing a generative model with the computer system, wherein the generative model has been trained on training data to generate maximum intensity projections (MIPs) from a radiographic image;(c) inputting the radiographic image data to the machine learning model, generating a plurality of MIPs corresponding to a plurality of spatial positions; and(d) generating the 3D model with the computer system by constructing a neural radiance field (NeRF) model from the plurality of MIPs. wherein the NeRF model represents the 3D model of the portion of the subject's anatomy.

15. The method of claim 14, wherein the generative model comprises a denoising diffusion probabilistic model (DDPM).

16. The method of claim 14, comprising accessing radiographic fingerprint data associated with the radiographic image data and inputting the radiographic fingerprint data to the generative model, wherein the radiographic fingerprint data comprises embedding vectors that encode anatomical features in the radiographic image data.

17. The method of claim 14, wherein the 3D model depicts bony structures in the subject.

Images