3D model reconstruction

Abstract

The present disclosure relates to systems, devices and methods for reconstructing three-dimensional models of anatomical structures using two-dimensional images.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 295,518, filed December 31, 2021, and entitled “THREE-DIMENSIONAL MODEL RECONSTRUCTION,” the disclosure of which is incorporated by reference in its entirety herein. [Background technology]

[0002] Various medical procedures involve the use of one or more devices configured to penetrate the human anatomy to reach a treatment site. Certain operational processes can include locating the medical instrument within a patient and visualizing an area of ​​interest within the patient. To do so, many medical instruments may include sensors to track the location of the instrument and may include vision capabilities, such as an embedded camera or compatible use with a visual probe. [Brief description of the drawings]

[0003] Various embodiments are illustrated in the accompanying drawings for illustrative purposes and should not be construed as limiting the scope of the present disclosure in any way. In addition, various features of different disclosed embodiments may be combined to form further embodiments that are part of the present disclosure. Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. [Figure 1] 1 illustrates an exemplary medical system for performing various medical procedures, according to aspects of the present disclosure. [Diagram 2] FIG. 1 illustrates a reconstructed three-dimensional model in accordance with one or more embodiments. [Diagram 3] 2 is a block diagram illustrating an example data flow of the model reconstruction module of FIG. 1 in accordance with an example embodiment. [Figure 4] 1 is a flowchart illustrating a method for reconstructing a three-dimensional model from two-dimensional images acquired during or as part of a medical procedure, in accordance with an illustrative embodiment. [Diagram 5] FIG. 1 is a system diagram illustrating a neural network generation system in accordance with an exemplary embodiment. [Figure 6] 1 is a flowchart illustrating a method for generating a trained neural network that can be used to reconstruct a three-dimensional model from a set of two or more two-dimensional images, in accordance with an example embodiment. [Figure 7] FIG. 1 is a system diagram illustrating a personalized neural network generation system, according to an exemplary embodiment. [Figure 8] FIG. 8 illustrates a reconstructed three-dimensional model 800 in accordance with an example embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] The directions provided herein are for convenience only and do not necessarily affect the scope or meaning of the disclosure. Although certain exemplary embodiments are disclosed below, the subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as to modifications and equivalents thereof. Thus, the scope of claims that may arise from this specification is not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described in sequence as multiple separate operations in a manner that may be helpful in understanding a particular embodiment, however, the order of description should not be construed to imply that these operations are order dependent. Furthermore, structures, systems, and / or devices described herein may be embodied as integrated or separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are realized by any particular embodiment. Thus, for example, various embodiments may be performed in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or suggested herein.

[0005] overview The present disclosure relates to systems, devices and methods for generating three-dimensional models of anatomical structures. Many medical procedures rely on accurate representation of a patient's anatomy and guidance in controlling instruments within that anatomy. For example, in percutaneous access kidney stone removal, accurate and safe stone removal may depend on the selection of the calyx where the percutaneous instrument enters the kidney. The selected calyx should not only provide easy access to the stone but also ensure that the needle trajectory from the patient's skin to the renal collecting system is consistent with the patient's anatomy. The selection of the calyx may include analysis of preoperative computed tomography (CT) images and intraoperative fluoroscopic images. Preoperative CT images are often acquired without contrast and therefore can only visualize stone location relative to the kidney volume and are not sufficient to visualize renal collecting system morphology. Intraoperative fluoroscopic images, which may have a relatively high resolution, may be acquired with contrast and reveal the relative position of the calyx. However, fluoroscopic images are two-dimensional and it may be difficult to understand the volumetric kidney morphology from single / multiple fluoroscopic views.

[0006] The embodiments described herein are capable of reconstructing a three-dimensional model of anatomy from a limited set of two-dimensional images.

[0007] 3D Pose Estimation System FIG. 1 illustrates an exemplary medical system 100 for performing various medical procedures according to an embodiment of the present disclosure. The medical system 100 includes a robotic system 110 configured to engage and / or control a medical instrument 120 and perform a procedure on a patient 130. The medical system 100 also includes a control system 140 configured to interface with the robotic system 110, provide information regarding the procedure, and / or perform various other operations. For example, the control system 140 may include a display 142 that presents certain information to assist a physician 160. The display(s) 142 may be a monitor, a screen, a television, virtual reality hardware, augmented reality hardware, a three-dimensional imaging device (e.g., a holographic device), or the like, or a combination thereof. The medical system 100 may include a platform 150 configured to hold the patient 130. The system 100 may further include an electromagnetic (EM) field generator 180, which may be held by one or more robotic arms 112 of the robotic system 110 or may be a standalone device. In an embodiment, the medical system 100 may also include an imaging device 190 that may be integrated into a C-arm and / or configured to provide imaging during a procedure, such as a fluoroscopy-type procedure.

[0008] In some implementations, the medical system 100 can be used to perform a percutaneous procedure. For example, if the patient 130 has a kidney stone that is too large to be removed through the urinary tract, the physician 160 can perform a procedure to remove the kidney stone through a percutaneous access point on the patient 130. To illustrate, the physician 160 can interact with the control system 140 to control the robotic system 110 to advance and navigate the medical instrument 120 (e.g., a scope) from the urethra through the bladder, up the ureter, and into the kidney where the stone is located. The control system 140 can provide information about the medical instrument 120 via the display 142, such as real-time images captured with it, to assist the physician 160 in navigating the medical instrument 120.

[0009] Upon reaching the location of the kidney stone (e.g., within the calyx), the medical instrument 120 can be used to designate / tag a target location (e.g., a desired point for accessing the kidney) for the medical instrument 170 (e.g., a needle) to percutaneously access the kidney. To minimize damage to the kidney and / or its surrounding anatomical structures, the physician 160 can designate a particular papilla as the target location for entering the kidney with the medical instrument 170. However, other target locations can be designated or determined. To assist the physician in driving the medical instrument 170 through a particular papilla and into the patient 130, the control system 140 can provide a visualization interface 144, which can include a rendering of a three-dimensional model of the anatomical structure generated based on two-dimensional images captured by the system 100, such as fluoroscopic images. As will be described in more detail, the visualization interface 144 can provide the operator with information to aid in driving the medical instrument 170 to the target location.

[0010] Once the medical instrument 170 reaches the target location, the physician 160 may use the medical instrument 170 and / or another medical instrument to remove the kidney stone from the patient 130. One such instrument may be a percutaneous catheter. The percutaneous catheter, like the instrument 120, may be an instrument with steering capabilities, but in some embodiments may lack a dedicated camera or position sensor. Some embodiments may use the augmented visualization interface 144 to render augmented images that are useful for navigating the percutaneous catheter within the anatomy.

[0011] Although the above percutaneous and / or other procedures have been discussed in the context of using the medical instrument 120, in some implementations, the percutaneous procedures may be performed without the assistance of the medical instrument 120. Additionally, the medical system 100 may be used to perform a variety of other procedures.

[0012] Additionally, while many embodiments are described as the physician 160 using the medical instrument 170, the medical instrument 170 may alternatively be used by components of the medical system 100. For example, the medical instrument 170 may be held / manipulated by the robotic system 110 (e.g., one or more robotic arms 112), and techniques discussed herein may be implemented to control the robotic system 110 to insert the medical instrument 170 in the proper pose (or aspect of the pose, such as orientation or placement) to reach the target location.

[0013] In the example of FIG. 1 , medical instrument 120 is implemented as a scope and medical instrument 170 is implemented as a needle. Thus, for ease of discussion, medical instrument 120 is referred to as a "scope 120" or a "luminal-based medical instrument 120" and medical instrument 170 is referred to as a "needle 170" or a "percutaneous medical instrument 170". However, medical instrument 120 and medical instrument 170 may each be implemented as any suitable type of medical instrument, including, for example, a scope (sometimes referred to as an "endoscope"), a needle, a catheter, a guidewire, a nephrolithotomy device, a basket retrieval device, a forceps, a vacuum, a needle, a scalpel, an imaging probe, a jaw, a scissors, a grasper, a needle holder, a micro-dissection instrument, a staple applier, a tacker, an aspirating / irrigation tool, a clip applier, and the like. In some embodiments, the medical instrument is a steerable device, and in other embodiments, the medical instrument is a non-steerable device. In some embodiments, a surgical tool refers to a device configured to puncture or be inserted through a body structure, such as a needle, scalpel, guidewire, etc. However, a surgical tool can also refer to other types of medical instruments.

[0014] In some embodiments, the medical instrument, such as the scope 120 and / or the needle 170, includes a sensor configured to generate sensor data that can be transmitted to another device. In examples, the sensor data can indicate the location / orientation of the medical instrument and / or can be used to determine the location / orientation of the medical instrument. For example, the sensor can include an electromagnetic (EM) sensor having a coil of conductive material. Here, an EM field generator, such as EM field generator 180, can provide an EM field that is detected by the EM sensor on the medical instrument. The magnetic field can induce a small current in the coil of the EM sensor, which can be analyzed to determine the distance and / or angle / orientation between the EM sensor and the EM field generator. Additionally, the medical instrument can include other types of sensors configured to generate sensor data, such as any one or more of a camera, range sensor, radar device, shape-sensing fiber, accelerometer, gyroscope, satellite-based positioning sensor (e.g., global positioning system, GPS), radio frequency transceiver, etc. In some embodiments, the sensor is positioned on the distal end of the medical instrument, while in other embodiments, the sensor is positioned elsewhere on the medical instrument. In some embodiments, sensors on the medical instrument can provide sensor data to the control system 140, which can implement one or more localization techniques to determine / track the position and / or orientation of the medical instrument.

[0015] In some embodiments, the medical system 100 may record or otherwise track runtime data generated during a medical procedure. This runtime data may be referred to as system data. For example, the medical system 100 may track or otherwise record sensor readings (e.g., sensor data) from instruments (e.g., the scope 120 and needle 170) in a data store 145A (e.g., a computer storage system such as a computer-readable memory, a database, a file system, etc.). In addition to sensor data, the medical system 100 may store other types of system data in the data store 145A. For example, in the context of FIG. 1 , system data can further include a time series of video images captured by the scope 120, a status of the robotic system 110, command data from I / O device(s) (e.g., I / O device(s) 146 described below), audio data (which may be captured by an audio capture device embedded within the medical system 100, such as a microphone on a medical instrument, a robotic arm, or elsewhere in the medical system), image data from an external (relative to the patient) imaging device (an RGB camera, a LIDAR imaging sensor, a fluoroscopic imaging sensor, etc.) and imaging device 190, etc.

[0016] 1, control system 140 includes model reconstruction module 141, which may include control circuitry that operates on two-dimensional image data stored in system data and case data store 145 to generate a reconstructed three-dimensional image of the anatomical structure using two or more two-dimensional images. As discussed in more detail below, model reconstruction module 141 may employ machine learning techniques to generate a three-dimensional volumetric model of the anatomical structure using a trained network and the two or more two-dimensional images.

[0017] The terms "scope" or "endoscope" are used herein according to their broad and ordinary meanings and may refer to any type of elongated medical instrument having imaging, viewing, and / or capture capabilities and configured to be introduced into any type of organ, cavity, lumen, chamber, and / or space in the body. For example, references herein to a scope or endoscope may refer to a ureteroscope (e.g., for accessing the urinary tract), a laparoscope, a nephroscope (e.g., for accessing the kidneys), a bronchoscope (e.g., for accessing the airways such as the bronchi), a colonoscope (e.g., for accessing the colon), an arthroscope (e.g., for accessing a joint), a cystoscope (e.g., for accessing the bladder), a borescope, etc.

[0018] The scope may comprise a tubular and / or flexible medical instrument configured to be inserted into a patient's anatomy to capture images of the anatomy. In some embodiments, the scope may house wires and / or optical fibers to transfer signals between the optical assembly and the distal end of the scope, and the scope may include an imaging device, such as an optical camera. The camera / imaging device may be used to capture images of an internal anatomical space, such as a target calyx / papilla of the liver. The scope may be further configured to house optical fibers to carry light from a proximally located light source, such as a light emitting diode, to the distal end of the scope. The distal end of the scope may include a port for a light source to illuminate the anatomical space when using the camera / imaging device. In some embodiments, the scope is configured to be controlled by a robotic system, such as the robotic system 110. The imaging device may comprise optical fibers, fiber arrays, and / or lenses. The optical components may move with the tip of the scope, such that movement of the tip of the scope results in a change in the image captured by the imaging device.

[0019] The scope may be articulatable, such as with respect to at least a distal portion of the scope, so that the scope may be maneuvered within the human anatomy. In some embodiments, the scope is configured to be articulated, for example, with 5 or 6 degrees of freedom, including, for example, X, Y, Z coordinate translation, as well as pitch, yaw, and roll. The position sensors of the scope may similarly have similar degrees of freedom with respect to the position information they create / provide. The scope may include telescopic parts, such as an inner leader portion and an outer sheath portion, which may be manipulated to telescopically extend the scope. The scope may, in some cases, comprise a rigid or flexible tube and may be sized to pass through an outer sheath, catheter, introducer, or other luminal device, or may be used without such a device. In some embodiments, the scope includes a working channel for deploying medical instruments (e.g., lithotriptors, basket devices, forceps, etc.), irrigation, and / or suction to the working area at the distal end of the scope.

[0020] The robotic system 110 can be configured to at least partially facilitate the performance of a medical procedure. The robotic system 110 can be arranged in various ways depending on the particular procedure. The robotic system 110 can include one or more robotic arms 112 configured to engage and / or control the scope 120 to perform the procedure. As shown, each robotic arm 112 can include multiple arm segments coupled to joints, thereby providing multiple degrees of mobility. In the example of FIG. 1, the robotic system 110 is positioned proximate to a leg of the patient 130, and the robotic arm 112 is actuated to engage and position the scope 120 for access to an access point, such as the urethra of the patient 130. Once the robotic system 110 is properly positioned, the scope 120 can be inserted into the patient 130 robotically using the robotic arm 112, manually by the physician 160, or a combination thereof. The robotic arm 112 can also be connected to an EM field generator 180, which can be positioned near the treatment site, such as within proximate range of the kidney of the patient 130.

[0021] The robotic system 110 may also include a support structure 114 coupled to the one or more robotic arms 112. The support structure 114 may include control electronics / circuitry, one or more power sources, one or more pneumatics, one or more light sources, one or more actuators (e.g., motors to move the one or more robotic arms 112), memory / data storage, and / or one or more communication interfaces. In some embodiments, the support structure 114 includes input / output (I / O) device(s) 116 configured to receive input, such as user input for controlling the robotic system 110, and / or provide output, such as a graphical user interface (GUI), information about the robotic system 110, information about the procedure, etc. The I / O device(s) 116 may include a display, a touch screen, a touch pad, a projector, a mouse, a keyboard, a microphone, a speaker, etc. In some embodiments, the robotic system 110 is mobile (e.g., the support structure 114 includes wheels) so that the robotic system 110 can be positioned wherever appropriate or desired for the procedure. In other embodiments, the robotic system 110 is a fixed system. Additionally, in some embodiments, the robotic system 112 is integrated into the table 150.

[0022] The robotic system 110 can be coupled to any of the components of the medical system 100, such as the control system 140, the platform 150, the EM field generator 180, the scope 120, and / or the needle 170. In some embodiments, the robotic system is communicatively coupled to the control system 140. In one example, the robotic system 110 can be configured to receive control signals from the control system 140 to perform actions, such as positioning the robotic arm 112 in a particular manner, manipulating the scope 120, etc. In response, the robotic system 110 can control the components of the robotic system 110 to perform the actions. In another example, the robotic system 110 can be configured to receive images from the scope 120 depicting the internal anatomy of the patient 130 and / or transmit the images to the control system 140, which can then be displayed on the display(s) 142. Additionally, in some embodiments, the robotic system 110 is coupled to components of the medical system 100, such as the control system 140, in a manner such that the robotic system 110 can receive fluids, optics, power, etc. from the components. Exemplary details of the robotic system 110 are discussed in further detail below with reference to FIG.

[0023] The control system 140 can be configured to provide various functions to assist in performing a medical procedure. In some embodiments, the control system 140 can be coupled to the robotic system 110 and operate in cooperation with the robotic system 110 to perform a medical procedure on the patient 130. For example, the control system 140 can communicate with the robotic system 110 via a wireless or wired connection (e.g., to control the robotic system 110 and / or the scope 120, receive images captured by the scope 120, etc.), provide fluids to the robotic system 110 via one or more fluid channels, provide power to the robotic system 110 via one or more electrical connections, provide optics to the robotic system 110 via one or more optical fibers or other components, etc. Additionally, in some embodiments, the control system 140 can communicate with the needle 170 and / or the scope 170 to receive sensor data from the needle 170 and / or the endoscope 120 (via the robotic system 110 and / or directly from the needle 170 and / or the endoscope 120). Additionally, in some embodiments, the control system 140 may be in communication with the table 150 to place the table 150 in a particular orientation or otherwise control the table 150. Additionally, in some embodiments, the control system 140 may be in communication with the EM field generator 180 to control the generation of the EM field around the patient 130.

[0024] The control system 140 includes various I / O devices configured to assist the physician 160 or others in performing a medical procedure. In this example, the control system 140 includes I / O devices 146 that are used by the physician 160 or other user to control the scope 120, such as to navigate the scope 120 within the patient 130. For example, the physician 160 can provide input via the I / O devices 146, and in response, the control system 140 can send control signals to the robotic system 110 to operate the scope 120. Although the I / O device(s) 146 are illustrated as controllers in the example of FIG. 1, the I / O device(s) 146 may be implemented as various types of I / O devices, such as a touch screen, a touch pad, a mouse, a keyboard, a surgeon's or physician's console, virtual reality hardware, augmented hardware, a microphone, a speaker, a haptic device, etc.

[0025] As also shown in FIG. 1 , the control system 140 can include a display 142 to provide various information regarding the procedure. As described above, the display(s) 142 can present a visualization interface 144 to assist the physician 160 in the percutaneous access procedure (e.g., maneuvering the needle 170 toward the target site). The display(s) 142 can also provide information regarding the scope 120 (e.g., via the qualification interface 144 and / or another interface). For example, the control system 140 can receive real-time images captured by the scope 120 and display the real-time images via the display(s) 142. Additionally or alternatively, the control system 140 can receive signals (e.g., analog, digital, electrical, acoustic / sonic, pneumatic, tactile, hydraulic, etc.) from medical monitors and / or sensors associated with the patient 130, and the display(s) 142 can present information regarding the health or environment of the patient 130. Such information may include information displayed via a medical monitor, including, for example, heart rate (e.g., ECG, HRV, etc.), blood pressure / rate, muscle biosignals (e.g., EMG), body temperature, blood oxygen saturation (e.g., SpO2), CO2, brain waves (e.g., EEG), environmental and / or local or core body temperature, etc.

[0026] To facilitate the functioning of the control system 140, the control system 140 may include various components (sometimes referred to as "subsystems"). For example, the control system 140 may include control electronics / circuitry, as well as one or more power sources, pneumatics, light sources, actuators, memory / data storage devices, and / or communication interfaces. In some embodiments, the control system 140 includes control circuitry, including a computer-based control system configured to store executable instructions that, when executed, cause various operations to be implemented. In some embodiments, as shown in FIG. 1, the control system 140 is mobile, while in other embodiments, the control system 140 is a stationary system. Although various functions and components are discussed as being implemented by the control system 140, any of the functions and / or components may be integrated into and / or performed by other systems and / or devices, such as the robotic system 110, the platform 150, and / or the EM generator 180 (or even the scope 120 and / or the needle 170). Exemplary details of the control system 140 are discussed in more detail below with reference to FIG. 13.

[0027] Imaging device 190 may be configured to capture / generate one or more images of patient 130 during a procedure, such as one or more X-ray or CT images. In an embodiment, images from imaging device 190 may be provided in real time to view anatomical structures and / or medical instruments, such as scope 120 and / or needle 170, within patient 130 to assist physician 160 in performing a procedure. Imaging device 190 may be used to perform fluoroscopy (e.g., with a contrast agent within patient 130) or another type of imaging technique.

[0028] The various components of the medical system 100 may be communicatively coupled to one another via a network, which may include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), Internet area networks (IANs), cellular networks, the Internet, etc. Additionally, in some embodiments, the components of the medical system 100 are connected for data communication, fluid / gas exchange, power exchange, etc., via one or more supporting cables, tubes, etc.

[0029] Although various techniques and systems are discussed as being implemented as a robotically-assisted procedure (e.g., a procedure that at least partially uses the medical system 100), these techniques and systems may be implemented in other procedures, such as fully robotic medical procedures, human-only procedures (e.g., not including a robotic system), etc. For example, the medical system 100 may be used to perform a procedure (e.g., a fully robotic procedure) without a physician holding / manipulating medical instruments. That is, medical instruments used during a procedure, such as the scope 120 and needle 170, may each be held / controlled by a component of the medical system 100, such as the robotic arm 112 of the robotic system 110.

[0030] 3D model reconstruction method and operation The details of the operation of the exemplary model reconstruction system are now described. The methods and operations disclosed herein are described with reference to the model reconstruction system 100 shown in Figure 1 and the modules and other components shown in Figures 2 and 3. However, it should be understood that the methods and operations may be performed by any of the components discussed herein, alone or in combination.

[0031] A model reconstruction system can generate a representation of a three-dimensional anatomical structure based on a relatively limited number of two-dimensional images acquired during or as part of a medical procedure. FIG. 3 is a block diagram illustrating an example data flow of the model reconstruction module 141 of FIG. 1, according to an example embodiment. As shown in FIG. 3, the model reconstruction module 141 receives as input two-dimensional images 310a-n and outputs a reconstructed model 330. The reconstruction module 320 may include a neural network previously trained on labeled images. Systems, methods, and apparatus for generating neural networks are described in more detail below. Some embodiments may additionally or alternatively include engineered or algorithmic solutions that process the two-dimensional images 310a-n to generate a reconstructed model 330 based on identifying or otherwise matching shape priors, calyces, Symantec templates, key points, and the like. The neural network can be trained to automatically recognize important anatomical landmarks, such as individual calyces, ureteropelvic junctions, in the two-dimensional images, thereby simplifying the recognition of renal anatomy. An alternative solution is to rely on clinical knowledge about the most common kidney anatomical structures. For each anatomical subcategory, examples of kidneys belonging to this subcategory are collected. For such examples, a principal component analysis is performed. The generated principal components will capture the main shape variability in the target kidney subcategory. Any linear combination of these components results in an example of a kidney belonging to the target subcategory. For a kidney observed on intraoperative fluoroscopy, an optimal linear combination of the principal components is searched. If such a combination accurately captures the kidney, the kidney belongs to the target subcategory. Otherwise, it belongs to the other subcategory. The principal component model for each common subcategory is fitted to the kidney fluoroscopy image in order to recognize which subcategory belongs to the kidney. According to the recognized kidney type, the appropriate nerve from module 141 is selected.Another solution may include single-shot estimation that exploits semantic connectivity of anatomical structures, for example, in the context of the kidney, the ureteropelvic junction may branch into a major calyx, which is connected to a minor calyx, etc. The reconstructed model 330 may represent the anatomical structures in a three-dimensional view.

[0032] FIG. 4 is a flow chart illustrating a method 400 of reconstructing a three-dimensional model from two-dimensional images acquired during or as part of a medical procedure, according to an exemplary embodiment. As FIG. 4 shows, the method 400 may begin at block 410, where the system 100 acquires a neural network trained with images from one or more computed tomography scans labeled according to parts of the anatomy. As used herein, a system or device may "acquire" data in any number of mechanisms, such as a push or pull model or access through local storage. For example, a neural network service (described below) may stream the neural network to the model reconstruction module 141 based on a determinable event or based on a schedule. In other examples, the model reconstruction module 141 may send a request for the neural network to the neural network service, and in response to the request, the neural network service may send the requested neural network to the reconstruction module 141. Additionally, some embodiments may maintain a local copy of the neural network or may acquire the neural network via a local request to a storage device.

[0033] In block 420, the system 100 may acquire a first fluoroscopic image of the patient's anatomy. In block 430, the system may acquire a second fluoroscopic image of the patient's anatomy. The first and second fluoroscopic images may capture images of the same anatomy but from different angles. For example, in some embodiments, the first fluoroscopic image may be acquired from a coronal renal projection. The coronal renal projection may provide the best visibility of the renal anatomy. The second fluoroscopic image may be acquired from a sagittal projection (lateral view) since the sagittal projection enhances the coronal projection. The angle between the two fluoroscopic images may be significantly in the interval from 75 degrees to 105 degrees. The fluoroscopic images acquired during blocks 420, 430 may be acquired during the procedure to visually determine the suitable calyx for percutaneous access.

[0034] In block 440, the system 100 generates a model of the anatomy using the neural network, the first fluoroscopic image, and the second fluoroscopic image. In some embodiments, the neural network may include a multi-pass recursive reconstruction neural network. The input to the neural network includes fluoroscopic images passed through two parallel encoder paths. The outputs of the two encoder paths are concatenated and then passed through a recursive unit. The recursive unit serves to unwrap the two-dimensional input into a three-dimensional volume that will contain a volumetric renal anatomy reconstruction. The output of the recursive unit may be passed through a decoder that populates the three-dimensional volume with probabilities to form the renal anatomy map. The resulting renal anatomy map is passed through a threshold to generate a resulting binary mask of the renal collection system. A rendering of the renal collection system (e.g., model) is displayed to a user in block 460.

[0035] It should be appreciated that in some embodiments, the input to the neural network may be data derived from the image data rather than the image data itself, for example, the input to the neural network may be a skeleton extracted from a pyelogram, or key points (calyx, ureteropelvic junction, etc.).

[0036] As mentioned above, some embodiments may utilize engineered solutions rather than neural networks. It should be understood that in these embodiments, engineered or algorithmic solutions are used rather than neural networks in the context of method 400. For example, as mentioned above, one engineered solution may include a principal component analysis. In this analysis, a 3D volume is reconstructed from a 2D image. It may be a vector of 10 2D landmarks that define the calyx for X samples acquired by system 100. Alternatively, it may be the outline of the kidney in a 2D fluoroscopic image defined by 100 2D points. Alternatively, it may be the surface of the kidney defined by 100k 3D points. When a new kidney volume is reconstructed from a 2D fluoroscopic image, the reconstruction is converted into a 3D mesh. This mesh is modeled by the most contributing components for each kidney subcategory. If there is a subcategory in which the reconstructed kidney mesh can be modeled with low error, the kidney is considered to belong to this subcategory. The most contributing components are used to refine the reconstruction.

[0037] Although the method 400 is discussed with respect to acquiring a first and a second fluoroscopic image, and the neural network is configured to operate on those first and second fluoroscopic images, other embodiments may acquire additional fluoroscopic images. For example, in some cases, it may be desirable to further improve the visibility of some calyces, and more fluoroscopic images may be acquired at different angles. The multi-pass recursive reconstruction neural network presented above can accommodate additional fluoroscopic images by changing the recursive unit from a two-to-many architecture, where the input is two fluoroscopic images, to a many-to-many architecture. Thus, these embodiments may allow a user to acquire any number of fluoroscopic images, with each new fluoroscopic image improving the quality of the resulting reconstruction until a required level of clarity and precision is achieved.

[0038] It should be appreciated that the embodiments described herein use two-dimensional imaging to reconstruct a three-dimensional model of an anatomical structure without the need to obtain a contrast-enhanced computed tomography scan. Thus, the embodiments described herein can provide visualization of an anatomical structure without requiring additional steps or procedures in the workflow, providing more efficient use of medical resources and providing a better experience for the patient.

[0039] Model Generation Module As mentioned above, referring to block 410 of FIG. 4, the system 100 can obtain a neural network trained by images from one or more computed tomography scans labeled according to parts of the anatomy. Also, as mentioned above, the neural network can be accessed via a neural network service. The concepts including neural network generation and neural network services will now be described in more detail. FIG. 5 is a system diagram illustrating a neural network generation system 500 according to an exemplary embodiment. The neural network generation system 500 can include network communication between the control system 140 of FIG. 1 and the neural network training system 510 via a network 520. The network 520 can include any suitable communication network that allows two or more computer systems to communicate with each other, which can include wireless and / or wired networks. Exemplary networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), Internet area networks (IANs), cellular networks, the Internet, and the like.

[0040] The neural network training system 510 may be a computer system configured to generate a neural network 512 that can be used to generate a reconstructed three-dimensional model of an anatomical structure from a two-dimensional image (e.g., a fluoroscopic image).

[0041] The neural network training system 510 may be coupled to a two-dimensional image database 530. The two-dimensional image database 530 may include annotated images of anatomical structures such as kidneys. For example, contrast-enhanced renal CT scans from a pool of patients may be collected and manually annotated by a trained human observer. As described in more detail below, the images and annotations from the CT scans may be modified to approximate the hardware and software of the medical system receiving the neural network. Additionally, in some embodiments, the neural network training system 510 may segment the annotated renal images based on patient or kidney features shown in the contrast-enhanced renal CT scans.

[0042] FIG. 6 is a flow chart illustrating a method 600 for generating a trained neural network that can be used to reconstruct a three-dimensional model from a set of two or more two-dimensional images, according to an example embodiment.

[0043] In block 610, the neural network training system 510 obtains a first set of two or more labeled two-dimensional images corresponding to a first patient. The first set of labeled two-dimensional images may be derived from a three-dimensional image of the first patient's anatomy. One example of a three-dimensional image may be a three-dimensional volume output of a CT scan. In this example, the neural network training system 510 may generate the first set of two-dimensional images by creating artificial fluoroscopic images from the three-dimensional volume of the CT scan. In some embodiments, for each CT scan, the neural network training system 510 may generate multiple to numerous artificial fluoroscopic images from the annotated CT database. By way of example and not limitation, the neural network training system 510 may generate a hundred or hundreds of fluoroscopic images from one orientation (e.g., coronal orientation) with random vibration and a hundred or hundreds of fluoroscopic images from another orientation (e.g., sagittal orientation) with random vibration. It should be understood that random vibrations (e.g., 15-20 degrees) can account for imperfections in patient positioning during the procedure and enrich the database of fluorescence CT kidney cases.

[0044] It should be understood that the neural network training system 510 may generate the artificial image in any suitable manner. For example, in one embodiment, a perspective projection of a three-dimensional image onto a two-dimensional plane is performed. In the context of a CT scan, this is one way to reconstruct a radiograph from a CT image. Another approach may be to take planar slices from the three-dimensional image. The slices can then be post-processed to more closely resemble a two-dimensional image that may be acquired by an imaging device.

[0045] In block 620, the neural network training system 510 obtains a second set of two or more labeled two-dimensional images corresponding to a second patient. Similar to block 610, the second set of labeled two-dimensional images may be derived from a three-dimensional image of anatomy, such as a three-dimensional volume output of a CT scan. The neural network training system 510 may generate the second set of two-dimensional images by creating artificial fluoroscopic images from the three-dimensional volume of the CT scan. In some embodiments, for each CT scan, the neural network training system 510 may generate multiple to numerous artificial fluoroscopic images from the annotated CT database. By way of example and not limitation, the neural network training system 510 may generate a hundred or hundreds of fluoroscopic images from one orientation (e.g., coronal orientation) with random vibration and a hundred or hundreds of fluoroscopic images from another orientation (e.g., sagittal orientation) with random vibration.

[0046] In block 630, the neural network training system 510 trains a neural network based on the first set of labeled two-dimensional images and the second set of labeled two-dimensional images to generate a trained neural network. As described above, the trained neural network may include a multi-pass recurrent reconstruction neural network. The trained network may be configured to receive two-dimensional images acquired from a medical system, which are then passed through two parallel encoder paths. The trained network is further configured to concatenate the outputs of the encoder paths and pass the concatenated output through a recursive unit. The recursive unit is responsible for unwrapping the two-dimensional input into a three-dimensional volume that includes a volumetric renal anatomical reconstruction. The trained neural network is configured to pass the output of the recursive unit through a decoder, which populates the three-dimensional volume with probabilities to form a renal anatomical structure map. Finally, the resulting map is passed through a threshold to generate a binary mask of the renal collection system.

[0047] It should be appreciated that in some embodiments, the input for training the neural network may be data derived from the image data rather than the image data itself, for example, the input to the neural network may be a skeleton extracted from a pyelogram, or key points (calyx, ureteropelvic junction, etc.).

[0048] At the end of method 600, in embodiments in which the neural network training system 510 is distributed from the control system 140, the neural network training system 510 can transmit the trained neural network to the control system 140. Once received, the control system 140 can use the trained neural network to reconstruct a three-dimensional model of the anatomical structure using the two-dimensional images acquired intraoperatively. This use of the trained neural network is described above with reference to FIG.

[0049] In some embodiments, the neural network generation system 500 may generate a personalized trained neural network. As used herein, a personalized trained neural network may refer to a neural network trained in a manner that takes into account the characteristics or features of a patient or a system that uses the trained neural network to reconstruct a three-dimensional model. FIG. 7 is a system diagram illustrating a personalized neural network generation system 700, according to an exemplary embodiment. The personalized neural network generation system 700 may include many of the components shown and described with reference to the neural network generation system 500 of FIG. 5. The personalized neural network generation system 700 may operate by receiving a request 710 for a trained neural network from the control system 140. The request 710 may include treatment characteristics. For example, the treatment characteristics may include characteristics of the equipment for the system 100 on which the control system 140 operates. The equipment characteristics may include the imaging capabilities (e.g., resolution) of the imaging device, the dimensions and distance between the imaging device and a known location such as a bed platform, the angle at which the imaging device acquires an image, etc. Additionally or alternatively, the treatment characteristics may include characteristics of the patient on whom the control system 140 is performing the procedure. The patient characteristics may include age, race, sex, condition, etc. In some cases, a particular anatomical structure may have a known structure, and an identifier of the known structure may be passed as a treatment characteristic. For example, the internal structure of a kidney may have one of four commonly known shapes, and the treatment characteristics that the control system 140 may send to the personalized neural network generation system 700 may include an identification of the type of shape to construct the trained neural network.

[0050] Based on the request 710, the personalized neural network generation system 700 can segment or derive the annotated kidney images based on the treatment characteristics. In some cases, each annotated kidney image can include characteristics that characterize the anatomical structure of the image, such as age, race, sex, condition, etc. The personalized neural network generation system 700 may filter out annotated kidney images with characteristics that do not match or are outside of an acceptable range compared to the treatment characteristics in the request 710. For example, the personalized neural network generation system 700 can filter out all annotated kidney images that are outside of an age range specified by the request 710.

[0051] Additionally or alternatively, the images and annotations from the CT scan can be modified to approximate the hardware and software of the medical system receiving the neural network. For example, some embodiments of the personalized neural network generation system 700 may rescale the images and their annotations to an isotropic resolution that approximately matches the image resolution of the imaging device 190. This resolution may be specified in the request 710 as a treatment characteristic, or the resolution may be inferred based on the equipment identification, such as a look-up table that maps the make and model of the imaging device to a known resolution. The personalized neural network generation system 700 may also process the images to match a known distance between the imaging device and a known reference point, such as a bed. Furthermore, if the treatment characteristic identifies the imaging device angle, the personalized neural network generation system 700 may generate a virtual two-dimensional image using the angle listed in the treatment characteristic of the request 710.

[0052] Once the personalized neural network generation system 700 segments or derives the annotated renal images based on the procedure characteristics in the request, the personalized neural network generation system 700 may use the segmented and / or derived annotated renal images to generate a personalized trained neural network. Generating the personalized neural network generation system 700 may be performed using a method similar to the embodiment described with reference to the method 500 of FIG. 5. It should be appreciated that generating a neural network trained with annotated images that share characteristics with the patient may result in a more accurate neural network for the patient. To that end, in some embodiments, the annotated images may include annotated images obtained from the patient. In this manner, images from the patient are used to train the neural network, and in these cases, some embodiments of the personalized neural network generation system 700 may weight images from the patient more heavily than other images when setting parameters of the neural network.

[0053] Feedback and Refresher Training In some embodiments, the control system 140 may provide feedback to a neural network generation system, such as that described with reference to Figures 5 and 7. Such feedback may include annotation of the reconstructed three-dimensional model. Annotations of the patient's anatomy may be derived from the neural network itself. Annotations of the patient's anatomy may also be input by an operator of the control system 140. Other types of feedback of the control system may include the number and corresponding angles of fluoroscopic images taken to achieve a given three-dimensional model, as well as a confidence level of the accuracy of the three-dimensional model overall or in a given region within the anatomy.

[0054] This level of feedback may be incorporated by the neural network generation system. Some embodiments may use this feedback to correlate the number and angle of fluoroscopic images that produce a desired result. This correlation may be provided to the control system as part of the neural network to help guide the control system in taking two-dimensional images with the imaging device 190. This guidance may be instructions and suggestions for the number and angle of two-dimensional images, or the guidance may be automated where the control system automatically controls the imaging device to take two-dimensional images according to the recommendations.

[0055] By way of example and not limitation, the internal structure of the kidney may have one of several known shapes. In a situation where it is difficult to reconstruct the three-dimensional orientation of the small calyx relative to a particular shape using two two-dimensional images, the neural network generation system may determine through detected correlations that three or more two-dimensional images would result in the neural network in generating a relatively suitable three-dimensional model. In this case, the neural network generation system may generate a recommendation to the control system to take three or more two-dimensional images using the imaging device. Additionally or alternatively, the angle of the two-dimensional image may be a factor, and the neural network generation system may detect correlations based on the angle, which may then be sent as guidance to the control system, which may ensure that the two-dimensional images are acquired from a suitable angle to capture the difficult-to-reconstruct calyx.

[0056] Confidence Level In some embodiments, the neural network may detect a confidence level for the reconstructed three-dimensional model. FIG. 8 is a diagram illustrating a reconstructed three-dimensional model 800, according to an example embodiment. The reconstructed three-dimensional model 800 includes a confidence indicator 810, which is a visual element that indicates a confidence level of the accuracy of a portion of the reconstructed three-dimensional model 800. In the example shown in FIG. 8, the confidence indicator 810 may indicate that the system has a low confidence (which may be measured based on a threshold confidence level) that the portion of the anatomy highlighted by the confidence indicator 810 is accurate. It should be appreciated that the confidence indicator may be represented as a shading, a pattern, a transparency, or the like.

[0057] The confidence level of the reconstructed anatomical structure (or a portion thereof) may be determined based on the output of the neural network. For example, the neural network may be trained to reconstruct a 3D kidney image in the form of a binary mask. In some embodiments, the binary mask may in fact be represented by a 3D array with values ​​ranging from 0 to 1. The network may assign values ​​closer to 0.5 to represent lower certainty for the corresponding pixel in the 3D array.

[0058] Based on the confidence, the system may recommend to the operator to take additional two-dimensional images, possibly at a different angle or with better quality images, if there were external conditions that reduced the overall quality of the initial two-dimensional image, such as insufficient distribution of contrast agent or insufficient depth or focus. In some embodiments, the initial confidence level, corrective actions, and resulting confidence levels are communicated back to the neural network generation system as feedback that may be used to improve the training data and generate recommendations for future use.

[0059] Mounting system and terminology Implementations disclosed herein provide systems, methods, and devices for reconstructing three-dimensional models of anatomical structures using two-dimensional images. Various implementations described herein provide improved visualization of anatomical structures during medical procedures using medical robots.

[0060] System 100 can include various other components. For example, system 100 can include one or more control electronics / circuitry, power sources, pneumatics, light sources, actuators (e.g., motors for moving a robotic arm), memory, and / or a communication interface (e.g., for communicating with another device). In some embodiments, the memory can store computer-executable instructions that, when executed by the control circuitry, cause the control circuitry to perform any of the operations discussed herein. For example, the memory can store computer-executable instructions that, when executed by the control circuitry, cause the control circuitry to receive input and / or control signals related to the operation of the robotic arm and, in response, control the robotic arm to position it in a particular arrangement.

[0061] The various components of system 100 may be electrically and / or communicatively coupled using certain connection circuits / devices / features, which may or may not be part of the control circuitry. For example, the connection feature(s) may include one or more printed circuit boards configured to facilitate mounting and / or interconnection of at least some of the various components / circuitry of system 100. In some embodiments, two or more of the control circuitry, data storage / memory, communication interface, power supply unit(s), and / or input / output (I / O) component(s) may be electrically and / or communicatively coupled to one another.

[0062] The term "control circuitry" is used herein according to its broad and ordinary meaning and may refer to any collection of the following: one or more processors, processing circuits, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connection circuits), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, graphics processing units, field programmable gate arrays, programmable logic circuits, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any device that manipulates signals (analog and / or digital) based on hard-coding of circuit and / or operational instructions. The control circuitry may further include one or more storage devices, which may be embodied in a single memory device, multiple memory devices, and / or embedded circuits of the device. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information. It should be noted that in embodiments where the control circuitry includes a hardware state machine (and / or implements a software state machine) and includes analog, digital, and / or logic circuitry, the data storage device(s) / register(s) storing any associated operational instructions may be embedded within or external to the circuitry including the state machine, analog, digital, and / or logic circuitry.

[0063] The term "memory" is used herein according to its broad and ordinary meaning and may refer to any suitable or desired type of computer-readable medium, including, for example, one or more volatile, non-volatile, removable, and / or non-removable data storage devices implemented using any suitable or desired technology, layout, and / or data structures / protocols that contain any suitable or desired computer-readable instructions, data structures, program modules, or other types of data.

[0064] Computer-readable media that may be implemented according to embodiments of the present disclosure may include, but are not limited to, phase-change memory, static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other non-transitory medium that may be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media may generally not include communication media such as modulated data signals and carrier waves. Thus, computer-readable media should generally be understood to refer to non-transitory media.

[0065] Further embodiments Depending on the embodiment, certain acts, events, or functions of any of the algorithms or processes described herein may be performed in a different order, added, merged, or omitted entirely, and thus, in a particular embodiment, not all of the described acts or events are necessary to the execution of a process.

[0066] In particular, conditional language used herein, such as "can," "could," "might," "may," "eg," and the like, unless specifically stated otherwise or understood otherwise within the context in which it is used, is intended to have its ordinary meaning and is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is not intended to imply that features, elements, and / or steps are generally required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included or performed in any particular embodiment, with or without author input or prompting. Terms such as "comprising," "including," "having," and the like, are synonymous and used in their ordinary sense and are used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, when the term "or" is used, for example, to connect a list of elements, the term "or" is used in its inclusive sense (and not its exclusive sense) to mean one, some, or all of the listed elements. Unless specifically stated otherwise, connective language such as the phrase "at least one of X, Y, and Z" is understood in the context as it is commonly used to convey that an item, term, element, etc. can be either X, Y, or Z. Thus, such connective language is not generally intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0067] In the above description of the embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the method of the disclosure should not be interpreted as reflecting an intention that any claim requires more features than are expressly recited in that claim. Moreover, any component, feature, or step illustrated and / or described in a particular embodiment herein may be applied to or used in conjunction with any other embodiment. Moreover, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Thus, it is intended that the scope of the present disclosure should not be limited by the particular embodiments described above, but should be determined solely by a fair reading of the following claims.

[0068] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify an element, such as a structure, component, operation, etc., do not necessarily indicate a priority or order of the element with respect to any other elements, but rather may generally distinguish the element from another element having a similar or identical name (apart from the use of the ordinal term). In addition, as used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events not expressly recited.

[0069] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It is further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0070] Spatially relative terms such as "outer", "inner", "upper", "lower", "below", "upper", "vertical", "horizontal", and similar terms may be used herein for ease of description to describe the relationship between one element or component and another element or component as illustrated in the drawings. It should be understood that the spatially relative terms are intended to encompass different orientations of the device during use or operation in addition to the orientation depicted in the drawings. For example, if the devices shown in the drawings are inverted, a device positioned "below" or "under" another device may be positioned "above" the other device. Thus, the illustrative term "lower" may include both lower and upper positions. Devices may also be oriented in other directions, and thus the spatially relative terms may be interpreted differently depending on the orientation.

[0071] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," "greater than," and the like are intended to encompass the notion of equality. For example, "less" can mean not only "less than" in the strict mathematical sense, but also "less than or equal to."

[0072] [Embodiment] (1) A method for generating a model of a patient's anatomy, comprising: obtaining a neural network trained with images from one or more computed tomography scans labeled according to portions of the anatomical structure; obtaining a first fluoroscopic image of the anatomy of the patient; acquiring a second fluoroscopic image of the patient's anatomical structure, the first fluoroscopic image and the second fluoroscopic image capturing the anatomical structure from different angles; generating the model of the anatomical structure using the neural network, the first fluoroscopic image, and the second fluoroscopic image; rendering said model on a display device; A method comprising: (2) The method of embodiment 1, further comprising acquiring a third fluoroscopic image of the patient's anatomical structure, and generating the model of the anatomical structure further uses the third fluoroscopic image. (3) The method of claim 1, wherein the different angles are substantially perpendicular to each other. (4) The method of embodiment 1, further comprising generating a confidence level associated with at least a portion of the model, the confidence level relating to a confidence in the accuracy of the portion. (5) The method of embodiment 4, further comprising rendering the representation of the confidence level together with the model on the display device.

[0073] (6) A method for generating a trained neural network usable to reconstruct a three-dimensional model from a set of two or more two-dimensional images, comprising: obtaining a first set of two or more labeled two-dimensional images corresponding to a first patient; obtaining a second set of two or more labeled two-dimensional images corresponding to a second patient; training a neural network based on the first set of two or more labeled two-dimensional images and the second set of two or more labeled two-dimensional images to generate the trained neural network; A method comprising: (7) The method of embodiment 6, further comprising acquiring characteristics of a third patient, wherein the first set of two or more labeled two-dimensional images corresponding to the first patient is acquired based on a comparison between the first patient and the characteristics. (8) The method of claim 7, wherein a second set of the two or more labeled two-dimensional images corresponding to the second patient is obtained based on a comparison between the second patient and the characteristics. (9) The method of embodiment 6, further comprising processing the first set of two or more labeled two-dimensional images based on characteristics of the medical system. (10) The method of claim 9, wherein the characteristics of the medical system include at least one of an imaging capability of an imaging device, a distance between the imaging device and a known location, or an angle of the imaging device.

[0074] (11) acquiring a first set of two or more unlabeled two-dimensional images corresponding to the first patient; generating the three-dimensional model using the first set of two or more unlabeled two-dimensional images and the trained neural network; and 7. The method of embodiment 6, further comprising: (12) Training the neural network based on the first set of two or more labeled two-dimensional images and the second set of two or more labeled two-dimensional images includes: deriving a first set of characteristics from the first set of two or more labeled two-dimensional images; deriving a second set of properties from a second set of the two or more labeled two-dimensional images; and training the neural network using the first set of characteristics and the second set of characteristics; 7. The method of embodiment 6, further comprising: (13) The method of claim 12, wherein the first set of features includes at least one of keypoints that identify portions of a skeleton or anatomical structure extracted from the first set of two or more labeled two-dimensional images. (14) A non-transitory computer-readable storage medium for generating a trained neural network usable to reconstruct a three-dimensional model from a set of two or more two-dimensional images, the non-transitory computer-readable storage medium storing instructions that, when executed, cause a processor of a device to at least: obtaining a first set of two or more labeled two-dimensional images corresponding to a first patient; obtaining a second set of two or more labeled two-dimensional images corresponding to a second patient; training a neural network based on the first set of two or more labeled two-dimensional images and the second set of two or more labeled two-dimensional images to generate the trained neural network; A non-transitory computer-readable storage medium for causing a program to be executed. (15) The instructions cause the processor to: 15. The non-transitory computer-readable storage medium of embodiment 14, further comprising acquiring characteristics of a third patient, wherein a first set of the two or more labeled two-dimensional images corresponding to the first patient is acquired based on a comparison between the first patient and the characteristics.

[0075] (16) The non-transitory computer-readable storage medium of embodiment 15, wherein a second set of the two or more labeled two-dimensional images corresponding to the second patient is obtained based on a comparison between the second patient and the characteristics. (17) The instructions cause the processor to: 15. The non-transitory computer-readable storage medium of embodiment 14, further comprising processing the first set of the two or more labeled two-dimensional images based on characteristics of the medical system. (18) The non-transitory computer-readable storage medium of embodiment 17, wherein the characteristics of the medical system include at least one of an imaging capability of an imaging device, a distance between the imaging device and a known location, or an angle of the imaging device. (19) The instructions cause the processor to: acquiring a first set of two or more unlabeled two-dimensional images corresponding to the first patient; generating the three-dimensional model using the first set of two or more unlabeled two-dimensional images and the trained neural network; and 15. The non-transitory computer-readable storage medium of embodiment 14, further comprising: (20) Training the neural network based on the first set of two or more labeled two-dimensional images and the second set of two or more labeled two-dimensional images includes: deriving a first set of characteristics from the first set of two or more labeled two-dimensional images; deriving a second set of properties from a second set of the two or more labeled two-dimensional images; and training the neural network using the first set of characteristics and the second set of characteristics; Further comprising: 15. The non-transitory computer-readable storage medium of claim 14, wherein the first set of features includes at least one of key points that identify portions of a skeleton or anatomical structure extracted from the first set of two or more labeled two-dimensional images.

Claims

1. A method for modeling an anatomical structure, comprising: acquiring a first two-dimensional (2D) image of the anatomical structure; acquiring a second 2D image of the anatomical structure captured from a different angle than the first 2D image; generating a three-dimensional (3D) model of the anatomical structure based on the first 2D image, the second 2D image, and a neural network trained to reconstruct the 3D model from the first 2D image and the second 2D image, wherein the 3D model indicates a confidence level for at least a portion of the reconstruction; A method comprising:

2. The method of claim 1, further comprising obtaining a third 2D image of the anatomical structure, wherein the generation of the 3D model is further based on the third 2D image.

3. The method of claim 1, further comprising rendering the 3D model including a representation of the confidence level for at least a portion of the reconstruction on a display device.

4. A method for training a neural network, comprising: acquiring a plurality of two-dimensional (2D) images of a first anatomical structure; acquiring a plurality of 2D images of a second anatomical structure of the same type as the first anatomical structure; training the neural network to reconstruct a three-dimensional (3D) model of an anatomical structure based on the plurality of 2D images of the first anatomical structure and the plurality of 2D images of the second anatomical structure, wherein the 3D model indicates a confidence level for at least a portion of the reconstruction; A method comprising:

5. The acquisition of the plurality of 2D images of the first anatomical structure comprises: determining a characteristic of a third anatomical structure; comparing the characteristic of the third anatomical structure with each characteristic of the first anatomical structure; selecting the plurality of 2D images of the first anatomical structure based on comparing the characteristic of the third anatomical structure with the respective characteristic of the first anatomical structure; The method of claim 4, comprising:

6. The acquiring of the plurality of 2D images of the second anatomical structure comprises: comparing the characteristic of the third anatomical structure with each characteristic of the second anatomical structure; selecting the plurality of 2D images of the second anatomical structure based on comparing the characteristic of the third anatomical structure with the respective characteristic of the second anatomical structure; The method of claim 5 , comprising:

7. The method of claim 4 , further comprising processing the plurality of 2D images of the first anatomical structure based at least in part on characteristics of a medical system.

8. The method of claim 7 , wherein the characteristics of the medical system include at least one of an imaging capability of an imaging device, a distance between the imaging device and a known location, or an angle of the imaging device.

9. Training the neural network: deriving a first set of properties from the plurality of 2D images of the first anatomical structure; deriving a second set of properties from the plurality of 2D images of the second anatomical structure; and training the neural network based at least in part on the first set of characteristics and the second set of characteristics; The method of claim 4, comprising:

10. 10. The method of claim 9, wherein the first set of features comprises at least one of a skeleton extracted from the plurality of 2D images of the first anatomical structure or key points identifying portions of an anatomical structure.

Images