Style transfer for intraoperative imaging

Abstract

A system receives a set of two-dimensional projection images associated with respective projection angles from an imaging device. The projection images include projections of an anatomical structure, an anatomical target, and possibly fiducial markers, with a flexible elongate device disposed therein. The system can reconstruct a three-dimensional image from the set of two-dimensional projection images. In some examples, prior to reconstruction, the system can detect and repair pixels (within the projection images) associated with the fiducial markers, the flexible elongate device, and / or other objects. The system can enhance the reconstruction using a style transfer model. The system or another system can train the style transfer model by generating two-dimensional projections for a training set based on three-dimensional ground truth images.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority and filing date benefit to U.S. Provisional Patent Application No. 63 / 581,541, filed September 8, 2023, entitled “STYLE TRANSFER FOR INTRAOPERATIVEIMAGING”. The entire contents of that U.S. Provisional Patent Application are expressly incorporated herein by reference. Technical Field

[0003] This disclosure relates to the planning and / or navigation of minimally invasive medical procedures, and more specifically to the use of style transfer models to enhance intraoperative imaging. Background Technology

[0004] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, physicians can insert minimally invasive medical instruments (including surgical, diagnostic, and / or therapeutic instruments) to reach target tissue locations. One such technique utilizes flexible and / or steerable elongated devices, such as flexible catheters, which can be inserted into an anatomical channel and navigated toward regions of interest within the patient's anatomy.

[0005] The combination of positioning sensors at the flexible elongation device and intraoperative imaging can greatly aid in the planning and navigation of minimally invasive procedures. Specifically, combining sensor data with intraoperative images enables accurate determination of the location, orientation, and / or pose of the flexible elongation device within the patient's anatomy. However, accessible intraoperative techniques such as tomosynthesis based on a limited projection set can suffer from low image quality and reconstruction artifacts. Improving the quality of intraoperative tomosynthesized images and extracting accurate intraoperative anatomical data using current techniques remains a challenge. Summary of the Invention

[0006] The following is a simplified overview of the various examples described herein and is not intended to identify key or important elements or define the scope of the claims.

[0007] In some examples, a tangible, non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause one or more processors to: compute a set of two-dimensional projected images corresponding to spans of projection angles less than 120 degrees, based on a three-dimensional ground-truth image. The instructions can also cause one or more processors to: compute a three-dimensional tomographic synthesis training image based on the two-dimensional projected image set. Furthermore, the instructions can cause one or more processors to: generate a three-dimensional output image using the three-dimensional tomographic synthesis training image as input to a style transfer machine learning model; and adjust the style transfer machine learning model to reduce the loss indication between the three-dimensional output image and the three-dimensional ground-truth image.

[0008] In other examples, a system for training a style transfer machine learning model for visualizing patient anatomy during medical procedures includes: one or more processors; and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the processors to: compute a set of two-dimensional projected images corresponding to spans of projection angles less than 120 degrees, based on three-dimensional real images. The instructions may also cause the processors to: compute three-dimensional tomographic synthesis training images based on the two-dimensional projected image set. Furthermore, the instructions may cause the processors to: generate a three-dimensional output image using the three-dimensional tomographic synthesis training image as input to the style transfer machine learning model; and adjust the style transfer machine learning model to reduce loss indicators between the three-dimensional output image and the three-dimensional real image.

[0009] In other examples, a method for training a style transfer machine learning model for visualizing patient anatomy during medical procedures includes: computing a set of two-dimensional projected images corresponding to spans of projection angles less than 120 degrees, using one or more processors and based on three-dimensional real images. The method further includes: computing three-dimensional tomographic synthesis training images using one or more processors and based on the two-dimensional projected image set. Further, the method includes: generating a three-dimensional output image using the three-dimensional tomographic synthesis training image as input to the style transfer machine learning model using one or more processors. Further still, the method includes: adjusting the style transfer machine learning model by one or more processors to reduce the loss indication between the three-dimensional output image and the three-dimensional real images.

[0010] In other examples, a tangible, non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause one or more processors to: acquire a plurality of two-dimensional images associated with a first viewpoint, the two-dimensional images depicting an anatomical structure. The instructions may also cause one or more processors to: compute a first three-dimensional image of at least a portion of the anatomical structure, at least partially based on the plurality of two-dimensional images. Furthermore, the instructions may cause one or more processors to: compute a second three-dimensional image based on at least a portion of the first three-dimensional image and using one or more style transfer models; and cause a display device to display a graphical user interface depicting the second three-dimensional image.

[0011] In other examples, a system for visualizing patient anatomy includes: a display device; one or more processors; and one or more non-transitory computer-readable media storing instructions, which, when executed by the one or more processors, cause the one or more processors to: acquire a plurality of two-dimensional images associated with a first viewpoint range, the two-dimensional images depicting the anatomical structure. The instructions may also cause the one or more processors to: compute a first three-dimensional image of at least a portion of the anatomical structure, at least partially based on the plurality of two-dimensional images. Furthermore, the instructions may cause the one or more processors to: compute a second three-dimensional image based on at least a portion of the first three-dimensional image and using one or more style transfer models. Even further, the instructions may cause the one or more processors to: cause the display device to display a graphical user interface depicting the second three-dimensional image.

[0012] In other examples, a method for visualizing patient anatomy includes: acquiring a plurality of two-dimensional images associated with a first viewpoint range by one or more processors, the two-dimensional images depicting the anatomy. The method further includes: calculating a first three-dimensional image of at least a portion of the anatomy by one or more processors and at least partially based on the plurality of two-dimensional images. Further, the method includes: calculating a second three-dimensional image by one or more processors based on at least a portion of the first three-dimensional image and using one or more style transfer models; and causing a display device to display a graphical user interface depicting the second three-dimensional image by one or more processors. Attached Figure Description

[0013] Figure 1A An example system for navigation during medical procedures within an operating environment is described.

[0014] Figure 1B This is a simplified diagram of a flexible elongation device installed within an anatomical structure.

[0015] Figure 1C Figure 1DThis is a simplified diagram depicting an example intraoperative imaging geometry used for tomography synthesis.

[0016] Figure 1E is based on Figure 1C, Figure 1D A simplified three-dimensional representation of a flexible elongation device set within the anatomical structure of the imaging geometry.

[0017] Figure 1F A portion of a projected image depicts a flexible elongation device positioned within an anatomical structure covered with reference markers.

[0018] Figure 1G Figure 1H The example reference plate is schematically shown at two projection angles.

[0019] Figure 2A to Figure 2D An example procedure for in-painting reference marks in a projected image of a flexible elongation device set within an anatomical structure is illustrated schematically.

[0020] Figure 2E to Figure 2H An example procedure for repairing a flexible elongation device in a projected image of an anatomical structure is illustrated schematically.

[0021] Figure 3A An example multi-projection image acquisition geometry for a flexible elongation device set within an anatomical structure is schematically shown.

[0022] Figure 3B Figure 3C It is used for Figure 3A Example projected images of the geometric structure of the image acquisition.

[0023] Figure 4A Rigid bodies in two example coordinate systems are depicted.

[0024] Figure 4B An example coordinate transformation process is illustrated schematically.

[0025] Figure 4C An example geometry is depicted for extraction and coordinate registration using a flexible elongation device that utilizes shape data and projected images.

[0026] Figure 5A , Figure 5B An example process for extracting, registering coordinates, and updating targets using a flexible elongation device that utilizes shape data and projected images is illustrated.

[0027] Figure 6A An example style transfer for 3D images is illustrated schematically.

[0028] Figure 6BAn example training process for a machine learning model used for style transfer is illustrated schematically.

[0029] Figure 7 It is a simplified diagram based on some examples of medical systems.

[0030] Figure 8A These are simplified diagrams of medical device systems based on some examples.

[0031] Figure 8B This is a simplified diagram of a medical device, including a medical tool within an elongation device, based on some examples.

[0032] Figure 9A and Figure 9B It is a simplified diagram of a patient coordinate space side view based on some examples, including a medical device mounted on an insertion component.

[0033] Figure 10 A flowchart depicts an example method for visualizing a patient's anatomical structures during a medical procedure.

[0034] Figure 11A , Figure 11B A flowchart depicts another example method for visualizing a patient's anatomy during a medical procedure.

[0035] Figure 12A A flowchart depicts an example method for training machine learning and / or deep learning models for style transfer.

[0036] Figure 12B A flowchart is depicted for an example method of using style transfer to visualize patient anatomy.

[0037] Figure 13A , Figure 13B The outputs of tomographic synthesis imaging and cone-beam computed tomography (CBCT) imaging of the patient's anatomy are depicted respectively.

[0038] Figure 14A , Figure 14B The performance of the style transfer model, which transfers to tomographic synthetic images to computed tomography-style images and CBCT-style images, is shown.

[0039] Examples of this disclosure and its advantages can be best understood by referring to the following detailed description. It should be understood that the same reference numerals are used to identify the same elements shown in one or more figures, wherein the illustrations in the figures are for illustrative purposes and not for limiting the scope of this disclosure. Detailed Implementation

[0040] In the following description, specific details are set forth in relation to some examples conforming to this disclosure. Numerous specific details are set forth to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that some examples can be practiced without some or all of these specific details. The specific examples disclosed herein are intended to be exemplary and not restrictive. Other elements, though not specifically described herein, can be implemented by those skilled in the art within the scope and spirit of this disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one example may be incorporated into other examples unless otherwise specifically described or if one or more features would render the example inoperable. In some cases, well-known methods, processes, components, and circuits have not been described in detail to avoid unnecessarily obscuring aspects of the examples.

[0041] This disclosure describes various instruments and parts thereof based on their state in three-dimensional space. As used herein, the term “position” refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term “orientation” refers to the rotational placement of an object or part of an object (e.g., one or more rotational degrees of freedom, such as roll, pitch, and yaw). As used herein, the term “pose” refers to the position of an object or part of an object in at least one translational degree of freedom and the orientation of that object or part of an object in at least one rotational degree of freedom (e.g., up to six total degrees of freedom). As used herein, the term “shape” refers to the set of poses, positions, and / or orientations measured along an object. As used herein, the term “distal” refers to a location closer to the process site, and the term “proximal” refers to a location further away from the process site. Thus, when an instrument is designed to perform a process, the distal portion or distal end of the instrument is closer to the process site than the proximal portion or proximal end of the instrument.

[0042] This disclosure generally relates to systems and methods for facilitating user planning and / or user navigation during endovascular medical procedures (e.g., by physicians). These systems and methods can provide improved visualization of patient anatomy based on tomographic synthesis of two-dimensional projected images from a finite set of angles. Furthermore, improvements in tomographic synthesis output can lead to improved detectability of lesions and / or other targets within the patient's anatomy. Additionally, the systems and methods described in this disclosure can facilitate registration between the imaging coordinate system and the coordinate system of instruments (e.g., flexible elongating devices) positioned within the patient's anatomy. The registration techniques described in this disclosure can further improve the accuracy and / or speed of the medical procedure by facilitating the transformation of target locations from intraoperative images to the instrument coordinate system.

[0043] The systems and methods described in this paper improve the three-dimensional output images of tomographic synthesis by, for example, repairing reference markers (and / or other image elements) from two-dimensional images prior to tomographic reconstruction. Reference markers, which may originate from a reference plate placed below, above, or around the patient's body, can help determine projection angles (e.g., the angular position of the arm of an X-ray imaging device), but once the angles are determined, they tend to interfere with tomographic reconstruction.

[0044] Restoring reference markers can involve several steps. First, for each in the 2D image, the system can generate a binary value mask that sets the pixel values ​​associated with the reference marker to zero. In some examples, the system can generate a mask for a given image based solely on the image itself. In other examples, the system can use image sequences from different angles to help determine the mask for each angle. To generate the mask, the system can detect reference objects based on reference size, reference shape, and / or spacing between reference objects. The system can use various signal processing techniques to compensate for partial detection of reference markers. For example, the system can “expand” the mask to ensure that all pixels of the reference markers are captured. After removing the reference markers, the system can use one or more of a variety of restoration techniques to fill in the removed pixels. For example, the system can use computer vision or image processing algorithms such as methods based on Navel-Stokes gradients or fast-walking methods. Additionally or alternatively, the system can use machine learning (ML) for restoration. For example, the system can use progressive restoration, attention-based restoration, and / or multi-faceted restoration. The system can implement restoration using various possible neural network architectures such as autoencoders, generative adversarial networks (GANs), and / or diffusion models.

[0045] In some examples, the systems and methods of this disclosure can apply reparative techniques to remove instruments (e.g., flexible elongating devices) positioned within a patient's anatomy. For this purpose, the systems and methods of this disclosure may need to identify pixels associated with the instrument within a two-dimensional projection. In some examples, the system can identify pixels associated with the instrument based at least in part on user input. For example, the system can generate a graphical user interface (GUI) on a display device displaying multiple projections, and the user can select pixels corresponding to the same reference point positioned at the instrument from at least two of the displayed projections. The system can calculate the coordinates of the reference point in three dimensions by tracing back along projection rays from the projected reference point and finding the intersection of projection rays from different projections. Furthermore, the system can identify pixels forming a continuous curve associated with a segment of the instrument in at least one of the projections. In some examples, the system can generate a GUI display and projections, prompting the user to select multiple points along the curve associated with the segment of the instrument. In other examples, the system can use one or more image processing techniques (e.g., thresholding, segmentation, contour detection, etc.) to automatically determine the pixels forming the curve associated with the segment of the instrument. Based on identified pixels forming a continuous curve (e.g., a centerline) associated with a segment of the instrument, the system can back-project the two-dimensional curve into three dimensions of the imaging coordinate system to calculate a three-dimensional curve associated with the segment of the instrument. In some examples, the system can perform three-dimensional image reconstruction based on the projection and trace a ray along the projection direction from the projection of the pixels forming the curve to find the instrument's maximum intensity and / or other indicators. In other examples, the system can trace the projection of a ray from the first projection image (where pixels associated with the curve are identified) in a second projection image and identify pixels associated with the projection of the instrument onto the second projection image. In this way, the system can reconstruct the curve associated with a segment of the instrument in three dimensions based on at least two projection images, as shown in the reference. Figure 4C More detailed description.

[0046] In some examples, the system can calculate a three-dimensional curve associated with an instrument positioned within an anatomical structure (e.g., at least one segment of the instrument) based at least in part on shape data obtained from a sensing system. Because the shape data from the sensing system can reside in a coordinate system different from the imaging coordinate system, methods using shape data can include a registration process between the two coordinate systems. [Reference] Figures 4A to 4C The registration of coordinate systems and the use of shape data to identify three-dimensional curves (associated with instruments) in imaging coordinates are discussed in more detail.

[0047] The three-dimensional curves associated with the instrument can be used to generate a repair mask for the instrument. For example, the system can project the curve onto each two-dimensional projected image in a set of two-dimensional projected images obtained from an imaging system to generate a repair mask. See below for reference. Figure 2A to Figure 2H As described, the repair mask can be expanded, and the repair process can be similar to the repair process used for baseline markers.

[0048] After obtaining a set of two-dimensional images with removed and repaired instruments and / or reference markers, the system can proceed to reconstruct three-dimensional cross-sections of the patient's anatomy based on the two-dimensional image set and corresponding angles. The system can use backprojection with appropriate regularization and / or any other suitable algorithm for three-dimensional image reconstruction. The system can be configured to use style transfer techniques to enhance the reconstructed images. That is, the system can be configured to generate enhanced images with the style of computed tomography (CT) or cone-beam computed tomography (CBCT) images from tomographic synthetic images. For this purpose, the operating system's approach may include training an ML model to transfer the CT / CBCT style to the tomographic synthetic images. The ML model may include a convolutional neural network (CNN) (e.g., U-net), a transformer network, a GAN, and / or a diffusion model. Training may include: generating simulated tomographic synthetic inputs by obtaining a subset of angles from CT / CBCT data; adding appropriate noise to the projection and corresponding angular uncertainties; generating tomographic synthetic data and training based on the simulated tomographic synthetic data to reduce loss relative to the ground truth based on CT or CBCT data. The system can then apply the trained style transfer model to real tomographic synthesis data to reduce tomographic synthesis artifacts and generate volumetric or tomographic images (slices) similar to CT and / or CBCT outputs. More generally, the techniques of this disclosure can use a style transfer model trained on a less sparse set of measurements to enhance inverse images obtained from a sparse set of measurements.

[0049] The system can render a graphical user interface on a display device, which depicts visual data at any stage of the process: a two-dimensional image with a reference object, an image with the reference object removed, a tomographic composite image before style transfer, and / or a tomographic composite image after style transfer. The system can enable the user to select parameters and / or techniques for any stage of image processing (e.g., as briefly described above) via the graphical user interface.

[0050] In other examples, the system can enhance one or more 2D images that are not associated with a corresponding viewpoint by repairing one or more reference markers within one or more 2D images. In still other examples, the system can enhance tomographic images using style transfer (e.g., transferring CT / CBCT style to tomographic composite images). In a broader sense, reference object repair does not need to be associated with tomographic composites or style transfer, and style transfer enhancement of tomographic composites or other imaging modalities does not need to be associated with reference object repair (e.g., it can be applied to methods where the system determines the viewpoint without using a reference plate).

[0051] The technique for coordinate registration between the instrument coordinate system and the imaging coordinate system, or the combination of said technique with enhancement of tomographic synthetic images, can help (e.g., refer to...) Figure 5A , Figure 5B (As described) accurately pinpointing the location of targets within the patient's anatomical structures, thereby improving the speed and / or accuracy of medical procedures.

[0052] Figure 1A An example system 100 for navigation during a medical procedure within an operating environment 101 is depicted. System 100 can acquire images from a portion of the operating environment 101 positioned within the field of view F (generally indicated by the dashed line) of an imaging unit 110. For this purpose, system 100 can be communicatively connected to the imaging unit 110. Additionally, system 100 can acquire data from one or more sensors positioned within the operating environment. For this purpose, system 100 can be communicatively connected to one or more sensors via a communication connection with a sensing unit 115.

[0053] System 100 includes a processing unit 120 and a display unit 130 that are communicatively connected to each other. Although in Figure 1A In this embodiment, the imaging unit 110 and the sensing unit 115 are depicted differently from system 100; however, in other examples, system 100 may include imaging unit 110 and / or sensing unit 115. In any case, one or more processors of the processing unit 120 of system 100 may be configured to receive images and / or processed image information from imaging unit 110 and to receive data from one or more sensors via sensing unit 115.

[0054] Throughout this disclosure, the following description of the example operations performed by processing unit 120 should be understood as being performed by one or more processors of processing unit 120. In some examples, one or more processors may include hardware specifically configured (e.g., hardwired or programmable) to perform at least a portion of the example operations described in this disclosure. Additionally or alternatively, one or more processors may be configured to perform at least a portion of the example operations described in this disclosure by executing a software instruction set. For this purpose, system 100 may include a tangible non-transitory computer-readable medium or communicatively connected to a tangible non-transitory computer-readable medium. The medium may store instructions that, when executed by processing unit 120, perform any or more of the example operations described below. For example, the instructions may cause processing unit 120 to perform image processing operations on an image received from imaging unit 110 and / or perform calculations (e.g., for coordinate registration) based on data received through sensing unit 115. Furthermore, instructions may cause the processing unit 120 to cause the display unit 130 to display information via a GUI regarding the processing of the image received from the imaging unit 110 and the processing of the data received by the sensing unit 115. For example, the processing unit 120 may send information or data representing the entire GUI including the information to the display unit 130.

[0055] Operators of medical systems (e.g., physicians, other medical practitioners, or fully automated robotic surgical systems) can use information displayed at display unit 130 to perform medical procedures (e.g., endoscopy, biopsy, drug therapy, and / or treatments such as ablation). During a medical procedure, the operator can control a flexible elongation device 140, which is inserted through an orifice O (or through a suitable surgical incision) into an anatomical structure A of a patient P positioned at table T. For example, a medical procedure may include navigating the flexible elongation device 140 (indicated by solid lines outside the patient P and dashed lines inside the patient P) toward a Region of Interest (ROI) R within anatomical structure A using information displayed at display unit 130. For example, an ROI R may be a designated procedure site for visual examination, biopsy, treatment, or any other medical procedure. Throughout this disclosure, an ROI R may be referred to as an area or target.

[0056] One or more reference elements may be disposed at the flexible elongation device 140 (e.g., physically contacting the flexible elongation device 140 in a manner that forms a rigid relationship with the flexible elongation device 140 during operation / use, integrated within the flexible elongation device 140, fixedly attached to the flexible elongation device 140, or removably attached to the flexible elongation device 140). The reference elements are configured to be visible in the image acquired by the imaging unit 110, thereby enhancing the visibility of the flexible elongation device 140 and / or aiding in the identification of certain points along the device 140. For this purpose, the reference elements may include elements of various materials and / or structures such as metals, plastics, etched glass, dyes, radioactive or fluorescent markings, confined fluids (e.g., bubbles), etc. At least some of the elements of the reference element 142 may be integrated (e.g., etched, deposited, painted, or otherwise fixedly attached) to the flexible elongation device 140. Additionally or alternatively, the reference elements may include elements removably disposed at the flexible elongation device 140. For example, the reference object can be integrated into a removable structure such as a sleeve or mandrel, which can then be removably attached to the flexible elongation device 140.

[0057] One or more sensors may also be disposed at the flexible elongation device. The sensors may be mechanical, optical, electromagnetic (EM), or any other suitable sensors. The sensors may be integrated into the flexible elongation device 140 or removably attached to it. The sensors may be configured to communicate with the sensing unit 115.

[0058] In some examples, the sensor is an active sensor configured to emit electromagnetic radiation (e.g., optical, radio, low-frequency magnetic) or radioactive radiation. Sensing unit 115 may include components for receiving the radiation emitted by the sensor and performing triangulation or trilateration of the sensor's position in sensor system coordinates. Sensor system coordinates may be the coordinates of a robot-assisted system configured to manipulate, control, or guide the flexible elongation device 140. Therefore, sensor system coordinates may be referred to as machine coordinates.

[0059] In other examples, the sensor is passive and does not emit radiation. A passive sensor can sense radiation emitted by components of the sensing unit 115 disposed within the operating environment 101. For example, in an EM sensing system, one or more charged coils may be disposed within the operating environment 101 to generate a static or dynamic magnetic field. The sensor can be configured to pick up changes in the dynamic magnetic field, or to pick up changes in the sensed static or quasi-static magnetic field when the sensor moves, and convert the sensed changes into a current received by the sensing unit 115. The sensing unit 115 can then calculate an indication of the sensor's position based on the received current.

[0060] The sensor can be an optical fiber sensor positioned along the length of the flexible elongation device. The optical fiber sensor may include a Bragg grating and / or materials for enhancing nonlinear scattering. The optical fiber sensor can be configured to change its spectral reflectivity based on material strain. Such a sensor can scatter light, for example, emitted by the sensing unit 115, in a manner indicating the location and degree of bending in the flexible elongation device 140.

[0061] In some examples, the sensors may include accelerometers, gyroscopes, and / or magnetometers. Additionally, point sensors may include an inertial measurement unit (IMU) combining multiple sensors (e.g., accelerometers, gyroscopes) and / or an inertial and magnetic measurement unit (IMMU) combining multiple sensors (e.g., accelerometers, gyroscopes, magnetometers). The IMU and / or IMMU may generate signals indicating the orientation of the flexible elongation device at a given point relative to (e.g., the Earth's) gravitational and / or magnetic fields. In some examples, an additional magnetic field may be introduced into the sensor environment. Additionally or alternatively, the IMU and / or IMMU may generate signals indicating the motion of the flexible elongation device (e.g., caused by motion of an anatomical structure due to breathing and / or other factors, and / or by independent motion of the flexible elongation device within an anatomical structure). In some examples, sensing unit 115 may combine indications of orientation (e.g., up to three degrees of freedom) from the IMU with indications of position from other (e.g., EM) sensors to generate more complete data indicating the pose of the flexible elongation device. Furthermore, in some examples, sensing unit 115 can combine data from sensors in multiple sensor coordinate systems. Sensing unit 115 and / or processing unit 120 can register multiple sensor coordinate systems with each other.

[0062] Generally, system 100 can use any suitable combination of sensors described above to obtain shape data indicating the shape of flexible elongation device 140 (e.g., pose, including position and orientation, at different points along the length of flexible elongation device 140).

[0063] Figure 1B This is a simplified diagram of a flexible elongation device 140 installed within anatomical structure A. (Includes...) Figure 1BThis is to provide an expanded and more detailed view of the portion of the operating environment 101 set within the field of view F. Anatomical structure A may be the lungs of patient P. The flexible elongation device 140 can be inserted by an operator into and navigate toward region R (e.g., the target of a medical procedure), for example, to study or treat pathology in region R. The techniques described in this disclosure can facilitate the navigation process by generating and displaying timely and accurate imaging and sensing of the flexible elongation device 140 and imaging of the anatomical structure A. These techniques combine position, pose, and / or shape data in one coordinate system obtained using sensors and position, pose, and / or shape data obtained using imaging in another coordinate system. Combining the data can improve the speed, accuracy, reliability, and / or safety of the medical procedure. For example, processing unit 120 can combine data from imaging unit 110 and / or sensing unit 115 to quickly and accurately determine the position, orientation, and / or pose of at least a portion of the flexible elongation device 140 relative to the anatomical structure A of patient P, and particularly the ROI or target R.

[0064] Furthermore, processing unit 120 can generate a graphical user interface (GUI) or update GUI data for display on display unit 130, thereby assisting the operator in the medical procedure. In some examples, processing unit 120 can generate data and / or control signals for a control unit of a robotic system configured to manipulate and / or navigate the flexible elongation device 140. Additionally or alternatively, processing unit 120 can be configured to generate one or more alarms based on combined imaging and sensing data. Alarms may include, for example, alarms indicating proximity to region R, alarms indicating potential navigation errors, and alarms indicating that the confidence level of the position of the tip of the flexible elongation device 140 has fallen below a threshold level. It should be noted that combining intraoperative data from imaging unit 110 and sensing unit 115 requires registration of two coordinate systems relative to each other, as referenced below. Figures 4A to 4B As described.

[0065] Generally, a suitably configured imaging unit (e.g., imaging unit 110) can generate intraoperative imaging data. Intraoperative imaging data may include, for example, computed tomography (CT), particularly cone-beam computed tomography (CBCT) data. For this purpose, the imaging unit may include a C-arm CBCT imaging system and / or magnetic resonance imaging (MRI) data. Additionally or alternatively, intraoperative imaging data may be obtained using thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, or any other suitable imaging technique.

[0066] This disclosure focuses in particular on three-dimensional (3D) image reconstruction based on two-dimensional (2D) projected images. Such intraoperative imaging data may include fluorescence fluoroscopy X-ray data (e.g., generated by a C-arm X-ray device), and especially tomographic composite data (reconstructed into 3D volume from 2D X-ray images, such as fluorescence fluoroscopy images).

[0067] Figure 1C Figure 1D This is a simplified diagram depicting an example intraoperative imaging geometry used for tomography. As shown in Figure 1C, the patient P is placed within the operating environment 101. An X-ray source 150 (which may be included in the imaging unit 110) may be positioned along an arc C (e.g., C1, C2, C3, etc.) and directed toward a detector 155 (which may also be included in the imaging unit 110) radially positioned on the opposite side of the patient P. The arc C may lie in a plane orthogonal to an axis A longitudinally positioned through the patient P, and the center of the arc C may be located at axis A. More generally, axis A does not need to extend longitudinally through the patient P. Generally, the axis of rotation of the arc C does not need to extend through the patient, but may be below or above the patient P and at any suitable angle relative to the patient P's body. Furthermore, the arc C does not need to be circular to apply the techniques of this disclosure. In some examples, the arc C may be determined by, for example, the movement of the C-arm of a fluoroscopic imaging device. Based on the geometry of Figure 1C, the system can generate projections according to a finite set of projection angles (e.g., 120°, 100°, 110°, 90°, 80°, 70°, 60° or any other suitable span).

[0068] In addition to passing through the patient P, X-rays emitted from the X-ray source 150 may pass through a reference plate 160, for example, positioned below the patient P (or at or near the patient P's body). In Figure 1C, the reference plate 160 is shown separately to illustrate example reference markings configured as an example, permeating the plate 160. Reference markings may be high-X-ray density spheres and / or cylinders (with regular or varying diameters of 1 mm, 2 mm, 5 mm, 10 mm, or any other suitable size) placed throughout the plate 160 at regular or varying intervals of 10 mm, 20 mm, 30 mm, 40 mm, or any other suitable interval. Other reference markings, such as linear, square, or other suitable markings with high X-ray attenuation or scattering, may be included in the plate 160. As described in more detail below, the system 100 may use the reference plate 160 to determine the projection angle of a 2D image (e.g., a fluorescence fluoroscopic image) detected by the detector 155 of the imaging unit 110.

[0069] Figure 1DThe example imaging coordinate system is defined. Axis A can be referred to as the x-axis. The orthogonal horizontal axis can be referred to as the y-axis. Finally, the vertical axis can be referred to as the z-axis. Planes Sx, Sy, and Sz are orthogonal to the x-axis, y-axis, and z-axis, respectively, and can be referred to as the axial plane, sagittal plane, and coronal plane, respectively. It should be noted that the axial plane Sx is parallel to the plane of arc C and does not correspond to any possible projection angle. In contrast, the coronal plane Sz corresponds to the projection acquired at the vertex of curve C using X-ray source 150. The angular range relative to the vertex of curve C may not allow for a projection to the sagittal plane, which would require an angle of ±90° relative to the vertical direction. The fluorescence fluoroscopy acquisition angle can be, for example, between ±60° relative to the vertical direction, for an arc that can be less than 120°. On the other hand, the reconstructed tomographic composite image can be presented as a tomographic image parallel to any one or more of the three planes Sx, Sy, and / or Sz.

[0070] Figure 1E is based on Figure 1C, Figure 1D A simplified three-dimensional representation of the flexible elongation device 140 disposed within the anatomical structure A within the imaging geometry. It is included to aid in the representation of the flexible elongation device 140 within the anatomical structure A. Figure 1D The coordinate system visualization is a projection acquired by the tomographic synthetic imaging system shown in Figure 1C. The flexible elongation device 140 can have a higher X-ray density than the surrounding anatomical structures, such as in... Figure 1F See the example projection in the example.

[0071] Figure 1F A portion of a projection image 180 depicts a flexible elongation device 141 (which may be device 140) disposed within an anatomical structure (e.g., anatomical structure A) covered with reference markers (e.g., from reference plate 160). The flexible elongation device 141 appears darker than the surrounding tissue, but can be converted to the high-intensity portion of the image from a negative. The high X-ray density reference markers (e.g., marker 161) are highly visible in image 180.

[0072] Figure 1G Figure 1H The projection (i.e., projected image) of an example reference plate 160 at two projection angles is schematically shown. For example, it can be assumed that the plate 160 has reference marks arranged at equal intervals in a regular square grid along both directions. In Figure 1G, the projection angle can then be 0° relative to the z-axis (i.e., perpendicular to the plate), thereby producing a regular grid of reference marks within the projected image of the plate 160. On the other hand, in Figure 1H In the projection, the angle deviates from the normal, causing the columns of the reference marks to be closer than the rows, as well as other distortions. Some of these distortions can be caused by the two axes of symmetry relative to the reference plate 160 (as shown in Figure 1G). Figure 1H The dashed line in the diagram represents the angle of projection that deviates from the normal. Figure 1HThe projection may additionally have perspective distortion, for example, due to the slightly non-parallel X-rays allowed by the X-ray source 150.

[0073] System 100 can use the geometric distortion in the projection of reference plate 160 to determine the X-ray incident angle, and thus determine the projection angle and position of X-ray source 150 along arc C. In some examples, system 100 may rely on geometric formulas (e.g., the spacing of reference elements in a given direction is proportional to the cosine of the corresponding angle) to determine the projection angle. Additionally or alternatively, system 100 can obtain and use calibration data from known projection angles.

[0074] Figure 2A to Figure 2D A schematic example procedure for repairing reference marks in a projected image of a flexible elongation device positioned within an anatomical structure is illustrated. Figure 2A schematically shows a projected image of an anatomical structure A with a ROI or target R, a flexible elongation device 140, and a grid of reference marks. Figure 2A can be considered as... Figure 1F A schematic example of the projected image 180 in the image.

[0075] A portion of the restoration process performed by one or more processors of the system (e.g., processing unit 120 of system 100) may include identifying reference markers in the projected images. For this purpose, the system may use blob detection, spatial frequency analysis, cyclic Hough transform, or any other suitable image processing algorithm. In some examples, the system may use a machine learning model to detect the reference markers. The system may identify reference markers for each individual projection, or in some examples, the system may use correlations within a sequence of projections (e.g., within a fluorescence perspective video) to identify the reference markers. Additionally or alternatively, the system may use prior information about the markers, such as size, shape, and spacing, to aid in the detection and identification of the reference markers.

[0076] Another part of the repair process, performed by one or more processors of the system, includes generating a mask for masking and removing reference markers from the projected image. Figure 2B An example mask is shown. In some examples, the mask can be based on segmentation reference markers in the projected image. For example, the segmentation markers can be expanded to include pixels around the segmented region, thereby compensating for possible segmentation errors that might cause missed pixels. In other examples, detecting the reference markers can lead to the identification of the marker's center point, and then using prior knowledge about the markers to generate the mask. That is, if the system identifies the projection angle and center point of the markers in Figure 2A, the system can generate the mask based on prior knowledge about the marker's size, shape, and / or spacing. Figure 2B The mask in the middle.

[0077] Another part of the repair process, executed by one or more processors of the system, involves removing or zeroing out pixels from the projected image based on the generated mask, as shown in Figure 2C. Depending on the algorithm and data structures used in the repair of the projected image, pixel removal leaves the pixel value unknown, zero, or any other suitable value. In any case, the resulting gaps in information within the projected image are schematically shown in Figure 2C. The system can use appropriate repair techniques to fill these gaps, thereby producing... Figure 2D The image shown schematically illustrates the repaired image. The result of repairing the reference markers preserves as much information as possible related to the projection of the anatomical structure A, including the ROI or target R, and the projection of the flexible elongation device 140.

[0078] Inpainting techniques can include explicitly defined image processing algorithms and / or machine learning-based models (e.g., trained on projected image data). Algorithms can include hydrodynamic-based algorithms, such as methods based on Navel-Stokes gradients and / or diffusion-based methods. Additionally or alternatively, inpainting algorithms can be based on fast-marching methods. ML-based techniques can include progressive inpainting, attention-based inpainting, and / or diversification-based inpainting. Progressive inpainting fills the image in a stepwise manner based on surrounding pixels or structures. Attention-based inpainting takes into account information from distant spatial locations. Diversification inpainting produces multiple results for a single image. Additional information, such as the correlation (or other suitable considerations) between projected images in a sequence, can guide the system's decisions regarding the selection of results generated by diversification inpainting. From a network architecture perspective, the ML model used for inpainting can be based on autoencoders, variational autoencoders (VAEs), generative adversarial networks (GAEs), or diffusion models.

[0079] In some examples, the system can be configured to repair pseudo-images of other aspects and / or elements of the projected image, such as objects in the operating environment and / or instruments (e.g., flexible elongation devices) set within the patient's anatomy.

[0080] Figure 2E to Figure 2H An example procedure for repairing a flexible elongation device in a projected image of an anatomical structure is schematically illustrated. Generally, the system can implement a repair procedure for a flexible elongation device 240 (which may be a flexible elongation device 140) disposed within an anatomical structure A, including a ROI or target R, in a similar sequence to the repair of reference markers discussed above. Figure 2E shows the original projected image, while... Figure 2F A mask 241 corresponding to the flexible elongation device 240 within the projected image is shown. (Reference) Figure 4CAn example process for generating mask 241 is discussed in more detail. Figure 2G shows a projected image of Figure 2E, in which pixels associated with mask 241 are removed from the image (as discussed with reference to Figure 2C), resulting in gaps G within the projected image of anatomical structure A. Figure 2H A projected image of Figure 2E is shown, in which the flexible elongation device 240 is removed and the gap G is filled using repair techniques, such as those discussed above.

[0081] Figure 3A An example multi-projection image acquisition geometry is schematically shown for a flexible elongation device 340 (which may be flexible elongation device 140 or 240) disposed within an anatomical structure A. The geometry is similar to that shown in Figure 1C above, where the anatomical structure A replaces the patient P to more clearly illustrate the generation of projected images of the anatomical structure A. For example, one projected image of the anatomical structure A and the flexible elongation device 340 may correspond to the X-ray source 150 at position C1 and the detector 155 at corresponding position D1. Another projected image of the anatomical structure A and the flexible elongation device 340 may correspond to the X-ray source 150 at position C2 and the detector 155 at corresponding position D2.

[0082] Figure 3B Figure 3C It is used for Figure 3A The image acquisition geometry is an example projected image. For example, image 301 may correspond to an X-ray source at position C1, and image 302 may correspond to an X-ray source at position C2. The processing unit of the system (e.g., system 100) (e.g., processing unit 120) may enable the display device (e.g., display unit 130) to generate a GUI and display projected image 301 and / or projected image 302 within the GUI.

[0083] The system can identify reference point 310 within the projected image 301. Reference point 310 can be the tip of the flexible elongation device 340 or another suitable reference point (e.g., it can be identified by a reference object located at the flexible elongation device 340, as mentioned above). Figure 1A(As described). In some examples, the system can automatically identify reference point 310. In other examples, the system can use, for example, a GUI generated at the display to prompt the user to identify reference point 310 within the displayed projected image 301. Furthermore, the system can identify reference point 320 within the projected image 302, which may correspond to the same physical point at the flexible elongation device 340 as reference point 310. As described above, the system can identify point 320 automatically or based on user input. Additionally, the system can identify additional points 322, 324 disposed at the flexible elongation device 340 within the projected image 302. Identifying points 310 to 324 may include identifying their respective two-dimensional coordinates within the projected images (e.g., images 301, 302). The system can use the identified points 310 to 320 (e.g., the identified two-dimensional coordinates) to identify curves corresponding to at least a portion of the flexible elongation device 340 within at least one of the projected images (e.g., image 302). Furthermore, the system can use the identified points 310 to 320 and / or the identified two-dimensional curves to reconstruct the corresponding portion of the flexible elongation device 340 in three dimensions. Additionally or alternatively, the system can use at least two projected images (e.g., projected images 301, 302) to register the imaging coordinate system with the sensing or instrument coordinate system using shape data received from the sensing system (e.g., sensing unit 115), as referenced. Figures 4A to 4C As described.

[0084] Figure 4A A rigid body 402 in two example coordinate systems is depicted to illustrate the concept of registration between the two coordinate systems. A first coordinate system 404 may correspond to coordinates, for example, shape data acquired or generated by a sensing unit 115. The sensing unit 115 may be part of a robotic unit configured to actuate and / or manipulate a flexible elongation device 140. Therefore, the first coordinate system 404 may represent the coordinate system of the robotic unit. A second coordinate system 406 may correspond to coordinates of an image acquired or generated by an imaging system, such as imaging unit 110.

[0085] Rigid body 402 may represent a portion (e.g., a length segment) of a flexible elongation device (e.g., flexible elongation device 140, 240, or 340). Although the flexible elongation device 140 is flexible, the short portion (e.g., an infinitesimal segment) may be considered rigid for all practical purposes.

[0086] The state (e.g., position and orientation) of the rigid body 402 in three-dimensional space can be described using a first coordinate system 404 and / or a second coordinate system 406. A rigid body without symmetry (e.g., rigid body 402) has six degrees of freedom (6 DOFs), and its position and orientation can be described using six coordinates. In the first coordinate system 404, the rigid body 402 can have coordinates (x, y, z, ...). , , (x, y, and z) where x, y, and z are the position coordinates of the center 408 of the rigid body 402 relative to the origin O along the axis of the first coordinate system 404. In other examples, position coordinates can be specified for any point within the rigid body 402, or in fact, for any point with a rigidly defined geometric relationship to the rigid body 402. , and The orientation of rigid body 402 can be described by orientation vector 409, which in... Figure 4A In the example, it originates from center 408 and passes through the middle of one of the small planes of rigid body 402. For example, It can be the elevation angle relative to the z-axis. It can be an azimuth angle parallel to the xy plane, and These can be the rotation angle of rigid body 402 about orientation vector 409. Alternatively, the three orientation coordinates can be the roll, pitch, and yaw of rigid body 402 relative to any suitable reference direction. Similar to the coordinates (x, y, z) of the first coordinate system 404. , , The coordinates of the second coordinate system 406 are (x', y', z'). ', ', ') describes the position (x', y', z') relative to the origin O' and the orientation relative to, for example, the z' axis of rigid body 402. ', ', Registration of the first coordinate system 404 with the second coordinate system 406 at least near rigid body 402 includes at least the coordinates (x, y, z, ...) , , ) and coordinates (x', y', z', ', ', Find a mapping (e.g., transformation, mathematical relation, etc.) between the points (x', y', z'). Another way to view the registration is that the mapping limits the corresponding points (u, v, w) near the position (x, y, z) to any point (u', v', w') near the position (x', y', z').

[0087] In the above discussion, it was assumed that the first coordinate system 404 and the second coordinate system 406 have the same scale, and that the transformation from the first coordinate system 404 to the second coordinate system 406 is rigid. However, in some examples, the mapping from the first coordinate system 404 to the second coordinate system 406 may include one or more scaling factors for the axes. Therefore, the mapping may include three translation variables, three rotation variables, and / or three scaling variables. Furthermore, each of these variables may depend on the position, and the mapping may include deformation.

[0088] In some examples, each of coordinate systems 404 and 406 is independently calibrated to achieve accurate and consistent scaling within a shared operating volume. The coordinate registration process can then be defined based on three translation constants and three rotation constants of the shared operating volume. In other examples, gradual changes in scaling within at least one of coordinate systems 404 and / or 406 may require the use of up to three translation variables and up to three rotation variables, each a function of the position within the shared operating volume.

[0089] Figure 4B An example coordinate transformation process is illustrated schematically. In one example, a processing unit (e.g., processing unit 120) can obtain or determine the position and orientation of a rigid body (e.g., rigid body 402) within a first coordinate system 404 as (x, y, z, ...). , , ), and its position and orientation within the second coordinate system 406 are (x', y', z', ', ', The processing unit can then generate a mapping M between the two coordinate systems 404 and 406. The processing unit can be configured to map a new position (u', v', w') within the second coordinate system 406 to the corresponding position (u, v, w) within the first coordinate system 404. Alternatively or additionally, the processing unit can be configured to map coordinates from the first coordinate system 404 to the second coordinate system 406. The mapping can be valid only in the region around (x, y, z). By collecting rigid body coordinates in both coordinate systems 404 and 406, the processing unit can extend the validity of the mapping over any portion of the shared operational volume of the two coordinate systems 404 and 406. For example, the processing unit can implement coordinate registration as a linear mapping:

[0090]

[0091] in, , and These are rotational parameters (e.g., roll, pitch, and yaw), S 11 S 22 and S33 d1, d2, and d3 are scaling parameters (which can be units, as discussed above), and d1, d2, and d3 are displacement factors. Therefore, using scaling factors, there can be nine mapping parameters, and six mapping parameters can exist when scaling can be ignored. As discussed above, the linear mapping can be a function of the input position coordinates (u', v', w'). For example, the processing unit can store and / or access a lookup table to find entries for mapping parameters corresponding to the input position coordinates. Because the lookup table can only have a finite number of recorded input coordinates (in this case, recorded coordinates), the system can use the entry corresponding to the recorded coordinate closest to the input coordinates. Alternatively, the system can interpolate mapping parameters corresponding to a set of recorded coordinates near the input coordinates. In other examples, the processing unit can store and / or access a polynomial, spline function, or another suitable fitting function that correlates the input coordinates with the mapping parameters. The mapping can correlate the shape data of the flexible elongation device obtained from the sensing system with the shape data of the flexible elongation device in the imaging coordinate system, as referenced... Figure 4C The above is discussed. Furthermore, the system can use mapping to update the position of the target (e.g., target R) within the instrument coordinate system, as referenced. Figure 5A As described.

[0092] Figure 4C An example geometry for extraction and coordinate registration of a flexible elongation device using shape data and projected images is depicted. Two-dimensional projections 410, 420 of the three-dimensional curve 430 represent at least a portion of the flexible elongation device (e.g., flexible elongation device 140, 240, or 340). Projections 410, 420 (which may be referred to as projection curves 410, 420) may resemble and / or represent two-dimensional projections of the flexible elongation device 340 in projected images 301, 302. Projections 410, 420 are depicted on orthogonal planes—the xz plane and the xy plane, respectively. Although orthogonal projections (e.g., at ±45°) can be obtained in the imaging geometry described above, reference... Figure 4C The discussion can be extended to nonorthogonal projections.

[0093] Depend on Figure 4C The three-dimensional image and coordinate system defined by the x, y, and z axes can be defined relative to the origin located at a reference point set at the flexible elongation device. For example, reference point 310 in projected image 301 (which may be the tip of the flexible elongation device 340) and reference point 320 in projected image 302 can be projections along the same reference point of the flexible elongation device 340, and the origin of the three-dimensional imaging coordinate system can be defined without any loss of generality. Therefore, as Figure 4C The depicted three-dimensional curves 430 of projections 410, 420 and the flexible elongation device section all intersect at the origin of the coordinate system.

[0094] A system (e.g., system 100) can be configured to reconstruct curve 430 based on a two-dimensional projected image including projection curves 410 and 420. To do this, the system can first identify the projection curve (e.g., projection curve 420) within at least one of the projection images (e.g., corresponding to a projection onto the xy-plane). When identified, the two-dimensional curve 420 can be represented as the centerline of the projection of the flexible elongating device. In some examples, the identification of the centerline curve within the projection image can be based at least in part on identifying several points along the curve (e.g., points 320, 322, and 324). The system can identify the points along the curve based on user input via a GUI displaying the corresponding projection or based on automatically extracting the two-dimensional curve corresponding to the flexible elongating device from the corresponding projection image using a suitable image processing algorithm.

[0095] In some examples, the system can extract a two-dimensional curve (e.g., curve 410) in a second projected image based on a two-dimensional curve (e.g., curve 420) extracted from the first projected image. The curves in the two projected images necessarily share the same coordinates along an axis defined by the intersection of the two projection planes (e.g., the x-axis relative to curves 410 and 420). Therefore, for a given point along one two-dimensional projection curve, a corresponding point can be found on the second two-dimensional projection curve. For example, a point on curve 420 can be projected onto the x-axis along projection ray 462. The system can find the corresponding point on curve 410 by tracing along projection ray 464 to find its intersection with curve 410. The system can identify the intersection as a high or low (depending on the format of the projected image) intensity point in the projection that intersects projection ray 464 in the projected image. In an example where another point along curve 420 is projected onto the x-axis via ray 466, the corresponding grayscale 468 intersects curve 410 at at least two points. The system can resolve ambiguity by selecting points that are adjacent to the previously detected points along curve 410.

[0096] Once the system identifies two corresponding points in the projected image, the corresponding point in the three-dimensional image can be found, for example, at the intersection 470 of the back-projection rays emanating from the two corresponding points in the two-dimensional projected image along their respective projection angles. In some examples, the system can use more than two projections to identify corresponding points in each of the projections, and back-project the identified points to find the corresponding point in the three-dimensional image. In the presence of measurement uncertainties, the back-projected rays may not intersect. The system can then find in the three-dimensional image the point that minimizes a metric of the distance between the three-dimensional points in the back-projection rays (e.g., root mean square or L2 norm). In the manner described above, the curve 430 in the three-dimensional image can be reconstructed from the curves 410 and 420 in the two-dimensional projection.

[0097] In other examples, the system can reconstruct curve 430 in 3D based on curve 420 (or curve 410) by first reconstructing the imaging volume based on the entire set of 2D projections. Curve 430 in the resulting 3D reconstructed image can be calculated by back-tracing or back-projecting from points along curve 420 along rays corresponding to the projection angles. The system can identify corresponding points along curve 430 in a manner similar to identifying corresponding points on 2D curve 410, based on appropriate intensity thresholds and continuity requirements, as discussed above.

[0098] In some examples, the system can use the reconstructed curve 430 to register the imaging coordinate system with the sensing coordinate system. For this purpose, the system can receive shape data from a sensing unit (e.g., sensing unit 115) and identify and define shapes within the shape data. Figure 4C The coordinates of the sensing system correspond to the reference point of the origin of the imaging coordinate system (e.g., the tip of the flexible elongation device). The first step of registration may be: limiting transformation (e.g., reference...) Figure 4B The translation portion of the transformation (M) describes placing the reference point at the common origin of the coordinate systems registered by translation. The shape data from the sensing system can then be represented by the dashed curve 480, assuming identical scales in both coordinate systems, as discussed above. The remainder of the registration may include finding the roll, pitch, and yaw angles that align shape data curve 480 with curve 430. Although perfect alignment may be impossible due to noise and distortion in the shape data received from the sensing unit and calculated based on imaging data from the imaging unit, optimization algorithms can minimize the cost function indicating misalignment or misregistration. For example, with or without regularization taking into account previous transformation data and / or other suitable constraints, the system can compute the registration transformation between shape data 480 and curve 430 in a least-squares sense. Additionally or alternatively, the system can minimize the weighted least-squares difference, thus assigning higher weight to alignments near the origin than alignments far from the origin. Assigning higher weight to alignment near the origin may be particularly useful in the following example: when the tip of a flexible elongation device is close to an anatomical target (e.g., target R), the reference point at the origin of the registration coordinate system is at the tip.

[0099] In some examples, registration between the imaging coordinate system and the sensing coordinate system does not need to follow the extraction of curve 430 in 3D. Instead, the system can use shape data 480 to simultaneously extract curve 430 and register the coordinate system. To do this, the system can calculate a coordinate transformation to align the projection of shape data 480 with at least one of the projected curves 410 and 420. In this way, shape data 480 from the sensing unit can constrain the reconstruction of curve 430.

[0100] Regardless of the process by which the system calculates the coordinate registration and extraction of the 3D curve 430, the system can use the registration transformation and curve extraction results in at least two ways. First, reprojecting the 3D curve 430 onto a 2D projection set facilitates the generation of a repair mask for each projection. For example, the projected curve can be expanded and used as a mask, such as a reference. Figure 2E to Figure 2H The second point discussed is that registration between coordinate systems can facilitate updating the position of anatomical targets in the sensing (instrument) coordinates, such as reference... Figure 5A , Figure 5B This will be discussed in more detail. Furthermore, the restoration can help update the target location by implementing image reconstruction that minimizes artifacts caused by flexible elongation devices near the target. Improved image quality allows the operator to more accurately identify the target location in the reconstructed image.

[0101] Figure 5A , Figure 5B Example processes 510, 520, 530, 540, 550, 554, 556, 558, and 560 for extraction, coordinate registration, and target updating using a flexible elongation device with shape data and projected images are schematically illustrated. A system (e.g., system 100) may use one or more processors of a processing unit (e.g., processing unit 120) and a display unit (e.g., display unit 130) to execute processes 510, 520, 530, 540, 550, 554, 556, 558, and 560.

[0102] The system can (e.g., from imaging unit 110) obtain a set of two-dimensional projected images 502 corresponding to a set of corresponding projection angles, for example, obtained using the imaging geometry discussed with reference to FIG. 1C. The system can select projected images 504, 506 from set 502, which can be, for example, projected images 301, 302. Within each of projected images 504, 506, the system can identify corresponding pairs of reference points (e.g., corresponding to the tip or distal end of the flexible elongation device), such as, for example, points 310 and 320. For this purpose, the system can generate a GUI on the display unit and prompt the operator to select the corresponding point. In other examples, the system can automatically select the corresponding point.

[0103] Using corresponding point pairs in projected images 504 and 506 as input, the system can perform a triangulation process 510 to calculate the three-dimensional coordinates of the reference point in the imaging coordinate system. Without any general loss, the output of the triangulation process 510 can be an imaging coordinate system 512 with the reference point as its origin, such as the reference... Figure 4C As described.

[0104] Using the projected image 504 as input, the system can perform a 2D centerline extraction process 520 to identify within the projection 504 a curve 522 (e.g., curve 420) corresponding to at least a portion of the flexible elongation device, including reference points. For this purpose, the system can generate a GUI on the display unit and prompt the operator to select multiple points on the curve 522. Alternatively or additionally, the system can use one or more image processing algorithms to identify the curve 522.

[0105] Using coordinate system 512, curve 522, and projection 506 as input, the system can perform a reverse tracing process to generate a three-dimensional curve 532 (e.g., curve 430) in the imaging system coordinates 512. In some examples, the reverse tracing process 530 can use a projected image set 502, such as a reference image set 502. Figure 4C As described.

[0106] The system can use the generated curve 532 and the shape data 535 obtained from the sensor unit (e.g., sensing unit 115) to calculate the transformation 542 between the image coordinate system 512 and the instrument coordinate system, such as by referring to... Figure 4C As discussed, alternatively, the system may use registration process 540 when generating curve 532.

[0107] In any case, the system can use curve 532 and projection set 502 together as inputs for reconstruction using the repair process 550. (See above reference.) Figure 4C and Figure 2E to Figure 2H As discussed, process 550 may include backprojecting curve 532 onto a projection image set 502 to generate a repair mask. Reconstruction using repair process 550 can generate a reconstructed three-dimensional image 552 of the patient's anatomy with target 553. Three-dimensional image 552 can remove artifacts caused by the presence of flexible elongation devices disposed within the patient's anatomy, thereby providing a clear image of target 553. It should be noted that in some examples, tomographic synthesis reconstruction from image set 502 can be performed without repair. That is, process 550 can be replaced by a direct tomographic synthesis reconstruction process instead of referencing... Figure 5B The sequence of subprocesses 554, 556, and 558 under discussion.

[0108] The system can use the reconstructed image 552 and transformation 542 together as input to the target update process 560. In some examples, the target update process 560 may include generating a 3D image 552 and / or a corresponding tomographic image at the GUI and prompting the operator to identify the center of the target 563. In other examples, the system may automatically identify and / or segment the target 563 within image 552. Transformation 542 can be used to map the center of the target 553 identified based on image 552 in the image and coordinate system onto the instrument coordinate system 562. In some examples, the system may locate the target 563 based solely on transformation 542 and the target 553 from image 552. In other examples, the system may use the target update process 560 and prior information about the target 553 (e.g., previous updates based on intraoperative and / or preoperative images) to update the coordinates of the target 563 in the instrument coordinate system 562.

[0109] Figure 5B The sub-processes 554, 556, and 558, which include process 550, are schematically shown, for example, with reference to [reference needed]. Figure 2E to Figure 2H and Figure 4C The mask generation process 554, as discussed above, generates a mask 555 based on curve 532 by projecting an expanded version of curve 532 onto projection set 502. The repair process 556 can be performed as described above. Figure 2A to Figure 2H The repair discussed results in the output of projection set 557 after the flexible elongation device is removed. In some examples, the repair process may also remove the reference markers from projection set 557. The system can then reconstruct the repaired projection 557 using any suitable tomographic synthesis process 558 (e.g., using FDK reconstruction or any other suitable reconstruction technique) to generate a three-dimensional image 552.

[0110] It should be noted that the system can additionally enhance image 552 before identifying target 553. For example, the system can use the following reference... Figure 6A , Figure 6B The style transfer process discussed.

[0111] Figure 6AAn example style transfer for 3D images is schematically illustrated. Scene 600 (as discussed above, which may include a patient's anatomical structure with anatomical targets, devices set within the patient's anatomical structure, one or more reference objects for aiding, for example, image analysis and / or reconstruction) can be imaged using a selected imaging modality. Example image 601 may be the result of imaging using a first imaging modality (e.g., CT or CBCT), and image 602 may be the result of a second imaging modality (e.g., tomographic synthesis of fluorescence fluoroscopy images). Generally, the first modality may be a modality with higher fidelity than the second modality, and therefore image 601 may be an image with higher fidelity than image 602. In some examples, the system (e.g., system 100) may use a machine learning and / or deep learning (ML / DL) model 610 to generate an image with properties similar to image 601 based on image 602. That is, the image generated by model 610 can not only look like image 601, but can also have higher fidelity than image 602. In a sense, when effectively trained, Model 610 can be viewed as performing denoising, deconvolution, image restoration, and other image enhancement processes.

[0112] In some examples, model 610 may use or be based on a convolutional neural network (CNN) (e.g., U-net) and / or a transformer network. Additionally or alternatively, model 610 may be a generative adversarial network (GAN) model and / or a diffusion model. Model 610 may be based on a combination of models to generate multiple outputs from a single image input, and select one of the outputs based on the operator's choice or one or more computed metrics of the output.

[0113] In some examples, the system can render one or more outputs of the model on the system's display unit (e.g., display unit 130).

[0114] Figure 6B An example training procedure 620 for a style transfer ML / DL model 630 (e.g., model 610) is illustrated schematically. The training procedure 620 may, but does not need to, be implemented by the system using the model (e.g., system 100). The training procedure can be performed by another suitable system with appropriate computational resources, and the trained model can be transferred to a system for intraoperative imaging.

[0115] Training process 630 may begin by selecting example high-fidelity volumetric data 630 from, for example, a high-fidelity first modal image (e.g., an image such as image 1). The high-fidelity volumetric data 630 can be used as the ground truth for training process 620. The volumetric data 630 may be, for example, reconstructed CT or CBCT data and may include associated projection data. Based on the reconstructed data or based on the original projection data associated with the high-fidelity volumetric model 630, the process may generate a simulated second modal sine curve (e.g., a simulated set of fluorescence fluoroscopic images for tomographic synthesis). In some examples, generating the simulated sine curve may include selecting projections from a finite set of projection angles (e.g., 120°, 100°, 110°, 90°, 80°, 70°, 60°, or any other suitable span). In other examples, generating the simulated sine curve may include simulating projections based on interpolated volumetric data using, for example, an X-ray propagation model. Generating the simulated sine curve may include adding noise and / or distortion to the simulated projection and / or adding noise and / or distortion to the projection angle associated with the projection. Based on the simulated second-modal sine curve 640, process 620 may generate reconstructed second-modal volumetric data 650 (e.g., tomographic reconstruction) based on any suitable reconstruction algorithm (e.g., FDK). The reconstructed data 650 is used as training input to model 660, and model 660 is trained according to any suitable training technique to minimize the loss relative to the true values ​​of the high-fidelity volumetric data 630. Model 660, thus trained with an appropriate amount of training data, can then be used as a style transfer model to enhance images generated from lower-fidelity modalities (e.g., tomographic synthesis) to generate higher-fidelity outputs that may resemble higher-fidelity (e.g., CT or CBCT) reconstructions. The high-fidelity image output can facilitate, for example, the recognition of anatomical targets, such as reference images. Figure 5A , Figure 5B The subject of discussion.

[0116] Figures 7 to 9B A diagram depicts, in some examples, a medical system that can be used to manipulate a medical device including a flexible elongation device according to any of the methods and systems described above. For example, each reference above to "system" may refer to the system (e.g., system 700) or its subsystems discussed below.

[0117] Figure 7This is a simplified diagram of a medical system 700 based on some examples. Medical system 700 may include at least a portion of system 100 described with reference to Figure 1. Medical system 700 may be applicable to procedures such as surgery, diagnosis (e.g., biopsy), or treatment (e.g., ablation, electroporation, etc.). While some examples of such procedures are provided herein, any references to medical or surgical instruments and methods are non-limiting. The systems, instruments, and methods described herein can be used with animal or human cadavers, animal carcasses, portions of human or animal anatomy, for non-surgical diagnostics, and for industrial systems, general-purpose or special-purpose robotic systems, general-purpose or special-purpose remote operating systems, or robotic medical systems.

[0118] like Figure 7 As shown, the medical system 700 may include a manipulator assembly 702 that controls the operation of a medical device 704 to perform various procedures on a patient (e.g., patient P on a table T as shown in FIG. 1). The medical device 704 may include the flexible extension device 140 of FIG. 1. The medical device 704 may extend into an internal portion of the patient P's body via an opening within the patient P's body. The manipulator assembly 702 may be a remotely operated, non-remotely operated, or hybrid remotely and non-remotely operated assembly, having one or more degrees of freedom that can be electrically operated and / or one or more degrees of freedom that can be non-electrically operated (e.g., manually operated). The manipulator assembly 702 may be mounted to and / or positioned near the patient table T. A master assembly 706 enables an operator O (e.g., a surgeon, clinician, internist, or other user as described above) to control the manipulator assembly 702. In some examples, the master assembly 706 enables the operator O to view the procedure site or other graphical or information displays. In some examples, the manipulator component 702 may be excluded from the medical system 700, and the medical device 704 may be directly controlled by operator O. In some examples, the manipulator component 702 may be manually controlled by operator O. Direct operator control may include various handles and operator interfaces for handheld operation of the medical device 704.

[0119] The main component 706 may be located at a surgeon's console near the patient table T where the patient P is located (e.g., in the same room as the patient table T), such as on the side of the patient table T. In some examples, the main component 706 is located away from the patient table T, such as in a different room or a different building. The main component 706 may include one or more control devices for controlling the manipulator component 702. The control devices may include any number of various input devices, such as joysticks, trackballs, rollers, steering pads, buttons, data gloves, trigger guns, manual controllers, voice recognition devices, motion or presence sensors, etc.

[0120] Manipulator assembly 702 supports medical device 704 and may include a kinematic structure of links providing a setting structure. Links may include one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place) and / or one or more servo-controlled links (e.g., one or more links that can be controlled in response to commands, for example, from control system 712). Manipulator assembly 702 may include a plurality of actuators (e.g., motors) that drive inputs on medical device 704 in response to commands, for example, from control system 712. Actuators may include a drive system that moves medical device 704 in various ways when coupled to it. For example, one or more actuators may advance medical device 704 into a natural or surgically generated anatomical opening. Actuators may control engagement of medical device 704, for example, by moving the distal end (or any other part) of medical device 704 in multiple degrees of freedom. These degrees of freedom may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). One or more actuators may control the rotation of the medical device about its longitudinal axis. Actuators may also be used to move the engageable end effector of the medical device 704 (e.g., for grasping tissue in the jaws of a biopsy device, etc.), or may be used to move or otherwise control tools inserted within the medical device 704 (e.g., imaging tools, ablation tools, biopsy tools, electroporation tools, etc.).

[0121] The control system 712 may include at least a portion of the processing unit 120. Additionally or alternatively, the control system 712 may be communicatively connected to the processing unit 120. In some examples, the output of the processing unit 120 according to the technology described above may enable the control system 712 to autonomously (without input from operator O) control certain movements of the medical device 704.

[0122] The medical system 700 may include a sensor system 708 (which may include at least a portion of a sensing unit 115) having one or more subsystems for receiving information about the manipulator assembly 702 and / or the medical device 704. Such subsystems may include: a position sensor system (e.g., using an electromagnetic (EM) sensor or other type of sensor for detecting position or orientation); a shape sensor system for determining position, orientation, velocity, rate, pose, and / or shape along one or more segments and / or distal ends of the flexible body of the medical device 704; a visualization system for capturing images, for example, from the distal end of the medical device 704 or from some other location; and / or an actuator position sensor (e.g., a resolver, encoder, potentiometer, etc.) describing the rotation and / or orientation of the actuator controlling the medical device 704. The subsystem may include an imaging subsystem (e.g., using a color imaging device, an infrared imaging device, an ultrasound imaging device, an X-ray imaging device, a fluorescence imaging device, a computed tomography (CT) imaging device, a magnetic resonance imaging (MRI) imaging device, or some other type of imaging device), such as imaging unit 110.

[0123] It should be noted that the position and orientation of the sensors in sensor system 708 can be determined in the sensor coordinate system. In some examples, the sensor coordinate system is integrated with or the same as the coordinate system of the manipulator assembly 702.

[0124] Medical system 700 may include a display system 710 (e.g., display unit 130) for displaying images or representations of the procedure site and medical device 704. The display system 710 and main component 706 may be oriented such that a physician O can use remote presence to control the medical device 704 and main component 706. The display system 710 may include at least a portion of the display unit 130.

[0125] In some examples, medical device 704 may include a visualization system that includes an image capture component that records simultaneous or real-time images of the procedure site and provides the images to the operator O via one or more displays of display system 710. The image capture component may include various types of imaging devices. The simultaneous images may be, for example, two-dimensional or three-dimensional images captured by an endoscope positioned within the anatomical procedure site. The visualization system may obtain intraoperative images in image system coordinates different from the sensor system coordinates. In some examples, the visualization system may include an endoscope component that may be integrally or detachably coupled to medical device 704. Additionally or alternatively, a separate endoscope attached to a separate manipulator assembly may be used with medical device 704 to image the procedure site. The visualization system may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, such as control system 712.

[0126] Display system 710 can also display images of the process site and medical device, which can be captured by a visualization system. In some examples, medical system 700 provides operator O with a remote sense of presence. For example, an image captured by an imaging device at the distal portion of medical device 704 can be presented by display system 710 to provide operator O with a perception of the distal portion of medical device 704. Input provided by operator O to main component 706 can move the distal portion of medical device 704 in a manner corresponding to the nature of the input (e.g., the distal end turns to the right when the trackball rolls to the right), and cause a corresponding change in the viewing angle of the image captured by the imaging device at the distal portion of medical device 704. Thus, operator O's remote sense of presence is maintained when medical device 704 is moved using main component 706. Operator O can manipulate the hand controls of main component 706 and medical device 704 as if viewing a workspace in a substantially real-world setting, simulating the experience of physically manipulating medical device 704 from within the patient's anatomy.

[0127] In some examples, the display system 710 can present virtual images of the procedure site created using image data recorded preoperatively (e.g., before surgery performed by the medical device system 200) or intraoperatively (e.g., simultaneously with surgery performed by the medical device system 200), such as image data created using computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), fluoroscopy, thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc. The virtual images can include two-dimensional, three-dimensional, or higher-dimensional images (e.g., including information based on time or rate). In some examples, one or more models are created based on preoperative or intraoperative image datasets, and the virtual image is generated using one or more models.

[0128] In some examples, for the purpose of image-guided medical procedures, the display system 710 can display a virtual image generated based on the position of the tracking medical device 704. For example, the tracking position of the medical device 704 can be registered with a model generated using preoperative or intraoperative images (e.g., a dynamic reference), where different parts of the model correspond to different locations of the patient's anatomy. As the medical device 704 moves through the patient's anatomy, the registration is used to determine the parts of the model corresponding to the position and / or viewpoint of the medical device 704, and a virtual image is generated using the determined parts of the model. This can be accomplished to present a virtual image of the internal procedure site corresponding to the tracking position of the medical device 704 to the operator O from the viewpoint of the medical device 704.

[0129] The display system 710 may include a display unit 130 and may display an image including the position, orientation, and / or pose of the medical device 704 in a joint coordinate system based on the reference above. Figure 2A to The technique described in Figure 6 is used to register the sensor coordinate system with the imaging coordinate system.

[0130] The medical system 700 may also include a control system 712, which may include a processing circuitry (e.g., processing unit 120) that implements some or all of the methods or functions discussed herein. The control system 712 may include at least one memory and at least one processor for controlling the operation of the manipulator assembly 702, medical device 704, main assembly 706, sensor system 708, and / or display system 710. The control system 712 may include instructions (e.g., a non-transitory machine-readable medium storing instructions) that, when executed by at least one processor, configure one or more processors to implement some or all of the methods or functions discussed herein. Although the control system 712... Figure 7While shown as a single block, the control system 712 may include two or more separate data processing circuits, with some processing performed at the manipulator component 702, others at the main component 706, and so on. In some examples, the control system 712 may include other types of processing circuitry systems, such as application-specific integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs). The control system 712 may be implemented using hardware, firmware, software, or a combination thereof.

[0131] In some examples, the control system 712 may receive feedback from the medical device 704, such as force and / or torque feedback. In response to this feedback, the control system 712 may transmit a signal to the main component 706. In some examples, the control system 712 may transmit a signal instructing one or more actuators of the manipulator component 702 to move the medical device 704. In some examples, the control system 712 may transmit information about the feedback to a display system 710 for presentation or to perform other types of actions based on the feedback.

[0132] The control system 712 may include a virtual visualization system to provide navigational assistance to the operator O when controlling the medical device 704 during an image-guided medical procedure. Virtual navigation using the virtual visualization system may be based on a preoperative or intraoperative dataset of the anatomical pathways of the patient P acquired. The control system 712 or a separate computing device may, alone or in combination with operator input, use programmed instructions to transform recorded images into a model of the patient's anatomy. This model may include a segmented two-dimensional or three-dimensional synthetic representation of parts or entire anatomical organs or regions. The image dataset may be associated with the synthetic representation. The virtual visualization system may obtain sensor data from the sensor system 708 for calculating the (e.g., approximate) position of the medical device 704 relative to the anatomical structures of the patient P. The sensor system 708 may be used to register and display the medical device 704 and the images recorded preoperatively or intraoperatively. For example, PCT disclosure WO 2016 / 191298 (published December 1, 2016, entitled "Systems and Methods of Registration for Image Guided Surgery") discloses an example system, which is incorporated herein by reference in its entirety. Additional or alternative sites, registration can be based on the above references. Figure 2A to The technology discussed in Figure 6.

[0133] During the virtual navigation process, sensor system 708 can be used to calculate the (e.g., approximate) position of medical device 704 relative to the anatomical structure of patient P. This position can be used to generate both a macroscopic (e.g., external) tracking image of the anatomical structure of patient P and a virtual internal image of the anatomical structure of patient P. The system may include one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to register and display the medical device and preoperatively recorded medical images. For example, U.S. Patent No. 8,900,131 (filed May 13, 2011, entitled “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery”), which is incorporated herein by reference in its entirety.

[0134] The medical system 700 may also include operating and support systems (not shown), such as lighting systems, steering and maneuvering control systems, flushing systems, and / or suction systems. In some examples, the medical system 700 may include more than one manipulator assembly and / or more than one master assembly. The exact number of manipulator assemblies may depend on factors such as the medical procedure and space constraints within the operating room. Multiple master assemblies may be located in the same location or may be positioned in separate locations. Multiple master assemblies allow more than one operator to control one or more manipulator assemblies in various combinations.

[0135] Figure 8A This is a simplified diagram of a medical device system 800 based on some examples. The medical device system 800 includes a flexible elongation device 802 (e.g., device 140) (also referred to as elongation device 802), a drive unit 804, and a medical instrument 826, which together serve as an example of a medical instrument 704 of the medical system 700. The medical system 700 can be a remote operating system, a non-remote operating system, or a hybrid of remote and non-remote operating systems, as shown in reference... Figure 7 As described above. The visualization system 831, tracking system 830, and navigation system 832 are also included. Figure 8A The diagram shows an example component of the control system 712 of the medical system 700. In some examples, the medical device system 800 can be used in non-remotely operated exploration procedures or in procedures involving routine manual operation of the medical device (e.g., endoscopy). The medical device system 800 can be used to collect (e.g., measure) a set of data points corresponding to positions within the anatomical passage of a patient (e.g., patient P).

[0136] The elongation device 802 is coupled to the drive unit 804. The elongation device 802 includes a channel 821 through which a medical instrument 826 can be inserted. The elongation device 802 navigates within the patient's anatomy to deliver the medical instrument 826 to the procedure site. The elongation device 802 includes a flexible body 816 having a proximal end 817 and a distal end 818. In some examples, the flexible body 816 may have an outer diameter of approximately 3 mm. Other flexible bodies may have larger or smaller outer diameters.

[0137] Medical device system 800 may include a tracking system 830 for determining the position, orientation, velocity, rate, pose, and / or shape of a flexible body 816 at its distal end 818 and / or along one or more segments 824 of the flexible body 816, as will be described in further detail below. Tracking system 830 may include one or more sensors and / or imaging devices. The flexible body 816 (e.g., the length between the distal end 818 and the proximal end 817) may include multiple segments 824. Tracking system 830 may be implemented using hardware, firmware, software, or a combination thereof. In some examples, tracking system 830 is... Figure 7 This is part of the control system 712 shown. The tracking system 830 can realize a reference. Figure 1A At least some of the techniques described in Figure 6, and for this purpose, may include Figure 1A At least a portion of the processing unit 120 or related to Figure 1A The processing unit 120 is connected to the communication network.

[0138] The tracking system 830 can use a shape sensor 822 to track the distal end 818 and / or one or more segments 824 of the flexible body 816. It should be noted that the shape sensor 822 can be omitted when utilizing the techniques of this disclosure. The shape sensor 822 may include an optical fiber aligned with the flexible body 816 (e.g., disposed within an internal channel of the flexible body 816 or mounted externally along the flexible body 816). In some examples, the optical fiber may have a diameter of about 800 μm. In other examples, the diameter may be larger or smaller. The optical fiber of the shape sensor 822 can form an optical fiber bending sensor for determining the shape of the flexible body 816. An optical fiber including a fiber Bragg grating (FBG) can be used to provide strain measurements of the structure in one or more dimensions. Various systems and methods applicable to monitoring the shape and relative position of optical fibers in three dimensions are described in U.S. Patent Application Publication No. 2006 / 0013523 (filed July 13, 2005, entitled "Fiber optic position and shape sensing device and method relating thereto"), U.S. Patent No. 7,772,541 (filed March 12, 2008, entitled "Fiber Optic Position and / or Shape Sensing Based on Rayleigh Scatter"), and U.S. Patent No. 8,773,650 (filed September 2, 2010, entitled "Optical Position and / or Shape Sensing"), all of which are incorporated herein by reference in their entirety. In some examples, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering.

[0139] In some examples, other techniques may be used to determine the shape of the flexible body 816. For example, the history of the position and / or pose of the distal end 818 of the flexible body 816 may be used to reconstruct the shape of the flexible body 816 over time intervals, such as when the flexible body 816 advances or retracts within a patient's anatomy. In some examples, the tracking system 830 may alternatively and / or additionally use a position sensor system 820 to track the distal end 818 of the flexible body 816. The position sensor system 820 may be a component of an EM sensor system, wherein the position sensor system 820 includes one or more position sensors. Although the position sensor system 820 is shown proximity to the distal end 818 of the flexible body 816 to track the distal end 818, the number and position of the position sensors in the position sensor system 820 may vary to track different regions along the flexible body 816. In one example, the position sensors include conductive coils that can withstand externally generated electromagnetic fields. Each coil of the position sensor system 820 may generate an induced electrical signal having characteristics that depend on the position and orientation of the coil relative to the externally generated electromagnetic field. The position sensor system 820 can measure one or more position coordinates and / or one or more orientation angles associated with one or more portions of the flexible body 816. In some examples, the position sensor system 820 can be configured and positioned to measure six degrees of freedom, such as three position coordinates X, Y, and Z, and three orientation angles indicating pitch, yaw, and roll of a reference point. In some examples, the position sensor system 820 can be configured and positioned to measure five degrees of freedom, such as three position coordinates X, Y, and Z, and two orientation angles indicating pitch and yaw of a reference point. Further description of the position sensor system applicable to some examples is provided in U.S. Patent No. 6,380,732 (filed August 11, 1999, entitled "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked"), which is incorporated herein by reference in its entirety.

[0140] According to the present disclosure described above with reference to FIG2A-6, a processing unit (e.g., processing unit 120) can enhance the accuracy of the position obtained by the position sensor system 820 by combining data obtained by the position sensor system 820 with data obtained by an external imaging system (e.g., by means of imaging unit 110).

[0141] In some examples, the tracking system 830 may alternatively and / or additionally rely on a set of stored pose, position, and / or orientation data for points on the elongation device 802 and / or medical instrument 826, captured during one or more cycles of alternating movement (e.g., breathing). This stored data can be used to develop shape information about the flexible body 816. In some examples, a series of position sensors (not shown), such as EM sensors like those in the position sensor system 820, or some other type of position sensor, may be positioned along the flexible body 816 and used for shape sensing. In some examples, data history acquired during surgery from one or more of these position sensors can be used to represent the shape of the elongation device 802, particularly where the anatomical passage is typically static.

[0142] Figure 8B This is a simplified diagram of a medical tool 826 within an elongation device 802, based on some examples. The flexible body 816 of the elongation device 802 may include a channel 821 sized and shaped to accommodate the medical tool 826. In some examples, the medical tool 826 may be used for procedures such as imaging, surgery, biopsy, ablation, illumination, irrigation, aspiration, electroporation, etc. The medical tool 826 can be deployed through the channel 821 of the flexible body 816 and operated at a procedure site within an anatomical structure. The medical tool 826 may be, for example, an image capture probe, a biopsy tool (e.g., a needle, gripper, brush, etc.), an ablation tool (e.g., a laser ablation tool, a radiofrequency (RF) ablation tool, a cryoablation tool, a thermal ablation tool, a heated liquid ablation tool, etc.), an electroporation tool, and / or another surgical, diagnostic, or therapeutic tool. In some examples, the medical tool 826 may include an end effector with a single working member, such as a scalpel, a blunt blade, an optical fiber, an electrode, etc. Other end effector types can include, for example, forceps, grippers, scissors, sutures, clamps, etc. Other end effectors can also include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, etc.

[0143] Medical tool 826 may be a biopsy tool for removing a sample of tissue or cells from a target anatomical location. In some examples, the biopsy tool is a flexible needle. The biopsy tool may also include a sheath that can surround the flexible needle to protect the needle and the inner surface of the channel 821 when the biopsy tool is located within the channel 821. Medical tool 826 may be an image capture probe that includes a distal portion having a stereo or single-field-of-view camera, which may be positioned at or near the distal end 818 of the flexible body 816 for capturing images (e.g., still or video images). The captured images may be processed by visualization system 831 for display and / or provided to tracking system 830 to support tracking of one or more segments of the distal end 818 of the flexible body 816 and / or segments 824 of the flexible body 816. The image capture probe may include a cable for transmitting captured image data, the cable being coupled to an imaging device at the distal portion of the image capture probe. In some examples, the image capture probe may include a bundle of optical fibers, such as a fiber optic endoscope, coupled to a more proximal imaging device coupled to the visualization system 831. The image capture probe may be monospectral or multispectral, for example, capturing image data in one or more of the visible, near-infrared, infrared, and / or ultraviolet spectra. The image capture probe may also include one or more light emitters that provide illumination to facilitate image capture. In some examples, the image capture probe may use ultrasound, X-ray, fluoroscopy, CT, MRI, or other types of imaging techniques.

[0144] In some examples, an image capture probe is inserted within the flexible body 816 of the elongation device 802 to facilitate visual navigation of the elongation device 802 to the procedure site, and then the image capture probe is replaced within the flexible body 816 with another type of medical instrument 826 for performing the procedure. In some examples, the image capture probe may be located within the flexible body 816 of the elongation device 802 along with another type of medical instrument 826 to facilitate simultaneous image capture and tissue intervention, for example, within the same channel 821 or in different channels. The medical instrument 826 may advance from an opening in the channel 821 to perform the procedure (or some other function) and then retract into the channel 821 when the procedure is complete. The medical instrument 826 may be removed from the proximal end 817 of the flexible body 816 or along the flexible body 816 from another optional instrument port (not shown).

[0145] In some examples, the extension device 802 may include integrated imaging capabilities instead of utilizing a removable image capture probe. For example, the imaging device (or fiber bundle) and light emitter may be located at the distal end 818 of the extension device 802. The flexible body 815 may include one or more dedicated channels carrying cables and / or optical fibers between the distal end 818 and the visualization system 831. Here, the medical device system 800 can perform imaging and tooling operations simultaneously.

[0146] In some examples, the medical tool 826 is capable of controlled engagement. The medical tool 826 may house a cable (also referred to as a traction cable), linkage, or other actuation control (not shown), extending between its proximal and distal ends to controllably bend the distal end of the medical tool 826, such as those discussed herein with respect to the flexible elongation device 802. The medical tool 826 may be coupled to the drive unit 804 and the manipulator assembly 702. In these examples, the elongation device 802 may be excluded from the medical device system 800, or may be a flexible device without controlled engagement. The steerable maneuvering apparatus or tool applicable to some examples is further described in detail in U.S. Patent No. 7,316,681 (filed October 4, 2005, entitled "Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity") and U.S. Patent No. 9,259,274 (filed September 30, 2008, entitled "Passive Preload and Capstan Drive for Surgical Instruments"), which are incorporated herein by reference in their entirety.

[0147] The flexible body 816 of the elongation device 802 may also, or alternatively, accommodate cables, linkages, or other steering control (not shown) extending between the drive unit 804 and the distal end 818 to controllably bend the distal end 818, for example, as depicted by the dashed line 819 of the distal end 818 in FIG2A. In some examples, at least four cables are used to provide independent up-and-down steering control to control the pitch of the distal end 818 and left-and-right steering control to control the yaw of the distal end 881. In these examples, the flexible elongation device 802 may be a steerable conduit. Examples of steerable conduits suitable for some examples are described in detail in PCT Publication WO 2019 / 018736 (published January 24, 2019, entitled "Flexible Elongate Device Systems and Methods"), which is incorporated herein by reference in its entirety.

[0148] In examples where the elongation device 802 and / or medical tool 826 is actuated by a remotely operated component (e.g., manipulator component 702), the drive unit 804 may include a drive element (e.g., an actuator) removably coupled to and receiving power from the remotely operated component. In some examples, the elongation device 802 and / or medical tool 826 may include a grasping feature, a manual actuator, or other components for manually controlling the movement of the elongation device 802 and / or medical tool 826. The elongation device 802 may be steerable, or alternatively, it may be non-steerable and lack an integrated mechanism for operator control of bending of the distal end 818. In some examples, one or more channels 821 (also referred to as lumens) may be defined by the inner wall of the flexible body 816 of the elongation device 802, through which the medical tool 826 may be deployed and used at a target anatomical location.

[0149] In some examples, medical device system 800 (e.g., extension device 802 or medical tool 826) may include, for example, flexible bronchial instruments such as bronchoscopes or bronchial tubes for visual examination and diagnosis, biopsy, and / or treatment of the lungs. Medical device system 800 may also be adapted to navigate and treat other tissues in any anatomical system of a variety of anatomical systems via naturally or surgically generated access channels, including the colon, intestine, kidneys and renal calyces, brain, heart, circulatory system including the vascular system, etc.

[0150] Information from tracking system 830 can be sent to navigation system 832, where it can be combined with information from visualization system 831 and / or preoperatively acquired models to provide real-time location information to physicians, clinicians, surgeons, or other operators. Tracking system 830, navigation system 832, and visualization system 831 can be implemented using a reference system. Figure 1A The techniques described in Figure 6 at least partially collaborate to achieve the functionality of system 100. In some examples, real-time location information may be displayed on display system 710 for controlling medical device system 800. In some examples, navigation system 832 may utilize location information as feedback for locating medical device system 800.

[0151] Figure 9A and Figure 9B This is a simplified diagram based on some examples, including a side view of a medical device mounted on an insertion assembly in patient coordinate space. (e.g.) Figure 9A and Figure 9BAs shown, the surgical environment 900 may include a patient P located on a patient table T. Patient P may be stationary within the surgical environment 900 because overall patient movement is restricted by sedation, restraint, and / or other means. Periodic anatomical movements of patient P (including respiratory and cardiac movements) may continue. Within the surgical environment 900, a medical device 904 is used to perform medical procedures, which may include, for example, surgery, biopsy, ablation, illumination, irrigation, aspiration, or electroporation. The medical device 904 may also be used to perform other types of procedures, such as a registration process that associates position, orientation, and / or pose data captured by a sensor system 708 with a desired (e.g., anatomical or systemic) reference frame. The medical device 904 may be, for example, medical device 704. In some examples, the medical device 904 may include an elongation device 910 (e.g., a catheter) coupled to an instrument body 912. The elongation device 910 may be the elongation device 140 of FIG. 1. The elongation device 910 includes one or more channels sized and shaped to accommodate medical instruments.

[0152] The elongation device 910 may also include one or more sensors (e.g., components of sensor system 708). In some examples, a shape sensor 914 may be fixed at a proximal point 916 on the instrument body 912. The proximal point 916 of the shape sensor 914 may move with the instrument body 912, and the position of the proximal point 916 relative to a desired reference frame may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 914 may measure the shape from the proximal point 916 to another point (e.g., the distal end 918 of the elongation device 910). The shape sensor 914 may be aligned with the elongation device 910 (e.g., disposed within an internal channel or mounted externally). In some examples, the shape sensor 914 may use optical fibers to generate shape information of the elongation device 910.

[0153] In some examples, position sensors (e.g., EM sensors) may be incorporated into medical device 904. A series of position sensors may be positioned along flexible elongation device 910 and used for shape sensing. Position sensors may be used in place of or in conjunction with shape sensor 914, for example, to improve the accuracy of shape sensing or to verify shape information.

[0154] The extension device 910 may accommodate cables, linkages, or other steering control mechanisms that extend between the instrument body 912 and the distal end 918 to controllably bend the distal end 918. In some examples, at least four cables are used to provide independent up-and-down steering control to control the pitch of the distal end 918 and left-and-right steering control to control the yaw of the distal end 918. The instrument body 912 may include a drive input that is removably coupled to and receives power from a drive element (e.g., an actuator) to and from the manipulator assembly.

[0155] The instrument body 912 may be coupled to an instrument holder 906. The instrument holder 906 may be mounted to an insertion stage 908 fixed within the surgical environment 900. Alternatively, the insertion stage 908 may be movable but has a known position within the surgical environment 900 (e.g., via a tracking sensor or other tracking device). The instrument holder 906 may be a component of a manipulator assembly (e.g., manipulator assembly 702) coupled to the medical device 904 to control insertion motion (e.g., movement along insertion axis A) and / or movement of the distal end 918 of the extension device 910 in multiple directions, such as yaw, pitch, and / or roll. The instrument holder 906 or the insertion stage 908 may include actuators, such as servo motors, for controlling the movement of the instrument holder 906 along the insertion stage 908.

[0156] Sensor device 920 may be a component of sensor system 708. Sensor device 920 can provide information about the position of instrument body 912 as it moves relative to insertion stage 908 along insertion axis A. Sensor device 920 may include one or more rotary transformers, encoders, potentiometers, and / or other sensors that measure the rotation and / or orientation of actuators controlling the movement of instrument carriage 906, thereby indicating the movement of instrument body 912. In some examples, insertion stage 908 has, for example, Figure 9A and Figure 9B The linear track is shown. In some examples, the insertion stage 908 may have a curved track or a combination of curved track segments and linear track segments.

[0157] Figure 9A The device body 912 and device holder 906 are shown in the retracted position along the insertion stage 908. In this retracted position, the proximal point 916 is located at position L0 on the insertion axis A. The position of the proximal point 916 can be set to zero and / or other reference values ​​to provide a basic reference (e.g., corresponding to the origin of the desired reference system) to describe the position of the device holder 906 along the insertion stage 908. In the retracted position, the distal end 918 of the extension device 910 can be positioned precisely within the inlet orifice of the patient P. Also in the retracted position, data captured by the sensor device 920 can be set to zero and / or other reference values ​​(e.g., I=0). Figure 9BIn this configuration, the instrument body 912 and instrument holder 906 have advanced along the linear track of the insertion stage 908, and the distal end 918 of the extension device 910 has advanced into the patient P. In this advanced position, the proximal point 916 is located at position L1 on the insertion axis A. In some examples, rotation and / or orientation of the actuator measured by the sensor device 920 indicating the movement of the instrument holder 906 along the insertion stage 908 and / or one or more position sensors associated with the instrument holder 906 and / or the insertion stage 908 can be used to determine the position L1 of the proximal point 916 relative to position L0. In some examples, position L1 can also serve as an indicator of the distance or insertion depth of the distal end 918 of the extension device 910 into the channel of the patient P's anatomy.

[0158] Figure 10 A flowchart depicts an example method 1000 for visualizing a patient's anatomical structures during a medical procedure. Method 1000 can be implemented by a system (e.g., system 100) that may include one or more processors (e.g., disposed at processing unit 120) and a display device (e.g., display unit 130). Furthermore, instructions for executing method 1000 on one or more processors can be stored on a tangible, non-transitory computer-readable medium.

[0159] At box 1010, method 1000 includes: obtaining a first plurality of two-dimensional images depicting anatomical structures and one or more objects. Example systems may obtain images from imaging units such as, for example, fluorescence fluoroscopic imaging units. As discussed throughout this disclosure, the two-dimensional images may correspond to different projection angles. As discussed above, to better identify the projection angles associated with the projected images, example imaging setups may include a reference plate.

[0160] At box 1020, method 1000 includes: calculating a second plurality of two-dimensional images, at least partially based on a first plurality of two-dimensional images, by repairing at least a portion of pixels associated with at least one of one or more objects. Objects may include reference elements represented as reference markers in a set of projected images. Additionally or alternatively, objects may include instruments (e.g., flexible elongating devices) disposed within a patient's anatomy. When two-dimensional projected images are used to reconstruct a three-dimensional volumetric image, pixels associated with non-anatomical objects within the projection may cause artifacts. Therefore, as referenced above... Figure 2A to Figure 2H , Figure 4C and Figure 5BThe detailed repairs discussed can improve the quality of reconstructed images. In some examples, the system can repair reference markers to better identify pixels associated with the flexible elongation device, and subsequently repair pixels associated with at least a portion of the flexible elongation device. Reconstruction of the three-dimensional anatomy following repair can facilitate the identification of anatomical targets in medical procedures, thereby improving the speed and accuracy of the process.

[0161] At box 1030, method 1000 includes: causing a display device to display a graphical user interface depicting a visualization, the visualization being at least partially based on at least one of a second plurality of two-dimensional images. In some examples, the visualization may be used for assessing stages of a medical procedure and / or for navigational purposes. In other examples, the visualization may be part of an interactive display that prompts an operator for input, such as, for example, identifying anatomical targets. Additionally or alternatively, the visualization may be a portion of the GUI used to identify pixels within a projected image associated with a flexible elongating device, as shown, for example, with reference to FIG. 3B. Figure 3C and Figure 4C The subject of discussion.

[0162] At option box 1040, method 1000 includes: calculating a first three-dimensional image of at least a portion of an anatomical structure based at least partially on a second plurality of two-dimensional images. As discussed above, the three-dimensional image may be based on projection (e.g., Figure 5A Reconstruction of image 552 in the image.

[0163] At option 1050, method 1000 includes: calculating a second three-dimensional image based on a first three-dimensional image using a style transfer machine learning model, wherein visualization is at least partially based on the calculated second three-dimensional image. (See above for example, reference...) Figure 6A Figure 6C and the following references Figure 13A , Figure 13B and Figure 14A , Figure 14B The first reconstructed image discussed may have low fidelity due to, for example, a finite set of projection angles. A style transfer model that takes the first 3D image as input and computes a second 3D image as output can significantly enhance image quality (e.g., as shown below). Figure 14A , Figure 14B (As shown above). Figure 6B The training of the style transfer model is discussed below with reference to Figure 12.

[0164] Figure 11A , Figure 11BA flowchart depicts an example method 1100 for visualizing a patient's anatomical structures during a medical procedure. Method 1100 may be implemented by a system (e.g., system 100) that may include one or more processors (e.g., disposed at processing unit 120) and a display device (e.g., display unit 130). Furthermore, instructions for executing method 1100 on one or more processors may be stored on a tangible, non-transitory computer-readable medium.

[0165] At block 1110, method 1100 includes: obtaining a plurality of two-dimensional projection images corresponding to a plurality of respective projection angles and depicting anatomical structures and flexible elongation devices. At block 1120, method 1100 includes: identifying corresponding two-dimensional coordinates of a reference point disposed at the flexible elongation device within each of at least two of the plurality of two-dimensional projection images. As discussed above, identifying two-dimensional coordinates within the projection images may include: obtaining input from an operator via a GUI by one or more processors. Additionally or alternatively, image processing algorithms and / or ML models may assist the processing unit in autonomously identifying reference points or at least candidate reference points within the projection images. The operator and / or system may identify the reference point as the distal end (e.g., tip) of the flexible elongation device, and / or indicate the reference point by a reference object disposed at the flexible elongation device.

[0166] At box 1130, method 1100 includes: calculating the three-dimensional coordinates of a reference point in the imaging coordinate system, at least in part, based on corresponding two-dimensional coordinates and two corresponding projection angles. The system can calculate the three-dimensional coordinates by back-tracing rays from the reference point along the corresponding projection angles to find the intersection or nearest neighbor point of the back-tracing rays. The system can use more than two projected images with the coordinates of the identified projected reference point to calculate the coordinates of the reference point in three dimensions, making the calculation more robust to noise and / or errors in the projection of the identified reference point. For example, an incorrectly identified projected reference point may cause the back-tracing rays to stray far from the nearest neighbor point of other back-tracing rays.

[0167] At box 1140, method 1100 includes: identifying a curve (e.g., a centerline) corresponding to at least a portion of a reference point of the flexible elongating device within at least two of a plurality of two-dimensional projection images. The curve identification may include input from an operator via a GUI obtained by one or more processors. For example, as discussed above (e.g., reference point). Figure 3C The GUI can render a projected image depicting the projection of the flexible elongation device, where the user can select several points corresponding to the device. One or more processors can use appropriate image processing algorithms and user input to segment the curve corresponding to the device. In other examples, the system can autonomously identify the curve.

[0168] At block 1150, method 1100 includes: receiving shape data from a sensing unit in a sensing coordinate system for the flexible elongation device, including at least a portion of a reference point. The shape data may be generated by one or more sensors, such as a reference... Figure 1A The discussion focuses on this. At box 1160, method 1100 includes: registering a sensing coordinate system to an imaging coordinate system based at least in part on received shape data and curves identified within at least one of at least two of a plurality of two-dimensional projected images. (Reference) Figures 4A to 4C The registration process is discussed in detail. In some examples, the registration process may include: first calculating a curve corresponding to the flexible elongation device in three dimensions based on the projected image, and then aligning the calculated curve with the received shape data. In other examples, the shape data may be used as input to the registration process, such as a reference. Figure 4C and Figure 5A The subject of discussion.

[0169] At box 1170, method 1100 includes: reconstructing a three-dimensional image of an anatomical structure based at least in part on a plurality of two-dimensional projected images corresponding to a plurality of respective projection angles. The reconstruction may be, for example, based on tomographic synthesis of a set of fluorescence fluoroscopic images. As discussed throughout this disclosure, the reconstruction may include a remedial step or a remedial step performed prior to the reconstruction. Additionally or alternatively, the reconstruction may be enhanced by using a style transfer model.

[0170] At box 1180, method 1100 includes: identifying a target within a reconstructed 3D image of the anatomical structure. In some examples, target identification may include: receiving input from an operator via a GUI by one or more processors, as shown in the example above. Figure 5A The above is discussed. In other examples, the system may autonomously identify targets within the reconstructed image using appropriate image processing algorithms and / or ML models. Additionally or alternatively, the system may use prior information (e.g., from preoperative imaging) to identify targets within the reconstructed intraoperative image.

[0171] At block 1190, method 1100 includes: calculating the position of a target in an instrument coordinate system and at least in part based on identifying the target and registering the sensing coordinate system to the imaging coordinate system. The position of the target may be calculated based on user input identifying the center of the target within a reconstructed image, based on identifying the center of a region (e.g., a set of voxels) associated with the target, or by other suitable means. In some examples, calculating the target position may be based on applying a registration transformation to the target position in the reconstructed image to calculate the target position in instrument coordinates. Additionally or alternatively, calculating the target position may include: updating the target position according to a previously identified target position in the instrument coordinate system. Updating the target may involve integrating previous information, filtering the temporal evolution of the target position, and / or any other suitable signal processing techniques. At block 1195, method 1100 includes: displaying a graphical user interface on a display device depicting the calculated position of the target in the instrument coordinate system. Depicting the target in the instrument coordinate system can assist the operator in performing medical procedures. Therefore, the techniques of this disclosure for accurately and timely calculating the target position (which may move, for example, under anatomical deformation caused by pressure from the operating instrument, respiration and / or other physiological processes) can improve process speed and accuracy.

[0172] Figure 12A A flowchart depicts an example method 1200 for training machine learning and / or deep learning models for style transfer, as shown in, for example, reference... Figure 6A , Figure 6B The method can be implemented on one or more processors. At least some of the processors can be included in the processing unit 120. Alternatively or additionally, the one or more processors can be distributed (e.g., locally and / or in the cloud). At least one of the one or more processors can be a GPU.

[0173] At box 1210, method 1200 includes: calculating a set of two-dimensional projected images corresponding to the span of the projection angle, based on a three-dimensional real image. For example, the span of the projection angle may not exceed 120 degrees. In some examples, the method may include: calculating a set of two-dimensional images with a span greater than 120 degrees. Although fluorescence-based tomography typically uses spans less than 120 degrees, larger spans are possible. Furthermore, method 1200 can be applied to enhance images of other modalities. For example, method 1200 can enhance tomographic images obtained from angles with sparse intervals covering the full range of 180 degrees.

[0174] The three-dimensional real image can be, for example, a CT and / or CBCT image reconstructed from CT and CBCT data, respectively. Other possible real images may include MRI, 3D ultrasound, or any other suitable 3D modality or combination of modalities. Additionally or alternatively, the real image may be at least partially generated by a computer. For example, a set of two-dimensional projected images can be computed as a projection of a real 3D image using a suitable projection physics model.

[0175] The system for implementing method 1200 can compute multiple sets of two-dimensional projected images based on a single 3D real image. For example, one set of two-dimensional projected images can be computed for a set of projected angles spanning 120 degrees. Another set of two-dimensional projected images can be computed for a set of projected angles spanning 90 degrees. In any case, the angles can be distributed in any suitable manner at any suitable angular intervals (e.g., 10°, 5°, 3°, 2°, 1°, etc.). In some examples, the system can compute one set of projected images and use that one set of projected images to generate one or more additional sets of projected images. For example, the system can generate additional sets of projected images (or multiple sets of projected images) by adding noise and / or artifacts to the projected images. In other examples, additional sets of projected images can be generated from a first set of projected images by removing images associated with some of the projected images (e.g., on either side of the span of angles, every other angle, etc.). For example, the additional sets of projected images can have no more than half of the projection. That is, sparse sets of projected images can be generated from denser sets of projected images.

[0176] At box 1220, method 1200 includes: computing a three-dimensional tomographic synthesis training image based on a set of two-dimensional projected images. The system can generate the tomographic synthesis image using backprojection, filtered backprojection, iterative reconstruction, and / or any other suitable reconstruction method. More generally, the training image does not need to be a tomographic synthesis image. It can be an MRI image (e.g., an image degraded by adding noise or k-space errors relative to a real image) or any other suitable modal image used to generate a style transfer model.

[0177] Tomographic composite images can be generated from a set of projected images based on a corresponding set of angles. In some examples, the training system can add noise to at least some of the projected angles within the span of the projected angles. That is, a projected image can be computed using one set of angles, but reconstructed using a slightly different set of angles, which has systematic and / or random shifts relative to the correct set of angles. In this way, the training system can generate a model that is robust to angle uncertainty.

[0178] At box 1230, method 1200 includes: generating a 3D output image using a 3D tomographic training image as input to a style transfer machine learning model. The style transfer machine learning model may include a CNN, a recurrent neural network, a transformer network, a GAN, a diffusion model, or any other suitable model architecture and / or combination of models.

[0179] Style transfer models can generate at least one output for each of the input training images. In some examples, the model can be a combination of component models, and each of the component models can generate an individual output. The individual outputs can then be combined using, for example, a deterministic algorithm.

[0180] At box 1240, method 1200 includes: adjusting a style transfer machine learning model to reduce the loss indication between the 3D output image and the 3D real image. In some examples, the training system may use, for example, gradient descent methods to adjust the model parameters. Additionally or alternatively, the system may adjust the model architecture and / or other suitable hyperparameters. Method 1200 can be used to train a style transfer ML / DL model, which can be used as part of method 1000 or 1100 to enhance the visualization of patient anatomical structures, thereby aiding in the visualization of anatomical targets, as referenced below. Figure 12B As described.

[0181] Figure 12B A flowchart is depicted for an example method 1250 for visualizing patient anatomy using style transfer. Method 1250 can be implemented using system 100 or other suitable systems. In some examples, method 1250 can enhance image reconstruction after the repair and reconstruction techniques described throughout this disclosure. In other examples, method 1250 can enhance reconstructed images regardless of whether repair techniques are used.

[0182] At box 1260, method 1250 includes: obtaining a plurality of two-dimensional images associated with a first viewing range, the two-dimensional images depicting anatomical structures. For example, the two-dimensional images may be fluorescent perforated images obtained over a span of angles of 60°, 90°, 120° or any other suitable range and at any suitable regular or non-uniform angular intervals (e.g., 10°, 5°, 3°, 2°, 1°, etc.).

[0183] At box 1270, method 1250 includes: calculating (e.g., via tomographic synthesis) a first three-dimensional image of at least a portion of an anatomical structure based at least in part on a plurality of two-dimensional images. The calculation may include: repairing reference markers and / or other objects set within the anatomical structure, as discussed above.

[0184] At box 1280, method 1250 includes: computing a second three-dimensional image based on at least a portion of the first three-dimensional image and using one or more style transfer models. The style transfer model can be any of the style transfer models described above (e.g., refer to...). Figure 12A In addition, you can refer to the above. Figure 12A One or more models may be trained as described. For example, the source data of ground truth used in training one or more models may be CT and / or CBCT data, and may be associated with a viewpoint range greater than the first viewpoint range. One or more models may include models trained on inputs (e.g., training images) with different viewpoint ranges, different angular sparsity, different noise levels, etc.

[0185] At box 1290, method 1250 includes: displaying a GUI on a display device that depicts a second three-dimensional image. In some examples, method 1250 may include: displaying a GUI on a display device that depicts style transfer model options. The method may also include: obtaining user input regarding option selection. For example, the GUI may present selections and collect input for selection among models trained on input images with different angular ranges, different angular sparsity, etc. Additionally or alternatively, style transfer model options may include a target style for the second three-dimensional image. For a given input image, the target style may be CT, CBCT, MRI, etc. That is, the output image may resemble an image obtained using the corresponding modality. (See also: Regarding...) Figure 13A , Figure 13B and Figure 14A , Figure 14B The style transfer discussed here can improve the visualization of patient anatomy by improving the appearance of artifacts associated with tomographic synthesis of images with a limited angular range.

[0186] Figure 13A , Figure 13B The outputs of tomographic synthesis imaging and CBCT imaging were respectively depicted to illustrate the patient's anatomical structures. Figure 13A In the diagram, panel 1310 depicts an axial fault image passing through the reconstructed fault composite image, panel 1320 depicts a sagittal fault image, and panel 1330 depicts a coronal fault image. Figure 13BIn the diagram, subfigure 1340 depicts an axial tomographic image across the reconstructed CBCT image, subfigure 1350 depicts a sagittal tomographic image, and subfigure 1360 depicts a coronal tomographic image. The axial and sagittal tomographic images of the tomographic synthesis exhibit significant imaging artifacts (e.g., smearing artifacts) and have much lower fidelity than the axial and sagittal tomographic images of the CBCT image. The style transfer technique of this disclosure can at least partially remedy the deficiencies of tomographic synthesis, such as, for example... Figure 14A , Figure 14B As shown.

[0187] Figure 14A , Figure 14B The performance of style transfer models for transferring tomographic synthetic images to CT and CBCT images, respectively, is shown. See reference... Figure 6A , Figure 6B Train the style transfer model as discussed in Figure 12. Figure 14A , Figure 14B The example style transfer model based on the U-net architecture is shown to perform quite well (exhibiting fidelity comparable to real images) with moderate reconstruction noise. With high reconstruction noise, the U-net architecture model produces blurring and artifacts appearing as holes in the output image. On the other hand, for image reconstruction with significant reconstruction noise, the example model based on the GAN architecture demonstrates better performance than the U-net architecture example model.

[0188] One or more components of the examples discussed in this disclosure (e.g., control system 712) can be implemented in software to execute on one or more processors of a computer system. The software may include code that, when executed by one or more processors, configures the processors to perform the various functions discussed herein. The code may be stored in a non-transitory computer-readable storage medium (e.g., memory, magnetic storage device, optical storage device, solid-state storage device, etc.). The computer-readable storage medium may be part of a computer-readable storage device, such as electronic circuitry, a semiconductor device, a semiconductor memory device, a read-only memory (ROM), flash memory, an erasable programmable read-only memory (EPROM), a floppy disk, a CD-ROM, an optical disk, a hard disk, or other storage device. The code may be downloaded to the computer-readable storage medium for storage via a computer network such as the Internet, an intranet, etc. The code may be executed by any of a variety of centralized or distributed data processing architectures. The programming instructions of the code may be implemented as multiple separate programs or subroutines, or they may be integrated into multiple other aspects of the system described herein. Components of the computing system discussed herein may be connected using wired and / or wireless connections. In some examples, wireless connectivity can use wireless communication protocols such as Bluetooth, Near Field Communication (NFC), Infrared Data Association (IrDA), Home RF, IEEE 802.11, Digital Enhanced Cordless Telecommunications (DECT), and Wireless Medical Telemetry Service (WMTS).

[0189] Various general-purpose computer systems can be used to perform one or more of the processes, methods, or functions described herein. Additionally or alternatively, various special-purpose computer systems can be used to perform one or more of the processes, methods, or functions described herein. Furthermore, various programming languages ​​can be used to implement one or more of the processes, methods, or functions described herein.

[0190] While certain examples and instances have been described above and shown in the accompanying drawings, it should be understood that these examples and instances are merely illustrative and not limited to the specific constructions and arrangements shown and described, as those skilled in the art will recognize a variety of other alternatives, modifications and equivalents.

Claims

1. A tangible, non-transitory, computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: Based on three-dimensional real images, calculate the set of two-dimensional projected images corresponding to the span of projection angles less than 120 degrees; Based on the two-dimensional projection image set, calculate the three-dimensional tomographic synthesis training image; The three-dimensional tomographic training images are used as input to the style transfer machine learning model to generate three-dimensional output images; as well as Adjust the style transfer machine learning model to reduce the loss indication between the 3D output image and the 3D real image.

2. The tangible non-transitory computer-readable medium according to claim 1, wherein, The three-dimensional real image is a computed tomography (CT) or cone-beam computed tomography (CBCT) image.

3. The tangible non-transitory computer-readable medium according to claim 1 or 2, wherein, The style transfer machine learning model includes a convolutional neural network.

4. The tangible non-transitory computer-readable medium according to any one of claims 1 to 3, wherein, The style transfer machine learning model includes a transformer network.

5. The tangible non-transitory computer-readable medium according to any one of claims 1 to 4, wherein, The style transfer machine learning model includes generative adversarial networks (GANs).

6. The tangible non-transitory computer-readable medium according to any one of claims 1 to 5, wherein, The style transfer machine learning model includes a diffusion model.

7. The tangible non-transitory computer-readable medium according to any one of claims 1 to 6, wherein, Calculating the two-dimensional projected image set based on three-dimensional real images includes adding noise to at least some of the images in the set.

8. The tangible non-transitory computer-readable medium according to any one of claims 1 to 7, wherein, Calculating the three-dimensional tomographic training image based on the two-dimensional projection image set includes adding noise to at least some projection angles within the span of the projection angles.

9. The tangible non-transitory computer-readable medium according to any one of claims 1 to 8, wherein, The instructions also cause the one or more processors to: Additional three-dimensional tomographic training images are calculated based on a subset comprising no more than half of the set of two-dimensional projected images. The additional 3D tomographic training images are used as input to the style transfer machine learning model to generate additional 3D output images. as well as The style transfer machine learning model is adjusted to reduce the loss indication between the additional 3D output image and the 3D real image.

10. The tangible non-transitory computer-readable medium according to any one of claims 1 to 8, wherein, The instructions also cause the one or more processors to: Additional three-dimensional tomographic training images are calculated based on a subset comprising no more than half of the set of two-dimensional projected images. The additional 3D tomographic training images are used as input to an alternative style transfer machine learning model to generate additional 3D output images. as well as The alternative style transfer machine learning model is adjusted to reduce the loss indication between the additional 3D output image and the 3D real image.

11. A system for training a style transfer machine learning model for visualizing patient anatomy during a medical procedure, the system comprising: One or more processors; as well as A non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to: Based on three-dimensional real images, calculate the set of two-dimensional projected images corresponding to the span of projection angles less than 120 degrees; Based on the two-dimensional projection image set, calculate the three-dimensional tomographic synthesis training image; The three-dimensional tomographic training image is used as input to the style transfer machine learning model to generate a three-dimensional output image; as well as Adjust the style transfer machine learning model to reduce the loss indication between the 3D output image and the 3D real image.

12. The system according to claim 11, wherein, The three-dimensional real image is a computed tomography (CT) or cone-beam computed tomography (CBCT) image.

13. The system according to claim 11 or 12, wherein, The style transfer machine learning model includes a convolutional neural network.

14. The system according to any one of claims 11 to 13, wherein, The style transfer machine learning model includes a transformer network.

15. The system according to any one of claims 11 to 14, wherein, The style transfer machine learning model includes generative adversarial networks (GANs).

16. The system according to any one of claims 11 to 15, wherein, The style transfer machine learning model includes a diffusion model.

17. The system according to any one of claims 11 to 16, wherein, Calculating the two-dimensional projected image set based on three-dimensional real images includes adding noise to at least some of the images in the set.

18. The system according to any one of claims 11 to 17, wherein, Calculating the three-dimensional tomographic training image based on the two-dimensional projection image set includes adding noise to at least some projection angles within the span of the projection angles.

19. The system according to any one of claims 11 to 18, wherein, The instructions also cause the one or more processors to: Additional three-dimensional tomographic training images are calculated based on a subset comprising no more than half of the set of two-dimensional projected images. The additional 3D tomographic training images are used as input to the style transfer machine learning model to generate additional 3D output images. as well as The style transfer machine learning model is adjusted to reduce the loss indication between the additional 3D output image and the 3D real image.

20. The system according to any one of claims 11 to 18, wherein, The instructions also cause the one or more processors to: Additional three-dimensional tomographic training images are calculated based on a subset comprising no more than half of the set of two-dimensional projected images. The additional 3D tomographic training images are used as input to an alternative style transfer machine learning model to generate additional 3D output images. as well as The alternative style transfer machine learning model is adjusted to reduce the loss indication between the additional 3D output image and the 3D real image.

21. A method for training a style transfer machine learning model for visualizing patient anatomy during a medical procedure, the method comprising: Calculate a set of two-dimensional projected images corresponding to a span of projection angles less than 120 degrees, using one or more processors and based on three-dimensional real images; The three-dimensional tomographic training image is calculated by the one or more processors and based on the two-dimensional projected image set; The three-dimensional output image is generated by one or more processors using the three-dimensional tomographic training image as input to the style transfer machine learning model; as well as The style transfer machine learning model is adjusted by one or more processors to reduce the loss indication between the 3D output image and the 3D real image.

22. The method according to claim 21, wherein, The three-dimensional real image is a computed tomography (CT) or cone-beam computed tomography (CBCT) image.

23. The method according to claim 21 or 22, wherein, The style transfer machine learning model includes a convolutional neural network.

24. The method according to any one of claims 21 to 23, wherein, The style transfer machine learning model includes a transformer network.

25. The method according to any one of claims 21 to 24, wherein, The style transfer machine learning model includes generative adversarial networks (GANs).

26. The method according to any one of claims 21 to 25, wherein, The style transfer machine learning model includes a diffusion model.

27. The method according to any one of claims 21 to 26, wherein, Calculating the two-dimensional projected image set based on three-dimensional real images includes adding noise to at least some of the images in the set.

28. The method according to any one of claims 21 to 27, wherein, Calculating the three-dimensional tomographic training image based on the two-dimensional projection image set includes adding noise to at least some projection angles within the span of the projection angles.

29. The method according to any one of claims 21 to 28, further comprising: Additional three-dimensional tomographic training images are calculated based on a subset comprising no more than half of the set of two-dimensional projected images. The additional 3D tomographic training images are used as input to the style transfer machine learning model to generate additional 3D output images. as well as The style transfer machine learning model is adjusted to reduce the loss indication between the additional 3D output image and the 3D real image.

30. The method according to any one of claims 21 to 28, further comprising: Additional three-dimensional tomographic training images are calculated based on a subset comprising no more than half of the set of two-dimensional projected images. The additional 3D tomographic training images are used as input to an alternative style transfer machine learning model to generate additional 3D output images. as well as The alternative style transfer machine learning model is adjusted to reduce the loss indication between the additional 3D output image and the 3D real image.

31. A tangible, non-transitory, computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: Obtain multiple two-dimensional images associated with a first-viewpoint range, the two-dimensional images depicting anatomical structures; A first three-dimensional image of at least a portion of the anatomical structure is calculated, based at least in part on the plurality of two-dimensional images; A second three-dimensional image is calculated based on at least a portion of the first three-dimensional image and using one or more style transfer models; as well as The display device displays a graphical user interface that depicts the second three-dimensional image.

32. The tangible non-transitory computer-readable medium according to claim 31, wherein, The one or more style transfer models are trained using training image source data associated with a second viewpoint range, wherein the second viewpoint range is larger than the first viewpoint range.

33. The tangible non-transitory computer-readable medium according to claim 32, wherein, The one or more style transfer models are trained at least in part by the following methods: Based on the training image source data, generate a three-dimensional realistic image; Based on the training image source data, calculate a two-dimensional projection image set corresponding to a span of projection angle not exceeding 120 degrees; Based on the two-dimensional projection image set, calculate the three-dimensional tomographic synthesis training image; The three-dimensional tomographic training image is used as input to the style transfer model to generate a three-dimensional output image; as well as Adjusting the style transfer model to reduce the loss indication between the 3D output image and the 3D real image.

34. The tangible non-transitory computer-readable medium according to claim 32 or 33, wherein, The training image source data includes at least one of computed tomography (CT) and / or cone-beam computed tomography (CBCT) data.

35. The tangible non-transitory computer-readable medium according to any one of claims 31 to 34, wherein, The multiple two-dimensional images are fluorescent perspective images.

36. The tangible non-transitory computer-readable medium according to any one of claims 31 to 35, wherein, The one or more style transfer models include at least one of the following: (i) a convolutional neural network; (ii) a transformer network; (iii) a generative adversarial network (GAN); and / or (iv) a diffusion model.

37. The tangible non-transitory computer-readable medium according to any one of claims 31 to 36, wherein: The one or more style transfer models include at least two models trained with different viewpoint ranges; as well as Using one or more style transfer models includes selecting from at least two models.

38. The tangible non-transitory computer-readable medium according to any one of claims 31 to 37, wherein, The instructions, when executed by the one or more processors, also cause the one or more processors to: The display device displays a graphical user interface, which depicts style transfer model options, and User input is obtained from the graphical user interface, which depicts style transfer model options, and The calculation of the second three-dimensional image is based at least in part on the user input obtained from the graphical user interface, which depicts style transfer model options.

39. The tangible non-transitory computer-readable medium according to claim 38, wherein, The style transfer model options include at least one of the following: (i) view range; and / or (ii) view sparsity.

40. The tangible non-transitory computer-readable medium according to claim 38 or 39, wherein, The style transfer model options include a target style for the second 3D image.

41. A system for visualizing a patient's anatomical structures, the system comprising: Display device; One or more processors; as well as A non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to: Obtain multiple two-dimensional images associated with a first-viewpoint range, the two-dimensional images depicting anatomical structures; A first three-dimensional image of at least a portion of the anatomical structure is calculated, based at least in part on the plurality of two-dimensional images; A second three-dimensional image is calculated based on at least a portion of the first three-dimensional image and using one or more style transfer models; as well as The display device displays a graphical user interface that depicts the second three-dimensional image.

42. The system according to claim 41, wherein, The one or more style transfer models are trained using training image source data associated with a second viewpoint range, wherein the second viewpoint range is larger than the first viewpoint range.

43. The system according to claim 42, wherein, The one or more style transfer models are trained at least in part by the following methods: Based on the training image source data, generate a three-dimensional realistic image; Based on the training image source data, calculate a two-dimensional projection image set corresponding to a span of projection angle not exceeding 120 degrees; Based on the two-dimensional projection image set, calculate the three-dimensional tomographic synthesis training image; The three-dimensional tomographic training image is used as input to the style transfer model to generate a three-dimensional output image; as well as Adjusting the style transfer model to reduce the loss indication between the 3D output image and the 3D real image.

44. The system according to claim 42 or 43, wherein, The training image source data includes at least one of computed tomography (CT) and / or cone-beam computed tomography (CBCT) data.

45. The system according to any one of claims 41 to 44, wherein, The multiple two-dimensional images are fluorescent perspective images.

46. ​​The system according to any one of claims 41 to 45, wherein, The one or more style transfer models include at least one of the following: (i) a convolutional neural network; (ii) a transformer network; (iii) a generative adversarial network (GAN); and / or (iv) a diffusion model.

47. The system according to any one of claims 41 to 46, wherein: The one or more style transfer models include at least two models trained with different viewpoint ranges; as well as Using one or more style transfer models includes selecting from at least two models.

48. The system according to any one of claims 41 to 47, wherein: The instructions, when executed by the one or more processors, also cause the one or more processors to: The display device displays a graphical user interface, which depicts style transfer model options; and User input is obtained from the graphical user interface, which depicts style transfer model options; and The calculation of the second three-dimensional image is based at least in part on the user input obtained from the graphical user interface, which depicts style transfer model options.

49. The system according to claim 48, wherein, The style transfer model options include at least one of the following: (i) view range; and / or (ii) view sparsity.

50. The system according to claim 48 or 49, wherein, The style transfer model options include a target style for the second 3D image.

51. A method for visualizing a patient's anatomical structures, the method comprising: Multiple two-dimensional images associated with a first viewpoint range are obtained by one or more processors, the two-dimensional images depicting anatomical structures; A first three-dimensional image of at least a portion of the anatomical structure is calculated by the one or more processors and at least in part based on the plurality of two-dimensional images; The one or more processors compute a second three-dimensional image based on at least a portion of the first three-dimensional image and using one or more style transfer models; as well as The one or more processors cause the display device to display a graphical user interface, which depicts the second three-dimensional image.

52. The method according to claim 51, wherein, The one or more style transfer models are trained using training image source data associated with a second viewpoint range, wherein the second viewpoint range is larger than the first viewpoint range.

53. The method according to claim 52, wherein, The one or more style transfer models are trained at least in part by the following methods: Based on the training image source data, generate a three-dimensional realistic image; Based on the training image source data, calculate a two-dimensional projection image set corresponding to a span of projection angle not exceeding 52 degrees; Based on the two-dimensional projection image set, calculate the three-dimensional tomographic synthesis training image; The three-dimensional tomographic training image is used as input to the style transfer model to generate a three-dimensional output image; as well as Adjusting the style transfer model to reduce the loss indication between the 3D output image and the 3D real image.

54. The method according to claim 52 or 53, wherein, The training image source data includes at least one of computed tomography (CT) and / or cone-beam computed tomography (CBCT) data.

55. The method according to any one of claims 51 to 54, wherein, The multiple two-dimensional images are fluorescent perspective images.

56. The method according to any one of claims 51 to 55, wherein, The one or more style transfer models include at least one of the following: (i) a convolutional neural network; (ii) a transformer network; (iii) a generative adversarial network (GAN); and / or (iv) a diffusion model.

57. The method according to any one of claims 51 to 56, wherein: The one or more style transfer models include at least two models trained with different viewpoint ranges; as well as Using one or more style transfer models includes selecting from at least two models.

58. The method according to any one of claims 51 to 57, further comprising: The display device is made to display a graphical user interface by one or more processors, the graphical user interface depicting style transfer model options; as well as User input is obtained from the graphical user interface by one or more processors, the graphical user interface depicting style transfer model options; The calculation of the second three-dimensional image is based at least in part on the user input obtained from the graphical user interface, which depicts style transfer model options.

59. The method according to claim 58, wherein, The style transfer model options include at least one of the following: (i) view range; and / or (ii) view sparsity.

60. The method according to claim 58 or 59, wherein, The style transfer model options include a target style for the second 3D image.

Images