Method and system for using multi-view pose estimation

Abstract

The method includes receiving a series of medical images showing a region of interest including a plurality of targets, captured by the medical imaging device while the medical imaging device moves by rotation; determining a pose of each of a subset of the series of medical images in which the targets are visible; estimating a trajectory of the medical imaging device based on the determined poses of the subset and a trajectory constraint of the imaging device; determining a pose of one of the medical images in which the targets are not visible by extrapolation based on the assumption that the movement of the medical imaging device continues; and determining a stereoscopic reconstruction of the region of interest based on at least some of the poses of the subset and the pose of one of the medical images in which the targets are not visible.

Description

Technical Field

[0001] This application is a International (PCT) patent application related to and claiming priority from U.S. Provisional Patent Application No. 62 / 965,628, filed on January 24, 2020, by the same owner and entitled "METHODS AND SYSTEMS FOR USING MULTI VIEW POSE ESTIMATION", the entire content of which is incorporated herein by reference.

[0002] Embodiments of the present invention relate to invasive devices and methods of using the same.

Background Art

[0003] The use of minimally invasive procedures, such as endoscopic procedures, video-assisted thoracic surgery, or similar medical procedures, can be used as a diagnostic tool for suspected lesions or as a treatment means for cancerous tumors.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] In some embodiments, the present invention: Obtaining the first image from the first imaging modality, Extracting at least one element from a first image from a first imaging modality, Extraction includes, at least one element, an airway, blood vessel, body cavity, or any combination thereof. From a second imaging modality, at least (i) a first image of the radiopaque device in a first orientation and (ii) a second image of the radiopaque device in a second orientation, The radiopaque device is located inside the patient's body cavity, and it is necessary to obtain it. To generate at least two expanded bronchography images, The first expanded bronchography image corresponds to the first image from the radiopaque device in the first position. The second expanded bronchography image corresponds to the second image from the radiopaque device in the second position, and is generated accordingly. During the following period, that is, (i) The first position of the radiopaque equipment, (ii) Determining the mutual geometric constraints between the second orientation of the radiopaque device and, The first and second orientations of a radiopaque device are estimated by comparing the first and second orientations of the radiopaque device with the first image of the first imaging modality, To compare, (i) First dilated bronchography, (ii) A second dilated bronchography, and (iii) performed using at least one element, The estimated first orientation of the radiopaque device and the estimated second orientation of the radiopaque device satisfy the determined mutual geometric constraints, and The process includes generating a third image, the third image being an extended image derived from a second imaging modality, which highlights the target region, The method provides a method in which the target region is determined from data from a first imaging modality.

[0007] In some embodiments, at least one element from the first image from the first imaging modality further includes ribs, spine, diaphragm, or any combination thereof. In some embodiments, the relative geometric constraints are generated by: a. Estimating the difference between (i) the first posture and (ii) the second posture by comparing the first image from the radiopaque device with the second image from the radiopaque device, The estimation is performed using a device including an angle meter, accelerometer, gyroscope, or any combination thereof, and the device is attached to a second imaging modality. b. Extracting multiple image features in order to estimate relative posture changes, Multiple image features include anatomical elements, non-anatomical elements, or any combination thereof. The image features include a patch attached to the patient, a radiopaque marker disposed within the field of view of a second imaging modality, or any combination thereof, The image features are to be extracted as visible on a first image of the radiopaque device and a second image of the radiopaque device, c. (i) determining the difference between a first pose and (ii) a second pose by using at least one camera, where the camera includes a video camera, an infrared camera, a depth camera, or any combination thereof, The camera is in a fixed position, The camera is configured to track at least one feature, using at least one feature including a marker attached to the patient, a marker attached to the second imaging modality, or any combination thereof, tracking at least one feature, and thereby estimating, d. or any combination thereof.

[0008] In some embodiments, the method further includes tracking a radiopaque device to identify a trajectory and using the trajectory as an additional geometric constraint, where the radiopaque device includes an endoscope, an endobronchial device, or a robotic arm.

[0009] In some embodiments, the present invention is a method including: generating a map of at least one body cavity of a patient, where the map is generated using a first image from a first imaging modality, generating, acquiring an image of a radiopaque device having at least two attached markers from a second imaging modality, where the at least two attached markers are separated by a known distance, acquiring, Identifying the orientation of a radiopaque device from a second imaging modality to a map of at least one body cavity of the patient, Identifying the first position of a first marker attached to a radiopaque device on a second image from a second imaging modality, Identifying the second position of a second marker attached to a radiopaque device on a second image from a second imaging modality, Measure the distance between the first position of the first marker and the second position of the second marker, Projecting the known distance between the first marker and the second marker, To pinpoint the exact location of a radiopaque device within at least one body cavity of a patient, compare the measured distance with the projected known distance between a first marker and a second marker.

[0010] In some embodiments, radiopaque devices include endoscopes, endobronchial instruments, or robotic arms.

[0011] In some embodiments, the method further includes determining the depth of the radiopaque instrument using the trajectory of the radiopaque instrument.

[0012] In some embodiments, the first image from the first imaging modality is a preoperative image. In some embodiments, at least one image from a radiopaque device from the second imaging modality is an intraoperative image.

[0013] In some embodiments, the present invention includes the following methods: Obtaining the first image from the first imaging modality, Extracting at least one element from a first image from a first imaging modality, Extraction includes, at least one element, an airway, blood vessel, body cavity, or any combination thereof. From a second imaging modality, in two different orientations of the second imaging modality, to acquire at least (i) one image of the radiopaque device and (ii) another image of the radiopaque device, The first image from the radiopaque device is taken in the first orientation of the second imaging modality. The second image from the radiopaque device is taken in the second orientation of the second imaging modality. The radiopaque device is located inside the patient's body cavity, and it is necessary to obtain it. To generate at least two augmented bronchography images corresponding to each of two orientations of an imaging device, wherein the first augmented bronchography image is derived from a first image of a radiopaque device, and the second augmented bronchography image is derived from a second image of a radiopaque device. During the following period, that is, (i) The first stance of the second imaging modality, (ii) Reconstructing using the mutual geometric constraints between the second posture of the second imaging modality and, Estimating two poses of the second imaging modality relative to the first image of the first imaging modality, using at least one element extracted from the corresponding extended bronchography image and the first image of the first imaging modality, The two estimated postures satisfy each other's geometric constraints, and the estimation is as follows: The method involves generating a third image, which is an augmented image with the target region highlighted, derived from a second imaging modality based on data sourced from a first imaging modality.

[0014] In some embodiments, anatomical elements such as ribs, spine, diaphragm, or any combination thereof are extracted from the first imaging modality and the second imaging modality.

[0015] In some examples, the mutual geometric constraints are generated by the following: a. Estimating the difference between (i) the first posture and (ii) the second posture by comparing the first image from the radiopaque device with the second image from the radiopaque device, The estimation is performed using a device including an angle meter, accelerometer, gyroscope, or any combination thereof, and the device is attached to a second imaging modality. b. Extracting multiple image features in order to estimate relative posture changes, Multiple image features include anatomical elements, non-anatomical elements, or any combination thereof. Image features include a patch attached to the patient, a radiopaque marker placed within the field of view of the second imaging modality, or any combination thereof. Image features are those visible in the first image and the second image of the radiopaque device, and are to be extracted. c. The difference between (i) the first posture and (ii) the second posture is determined by using at least one camera, The cameras include video cameras, infrared cameras, depth cameras, or any combination thereof. The camera is in a fixed position. The camera is configured to track at least one feature, At least one feature includes a marker attached to the patient, a marker attached to a second imaging modality, or any combination thereof, and is used. By tracking at least one feature, and by estimating, d. Or any combination of these.

[0016] In some embodiments, the method further includes tracking a radiopaque instrument to identify its trajectory and using such trajectory as an additional geometric constraint, the radiopaque instrument including an endoscope, an intrabronchial instrument, or a robotic arm.

[0017] In some embodiments, the present invention is a method for determining the true location of a device within a patient's body, including: Using a map of at least one body cavity of the patient generated from the first image of the first imaging modality, The method involves acquiring an image of a radiopaque device from a second imaging modality, on which at least two markers are attached at a predetermined distance apart. The markers can be recognized from images located within at least two different body cavities in the patient's body, and can be acquired from those images. To obtain the attitude of the second imaging modality for the map, Identifying the first position of a first marker attached to a radiopaque device on a second image from a second imaging modality, Identifying the second position of a second marker attached to a radiopaque device on a second image from a second imaging modality, Measure the distance between the first position of the first marker and the second position of the second marker, Using the orientation of the second imaging modality, the known distance between markers is projected onto each of the recognized locations of the radiopaque device, To determine the true position of the device within the body, compare the measured distance with the projected distance between the two markers.

[0018] In some embodiments, radiopaque devices include endoscopes, endobronchial instruments, or robotic arms.

[0019] In some embodiments, the method further includes determining the depth of the radiopaque instrument using the trajectory of the radiopaque instrument.

[0020] In some embodiments, the first image from the first imaging modality is a preoperative image. In some embodiments, at least one image from a radiopaque device from the second imaging modality is an intraoperative image.

[0021] In some embodiments, the method includes receiving a series of medical images taken by a medical imaging device while the medical imaging device is being rotated by a rotation, the series of medical images showing a region of interest including multiple targets; determining the orientation of each of the subsets of medical images in which the multiple targets are visible; estimating the trajectory of the movement of the medical imaging device based on the determined orientations of the subsets of medical images and the trajectory constraints of the imaging device; determining the orientation of at least one of the medical images in which the multiple targets are not visible by extrapolation based on the assumption that the movement of the medical imaging device continues; and determining a volumetric reconstruction of the region of interest based on at least (a) at least some of the orientations of the subsets of medical images in which the multiple targets are visible, and (b) at least one of the orientations of at least one of the medical images in which the multiple targets are not visible.

[0022] In some embodiments, the orientation of each subset of a series of medical images is determined based on a 2D-3D correspondence between the 3D locations of multiple targets and the 2D locations of multiple targets observed in the subset of medical images. In some embodiments, the 3D locations of multiple targets are determined based on at least one preoperative image. In some embodiments, the 3D locations of multiple targets are determined by applying structure-from-motion technology.

[0023] In some embodiments, the method includes receiving a plurality of medical images while the medical imaging device is rotated by the movement of the C-arm having a constrained trajectory, wherein at least some of the plurality of medical images include a region of interest; determining the orientation of each subset of the plurality of medical images; calculating the positions of a plurality of 3D targets based on the 2D positions of the 3D targets in the subset of the plurality of medical images and based on the determined orientation of each subset of the plurality of medical images; determining the orientation of one further medical image in which at least some of the 3D targets are visible by determining the location and orientation of the imaging device based on known 3D-2D correspondences of at least the 3D targets; and calculating a three-dimensional reconstruction of the region of interest based on the orientation of at least one further medical image and one further medical image.

[0024] In some embodiments, the orientation of each subset of multiple medical images is determined based on the pattern of radiopaque markers visible in at least one subset of medical images. In some embodiments, the orientation is further determined based on a constrained trajectory.

[0025] In some embodiments, the method includes receiving a series of medical images taken by a medical imaging device while the medical imaging device is rotated by rotation, the series of medical images showing a region of interest including a target object having a 3D shape; calculating the orientation of at least some of the medical images based on the 3D-2D correspondence of the 2D projection of the target object in at least some of the medical images; and calculating a three-dimensional reconstruction of the region of interest based on at least some of the medical images and the calculated orientations of at least some of the medical images.

[0026] In some embodiments, the target is an anatomical target. In some embodiments, the 3D shape of the anatomical target is determined based on at least one preoperative image.

[0027] In some embodiments, the 3D shape of the target object is determined based on applying structure-from-motion technology to at least some of a series of medical images. In some embodiments, structure-from-motion technology is applied to all of a series of medical images.

[0028] In some embodiments, posture is calculated for all images in a series.

[0029] In some embodiments, the series of images does not show multiple radiopaque markers.

[0030] In some embodiments, calculating the pose of at least some of the medical images is further based on a known trajectory of rotation.

[0031] In some embodiments, the 3D shape of the target object is determined based on at least one preoperative image, and further based on applying structure-from-motion technology to at least some of a series of medical images.

[0032] In some embodiments, the target object is a device placed inside the patient's body in the target area.

[0033] In some embodiments, the target object is an object positioned outside the patient's body, but in close proximity to it. In some embodiments, the object is fixed to the patient's body.

[0034] The present invention will be further described with reference to the accompanying drawings, in which similar structures are referred to by similar numbers across several of the figures. The drawings shown are not necessarily drawn to exact scale, and instead the overall emphasis is on illustrating the principles of the present invention. Furthermore, some features may be exaggerated to illustrate the details of certain components. [Brief explanation of the drawing]

[0035] [Figure 1] This is a block diagram of a multi-view pose estimation method used in some embodiments of the method of the present invention. [Figure 2] This figure shows an exemplary embodiment of an intraoperative image used in the method of the present invention, and shows a fluoroscopic image acquired from one specific posture. The bronchoscope 240, instrument 210, ribs 220, and body boundary 230 are visible. The multi-view posture estimation method uses the elements visible in Figure 2 as input. [Figure 3] This figure shows an exemplary embodiment of an intraoperative image used in the method of the present invention, and shows a fluoroscopic image acquired from one specific posture. The bronchoscope 340, instrument 310, ribs 320, and body boundary 330 are visible. The multi-view posture estimation method uses the elements visible in Figure 3 as input. [Figure 4] This figure shows an exemplary embodiment of intraoperative images used in the method of the present invention, and shows fluoroscopic images acquired in a different posture compared to Figures 2 and 3 as a result of C-arm rotation. The bronchoscope 440, instrument 410, ribs 420, and body boundary 430 are visible. The multi-view posture estimation method uses the elements visible in Figure 4 as input. [Figure 5] This is a schematic diagram of the structure of the bronchial airway used in the method of the present invention. The airway centerline is represented by 530. The catheter is inserted into the airway structure and imaged by a fluoroscope device having an image plane 540. The projection of the catheter on the image is shown by a curve 550, to which radiopaque markers attached are projected to points G and F. [Figure 6] This is an image of the tip of a bronchoscope device attached to a bronchoscope, which can be used in embodiments of the method of the present invention. [Figure 7] This figure shows an embodiment of the method of the present invention, and is a fluoroscopic image of a tracked scope (701) used in a bronchoscopy procedure together with a surgical instrument (702) extending therefrom. The surgical instrument may include a radiopaque marker or a specific pattern attached thereto. [Figure 8] The figures show two views of epipolar geometry according to an embodiment of the method of the present invention, and the figures are of a pair of fluoroscopy images including a scope (801) used in a bronchoscopy procedure together with a surgical instrument (802) extending therefrom. The surgical instrument may include a radiopaque marker or a unique pattern attached thereto (points P1 and P2 represent parts of such a pattern). Point P1 has the corresponding epipolar line L1. Point P0 represents the tip of the scope, and point P3 represents the tip of the surgical instrument. O1 and O2 represent the focal points of the corresponding views. [Figure 9] This figure shows an exemplary method for 6-degree-of-freedom pose estimation using a 3D-2D correspondence. [Figure 10A] This figure shows the orientation of the X-ray imaging device mounted on the C-arm. [Figure 10B] This figure shows the orientation of the X-ray imaging device mounted on the C-arm. [Figure 11] This figure shows the use of a 3D target to estimate the trajectory of the C-arm. [Figure 12] This figure shows how the algorithm estimates the pose for each image frame using a known set of visible radiopaque markers. [Figure 13] This figure shows a method for estimating a 3D target using the structure-from-motion technique without using radiopaque markers. [Figure 14] This diagram shows the same feature points of an object visible in multiple frames. [Figure 15] This diagram shows the same feature points of an object visible in multiple frames. [Figure 16] This diagram shows a method for optimizing the determination of the position of feature points of an object visible across multiple frames. [Figure 17] This figure shows the process for determining 3D image reconstruction based on a series of received 2D images. [Figure 18] This diagram shows the process for training an image-to-image transformation using unaligned images. [Figure 19] This figure shows the training of a model for the transformation from Domain C to Domain B. [Figure 20] This diagram shows exemplary guidance for the user to position the fluoroscope. [Figure 21] This diagram shows exemplary guidance for the user to position the fluoroscope. [Modes for carrying out the invention]

[0036] These figures constitute part of this specification and include exemplary embodiments of the invention, illustrating various purposes and features. Furthermore, these figures are not necessarily to exact scale, and some features may be exaggerated to illustrate the details of certain components. Moreover, all measurements, specifications, etc., shown in the figures are intended to be illustrative and not restrictive. Accordingly, the specific details relating to the structure and function disclosed herein are not limiting and should be interpreted as merely representative grounds to teach those skilled in the art how to utilize the invention in various ways.

[0037] Among the benefits and improvements disclosed herein, other objectives and advantages of the present invention will become apparent from the following description, to be interpreted in conjunction with the accompanying figures. While detailed embodiments of the present invention are disclosed herein, it should be understood that the disclosed embodiments are merely illustrative examples of the various forms in which the invention may be embodied. Furthermore, each of the examples given in connection with the various embodiments of the present invention is intended to be illustrative and not restrictive.

[0038] Throughout the specification and claims, the following terms have the meanings expressly associated herein unless the context clearly indicates otherwise. The phrases “in one embodiment” and “in some embodiments” may, when used herein, refer to the same embodiment, though not necessarily. Furthermore, the phrases “in another embodiment” and “in some other embodiments” may, when used herein, refer to different embodiments, though not necessarily. In this case, various embodiments of the present invention can be readily combined without departing from the scope or spirit of the invention, as described below.

[0039] Furthermore, as used herein, the term “or” functions as an inclusive “or” and is equivalent to the term “and / or” unless explicitly stated otherwise by the context. The term “based on” is not exclusive and may be based on additional factors not listed unless explicitly stated otherwise by the context. Furthermore, throughout this specification, the meanings of “a,” “an,” and “the” include multiple references. The meaning of “in” includes “in” and “on.”

[0040] As used herein, “multiple” means a number greater than one, for example, but not limited to, two, three, four, five, six, seven, eight, nine, ten, etc. For example, multiple images could be two images, three images, four images, five images, six images, seven images, eight images, nine images, ten images, etc.

[0041] As used herein, “anatomical element” refers to a target object, which may be, for example, a region of interest, an incision point, a bifurcation, a blood vessel, a bronchial airway, a rib, or an organ.

[0042] As used herein, “geometrical constraint” or “geometric constraint” or “mutual constraint” refers to the geometric relationships between bodily organs (e.g., at least two bodily organs) within a subject’s body, where these constraints establish similar geometric relationships between ribs, body boundaries, etc., within the subject’s body. Such geometric relationships remain unchanged or their relative movement can be ignored or quantified when observed through different imaging modalities.

[0043] As used herein, “posture” refers to a set of six parameters that determine the relative location and orientation of an intraoperative imaging device source as an alternative to an optical camera device. In a non-limiting example, posture can be obtained as a combination of relative movement between the device, the patient bed, and the patient. Another non-limiting example of such movement is the rotation of the intraoperative imaging device combined with its movement around a stationary patient bed in which the patient is stationary.

[0044] As used herein, “location” means the position of any object (measurable in any coordinate system, e.g., a Cartesian coordinate system of x, y, and z) in 3D space, including the imaging device itself.

[0045] As used herein, “orientation” refers to the angle of the intraoperative imaging device. In non-limiting examples, the intraoperative imaging device may be oriented to face upward, downward, or to the side.

[0046] As used herein, “pose estimation method” refers to a method for estimating the parameters of a camera associated with a second imaging modality in the 3D space of a first imaging modality. A non-limiting example of such a method is obtaining the parameters of an intraoperative fluoroscopy camera in the 3D space of a preoperative CT. By using such estimated pose in a mathematical model, at least one 3D point in a preoperative computed tomography (CT) image is projected onto a corresponding 2D point in an intraoperative X-ray image.

[0047] As used herein, “multi-view pose estimation method” refers to a method for estimating some of at least two different poses of an intraoperative imaging device. In this case, the imaging device acquires images from the same scene / subject.

[0048] As used herein, “relative angular difference” refers to the angular difference between two orientations of an imaging device caused by a relative angular movement between them.

[0049] As used herein, “difference in relative posture” refers to the difference in both position and relative angle between two postures of an imaging device, caused by relative spatial movement between the subject and the imaging device.

[0050] As used herein, “epipolar distance” refers to a measured distance between a point and the epipolar line of the same point in another view. As used herein, “epipolar line” refers to a line calculated from one or more points x,y vectors or a two-column matrix in a given view.

[0051] As used herein, “similarity measure” refers to a real-valued function that quantifies the similarity between two objects.

[0052] In some embodiments, the present invention: Obtaining the first image from the first imaging modality, Extracting at least one element from a first image from a first imaging modality, Extraction includes, at least one element, an airway, blood vessel, body cavity, or any combination thereof. From a second imaging modality, at least (i) a first image of the radiopaque device in a first orientation and (ii) a second image of the radiopaque device in a second orientation, The radiopaque device is located inside the patient's body cavity, and it is necessary to obtain it. To generate at least two expanded bronchography images, The first expanded bronchography image corresponds to the first image from the radiopaque device in the first position. The second expanded bronchography image corresponds to the second image from the radiopaque device in the second position, and is generated accordingly. During the following period, that is, (i) The first position of the radiopaque equipment, (ii) Determining the mutual geometric constraints between the second orientation of the radiopaque device and, The first and second orientations of a radiopaque device are estimated by comparing the first and second orientations of the radiopaque device with the first image of the first imaging modality, To compare, (i) First dilated bronchography, (ii) A second dilated bronchography, and (iii) performed using at least one element, The estimated first orientation of the radiopaque device and the estimated second orientation of the radiopaque device satisfy the determined mutual geometric constraints, and The process includes generating a third image, the third image being an extended image derived from a second imaging modality, which highlights the target region, The method provides a method in which the target region is determined from data from a first imaging modality.

[0053] In some embodiments, at least one element from the first image from the first imaging modality further includes ribs, spine, diaphragm, or any combination thereof. In some embodiments, the relative geometric constraints are generated by: a. Estimating the difference between (i) the first posture and (ii) the second posture by comparing the first image from the radiopaque device with the second image from the radiopaque device, The estimation is performed using a device including an angle meter, accelerometer, gyroscope, or any combination thereof, and the device is attached to a second imaging modality. b. Extracting multiple image features in order to estimate relative posture changes, Multiple image features include anatomical elements, non-anatomical elements, or any combination thereof. Image features include a patch attached to the patient, a radiopaque marker placed within the field of view of the second imaging modality, or any combination thereof. Image features are those visible in the first image and the second image of the radiopaque device, and are to be extracted. c. The difference between (i) the first posture and (ii) the second posture is determined by using at least one camera, The cameras include video cameras, infrared cameras, depth cameras, or any combination thereof. The camera is in a fixed position. The camera is configured to track at least one feature, At least one feature includes a marker attached to the patient, a marker attached to a second imaging modality, or any combination thereof, and is used. By tracking at least one feature, and by estimating, d. Or any combination of these.

[0054] In some embodiments, the method further includes tracking a radiopaque instrument to determine its trajectory and using the trajectory as an additional geometric constraint, the radiopaque instrument including an endoscope, an intrabronchial instrument, or a robotic arm.

[0055] In some embodiments, the present invention includes the following methods: To generate a map of at least one body cavity of the patient, The map is generated using the first image from the first imaging modality, The method involves obtaining an image of a radiopaque device equipped with at least two attached markers from a second imaging modality, At least two attached markers are separated by a known distance, and to obtain them, Identifying the orientation of the radiopaque device from the second imaging modality to a map of at least one body cavity of the patient, Identifying the first position of a first marker attached to a radiopaque device on a second image from a second imaging modality, Identifying the second position of a second marker attached to a radiopaque device on a second image from a second imaging modality, Measure the distance between the first position of the first marker and the second position of the second marker, Projecting the known distance between the first marker and the second marker, To pinpoint the exact location of a radiopaque device within at least one body cavity of a patient, compare the measured distance with the projected known distance between a first marker and a second marker. 3D information inferred from a single view is still ambiguous and may lead to the instrument being mispositioned at multiple locations within the lung. The occurrence of such situations can be reduced by analyzing the planned 3D path before the actual procedure and calculating the most optimal fluoroscope orientation to avoid most of the ambiguity during guidance. In some embodiments, the fluoroscope positioning is carried out according to the method described in claim 4 of International Patent Application PCT / IB2015 / 00438, the entirety of which is incorporated herein by reference.

[0056] In some embodiments, radiopaque devices include endoscopes, endobronchial instruments, or robotic arms.

[0057] In some embodiments, the method further includes determining the depth of the radiopaque instrument using the trajectory of the radiopaque instrument.

[0058] In some embodiments, the first image from the first imaging modality is a preoperative image. In some embodiments, at least one image from a radiopaque device from the second imaging modality is an intraoperative image.

[0059] In some embodiments, the present invention includes the following methods: Obtaining the first image from the first imaging modality, Extracting at least one element from a first image from a first imaging modality, Extraction includes, at least one element, an airway, blood vessel, body cavity, or any combination thereof. From a second imaging modality, in two different orientations of the second imaging modality, to acquire at least (i) one image of the radiopaque device and (ii) another image of the radiopaque device, The first image from the radiopaque device is taken in the first orientation of the second imaging modality. The second image from the radiopaque device is taken in the second orientation of the second imaging modality. The radiopaque device is located inside the patient's body cavity, and it is necessary to obtain it. To generate at least two augmented bronchography images corresponding to each of two orientations of an imaging device, wherein the first augmented bronchography image is derived from a first image of a radiopaque device, and the second augmented bronchography image is derived from a second image of a radiopaque device. During the following period, that is, (i) The first stance of the second imaging modality, (ii) Determining the mutual geometric constraints between the second posture of the second imaging modality and, Estimating two poses of the second imaging modality relative to the first image of the first imaging modality, using at least one element extracted from the corresponding extended bronchography image and the first image of the first imaging modality, The two estimated postures satisfy each other's geometric constraints, and the estimation is as follows: The method involves generating a third image, which is an augmented image with the target region highlighted, derived from a second imaging modality based on data sourced from a first imaging modality.

[0060] During the guidance of an intrabronchial instrument, it is necessary to verify the instrument's position in 3D relative to the target and other anatomical structures. After reaching a certain position within the lung, the physician can change the position of the fluoroscope while keeping the instrument in the same position. Using these intraoperative images, a person skilled in the art can reconstruct the instrument's position in 3D and show the physician the instrument's position relative to the target in 3D.

[0061] To reconstruct the location of an instrument in 3D, it is necessary to pick up corresponding points in both views. These points could be specific markers on the instrument, or identifiable points on any instrument, such as the tip of an instrument or the tip of a bronchoscope. To achieve this, epipolar lines can be used to find correspondences between points. Furthermore, epipolar constraints can be used to filter out the detection of false positive markers and to eliminate markers that do not have a corresponding pair due to marker misdetection (see Figure 8).

[0062] (The epipolar is related to the geometry of stereoscopic vision, a special field of computational geometry.)

[0063] In some embodiments, virtual markers can be generated on any instrument, for example, an instrument without visible radiopaque markers. This is done by: (1) selecting an arbitrary point on the instrument in a first image; (2) calculating an epipolar line on a second image using a known geometric relationship between the two images; and (3) intersecting the epipolar line with a known or from the second image trajectory of the instrument to provide a virtual marker for matching.

[0064] In some embodiments, the present invention includes the following methods: Obtaining the first image from the first imaging modality, Extracting at least one element from a first image from a first imaging modality, Extraction includes, at least one element, an airway, blood vessel, body cavity, or any combination thereof. Acquiring at least two images from a second imaging modality in two different orientations of the second imaging modality at the same radiopaque equipment location for at least one or more different equipment locations, The radiopaque device is located inside the patient's body cavity, and it is necessary to obtain it. Using the mutual geometric constraints between the orientations of corresponding images, the 3D trajectory of each instrument is reconstructed from multiple corresponding images of the same instrument's location in a reference coordinate system. This involves estimating a transformation that aligns the reconstructed 3D trajectory of the radiopaque instrument with the 3D trajectory extracted from the image of the first imaging modality, thereby estimating the transformation between the reference coordinate system and the image of the first imaging modality. To generate a third image, which is an augmented image highlighting the target region, derived from the second imaging modality using a known orientation in the reference coordinate system, based on data sourced from the first imaging modality using a transformation between the reference coordinate system and the image of the first imaging modality.

[0065] In some embodiments, a method for acquiring images from different orientations of multiple radiopaque instrument locations comprises: (1) positioning the radiopaque instrument at a first location; (2) acquiring an image of a second imaging modality; (3) changing the orientation of the imaging device of the second modality; (4) acquiring another image of the second imaging modality; (5) changing the location of the radiopaque instrument; and (6) continuing step 2 until a desired number of unique radiopaque instrument locations are obtained.

[0066] In some embodiments, it is possible to reconstruct the location of any element that can be identified on at least two intraoperative images obtained from imaging devices in two different poses. When each pose of the second imaging modality is known relative to the first image of the first imaging modality, it is possible to show the reconstructed 3D location of an element relative to any anatomical structure from the image of the first imaging modality. An example of the use of this technique is the confirmation of the 3D location of a reference point marker deployed relative to an object.

[0067] In some embodiments, the present invention includes the following methods: Obtaining the first image from the first imaging modality, Extracting at least one element from a first image from a first imaging modality, Extraction includes, at least one element, an airway, blood vessel, body cavity, or any combination thereof. From a second imaging modality, in two different orientations of the second imaging modality, to obtain at least (i) one image of the radiopaque reference point and (ii) another image of the same radiopaque reference point, The first image of the radiopaque reference point is taken in the first orientation of the second imaging modality. The second image of the radiopaque reference point is acquired in the second orientation of the second imaging modality. The 3D location of the radiopaque reference point is determined from two orientations of the imaging device. (i) The first stance of the second imaging modality, (ii) Reconstructing using the mutual geometric constraints between the second posture of the second imaging modality and, To generate a third image that shows the 3D relative location of a reference point to a target area, based on data sourced from a first imaging modality.

[0068] In some embodiments, anatomical elements such as ribs, spine, diaphragm, or any combination thereof are extracted from the first imaging modality and the second imaging modality.

[0069] In some examples, the mutual geometric constraints are generated by the following: a. Estimating the difference between (i) the first posture and (ii) the second posture by comparing the first image from the radiopaque device with the second image from the radiopaque device, The estimation is performed using a device including an angle meter, accelerometer, gyroscope, or any combination thereof, and the device is attached to a second imaging modality. b. Extracting multiple image features in order to estimate relative posture changes, Multiple image features include anatomical elements, non-anatomical elements, or any combination thereof. Image features include a patch attached to the patient, a radiopaque marker placed within the field of view of the second imaging modality, or any combination thereof. Image features are those visible in the first image and the second image of the radiopaque device, and are to be extracted. c. The difference between (i) the first posture and (ii) the second posture is determined by using at least one camera, The cameras include video cameras, infrared cameras, depth cameras, or any combination thereof. The camera is in a fixed position. The camera is configured to track at least one feature, At least one feature includes a marker attached to the patient, a marker attached to a second imaging modality, or any combination thereof, and is used. By tracking at least one feature, and by estimating, d. Or any combination of these.

[0070] In some embodiments, the method further includes tracking a radiopaque instrument to identify its trajectory and using such trajectory as an additional geometric constraint, the radiopaque instrument including an endoscope, an intrabronchial instrument, or a robotic arm.

[0071] In some embodiments, the present invention is a method for determining the true location of a device within a patient's body, including: Using a map of at least one body cavity of the patient generated from the first image of the first imaging modality, The method involves acquiring an image of a radiopaque device from a second imaging modality, on which at least two markers are attached at a predetermined distance apart. The markers can be recognized from images located within at least two different body cavities in the patient's body, and can be acquired from those images. To obtain the attitude of the second imaging modality for the map, Identifying the first position of a first marker attached to a radiopaque device on a second image from a second imaging modality, Identifying the second position of a second marker attached to a radiopaque device on a second image from a second imaging modality, Measure the distance between the first position of the first marker and the second position of the second marker, Using the orientation of the second imaging modality, the known distance between markers is projected onto each of the recognized locations of the radiopaque device, To determine the true position of the device within the body, compare the measured distance with the projected distance between the two markers.

[0072] In some embodiments, radiopaque devices include endoscopes, endobronchial instruments, or robotic arms.

[0073] In some embodiments, the method further includes determining the depth of the radiopaque instrument using the trajectory of the radiopaque instrument.

[0074] In some embodiments, the first image from the first imaging modality is a preoperative image. In some embodiments, at least one image from a radiopaque device from the second imaging modality is an intraoperative image.

[0075] Multi-view pose estimation

[0076] International application PCT / IB2015 / 000438, which includes a description of a method for estimating the positional information (e.g., location, orientation) of a fluoroscope device relative to a patient during an endoscopic procedure, is incorporated herein by reference in its entirety. International application PCT / IB15 / 002148, filed on October 20, 2015, is also incorporated herein by reference in its entirety.

[0077] The present invention relates to a method comprising data extracted from a set of intraoperative images, each image being acquired from an imaging device in at least one (e.g., one, two, three, four, etc.) unknown pose. These images are used as input to a pose estimation method. As an exemplary embodiment, Figures 3, 4, and 5 show examples of three sets of fluoroscope images. The images in Figures 4 and 5 were acquired in the same unknown pose, while the image in Figure 3 was acquired in a different unknown pose. This set may or may not include additional known location data related to the imaging device, for example. For example, one set may include location data such as the location and orientation of a C-arm, which can be provided by the fluoroscope or acquired via a measuring device attached to the fluoroscope, such as an angle meter, accelerometer, gyroscope, etc.

[0078] In some embodiments, anatomical elements are extracted from additional intraoperative images, and these anatomical elements suggest geometric constraints that can be introduced into the posture estimation method. As a result, the number of elements extracted from a single intraoperative image can be reduced before using the posture estimation method.

[0079] In some embodiments, the multi-view pose estimation method further includes overlaying information sourced from a preoperative modality onto any image from a set of intraoperative images.

[0080] In some embodiments, a description of overlay information for intraoperative images sourced from preoperative modalities can be found in International Application PCT / IB2015 / 000438, which is incorporated herein by reference in its entirety.

[0081] In some embodiments, multiple second imaging modalities allow for changing the orientation of the fluoroscope relative to the patient to acquire multiple images (e.g., rotation or linear movement of the fluoroscope arm, rotation and movement of the patient bed, relative movement of the patient on the bed, or any combination of the above), in which case multiple images are acquired as any combination of rotational and linear movement between the patient and the fluoroscope device, from the aforementioned relative orientation of the fluoroscope as the source.

[0082] While several embodiments of the present invention have been described, it will be understood that these embodiments are illustrative and not restrictive, and that many modifications will become apparent to those skilled in the art. Furthermore, various steps may be performed in any desired order (and any desired steps may be added and / or removed).

[0083] Herein, we refer to the following examples, which illustrate some embodiments of the present invention in a non-limiting manner, in conjunction with the above description.

[0084] Example: Minimally invasive lung surgery

[0085] Non-limiting exemplary embodiments of the present invention can be applied to minimally invasive lung procedures, in which an endobronchial device is inserted into the patient's bronchial airway through the working channel of a bronchoscope (see Figure 6). Before commencing the diagnostic procedure, the physician performs a setup process in which the physician positions the catheter in several (e.g., two, three, four, etc.) bronchial airways around the area of ​​interest. Fluoroscope images of each position of the endobronchial catheter are obtained, as shown in Figures 2, 3, and 4. An example of a guidance system used to perform intraoperative fluoroscopy device orientation estimation is described in International Application No. PCT / IB2015 / 000438, and the present method of the present invention uses extracted elements (e.g., multiple catheter positions, anatomical structures of the ribs, and the patient's body boundaries, but not limited to these).

[0086] After estimating the position in the target area, the pathways for inserting the bronchoscope can be identified on the pre-procedural imaging modality and marked by highlighting or by overlaying information from the pre-operative images onto the intraoperative fluoroscopy image. After guiding the endobronchial catheter into the target area, the physician can rotate, change the zoom level of, or move the fluoroscopy device to verify, for example, that the catheter is positioned within the target area. Typically, such a change in the position of the fluoroscopy device, as shown in Figure 4, invalidates the previously estimated position, requiring the physician to repeat the setting process. However, since the catheter is already positioned within the possible target area, repeating the setting process is not necessary.

[0087] Figure 4 shows an exemplary embodiment of the present invention, illustrating how the angle and orientation of the fluoroscope are estimated using anatomical elements extracted from Figures 2 and 3 (wherein Figures 2 and 3, for example, show images obtained from the initial setup process, as well as additional anatomical elements extracted from the images, such as the position of the catheter, the anatomical structure of the ribs, and the body boundary). The orientation can be changed, for example, by (1) moving the fluoroscope (e.g., rotating the head around the C-arm), (2) moving the fluoroscope forward or backward, or alternatively by changing the location of the subject, or a combination of both. Furthermore, the mutual geometric constraints between Figures 2 and 4, such as location data related to the imaging device, can be used in the estimation process.

[0088] Figure 1 shows an exemplary embodiment of the present invention, which is as follows:

[0089] I. Component 120 extracts 3D anatomical elements such as the bronchial airways, ribs, and diaphragm from preoperative images such as CT, magnetic resonance imaging (MRI), and positron emission tomography-computed tomography (PET-CT), but not limited to these, using an automated or semi-automated segmentation process, or any combination thereof. Examples of automated or semi-automated segmentation processes are described in "Three-dimensional Human Airway Segmentation Methods for Clinical Virtual Bronchoscopy," Atilla P. Kiraly, William E. Higgins, Geoffrey McLennan, Eric A. Hoffman, and Joseph M. Reinhardt, which is incorporated herein by reference in its entirety.

[0090] II. Component 130 extracts 2D anatomical elements from a set of intraoperative images, such as fluoroscopic images and ultrasound images, but not limited to these (these are further shown in Figure 4 and include, for example, the bronchial airways 410, ribs 420, body boundaries 430, and diaphragm).

[0091] III. Component 140 calculates the mutual constraints between each subset of images in the set of intraoperative images, such as differences in relative angles, differences in relative posture, epipolar distance, etc.

[0092] In another embodiment, the method includes estimating mutual constraints between each subset of images in a set of intraoperative images. Non-limiting examples of such methods include: (1) the use of a measuring device attached to an intraoperative imaging device to estimate relative postural changes between at least two postures of a pair of fluoroscopic images; (2) the extraction of image features, including but not limited to anatomical or non-anatomical elements visible on both images, including a patch attached to the patient (e.g., an ECG patch) or a radiopaque marker placed within the field of view of the intraoperative imaging device, and the estimation of relative postural changes using these features; and (3) the use of a set of cameras, such as a video camera, infrared camera, depth camera, or any combination thereof, mounted at a designated location in the operating room, to track features, such as a patch or marker attached to the patient, or a marker attached to the imaging device. By tracking such features, the components can estimate relative postural changes of the imaging device.

[0093] IV. Component 150 matches 3D elements generated from preoperative images to their corresponding 2D elements generated from intraoperative images. For example, matching a given 2D bronchial airway extracted from a fluoroscopy image to a set of 3D airways extracted from a CT image.

[0094] V. Component 170 estimates the posture for each image in a set of intraoperative images in a desired coordinate system, such as a preoperative image coordinate system or a surgical environment-related coordinate system formed by other imaging or guidance devices.

[0095] The inputs to this component are as follows: • 3D anatomical elements extracted from preoperative patient images. • 2D anatomical elements extracted from a set of intraoperative images. As described herein, the images in the set may be sourced from the same or different imaging devices. • Mutual constraints between each subset of images in a set of intraoperative images

[0096] Component 170 evaluates the posture for each image from the set of intraoperative images as follows: • The extracted 2D elements are matched to the corresponding and projected 3D anatomical elements. The mutual constraints calculated by component 140 are applied to the estimated posture.

[0097] To match projected 3D elements from preoperative images to corresponding 2D elements from intraoperative images, similarity measures such as distance measurements are necessary. Such distance measurements provide a measure for evaluating the distance between projected 3D elements and their corresponding 2D elements. For example, the Euclidean distance between two polylines (e.g., a series of connected line segments created as a single object) can be used as a similarity measure between a 3D projected bronchial airway from a preoperative image and a 2D airway extracted from an intraoperative image.

[0098] Furthermore, in one embodiment of the method of the present invention, the method includes estimating a set of poses corresponding to a set of intraoperative images by identifying a pose that optimizes a similarity scale when mutual constraints between subsets of images from the intraoperative image set are satisfied. The optimization of the similarity scale can be viewed as a least-squares problem and can be solved in several ways, for example, (1) using a well-known bundle adjustment algorithm that implements an iterative minimization method for pose estimation, the whole of which is incorporated herein by reference: B. Triggs, P. McLauchlan, R. Hartley, A. Fitzgibbon (1999), "Bundle Adjustment—A Modern Synthesis", ICCV '99: Proceedings of the International Workshop on Vision Algorithms, Springer-Verlag, pp. 298-372; and (2) using a grid search method for searching for the optimal pose that optimizes the similarity scale by scanning the parameter space.

[0099] Marker

[0100] To reconstruct 3D information about the location of a device, radiopaque markers can be placed at predetermined locations on the medical device. Several pathways of 3D structures in internal body lumens, such as bronchial airways or blood vessels, can be projected onto similar 2D curves on intraoperative images. For example, as shown in International Application No. PCT / IB2015 / 000438, the 3D information obtained using the markers can be used to distinguish between such pathways.

[0101] In an exemplary embodiment of the present invention, as shown in Figure 5, the device is imaged by an intraoperative device and projected onto the imaging plane 505. Since both airways are projected onto the same curve on the imaging plane 505, it is unknown whether the device is placed in airway 520 or airway 525. To distinguish between airway 520 and airway 525, it is possible to use at least two radiopaque markers attached to a catheter, with a predetermined distance "m" between the markers. In Figure 5, the markers seen on the preoperative image are labeled "G" and "F".

[0102] The process of distinguishing between airway 520 and airway 525 can be carried out as follows:

[0103] (1) Point F from the intraoperative image is projected onto possible candidate airways 520 and 525 to obtain points A and B.

[0104] (2) Point G from the intraoperative image is projected onto possible candidate airways 520 and 525 to obtain points C and D.

[0105] (3) Measure the distance between the pair of projected markers |AC| and |BD|.

[0106] (4) Compare the distance |AC| above 520 and the distance |BD| above 525 with the distance m predetermined by the device manufacturer. Select the appropriate airway according to the similarity of the distances.

[0107] Tracking scope

[0108] As a non-limiting example, this specification discloses a method for registering a patient's CT scan with a fluoroscope device. This method uses anatomical elements detected in both the fluoroscope image and the CT scan as input to a posture estimation algorithm that generates the fluoroscope device posture (e.g., orientation and location) relative to the CT scan. Below, this method is extended by adding a 3D spatial trajectory corresponding to the location of the intrabronchial device as input to the registration method. These trajectories can be obtained by several means, such as by attaching a location sensor along the scope or by using a robotic endoscope arm. Such an intrabronchial device is hereafter referred to as a tracking scope. The tracking scope is used to guide surgical instruments extending from it to the area of ​​interest (see Figure 7). Diagnostic instruments may be catheters, forceps, needles, etc. Below, a method for using location measurements obtained by a tracking scope to improve the accuracy and robustness of the registration method described herein is described.

[0109] In one embodiment, registration between the trajectory of the tracking scope and the coordinate system of the fluoroscope device is achieved by positioning the tracking scope at various locations in space and applying a standard attitude estimation algorithm. For reference to the attitude estimation algorithm, see the following paper, which is incorporated herein by reference in its entirety: "EPnP: Efficient Perspective-n-Point Camera Pose Estimation" by F. Moreno-Noguer, V. Lepetit, and P. Fua.

[0110] The posture estimation method disclosed herein is performed by estimating posture such that selected elements in a CT scan are projected onto their corresponding elements in a fluoroscope image. In one embodiment of the present invention, this method is extended by adding the trajectory of a tracking scope as input to the posture estimation method. These trajectories can be transformed into the coordinate system of the fluoroscope device using the method herein. When the trajectories are transformed into the fluoroscope device coordinate system, they act as an additional constraint on the posture estimation method because the estimated posture is constrained by the condition that the segmented bronchial airways from the registered CT scan and their trajectories must be aligned.

[0111] By using the orientation estimated by the fluoroscope device, anatomical elements from preoperative CT can be projected onto the fluoroscope's live video to guide surgical instruments to a designated target within the lung. Such anatomical elements may include, but are not limited to, the target lesion, the pathway to the lesion, etc. The projected pathway to the target lesion provides the physician with only two-dimensional information, resulting in ambiguity of depth; that is, several airways segmented on the CT may correspond to the same projection on the 2D fluoroscope image. It is important to properly identify the bronchial airway to which the surgical instrument will be placed on the CT. One method used to reduce such ambiguity, as described herein, is performed by using radiopaque markers placed on the instrument that provide depth information. In another embodiment of the present invention, ambiguity can be reduced using a tracking scope because it provides a 3D location within the bronchial airway. When applied to a branching bronchial tree, such a technique makes it possible to eliminate potential ambiguous options up to the tip 701 of the tracking scope in Figure 7. Assuming that the surgical instrument 702 in Figure 7 does not have a 3D trajectory, the aforementioned ambiguity with respect to this part of the instrument 702 would still arise, although the likelihood of such a occurrence is far lower. Therefore, this embodiment of the present invention improves the ability of the method described herein to properly locate the instrument.

[0112] Digital Computational Tomography (DCT)

[0113] In some embodiments, tomographic reconstruction from intraoperative images can be used to calculate the location of a target relative to a reference coordinate system. A non-limiting example of such a reference coordinate system can be defined by a fixture equipped with radiopaque markers of known geometry, enabling the calculation of the relative orientation of each intraoperative image. Since the geometric relationship of each input frame of the tomographic reconstruction to the reference coordinate system is known, the location of the target can also be placed within the reference coordinate system. This makes it possible to project the target onto further fluoroscopic images. In some embodiments, the projected location of the target can be corrected for respiratory motion by tracking the tissue within the area of ​​the target. In some embodiments, this motion correction is performed according to an exemplary method described in International Patent Application PCT / IB2015 / 00438, the entirety of which is incorporated herein by reference.

[0114] A method for augmenting intraoperative images using a C-arm-based CT and reference posture device, comprising the following: Collecting multiple intraoperative images whose geometric relationship with the reference coordinate system is known, Reconstructing a 3D object (volume), Marking the target area on the reconstructed three-dimensional object, Projecting the object onto further intraoperative images where the geometric relationship with the reference coordinate system is known.

[0115] In other embodiments, a tomographically reconstructed object can be registered to a preoperative CT object. Given known central locations of the object or its associated anatomical structures, such as blood vessels, bronchial airways, or orbital bifurcations, in both the reconstructed and preoperative objects, both objects can be initially aligned. In other embodiments, ribs extracted from both objects can be used to find the alignment (e.g., initial alignment). In some embodiments, the step of finding the proper rotation between the objects can be used to match the reconstructed location and trajectory of the device to all conceivable airway trajectories extracted from the CT. The best matching determines the most optimal relative rotation between the objects.

[0116] In some embodiments, the tomographically reconstructed model can be registered to the preoperative CT model using at least three common anatomical targets that can be identified on both the tomographically reconstructed model and the preoperative CT model. Examples of anatomical targets may include airway bifurcations and blood vessels.

[0117] In some embodiments, tomographically reconstructed 3D objects can be registered to preoperative CT 3D objects using image-based similarity methods such as cross-referencing.

[0118] In some embodiments, the tomographically reconstructed object can be registered to the preoperative CT object using a combination of at least one common anatomical target (e.g., a 3D-3D constraint) and at least one additional 3D-2D constraint (e.g., ribs or thoracic boundary). In such embodiments, both types of constraints can be formulated as energy functions and minimized using standard optimization methods such as the steepest descent method.

[0119] In other embodiments, tomographically reconstructed 3D objects from different time points in time using the same procedure can be registered. One possible application of this is to compare two images and transfer manually entered markings from one image to the other to show chronological 3D information.

[0120] In other embodiments, only partial information may be reconstructed from DCT due to limitations in the quality of fluoroscopic imaging, interference of the target area by other tissues, and spatial constraints of the surgical environment. In such cases, the corresponding partial information can be identified between a partial 3D object reconstructed from intraoperative imaging and the preoperative CT. The two image sources can be fused into one to form an integrated data set. The aforementioned data set can be updated as needed with additional intraoperative images.

[0121] In other embodiments, a three-dimensional object reconstructed by tomography can be registered to a 3D target shape reconstructed by REBUS.

[0122] A method for performing fluororesistance registration from CT using tomography, consisting of the following: This involves marking the target on the preoperative image and extracting the bronchial tree, The endoscopic equipment is placed within the target lung lobe, The tomography is spun using the C-arm while the instrument is stable inside, Marking objects and equipment on the reconstructed three-dimensional object, Aligning the pre-operative three-dimensional object with the reconstructed three-dimensional object according to the location of the target or the location of the attached anatomical structure, For all possible airway trajectories extracted from CT, The calculation of the optimal rotation between three-dimensional objects to minimize the distance between the reconstructed trajectory of the equipment and each airway trajectory, Select the rotation that corresponds to the shortest distance, By using the alignment between two three-dimensional objects, the reconstructed three-dimensional object is reinforced with anatomical information obtained from the pre-operative three-dimensional object, Further highlighting of the target area on intraoperative images.

[0123] In other embodiments, the quality of digital tomosynthesis can be improved by using a 3D model of a preceding preoperative CT scan. By performing a known coarse registration between the intraoperative image and the preoperative CT scan, the relevant region of the target can be extracted from the 3D model of the preoperative CT scan. The quality of the reconstructed image is greatly improved by imposing constraints on well-known reconstruction algorithms, which is incorporated herein by reference as a whole: Sehopoulos, Ioannis (2013), "A review of breast tomosynthesis. Part II. Image reconstruction, processing and analysis, and advanced applications," Medical Physics. 40 (1): 014302. As an example of such constraints, the initial 3D model can be initialized using a 3D model extracted from the preoperative CT.

[0124] In some embodiments, methods for improving tomographic reconstruction using preoperative CT scans include: This involves registering the images taken during the surgery and the preoperative CT scan, Extracting the region of the target object from the preoperative CT scan, Adding constraints to a well-known reconstruction algorithm, Reconstructing the image using the added constraints, Estimating the pose during tomography using a target object.

[0125] In some embodiments, multiple images of the same region from different poses are required to perform tomographic reconstruction.

[0126] In some embodiments, pose estimation can be performed using a fixed pattern of 3D radiopaque markers, as described in International Patent Application PCT / IB17 / 01448, "Jigs for use in medical imaging and methods for use thereof" (which is thus incorporated herein by reference). For example, using such a 3D pattern with radiopaque markers imposes a physical constraint that the pattern must be at least partially visible in the image frame along with the area of ​​the patient in question.

[0127] For example, one such C-arm-based CT system is described in the prior art, U.S. Patent Application No. 9044190B2, relating to a "C-arm computerized tomography system." In this application, a three-dimensional object is typically used, positioned in a fixed location relative to the subject, and a series of video images of the object area of ​​the subject are acquired while the C-arm is moved manually or by a scanning motor. Images from the video sequence are analyzed, and the pose of the C-arm relative to the subject is determined by analyzing the image pattern of the object.

[0128] However, this system utilizes a tertiary object with an opaque marker, and this marker must be within the field of view in every frame in order to determine its orientation. This requirement either severely limits the imaging angle of the C-arm or requires that such a 3D object (or part of an object) be positioned on or around the patient, which is a limiting factor from a clinical application standpoint as it restricts access to the patient or movement of the C-arm itself. The quality and dimensionality of tomographic reconstructions are known to depend, among other factors, particularly the rotation angle of the C-arm. From the perspective of tomographic reconstruction quality, the range of C-arm rotation angles is critical for tomographic reconstruction of small soft tissue objects. A non-limiting example representing such objects is a soft tissue lesion in the human lung measuring 8-15 mm in size.

[0129] Therefore, there is a need for a system that can obtain wide-angle imaging without requiring a restrictive three-dimensional object (or part thereof) with an opaque marker within any frame to determine a suitable C-arm pose for any imaged frame of the C-arm, as is the case with conventional C-arm fluoroscopy imaging systems.

[0130] In some embodiments of the present invention, the pose of any image can be extracted using anatomical targets that are already part of the image, using the anatomical structures of a subject (patient). Non-limiting examples of such targets include ribs, lungs, diaphragm, trachea, etc. This technique can be implemented by using a 6-degree-of-freedom pose estimation algorithm from a 3D-2D correspondence. Such a method is also described in this patent disclosure. See Figure 9.

[0131] In some embodiments, the C-arm's motion continuity and the missing frame orientation can be extrapolated from known frames. Alternatively, in such cases, a hybrid approach can be used, in which the orientation is estimated for a subset of frames by assuming that the pattern or part thereof is visible with respect to such calculations, based on the pattern of radiopaque markers.

[0132] In some embodiments, the present invention includes pose estimation for any frame from known trajectory movements of an imaging device, assuming that the trajectory of the X-ray imaging device is known or extrapolable and bounded. A non-limiting example in Figure 10A shows the pose of an X-ray imaging device attached to a C-arm, covering a pattern of radiopaque markers. A subset of all frames with the radiopaque marker pattern is used to estimate the 3D trajectory of the imaging device. This information is used to greatly limit the solution search space by restricting the pose estimation of the pose in Figure 10B to a specific 3D trajectory.

[0133] In some embodiments, once the 3D trajectory of the C-arm's movement is estimated, such movement can be represented by a small number of variables. In the non-restrictive example depicted in Figure X1, the C-arm has an isocenter such that its 3D trajectory can be estimated using at least two known orientations of the C-arm, and this can be represented by a single parameter "t". In this case, having at least one known visible 3D target in the image is sufficient to estimate the parameter "t" in the trajectory corresponding to each orientation of the C-arm. See Figure 11.

[0134] In some embodiments, assuming the use of trigonometry and known intrinsic camera parameters, at least two known poses of the C-arm are required to estimate the 3D location of the target. Additional poses can be used for more stable and robust target location estimation.

[0135] In some embodiments, the method for performing tomographic reconstruction of a three-dimensional object according to the embodiments of the present invention includes the following: Rotating the X-ray imaging device, The method involves estimating the trajectory of an X-ray imaging device using frames, wherein the estimated 3D target becomes visible by solving the camera's location and orientation using 3D trajectory constraints and known 3D-2D correspondence features. The evaluation involves assessing the position of a frame on its trajectory, so that the estimated 3D target becomes visible or partially visible by an extrapolation algorithm based on the assumption that its movement is continuous. By solving for the camera's location and orientation using 3D trajectory constraints and known 3D-2D corresponding features, the pose of each frame is estimated. The process of calculating a three-dimensional reconstruction of the target region.

[0136] In some embodiments, the method for performing tomographic reconstruction of a three-dimensional object according to the present invention includes the following: Rotating the X-ray imaging device, This involves estimating the trajectory of an X-ray imaging device using frames, where the pattern of radiopaque markers is visible and the orientation can be estimated. This involves estimating the orientation of the frame, where only the estimated 3D target becomes visible by solving the camera's location and orientation using 3D trajectory constraints and known 3D-2D correspondence features. The process of calculating a three-dimensional reconstruction of the target region.

[0137] In some embodiments, the present invention relates to a solution to the imaging device pose estimation problem that does not have any 2D-3D correspondence features (e.g., does not require prior CT images). A camera calibration process, such as that described in Furukawa, Y. and Ponce, J., "Accurate camera calibration from multi-view stereo and bundle adjustment," International Journal of Computer Vision, 84(3), pp. 257-268 (2009) (this document is incorporated herein by reference), can be applied online or offline. Using the calibrated cameras, the 3D structure of an object visible in multiple images can be estimated by applying Structure from Motion (SfM) techniques. Such objects may include, but are not limited to, anatomical objects such as ribs, blood vessels, and the spine; devices located inside the body such as intrabronchial instruments, wires, and sensors; or devices located close to the outside of the body, such as those attached to the body. In some embodiments, all cameras are solved simultaneously. Such structure-from-motion techniques are described in Torr, PH and Zisserman, A, "Feature based methods for structure and motion estimation. In International workshop on vision algorithms" (pp. 278-294) (September 1999), Springer, Berlin, and Heidelberg (this document is incorporated herein by reference).

[0138] In some embodiments, the present invention enables overcoming the limitations of using known patterns of 3D radiopaque markers by combining a target 3D pattern and a 3D target that is dynamically estimated from the time of C-arm rotation, or possibly before such rotation, for the purpose of acquiring an imaging sequence for tomographic reconstruction. Non-limiting examples of such targets are represented by objects inside the patient's body, such as markers on endobronchial devices, device tips, etc., or by objects attached to the outside of the body, such as patches, wires, etc.

[0139] In some embodiments, as shown in Figure 12, the 3D target is estimated using prior art tomography or stereo algorithms that estimate the pose for each image frame using a known set of visible radiopaque markers.

[0140] In some embodiments, as an alternative, the 3D targets described above are estimated using the Structure from Motion (SfM) method, independent of radiopaque markers in the frame, as shown in Figure 13. In the next step, additional 3D targets are estimated. The estimated 3D targets are used to estimate the pose for frames that do not have a known 3D pattern of markers. Finally, a stereoscopic reconstruction is calculated using a series of all available images.

[0141] In some embodiments, the present invention is a method for reconstructing a three-dimensional object from a series of X-ray images, including: Estimating a 3D target from at least two frames with known poses, Using the reconstructed target, estimate the pose for other frames that do not have a pattern of radiopaque markers in the image frame, Calculate the 3D reconstruction using all frames.

[0142] In some embodiments, the present invention is an iterative reconstruction method that maximizes output imaging quality by iteratively fine-tuning the input to the reconstruction algorithm. A non-limiting example of image quality measurement could be the measurement of image sharpness. Since sharpness is related to the contrast of an image, a contrast measure can therefore be used as a sharpness or "autofocus" function. Such measurements are defined in Groen, F., Young, I., and Lighthart, G., "A comparison of different focus functions for use in autofocus algorithms," Cytometry 6, 81-91 (1985). As an example, the squared value Φ(a) of the gradient of a focus measure for an image in area a is given by:

[0143] Φ(a)=ΣΣΣ(f(x,y,z+1)-f(x,y,z))^2

[0144] Since the target area should be roughly in the center of the reconstructed 3D object, it makes sense to limit the calculation to the small rectangular area in the center.

[0145] In some embodiments, this can be formulated as an optimization problem and solved using techniques such as the steepest descent method. Fine-tuning of posture is done using the following function: p n+1 =p n +α∇F(p n ), in the formula, F is the posture p n This is done by updating the reconstruction function, which represents the reconstruction function given the condition, and then calculating the value of the sharpness function Φ().

[0146] In some embodiments, the present invention is an iterative pose matching method that improves output imaging quality by iteratively fine-tuning the camera pose to satisfy certain geometric constraints. Non-limiting examples of such constraints could be the same feature point of an object visible in multiple frames, and therefore necessarily, located at the intersection of rays connecting that object to the focal point of each camera (see Figure 14).

[0147] Initially, this is rarely the case due to inaccuracies in pose estimation and further due to object displacement (e.g., due to respiration). By correcting the camera's pose to satisfy the ray crossing constraint, pose determination errors and movement of the imaged target area are locally compensated, resulting in better reconstructed image quality. Examples of such feature points may be the tip of a device in the patient, or an opaque marker on the device, etc.

[0148] In some embodiments, this process can be formulated as an optimization problem and can be solved using methods such as the steepest descent method. See Figure 16 for this method. The cost function can be defined as the square of the distance between the feature points of the object and the nearest point on each ray (see Figure 15).

number

[0149] Fluoroscope device placement guidance

[0150] In some embodiments, each fluoroscope is calibrated before its first use. In some embodiments, the calibration process includes calculating the rotation angle of the actual fluoroscope, registering preoperative and intraoperative imaging modalities, and displaying the target on the intraoperative image.

[0151] In some embodiments, the fluoroscope is positioned before the rotation of the C-arm begins so that the object projected from the preoperative image remains centered in the image throughout the rotation of the C-arm.

[0152] In some embodiments, positioning the fluoroscope so that the object is centered in the fluoroscope image is not sufficient in itself because the height of the fluoroscope is very important, while the center of rotation is not always in the center of the image, causing unwanted shifts of the object outside the image area during C-arm rotation.

[0153] In some embodiments, the object's position is known relative to a reference frame, so the optimal 3D location of the C-arm is calculated. In some embodiments, optimizing the 3D location of the C-arm means minimizing the maximum distance of the object from the image center during C-arm rotation.

[0154] In some embodiments, to optimize the 3D position of the C-arm, the user first takes a snapshot of the fluoroscopy. In some embodiments, based on calculations, the user is instructed to move the fluoroscopy along three axes: up and down, left and right (relative to the patient), and head and foot (relative to the patient). In some embodiments, this instruction guides the fluoroscopy toward its optimal position. In some embodiments, the user moves the fluoroscopy according to the instruction and then takes another snapshot to obtain new instructions for the new position. Figure 20 shows exemplary guidance that may be provided to the user according to the above.

[0155] In some embodiments, positional quality is calculated for each snapshot by determining the percentage of sweeps (assuming + / - 30 degrees from the AP) in which the lesion is completely contained within a ROI, which is a small circle located in the center of the image.

[0156] In some embodiments, an alternative method for conveying instructions is to display static and similar dynamic patterns on the image, where the static pattern represents the desired position and the dynamic pattern represents the current object. In such embodiments, the user uses a continuous fluoroscope video, and the dynamic pattern moves in accordance with the movement of the fluoroscope. In some embodiments, the dynamic pattern moves along the x and y axes in accordance with the movement of the fluoroscope along the left-right axis and the head-foot axis, and the scale of the dynamic pattern changes in accordance with the movement of the fluoroscope device along the vertical axis. In some embodiments, the user properly positions the fluoroscope device by aligning the dynamic and static patterns. Figure 21 shows exemplary static and dynamic patterns as discussed above.

[0157] An example of improved angle-limited X-ray CT reconstruction using unsupervised deep learning.

[0158] Various algorithms exist for 3D image reconstruction from 2D images, which receive a set of 2D images of an object along with the camera orientation for each image as input and compute a 3D reconstruction of the object. These algorithms produce lower quality results when the 2D images are taken from a limited angle (an angular range of less than 180 degrees with respect to X-rays) due to loss of information. The proposed method provides a significant improvement in 3D image quality compared to other methods for reconstructing 3D images from 2D images taken from a limited angle.

[0159] In some embodiments, the present invention includes a method for performing improved angle-limited X-ray CT reconstruction using an unsupervised deep learning model: Applying existing methods to reconstruct low-quality CT images from X-ray images, Applying an image transformation algorithm from domain A to domain C, Apply an image transformation algorithm from Domain C to Domain B.

[0160] For further investigation, we will use domains A, B, and C. Domain A is defined as the "low-quality tomography reconstruction" domain, domain B is defined as the CT scan domain, and domain C is defined as the "simulated low-quality tomography reconstruction" generated from pre-procedure CT data.

[0161] In some embodiments, Section 1 can calculate the pose for all images, and then, in order to reconstruct the low-quality 3D reconstruction using, for example, the method of “estimating pose in tomography using a target object” described above, this method converts the 2D images within Domain A into low-quality CT images.

[0162] Continuing from the previous paragraph, the simulated low-quality reconstruction can be achieved by applying the FP (forward projection) algorithm, which calculates the intensity integral along the selected CT axis, to a given CT, thereby obtaining a series of simulated 2D X-ray images. In the following step, Method 1 described above is applied to reconstruct the low-quality 3D object, which is done, for example, by the SIRT (Simultaneous Iterative Reconstruction Technique) algorithm, which iteratively reconstructs the object by starting with an initial estimate of the reconstruction result and iteratively applying FP to change the current reconstruction result by the FP difference from the 2D image (https: / / tomroelandts.com / articles / the-sirt-algorithm).

[0163] In some embodiments, the domain transformation model used to translate the reconstruction from domain A to C cannot be supervised (because the simulation is aligned to a CT, and for 2D images there is no CT to which it is aligned). The simulated data can be obtained by the method described above. To train the required model, it is possible to use a cycle-consistent adversarial network (Cycle Gan) that translates the reconstruction to its aligned simulation. Training of a Cycle Gan is performed by combining adversarial loss, cycle loss, and identity loss (as described in Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros, 2017, Unpaired image-to-image translation using cycle-consistent adversarial networks, Proceedings of the IEEE international conference on computer vision, 2223-2232), which makes it possible to train using unaligned images, as shown in Figure 18.

[0164] In some embodiments, the transformation model from domain C to domain B can be supervised because, as defined by the process, the generation of a simulation given a CT is aligned to that CT. For example, as shown in Figure 19, an iCNN-based neural network using perceptual loss (as described in Justin Johnson, Alexandre Alahi, and Li Fei-Fei, Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016) and L2 distance loss can be used to train such a model.

[0165] In some embodiments, combinations of all the methods described above are shown in Figure 17, which describes a process that starts with a series of 2D images and receives 3D image reconstruction.

[0166] Equivalents The present invention provides, in particular, novel methods and compositions for treating mild to moderate acute pain and / or inflammation. While specific embodiments of the subject invention have been discussed, the above specification is illustrative and not limiting. Those skilled in the art will find many variations of the invention apparent upon closer examination of this specification. The full scope of the invention should be determined by referring to the claims together with the full scope of their equivalents, and the specification together with such variations.

[0167] Built-in by reference All publications, patents, and entries in sequence databases referenced herein are incorporated herein by reference in whole, as if each individual publication or patent were specifically and individually indicated as being incorporated by reference.

[0168] While several embodiments of the present invention have been described, it will be understood that these embodiments are illustrative and not restrictive, and that many modifications will become apparent to those skilled in the art. Furthermore, various steps may be performed in any desired order (and any desired steps may be added and / or removed).

Claims

1. The medical imaging device receives a series of medical images captured by the medical imaging device while it is being rotated by rotation, Determining the orientation of the medical imaging device when capturing each image of the subset of the series of medical images in which multiple target objects are shown, Based on the determined orientation of the medical imaging device and the trajectory constraints of the medical imaging device when the subset of the series of medical images is captured, the trajectory of the medical imaging device is estimated. The orientation of the medical imaging device when at least one of the medical images in which the plurality of target objects are not shown is captured is determined by extrapolation based on the assumption that the movement of the medical imaging device continues, To generate a three-dimensional reconstruction of a target region including the plurality of targets based on at least (a) at least some of the orientations of the medical imaging device when capturing the subset of the series of medical images in which the plurality of targets are shown, and (b) at least one of the orientations of the medical imaging device when capturing the at least one of the medical images in which the plurality of targets are not at least partially shown, Methods that include...

2. The method according to claim 1, wherein the orientation of the medical imaging device when capturing each image of the subset of the series of medical images is determined based on the correspondence between the 3D positions of the plurality of targets shown in the subset of the series of medical images and the 2D positions of the plurality of targets.

3. The method according to claim 2, wherein the 3D positions of the plurality of target objects are determined based on at least one preoperative image.

4. The method according to claim 2, wherein the 3D positions of the plurality of targets are determined by applying structure-from-motion technology.

Images