Systems and methods for providing surgical guidance

Abstract

Surgical guidance systems and methods visually mark a structure of interest (SOI) of an organ on intraoperative images captured by a localizable imaging device. The location of the SOI relative to a body structure (e.g., a passageway system) associated with the organ is determined in preoperative images captured by a medical diagnostic imaging (MDI) system. The location of the SOI relative to the body structure in the preoperative images is then mapped onto a reconstructed version of the body structure based on location information provided by localization sensors distributed on the body structure. Intraoperative images taken by the localizable imaging device are aligned with the reconstructed version of the body structure, and an image object representing the SOI is superimposed on the intraoperative images at a location mapped from the reconstructed version of the body structure.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application Serial No. 62 / 897,436, filed September 9, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to intraoperative guidance in medical procedures. More specifically, this disclosure relates to systems and methods for guiding a surgeon to one or more structures of interest (SOIs) within a patient, said structures being invisible to the surgeon, such as one or more diseased organ portions (DOPs) residing within one or more organs during the surgical procedure (and thus invisible to the surgeon). Background Technology

[0004] Endoscopy allows doctors to examine abnormalities in passageways within a patient's organs (such as blood vessels, the gastrointestinal tract, stomach, airways, etc.), such as foreign bodies, bleeding, tumors, or inflammation. Bronchoscopy is an example of endoscopic technique that involves visualizing the interior of the airways for diagnostic and therapeutic purposes. During bronchoscopy, a bronchoscope is typically inserted into the airway through the nose or mouth, or occasionally through a tracheostomy tube. This allows the practitioner to navigate to various areas of the lungs, enabling them to examine the patient's airway for abnormalities such as foreign bodies, bleeding, tumors, or inflammation.

[0005] In some cases, it is necessary to reach a specific location within the patient's body, such as to remove or excise a diseased portion of an organ (e.g., a lobe of the left or right lung) or a growth that may be benign or cancerous. This is typically done by using a laparoscope to remove the growth through the rib cage, for example, from the lung. Surgical instruments used in laparoscopic surgery include forceps, scissors, probes, dissecters, hooks, retractors, etc. In some cases, it is necessary to remove blood vessels connected to a lung lobe, not because the blood vessel or the lung lobe itself is diseased, but because the lung lobe contains a diseased portion, and the entire lobe needs to be removed due to a tumor.

[0006] Surgery can be performed using one or more imaging devices employing one or more imaging techniques, such as optical imaging (e.g., via a camera), thermal imaging, fluorescence imaging, and ultrasound imaging. However, each imaging technique has its limitations for each specific SOI. This disclosure aims to address these limitations. Summary of the Invention

[0007] In one aspect, a method for guiding a surgeon may include identifying (e.g., via a controller) a structure of interest (SOI) in a body structure and organ from preoperative images of an organ and an associated body structure; determining the position of the SOI relative to the body structure in the preoperative images; generating a reconstructed body structure; determining (e.g., via a controller) the position of the SOI relative to the reconstructed body structure based on the position of the SOI relative to the body structure in the preoperative images; capturing an intraoperative image via an imaging device; determining (e.g., via a controller) the position of the imaging device relative to the reconstructed body structure; determining (e.g., via a controller) the position of the SOI relative to the intraoperative image based on the determined position of the SOI relative to the reconstructed body structure and the position of the imaging device relative to the reconstructed body structure; generating (e.g., via a controller) an image object representing the SOI; and displaying the intraoperative image, wherein the image object is superimposed on the intraoperative image at the position of the SOI relative to the intraoperative image.

[0008] In some aspects, the body structure is a channel system including channels, and the method further includes reconstructing the channels to obtain a reconstructed channel system, and determining the spatial relationship between the reconstructed channels and the channels of the channel system in the preoperative image. In some aspects, the reconstructed channels may include transmitting positioning signals between a positioning system and positioning sensors distributed in the channels. The method may further include positioning the positioning sensors in the channels using one or more catheters.

[0009] In some aspects, determining the position of the SOI relative to the reconstructed channel system may include mapping the position of the SOI relative to the channels in the preoperative image to the reconstructed channels. In some aspects, determining the position of the imaging device relative to the reconstructed body structure includes transmitting positioning signals between a positioning system and one or more positioning sensors located in or on the imaging device. In some aspects, determining the position of the SOI relative to the reconstructed body structure includes obtaining positioning information related to the position of the positioning sensors contained in the body structure, and identifying the spatial relationship between the body structure in the preoperative image and the body structure containing the positioning sensors.

[0010] In various aspects, the method may include modifying a preoperative image based on changes in the imaging angle of the imaging device relative to the organ. In various aspects, changes in the imaging angle of the imaging device may include changes in the distance between the imaging device and the organ or changes in the imaging angle of the imaging device. In various aspects, modifying the preoperative image may include modifying the size, shape, scale, or orientation of the image content of the preoperative image. In various aspects, the method may include displaying the modified preoperative image together with intraoperative images.

[0011] In various aspects, the method may include overlaying a modified preoperative image onto an intraoperative image. In various aspects, the method may include modifying the preoperative image such that the viewpoint of the preoperative image matches the viewpoint of the intraoperative image. In various aspects, modifying the preoperative image may include aligning the viewpoint of the organ in the preoperative image, the viewpoint of the organ-associated channel system, and the viewpoint of the SOI with the viewpoint of the image captured by the imaging device. In various aspects, the SOI includes a diseased organ portion, a diseased lung portion, abnormal tissue, an anatomical structure of an abnormal organ, or a benign anatomical structure. In some aspects, the SOI includes one or more organ objects. In various aspects, the organ is a lung, the body structure is the pulmonary airway, and the imaging device is a camera.

[0012] In some aspects, generating an image object representing an SOI may include assigning morphological properties (e.g., size, shape, and / or orientation) of the SOI to the image object in a preoperative image. The organ subjected to the systems and methods described herein may be the lung, liver, kidney, ureter, or heart. The organ access may be the pulmonary airway, hepatic duct, or cardiac vessels. Preoperative images can be obtained, for example, by a medical diagnostic imaging (MDI) system. The result of imaging via an MDI system is referred to herein as an “MDI image.” The terms “preoperative image” and “MDI image” are used interchangeably herein, but preoperative images may also be obtained using other imaging systems.

[0013] In another aspect, a method for guiding a surgeon may include: receiving or acquiring an image of a channel system associated with an organ, said organ including one or more channels and a structure of interest (SOI) at least partially residing in the organ; determining the position of the SOI relative to a reconstructed channel system based on the position of the SOI relative to the channel system in the received image; capturing multiple images with an imaging device, and simultaneously performing on each of the multiple images: determining the position of the SOI relative to each image based on the position of the SOI relative to the reconstructed channel system and the relationship between each image and the reconstructed channel system; generating an image object representing the SOI; and overlaying the image object onto each image at the determined position of the SOI relative to each image. In various aspects, generating the image object representing the SOI may include assigning the image object a size, shape, and orientation that conforms to the size, shape, and orientation of the SOI relative to the channel system in the received image. In various aspects, the method may include modifying the received image such that the viewpoint of the received image matches the viewpoint of the multiple images. In various aspects, the method may include displaying the modified preoperative image along with the multiple images.

[0014] In another aspect, a system may include a positioning system configured to determine the location of channels in a channel system associated with an organ. The number of channels in the channel system of an organ, including positioning sensors, may depend on the organ undergoing the medical procedure. The system may further include an imaging device configured to generate an intraoperative image stream. In some aspects, the imaging device may include one or more positioning sensors to position the imaging device via the positioning system. The system may further include a controller configured to: reconstruct the channel system based on the locations of multiple channels; determine the location of the SOI relative to the channel system in the preoperative image based on a preoperative image of the organ channel system and the SOI; determine the location of the SOI relative to the reconstructed channel system based on the location of the SOI relative to the channel system in the preoperative image; capture intraoperative images using the imaging device; determine the location of the imaging device relative to the reconstructed channel system; and determine the location of the SOI relative to the intraoperative image based on the determined location of the SOI relative to the reconstructed channel system and the determined location of the imaging device relative to the reconstructed channel system. The controller may then overlay an image object representing the SOI onto the intraoperative image at the determined location of the SOI relative to the intraoperative image.

[0015] In various aspects, the controller can be configured to determine the position of the SOI relative to the reconstructed channel system by determining the spatial relationship between the reconstructed channel system and the channel system in the preoperative image. The reconstructed channel may include transmitting position information between a positioning system and positioning sensors distributed within the channel. The positioning sensors may be mounted on catheters located within the channel.

[0016] In various aspects, determining the position of the SOI relative to the reconstructed channel system may include mapping the position of the SOI in the preoperative image to the reconstructed channel system. In various aspects, the position of the imaging device relative to the reconstructed channel system is determined based on position information from one or more positioning sensors located within or on the imaging device relative to the reconstructed channel system. In various aspects, the organ may be the lung, and the channel may be the pulmonary airway. In various aspects, the controller may be further configured to modify the preoperative image based on changes in the distance of the imaging device from the SOI or changes in the angle at which the imaging device captures intraoperative images; and to display the modified preoperative image together with the intraoperative image on a display device. In various aspects, the imaging device is a camera. Attached Figure Description

[0017] Various exemplary aspects are shown in the accompanying drawings, but these exemplary aspects are not limiting. It should be understood that, for simplicity and clarity of illustration, the elements shown in the following accompanying drawings are not necessarily drawn to scale. Moreover, where deemed appropriate, reference numerals may be repeated in the drawings to indicate similar, corresponding, or analogous elements. In the accompanying drawings:

[0018] Figure 1 This is a flowchart illustrating a method for guiding a surgeon to an SOI in an organ according to the present disclosure;

[0019] Figure 2A This is a flowchart illustrating a method for guiding a surgeon to an SOI in an organ, based on an example aspect;

[0020] Figure 2B This is a flowchart illustrating a method for guiding a surgeon to an SOI in an organ, based on another example.

[0021] Figure 2C This is a flowchart illustrating a method for guiding a surgeon to an SOI in an organ, according to yet another example.

[0022] Figure 3A This is a schematic diagram showing preoperative images of the organ pathway and SOI based on an example.

[0023] Figure 3B This is a schematic diagram illustrating the deployment of positioning sensors in the channels or organs of an organ, based on specific examples.

[0024] Figure 3C It is shown that... Figure 3B A schematic diagram of the reconstructed channel corresponding to the channel, where Figure 3A SOI positioning relative to the channel;

[0025] Figure 3D It shows an image superimposed on the relevant organ based on the example aspect. Figure 3C A schematic diagram of the reconstructed channel and SOI;

[0026] Figure 3E It shows an image superimposed on the relevant organ based on another instance. Figure 3C A schematic diagram of SOI;

[0027] Figure 4 This is a schematic diagram illustrating a system for guiding a surgeon in performing surgery, based on an example.

[0028] Figure 5 This is a flowchart illustrating the intraoperative guidance method based on case studies; and

[0029] Figure 6 This is a flowchart illustrating an intraoperative guidance method based on another example. Detailed Implementation

[0030] The following description provides various details regarding examples. However, this description is not intended to limit the scope of the claims, but rather to explain various principles and their practice.

[0031] The benefits of using a camera are limited when the structure of interest (SOI) is inside an organ, as it cannot guide or show the surgeon the location of the SOI. For example, laparoscopy enables surgery on body organs (e.g., the lungs) by allowing observation of the organ from a distance (e.g., millimeters or centimeters). If the growth to be removed is located inside a body organ, where it cannot be accessed through anatomical passages (e.g., airways, blood vessels, etc.) or through natural openings, laparoscopic instruments can penetrate the organ (e.g., the lungs) to reach the desired location inside the lung. However, a camera may not be able to follow into the lung because it can be blocked by blood, fat, tissue, etc. Therefore, cutting into and through the lung, for example, in such cases (e.g., when the surgeon does not know what she is cutting below the surface of the lung), could lead to a trial-and-error surgical procedure, endangering healthy or sensitive tissues or critical structures (e.g., nerves, blood vessels, the heart, etc.) and potentially causing significant bleeding. This can be avoided if the surgeon knows the exact location of the portion of the lung to be operated on and / or the exact location of critical structures adjacent to that portion of the lung.

[0032] Therefore, it would be beneficial to have a surgical system and method that would enable surgeons to operate on SOIs inside body organs using laparoscopic surgery (e.g., operating on a portion of the lung within the lung), while allowing surgeons to know (e.g., see) the exact location of the SOI during the procedure, while minimizing the risk of damaging other nearby anatomical structures (e.g., healthy or sensitive anatomical structures). As used herein, the terms lung, airway, and diseased lung portion (DLP) are examples of organ, passageway, and diseased organ portion (DOP), respectively. DLP and DOP are example anatomical SOIs.

[0033] For example, in laparoscopic surgery, a camera only captures what is in front of it. Therefore, the user may not see the SOI (Site of Interest) in the video image captured by the camera, as the SOI may be located inside an organ while the camera may be outside the organ. Additionally, the user may not see critical structures (e.g., other organs or blood vessels) that may be nearby in the video image. The systems and methods of this disclosure detect or determine the location of unseen SOIs (e.g., targets and / or critical structures) relative to two-dimensional video images captured by a camera and add information about the unseen SOIs to the video images captured in real time. Information about the unseen SOIs can be added to the video images by combining the information with the video images (e.g., enhancement), or by overlaying, covering, or replacing the video images with information or data indicating the location of the SOI. As used herein, the term "capture image" can include the process of receiving, acquiring, or recording images, or obtaining image data (e.g., frames of a video image) from an image sensor of an electronic imaging device (e.g., a camera).

[0034] To add information about unseen SOIs to video images, the position of the SOI relative to the camera needs to be determined in real time. The system and method of this disclosure use preoperative images of body structures (e.g., airways in the lungs) and the SOI to determine the position of the camera relative to the body structures and the position of the SOI relative to the body. The position of the SOI is then determined via the body structures relative to the camera.

[0035] The orientation of the body structure, as well as the orientation and orientation of the camera, can be determined by placing positioning or location sensors within and / or on the body structure and camera. For example, an EM sensor can be placed in and anchored to the airway in the lungs via electromagnetic navigation bronchoscopy (ENB). Therefore, the airway structure can be constructed in real time, and the camera's position relative to the constructed airway can be determined in real time. The position of the SOI relative to the body structure (e.g., the airway in the lungs) is also determined. This can be performed using preoperative 3D images that capture the entire area, such as a CT scan. The position of the SOI relative to the body structure can be determined from the preoperative 3D images. The constructed body structure is then aligned with the preoperative 3D images to determine the position of the SOI relative to the constructed body structure.

[0036] In some respects, it is assumed that the SOI moves along with the body structure. Therefore, the SOI and the body structure may need to be positioned close to or adjacent to each other (e.g., within the same organ or anatomical region, such as the lungs). In other respects, the SOI and the body structure may be located in areas where they do not move together. In those respects, models or other methods can be used to determine the movement of the SOI relative to the body structure. In all respects, the systems and methods of this disclosure can be applied to imaging techniques other than cameras (e.g., spectral cameras) to allow the display of body structures that might be invisible using other imaging techniques.

[0037] Figure 1 A simplified diagram illustrating the method of this disclosure is shown. The purpose of this disclosure is to overlay (e.g., via registration 10) images representing one or more SOIs (e.g., SOI of organ 5) at least partially residing in one or more organs at their correct location relative to an intraoperative image (e.g., a camera image of organ 15) to guide the surgeon as they cut open one or more organs to reach one or more hidden or partially hidden SOIs or to reach areas adjacent to one or more hidden or partially hidden SOIs. However, because one or more SOIs are inside or partially inside one or more organs to be operated on (hidden or partially hidden within one or more organs), one or more SOIs are not visible to the surgeon and the camera. The camera captures or photographs red-green-blue (RGB) images or pictures of the organ from a distance. According to this disclosure, this problem is solved by using the organ's inherent anatomical access system (e.g., access system 25) as a "registration intermediary." (The following...) Figure 1 The description pertains to one SOI, but if there are more than one SOI, the registration process described herein will be applied to all SOIs; that is, each SOI can be processed in the same way.

[0038] The process of overlaying (e.g., via registration 10) an image object representing the SOI of organ 5 in the correct position relative to the intraoperative image 15 of the organ includes a stage of registering the SOI image object (e.g., via registration 20) to a preoperative image of the organ's inherent anatomical access system 25, and another stage of registering the intraoperative image 15 of the organ (e.g., via registration 30) to the same organ's inherent access system 15. Since the organ's access system 15 is located inside or near the organ, it can be hidden by the organ. Therefore, the organ is invisible to the surgeon and the imaging device (e.g., a camera, such as an RGB camera).

[0039] According to this disclosure, an organ's passage system can be synthesized, reconstructed, or reproduced, for example, by distributing positioning sensors along the length of one or more passages of an organ (e.g., by inserting a catheter having multiple positioning sensors distributed along all or part of the catheter's length) and processing positioning information obtained from or using the positioning sensors. In other aspects, an organ's passage system can be reconstructed by moving a catheter with positioning sensors through one or more passages of an organ, obtaining positioning information at multiple time points as the catheter moves through one or more passages of the organ, and then processing the positioning information. The positioning sensors may be or may include transducers configured to sense one or more positioning signals originating from another device or system and output position signals or position data, thereby determining the position of the positioning sensors. The positioning sensors may be configured to transmit positioning signals to an external device or system so that the external device or system can determine the position of the positioning sensors. A passage or airway including at least one positioning sensor is referred to herein as a "sensing passage" or a "sensing airway," respectively.

[0040] The actual organ's passage / airway undergoing the passage / airway reconstruction process and the actual organ's passage / airway system are referred to herein as the "actual passage / airway" and the "actual passage / airway system," respectively. Since the spatial, three-dimensional (3D) size, shape, and scale of the reconstructed passage / airway in the passage / airway system are determined using positional information obtained from positioning sensors distributed within the actual passage / airway, the reconstructed or estimated passage / airway is similar in size, shape, and scale to the actual passage / airway. Therefore, the registration between the SOI and the reconstructed passage / airway is equivalent to or similar to the registration between the SOI and the actual passage or airway, and the registration between intraoperative images and the reconstructed passage / airway is equivalent to or similar to the registration between intraoperative images and the actual passage / airway.

[0041] SOI-to-organ pathway registration 20 and intraoperative image-to-organ pathway registration 30 are ultimately used to perform registration 10, which is between the object image of the SOI and the intraoperative image. The concept behind the three-step registration process is that if the first spatial relationship between the SOI and the body pathway and the second spatial relationship between the body pathway and the intraoperative image are known (e.g., measurable), the third spatial relationship between the SOI and the intraoperative image can be obtained, inferred, derived, and / or calculated from the first and second spatial relationships.

[0042] According to an example aspect of this disclosure, a method for guiding a surgeon to an SOI inside an organ during surgery may include determining (e.g., via a controller) (1) one or more channels of the organ and (2) the location of one or more SOIs associated with the organ (e.g., within or on the organ), these SOIs being included in or displayed in one or more preoperative medical diagnostic imaging (MDI) images of the organ. The method may further include determining (e.g., via a controller) the location of one or more SOIs relative to one or more channels of the organ based on the location of one or more SOIs shown in one or more preoperative MDI images and the location of one or more channels.

[0043] The method may further include capturing or photographing a video stream via a camera, the video stream comprising a series of subsequent video images (e.g., intraoperative images) of an organ or a portion of an organ, and simultaneously, via a controller, co-aligning each video image with a channel of the organ; via the controller, determining the position of the SOI relative to the intraoperative images based on the position of the SOI relative to the channel of the organ and the alignment between the video images and one or more preoperative MDI images showing the channel of the organ; and via the controller, generating and overlaying an image object representing the SOI onto the video images at the determined position of the SOI relative to the intraoperative images. Overlaying the image object representing the SOI onto the video images at the determined position may include showing the video images with the SOI image object overlaid (e.g., superimposed) on the images at the determined position, enabling the surgeon to know where the relative position of the organ's SOI is in the intraoperative images.

[0044] In some aspects, determining the position of the SOI relative to the organ channel may include reconstructing the organ channel and aligning the resulting reconstructed channel with a video image channel. The organ channel reconstruction may be based on positioning signals that can be exchanged between the positioning system and multiple positioning sensors distributed along the length of one or more organ channels.

[0045] Aligning intraoperative images and reconstructed organ access can include exchanging positioning signals between a positioning system and at least one positioning sensor located within a camera (e.g., at the tip or distal end of the camera). Positioning two positioning sensors within the camera allows the controller to determine the camera's line of sight, such as the spatial angle relative to the body organ being operated on and relative to the intended location of the SOI. The spatial angular characteristics of the camera can enhance the surgeon's orientation at the surgical site. The organ can be, for example, the left or right lung, and the organ access can be, for example, the pulmonary airway.

[0046] In some aspects, determining the position of the SOI relative to the organ pathway may include determining the spatial relationship between the pathway in the MDI image and the reconstructed pathway, and mapping the position of the SOI in the MDI image relative to the reconstructed pathway system to a similar or identical location. Determining the position of the SOI relative to the pathway system may include obtaining positioning information related to the position of the positioning sensors contained in the pathway system, and identifying the spatial relationship between the pathway in the preoperative image (in the MDI image) and the pathway containing the positioning sensors.

[0047] Figure 2A A method for guiding a surgeon to an organ's SOI, based on an example aspect, is shown. Figure 2A Methods relating to the lungs and pulmonary airways are described; however, the same or similar methods can be applied to any other organ, such as the heart, liver, stomach, cardiovascular system, etc. Similar methods can be used to guide surgeons during any surgical procedure, wherein positioning sensors can be fixedly deployed in the organ's access system during said procedure.

[0048] The method may include two main phases: (1) a preoperative phase and (2) an intraoperative phase. In the preoperative phase, for example in step 201, images of the patient's lungs may be captured using an MDI system to capture images showing the lungs, the airways (or at least some of the airways), and additional SOIs, which may be candidate anatomical structures for surgery or anatomical structures that must not be damaged during surgery. One or more of the following steps (e.g., steps 203 and 205) may also be performed in the preoperative phase. After capturing images of the patient's lungs, the lungs, airways, and SOIs can be identified in the images. For example, the lungs, airways, and SOIs can be labeled in the patient's lung images, for example, by a clinician or by image recognition or segmentation algorithms.

[0049] In step 203, the location of the SOI (e.g., diseased portion of the lung (DLP)) and airways in the lung can be determined (e.g., via a controller), such as by a medical diagnostic imaging (MDI) system. For example, a reference frame (e.g., a Cartesian coordinate system) can be used to locate the SOI and airways, such that the location of the SOI relative to the airways in the MDI image can be determined, for example, by a computer system, as described below with respect to step 205. A set of lung images can be acquired via an MDI system (e.g., via a computed tomography (CT) system) before or during preparation for surgery. The type of MDI system used can be selected such that the images captured by the MDI system can clearly distinguish (e.g., visually) (1) the SOI (e.g., a malignant or benign tumor) that needs to be removed (e.g., ablated), and (2) at least some airways or other anatomical passages in the lung. In practice, at least one airway in the lung can be distinguishable, for example, by being visible or detectable in the MDI image, in relation to reference markers. One or more identifiable airways or other anatomical pathways in the lungs can be used as a basis for indicating the estimated location of the SOI on intraoperative images of the lungs (e.g., by overlaying image objects onto intraoperative images of the lungs) to provide guidance to the surgeon during the surgical procedure.

[0050] In step 209, a reconstructed airway is generated based on EM sensor information received or obtained from, for example, one or more positioning sensors described herein. In step 205, the spatial location of the SOI relative to the reconstructed or synthesized airway is determined. The spatial location of the SOI relative to the reconstructed airway can be found using the location of the SOI in one or more MDI images relative to the location of the airway in one or more MDI images.

[0051] The spatial location of the pulmonary airway system can be determined by: distributing positioning sensors along the length of the pulmonary airways; transmitting positioning signals (e.g., electromagnetic signals) between the positioning sensors, which are fixedly positioned along the length of the airways, and an external positioning system (e.g., an electromagnetic positioning system); and determining the spatial location, shape, and size (e.g., length) of these airways based on or using the position information obtained from the positioning sensors. The shape of each pulmonary airway containing the positioning sensors can then be reconstructed or reproduced based on or according to the spatial location information obtained from the positioning sensors (e.g., mathematically, and optionally, visually). Alternatively, the positioning sensors can be swept or moved through the pulmonary airways to obtain spatial location information, which can be used to reconstruct or reproduce the shape of the pulmonary airways.

[0052] Some features of the internal anatomy or structure of the lungs are common medical knowledge, such as airway diameter (e.g., for a specific size and time of day in a patient's respiratory cycle) and the main structure of the airway tree. This knowledge can help in the anatomical identification of airways in MDI images and in relating actual airways to corresponding reconstructed airways in the organ's airway system. After anatomically identifying each reconstructed airway (or after identifying a reconstructed airway system), it can be correlated, for example, with the corresponding airway or airway system shown in the MDI image based on physical shape and / or anatomical function. This correlation process can result in alignment between the reconstructed airway or reconstructed airway system and the corresponding airway or airway system shown in the MDI image. Spatially aligning MDI and intraoperative images can include spatially matching channels appearing in the MDI image with those in the intraoperative image based on the location of a positioning sensor. That is, when positioning sensors are placed in the channel system, their location can be adapted in a way that adapts to the structurally non-repetitive three-dimensional anatomy of the anatomical organ's channel system. Therefore, medical knowledge of the anatomical structure of the channel system with anatomical organs enables the computer system to know which channels of the channel system contain positioning sensors and to align the image content of the MDI image with the image content of the image.

[0053] Once the reconstructed airway / channel or airway / channel system is spatially aligned with the airway or airway system in the MDI image, the spatial position of the SOI relative to the reconstructed airway (within the reconstructed airway system) can be determined in step 205 based on or according to the position of the SOI in the MDI image relative to the corresponding airway or airway system in the MDI image. In other words, the orientation of the SOI relative to the reconstructed airway should match or correspond to the orientation of the SOI relative to the airway in the MDI image. Since each reconstructed airway spatially represents a specific airway, determining the position of the SOI relative to the reconstructed airway is similar to determining the position of the SOI relative to the relevant airway, as seen in the pre-procedure phase.

[0054] In step 207, an endoscopic camera, which may include one or more positioning sensors, captures images, and the two-dimensional (2D) position (or three-dimensional (3D) position) of the camera can be detected, for example, by a positioning system, when capturing images of the surgical site. During the surgical procedure, the camera outputs, for example, a stream of images of the lungs. One or more positioning sensors of the camera may be adhesively or removably attached to, for example, the distal end, portion, or tip of an endoscope.

[0055] In step 211, the position of the camera and the reconstructed airway are aligned. Since the position of the SOI and the orientation of the camera (and therefore the relative orientation of the image captured by the camera) are both determined relative to the same reconstructed airway (in steps 205 and 211, respectively), in step 213, the position of the SOI can also be determined relative to the image captured by the camera.

[0056] In step 215, the controller or computer system performing the method may display, for example, an image captured by a camera on a computer screen or monitor, wherein an image object that can graphically represent the SOI is displayed at a location corresponding to or consistent with the location of the SOI in the MDI image of the lung.

[0057] Figure 2B The example method shown performs two registrations: (1) “MDI image to airway” registration (performed in steps 202 and 204), and (2) “intraoperative image to airway” registration (performed in steps 208 and 212). Steps 204 and 206 are performed sequentially, and steps 208 and 212 are performed sequentially. Steps 204 and 206, as well as steps 208 and 212, can typically be performed in parallel.

[0058] Figure 2B A method for guiding a surgeon during thoracic lung surgery is illustrated, based on examples. The method comprises two main phases: (1) a preoperative phase and (2) an intraoperative phase. In the preoperative phase, for example in step 202, images of the patient's affected lung are acquired or captured using an MDI system to capture images of the airways and SOIs of some or all of the lungs. Moreover, for example, in step 202 or at any other suitable step (e.g., when such information is required for further processing or calculation), the location of the SOI relative to the airways imaged by the MDI is determined.

[0059] In step 204, the spatial three-dimensional relationship between the reconstructed airway and the MDI image of the airway (i.e., the airway visible or detectable in the lungs as seen in the MDI image) is determined. The step of spatially determining the relationship between the reconstructed airway and the airway in the MDI image may include, for example, aligning the two types of airway information by aligning the reconstructed airway and the corresponding MDI image of the airway.

[0060] Once the reconstructed or synthesized airway or airway structure is aligned with the airway or airway structure in the MDI image, in step 206, the spatial position of the SOI relative to the reconstructed airway or airway structure is determined based on or according to the MDI image. That is, the spatial position of the SOI relative to the reconstructed airway or airway structure is determined based on or according to the position of the SOI in the MDI image relative to the corresponding, matching, or associated airway or airway structure in the MDI image. In other words, the orientation of the SOI relative to the reproduced airway should match or substantially match the orientation of the SOI relative to the airway or airway structure in the MDI image.

[0061] In step 208, images are captured by an endoscopic camera that may include one or more positioning sensors, and the two-dimensional (2D) or three-dimensional (3D) position of the camera is detected, for example, by a positioning system, while images are captured by the endoscopic camera. During the surgical procedure, the camera may output a stream of images of the lungs (e.g., video images). The camera's positioning sensors may be adhesively or removably attached to the distal end, portion, or tip of the endoscope carrying the camera.

[0062] In step 212, the 2D or 3D position of the camera relative to the reconstructed airway is determined. Since the position of the SOI and the orientation of the camera (and therefore the image captured by the camera) are determined relative to the same reconstructed airway (in steps 206 and 212, respectively), the position of the SOI relative to the image captured by the camera can also be determined in step 214. The position of the camera relative to the reconstructed airway can be monitored while the camera is running and as long as the camera is running and the surgical procedure is in progress. In step 216, an image object representing the SOI (and optionally an image of the airway system) can be overlaid on the image at the corresponding position relative to the image to produce a complex image, which can be displayed, for example, on a computer screen or a television.

[0063] Figure 2B The method shown performs two registrations: (1) MDI image-airway registration (performed in steps 202 and 204), and (2) intraoperative image-airway registration (performed in steps 208 and 212). Steps 204 and 206 are performed sequentially, as are steps 208 and 212. Steps 204 and 206 can typically be performed in parallel with steps 208 and 212.

[0064] Figure 2C A method for guiding a surgeon to the surgical site during lung surgery is illustrated, based on an example. In step 210, for example, prior to performing surgery, a set of lung images, including SOIs, is acquired via an MDI system.

[0065] Step 210 may also include determining the spatial 3D location (e.g., {x,y,z} location) of the SOI relative to an airway or passage in the lung that is discernible in the MDI images from the MDI image set. During organ scanning, the MDI system may capture multiple images of the organ, each depicting a different “slice” of the organ, and three-dimensional information (e.g., a 3D model) of the organ can be obtained from a series of subsequent slices. While each image slice shows a planar image of the organ (e.g., in the {X,Y} plane), the image slice set (e.g., CT image slices) adds a third dimension to the organ (e.g., in the Z direction to maintain consistency relative to the instance {X,Y} plane). Therefore, the shape, size, and spatial orientation of each object discernible (e.g., visible) in the image slice set can be determined (e.g., calculated) relative to the location, shape, size, and spatial orientation of each other object discernible (e.g., visible) in the image slice set.

[0066] In step 220, a set of positioning sensors (e.g., magnetic field sensing transducers) are placed or distributed in one or more airways within the lung. The set of positioning sensors is placed or distributed in one or more lung airways that are discernible (e.g., visually) in the MDI image. This placement of the positioning sensors enables, for example, a controller to register the MDI image content, specifically the SOI and the lung airways where the positioning sensors are located during surgery, to airways sensed in real-time during the surgical procedure, for example, by a positioning system. The longitudinal distribution of positioning sensors within the lung airways, which are visually discernible in the MDI image and sensed by the positioning system during the surgical procedure, enables, for example, a controller or computer system to reconstruct (image-by-image) the airways accommodating the positioning sensors and optionally generate a composite image showing some or all of the reconstructed airways, among other image objects.

[0067] Synthetic images (which can be two-dimensional (2D) or three-dimensional (3D) images) are computer-generated images that may include image objects, or multiple image objects, graphically and morphologically representing a reconstructed airway containing a positioning sensor. Synthetic images may also include image objects graphically and morphologically representing a SOI (Self-Induced Infrared) space. Synthetic images may include image objects graphically and morphologically representing both the reconstructed airway and the SOI. Reconstruction of the lung airway may include generating image objects graphically representing the sensed lung airway via a controller or computer system.

[0068] In some respects, unsensored airways will not be reconstructed; therefore, such airways can be excluded from the synthetic or reconstructed images. However, as discussed herein, at least the overall anatomy and function of the airway system are common medical knowledge. By using this medical knowledge, for example, the spatial orientation of unsensored airways can be interpolated from the reconstructed airways. When used in conjunction with the reconstructed airways, interpolation can improve the accuracy of determining the position of the SOI relative to the reconstructed airways and thus relative to the intraoperative images.

[0069] Positioning sensors can be placed in some lung airways (e.g., in the patient's bronchial tree), for example, through natural body orifices or via catheters inserted into the lung airways (e.g., airways in the lung segment or lobe to be operated on). Multiple positioning sensors can be longitudinally positioned in or on a catheter. One or more catheters carrying one or more positioning sensors can be inserted into one or more lung airways to facilitate the reconstruction of the sensed lung airways. Throughout the surgical procedure, one or more catheters carrying one or more positioning sensors can be inserted into and attached to the lung airways. Attachment of the catheters to the airways can be performed, for example, by using a balloon, clips, or any other suitable attachment mechanism.

[0070] One or more catheters, including positioning sensors, can be inserted into the lung airways through, for example, the working channel of a bronchoscope. Catheters including positioning sensors can optionally be inserted into the lung airways through the working channel of a double-lumen endotracheal tube (DLT). A DLT is a type of endotracheal tube used for endotracheal intubation during thoracic surgery and other medical conditions to achieve selective unilateral ventilation of the right or left lung. A DLT consists of two small-lumen endotracheal tubes of unequal length, fixed side-by-side. The shorter tube terminates in the trachea, while the longer tube is placed in either the left or right bronchus to provide ventilation to the left or right lung, respectively. Catheters can be inserted into the lung airway system at once. Alternatively, multiple catheters can be inserted into the lungs simultaneously through the same DLT tube (e.g., a lead channel), and each catheter can then be navigated to a different airway.

[0071] Electromagnetic navigation bronchoscopy (ENB) can be used to guide a catheter to a specific airway in the lungs. This minimally invasive procedure allows access to hard-to-reach areas of the lungs and aids in the diagnosis of lung diseases. Regardless of how the catheter is inserted into any airway or channel, it can be anchored to the airway or channel using any suitable method, such as a balloon, hook, or anchoring arm, once it is correctly positioned.

[0072] In step 230, the 3D position of each sensor can be detected by an external or in vitro positioning system (e.g., by using {X,Y,Z} coordinates). Spatial details or parameters, such as the 3D shape, size, position, and orientation of the lung airway for each sensor, can be determined, for example, based on the 3D position of the sensor, and each specific lung airway can be reconstructed based on the spatial details or parameters associated with a specific lung airway.

[0073] Because the reconstructed airways are obtained using positioning sensors longitudinally distributed throughout the airways, which have now or previously been imaged by the MDI system, the reconstructed airways and the corresponding MDI-imagined airways are morphologically similar. For example, they may be similar in shape, proportion, relative orientation, angular relationships, etc. The similarity between the reconstructed airways and the MDI-imagined airways is the result of scanning the same physical airway system with the MDI system (e.g., preoperatively) and reconstructed using positional information obtained from the positioning sensors or using positional information obtained from the positioning sensors.

[0074] In step 240, the reconstructed airway system and the MDI image (e.g., a CT image) including at least a portion of the airway system and the SOI can be spatially co-aligned, for example mathematically and / or visually, so that the controller or computer system can use the same coordinate system for both types of airways (i.e., the reconstructed airways and the airways shown in the MDI image). Co-alignment of the reconstructed airways and the associated airways imaged by the MDI system can include the steps of adjusting the size of one type of airway relative to the other (e.g., reducing the size of the reconstructed airway relative to the airways imaged by the MDI system) and spatially aligning the two types of airways. In all respects, the spatial alignment of the two types of airways does not necessarily have to be done graphically; this can be done mathematically. Co-aligning the reconstructed airways and the associated airways imaged by the MDI system, or determining the spatial relationship between the two types of airways, is advantageous because it allows the controller to generate and position (e.g., visually) an image object representing the SOI at an appropriate location relative to the reconstructed airway system, corresponding to the position of the SOI relative to the airway system shown in the MDI image.

[0075] Preoperative MDI images show the airways of the lungs in a certain static state. However, airways reconstructed in real time during the surgical procedure may be deformed due to, for example, patient and / or lung movement. Additionally, while the airways of a patient can be imaged by an MDI system when lung function is normal (e.g., when the patient is awake), the reconstructed airways during the surgical procedure, when the patient is anesthetized and the lungs collapse during the medical procedure, result in some or all of the airways being deformed. Therefore, some differences or mismatches may exist between the airways imaged in the MDI and the deformed reconstructed airways. However, since both types of airways originate from the same physical anatomical structure (e.g., from the same tracheal tree), sufficient similarity (e.g., in shape, size, proportion, angle, and / or length) can exist between the two types of airways, allowing a controller or computer system to co-align the airways shown in the MDI images with the deformed reconstructed airways.

[0076] Steps 220-240 embody the first registration stage (e.g., MDI image-reconstructed image registration), where the airway image captured by the MDI system in step 210 and the reconstructed airway obtained in step 230 are co-registered. The result of this registration process is that the location of the SOI can be spatially determined relative to the reconstructed airway. The SOI may need to be cut off or protected from accidental cutting or other damage. That is, knowing the location of the SOI relative to the airway in the MDI image, and the spatial relationship between the airway imaged by the MDI system and the reconstructed airway, the location of the SOI can be determined relative to the reconstructed airway in step 250 (e.g., via a controller or computer system). When determining the location of the SOI relative to the airway imaged by the MDI system from one or more MDI images, the location of the SOI in the MDI image relative to the reconstructed airway is determined by using the spatial correlation between the airway shown in the MDI image and the reconstructed airway.

[0077] In step 260, the two-dimensional (2D) or three-dimensional (3D) position of the endoscopic camera, including one or more positioning sensors, is detected. During the procedure, the camera outputs a stream of lung images. One or more positioning sensors of the camera may be adhesively or removably attached to, for example, the distal end, portion, or tip of the endoscope carrying the optical fiber of the camera.

[0078] In step 270, the camera's position is used to determine the 2D or 3D relationship between the camera and the reconstructed airway. Since the location of the diseased lung portion was determined relative to the reconstructed airway in step 250, and the camera's orientation (and therefore the image captured by the camera) was determined relative to the same reconstructed airway in step 270, the location of the SOI can also be determined relative to the image captured by the camera in step 280. The camera's position relative to the reconstructed airway can be monitored as long as the camera is running and the surgical procedure is in progress.

[0079] Steps 260 and 270 embody the second registration stage (camera image-synthetic image registration), wherein (1) the position of the camera detected in step 260 and (2) the airway reconstructed in step 230 are co-registered. As a result of the camera image-synthetic image registration process, the spatial relationship between the camera image and the reconstructed airway can also be determined and displayed on a display screen (e.g., on a thoracoscopy view on the display screen).

[0080] In step 290, an image of the lung undergoing the surgical procedure can be displayed on a computer screen, for example, by overlaying (e.g., superimposing) the SOI and optionally the reconstructed airway onto the camera image. The surgeon can use the reconstructed airway as, for example, a reference / guiding frame to improve the precision and efficiency of the surgery. Thus, for example, the surgeon can view a computer monitor or screen and see an image object representing the position of the SOI relative to the camera image in real time (essentially at any time during the surgery), eliminating the need for the surgeon to guess the exact location of the SOI.

[0081] When a surgeon moves a camera (e.g., laterally) from one point at the surgical site to another, changes the distance of the camera from the surgical site, or changes the angle of the camera's line of sight relative to the surgical scene, the external positioning system can continuously sense the new position, angle, and / or distance of the camera and can visually readjust (e.g., rescale, distort, etc.) subsequent camera images relative to the reconstructed airway based on the camera's changed field of view (FOV), position, angle, and distance, in order to maintain alignment, relative relationship, and scale between the camera images and the reconstructed airway.

[0082] For example, if the camera is moved a certain amount in a certain direction (e.g., five centimeters to the left), the reconstructed airways will be repositioned in the opposite direction in the camera image (e.g., they will be translated) by that amount (e.g., five centimeters to the right). In another instance, if the camera is moved completely away from the surgical site, so that the FOV does not contain the surgical site, the reconstructed airways will completely disappear from the camera image. And if the camera is moved so that the camera's FOV contains a portion of the surgical site, depending on the portion of the surgical site captured by the camera, some reconstructed airways may be included in the camera image, while others may be excluded from the camera image.

[0083] Since the position of the SOI is maintained relative to the reconstructed airway throughout the entire surgical procedure (because the position of the SOI in the MDI system imaging is fixed relative to the airway (within a certain range), regardless of the camera position), if the position of the image object is determined to be outside the camera's FOV, then the image object representing the SOI will not appear in the image.

[0084] When displaying a diseased lung portion (and optionally reconstructed airways) superimposed (e.g., overlaid) on a camera image, a computer-generated object representing the diseased lung portion can be produced such that it is visually, graphically, or otherwise highlighted or enhanced to make the location, orientation, shape, and / or size of the diseased lung portion more or more apparent relative to the video content of the camera image. For example, the computer-generated object representing the location, shape, and size of the diseased lung portion may include a visually distinct boundary line, or it may have one or more colors that are more apparent relative to one or more colors of the vicinity or background.

[0085] Figure 3A An example preoperative MDI image of a lung airway system 300 according to the present disclosure is schematically shown. For simplicity, the lung airway system 300 includes a simplified lung structure, such as a bronchial tree including airways, and a diseased lung portion (DLP) 305. Figure 3A Corresponding to Figure 2C Step 210 involves capturing one or more MDI images, and the relative position (e.g., coordinates {x1, y1, z1}) of the DLP 305 in the lung airway system within the MDI image 300 can be determined, for example, by using the electro-optical characteristics of the MDI system (e.g., image signal gain, pixel output signal, scaling level FOV, etc.) during the capture of one or more images. For instance, the lung structure includes four connection points, denoted as J1, J2, J3, and J4, and eight branches, denoted as B1, B2, B3, B4, B5, B6, B7, and B8.

[0086] Figure 3BA pulmonary airway system 301 having catheters placed in several airways is schematically shown according to the present disclosure. The pulmonary airway system 301 (including the actual airways) corresponds to the pulmonary airway system (including the imaged airways) imaged in the MDI image 300. Figure 3B Corresponding to Figure 2C In step 220, three catheters 310-330 are placed in some of the lung airways of the pulmonary airway system 301. The three catheters include a left-hand side catheter 310, a middle catheter 320, and a right-hand side catheter 330. Figure 3A Each airway branch in the imaging is in Figure 3B Each of these corresponds to a specific airway branch. For example, Figure 3A The airway branch B1 in the imaging Figure 3B It contains the corresponding actual airway branch B1'. Figure 3A The airway branch B2 in the imaging Figure 3B Each of these has a corresponding actual airway branch B2', and so on.

[0087] Each catheter may have one or more positioning sensors disposed thereon or in it. For example, catheter 310 includes six positioning sensors S1, S2, S3, S4, S5 and S6; catheter 320 includes three positioning sensors S21, S22 and S23; and catheter 330 includes three positioning sensors S31, S32 and S33.

[0088] Figure 3C The image 302, which is a composite or reconstructed image, is schematically shown, depicting a scene similar to... Figure 3B The airways in the actual lung airway system 301 that correspond to, are related to, or are derived from the synthetic or reconstructed airways. Figure 3C Corresponding to Figure 2C Steps 230-250 in the process involve the registration of the MDI image to the synthetic image and the registration of the DLP to the synthetic airway.

[0089] The relative positions of some or all of the positioning sensors can be determined using an external positioning system. In step 230, the positioning information from the positioning sensors can be used to synthesize or reconstruct the sensed airway.

[0090] Figure 3B Each airway branch in the lung airway system 301, including the duct carrying sensors, has a corresponding synthetic or reconstructed airway branch in the synthetic image 302. For example, Figure 3B airway branch B5' in Figure 3C The corresponding synthetic or reconstructed airway branch B5” is present in the middle. Figure 3B airway branch B7' in Figure 3C The corresponding synthetic or reconstructed airway branch B7” is present in the middle. Figure 3Bairway branch B8' in Figure 3C The corresponding synthetic or reconstructed airway branch B8 is present.

[0091] Synthesized or reconstructed catheters 310', 320', and 330' are graphically represented, respectively. Furthermore, since the catheters exhibit the shape of the airway containing them, the synthesized or reconstructed catheters 310', 320', and 330' also exhibit the shape of the airway. For example, the 3D relationship between the synthesized image 302 and the MDI image 300 can be determined according to step 240 using the synthesized or reconstructed airway branches of the synthesized image 302 and the airway branches of the MDI image 300.

[0092] After determining (e.g., mathematically) the spatial relationship between the composite image 302 and the MDI image 300, or after images 300 and 302 are spatially co-aligned (e.g., mathematically), a reconstructed position {x1′,y1′,z1′} is found in the composite image 302 (at the location of the composite or reconstructed DLP 305'), which corresponds to or coincides with the position {x1,y1,z1} of DLP 305 in the MDI image 300. The position of the reconstructed DLP 305' may be consistent with the position of DLP 305, or they may be slightly different. The position of the reconstructed DLP 305' is spatially related to the composite or reconstructed airway in the composite image 302 in the same way or similarly as the position of DLP 305 in the MDI image 300 is spatially related to the airway in the MDI image 300.

[0093] Figure 3D A camera image 340 is shown based on an example. Figure 3D Corresponding to Figure 2C Steps 260-280 in the process involve camera image-synthetic image registration and DLP second registration (DLP-to-camera image registration). Camera image 340 symbolically represents an image of the lung being operated on, captured by a camera at a certain distance from the lung.

[0094] As described herein, the camera may include a positioning sensor that enables, for example, a controller or computer system to determine the 3D position of the camera relative to the duct (or duct positioning sensor). Knowing the 3D position of the camera relative to the duct (e.g., position 350), and optionally knowing the electro-optical parameters of the camera, the controller or computer system may display a camera image 340, in which a synthesized or reconstructed airway 360' (shown in dashed lines) is superimposed on the camera image 340. The controller or computer system may scale, resize, or otherwise adjust or manipulate the camera image 340 and / or the synthesized or reconstructed airway 360' to maintain their proportions when displayed, for example, on a computer screen. Figure 3D The synthetic or reconstructed airway 360' in the middle can be Figure 3C The synthesized or reconstructed airway 360 is a manipulated version (e.g., a resized version, a resized version, etc.).

[0095] refer to Figure 3C By knowing the reconstructed position {x1′, y1′, z1′} of the reconstructed DLP 305' relative to the synthesized or reconstructed airway 360 in the synthetic image 302, and by performing mathematical operations, the controller or computer system can determine the position of the DLP 305' in the synthetic image 340. Figure 3D The position 370 (position {x1″, y1″, z1″}) of the DLP (in the middle) relative to the manipulated reconstructed airway (i.e., relative to the reconstructed airway 360') maintains all relationships / proportions between the reconstructed DLP 305', the reconstructed airway 360', and the camera image 340. The reconstructed airway 360' does not need to be visually superimposed on the camera image 340, but if the reconstructed airway is superimposed on the camera image 340, it can provide additional visual guidance to the surgeon performing the procedure, for example, in the form of orientation information. Figure 3E The image 340 of the camera at position 370 is shown.

[0096] The position of the DLP 305 relative to camera image 340 is determined in the manner described herein. The shape and size of the DLP 305 may be determined, for example, based on one or more preoperative MDI images (e.g., one or more preoperative CT images), and may be rescaled, resized, realigned, etc., relative to or relative to one or more camera images to maintain the scale and / or realignment of the image content relative to one or more camera images. The scale and / or realignment between one or more camera images and the reproduced or synthesized airway may also be maintained so that, as the surgery progresses, the DLP 305 is correctly (e.g., in the correct orientation) (e.g., visually) superimposed on one or more camera images, thereby guiding the surgeon to the correct location of the DLP 305 in the lung. The guidance method disclosed herein also enables medical personnel to use the camera more effectively when the surgeon makes an incision through the lung to the DLP 305.

[0097] Figure 4 A block diagram of a system 400 for overlaying SOI onto an image is shown, according to an example aspect. System 400 may include a positioning system 410 for determining the location of a sensing channel in a channel system of an organ 462 in a patient 460, wherein one or more positioning sensors are distributed along the length of each sensing channel. System 400 may also include a camera 450 for generating an image stream 456 of the organ 462. Camera 450 may include one or more positioning sensors 454 for positioning camera 450 relative to the channel system via positioning system 410. System 400 may also include a controller 440 for controlling positioning system 410 and a computer system 480 for controlling system 400. Computer system 480 may assume the control tasks of controller 440 and therefore may also function as controller 440, and vice versa; controller 440 may assume the control tasks of computer system 480 and therefore may also function as computer system 480.

[0098] In some respects, controller 440 may be configured to: determine the position of SOI in a preoperative image relative to the channel system in a preoperative image, based on a preoperative image including organ 462, the channel system in organ 462, and SOI of organ 462; determine the position of SOI relative to the reconstructed channel based on the position of SOI relative to the channel system in the preoperative image; capture an image of the organ using a camera; co-align the channel system with the captured image; determine the position of SOI relative to the captured image based on the position of SOI relative to the channel system and the alignment between the channel system and the captured image; and overlay an image object representing SOI onto the captured image at the determined position of SOI relative to the captured image.

[0099] The controller 440 can control the operation of the electromagnetic signal transmitter 420. For example, the controller 440 can time the transmission of the electromagnetic signal 422. The controller 440 can also control the operation of the electromagnetic positioning signal receiver 430. For example, the controller 440 can time the reception of the electromagnetic signal 432 and the transmission of the electromagnetic signal 422.

[0100] The controller 440 can determine the position of the SOI relative to the channel system by synthesizing channels in the channel system and by determining the spatial relationship between the resulting synthesized or reconstructed channels and the channel system in the preoperative image.

[0101] The synthesized organ channel may include exchanging positioning signals (e.g., 422, 432) between the positioning system 410 and multiple positioning sensors distributed along the length of the channel to be synthesized. The channel is located within organ 462. The position of the diseased organ portion (DOP) relative to the channel system may include mapping the relative position of the DOP in preoperative images to the reconstructed channel system. For example, in Figure 3A The position of DOP relative to the channel in the preoperative image, shown at the location of DLP 305, is mapped to... Figure 3C The reconstructed DLP 305' position is shown in the diagram. Aligning the image and organ access system may include exchanging positioning signals between the positioning system 410 and at least one positioning sensor 454, which may be located within or related to the camera 450. The organ 462 may be, for example, a lung lobe, and the organ access may be, for example, a pulmonary airway.

[0102] The positioning system 410 may be an electromagnetic positioning system. The positioning system 410 may include an electromagnetic signal transmitter 420 to transmit electromagnetic signals 422. The electromagnetic signals 422 may be configured such that they can be used to locate a device including a positioning sensor. The positioning sensor may be or include a conductive coil or a set of conductive coils (e.g., three mutually orthogonal coils) and outputs a signal (e.g., a positioning signal) based on the strength of the electromagnetic field, which is sensed in a certain direction and further based on the spatial angle (e.g., orientation) between the plane of the conductive coil and the orientation of the electromagnetic field. The positioning system 410 may also include an electromagnetic positioning signal receiver 430 to receive electromagnetic signals 432 from one or more positioning sensors.

[0103] System 400 may also include a camera 450, which may include a camera and a light source. Camera 450 may also include a rod or tube 452 (e.g., which may form part of an endoscope), through which camera 450 can capture images of patient 460, for example, images of the lungs of a patient suspected of or diagnosed with SOI, such as DOP. An image sensor for camera 450 may be positioned at or near the distal end of tube 452 or at the proximal end of tube 452. A positioning sensor 454 may be positioned at the distal end of tube 452 to enable the determination of the 2D or 3D position of camera 450, and thus the 2D or 3D position of the image captured by camera 450 relative to the airways in the patient's lungs. In other words, positioning sensor 454 enables the registration of the image captured by camera 450 to the airways of patient 460's lungs, and to SOI via the patient's lung airways.

[0104] Electromagnetic signal 422 is intended to be sensed by a positioning sensor, which is inserted through the natural orifice of the patient 460 and into the airway of the patient's lungs. Positioning sensor ( Figure 4 Positioning sensors (not shown) are inserted into the airways of some lungs and can remain there during lung surgery (e.g., by connecting them to the airways). Positioning sensors can be inserted into the airways of the lungs using one or more catheters. One or more positioning sensors can be placed on or within the catheter. The positioning sensors can be configured to sense electromagnetic signals 422 and, depending on the orientation (position and / or orientation) of the positioning sensor relative to the electromagnetic signal transmitter 420, simultaneously transmit (432) one or more corresponding output signals to the positioning signal receiver 430. The positioning sensors in the patient's airway can wirelessly or via a communication cable transmit one or more output signals to the positioning signal receiver 430.

[0105] System 400 may also include a medical diagnostic imaging (MDI) system 470 for preoperatively and non-invasively capturing images of the internal tissues, organs, etc., of patient 460. Computer system 480 may be functionally coupled to (1) a positioning system 410 to receive (412) positioning data therefrom, (2) a camera 450 to receive (456) real-time images therefrom during surgery, and (3) the MDI system 470 to receive (472) images therefrom. Computer system 480 may determine from said images the orientation / location and / or orientation of the SOI relative to the airways in the patient's lungs, and optionally its shape and size. Computer system 480 may be configured to perform any of the methods disclosed herein. For example, computer system 480 may be configured to perform combined... Figure 2AThe methods described in –2C and 3A–3E. The computer system 480 can also be used as a controller 440. For example, the computer system 480 can control the operation of the positioning system 410, thereby replacing the controller 440. The computer system 480 may include or be connected to a display device 482 to display an image of the patient's lungs, wherein the object representing SOI 462 is superimposed on the image at the correct orientation, i.e., at the position corresponding to the orientation of the DLP relative to the airway, for example in… Figure 3D and 3E As shown in the image.

[0106] Throughout the surgical procedure, the position of camera 450 can be monitored continuously or in real time. The camera's position can be determined relative to the reconstructed airway. Determining the camera's orientation relative to the reconstructed airway may be straightforward if the same positioning system is used for: (1) positioning sensors for reconstructing or synthesizing the airway, and (2) positioning sensors for positioning the camera. However, if a different positioning system is used, some coordinate transformation may be required to determine the camera's orientation relative to the reconstructed airway.

[0107] In one aspect of this disclosure, display device 482 can simultaneously display two spatially aligned images in two display areas (DAs) (e.g., two juxtaposed DAs) on display device 482. One display area DA1 can display an MDI image (e.g., as a reference image), the MDI image including a body organ undergoing the registration and surgical procedure described herein. The other display area DA2 can display a real-time image of the surgical site. The real-time image of the surgical site includes images continuously captured by a camera (e.g., by camera 450) and displayed on display area DA2 while the surgical procedure is in progress and as long as the surgical procedure is in progress.

[0108] The imaging perspectives (e.g., angles and distances) of the captured MDI images and camera images may differ. For example, a surgeon may move the camera laterally toward the organ (e.g., when zoomed in) or away from the organ (e.g., when zoomed out) relative to the surgical site, thereby changing the field of view (FOV) and / or imaging angle. In contrast, MDI images are typically preoperative snapshots taken at a single time at certain FOVs, imaging angles, and distances from the patient's body. On one hand, whenever the camera operator (e.g., the surgeon) moves the camera, for example, changing its imaging angle and / or FOV relative to the organ on which surgery is being performed (e.g., the lung), the computer system 480 manipulates the MDI images during the surgical procedure to maintain spatial alignment and scale between the camera images and the MDI images. For example, the computer system 480 rotates, translates, aligns, resizes, and / or zooms in / out on image content, features, and / or objects in the MDI images to follow or track corresponding changes in the image content of the camera images (e.g., changes in the size, position, angle, and / or orientation of the SOI relative to the organ on which surgery is being performed).

[0109] For example, manipulating MDI images in the manner described herein is advantageous because it eliminates the need to overlay a reconstructed airway system of body organs onto the camera image (e.g., Figure 3D As shown in the image, the airway system is already shown in the MDI image. Another benefit of manipulating MDI images is that they contain additional image information that allows surgeons to see not only the airway system, but also various other types of tissue around the SOI.

[0110] Figure 5 This illustrates a guided approach based on examples. The following text is related to... Figure 4 Related description Figure 5 In step 510, the MDI system 470 captures an MDI image (e.g., a CT image) containing an organ (e.g., a lung), the organ's channel system, and SOIs within or on the organ. The computer system 480 analyzes the MDI image to determine the viewpoint from which the MDI image was captured. The computer system 480 may determine the viewpoint based on, for example, voxel information constituting the MDI image.

[0111] In step 520, camera 450 captures or photographs intraoperative images (e.g., video images of the outer surface of an organ containing a channel) at the surgical site. In step 520, computer system 480 optionally overlays image objects representing one or more SOIs onto the camera images. In some respects, the image of the organ's outer surface captured by the camera is the same organ shown in the MDI image. Camera 450 continues (522) to capture or photograph images while the surgical procedure is in progress, repeating steps 530-570 as described below for all or some of the newly captured images. In some respects, if an MDI image or a modified version of an MDI image is overlaid on the camera images, the image objects representing the SOI may not be overlaid on the camera images because the MDI images can provide sufficient information to the surgeon.

[0112] In step 530, computer system 480 determines the viewpoint of the captured image, for example, based on positioning information obtained from positioning sensors distributed in the channels of the organ and from positioning sensors mounted in or on camera 450. In step 540, computer system 480 compares the viewpoint of the MDI with the viewpoint of the captured image. If the two viewpoints match or are similar (this condition is indicated as "Yes" at step 550), computer system 480 displays the captured image or an image with the MDI image in step 560. However, if the two viewpoints do not match (this condition is indicated as "No" at step 550), computer system 480 modifies the viewpoint of the MDI image in step 570, compares the two viewpoints in step 540, and if, in step 550, the viewpoints match or are similar, computer system 480 displays the captured image or an image with the modified MDI image in step 560.

[0113] Figure 6 This illustrates a guidance method based on another instance. (And...) Figure 4 Related description Figure 6 In step 610, the MDI system 470 captures an MDI image (e.g., a CT image of the lungs), and the computer system 480 morphologically analyzes the MDI image, for example, to detect or identify airways of the airway system in the MDI image. The morphometric analysis performed by the computer system 480 results in morphological data relating to the image content of the MDI image (e.g., channels, one or more SOIs, anatomical structures, etc.).

[0114] CT images, as examples of MDI images, comprise multiple image slices of an organ, where each slice provides two-dimensional image information, while adjacent slices provide image information in a third (e.g., vertical) direction or dimension. For example, the spatial orientation and / or orientation of anatomical structures (e.g., channels, one or more SOIs, etc.) and how some image features (e.g., some channels) are spatially related to other image features (e.g., other channels) can be determined using voxels. A voxel represents a value on a grid in three-dimensional space. The orientation or coordinates of a voxel are not explicitly encoded along with its value. Instead, the orientation of a voxel is determined based on its orientation relative to other (e.g., adjacent) voxels. The relative orientation of voxels is determined within the image data structure that constitutes a single volumetric image. Thus, a voxel is the fundamental unit of information providing three-dimensional image information.

[0115] Therefore, CT images inherently include a coordinate system. This inherent coordinate system of CT allows computer systems not only to analyze CT image information and compare it with other image information, but also to manipulate voxels, such as modifying (e.g., translating, rotating, resizing, rescaling, etc.) image objects (e.g., anatomical structures, SOIs, other tissues, etc.) or image content, typically as needed, such as according to display requirements, or by virtually changing the imaging viewpoint. In the context of this disclosure, voxels can be manipulated and the associated CT image modified, if necessary, to make the viewpoint of the CT image or another MDI system equal, adjusted, matched, or adapted to the imaging viewpoint of the camera.

[0116] In step 620, camera 450 captures or photographs a camera image at the surgical site (e.g., a camera image of the outer surface of an organ), and optionally, computer system 480 overlays an image object representing one or more SOIs onto the captured image. In some respects, the image of the organ's outer surface captured by the camera is identical to the organ shown in the MDI image. This document describes how computer system 480 determines where the image object representing one or more SOIs will be positioned on the captured image, and how the image object should appear. While the surgery is in progress, camera 450 continues (622) to capture or photograph images. If, for example, in conjunction with this document... Figure 5 and 6 The method described above, which overlays an MDI image, a modified MDI image, or different preoperative images onto one or more preoperative images, indicates that the SOI image object may not be overlaid on one or more captured images.

[0117] In step 630, the computer system 480 morphologically analyzes the reconstructed channels in the organ, and in step 640, the computer system 480 may morphologically compare the channels imaged by the MDI system (e.g., by a CT system) with the reconstructed channels. The higher the morphological similarity between the channels in the MDI image and the reconstructed channels, the more similar the viewpoint (or virtual viewpoint) of the channels in the MDI image is to the viewpoint of the camera that captured the image of the organ containing the reconstructed channels. If the morphological similarity between the imaged channels and the reconstructed channels is low or nonexistent, it means that the digital data representing the image content of the MDI image needs to be manipulated in a way that matches the viewpoint of the MDI image (e.g., a virtual viewpoint or pseudo-viewpoint) with the viewpoint of the camera capturing or photographing the organ containing the reconstructed channels.

[0118] Positioning information related to the camera's location can be compared with positioning information obtained from positioning sensors distributed in the channel in order to determine the camera's spatial imaging angle or imaging viewpoint.

[0119] In step 650, computer system 480 checks for morphological differences between the reconstructed channels and their corresponding channels in the MDI image to determine if the two relevant viewpoints match. If computer system 480 determines in step 650 that there are no morphological differences between the reconstructed channels and their corresponding channels in the MDI image, it means that the two viewpoints match (i.e., the imaging viewpoint of the channel in the MDI image is similar to the imaging viewpoint of the reconstructed channel); this condition is marked "yes" in step 650, and in step 660, computer system 480 displays the MDI image along with one or more images as is (without modification). The MDI image can be superimposed on the image in a manner where the corresponding channels of the MDI image are consistent with the reconstructed channels. The MDI image can alternatively be displayed in combination with or in association with one or more images captured by camera 450.

[0120] However, if computer system 480 determines in step 650 that the imaged channels are morphologically dissimilar to the reconstructed channels (this condition is indicated as "No" in step 650), then computer system 480 modifies the image content of the MDI image in step 670 (e.g., by manipulating image voxels) to make the corresponding channels in the MDI image similar to, match, or fit the reconstructed channels. As described herein, if the two types of channels are similar to each other, the viewpoint of the modified MDI image (i.e., the final pseudo-viewpoint of the MDI image) matches, conforms to, or corresponds to the actual imaging angle of the camera. The MDI image is modified in a manner that modifies the viewpoint of the MDI image. Since the MDI image is captured once from the true viewpoint, which is the viewpoint of the MDI system, the modified viewpoint of the MDI image is a virtual viewpoint or pseudo-viewpoint. Alternatively, the MDI image content is modified in step 670, and steps 640 and 650 are repeated. Steps 640, 650, and 670 may be repeated until the modified MDI image of the channels morphologically matches or substantially matches the reconstructed channels.

[0121] The entire content of an MDI image can be modified as a whole; that is, each image element (e.g., each anatomical structure) in an MDI image can be modified image by image in the same way (e.g., rotated in the same direction and / or at the same angle) so that all anatomical structures in the MDI image can be displayed realistically, that is, in a way that is consistent with or matches the camera's viewpoint, even if some or all of the real anatomical elements are only partially visible in some or all of the images, thus consistent with the real anatomical structure of the organ. In other words, the content of an MDI image or any preoperative image can be modified in a way that makes the various anatomical structures in the MDI image appear as if captured from the viewpoint of camera 450. In this way, medical personnel performing medical procedures can have a realistic sense of the position or orientation of surgical instruments relative to the various anatomical structures of the organ being operated on, and can perform surgical procedures with improved precision.

[0122] After computer system 480 modifies the viewpoint of the MDI image in step 670 to match the viewpoint of the reconstructed channels, computer system 480 displays the image captured by camera 450 and the MDI image after the viewpoint of the modified MDI image in step 660. Alternatively, steps 640 and 650 can be repeated after step 670 until the channels match in step 650. The method steps can be applied, for example, to each image captured by camera 450, or to every nth image captured by camera 450 (e.g., every 5th image, every 20th image, etc.), and so on.

[0123] Computer system 480 calculates imaging parameters of or related to the MDI image and uses these calculations to adjust the image content of the MDI image (e.g., rotates, resizes, and / or translates it) so that whenever the camera's imaging perspective changes (e.g., when the surgeon moves or repositions camera 450), the spatial orientation and other display parameters of the MDI image content will change accordingly to display the MDI image content (e.g., together with the image captured by camera 450) as if the two images (camera image and MDI image) were captured from the same imaging viewpoint (e.g., as if they were captured from the same or similar distance and / or from the same or similar imaging angle). As used herein, the camera's imaging perspective can be measured or calculated relative to the anatomical organ or channel system of the organ. Spatially aligning the MDI image and the camera image can include, for example, spatially aligning channels appearing in the MDI image and reconstructed channels via computer system 480.

[0124] The aspects described herein refer to channels that are at least partially located in an organ (i.e., the organ in which the SOI is located). However, according to this disclosure, any channel system or channel dynamically associated with an organ may be used or utilized. A channel system or channel dynamically associated with an organ may include channels that move at least substantially with or in substantially the same manner as the organ (e.g., when the channel is in the organ) or channels in which their movement relative to the organ and / or vice versa can be determined (e.g., modeled, measured, or calculated). The term "movement" may include intrinsic movement, such as breathing, and / or extrinsic movement, such as movement caused by surgical instruments or devices.

[0125] The articles “a” and “an” are used herein to refer to one or more (i.e., at least one) grammatical object of the article, depending on the context. For example, depending on the context, “an element” means one or more elements. The term “including” is used herein to mean the phrase “including but not limited to” and may be used interchangeably with the phrase “including but not limited to”. Unless the context clearly indicates otherwise, the terms “or” and “and” are used herein to mean the term “and / or” and may be used interchangeably with the term “and / or”. The term “such” is used herein to mean the phrase “such as but not limited to” and may be used interchangeably with the phrase “such as but not limited to”.

[0126] The aspects may include a computer, processor, controller, and / or non-transitory storage medium, such as memory, disk drive, or USB flash drive, whose encoding includes or stores instructions, such as computer-executable instructions, which, when executed by the computer, processor, or controller, perform the methods disclosed herein.

[0127] Therefore, having described exemplary aspects of this disclosure, it will be apparent to those skilled in the art that modifications to the disclosed aspects will be within the scope of this disclosure. Thus, alternative aspects may include more modules, fewer modules, and / or functionally equivalent modules.

[0128] This disclosure relates to various types of MDI systems (e.g., CT, MRI, etc.), various cameras, and various types of positioning systems, devices, or sensors. Therefore, the scope of the appended claims is not limited to any particular MDI system, camera, positioning system, device, or sensor. This disclosure also relates to various types of body organs and preoperative images other than MDI images, and is applicable to multiple SOIs, DOPs, and DLPs that may be located in the same or different body parts and are performed during the same surgical procedure.

[0129] As referred to herein, the term "overlay" and its various derivatives can include, for example, overlapping information by adding or combining information such as image information (also referred to herein as "image"). For example, an image can be overlaid on a view (e.g., another image) to provide a composite view. As referred to herein, the term "overlay" and its various derivatives can also include, for example, replacing information by removing one piece of information (e.g., a portion of an image) and replacing it with another piece of information (e.g., a portion of another image) or by covering the first piece of information with another piece of information.

[0130] As referred to herein, the term "structure of interest" (SOI) can include biological or artificial elements, such as anatomical regions of interest in a patient's body (i.e., anatomical SOIs). For example, an SOI can be or can include tissues (including soft tissue and bone), organs, implants, or reference markers. An anatomical SOI can be or can include, for example, a diseased portion of tissue or organ, or a healthy portion of tissue or organ, or both a diseased and a healthy portion of tissue or organ.

[0131] As used herein, the term "morphology" refers to "mathematical morphology," which is the theory and techniques used in the analysis and processing of geometric structures. Mathematical morphology is most commonly applied to digital images. As used herein, morphology can be viewpoint-dependent, meaning that image objects can have different morphologies when viewed from different angles. As used herein, morphometrics refers to the quantitative analysis of forms, which include, for example, the size, shape, geometry, angles, and proportions between image elements contained in individual channels within MDI images and video images. Morphometric analysis can be performed using various image processing techniques to identify, for example, anatomical structures in the analyzed image, with the aim of comparing, for example, the spatial relationships between anatomical structures contained in images used as comparison objects.

[0132] While not limited in this respect, the use of terms such as “processing,” “calculating,” “operating,” “determining,” “establishing,” “evaluating,” “analyzing,” and “checking” can refer to one or more operations and / or processes of a computer, computing system, or other electronic computing device (e.g., a controller) that manipulates and / or transforms data represented as physical (e.g., electronic) quantities in the computer's registers and / or memory into other data represented similarly as physical quantities in the computer's registers and / or memory or other non-transitory storage media capable of storing instructions to perform operations and / or processes. Unless explicitly stated otherwise, the exemplary methods described herein are not subject to a particular order or sequence. Furthermore, some of the described methods or steps may occur or be performed, for example, at the same point in time.

[0133] Medical diagnostic imaging (MDI) refers to a variety of non-invasive methods used to detect, identify, and monitor diseases or injuries by generating images of various internal anatomical structures, organs, and pathologies (such as tumors) within a patient's body. The detailed images produced by these procedures are used to inform patients and physicians about the anatomy and function of the patient's internal organs and structures, and are also used for treatment and surgery on, for example, diseased organs. Radiologists and other physicians interpret the resulting MDI images for diagnosis, such as medical diseases or injuries, enabling the planning and implementation of specific patient treatments and therapies. Diagnostic imaging is also used to guide surgical planning and is often used to track the outcomes of surgical and / or treatment procedures. Diagnostic imaging techniques include, for example, ultrasound (US), magnetic resonance (MR), and computed tomography (CT), as well as traditional X-rays or radiology.

[0134] Video-assisted thoracoscopic surgery (VATS) is a type of thoracic surgery performed using a small camera inserted into the patient's chest through a small incision. The VATS instrumentation consists of a fiber optic oscilloscope connected to the camera and conventional thoracic or laparoscopic instruments. Using the VATS camera, the surgeon can view the instruments being used and the anatomical structures being operated on. The camera's fiber optic cable and instruments are inserted through a separate hole (or port) in the chest wall. The camera (e.g., a charge-coupled device (CCD) type camera) providing "conventional" (e.g., RGB) images (i.e., video images) of the treated body organ can be maintained at a certain distance from and outside the treated body organ to visualize the entire treated body organ and its surrounding area. If the surgeon's attention is focused on a specific area or spot, the surgeon can manipulate the camera to magnify that area or spot, or the surgeon can move the camera closer to that specific area or spot. VATS has many applications, one of which is in lung surgery. As described in this article, VATS cameras can capture images of the surface of the organ to be operated on, but not of the internal parts or anatomical structures of the organ, unless the surgeon makes a cut through the organ to make the internal parts visible to the camera. In the case of lung surgery, VATS cameras cannot see the diseased part of the lung to be operated on. Therefore, without a route to the internal anatomical structures of the lung, the surgeon may cut into the lung in the wrong location and / or unintentionally cut healthy or sensitive tissue or organs.

[0135] Some of the exemplary methods and systems shown in the accompanying drawings and described herein are described in the context of surgery on the lungs using a camera. However, the same or similar methods and systems (including imaging devices other than cameras) can be used to guide the surgeon when operating on other organ systems that include anatomical access / lumen systems.

[0136] Surgeons may want to manipulate multiple SOIs (e.g., remove or excise them) during the same surgical procedure, and the systems and methods described herein are also applicable to these situations. For example, lobectomy involves the removal of the pulmonary artery, pulmonary vein, and bronchi leading to the lung lobes; therefore, each lung element can be considered an anatomical SOI. An anatomical SOI may include one or more organ objects that may be in or related to the same organ to be operated on during the same surgical procedure, or that may be related to multiple organs to be operated on during the same surgical procedure. An anatomical SOI may be or include anatomical structures such as diseased organ portions, diseased lung portions, abnormal tissues, abnormal organs, or benign anatomical structures.

[0137] Anything described in this document as applicable, attributable to, or relating to MDI images (e.g., any operation, methodological steps, benefits, etc.) is equally applicable, attributable to, or relating to any type of preoperative image.

[0138] It should be understood that the various aspects disclosed herein can be combined in combinations different from those specifically presented in the specification and drawings. It should also be understood that, depending on the instance, certain actions or events of any process or method described herein may be performed in a different order, or may be added, combined, or omitted entirely (e.g., all described actions or events may not be necessary for performing these techniques). Furthermore, although some aspects of this disclosure are described for clarity as being performed by a single module or unit, it should be understood that the techniques of this disclosure can be performed by a combination of units or modules associated with, for example, a medical device.

[0139] In one or more instances, the described techniques can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include non-transitory computer-readable media, which correspond to tangible media such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0140] The instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other physical structure suitable for implementing the described techniques. Similarly, the techniques can be fully implemented in one or more circuit or logic elements.

Claims

1. A surgical guidance system comprising: A positioning system configured to determine the location of multiple channels in a channel system associated with an organ; An imaging device configured to generate an intraoperative image stream; and The controller is configured to: Based on the positions of the multiple channels, the channel system is reconstructed; Based on the channel system and the preoperative image of the structure of interest (SOI), determine the position of the SOI relative to the channel system in the preoperative image; Based on the position of the SOI relative to the channel system in the preoperative image, the position of the SOI relative to the reconstructed channel system is determined; Receive intraoperative images from the imaging device; The position of the imaging device relative to the position of the reconstructed channel system is determined; The position of the SOI relative to the intraoperative image is determined based on the position of the SOI relative to the reconstructed channel system and the position of the imaging device determined relative to the position of the reconstructed channel system. and At the determined position of the SOI relative to the intraoperative image, an image object representing the SOI is superimposed on the intraoperative image. The controller is configured to determine the position of the SOI relative to the reconstructed channel system by determining the spatial relationship between the reconstructed channel system and the channel system in the preoperative image. Determining the position of the SOI relative to the reconstructed channel system includes mapping the position of the SOI in the preoperative image to the reconstructed channel system, and wherein multiple positioning sensors are respectively mounted on multiple catheters located in the multiple channels, and Reconstructing the multiple channels includes transmitting location information between the positioning system and the multiple positioning sensors.

2. The system of claim 1, wherein the position of the imaging device relative to the reconstructed channel system is determined based on position information from one or more positioning sensors located in or on the imaging device relative to the reconstructed channel system.

3. The system according to claim 1 or 2, wherein the controller is further configured to: The preoperative image is modified based on changes in the distance between the imaging device and the SOI, or based on changes in the angle at which the imaging device captures the intraoperative image; and The modified preoperative images, along with the intraoperative images, are displayed on the display device.

4. The system of claim 1, wherein the image object representing the SOI is given morphological properties of the SOI in the preoperative image.

5. The system of claim 3, wherein modifying the preoperative image includes modifying the size, shape, proportion, or orientation of the image content of the preoperative image.

6. The system of claim 1, wherein the controller is further configured to modify the preoperative image such that the viewpoint of the preoperative image matches the viewpoint of the intraoperative image. Modifying the preoperative image includes aligning the viewpoint of the organ, the viewpoint of the body structure associated with the organ, and the viewpoint of the SOI in the preoperative image with the viewpoint of the intraoperative image.

7. The system of claim 1, wherein the SOI comprises: The diseased organ portion.

8. The system of claim 1, wherein the SOI comprises: The diseased part of the lung.

9. The system of claim 1, wherein the SOI comprises: Abnormal tissue.

10. The system of claim 1, wherein the SOI comprises: Abnormal organ.

11. The system of claim 1, wherein the SOI comprises: Benign anatomical structure.

12. The system of claim 1, wherein the SOI comprises: Organ structure.

Images