Systems and methods for model fusion based on surgical tools

Abstract

The present teachings relate to automated model fusion based on surgical tools. A two-dimensional (2D) image is received, and a surgical instrument is captured. A 2D position at which the surgical tool is attached to the instrument is detected. A particular type of focus information is determined based on the surgical tool for assisting a surgical task using the surgical tool. A 2D / 3D corresponding focal region is identified via model fusion. Visual guidance is created by projecting the type of focus information onto the 2D focus region based on the type of focus information from the 3D focus region of the 3D model of the organ and the surrounding anatomical structure.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. patent application Ser. No. 18 / 172,428, filed on February 22, 2023, entitled “Systems and Methods for Surgical Tool-Based Model Fusion,” which is incorporated herein by reference in its entirety.

[0002] This application is related to U.S. patent application Ser. No. 18 / 172,447, entitled “System and Method for Multimodal Display via Surgical Tool Assisted Model Fusion” (Attorney Docket No. 140551.569673), and International Application Ser. No. ___, entitled “System and Method for Multimodal Display via Surgical Tool Assisted Model Fusion” (Attorney Docket No. 140551.589776), both of which are incorporated herein by reference in their entirety. Background Art 1. Technical Field

[0003] The present teachings relate generally to computers and more particularly to signal processing. 2. Background Technology

[0004] With the advancement of technology, more and more tasks are now completed with the assistance of computers. Different industries have benefited from such technological advances, including the medical industry, in which large amounts of image data can be processed by computers to capture the patient's anatomical information to identify anatomical structures of interest (e.g., organs, bones, blood vessels, or abnormal nodules), obtain measurements of each object of interest (e.g., the size of a nodule growing in an organ), and visualize related features (e.g., three-dimensional (3D) visualization of an abnormal nodule). Such technologies enable healthcare professionals (e.g., doctors) to use high-tech means to assist them in treating patients in a more effective manner. Today, many surgeries are performed under laparoscopic guidance, so there is usually no need to open the patient's body, thereby minimizing damage to the body.

[0005] Through laparoscope guidance, the surgeon can be guided to approach the target organ and perform the desired operation. Although each operation may have a predetermined goal, in order to achieve the predetermined goal, each operation may have different subtasks that need to be completed. Examples include manipulating surgical instruments to specific positions under the guidance of laparoscopic images, targeting the anatomical structure to perform certain subtasks, and then using surgical instruments to perform the subtasks. Example subtasks include using surgical tools to separate blood vessels from the organ to be operated on, clamping blood vessels to stop blood flow before removing a portion of the organ, etc. Different surgical instruments or tools may be required to perform different subtasks.

[0006] Although laparoscopic images can provide users with real-time visualization of the anatomical structures inside the patient's body, there are limitations on how to interpret these images. First, these laparoscopic images are typically two-dimensional (2D) images, so that some information of the 3D anatomical structure may not be displayed in the 2D image. Second, because the laparoscopic camera has a limited field of view, the acquired 2D image may only capture a partial view of the target organ. In addition, some basic anatomical structures (such as blood vessels) may be located inside the organ, making them invisible in the 2D image. Due to these limitations, the user needs to reconstruct the invisible objects in his mind, which requires a lot of experience to digest what is seen from the laparoscopic images to align the observable objects with the pre-planned 3D surgical procedure. Therefore, a solution that can address the above challenges is needed. Summary of the Invention

[0007] The teachings disclosed herein relate to methods, systems, and programming for information management. More specifically, the teachings relate to methods, systems, and programming related to hash tables and storage management using hash tables.

[0008] In one example, a method implemented on a machine having at least one processor, a storage device, and a communication platform capable of connecting to a network for automated model fusion based on a surgical tool. A two-dimensional (2D) image is received, capturing a surgical instrument. A 2D position of the surgical instrument attached to the instrument is detected. Based on the surgical tool, a specific type of focus information is determined for use with the surgical tool to assist in a surgical task. Corresponding 2D / 3D focus regions are identified via model fusion. Based on the type of focus information from a 3D focus region of a 3D model of an organ and surrounding anatomy, a visual guide is created by projecting the type of focus information onto the 2D focus region.

[0009] In a different example, a system for automatic model fusion based on surgical tools is disclosed, the system including a surgical tool-assisted model fusion mechanism and a focus information display unit. The surgical tool-assisted model fusion mechanism is provided for detecting a surgical instrument to which a surgical tool is attached for performing a surgical task and its position from a 2D image capturing an anatomical structure associated with an organ. Based on the detected surgical tool, a type of focus information is determined. Based on the position of the surgical tool, a 2D / 3D corresponding focus region is identified so that the type of focus information can be retrieved from a 3D focus region in a 3D model representing the organ and surrounding anatomical structures. A focus information display unit is provided for creating a visual guide based on the type of focus information obtained from the 3D focus region and projecting the type of focus information from the 3D focus region onto a 2D focus region to assist a user in performing a surgical task using the surgical tool.

[0010] Other concepts relate to software for implementing the present teachings. A software product according to this concept includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium can be executable program code data, parameters associated with the executable program code, and / or information related to a user, a request, content, or other additional information.

[0011] Another example is a machine-readable, non-transitory, and tangible medium having recorded thereon information for automatic model fusion based on a surgical tool. When read by a machine, the information causes the machine to perform various steps. A two-dimensional (2D) image is received, capturing a surgical instrument. A 2D position of the surgical tool attached to the instrument is detected. A specific type of focus information is determined based on the surgical tool to assist in a surgical task using the surgical tool. 2D / 3D corresponding focus regions are identified via model fusion. Based on the type of focus information of the type from the 3D focus region of the 3D model of the organ and surrounding anatomical structure, a visual guide is created by projecting the type of focus information onto the 2D focus region.

[0012] Additional advantages and novel features will be set forth in part in the following description and in part will be apparent to those skilled in the art upon examination of the following and accompanying drawings, or may be learned by production or operation of the examples. The advantages of the present teachings may be realized and obtained by practice or use of various aspects of the methods, tools, and combinations set forth in the detailed examples discussed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The methods, systems, and / or programs described herein will be further described by way of exemplary embodiments. These exemplary embodiments will be described in detail with reference to the accompanying drawings. These embodiments are non-limiting exemplary embodiments, in which like reference numerals denote similar structures throughout the several views of the drawings, and in which:

[0014] Figure 1A A surgical instrument is shown having a hook at one end proximal to a target organ;

[0015] Figure 1B A surgical instrument is shown having a cutter at one end proximal to a target organ;

[0016] Figure 2A depicts an exemplary high-level system diagram of a surgical tool-assisted model fusion mechanism according to an embodiment of the present teachings;

[0017] Figure 2B is a flow chart of an exemplary process of surgical tool-assisted model fusion mechanism according to an embodiment of the present teachings;

[0018] Figure 3Adepicts an exemplary high-level system diagram of a surgical tool detection unit according to an embodiment of the present teachings;

[0019] Figure 3B is a flow chart of an exemplary process of a surgical tool detection unit according to an embodiment of the present teachings;

[0020] Figure 3C An exemplary implementation of a surgical tool detection unit according to an embodiment of the present teachings is shown;

[0021] Figure 4A An exemplary surgical tool-based focus information configuration according to an embodiment of the present teachings is shown;

[0022] Figure 4B FIGURES illustrate examples of different focus information determined due to detection of different surgical tools according to an embodiment of the present teachings;

[0023] Figure 4C FIGURES illustrate an example of focus information determined based on a detected surgical tool according to an embodiment of the present teachings;

[0024] Figure 5A depicts an exemplary high-level system diagram of a tool-based focused information identifier according to an embodiment of the present teachings;

[0025] Figure 5B is a flow chart of an exemplary process of a tool-based focused information identifier according to an embodiment of the present teachings;

[0026] Figure 5C depicts an exemplary high-level system diagram of a 2D / 3D focus region marker according to an embodiment of the present teachings;

[0027] Figure 5D is a flow chart of an exemplary process of a 2D / 3D focus region identifier according to an embodiment of the present teachings;

[0028] Figure 6 is an exemplary high-level system diagram of a focused information display unit according to an embodiment of the present teachings;

[0029] Figure 7A is a flowchart of an exemplary process of focusing on an information display unit according to an embodiment of the present teachings;

[0030] Figure 7B is a flow chart of an exemplary process of a focus overlay renderer for displaying focus information based on a detected tool identification according to an embodiment of the present teachings;

[0031] Figure 7CFIGURE 1 illustrates an exemplary display of focus information near a surgical instrument according to an embodiment of the present teachings, with details of the focus information being shown using a magnified view;

[0032] Figure 8 is a schematic diagram of an exemplary mobile device architecture that can be used to implement a dedicated system for implementing the present teachings in accordance with various embodiments; and

[0033] Figure 9 is a schematic diagram of an exemplary computing device architecture that can be used to implement a special-purpose system for implementing the present teachings in accordance with various embodiments. DETAILED DESCRIPTION

[0034] In the following detailed description, numerous specific details are set forth by way of example in order to facilitate a thorough understanding of the relevant teachings. However, it should be apparent to one skilled in the art that the present teachings may be practiced without these details. In other instances, well-known methods, processes, components, and / or systems have been described at a relatively high level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

[0035] The present teachings disclose exemplary methods, systems, and implementations for automatically determining instantaneous focus information based on detected surgical tools and displaying such focus information. A surgical instrument is inserted into the human body to perform a predetermined surgical procedure. The surgical instrument may carry tools at its distal end for performing a task. The position of the surgical instrument in a 3D workspace is tracked so that the 3D position of the surgical instrument is known. A 3D model of the target organ to be operated on is registered with 2D images acquired during the procedure. In addition to tracking the 3D positions of n surgical instruments, tools attached to the instrument tips may also be detected based on the 2D images.

[0036] Each tool may have a specific (multiple) designated functions during the surgical procedure. Therefore, the presence of the tool can signal what the task is to be performed, what the user (surgeon) is focused on at the moment, and what information may be relevant because it can assist the user in completing the task more easily. For example, when the surgical instrument is present near the target organ 100, the hook 120 can be attached to the tip of the surgical instrument 110, such as Figure 1A As shown in . Since hooks are known to be commonly used to perform tasks related to blood vessels, detecting the presence of a hook from a 2D image can indicate what the user intends to do with the blood vessels associated with the target organ. Such tasks may include hooking a blood vessel in order to separate the blood vessel from the organ before removing it, stopping blood flow by clamping the blood vessel, or by cauterizing the blood vessel. In this case, it can be inferred that the user's attention is on the blood vessel. Given this, enhancing the display of blood vessels close to the hook can help the user perform the intended task. For example, as Figure 1AAs shown, if the region 130 is located where the hook 120 is located, a magnified view 140 of the blood vessels in the region 130 may be displayed, the magnified view 140 providing a better view of the details in the region of interest.

[0037] Figure 1B Another example is shown in which a different tool (such as a cutter 160) can be attached to the tip of the surgical instrument 150 and appear near the organ 100. As is known, such a cutter 160 is often used to remove organs. Since resection operations often have a pre-planned surgical trajectory, when the cutter is detected in the 2D image, this can indicate to the user that the organ 100 is ready to be removed. Based on this assessment, the focused information is the portion of the organ to be removed. In order to assist the user in better visualizing the boundaries of the resection, the cutter 160 can be positioned on the portion of the organ to be removed (e.g., Figure 1B The focus information is displayed in an emphasized manner on the surgical tool 170 in the figure and the removal trajectory (e.g., dashed boundary) on the organ 100. In this case, the focus information determined based on the detected surgical tool can be displayed in a special way. For example, the corresponding resection area in the 3D model can be projected onto the 2D image using, for example, highlighting, and the boundary of the planned resection trajectory can be displayed, for example, in some different color, making it easier for the user to see and follow the boundary. The details of the focus information can also be magnified as an overlay on a portion of the screen or in a pop-up window.

[0038] The teachings disclosed herein determine what focused information is available that may assist a user in completing a current task based on a currently detected surgical tool, and accordingly display such identified focused information to the user in a manner that increases effectiveness in completing the task at hand. Figure 2A An exemplary high-level system diagram of a surgical tool-assisted model fusion mechanism 200 according to an embodiment of the present teachings is depicted. In this illustrated embodiment, the surgical tool-based model fusion mechanism 200 includes a surgical tool detection unit 210, a tool-based focus region identifier 240, and a focus information display unit 260.

[0039] 2D video images are provided as input to these three components. As discussed herein, in the surgical workspace, calibration can be performed so that the 2D image is aligned with the 3D coordinates of the points in the 3D workspace. In some embodiments, such calibration can be performed via a tracking mechanism (not shown), which tracks the surgical instrument, which has some (multiple) tracking devices attached to one end of the instrument that is retained outside the patient's body. The feature points present in the 2D image can be aligned with the 3D workspace. The transformation matrix can be derived based on a set of feature points identified from the 2D image via the surgical instrument (so that the 3D coordinates of such feature points can be derived) and the corresponding feature points on the 3D model of the organ. Based on the transformation matrix, any point on the 3D organ model can be projected onto the 2D image to provide 3D visualization of the organ at the position where the organ appears in the 2D image. Similarly, any type of information (such as blood vessels within the organ) can also be projected onto the 2D image.

[0040] exist Figure 2A In the illustrated embodiment, a surgical tool detection unit 210 is provided for identifying surgical tools appearing in 2D video images, such as those captured by a laparoscopic camera within a patient's body. Detection results regarding the type of surgical tool are provided to a tool-based focus region identifier 240. Tool-based focus region identifier 240 then determines focus information associated with the detected surgical tool based on a tool / focus information configuration stored in 250, thereby obtaining focus information for a target organ from a 3D organ model stored in 230. Tool / focus information configuration 250 is provided to specify focus information for each surgical tool used in different surgical procedures. Such configurations can be provided based on medical practice, for example, by professionals in the field.

[0041] The 3D organ model in storage 230 can be generated offline before the surgical procedure based on the image data obtained from the patient. Such a 3D model can include different types of information. For example, it can include 3D modeling of the organ using volume representation and / or surface representation. It can also include modeling of internal anatomical structures, such as nodules growing in the organ and vascular structures around and within the organ. The modeling of the organ and internal anatomical structures can also provide different features and measurements of, for example, the organ itself, the nodules therein or the different blood vessels. In addition, the 3D model of the patient's organ can also be combined with information about a pre-planned surgical trajectory, which has planned cutting points with defined positions on the surface of the organ. In addition, the 3D organ model can also include modeling of some areas near the organ. For example, when blood vessels or other anatomical structures (such as bones outside the organ) are connected to the organ or form part of the blood supply network connected to the organ, they can also be modeled so that they can affect the way the operation is performed.

[0042] Different types of information included in the 3D organ model can be used to project onto the 2D video image to assist the user in performing different tasks involved in the surgery. At each moment, depending on the type of surgical tool detected, (multiple) different segments of information from the 3D model can correspond to focus information. For example, when a surgical hook is detected from the 2D image, the blood vessels can be considered as the focus task at this time because surgical hooks can generally be used to handle tasks associated with blood vessels. In this case, the information characterizing the blood vessels in the 3D organ model 230 can be obtained from 230 as focus information and can be used for special displays. Although other types of information from the 3D organ model can also be retrieved and used for projection onto the 2D image (for example, the 3D shape and measurements of the organ), the portion considered as focus information can be displayed in different ways. An example such as Figure 2A , in which a 3D model of an organ is projected onto a 2D image, but since a surgical cutter is detected from the 2D image, the portion of the organ to be removed is displayed, for example, in a highlighted manner. As shown in the figure, boundary information can also be specifically marked in a conspicuous manner in the highlighted area corresponding to the resection area, because this is where the surgical cutter needs to approach to make the cut.

[0043] Figure 2B 2 is a flow chart of an exemplary process for a surgical tool-assisted model fusion mechanism 200 according to an embodiment of the present teachings. At 205 , a 2D image may be acquired, for example, from a laparoscopic camera inserted into a patient. Based on the 2D image, the surgical tool detection unit 210 identifies the type of surgical tool deployed at the time of surgery at 215 . This detection result is sent to a tool-based focus region identifier 240 , which, based on a tool / focus information configuration 250 , determines at 225 the type of focus information corresponding to the detected tool type, given the type of surgical procedure. As discussed herein, the type of focus information can be identified from a focus region determined based on the detected position of the surgical tool. In some embodiments, the focus region is determined relative to the view angle of the detected surgical tool. To this end, a spatial relationship between the detected tip of the surgical tool and a portion of the anatomical structure near the tip location is determined at 235 . Based on this spatial relationship, the view angle of the surgical tool can be estimated. For example, a 2D region corresponding to the portion of the anatomical structure directly facing the tool tip may include information relevant to the procedure performed using the surgical tool. Figure 1A An example is provided in , in which focus information to be displayed for assisting a surgeon in performing an operation using a detected surgical tool corresponds to region 130 because region 130 is a region that the tip of the detected surgical hook directly faces.

[0044] Based on the spatial relationship between the tool tip and the relevant anatomical portion, the tool-based focus region identifier 240 then identifies both a 2D focus region in the 2D image and a corresponding 3D focus region for the appropriate portion(s) of the 3D organ model at 245. Such identified 2D / 3D focus regions are where focus information is located and will be displayed to assist the surgeon in performing the operation using the surgical tool at that moment. Based on the identified 2D / 3D focus regions, the focus information display unit 260 then renders the content of the 3D model from the 3D focus region at 255 by projecting such 3D content onto the 2D focus region in the 2D image in a registered manner. In some embodiments, the 3D organ model can be retrieved at the beginning of the surgery, and the relevant portion can then be used to project onto the 2D image during the surgery in a manner determined by the detected surgical tool. For example, at the beginning of the surgery, a surface representation of the 3D organ in the 3D model can be used to project onto the 2D image. During the surgery, the display can be dynamically adjusted. When the surgical instrument changes its pose (eg it is oriented towards a different part of the organ), the projection of the surface representation of the 3D model needs to be adjusted so that the part of the organ's surface directly facing the instrument is displayed via the projection.

[0045] As another example, when different surgical tools (such as hooks) are detected, the vascular tree within the organ may need to be displayed in a manner that enables the user to "see through" the organ to observe the vessels. In this case, the vascular representation of the 3D model can be used to present such focus information. In some embodiments, the vessels can be projected with an appropriate orientation relative to the perspective of the tip of the surgical hook, as discussed herein. While the vessels are in focus and rendered in a special manner, other parts of the organ can also be rendered in a manner that is faded so that the focus information stands out more. Thus, at each moment, different parts of the 3D model can be used for different displays in different ways depending on the situation. The rendering can take into account different rendering treatments for focused and non-focused information. For example, when a vessel is in focus, other anatomical structures can be displayed to appear faded so that the vessel can appear more visible. At the same time, the focus information can also be displayed in a highlighted manner to increase contrast, such as by increasing the intensity of pixels on the vessel or using bright colors to display the vessel pixels.

[0046] That is, each time the focus changes, the projection of different pieces of the 3D model information on the 2D image may also change accordingly. Continuing with the above example, when a surgical cutter is detected later (after the surgical hook), the blood vessels previously projected in the 2D image may now need to be completely hidden, and the portion of the organ in front of the detected cutter may now need to be displayed in a special and highlighted manner so that the portion of the organ near the cutter can be clearly visualized. Therefore, each time the situation changes, the 3D organ model, once retrieved from memory 230, may need to be used, and each time a different portion of the 3D model may be used for the focused display and the other for the non-focused display.

[0047] Figure 3A An exemplary high-level system diagram of a surgical tool detection unit 210 according to an embodiment of the present teachings is depicted. In this illustrated embodiment, the surgical tool detection unit 210 includes an image-based surgical tool classifier 300 and a surgical tool pose estimator 350. In some embodiments, the image-based surgical tool classifier 300 may be as follows: Figure 3A As shown, a 2D image data pre-processor 310 , an image feature selector 320 , an image feature extractor 330 , a feature-based tool classifier 340 , and a surgical tool pose estimator 350 is provided. Figure 3B is a flow chart of an exemplary process of the surgical tool detection unit 210 according to an embodiment of the present teachings. In operation, the 2D image data preprocessor 310 receives a 2D image as input and then processes the received image at 305. Such preprocessing may be performed to improve image quality to facilitate feature extraction. Such enhancement may include filtering or intensity amplification for improved contrast.

[0048] In the illustrated embodiment, in order to identify surgical tools from a 2D image, features can be explicitly extracted from the preprocessed 2D image. Different features may be relevant for different types of surgical tools. In addition, depending on the specific surgical procedure, the types of surgical tools that may be used during the operation may be known. Therefore, the image feature selector 320 can access the surgical tool detection model 220, which can specify different types of features that are critical for different types of surgical tools. Based on this configuration, at 315, the image feature selector 320 can determine the types of features that may need to be extracted from the 2D image during the specific surgical procedure. For example, if the surgery is a liver resection, it may be known that the only surgical tools used are a cutter and a hook. Given this, the image feature selector 320 can obtain information about the features to be extracted in order to identify these two types of tools.

[0049] The features to be extracted are sent to notify the image feature extractor 330. Upon receiving instructions regarding which image features to identify, the image feature extractor 330 extracts such image features from the preprocessed image at 325. The feature-based tool classifier 340 can use the extracted features to classify the type of tool present in the image based on the extracted features at 335. Because image features for multiple types of surgical tools may need to be extracted from the same image, some features expected for a particular tool may not represent physical features of the surgical tool (e.g., features extracted for a surgical cutter from an image containing a surgical hook). In such cases, the confidence level for the tool type classification may remain low. In such cases, the classification result with a low confidence level determined at 345 may not be adopted, and no classification will be performed. The process continues at 325 with the next image. If the confidence score for the classification satisfies certain conditions, as determined at 345, the classification of the tool type is accepted. In such cases, the surgical tool pose estimator 350 continues to estimate the pose of the surgical tool at 355. For example, if the detection identifies that a surgical tool observed in a 2D image corresponds to a hook, the 3D position and orientation of the hook is determined. As discussed herein, such pose information is important in determining what the user's focus is, so that the focus information to be displayed in a particular manner is accurately determined.

[0050] Figure 3C Different exemplary implementations of an image-based surgical tool classifier 300 according to different embodiments of the present teachings are shown. In the illustrated embodiment, the image-based surgical tool classifier 300 performs classification via a machine-learned surgical tool classification model 360, which can be trained based on training data to recognize K different types of surgical tools. In some embodiments, the model 360 can have K outputs, each corresponding to a type of surgical tool and being a numerical value between 0 and 1 representing, for example, the probability that the input image includes the corresponding tool type. In this illustration, the surgical tool classification model 360 is trained to implicitly learn to perform image feature selection, extraction, and then classification for all K types of tools based on characteristics observed in the 2D image. Using the K probabilities output, a model-based tool type determiner 370 can then select one of the K tool type classifications as the most likely classification based on the probabilities from the model 360. The surgical tool pose estimator 350 then uses such detected surgical tool types to analyze the image to determine the position and orientation of the detected tool. When the 2D image is registered with the 3D workspace coordinate system, the appearance of the detected surgical tool can be analyzed to derive the 3D position and orientation of the tool in the 3D workspace coordinate system. As mentioned above, both the 3D position and orientation of the tool (particularly the tip of the tool) are important in identifying focus information.

[0051] The output of the surgical tool detection unit 210 includes the surgical tool type and the posture of the tool tip, and the output is sent to the tool-based focus area identifier 240, such as Figure 2A As shown, focus information is identified based on the tool type and the focus region determined based on the pose of the tool tip. As previously described, the type of focus information can be determined based on the tool / focus information configuration 250. Specific focus information can then be identified based on the tip orientation and its spatial relationship relative to the 3D model of the organ. Figure 4A An exemplary tool / focus information configuration stored in 250 according to an embodiment of the present teachings is shown. In this example, there may be a table in which rows may represent different tool types and columns may represent procedure types. The configuration may be structured in this manner because the same type of surgical tool may be used for different functions in different types of surgical procedures.

[0052] like Figure 4A As shown, this exemplary configuration can include rows corresponding to different surgical tool types (such as cutter scissors 400, surgical hooks 420, etc.). Different columns correspond to different procedures, so that each entry in the table corresponds to a specific combination of surgical tools used in a specific surgical procedure. The content provided in each entry can specify the type of focused information that a user (e.g., a surgeon) may want to clearly see. For example, if the cutter scissors 400 are used in a laparoscopic procedure, the type of focused information is specified as "resection boundary near the cutter tip" (410). Figure 4B An example of a resection boundary near a cutter tip displayed according to an embodiment of the present teachings is shown. In this example, cutter 160 is detected and its pose is determined. Based on the estimated cutter pose, the spatial relationship between the cutter opening (tip) and the anatomical structure of the target organ can be determined via a 3D model of the organ, so that a specific portion of the resection boundary can be identified on the 3D model for special display. For example, the portion of the resection boundary to be used as focus information can correspond to the portion of the resection boundary facing the cutter opening. In particular, resection boundary 170 close to the cutter and facing the cutter opening is treated as focus information and displayed in a special manner, even though organ information near the detected tool may also be displayed. For example, the portion of the organ in 440 can also be displayed when the cutter is detected, but because it does not face the cutter opening 160, it is not rendered in a special manner.

[0053] As another example, if a surgical hook 420 is detected in a 2D image during a laparoscopic procedure, the type of focus information is designated as "vessel branch in front of the tip of the hook" (430), as shown in FIG. Figure 4AIn this case, because the type is a hook, the relevant information is the vessel branch, as configured. However, it is possible to determine which part of the vessel tree corresponds to the focus information based on the detected posture of the hook. Figure 4C As shown, a hook 120 is detected with an orientation such that the hook tip is in a specific direction. In this case, the type of focused information is the vascular tree associated with the organ, but the portion of the vascular tree considered focused information is determined based on the spatial relationship between the hook tip and the organ's anatomy. In this example, the portion of the vascular tree at position 130, for example, directly faces the hook tip and can be considered focused information. However, a vascular branch in region 450, even though it is near the detected tool, may not be considered focused information because it is not in the direction of the hook tip.

[0054] Figure 5A An exemplary high-level system diagram of a tool-based focus region identifier 240 according to an embodiment of the present teachings is depicted. In this illustrated embodiment, the tool-based focus region identifier 240 includes a focus information type determiner 510, a focus information location determiner 520, and a 2D / 3D focus region identifier 530. As discussed herein, to determine focus information, both a specific information type and a specific region of interest (on both the 2D image and the 3D model) need to be identified. The focus information type determiner 510 is provided for determining the type of focus information based on the configuration specified in the tool / focus information configuration 250. On the other hand, the focus information location determiner 520 is provided for identifying a 2D focus region of interest in the 2D image within which the type of focus information of interest resides. As discussed herein, the 2D focus region is determined based on the detected pose (e.g., position and orientation) of the tip of the surgical tool. Based on the 2D focus region determined in the 2D image, a 3D focus region corresponding to the 2D focus region can then be correspondingly identified. This is achieved by the 2D / 3D focus region marker 530 .

[0055] Figure 5B 5 is a flow chart of an exemplary process of the tool-based focus region identifier 240 according to an embodiment of the present teachings. At 540, the focus information type determiner 510 receives the detection results about the type of surgical tool and its posture, and at 550, determines the focus region identifier 240 based on the surgical tool type, the configuration from 250 (e.g., Figure 4A) to determine the type of focused information (e.g., a vessel). To determine the specific 2D region of interest where the type of focused information resides, the focused information location determiner 520 analyzes the pose of the tool (e.g., the position and orientation / direction of the tool tip) at 560. At 570, the 2D / 3D focus region identifier 530 uses this information to first identify a region of interest in the 2D image, i.e., a 2D focus region relative to the pose of the tool tip, and correspondingly identify a portion of the anatomical structure in the 3D model, i.e., a 3D focus region based on the 2D focus region. The thus identified specifically focused 2D and 3D focus regions can then be output at 580. As discussed herein, a 3D focus region is a location where a specific type of focused anatomical information (determined based on the type of surgical tool detected) resides, such that 3D information in the 3D focus region can be retrieved and projected onto the 2D focus region to assist the surgeon in performing surgery using the surgical tool.

[0056] In some embodiments, other information near the tool position detected in the 3D model can also be retrieved and displayed to provide, for example, better surgical context. For example, when a hook is detected, the type of focus information can correspond to a blood vessel. Although a specific portion of the vascular tree facing the tool tip can be considered as specific focus information, 3D representations of other portions of the vascular tree near the tool can also be output and displayed. In addition, portions of the organ surrounding the blood vessels near the tool can also be output and displayed. The specific focus information can be displayed in a special manner (e.g., highlighted or colored or with higher contrast), and other related information surrounding the specific focus information can be displayed in a manner that does not interfere with or diminish the special effect of the specific focus information.

[0057] Figure 5C An exemplary high-level system diagram of a 2D / 3D focus region identifier 530 according to an embodiment of the present teachings is depicted. As discussed herein, 2D and 3D focus regions can be identified from a 2D image and a 3D model, respectively. A 2D focus region can be determined based on a detected input position near the tip of a surgical instrument. This 2D focus region can have a corresponding 3D focus region. In some embodiments, to identify a 3D focus region corresponding to a 2D focus region, significant anatomical features observed in the 2D focus region in the 2D image can be extracted and used to identify corresponding 3D anatomical feature points in the 3D model. The 2D / 3D focus region identifier 530 is provided to accomplish this via automatic or semi-automatic operation, as disclosed herein.

[0058] In the illustrated embodiment, the 2D / 3D focal region identifier 530 can take a 2D instrument tip position as input and generate both a 2D and 3D focal region as output. To facilitate operation, the 2D / 3D focal region identifier 530 includes a 2D anatomical feature detector 505, an operational model determiner 515, an automatic 3D corresponding feature identifier 525, a manual 2D / 3D feature selector 535, a 3D model rendering unit 545, and a focal region determiner 555. Figure 5D is a flow chart of an exemplary process of the 2D / 3D focal region identifier 530 according to an embodiment of the present teachings. In operation, when the 2D anatomical feature detector 505 receives a 2D instrument tip position as input, it may first align the 3D model with the 2D image via anatomical feature points. To this end, the 2D anatomical feature detector 505 may automatically detect 2D features at 562. In some embodiments, such 2D feature points may be obtained near or around the input tip position. In some cases, such 2D features may correspond to some anatomically distinct features, such as a concave point between the two halves of the liver (e.g., Figure 1A – Figure 1B and Figure 4B – Figure 4C The type of such feature points may also depend on the type of instrument detected. For example, if the instrument or tool detected is a surgical hook, this may indicate that the surgeon intends to treat a vessel near the tip of the tool. In this case, a branch point in the vessel tree may represent a better feature than a concave point between different parts of the liver. This is also true when such different features can help identify corresponding 3D features from a portion of the 3D model used for the 3D representation of the vessel.

[0059] In some cases, the input tip position may be such that no such different features exist. For example, the instrument tip may be close to the surface of the liver. In addition, in some cases, the detected 2D features may not be as good as needed. In the event that a sufficiently good 2D feature is not detected by the automatic device, the present teachings achieve high-quality 2D feature detection by means of manual assistance. The operating model determiner 515 can evaluate at 565 whether the automatically detected 2D features have the desired level of uniqueness or quality for identifying the corresponding 3D features. If so, the automatic operating mode is applied and the automatic 3D corresponding feature identifier 525 is activated, which then accesses the 3D model 230 at 567 and automatically detects 3D anatomical features corresponding to the detected 2D anatomical features from the 3D model at 569.

[0060] If it is determined at 565 that the 2D anatomical features are unsatisfactory (i.e., not detected or of poor quality), the operational model determiner 515 can control the process to identify the 2D features in manual mode by activating the manual 2D / 3D feature selector 535. The manual 2D / 3D feature selector 535 can control an interface for communicating with a user to facilitate manual identification of the 2D features by the user at 572. Such manually identified 2D anatomical features are then provided to the automatic 3D corresponding feature identifier 525 for accessing the 3D model at 567, and then automatically detecting the 3D anatomical features corresponding to the 2D features from the 3D model at 569. To ensure the quality of the identified 3D corresponding anatomical features, at 574, the automatic 3D corresponding feature identifier 525 can evaluate whether the identified 3D features are satisfactory based on some criteria. If they are satisfactory, both the 2D features and the corresponding 3D features can be fused to identify the 2D and 3D focus areas.

[0061] In the event that the 3D anatomical features are not satisfactory, the operation mode determiner 515 may activate the 3D model rendering unit 545 to render the 3D model so that the manual 2D / 3D feature selector 535 interacts with the user to manually identify 3D corresponding anatomical features from the rendered 3D model at 575. With satisfactory corresponding 3D anatomical features (automatically identified or manually selected), the 2D features and the corresponding 3D features may now be used to align (or orient) the 3D model with what is observed in the 2D image (consistent with the camera's perspective). That is, at 577, the 2D information observed in the 2D image (represented by, for example, the 2D anatomical features) is fused with the 3D information from the 3D model (represented by, for example, the corresponding 3D anatomical features) so that the focus region determiner 555 may then proceed to determine the 2D focus region and the corresponding 3D focus region accordingly based on the fused 2D / 3D information at 579. Figure 5A As shown in FIG, the focus information display unit 260 can then use the 2D and 3D focus areas to display tool-related information to the user to facilitate the surgical procedure.

[0062] Figure 6 FIG2 is an exemplary high-level system diagram of a focus information display unit 260 according to an embodiment of the present teachings. As disclosed herein, the function of the focus information display unit 260 is to visualize specific information dynamically determined based on the currently deployed surgical tool. These visualizations can then assist the user in performing specific procedures. As can be seen from the perspective of the tool tip, the display is performed near the tool tip. The focus information identified in this manner is intended to be consistent with the current purpose of the tool, allowing the focus information to be rendered to better facilitate the user in performing specific procedures.

[0063] In the illustrated embodiment, the focus information display unit 260 includes a registration mode determiner 650, a dynamic registration unit 600, and a focus overlay renderer 660. The registration mode determiner 650 may be provided to determine a registration mode before rendering 3D model information representing 3D focus information onto a 2D image. In some cases, the registration mode may be used to register a rigid body (e.g., when the organ being operated on is largely rigid, such as a bone). In other cases, during surgery, the registration mode may need to be directed to the registration of a deformable object (e.g., a heart), which may deform over time due to, for example, pumping blood or the patient's breathing. The dynamic registration unit 600 is provided for registering the 3D information to be rendered with the 2D image (including, for example, both focused and unfocused information) in the registration mode determined by the registration mode determiner 650. The focus overlay renderer 660 is provided for rendering the 3D focused and unfocused information in the 2D image based on the registration result.

[0064] In some embodiments, the dynamic registration unit 600 may further include a 2D focus registration feature extractor 610 for extracting 2D feature points to be used for registration, a 3D focus correspondence feature extractor 620 for identifying 3D feature points corresponding to the 2D feature points, a rigid body registration unit 630 provided for performing rigid registration if necessary, and a deformable registration unit 640 for performing deformable registration in a deformable registration mode. The registration results from the rigid registration or the deformable registration are sent to the focus overlay renderer 660 so that the focused and unfocused information of the 3D model can be rendered by appropriately projecting the 3D information onto the 2D image based on the registration results.

[0065] In some embodiments, the dynamic registration unit 600 can also be implemented using a model-based registration implementation, wherein a deep learning model (not shown) can be obtained via training data, so that the feature extraction and registration steps explicitly performed by the 2D focus registration feature extractor 610, the 3D focus correspondence feature extractor 620, the rigid body registration unit 630, and the deformable registration unit 640 can be replaced by being implicitly performed via the deep learning model, which incorporates the knowledge learned during training into the parameters of its embedding layer and other layers. Such a model-based solution can take a 2D image, a 3D model, and an identified 2D / 3D focus region of interest as input and generate a registration result as output to be provided to the focus overlay renderer 660.

[0066] Figure 7Ais a flowchart of an exemplary process of the focus information display unit 260 according to an embodiment of the present teachings. When the 2D focus registration feature extractor 610 receives the designated 2D / 3D focus region at 700, the 2D focus registration feature extractor 610 extracts registration feature points from the 2D image at 710. In some embodiments, feature points can be extracted from the designated focus region. In some embodiments, feature points used for registration can be extracted from any location on the target organ, regardless of whether these feature points are from the focus region of interest. In some embodiments, the 2D feature points used for registration can be manually identified by the user through interaction with the 2D image. For the identified 2D feature points, the 3D focus corresponding feature extractor 620 can extract 3D feature points corresponding to the 2D feature points from the 3D model 230 of the target organ at 720.

[0067] At the same time, at 730, the registration mode determiner 650 can determine the registration mode to be applied based on the type of surgical procedure issued. As discussed herein, in some cases, if the anatomical structure involved is rigid (e.g., bone), rigid registration can be performed, but in some cases, when the anatomical structure involved deforms over time, deformable registration can be used. When it is determined at 740 that rigid registration is applicable, at 750, the rigid registration unit 630 performs registration based on the 2D feature points and their corresponding 3D feature points. When deformable registration is applied, the deformable registration unit 640 performs deformable registration based on the 2D and 3D corresponding feature points at 760. After the registration is completed, the focus overlay renderer 660 renders the focus information at 770 by projecting the 3D focus information from the 3D model onto the 2D image based on the registration results. When presenting information about the detected tool surroundings, other information that is not focus information but is still near the surgical tool may also be rendered. As discussed herein, focused and non-focused information may be rendered differently such that focused information is displayed in a particular manner to create a clearer visual for the user, while non-focused information may be displayed in a manner that provides context without distracting from the focused information.

[0068] As discussed herein, there are various ways to render focused information. Any implementation can be used as long as the focused information provides a clear visual contrast to the surrounding non-focused information. In this disclosure, some exemplary methods for rendering focused information are provided, but these methods are for illustration only and not limitation. Figure 7BFIG2 is a flow chart of an exemplary process of a focus overlay renderer 660 for rendering focus information based on a detected tool identification according to an embodiment of the present teachings. When the focus overlay renderer 660 receives the registration results and the 2D / 3D focus region designation, the focus overlay renderer 660 retrieves the relevant information from the 3D model 230 and displays the 3D focus and non-focus information by projecting such information onto the 2D image according to the registration results at 705.

[0069] For example, when the detected surgical tool is a cutter scissors and the surgery is a laparoscopic liver resection, the identified focused information can be a portion of the liver surface having a portion of the resection boundary represented by the 3D model of the target liver. The portion of the resection boundary identified as focused information can be a portion of the resection trajectory facing the cutter's opening, determined based on the cutter's orientation. In this example, the unfocused information can include a portion of the modeled liver surface, similarly determined to be proximate to the cutter's opening position or orientation detected from the 2D image. When both focused and unfocused information are determined, their 3D representations from the 3D model are projected onto the 2D image. For example, a portion of the liver surface (unfocused but providing relevant information about the visual context of the focused information) is rendered, with a portion of the resection boundary (focused information) on the surgical trajectory superimposed as, for example, a cutting point at an appropriate location on the surface.

[0070] In some embodiments, the focused and unfocused information can be rendered in a special manner to enhance the quality of visual assistance to the user during surgery. As described herein, some enhanced display options can be provided to improve the quality of visual guidance. If, at 715, it is determined that the focus overlay renderer 660 does not support such enhanced display options, the display operation of the focus overlay renderer 660 ends at 725. Otherwise, the focus overlay renderer 660 can be configured by the user on a per-procedure basis to apply certain enhanced display options based on the user's needs. In this illustration, for example, the enhanced display process can check at 735 whether it is configured to dim the unfocused information or highlight the displayed focused information. If it is configured to dim the unfocused information (or even other background information in the 2D image), then at 745, the focus overlay renderer 660 dims the displayed unfocused information (and other portions of the 2D image). If it is configured to highlight the focused information, then the focus overlay renderer 660 modifies the display of the focused information at 755. This modification can be applied to make the focused information more visually prominent. Using the focus information above as an example of a portion of a cutout boundary, the cutout point of the portion projected on the 2D image can be rendered using a bright color or at maximum intensity. In some embodiments, dimming and highlighting can be applied ( Figure 7B(not shown) to further increase the contrast between the focused information and other information in the 2D image.

[0071] Another exemplary augmented display option may be to provide a magnified view of the portion of the 2D image where focus information is rendered. If, at 765, it is determined that the focus overlay renderer 660 is configured to do so, then, at 775, an area on the display screen may be identified for presenting a magnified view of the region of interest in the 2D image where focus information is rendered. Then, at 785, a magnified view of the content in the region of interest may be generated and displayed in the area of ​​the display screen for the magnified view. Figure 7C An exemplary display of focus information on a blood vessel 702 near the tip of a surgical hook 120, according to an embodiment of the present teachings, is illustrated in a magnified view 712 of the blood vessel 702. As shown, the magnified view 712, while containing the same information as 702, provides a better visualization to the user and, therefore, provides improved guidance to the user during surgery. Other enhanced display options may be present. If, at 795, it is determined that other enhanced display options are available, the focus overlay renderer 660 continues the process of generating an enhanced visualization of the focus information. If no other options are available, the process moves to the end of the process at 725.

[0072] Figure 8 is a schematic diagram of an exemplary mobile device architecture that may be used to implement a dedicated system for implementing the present teachings in accordance with various embodiments. In this example, the user device on which the present teachings may be implemented corresponds to a mobile device 800, including but not limited to a smartphone, a tablet computer, a music player, a handheld game console, a global positioning system (GPS) receiver, and a wearable computing device, or in any other form factor. The mobile device 800 may include one or more central processing units ("CPUs") 840, one or more graphics processing units ("GPUs") 830, a display 820, a memory 860, a communication platform 810 (such as a wireless communication module), a storage device 890, and one or more input / output (I / O) devices 850. Any other suitable components, including but not limited to a system bus or controller (not shown), may also be included in the mobile device 800. As Figure 8 As shown, a mobile operating system 870 (e.g., iOS, Android, Windows Phone, etc.) and one or more applications 880 can be loaded from storage device 890 into memory 860 for execution by CPU 840. Application 880 can include, at least in part, a user interface for information analysis and management according to the present teachings or any other suitable mobile application on mobile device 800. User interaction, if any, can be implemented via I / O device 850 and provided to various components connected via network(s).

[0073] In order to realize each module, unit and function thereof described in the present disclosure, computer hardware platform can be used as (multiple) hardware platform of one or more elements described herein.The hardware components, operating system and programming language of this type of computer are conventional in nature, and it is assumed that those skilled in the art are fully familiar with to adapt these technologies to the appropriate settings described herein.The computer with user interface element can be used to realize personal computer (PC) or other types of workstations or terminal equipment, but if suitable programming, computer can also serve as server.It is believed that those skilled in the art are familiar with the structure, programming and general operation of this type of computer equipment, so accompanying drawing should be self-explanatory.

[0074] Figure 9 800 is a schematic diagram of an exemplary computing device architecture that can be used to implement a dedicated system for implementing the present teachings according to various embodiments. Such a dedicated system in conjunction with the present teachings has a functional block diagram of a hardware platform including user interface elements. The computer can be a general-purpose computer or a special-purpose computer. Both can be used to implement a dedicated system for the present teachings. The computer 800 can be used to implement any component or aspect of the framework disclosed herein. For example, the information analysis and management methods and systems disclosed herein can be implemented on a computer such as computer 900 via the computer's hardware, software program, firmware, or a combination thereof. Although only one such computer is shown for convenience, the computer functions described herein in connection with the present teachings can be implemented in a distributed manner on several similar platforms to distribute the processing load.

[0075] The computer 900 includes, for example, a COM port 950 that is connected to a network connected to the COM port 950 and is connected from a network connected to the COM port 950 to facilitate data communication. The computer 900 also includes a central processing unit (CPU) 920 in the form of one or more processors for executing program instructions. The exemplary computer platform includes an internal communication bus 910, different forms of program storage and data storage (e.g., disk 970, read-only memory (ROM) 930, or random access memory (RAM) 940) for various data files to be processed and / or transferred by the computer 900 and possible program instructions to be executed by the CPU 920. The computer 900 also includes an I / O component 960 that supports input / output flow between the computer and other components in the computer (such as user interface elements 980). The computer 900 can also receive programming and data via network communications.

[0076] Thus, as described above, aspects of the information analysis and management methods and / or other processes may be embodied in programming. The programmatic aspects of the technology may be considered to be a "product" or "article of manufacture" typically in the form of executable code and / or associated data executed on or implemented in some type of machine-readable medium. Tangible, non-transitory "storage" type media include any or all of memory or other storage for a computer, processor, or the like, or its associated modules (such as various semiconductor memories, tape drives, disk drives, etc.) that may provide storage for software programming at any time.

[0077] All or part of the software may sometimes be delivered over a network, such as the Internet or various other telecommunication networks. Such communications, for example, may enable software to be loaded from one computer or processor to another, for example, in connection with information analysis and management. Thus, another type of medium that may carry software elements includes optical, electrical, and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical ground networks, and through various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, or the like, may also be considered to be the medium that carries the software. As used herein, unless limited to tangible "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to a processor for execution.

[0078] Thus, a machine-readable medium can take many forms, including but not limited to tangible storage media, carrier media, or physical transmission media. Non-volatile storage media include, for example, optical or magnetic disks that can be used to implement the system shown in the accompanying drawings or any of the components of the system, such as any of the storage devices or the like in any (multiple) computers. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wire and optical fiber, including the wires that form a bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, a DVD or DVD-ROM, any other optical medium, punched card stock tape, any other physical storage medium with a pattern of holes, RAM, PROM and EPROM, FLASH-EPROM, any other memory chip or cassette, a carrier wave that transports data or instructions, a cable or link that transports such a carrier wave, or any other medium from which a computer can read programming code and / or data. Many of these forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to a physical processor for execution.

[0079] Those skilled in the art will recognize that the teachings herein are amenable to various modifications and / or enhancements. For example, while the implementation of the various components described above may be embodied in a hardware device, it may also be implemented as a software-only solution, for example, installed on an existing server. Additionally, the technology disclosed herein may be implemented as firmware, a firmware / software combination, a firmware / hardware combination, or a hardware / firmware / software combination.

[0080] Although the foregoing has described what is considered to constitute the present teachings and / or other examples, it should be understood that various modifications may be made thereto, and the subject matter disclosed herein may be implemented in various forms and examples, and the teachings may be applied to many applications, only some of which have been described herein. It is intended that the appended claims claim any and all applications, modifications, and variations that fall within the true scope of the present teachings.

Claims

1. A method implemented on at least one processor, memory, and communication platform, the method comprising: receiving a two-dimensional (2D) image capturing a surgical instrument and anatomical structures associated with an organ to be operated on during the surgery; receiving an input specifying a 2D position of a surgical tool detected from the 2D image, wherein the surgical tool is attached to the surgical instrument for performing a surgical task and the position is near a tip of the surgical tool; determining, based on the detected surgical tool, a type of focused information to be displayed to assist a user in performing the surgical task using the surgical tool; determining, based on the position and via model fusion, a 2D focus region in the 2D image and a corresponding 3D focus region from a 3D model representing the organ and surrounding anatomical structures; A visual guide is created based on the type of focus information obtained from the 3D focus area, the visual guide projecting the type of focus information onto the 2D focus area to assist the user in performing the surgical task using the surgical tool.

2. The method according to claim 1, characterized in that The detected surgical tool is one from a list consisting of a scalpel and a surgical hook; and The type of focus information associated with the scalpel corresponds to information associated with the organ; and The type of focus information associated with the surgical hook corresponds to information associated with blood vessels near or in the organ.

3. The method according to claim 1, characterized in that The step of determining the 2D focus area and the 3D focus area comprises: Identify 2D features from 2D images; identifying, from the 3D model, a 3D feature corresponding to the 2D feature, wherein the 2D feature and the 3D feature satisfy some predetermined criteria; performing the model fusion based on the 2D features and the corresponding 3D features to align the 3D model relative to the detected position of the surgical tool; The 2D focus area and the 3D focus area are determined based on the model fusion result.

4. The method according to claim 3, characterized in that The 2D focus area in the 2D image is determined based on at least one of a type, a position, and an orientation of the detected surgical tool.

5. The method according to claim 1, wherein The steps of creating the visual guide include: projecting 3D focus information retrieved from the 3D focus area onto the 2D focus area in the 2D image; and The visual guide is generated based on the 2D image having projected 3D focus information therein.

6. The method of claim 5, further comprising identifying one or more 3D non-focused regions in the 3D model near the 3D focused region, wherein The one or more 3D non-focus regions are determined based on the detected surgical tool. 7 . The method of claim 6 , further comprising projecting 3D out-of-focus information from the one or more 3D out-of-focus areas onto the 2D image to provide a context for the 3D focused information.

8. The method according to claim 7, further comprising: The visual guidance is enhanced by performing at least one of the following: highlighting the presentation of the projected 3D focus information, dimming the presentation of the projected 3D out-of-focus information, and An enlarged view of the sub-region of the 2D image where the 3D focus information is projected is generated.

9. A machine-readable and non-transitory medium having information recorded thereon, wherein: The information, when read by the machine, causes the machine to perform the following steps: receiving a two-dimensional (2D) image capturing anatomical structures and surgical instruments associated with an organ to be operated on during the surgery; receiving an input specifying a 2D position of a surgical tool detected from the 2D image, wherein the surgical tool is attached to the surgical instrument to perform a surgical task and the position is near a tip of the surgical tool; determining, based on the detected surgical tool, a type of focused information to be displayed to assist a user in performing the surgical task using the surgical tool; determining, based on the position and via model fusion, a 2D focus region in the 2D image and a corresponding 3D focus region from a 3D model representing the organ and surrounding anatomical structures; A visual guide is created based on the type of focus information obtained from the 3D focus area, the visual guide projecting the type of focus information onto the 2D focus area to assist the user in performing the surgical task using the surgical tool.

10. The medium according to claim 9, characterized in that The detected surgical tool is one from a list consisting of a scalpel and a surgical hook; and The type of focus information associated with the scalpel corresponds to information associated with the organ; and The type of focus information associated with the surgical hook corresponds to information associated with blood vessels near or in the organ.

11. The medium according to claim 9, characterized in that The step of determining the 2D focus area and the 3D focus area comprises: Identify 2D features from 2D images; 3D features corresponding to the 2D features are identified from the 3D model, wherein The 2D features and the 3D features satisfy some predetermined criteria; performing the model fusion based on the 2D features and the corresponding 3D features to align the 3D model relative to the detected position of the surgical tool; The 2D focus area and the 3D focus area are determined based on the model fusion result.

12. The medium according to claim 11, characterized in that The 2D focus area in the 2D image is determined based on at least one of a type, a position, and an orientation of the detected surgical tool.

13. The medium according to claim 9, characterized in that The steps of creating the visual guide include: projecting 3D focus information retrieved from the 3D focus area onto the 2D focus area in the 2D image; and The visual guide is generated based on the 2D image having projected 3D focus information therein.

14. The medium according to claim 13, characterized in that When the information is read by the machine, the machine is further caused to perform the step of identifying one or more 3D non-focused areas near the 3D focused area in the 3D model, wherein the one or more 3D non-focused areas are determined based on the detected surgical tool.

15. The medium according to claim 14, characterized in that When the information is read by the machine, the machine is further caused to perform the step of projecting 3D non-focus information from the one or more 3D non-focus areas onto the 2D image to provide a background for the 3D focus information.

16. The medium according to claim 15, characterized in that The information, when read by the machine, further causes the machine to perform the step of enhancing the visual guidance by performing at least one of the following: highlighting the presentation of the projected 3D focus information; dimming a presentation of the projected 3D out-of-focus information; as well as An enlarged view of the sub-region of the 2D image where the 3D focus information is projected is generated.

17. A system comprising: A surgical tool-assisted model fusion mechanism, wherein the surgical tool-assisted model fusion mechanism is implemented by a processor and configured to: receiving a two-dimensional (2D) image capturing anatomical structures and surgical instruments associated with an organ to be operated on during the surgery, receiving an input specifying a 2D position of a surgical tool detected from the 2D image, wherein the surgical tool is attached to the surgical instrument for performing a surgical task and the position is near a tip of the surgical tool, determining, based on the detected surgical tool, a type of focused information to be displayed to assist a user in performing the surgical task using the surgical tool, and determining, based on the position and via model fusion, a 2D focus region in the 2D image and a corresponding 3D focus region from a 3D model representing the organ and surrounding anatomical structures; as well as and a focus information display unit implemented by a processor and configured to create a visual guide based on the type of focus information obtained from the 3D focus area, the visual guide projecting the type of focus information onto the 2D focus area to assist the user in performing the surgical task using the surgical tool.

18. The system according to claim 17, wherein: The detected surgical tool is one from a list consisting of a scalpel and a surgical hook; and The type of focus information associated with the scalpel corresponds to information associated with the organ; and The type of focus information associated with the surgical hook corresponds to information associated with blood vessels near or in the organ.

19. The system according to claim 16, wherein: The step of determining the 2D focus area and the 3D focus area comprises: Identify 2D features from 2D images; 3D features corresponding to the 2D features are identified from the 3D model, wherein The 2D features and the 3D features satisfy some predetermined criteria; performing the model fusion based on the 2D features and the corresponding 3D features to align the 3D model relative to the detected position of the surgical tool; The 2D focus area and the 3D focus area are determined based on the model fusion result.

20. The system according to claim 19, wherein: The 2D focus area in the 2D image is determined based on at least one of a type, a position, and an orientation of the detected surgical tool.

21. The system according to claim 17, wherein: The steps of creating the visual guide include: projecting 3D focus information retrieved from the 3D focus area onto the 2D focus area in the 2D image; and The visual guide is generated based on the 2D image having projected 3D focus information therein.

22. The system according to claim 21, wherein: The surgical tool-assisted model fusion mechanism is further configured to identify one or more 3D non-focused regions near the 3D focused region in the 3D model, wherein the one or more 3D non-focused regions are determined based on the detected surgical tool.

23. The system according to claim 6, wherein: The focus information display unit is further configured to project 3D non-focus information from the one or more 3D non-focus areas onto the 2D image to provide a context for the 3D focus information.

24. The system according to claim 23, wherein: The focus information display unit is further configured to enhance the visual guidance by performing at least one of the following: highlighting the presentation of the projected 3D focus information; dimming a presentation of the projected 3D out-of-focus information; as well as An enlarged view of the sub-region of the 2D image where the 3D focus information is projected is generated.

Images