Method and system for estimating 3d camera pose based on 2d masks and ridges and application in laparoscopic surgery

Abstract

This teaching relates to estimating 3D camera pose based on 2D features detected from 2D images. A virtual 3D camera pose is generated relative to a 3D model of a target organ and associated anatomical structures. Virtual 2D images are created by projecting the 3D model from a viewpoint determined based on the virtual 3D camera pose. Each virtual 2D image includes the 2D structure of the target organ and / or some 3D anatomical structures visible from the corresponding viewpoint. A 2D feature / camera pose mapping model is then obtained based on the 2D features extracted from the virtual 2D images and the corresponding virtual 3D camera pose, where the 2D features include 2D ridge lines projected from 3D ridges on the target organ represented in the 3D model.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Patent Application No. 18 / 528,475, filed December 4, 2023, entitled “METHOD AND SYSTEM FOR ESTIMATING A 3D CAMERAPOSE BASED ON 2D MASK AND RIDGES AND APPLICATION IN A LAPAROSCOPIC PROCEDURE”, the entire contents of which are incorporated herein by reference. Background Technology 1. Technical Field

[0002] This teaching generally deals with computers. More specifically, this teaching deals with signal processing. 2. Background Technology

[0003] With advancements in technology, an increasing number of tasks are now performed with the help of computers. Various industries benefit from these advancements, including the medical field, where computers can process vast amounts of image data, capturing anatomical information about patients to identify anatomical structures of interest (e.g., organs, bones, blood vessels, or abnormal nodules), obtaining measurements of each object of interest (e.g., the size of nodules growing within organs), and quantifying different anatomical structures (e.g., the size and shape of abnormal nodules). This information can be used for a variety of purposes, including preoperative planning and surgical guidance. Modern laparoscopic surgery can also leverage these advancements to obtain information during the procedure to facilitate navigation and guidance for the surgeon.

[0004] This is Figure 1A The diagram illustrates a setup for laparoscopic surgery where a patient 120 is on an operating table 110 and a laparoscopic camera 130 can be inserted into the patient's body to observe locations of interest, capturing, for example, surgical instruments 140 (also inserted into the patient's body) present at that location, the vicinity of organs, and other anatomical structures that may be close to the surgical instruments 140. Two-dimensional (2D) images captured by the laparoscopic camera 130 can be displayed (150) so that the surgeon can use them as visual guidance. The surgeon then mentally correlates what is seen in the 2D image (e.g., surgical instruments near the surface of an organ) with the actual three-dimensional (3D) object of interest (e.g., the liver to be removed during surgery) to determine which part of the organ is close to the surgical instruments, thereby figuring out how to manipulate the surgical instruments.

[0005] In laparoscopic surgery, 3D models representing organs of interest can be used to provide 3D information corresponding to what is seen in 2D images, enhancing the effectiveness of visual guidance. Such 3D models can represent both the physical structure of an organ (e.g., the liver) and its internal anatomical structures (e.g., blood vessels and nodules within the liver). If such a 3D model can be registered with what is seen in a 2D image, then the projection of such a 3D model at the registered location allows the surgeon to see all 3D objects around or inside the organ. This can provide valuable information to help the surgeon navigate surgical tools to achieve the intended task (e.g., nodule removal) in a way that avoids damage to other parts of the body, such as blood vessels.

[0006] To utilize 3D models for enhancement in laparoscopic surgery, 2D laparoscopic images need to be registered with the 3D model. In some cases, the surgeon or assistant can manually select 2D feature points from the 2D image and corresponding 3D points from the 3D model to facilitate registration. However, such manual methods are impractical in actual surgery because they are slow, cumbersome, and impossible to operate continuously with changing 2D images as surgical instruments move.

[0007] Therefore, a solution capable of addressing the aforementioned challenges is needed. Summary of the Invention

[0008] The teachings disclosed herein relate to methods, systems, and programming for information management. More specifically, these teachings relate to methods, systems, and programming related to hash tables and storage management using hash tables.

[0009] In one example, a method implemented on a machine having at least one processor, storage device, and communication platform is used to estimate a 3D camera pose based on 2D features detected from 2D images. A virtual 3D camera pose is generated relative to a 3D model of a target organ and associated anatomical structures. Virtual 2D images are created by projecting the 3D model from a viewpoint determined based on the virtual 3D camera pose. Each virtual 2D image includes the 2D structure of the target organ and / or some 3D anatomical structures visible from the corresponding viewpoint. A 2D feature / camera pose mapping model is then obtained based on the 2D features extracted from the virtual 2D images and the corresponding virtual 3D camera pose, wherein the 2D features include 2D ridge lines projected from 3D ridges on the target organ represented in the 3D model.

[0010] In various examples, a system for estimating 3D camera pose based on 2D features detected from 2D images is disclosed, and includes a camera pose generator and a 2D feature / camera pose mapping model generator. The camera pose generator is provided to generate virtual 3D camera poses relative to a previously constructed 3D model modeling a 3D target organ and its associated 3D anatomical structures, wherein each of the virtual 3D camera poses corresponds to a viewpoint for viewing the 3D model. The 2D feature / camera pose mapping model generator is provided to create virtual 2D images corresponding to the virtual 3D camera poses by projecting the 3D model according to the corresponding viewpoint, wherein each of the virtual 2D images includes 2D structures in the 2D projected target organ and / or some 3D anatomical structures visible from the corresponding viewpoint, and a 2D feature / camera pose mapping model is obtained based on 2D features extracted from the virtual 2D images and the corresponding virtual 3D camera poses. The 2D features include 2D ridge lines projected from 3D ridges on the target organ represented in the 3D model.

[0011] Other concepts relate to software used to implement this teaching. A software product according to these concepts includes at least one machine-readable non-transient medium and information carried by that medium. The information carried by the medium may be executable program code data, parameters associated with the executable program code, and / or user-related information, requests, content, or other additional information.

[0012] Another example is a machine-readable, non-transient, and tangible medium on which information is recorded for estimating 3D camera pose based on 2D features detected from 2D images. A virtual 3D camera pose is generated relative to a 3D model of the target organ and associated anatomical structures. Virtual 2D images are created by projecting the 3D model from a viewpoint determined based on the virtual 3D camera pose. Each virtual 2D image includes the 2D structure of the target organ and / or some 3D anatomical structures visible from the corresponding viewpoint. Then, based on the 2D features extracted from the virtual 2D images and the corresponding virtual 3D camera pose, a 2D feature / camera pose mapping model is obtained accordingly. The 2D features include 2D ridge lines projected from 3D ridges on the target organ represented in the 3D model.

[0013] Additional advantages and novel features will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art upon review of the following description and the accompanying drawings, or may be learned by example from production or operation. The advantages of this teaching can be realized and obtained by practice or by using various aspects of the methods, tools, and combinations set forth in the detailed examples discussed below. Attached Figure Description

[0014] The methods, systems, and / or programming described herein will be further described through exemplary embodiments. These exemplary embodiments will be detailed with reference to the accompanying drawings. These embodiments are non-limiting exemplary embodiments, wherein the same reference numerals denote similar structures throughout several views of the drawings, and wherein:

[0015] Figure 1A An exemplary setup for laparoscopic surgery is shown;

[0016] Figure 1B An exemplary 3D model of an organ and the viewpoint from which the 3D model is viewed from a 3D camera pose are shown;

[0017] Figure 1C The image shows a 2D image created by projecting a 3D model onto a 2D plane based on the viewpoint determined by the pose of a 3D camera.

[0018] Figure 1D An exemplary 3D model of an organ having a 3D surface and an identified spine, according to an embodiment of this teaching, is shown.

[0019] Figure 1E-1F The illustrations show different types of 3D features representing organs according to embodiments of this teaching;

[0020] Figure 1G This shows exemplary 2D features that can be observed when a 3D-modeled organ is projected onto a 2D plane;

[0021] Figure 2 A 3D camera pose estimation framework according to embodiments of this teaching is depicted, and an exemplary high-level system diagram of its use is provided.

[0022] Figure 3A This is a flowchart illustrating an exemplary process for obtaining a model for mapping 2D features to 3D camera pose within a 3D camera pose estimation framework, according to embodiments of this teaching.

[0023] Figure 3B This is a flowchart illustrating an exemplary process for estimating the pose of a 3D laparoscopic camera based on 2D features extracted from 2D laparoscopic images using a camera pose mapping model, according to an embodiment of this teaching.

[0024] Figure 4A An exemplary high-level system diagram depicts a 2D feature / camera pose mapping model generator according to embodiments of this teaching;

[0025] Figure 4B An exemplary correspondence between 2D features and 3D camera poses that can be used to construct a mapping model according to embodiments of this teaching is shown;

[0026] Figure 4CThe image shows encoded 2D features and 3D camera poses to be used to construct a mapping model according to an embodiment of this teaching;

[0027] Figure 5A An exemplary scheme is shown for decoding encoded 2D features to reconstruct 2D features for camera pose estimation refinement, according to embodiments of the present teachings.

[0028] Figure 5B An exemplary type of mapping model that can be trained based on discrete training data according to embodiments of this teaching is shown;

[0029] Figure 5C This is a flowchart of an exemplary process for a 2D feature / camera pose mapping model generator according to embodiments of this teaching;

[0030] Figure 6A An exemplary high-level system diagram depicts a camera pose estimator according to embodiments of this teaching;

[0031] Figure 6B This is a flowchart illustrating an exemplary process of a camera pose estimator according to an embodiment of this teaching;

[0032] Figure 7 These are schematic diagrams of exemplary mobile device architectures that can be used to implement dedicated systems for carrying out the present teachings, according to various embodiments; and

[0033] Figure 8 This is a schematic diagram of an exemplary computing device architecture that can be used to implement a dedicated system for carrying out the present teachings, according to various embodiments. Detailed Implementation

[0034] In the following detailed description, numerous specific details are illustrated by way of example to facilitate a thorough understanding of the teachings. However, it will be apparent to those skilled in the art that these teachings can be practiced without such details. In other instances, well-known methods, processes, components, and / or systems have been described at a relatively high level without detail in order to avoid unnecessarily obscuring various aspects of these teachings.

[0035] This teaching discloses exemplary methods, systems, and implementations of a framework for estimating 3D camera pose based on 2D image features detected from laparoscopic images, as well as exemplary applications in laparoscopic surgery. 3D models of organs can be constructed to represent the organ and various anatomical structures present in or near the organ in terms of their physical appearance (such as size, volume, shape) and structural features (such as their spine). Such 3D models can be used to generate projections of different perspective views of relevant portions in a 2D plane. Each perspective view can be determined based on the corresponding 3D camera pose.

[0036] In some embodiments, different 3D camera poses may be assumed, each of which can be used to determine a viewpoint corresponding to the 3D camera pose, which can be used to render a 3D model on a 2D plane to create a projection. 2D features of the projected 3D model appearing on the 2D plane can be detected and utilized to obtain a mapping from 2D features to the 3D camera pose. Figure 1B An example 3D model 160 of an organ and a hypothetical 3D camera pose 170 are shown, which can be used to determine the viewing angle of the 3D model 160. Figure 1C A 2D image plane 180 is shown, and a 2D projection 190 of the 3D model 160 onto the image plane 180 is shown according to the viewpoint determined by the 3D camera pose 170. Such a projection 190 corresponds to a 2D structure with different features (such as shape, size, etc.). When 2D features appear in the 2D image plane 180, the 2D features can correspond to their 3D properties. Figure 1D The illustration shows some exemplary 3D features of a liver modeled in 3D model 160, including the 3D surface of liver 160-1 and ridge 160-2. When model 160 is positioned perpendicular to... Figure 1D When the viewpoint is projected onto a 2D image plane, these 3D features can be rendered in the 2D image plane, such as... Figure 1E (Surface of the liver model) and 1F (ridge of the liver model) are shown.

[0037] As discussed in this paper, 2D projection based on a 3D model can first obtain a segmentation of the object of interest (e.g., liver). Based on the segmentation results, different 2D image features can be extracted from the segmented object. Figure 1GExemplary types of 2D features that can be extracted from a projection of a 3D model are shown, including features associated with a segmented object of interest, such as intensity-related features (e.g., texture or color) within the segmented region, and geometric features, such as the contour or shape of the segmented region. Additional features, such as ridges along the surface of a liver, can also be extracted from the segmented region. Such 2D features detected from the projected 2D image and the 3D camera pose used to determine the viewpoint for projection can form a correspondence. When multiple 3D camera poses are used to generate corresponding 2D projections and 2D features extracted from them, the corresponding 3D camera poses and detected 2D features can be used to create a discrete mapping model between the detected 2D features and the 3D camera poses. The discrete mapping model can be constructed as a lookup table (LUT) with pairs of 2D features and 3D camera poses. Multiple LUTs can be constructed, each of which can be based on different 2D features. For example, a LUT can be used to discretely map 2D intensity features to 3D camera poses. Another LUT can be used to map 2D geometric features, such as shape, to poses. Another LUT can map a combination of 2D masks and ridge features to a 3D camera pose. Although such discrete mappings are not continuous, in each case, it can identify the closest mapping through approximation. The estimated 3D camera pose obtained via approximation can optionally be refined or optimized to improve the accuracy of the 3D camera pose.

[0038] In various embodiments, a continuous mapping model can be constructed by training a mapping model, for example via machine learning, using discrete 2D feature / 3D camera pose mappings as training data. This allows the model to learn the complex relationship between 2D features and 3D camera poses, enabling it to map any given set of 2D features to some candidate 3D camera poses. Such a continuous mapping model can output multiple discrete outputs, each of which can correspond to a 3D camera pose with a score (e.g., probability) indicating the confidence level of the estimated 3D camera pose. In another embodiment, the multiple outputs of the trained mapping model can correspond to different degrees of freedom associated with the 3D pose. Such a continuous mapping model can also have multiple outputs, each of which can be associated with an estimated pose or dimensional parameter and has a corresponding confidence score associated with it.

[0039] As discussed in this paper, different discrete mapping LUTs can be obtained, each of which can be based on different types or combinations of 2D features. This also applies to continuous mapping models. For example, a trained mapping model can be used to map a combination of a mask and ridge-related 2D features to a 3D camera pose. Different mapping models can be trained to map 2D geometric features of objects detected from a 2D image to a 3D pose. Yet another type of mapping model can be trained to map combinations of features (e.g., related to intensity and geometry) to a 3D camera pose. In each application scenario, an appropriate type of model (e.g., a mapping based on 2D masks and ridge features) can be invoked to estimate the underlying 3D camera pose. In some applications, more than one type of model (e.g., a model based on geometry, intensity, and ridge features) can be invoked to estimate the 3D pose, and the estimates from the different models can then be combined in some way to derive an overall estimate of the 3D camera pose. In some implementations, 2D features detected from 2D projection can be encoded, enabling the mapping between 2D features and 3D camera poses to be performed based on the code of the 2D features. Since code can be much lighter than 2D features such as masks or ridges, the mapping model trained on the code can also be computationally more efficient, allowing the process of estimating 3D camera poses based on the code of 2D features to be performed more efficiently. An encoding scheme for generating such code for 2D features can be determined, enabling the 2D features to be reconstructed in the 2D image plane when needed.

[0040] As discussed herein, the output from a continuous mapping model can be multiple 3D camera poses, each associated with a score such as a probability indicating confidence in the estimate. To determine the 3D camera pose, in some embodiments, the estimate with the highest confidence score can be selected as the estimated 3D camera pose. Other implementations may also derive the final 3D camera pose estimate. In some embodiments, multiple (e.g., K) 3D camera pose estimates can be combined to generate an aggregated 3D pose estimate. For example, the top K estimates with, for example, sufficient confidence scores can be aggregated in a weighted sum manner to generate the final 3D camera pose estimate. In some cases, the weights applied to individual estimates can be obtained based on their ranking, which is determined, for example, based on the associated confidence score. Aggregation can be performed by taking a weighted sum of parameters in each dimension (each degree of freedom).

[0041] The 3D camera pose estimated based on 2D features such as masks and ridges, according to this teaching, can be considered an initial estimate and can optionally be further optimized or refined. According to this teaching, in some embodiments, differential rendering can be used to facilitate optimization. Based on the initially estimated 3D camera pose, a 3D model 160 can be rendered using slightly perturbed rendering parameters (such as parameters that have been slightly shifted or rotated) to create a differential projection result. During iterative optimization, each of the differential rendering results can be evaluated against a loss function defined relative to the pose parameters (e.g., 6 degrees of freedom), allowing the 3D pose-related parameters to be iteratively adjusted until convergence. The refined or optimized 3D camera pose estimate can then be used as the estimated 3D camera pose.

[0042] Trained 2D feature / 3D camera pose mapping models, whether as discrete LUTs or in continuous form, can be deployed in various applications. In one example, such models can be used in laparoscopic surgery on organs to estimate the 3D pose of the laparoscopic camera based on 2D features extracted from laparoscopic images. The estimated 3D camera pose can then be combined with a 3D model of the organ to determine the viewing angle for projecting the 3D model onto a display, thereby providing 3D visual guidance aligned with what is seen in the 2D laparoscopic image. In some embodiments, the display may correspond to a superimposed 3D projection of the 3D model. In some embodiments, a separate screen can be rendered using a 3D model projected alongside the laparoscopic image. Such projections of the 3D model can also include different anatomical structures beneath the organ surface, allowing the aligned 3D model to provide effective visual assistance to the surgeon during laparoscopic surgery. The following references... Figure 2 – Figure 6B Details related to this teaching on estimating 3D camera pose based on 2D image features are provided.

[0043] Figure 2An exemplary high-level system diagram of a 3D camera pose estimation and use case framework 200 according to embodiments of this teaching is depicted. Framework 200 comprises two parts: a preoperative part and an intraoperative part. The preoperative part is provided for building a 2D feature / camera pose mapping model 240 and includes a camera pose generator 210 and a 2D feature / camera pose mapping model generator 230. In the illustrated embodiment, the camera pose generator 210 is provided for generating a series of hypothetical 3D camera poses based on a camera pose generation configuration 220 (which may specify, for example, a resolution for generating 3D camera poses). The generated 3D camera poses can then be provided to the 2D feature / camera pose mapping model generator 230, which can utilize the series of hypothetical 3D camera poses to generate a 2D feature / camera pose mapping model of a 3D model 160 of an organ.

[0044] Specifically, as discussed herein, in order to create the mapping model 240, for each of the hypothetical 3D camera poses generated by the camera pose generator 210, the corresponding viewing angle is determined (see [link to documentation]). Figure 1B (An example viewpoint determined based on the 3D camera pose 170) and used to project the 3D model 160 onto a 2D image plane, such as Figure 1C On the image plane shown. Based on projection, 2D features are extracted (e.g., 2D masks for the object of interest and ridges on the object of interest) and then used to pair with the 3D camera pose used for projection. In this way, a pair of 2D features and 3D camera poses is created and used to derive a 2D feature / 3D camera pose mapping model 240. As discussed herein, in some embodiments, this pairing can be used to construct a discrete model as a lookup table (LUT). This works well when the resolution used to generate the 3D camera pose is sufficiently fine, resulting in relatively accurate 3D camera poses estimated via the LUT. In other embodiments, the created pair of 2D features and 3D camera poses can be used as training data for training during a machine learning process to obtain a continuous mapping model as discussed herein. Such a obtained mapping model 240 can then be used intraoperatively to estimate the 3D camera pose based on 2D features extracted from 2D images acquired during surgery.

[0045] Figure 3AThis is a flowchart illustrating an exemplary process for the first part of a framework 200 for deriving a model that maps 2D features to 3D camera poses, according to embodiments of this teaching. First, a 3D model 160 of the relevant organ is retrieved at 300. To generate the mapped model, a camera pose generator 210 generates various virtual 3D camera poses at 315 based on some generation parameters (e.g., resolution, etc.) specified in configuration 220. Such generated virtual 3D camera poses are provided to a 2D feature / camera pose mapping model generator 230, which at 320 projects the 3D model 160 onto different viewpoints determined according to the virtual 3D camera poses to obtain corresponding 2D projected images. Then, at 325, 2D features can be extracted from each of the 2D projected images and used relative to the corresponding virtual 3D camera poses to create a pairing or mapping between the 2D features and each corresponding 3D camera pose at 330. The mappings created in this way correspond to discrete pairs and can be used to derive discrete or continuous mapping models. In practice, the operation of deriving the 2D feature / camera pose mapping model generator 230 can differ depending on the type of model to be created (e.g., discrete or continuous). For example, the mapping created between a virtual 3D camera pose and extracted 2D features can be directly used as a discrete mapping model. On the other hand, to derive a continuous mapping model, such discrete mappings can be used as training data for machine learning against their respective continuous mapping models.

[0046] The second part of framework 200 is used to apply such a derived mapping model to estimate the 3D camera pose based on 2D features identified from 2D images acquired during a medical procedure. In some embodiments, the acquired 2D images may correspond to laparoscopic images obtained via a laparoscopic camera, and the task is to estimate the 3D pose of the laparoscopic camera. Figure 2 As shown, the second part of frame 200 includes an intraoperative monitor 250, a camera pose estimator 260, and a pose-based 3D model renderer 270. The intraoperative monitor 250 is provided in the operating room for displaying a laparoscopic camera (such as...) inserted into the patient's body. Figure 1A The 2D images acquired by the camera 130 shown. One of the acquired 2D images can be selected and processed by the camera pose estimator 260 to segment the object of interest (e.g., an organ such as the liver) and its 2D features for estimating the camera's 3D pose. In some embodiments, when a laparoscopic camera acquires a 2D image (video) stream via interaction with a surgeon, a particular 2D image can be identified as a selected 2D image for estimating the corresponding 3D camera pose. For example, a laparoscopic image of satisfactory quality can be selected for estimating the 3D camera pose.

[0047] According to this teaching, a camera pose estimator 260 is provided for estimating a 3D camera pose relative to a selected 2D image based on a 2D feature / camera pose mapping model 240. Utilizing the estimated 3D camera pose from the camera pose estimator 260, a pose-based 3D model renderer 270 is provided to determine the viewpoint for projecting a 3D model 160 of the organ in question using the estimated 3D camera pose. This creates a 3D rendering of the model 160 aligned with the selected laparoscopic image. The rendered 3D model 160 provides more effective visual guidance to the surgeon, not only because it is aligned with the laparoscopic image, but also because it reveals the internal anatomical structures of the organ that are not visible from the 2D laparoscopic image. As discussed herein, the 3D model 160 can be rendered by overlaying it onto a 2D image. In other embodiments, the 3D model 160 can be rendered on a separate display, e.g., on a different display window of the same display showing the 2D image, or on different display devices. This type of rendering can be displayed side-by-side with 2D images.

[0048] Figure 3B This is a flowchart illustrating an exemplary process for estimating the 3D laparoscopic camera pose for aligning a 3D model with a laparoscopic image during surgery, according to embodiments of this teaching. In laparoscopic surgery, the camera pose estimator 260 first receives a selection of a 2D laparoscopic image at 305. Then, at 315, the selected laparoscopic image is processed to segment an object of interest (e.g., liver). Based on the segmentation result, a 2D mask and ridge associated with the segmented object are obtained at 325 and used as 2D features at 335 to estimate the 3D pose of the laparoscopic camera based on the 2D feature / camera pose mapping model 240. As discussed herein, in some embodiments, the 2D feature / camera pose mapping model 240 may be trained to take the code of the 2D features as input. In this case, the 2D features (including masks and ridges) can be encoded first to generate code, which is then fed as input to the mapping model 240 to obtain the estimated 3D camera pose.

[0049] As discussed herein, depending on the type of mapping model, an initial 3D camera pose can be generated as the output of the camera pose estimator 260. In some embodiments, alternatively, the camera pose estimator 260 may optionally further optimize the initial 3D camera pose estimate to produce an optimized estimated 3D camera pose. The pose-based 3D model renderer 270 can then use the 3D camera pose estimate (initial or optimized) from the camera pose estimator 260 to determine the rendering viewpoint at 345 based on the estimated 3D camera pose, and then project the 3D model 160 of the organ onto a display at 355 according to that viewpoint. The following references... Figure 4A – Figure 6B Details related to the camera pose estimator 270 are provided.

[0050] Figure 4A An exemplary high-level system diagram depicts a 2D feature / camera pose mapping model generator 230 according to an embodiment of this teaching; as discussed herein, the 2D feature / camera pose mapping model generator 230 is provided to create a mapping model 240 between 2D features detected from laparoscopic images or codes encoded thereto and 3D camera poses via virtual 2D images created in different viewpoints corresponding to a hypothetical 3D camera pose. The generator 230 can take a virtual camera pose (from a camera pose generator 210) as input and generate the 2D feature / camera pose mapping model 240. The illustrated embodiment is provided for deriving discrete or continuous mapping models and includes a model generation controller 400, a camera pose-based 3D model projector 410, a 2D projection mask recognizer 420, a 2D projection ridge extractor 430, a mapping data generator 440, and a machine learning engine 460.

[0051] A model generation controller 400 is provided to take an input virtual camera pose as input and control the generation of a 2D feature / camera pose mapping model 240 accordingly. Based on each input virtual 3D camera pose, a camera pose-based 3D model projector 410 is invoked to determine the corresponding projection viewpoint based on the input virtual camera pose, and then projects a 3D model 160 according to the corresponding viewpoint to generate a 2D virtual projected image. Such virtual projected images can then be used by a 2D projection mask recognizer 420 to identify masks of objects of interest (e.g., liver), and by a 2D projection ridge extractor 430 to extract ridges present therein. A mapping data generator 440 can be provided to take the camera pose and 2D features (masks and ridges) detected from the projected 2D images as input, and form pairs between the 2D features and the 3D camera pose. Mappings can be formed based on multiple such pairs based on multiple input virtual camera poses and corresponding 2D features extracted from the corresponding projected 2D virtual images.

[0052] Figure 4B An exemplary pairing of 2D features and camera poses according to embodiments of this teaching is shown. Figure 4B As shown, each pair corresponds to a row including 2D features 470 and 3D camera pose 480. Figure 4B In the illustrations shown, each 3D camera pose can be represented as a tuple with six parameters (X, Y, Z, p, r, y) corresponding to six degrees of freedom, including (X, Y, Z) representing coordinates in 3D camera space, and (p, r, y) representing orientations measured according to pitch (p), roll (r), and yaw (y), respectively. In some embodiments, 2D features can be encoded such that pairing can occur between 2D feature code 490 and 3D camera pose 480, as in embodiments according to this teaching. Figure 4C As shown in the illustration, 2D features detected from a 2D image (e.g., masks and ridges) can be combined into a feature vector, which can be encoded to generate codes and then paired with corresponding 3D camera poses. Using such encoded 2D features makes the 2D feature / camera pose mapping model 240 more efficient because it eliminates the need to use images with 2D features to pair with 3D camera poses. This is especially true when the mapping model 240 is built based on millions of pairings.

[0053] In some embodiments, it may be necessary to reconstruct 2D features by decoding code. For example, mapping model 240 may generate multiple candidate camera pose estimates with different confidence levels, and 2D features may be needed to facilitate selection from such initial outputs. The 2D features can be encoded using an appropriate encoding scheme, allowing the underlying 2D features to be efficiently reconstructed based on the code. Figure 5A An exemplary process for encoding 2D features to generate code during the construction of mapping model 240 is depicted, and the code can be used to reconstruct the 2D features via decoding when needed. As shown, mask image 510 and ridge image 520 can be processed by encoder 530 to generate feature vector code 540 to efficiently represent 2D features 510 and 2D features 520. When needed, such code 540 can be decoded by decoder 550 to generate reconstructed mask image 560 and reconstructed ridge image 570. In some embodiments, the reconstructed image (e.g., reconstructed mask image 560) can be used, for example, to compare with a mask image obtained based on laparoscopic images to evaluate similarity.

[0054] The pairing between 2D features (or their codes) and 3D camera poses can be directly used as a LUT for a discrete mapping model. In this case, the mappings created by the mapping data generator 440 based on the pairings can be stored as a 2D feature / camera pose mapping model 240. In some embodiments, these paired mappings can be used by the machine learning engine 460 as training data 450 for machine learning to obtain a continuous 2D feature / camera pose mapping model 240.

[0055] Even though this paper discloses 2D feature masks and ridges for illustration purposes, other 2D features can also be used to pair with 3D camera poses to derive a mapping model 240 as needed for any application at hand. Figure 5B Different 2D feature / camera pose mapping models that can be derived based on 2D features according to embodiments of this teaching are illustrated. For example, mapping models can be obtained based on individual types of 2D features to create discrete or continuous models (e.g., individual mask-based mapping models or ridge-based mapping models). Mapping models can also be obtained based on combinations of different 2D features (e.g., both masks and ridges).

[0056] Figure 5C This is a flowchart of an exemplary process for a 2D feature / camera pose mapping model generator 230 according to an embodiment of this teaching. In operation, when an input virtual 3D camera pose is received at 505, the pattern generation controller 400 sends the camera pose to a camera pose-based 3D model projector 410, which determines the corresponding viewpoint for projection at 515 and projects a 3D model 160 accordingly at 525 to generate a 2D projected virtual image. Then, a 2D projected mask recognizer 420 can identify the mask of the object of interest in the 2D projected virtual image at 535, and a 2D projected ridge extractor 430 can extract the ridge of the object of interest from the 2D projected virtual image at 545. Then, at 555, the obtained mask and ridge can be paired with the corresponding 3D camera pose by a mapping data generator 440. Depending on the operating mode controlled by the model generation controller 400, such generated mapping data can be directly stored in 240 as a discrete mapping model. If the operating mode is to generate a continuous mapping model, the mapping data generated by generator 440 can then be stored as training data in 450 (see [link to documentation]). Figure 4A Then the training data can be used by the machine learning engine 460 to learn via machine learning at 565, so as to obtain a continuous 2D feature / camera pose mapping model 240 at 575.

[0057] Figure 6AAn exemplary high-level system diagram of a camera pose estimator 260 according to an embodiment of the present teachings is depicted. In the illustrated embodiment, two parts may be included. A first part may be provided for estimating an initial 3D camera pose based on a 2D mask and ridge features. A second part may be provided as an option to optimize the initially estimated 3D camera pose to derive a refined 3D camera pose estimate. The first part may include a mask detection unit 600, a ridge detection unit 610, a top K camera pose candidate determiner 620, and a similarity-based selector 640, and optionally a 2D feature reconstructor 650.

[0058] Given an input laparoscopic image, mask detection unit 600 and ridge detection unit 610 are used to identify the mask of the object of interest and extract the ridges associated with the object, respectively. These detected 2D features are then used by top K camera pose candidate determiner 620 to estimate the top K camera pose candidates based on 2D feature / camera pose mapping model 240. In some embodiments, when the 2D feature / camera pose mapping model 240 corresponds to a LUT, the top K camera pose candidates can be identified based on the best match of the 2D features. For example, the top 5 camera poses can be obtained by selecting the 5 rows that produce the closest matches to the detected 2D features. If the mapping model 240 is a continuous model, the top K camera pose candidates can correspond to those with the top 5 highest rank confidence levels. In some embodiments, if the 2D feature / camera pose mapping model 240 operates based on encoded 2D features (i.e., codes), the first K camera pose candidate determiner 620 may first encode the detected 2D features to obtain codes, and then operate on the codes to derive estimated 3D camera pose candidates based on the mapping model 240.

[0059] The top K candidate camera poses provide a range of choices from which one is identified as the initial 3D camera pose estimate. To achieve this, the top K camera pose candidates, each with a corresponding set of top K 2D features (or their codes), can be provided to a similarity-based selector 630 for selection. In some embodiments, this can be achieved by comparing 2D features detected from a given input laparoscopic image with 2D features from a mapping model 240. When the 2D features presented by the mapping model 240 are codes, a 2D feature reconstructor 650 can be invoked first to reconstruct the 2D features based on the codes before comparing them with the 2D features detected from the laparoscopic image. Through this comparison, the camera pose estimate with the best-matching 2D feature pairing from the laparoscopic image can then be selected as the initial 3D camera pose estimate. The first part of the camera pose estimator 260 outputs the initial camera pose estimate.

[0060] As discussed herein, a second portion of the camera pose estimator 260 may optionally be provided to refine the initial camera pose estimate to generate an optimized 3D camera pose estimate. The second portion includes a camera pose estimation optimizer 660, a pose-based differential projection unit 670, and a similarity-based selector 640. In some embodiments, the operation of the camera pose estimation optimizer 660 can be controlled according to an operating mode specified in 630. In some cases, operating mode 630 may be configured to no optimization, such that the initial camera pose estimates selected from the first K candidates can be directly output as the estimated 3D camera pose.

[0061] When operating mode 630 is configured to further optimize the initial 3D camera pose estimation, further optimization can be performed based on the optimization parameters specified in 630. In some embodiments, optimization can be based on differential projection using perturbed pose parameters (with respect to different degrees of freedom, including coordinates, their pitch, roll, and yaw) and similarity-based 2D feature comparison. Based on the initial camera pose estimation, a camera pose estimation optimizer 660 can be provided to generate a perturbed camera pose according to the optimization parameters specified in 630 (e.g., the range and resolution of the perturbation for each degree of freedom). A pose-based differential projection unit 670 can be invoked to create a differential 2D mask / ridge image via differential projection of a 3D model 160 using the perturbed camera pose. In some embodiments, an optimization scheme may be deployed that selects the optimal perturbed 3D camera pose corresponding to a differential 2D mask / ridge image, which produces, for example, maximum similarity to features extracted from an input laparoscopic image (evaluated by, for example, a similarity-based selector 540).

[0062] Figure 6B This is a flowchart of an exemplary process of a camera pose estimator 260 according to an embodiment of this teaching. In operation, when an input 2D laparoscopic image is received at 605, a mask detection unit 600 segments the 2D laparoscopic image at 615 to identify a mask corresponding to an object of interest (e.g., liver), and a ridge detection unit 610 extracts the ridges associated with the object of interest at 625. Based on the obtained 2D mask / ridge, a top K camera pose candidate determiner 620 estimates the top K camera poses at 635 based on a 2D feature / camera pose mapping model 240. As discussed herein, in some embodiments, the 2D features used for mapping may first be encoded as code (when the 2D feature / camera pose mapping model 240 is constructed based on the encoded 2D features) and then used to obtain the top K candidates via the 2D feature / camera pose mapping model 240 (e.g., LUT or continuous model).

[0063] At 645, the similarity-based selector 640 can select an initial camera pose estimate from the top K camera pose candidates based on the similarity between the 2D features of the top K camera pose candidates and the 2D features extracted from the input laparoscopic image. In some embodiments, if the mapping model 240 is constructed using encoded 2D features (i.e., code), the 2D feature reconstructor 650 can be invoked first to decode the code of the top K camera pose candidates to obtain the reconstructed 2D features of the top K candidates, which are then used to evaluate the similarity with the 2D features of the input laparoscopic image. Depending on the operating mode configured in 630, if it is determined at 655 that no additional optimization is required, the initial camera pose selected at 695 is output as the estimated 3D camera pose for the laparoscopic camera. Otherwise, the camera pose estimation optimizer 660 can continue with further optimization by first generating a perturbed camera pose at 665 based on the initial camera pose estimation according to the operating mode configuration 630 (specifying, for example, the range and resolution of perturbations in different dimensions). This perturbed camera pose is then used by the pose-based differential projection unit 670 to obtain a differential 2D mask / ridge image at 675. This can be achieved by projecting the 3D model 160 using the viewpoint determined based on the perturbed camera pose. The 2D features of such differential 2D mask / ridge images can then be evaluated in terms of their similarity to the 2D features of the input laparoscopic image, and at 685, one of the perturbed camera poses can be selected as the optimal 3D camera pose estimation when the corresponding differential 2D mask / ridge image in that perturbed camera pose produces the maximum similarity to the 2D features of the input laparoscopic image. This optimized 3D camera pose estimation can then be output at 695.

[0064] like Figure 2 As shown in Figure A, the pose-based 3D model renderer 280 can use estimated 3D camera poses to render a 3D model 160 to provide effective visual guidance to the user during medical procedures. Depending on the application requirements, the pose-based 3D model renderer 280 can be configured to render the 3D model 160 in different ways. In some cases, different stages of the same procedure can be configured differently to render the 3D model 160 to show different types of information. For example, to remove a tumor inside the liver, the surgeon may first clamp some of the major blood vessels connected to the liver to prevent blood loss when cutting open the liver to remove the tumor. Once the external blood vessels are clamped, the surgeon may need to see the internal anatomy, including the tumor and the blood vessels supplying blood to it. At this point, the surgeon may want to view what lies beneath the surface of the liver via 3D model rendering, so that the 3D model 160 can be rendered to show all the anatomical structures and their spatial relationships to help the surgeon manipulate surgical instruments.

[0065] Thus, the pose-based 3D model renderer 280 can be configured to render a 3D model 160 based on the estimated 3D camera pose, according to the needs during different stages of laparoscopic surgery. For example, while surgical instruments are still approaching an object of interest (such as the liver), the 3D model 160 can be rendered to show the liver in terms of its physical properties (e.g., shape and size) and its surrounding anatomical structures (such as nearby blood vessels or bones). This rendered information can help the surgeon, for example, by clamping some blood vessels to stop the blood supply to the tumor, for example, within the liver, before removing it. Once the surgeon is ready to remove the tumor inside the liver, the 3D model 160 can be rendered to provide visual guidance about what lies beneath the surface of the object of interest (such as the liver) (e.g., the location of the tumor and the blood vessels connected to it), allowing the surgeon to perform the necessary operations.

[0066] Figure 7 This is a schematic diagram of an exemplary mobile device architecture that can be used to implement a dedicated system for performing the present teachings according to various embodiments. In this example, a user device on which the present teachings can be performed corresponds to mobile device 700, including but not limited to smartphones, tablets, music players, handheld game consoles, GPS receivers, and wearable computing devices, or any other form of factor. Mobile device 700 may include one or more central processing units (“CPU”) 740, one or more graphics processing units (“GPU”) 730, a display 720, memory 760, a communication platform 710 (such as a wireless communication module), a storage device 790, and one or more input / output (I / O) devices 750. Any other suitable components (including but not limited to a system bus or controller (not shown)) may also be included in mobile device 700. Figure 7 As shown, a mobile operating system 770 (e.g., iOS, Android, Windows Phone, etc.) and one or more applications 780 can be loaded from storage device 790 into memory 760 for execution by CPU 740. Application 780 may include a user interface or any other suitable mobile application for information analysis and management on mobile device 700, at least in part, in accordance with this teaching. User interaction (if any) may be implemented via I / O device 750 and provided to various components connected via a network(s).

[0067] To implement the various modules, units, and functions described herein, a computer hardware platform may be used as the hardware platform for one or more of the elements described herein. The hardware components, operating systems, and programming languages ​​of such computers are conventional in nature, and it is assumed that those skilled in the art are fully familiar with adapting those technologies to the appropriate settings described herein. A computer with user interface components can be used to implement a personal computer (PC) or other type of workstation or terminal device, but if properly programmed, the computer can also act as a server. It is believed that those skilled in the art are familiar with the structure, programming, and general operation of such computer equipment, and therefore the accompanying drawings should be self-evident.

[0068] Figure 8 This is a schematic diagram of an exemplary computing device architecture that can be used to implement a dedicated system for executing the present teachings according to various embodiments. Such a dedicated system in conjunction with the present teachings has a functional block diagram of a hardware platform including user interface elements. The computer can be a general-purpose computer or a dedicated computer. Both can be used to implement a dedicated system for the present teachings. The computer 800 can be used to implement any component or aspect of the framework disclosed herein. For example, the information analysis and management methods and systems disclosed herein can be implemented on a computer such as computer 800 via the computer's hardware, software programs, firmware, or a combination thereof. Although only one such computer is shown for convenience, the computer functions related to the present teachings as described herein can be implemented in a distributed manner on several similar platforms to distribute the processing load.

[0069] Computer 800 includes, for example, a COM port 850 connected to and from a network connected to the COM port 850 to facilitate data communication. Computer 800 also includes a central processing unit (CPU) 820 (in the form of one or more processors) for executing program instructions. An exemplary computer platform includes an internal communication bus 810, various forms of program storage devices and data storage devices (e.g., disk 870, read-only memory (ROM) 830, or random access memory (RAM) 840) for various data files to be processed and / or transferred by computer 800, and possibly program instructions to be executed by CPU 820. Computer 800 also includes an I / O component 860 that supports input / output flows between the computer and other components within the computer, such as user interface element 880. Computer 800 can also receive programming and data via network communication.

[0070] Therefore, as mentioned above, aspects of information analysis and management methods and / or other processes can be embodied in programming. The programmatic aspect of a technology can be considered generally as a "product" or "artifact" in the form of executable code and / or associated data executed on or implemented on a type of machine-readable medium. Tangible, non-transient "storage" type media includes any or all of memory or other storage devices for computers, processors, or the like, or their associated modules (such as various semiconductor memories, tape drives, disk drives, etc.) that can be provided for software programming at any time.

[0071] Software, in whole or in part, can sometimes be transmitted over a network, such as the Internet or various other telecommunications networks. Such communication, for example, can enable software to be loaded from one computer or processor to another, for instance, in relation to information analysis and management. Therefore, another type of medium that can carry software elements includes light waves, radio waves, and electromagnetic waves, such as those used through physical interfaces between local devices, through wired and optical ground networks, and through various air links. Physical elements carrying such waves (such as wired or wireless links, optical links, or the like) can also be considered as media carrying software. As used herein, unless limited to tangible “storage” media, terms such as “computer or machine-readable medium” refer to any medium involved in providing instructions to a processor for execution.

[0072] Therefore, machine-readable media can take many forms, including but not limited to tangible storage media, carrier media, or physical transmission media. Non-volatile storage media include, for example, optical discs or magnetic disks, any or similar storage devices in any computer(s) that can be used to implement a system as shown in the accompanying drawings or any component of that system. Volatile storage media include dynamic memory, such as the main memory of a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, which include lines forming a bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves (such as those generated during radio frequency (RF) and infrared (IR) data communications). Therefore, common forms of computer-readable media include, for example: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves for transmitting data or instructions, cables or links for transmitting such carrier waves, or any other media from which a computer can read programming code and / or data. Many of these forms of computer-readable media may be involved when carrying one or more sequences of one or more instructions to a physical processor for execution.

[0073] Those skilled in the art will recognize that this teaching is adaptable to various modifications and / or enhancements. For example, while the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a software-only solution, such as an installation on an existing server. Furthermore, the techniques disclosed herein can be implemented as firmware, a firmware / software combination, a firmware / hardware combination, or a hardware / firmware / software combination.

[0074] Although the foregoing has described what is considered to constitute this teaching and / or other examples, it should be understood that various modifications may be made thereto, and the subject matter disclosed herein may be implemented in various forms and examples, and the teachings may be applied to many applications, only some of which have been described herein. The appended claims are intended to claim protection for any and all applications, modifications, and variations that fall within the true scope of this teaching.

Claims

1. A method, the method comprising: A virtual 3D camera pose is generated relative to a previously constructed 3D model that models a 3D target organ and a 3D anatomical structure associated with the 3D target organ, wherein each of the virtual 3D camera poses corresponds to a viewpoint for viewing the 3D model. Virtual 2D images corresponding to the pose of the virtual 3D camera are created by projecting the 3D model according to a corresponding viewpoint, each of the virtual 2D images including a 2D projected target organ and / or 2D structures within some of the 3D anatomical structures visible from the corresponding viewpoint; and A 2D feature / camera pose mapping model is obtained based on the 2D features extracted from the virtual 2D image and the corresponding virtual 3D camera pose, wherein the 2D features include 2D ridge lines projected from the 3D ridges on the target organ represented in the 3D model.

2. The method of claim 1, wherein the 3D model models at least one of the following: The target organ, At least one blood vessel; At least one tumor; and One or more 3D ridges on the target organ.

3. The method according to claim 1, wherein Each of the virtual 3D camera poses is characterized according to six degrees of freedom; and The virtual 3D camera pose is generated to cover different viewpoints relative to the 3D model by incrementing in each of the six degrees of freedom according to a predetermined resolution.

4. The method according to claim 1, wherein, The 2D features extracted from each of the virtual 2D images include one or more of the following: A 2D structure corresponding to the 2D projection of the target organ in the virtual 2D image; A mask corresponding to the 2D structure of the target organ; A 2D spine is projected from the 3D spine on the target organ modeled by the 3D model.

5. The method according to claim 1, wherein, The steps for obtaining the 2D feature / camera pose mapping model include: Each of the virtual 3D camera poses is paired with 2D features extracted from a corresponding virtual 2D image, which is created by projecting the 3D model based on a viewpoint determined by the virtual 3D camera poses. The 2D feature / camera pose mapping model is created based on the pairing of the 2D features and the virtual 3D camera pose.

6. The method according to claim 5, wherein, The 2D feature / camera pose mapping model corresponds to the lookup table of the pairings that includes the 2D features and the virtual 3D camera pose, such that, given an input 2D feature extracted from a 2D image, at least one 3D camera pose is identified from the pairings in the lookup table that stores 2D features similar to the input 2D feature.

7. The method according to claim 5, wherein, The steps for creating the 2D feature / camera pose mapping tool include: Training data is generated based on the pairing of the 2D features and the virtual 3D camera pose; Machine learning is performed using the training data to learn the 2D feature / camera pose mapping tool.

8. The method of claim 1, further comprising: During a medical procedure, 2D images are received from a camera inserted into the patient's body near the target object to capture surrounding information; Detect 2D objects corresponding to the target organ and / or 2D structures corresponding to some of the 3D anatomical structures from the 2D image; Extract the 2D features of the detected 2D objects and / or 2D structures; The estimated 3D camera pose is predicted based on the 2D feature / camera pose mapping model. and The 3D model is projected to visualize the target organ and / or some of the anatomical structures associated with the target organ from a perspective determined based on the estimated 3D camera pose.

9. A machine-readable medium having information recorded thereon, wherein, When the information is read by the machine, the machine performs the following steps: A virtual 3D camera pose is generated relative to a previously constructed 3D model that models a 3D target organ and a 3D anatomical structure associated with the 3D target organ, wherein each of the virtual 3D camera poses corresponds to a viewpoint for viewing the 3D model. Virtual 2D images corresponding to the pose of the virtual 3D camera are created by projecting the 3D model according to a corresponding viewpoint, each of the virtual 2D images including a 2D projected target organ and / or 2D structures within some of the 3D anatomical structures visible from the corresponding viewpoint; and A 2D feature / camera pose mapping model is obtained based on the 2D features extracted from the virtual 2D image and the corresponding virtual 3D camera pose, wherein the 2D features include 2D ridge lines projected from the 3D ridges on the target organ represented in the 3D model.

10. The medium as claimed in claim 9, wherein, The 3D model models at least one of the following: The target organ, At least one blood vessel; At least one tumor; and One or more 3D ridges on the target organ.

11. The medium according to claim 9, wherein, Each of the virtual 3D camera poses is characterized according to six degrees of freedom; and The virtual 3D camera pose is generated to cover different viewpoints relative to the 3D model by incrementing in each of the six degrees of freedom according to a predetermined resolution.

12. The medium according to claim 9, characterized in that, The 2D features extracted from each of the virtual 2D images include one or more of the following: A 2D structure corresponding to the 2D projection of the target organ in the virtual 2D image; A mask corresponding to the 2D structure of the target organ; A 2D spine is projected from the 3D spine on the target organ modeled by the 3D model.

13. The medium according to claim 9, wherein, The steps for obtaining the 2D feature / camera pose mapping model include: Each of the virtual 3D camera poses is paired with a 2D feature extracted from a corresponding virtual 2D image, which is created by projecting the 3D model based on a viewpoint determined by the virtual 3D camera poses; and The 2D feature / camera pose mapping model is created based on the pairing of the 2D features and the virtual 3D camera pose.

14. The medium according to claim 13, wherein, The 2D feature / camera pose mapping model corresponds to the lookup table of the pairings that includes the 2D features and the virtual 3D camera pose, such that, given an input 2D feature extracted from a 2D image, at least one 3D camera pose is identified from the pairings in the lookup table that stores 2D features similar to the input 2D feature.

15. The medium according to claim 13, wherein, The steps for creating the 2D feature / camera pose mapping tool include: Training data is generated based on the pairing of the 2D feature pairs and the virtual 3D camera pose; Machine learning is performed using the training data to learn the 2D feature / camera pose mapping tool.

16. The medium according to claim 9, wherein, When the information is read by the machine, the machine further performs the following steps: During a medical procedure, 2D images are received from a camera inserted into the patient's body near the target object to capture surrounding information; Detect 2D objects corresponding to the target organ and / or 2D structures corresponding to some of the 3D anatomical structures from the 2D image; Extract the 2D features of the detected 2D objects and / or 2D structures; The estimated 3D camera pose is predicted based on the 2D feature / camera pose mapping model. and The 3D model is projected to visualize the target organ and / or some of the anatomical structures associated with the target organ from a perspective determined based on the estimated 3D camera pose.

17. A system comprising: A camera pose generator, implemented by a processor and configured to generate virtual 3D camera poses relative to a 3D model previously constructed to model a 3D target organ and 3D anatomical structures associated with the 3D target organ, wherein each of the virtual 3D camera poses corresponds to a viewpoint for viewing the 3D model. A 2D feature / camera pose mapping model generator, which is implemented by a processor and configured for use with Virtual 2D images corresponding to the pose of the virtual 3D camera are created by projecting the 3D model according to a corresponding viewpoint, each of the virtual 2D images including a 2D projected target organ and / or some 2D structures in the 3D anatomical structures visible from the corresponding viewpoint. A 2D feature / camera pose mapping model is obtained based on the 2D features extracted from the virtual 2D image and the corresponding virtual 3D camera pose, wherein the 2D features include 2D ridge lines projected from the 3D ridges on the target organ represented in the 3D model.

18. The system of claim 17, wherein, The 3D model models at least one of the following: The target organ, At least one blood vessel; At least one tumor; and One or more 3D ridges on the target organ.

19. The system according to claim 17, characterized in that, Each of the virtual 3D camera poses is characterized according to six degrees of freedom; and The virtual 3D camera pose is generated to cover different viewpoints relative to the 3D model by incrementing in each of the six degrees of freedom according to a predetermined resolution.

20. The system according to claim 17, wherein, The 2D features extracted from each of the virtual 2D images include one or more of the following: A 2D structure corresponding to the 2D projection of the target organ in the virtual 2D image; A mask corresponding to the 2D structure of the target organ; A 2D spine is projected from the 3D spine on the target organ modeled by the 3D model.

21. The system according to claim 17, wherein, The steps for obtaining the 2D feature / camera pose mapping model include: Each of the virtual 3D camera poses is paired with a 2D feature extracted from a corresponding virtual 2D image, which is created by projecting the 3D model based on a viewpoint determined by the virtual 3D camera poses; and The 2D feature / camera pose mapping model is created based on the pairing of the 2D features and the virtual 3D camera pose.

22. The system according to claim 21, wherein, The 2D feature / camera pose mapping model corresponds to the lookup table of the pairings that includes the 2D features and the virtual 3D camera pose, such that, given an input 2D feature extracted from a 2D image, at least one 3D camera pose is identified from the pairings in the lookup table that stores 2D features similar to the input 2D feature.

23. The method according to claim 21, wherein, The steps for creating the 2D feature / camera pose mapping tool include: Training data is generated based on the pairing of the 2D feature pairs and the virtual 3D camera pose; Machine learning is performed using the training data to learn the 2D feature / camera pose mapping tool.

24. The system of claim 1, further comprising a camera pose estimator, the camera pose estimator being implemented by a processor and configured to: During a medical procedure, 2D images are received from a camera inserted into the patient's body near the target object to capture surrounding information; Detect 2D objects corresponding to the target organ and / or 2D structures corresponding to some of the 3D anatomical structures from the 2D image; Extract the 2D features of the detected 2D objects and / or 2D structures; The estimated 3D camera pose is predicted based on the 2D feature / camera pose mapping model. and The 3D model is projected to visualize the target organ and / or some of the anatomical structures associated with the target organ from a perspective determined based on the estimated 3D camera pose.

Images