Method and system for 3D registration of an ultrasound probe in laparoscopic ultrasound surgery and applications thereof

Abstract

A system, method, medium, and implementation for registering an ultrasound probe in laparoscopic ultrasound surgery, and its applications are disclosed. The two-dimensional (2D) position of the ultrasound probe in a 2D laparoscopic (LP) image is detected, wherein the LP camera has been previously calibrated in three-dimensional (3D) space. Based on the detected 2D position of the ultrasound probe and the ultrasound model used for the probe, the 3D pose of the deployed ultrasound probe is estimated and registered in 3D space.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Patent Application No. 18 / 517,735, filed November 22, 2023, entitled “METHOD AND SYSTEM FOR 3D REGISTERING OF ULTRASOUND PROBE IN LAPROSCOPIC ULTRASOUND PROCEDURES AND APPLICATION STHEREOF”, which is incorporated herein by reference in its entirety. 1. Technical Field

[0002] This teaching generally deals with computers. More specifically, this teaching deals with signal processing. 2. Background Technology

[0003] With advancements in technology, an increasing number of tasks are now performed with the aid of computers. Various industries have benefited from these technological advancements, including the medical field, where vast amounts of image data acquired via modern sensing technologies and the capture of patient anatomy can be processed by computers to identify different types of useful information and used more effectively in various applications, such as visualization, to aid in diagnosis, preoperative planning, and surgical instrument navigation during surgery. With advancements in signal processing technology, it is now frequently possible for computers to automatically identify anatomical structures of interest (e.g., organs, bones, blood vessels, or abnormal nodules) from diverse sensed data (such as images), obtain measurements of each object of interest (e.g., the size of a nodule growing in an organ), and visualize the features of interest with sufficient detail from any desired angle (e.g., three-dimensional (3D) visualization of abnormal nodules).

[0004] Such information can be used for a variety of purposes. For example, a 3D model can be built for a target organ, representing the physical properties of the organ (e.g., the liver based on its volume, shape, or size along different dimensions), its parts (e.g., nodules growing inside the liver, vascular structures inside and near the organ), and its spatial relationships with other surrounding anatomical structures. Such 3D models can be utilized in various applications. For instance, organ 3D models can be used for preoperative planning to derive surgical pathways from the patient's skin and targets (such as nodules in the organ to be surgically removed during surgery). During surgery, the 3D model of the organ can be rendered to show the user (e.g., the surgeon) certain desired information when projected at the appropriate angle. Furthermore, while laparoscopic sensing technology makes it possible to acquire visual information about organs inside a patient to provide guidance for performing non-invasive surgery, ultrasound probes can now be deployed in laparoscopic ultrasound (LUS) surgical settings to help visualize what lies beneath the surface of organs visible to the laparoscopic camera.

[0005] exist Figure 1A The diagram illustrates an LUS setup where a patient 120 lies on an operating table 110 during laparoscopic surgery. A laparoscopic camera 130 and an ultrasound probe 140 are inserted into the patient's body to obtain two-dimensional (2D) images, including a laparoscopic image 150 and an ultrasound image 160. The 2D laparoscopic image 150 provides the view in front of the surgical instruments 170, and the ultrasound image 160 provides information sensed by the ultrasound probe 140 beneath the surface of organs visible to the laparoscopic camera 130. Together, the laparoscopic image 150 and the ultrasound image 160 provide visual guidance to the surgeon regarding how the surgeon can manipulate the surgical instruments 170 during the procedure. Due to the proximity of the laparoscopic camera 130 and the ultrasound probe 140, the visual information in the laparoscopic image 150 may also include a view from the ultrasound probe 140.

[0006] Ultrasound images capture what exists at a certain depth from the ultrasound probe and are known to be noisy, with each image typically containing only incomplete information. For these reasons, piecing together information from ultrasound images to identify anatomical structures such as blood vessels requires considerable experience and skill. Furthermore, even when a skilled surgeon can distinguish what corresponds to blood vessels and what corresponds to other anatomical structures, determining the type of each blood vessel is, if not impossible, extremely difficult. Figure 1BExample ultrasound images corresponding to organ slices are shown, some of which are 2D structures that can be identified by humans, and based on experience, such structures can be manually labeled, for example, as boundary slices for vessels 180 and 185, and tumor 190. In this example, the labels assigned to vessels 180 and 185, based on the user's observation and experience, could correspond to the portal vein and hepatic vein. Accurate labeling of vessels can be crucial in some surgical procedures. For example, in some surgeries, surgeons may need to clamp the portal vein before removing a tumor of the same degree of proximity. Currently, labeling different anatomical structures in ultrasound images is still performed manually by humans.

[0007] Therefore, solutions to the aforementioned existing technologies need to be improved. Summary of the Invention

[0008] The teachings disclosed herein relate to methods, systems, and programming for information management. More specifically, these teachings relate to methods, systems, and programming related to hash tables and storage management using hash tables.

[0009] In one example, a method and its application are disclosed, implemented on a machine having at least one processor, storage device, and a communication platform capable of connecting to a network, for registering an ultrasound probe in laparoscopic ultrasound surgery. The two-dimensional (2D) position of the ultrasound probe in a 2D laparoscopic (LP) image is detected, where the LP camera has previously been calibrated in three-dimensional (3D) space. Based on the detected 2D position of the ultrasound probe and an ultrasound model for the probe, the 3D pose of the deployed ultrasound probe is estimated and registered in 3D space.

[0010] In various examples, a system for registering an ultrasound probe in laparoscopic ultrasound surgery and its application are disclosed. The system includes an LP U-probe position detector and a 3D U-probe pose estimator. The LP U-probe position detector is provided to detect the two-dimensional (2D) position of the ultrasound probe visible in 2D laparoscopic (LP) images acquired by an LP camera previously calibrated in three-dimensional (3D) space and inserted into the patient during laparoscopic ultrasound (LPUS) surgery. The 3D U-probe pose estimator is provided to estimate the 3D pose of the ultrasound probe deployed during LPUS surgery based on the detected 2D position of the ultrasound probe and an ultrasound model for the ultrasound probe.

[0011] Other concepts relate to software used to implement this teaching. A software product according to these concepts includes at least one machine-readable non-transient medium and information carried by that medium. The information carried by the medium may be executable program code data, parameters associated with the executable program code, and / or user-related information, requests, content, or other additional information.

[0012] Another example is a machine-readable, non-transient, and tangible medium and its application, on which information for registering an ultrasound probe during laparoscopic ultrasound surgery is recorded. When this information is read by a machine, it causes the machine to perform the following steps: Detecting the two-dimensional (2D) position of the ultrasound probe in a 2D laparoscopic (LP) image, where the LP camera was previously calibrated in three-dimensional (3D) space. Based on the detected 2D position of the ultrasound probe and the ultrasound model used for the probe, estimating the 3D pose of the deployed ultrasound probe and registering the 3D pose of the ultrasound probe in 3D space.

[0013] Additional advantages and novel features will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art upon review of the following description and the accompanying drawings, or may be learned by example from production or operation. The advantages of this teaching can be realized and obtained by practice or by using various aspects of the methods, tools, and combinations set forth in the detailed examples discussed below. Attached Figure Description

[0014] The methods, systems, and / or programming described herein will be further described through exemplary embodiments. These exemplary embodiments will be detailed with reference to the accompanying drawings. These embodiments are non-limiting exemplary embodiments, wherein the same reference numerals denote similar structures throughout several views of the drawings, and wherein:

[0015] Figure 1A An exemplary laparoscopic surgical setup is shown, in which an ultrasound probe is also deployed to obtain information;

[0016] Figure 1B Ultrasound images of some 2D structures with human-provided labels are shown;

[0017] Figure 2A An exemplary high-level system diagram is depicted for estimating 3D ultrasound probe pose (USPP) based on 2D laparoscopic images and applying it to vascular labeling, according to embodiments of this teaching.

[0018] Figure 2B This is a flowchart illustrating an exemplary process for estimating 3D USPP based on 2D laparoscopic images and its application in vascular labeling, according to embodiments of this teaching.

[0019] Figure 3A This is an exemplary high-level system diagram of an LPU-probe position detector according to an embodiment of this teaching;

[0020] Figure 3B An exemplary ultrasound probe detected in a laparoscopic image according to an embodiment of this teaching is shown;

[0021] Figure 3C An exemplary 2D region of a detected ultrasound probe having some features, representing an embodiment of the present teaching, is shown, along with the 2D position of the ultrasound probe at the centroid of the 2D region, as defined therein.

[0022] Figure 3D This is a flowchart of an exemplary process of an LPU-probe position detector according to an embodiment of this teaching;

[0023] Figure 3E An exemplary scheme for determining the 3D coordinates of a U-probe by back-projection from a 2D U-probe position detected from a laparoscopic image, according to an embodiment of this teaching, is described.

[0024] Figures 3F-3G The illustration shows the 3D structure of an opening on an ultrasound probe according to different embodiments of this teaching and its properties relative to changes in viewing angle;

[0025] Figure 4A An exemplary high-level system diagram of a 3D U-probe pose estimator according to an embodiment of this teaching is depicted;

[0026] Figure 4B This is a flowchart illustrating an exemplary process of a 3D U-probe pose estimator according to an embodiment of this teaching;

[0027] Figure 5A An exemplary high-level system diagram of a 2D blood vessel tag generator according to an embodiment of this teaching is depicted;

[0028] Figures 5B-5C A 2D structure and its vascular boundary detected from a 2D ultrasound image according to an embodiment of this teaching is shown;

[0029] Figure 5D An exemplary virtual ultrasound slice image with certain vascular structures, having labels generated based on estimated 3D U-probe pose, is illustrated according to an embodiment of this teaching.

[0030] Figure 5E The illustration shows a virtual vascular structure rendered using a 3D model superimposed on a vascular structure extracted from an ultrasound image, according to an embodiment of this teaching.

[0031] Figure 5F This is a flowchart illustrating an exemplary process of a 3D U-probe pose estimator according to an embodiment of this teaching;

[0032] Figure 5G This is a flowchart illustrating an exemplary process of a 2D blood vessel tag generator according to an embodiment of this teaching;

[0033] Figure 6These are schematic diagrams of exemplary mobile device architectures that can be used to implement dedicated systems that implement the present teachings according to various embodiments; and

[0034] Figure 7 This is a schematic diagram of an exemplary computing device architecture that can be used to implement a dedicated system that implements the teachings according to various embodiments. Detailed Implementation

[0035] In the following detailed description, numerous specific details are illustrated by way of example to facilitate a thorough understanding of the teachings. However, it will be apparent to those skilled in the art that these teachings can be practiced without such details. In other instances, well-known methods, processes, components, and / or systems have been described at a relatively high level without detail in order to avoid unnecessarily obscuring various aspects of these teachings.

[0036] This teaching discloses exemplary methods, systems, and implementations for estimating 3D USPP during LUS surgery and the use of the estimated USPP in labeling different anatomical structures observed in ultrasound images during LUS surgery. LUS surgery has been used in various surgical procedures, such as laparoscopic hepatectomy, because the use of an ultrasound probe can provide a view of anatomical structures, such as blood vessels and tumors beneath the organ surface. Prior to surgery, a 3D organ model can be constructed based on 3D medical image data obtained from scans, such as computed tomography (CT) or magnetic resonance imaging (MRI). Ultrasound and CT registration during surgery can be possible. However, this is often not possible because images acquired by a laparoscopic camera during LUS surgery have a limited field of view, as laparoscopic images do not reveal sufficient anatomical structures. On the other hand, ultrasound images acquired via an ultrasound probe during LUS surgery are often quite noisy and contain only partial information, making them insufficient to provide enough information about the anatomical structures the user needs to see during surgery.

[0037] This teaching discloses a framework for estimating 3D USPP and then using such estimated 3D USPP to label anatomical structures partially observed in 2D ultrasound images. The estimation of 3D USPP is performed based on an ultrasound probe captured in 2D laparoscopic images by a laparoscopic camera during LUS surgery. When estimating the pose of the laparoscopic camera relative to a 3D model of the target organ (this estimation is outside the scope of this teaching), each pixel in the laparoscopic image, including the 2D position of the ultrasound probe as detected in the 2D laparoscopic image, can be mapped to a 3D position in the 3D space of the 3D model. To determine the orientation of the ultrasound probe, a 3D model of features on the ultrasound probe can be used. In some embodiments, such features may correspond to an opening located on the ultrasound probe and can be detected from a 2D laparoscopic image, which can be used to compare different projections of the opening using a 3D model with an orientation different from the estimated 3D position of the ultrasound probe. One of the projections created using a specific orientation can be identified as corresponding to the best match, such that the orientation can be identified as the orientation of the ultrasound probe (i.e., the rotation angle around the US probe axis). The 3D USPP can be determined using the 3D position and orientation of the ultrasound probe estimated by utilizing the 2D features of the ultrasound probe observed in 2D laparoscopic images. (Refer to Figure 2-) Figure 4B Details related to estimating the 3D USPP of the ultrasound probe have been disclosed.

[0038] Such 3D USPPs can be used to identify certain 2D structures detected from 2D ultrasound images, which may correspond to partially visible 2D anatomical structures with significant noise. Based on a 3D model of the target organ in the same 3D space as the estimated 3D USPP according to this teaching, such partially visible anatomical structures can be labeled to provide effective guidance to the surgeon during surgical procedures. For example, portal vein and hepatic vascular structures can be labeled in this way, even though they may only be partially visible in 2D ultrasound images, based on the teaching disclosed herein. Based on the estimated 3D USPP of the ultrasound probe, the 3D anatomical structures that should be seen by the ultrasound probe according to known operating parameters associated with the ultrasound sensor (modeled in a 3D model) can be determined such that 3D structures with known labels corresponding to the 3D anatomical structures visible by the ultrasound probe can be projected onto the 2D image plane to produce 2D projected structures with associated ground truth labels. Such 2D projected structures with ground truth labels can then be used to determine how to label 2D structures detected from 2D ultrasound images. Reference Figures 5A-5E Details are provided regarding the use of estimated 3D USPP to label 2D structures visible from 2D ultrasound images.

[0039] Figure 2A An exemplary high-level system diagram depicts a framework 200 for estimating 3D USPP based on 2D laparoscopic images and applying it to vascular labeling, according to embodiments of this teaching. The illustrated framework 200 includes two parts: one for estimating the 3D USPP of an ultrasound probe in an LUS setup, and another for the specific application of the estimated 3D USPP to label blood vessels observed in 2D ultrasound images. This exemplary framework 200 is applied in LUS procedures involving both a laparoscopic camera and an ultrasound probe to provide 2D laparoscopic images and 3D ultrasound images, respectively. For example, an LUS procedure may correspond to a surgery for removing a tumor from a patient's liver. In this case, a 3D model 240 of the target organ, the liver, can be constructed prior to the LUS procedure based on, for example, various image scan data. The 3D model 240 can model different relevant parts of the target organ, including, for example, the physical structure of the target organ, the tumor within the liver, and other relevant anatomical structures such as blood vessels.

[0040] According to the framework 200 of this embodiment, 2D images (both laparoscopic and ultrasound images) can be used as input, and certain anatomical structures, such as blood vessels, can be marked in the 2D ultrasound images to facilitate actions taken by the surgeon during LUS surgery based on the marked blood vessels (e.g., clamping the portal vein). It should be understood that this exemplary application of marking blood vessels using estimated 3D USPP is provided only as illustrative and not limiting. Other applications are also possible by using estimated 3D USPP from an ultrasound probe during LUS surgery to generate information or visual guidance for the user operating during the procedure.

[0041] The first part of frame 200 includes an LP U-probe position detector 210 and a 3D U-probe pose estimator 220. The LP U-probe position detector 210 is provided for detecting the presence of a visible ultrasound probe and determining its 2D position in a 2D laparoscopic image. The 3D U-probe pose estimator 220 is provided for estimating the 3D pose (including 3D position and orientation) of the ultrasound probe based on the detected 2D position and a 3D model 240 of the target organ. As discussed herein, the second part of frame 200 is used to label partially visible vascular structures in a 2D ultrasound image using the estimated 3D U-probe pose (USPP). To achieve this, the second part includes a U-image structure detector 250 and a 2D vascular label generator 260. The U-image structure detector 250 is provided for processing 2D ultrasound images generated based on content sensed by the ultrasound probe to identify 2D structures, such as regions or edge points. Because ultrasound images are typically 2D partial views of 3D structures, such detected 2D structures may not provide sufficient information to correlate them with certain 3D anatomical structures, for example, to determine whether a linear structure in a 2D U-image originates from the portal vein or the hepatic vein. Using the estimated 3D USPP from the 3D U-probe pose estimator 220, a 2D vessel label generator 260 is provided to label the 2D structures corresponding to the vessels based on the 3D model 240.

[0042] Figure 2B This is a flowchart illustrating an exemplary process for estimating 3D USPP and its application in labeling blood vessels, based on an embodiment of this teaching. During LUS surgery, the LPU-probe position detector 210 receives laparoscopic images acquired by a laparoscopic camera and continues to detect the region in the image corresponding to the ultrasound probe at 205. For example, from... Figure 1A As can be seen in the laparoscopic image 150, an ultrasound probe is present. Based on the detected area corresponding to the ultrasound probe, the LPU-probe position detector 210 determines the position of the U-probe in the LP image at 215, which can be defined according to some specified definitions. The 2D position of the U-probe in the laparoscopic image can then be used by the 3D U-probe pose estimator 220 to estimate the 3D pose of the ultrasound probe detected from the 2D laparoscopic image at 225. Such 3D pose includes 3D coordinates (e.g., (x, y, z) in 3D space) and orientation (i.e., pitch, roll, and yaw). Reference Figures 3A-4B Details are provided regarding obtaining the 3D pose of the ultrasound probe based on the 2D position.

[0043] On the other hand, when receiving 2D ultrasound images from an ultrasound probe, the U-image structure detector 250 processes the ultrasound image at 235 to detect various 2D structures at 245. Based on these detected 2D structures, the U-image structure detector 250 can estimate structures corresponding to blood vessels at 255. To label the estimated 2D vascular structures in the 2D image, the 2D vascular label generator 260 can retrieve a 3D model 240 for the target organ at 265, and can then continue at 275 to estimate the type or label of each vascular structure detected in the ultrasound image by utilizing 3D USPP estimation and the 3D model 240. Based on the estimated label of each 2D vascular structure, the 2D vascular label generator 260 can then label each of the 2D vascular structures in the ultrasound image at 285.

[0044] Figure 3A This is an exemplary high-level system diagram of an LP U-probe location detector 210 according to an embodiment of this teaching. As described above, the LP U-probe location detector 210 is provided for detecting regions in a laparoscopic image corresponding to an ultrasound probe and then determining their location according to certain definitions. In the illustrated embodiment, the LP U-probe location detector 210 includes an LP image preprocessor 310, a 2D U-probe segmentation unit 320, and a 2D U-probe location determiner 330. The LP image preprocessor 310 may optionally be provided for preprocessing the received laparoscopic image to generate an enhanced image to facilitate further processing. In some embodiments, the input laparoscopic image may be used directly without preprocessing. The 2D probe segmentation unit 320 may then use the input or preprocessed laparoscopic image to identify regions in the 2D image where an ultrasound probe is present. In some embodiments, segmentation may be performed based on a U-probe model 230 to segment the 2D probe region in the laparoscopic image. In some embodiments, some features within the segmented region may also be extracted. In some other embodiments, exemplary images of the LP view of the U-probe and the ground truth location of the U-probe region can be used to train a neural network via machine learning for U-probe segmentation and location determination. After training, the trained neural network can be used to segment the U-probe region and its location based on any input LP image containing the U-probe.

[0045] Figure 3BAn exemplary detected 2D region 350 corresponding to an ultrasound probe when it appears in a laparoscopic image, according to an embodiment of this teaching, is shown. As shown, in laparoscopic image 150, a 2D U-probe segmentation unit 320 can detect the 2D region 350 corresponding to the ultrasound probe based on, for example, a U-probe model 230. The U-probe model 230 can model physical properties based on boundaries and some distinguishing features, such as an opening 360 located on the ultrasound probe. This is in Figure 3C The image shows an exemplary segmentation result or 2D boundary 350 of an ultrasound probe detected from a 2D image, and an opening 360 extracted therefrom. In some embodiments, based on the segmentation of the ultrasound probe, the 2D U-probe location determiner 330 can determine a representative 2D location of the detected probe based on a specified configuration, for example, specified in 340, regarding how to determine the location based on the segmented region. In some embodiments, the representative location of the 2D region (representing the boundary of the ultrasound probe) can be defined as the centroid 370 of that region, such as... Figure 3C As shown in the diagram. Other methods can also be used to define representative locations of the detected ultrasound probe.

[0046] When a U-probe appears in a 2D image, there are different ways to detect its location. In some embodiments, conventional methods can be used to detect a first feature associated with the U-probe, and then a classification can be made based on whether the feature reveals the presence of the U-probe. In some embodiments, a model can be trained via deep learning based on training data. Figure 3D This is a flowchart illustrating an exemplary process by which an LP U-probe position detector 210, according to an embodiment of this teaching, identifies U-probes from 2D images using a segmentation model obtained via machine learning. The position detector 210 can identify the position of the U-probes in LP images based on a model trained for segmenting U-probes. This model can be trained offline based on training data and used to perform segmentation during LP surgery. At 305, laparoscopic images from previous surgeries can be retrieved using the U-probes at various locations and orientations. At 315, manual or semi-automatic segmentation can be performed to segment the U-probes at different locations and orientations from the laparoscopic images. At 325, a deep learning neural network can be trained using training data based on the laparoscopic images and U-probe segmentation as ground truth U-probe positions. Training can be performed offline before surgery. During surgery, when real-time laparoscopic images are received at 335, they are fed to the trained model as input to the deep learning model to segment the U-probes appearing in the real-time LP images and their positions at 345. Then, the specific 2D U-probe position can be output at 355.

[0047] Using the estimated 2D position of the ultrasound probe, the 3D pose of the ultrasound probe can be further estimated by the 3D U-probe pose estimator 220 (see [link]). Figure 2A ). Figure 4A An exemplary high-level system diagram of a 3D U-probe pose estimator 220 according to an embodiment of this teaching is depicted. The 3D pose has six degrees of freedom, including (x, y, z, r, y, p), corresponding to (x, y, z) as 3D coordinates in 3D space, and roll, yaw, and pitch associated with orientation. The 3D probe pose estimator 220 is provided to estimate both the 3D coordinates and orientation of the U-probe based on the 2D position of the U-probe estimated from a 2D laparoscopic image. In the illustrated embodiment, the 3D U-probe pose estimator 220 includes a 3D probe position determiner 410, a 2D probe opening extractor 450, a 2D virtual image generator 430, and a comparison-based probe orientation determiner 460.

[0048] A 3D probe position determiner 410 may be provided for estimating the corresponding 3D position of the U-probe based on the 2D probe position. A 2D U-probe opening extractor 450 may be provided for extracting 2D features associated with the opening of the U-probe from the laparoscopic image when the U-probe appears in the 2D laparoscopic image, including but not limited to the 2D region corresponding to the opening on the U-probe and the boundary points of the 2D region. A comparison-based probe orientation determiner 460 may be provided to estimate the orientation of the U-probe based on the 2D features detected from the laparoscopic image according to the probe model 230. In some embodiments, optionally, the estimation of the U-probe orientation may also be based on a 2D virtual ultrasound image 440, which is generated via simulation by projecting a 3D model 240 onto a 2D image plane based on estimated 3D coordinates and ultrasound sensor parameters specified by the probe model 230. Such generated virtual 2D ultrasound images may be generated using different assumed U-probe orientations and then used for comparison with 2D ultrasound images actually acquired by the U-probe to determine the U-probe orientation.

[0049] Regarding the estimation of the 3D coordinates of the U-probe based on the 2D position detected from the laparoscopic image, when calibrating the laparoscopic camera in 3D space, each 2D pixel in the laparoscopic image can correspond to a line of sight in 3D space, which can be obtained via a transformation using a transformation matrix obtained during calibration. When the 3D model 240 is also transformed to the same 3D coordinate system, each pixel in the laparoscopic image 375 can be back-projected onto the 3D model 240 in 3D space. How such a transformation is obtained is beyond the scope of this teaching. Therefore, the estimated 2D position of the U-probe in the laparoscopic image can be used to obtain the 3D coordinates of the ultrasound probe via backprojection. Figure 3EThis exemplary scheme is depicted, in which a laparoscopic camera 370 is calibrated in a 3D space, also displaying a 3D model 240. With this setup, the position of each pixel in the laparoscopic image 375 can be back-projected onto 3D coordinates on the 3D model 240. Therefore, the 2D U-probe segmentation 380 can be back-projected onto 390 of the 3D model 240. Consequently, the estimated 2D U-probe position can also be back-projected onto 3D coordinates in 3D space.

[0050] Therefore, assuming such calibration has been completed and transformation matrix 420 is available, the 3D U-probe position determiner 410 can take the detected 2D U-probe position and the 3D model in the camera coordinate system as input, and then use the transformation matrix to calculate the 3D position or 3D coordinates of the U-probe. The orientation of the U-probe (besides the rotation angle around the probe's axis) can be estimated, for example, by fitting 3D lines to 3D points obtained via backprojection from the 2D U-probe pixels onto the 3D model. To determine the additional degree associated with the rotation angle of the U-probe around its axis, additional image features associated with the U-probe can be detected from the 2D laparoscopic image, and these additional image features can be used to facilitate the estimation of the U-probe's 3D orientation. Given the known 3D position of the U-probe, different means may exist to estimate the 3D orientation of the U-probe. One exemplary approach could be based on additional features on the U-probe compared to those observed from the 2D laparoscopic image. Another exemplary approach is to utilize observable anatomical structures in a laparoscopic image, compared to a 2D virtual ultrasound image generated by projecting a portion of a 3D model 240 under given operating parameters of the U-probe. Such a 2D virtual ultrasound image can be created with respect to different orientations at a given 3D location. Yet another approach could be to integrate both U-probe-based and virtual 2D ultrasound image-based methods to optimize the estimated 3D orientation of the U-probe.

[0051] As discussed in this article, a U-probe can have an opening or a 360° aperture on it, such as... Figures 3B-3C As shown in the diagram. The features associated with the aperture on the U-probe can be modeled by the U-probe model 230. Figures 3F-3G The illustration shows a modeled 3D structure of an opening 360 on an ultrasound probe according to different embodiments of this teaching. For example... Figure 3F As seen in some embodiments, the aperture 360 ​​may correspond to a 3D cylindrical structure embedded in a U-probe, which has visible and invisible portions. For example, the visible portion of the 3D aperture 360 ​​may include the closed circle or boundary 360-1a of the opening and the visible portion of the inner wall region 360-2a of the cylindrical structure. The invisible portion of the 3D aperture 360 ​​may include the dashed portion 360-3a. Once the 3D coordinates of the U-probe are estimated, these modeled features can be used to estimate the orientation of the U-probe.

[0052] Depending on the orientation of the U-probe, the characteristics of the visible portion on the U-probe (e.g., the shape of the opening edge 360-la and the shape of the wall surface region 360-2a) can be changed accordingly. This is in Figure 3G As shown, the visible shape of the opening edge 360-1b and the corresponding visible shape of the region corresponding to the observable wall region 360-2b of the orifice 360 ​​differ from the corresponding observed properties associated with 360-1a and 360-2a, respectively, due to different orientations. Assuming the shape of the opening edge 360-1 is known (e.g., circular), the shape of the edge and the shape of the wall region detected from the 2D laparoscopic image (e.g., elliptical) can then be used by a comparison-based U-probe orientation determiner 460 to estimate the probe orientation. For example, a U-probe model 230 can be used to create 2D virtual images of the U-probe with respect to different orientations, and the visible 2D features of the projected U-probe can be extracted and compared with corresponding observations obtained from the laparoscopic image. The optimal matching orientation can be used as the estimated orientation of the U-probe.

[0053] On the other hand, utilizing the estimated 3D position of the U-probe, given a 3D model 240 and a U-probe model 230, a 2D virtual U-image generator 430 can be provided to generate 2D virtual ultrasound images 440 based on slices from the 3D model 240 when viewed from different angles, where the angles are determined based on different orientations, and the slices are determined based on the operating parameters of the U-probe (e.g., modeled by the U-probe model 230). 2D image features of observed 2D structures can be detected from such 2D virtual ultrasound images and compared with features detected from actual ultrasound images. Optimal matches can be identified to determine the orientation that produces the optimal match. As discussed herein, a comparison-based U-probe orientation determiner 460 can be configured to estimate the U-probe based on any of the possible methods according to a pre-configured operating mode.

[0054] Figure 4BThis is a flowchart of an exemplary process of a 3D U-probe pose estimator according to an embodiment of this teaching. After receiving the estimated 2D position of the U-probe at 405, the 3D U-probe position determiner 410 transforms the 2D position into 3D coordinates representing the 3D position of the U-probe at 415 based on a transformation matrix 420 and a 3D model. At 417, a partial orientation of the U-probe can be estimated, for example, via fitting a linear structure to 3D coordinates. The estimation of the remaining 3D orientation of the U-probe (i.e., rotation about the probe's axis) is performed according to the configured operating mode determined at 425. If the 3D U-probe pose estimator is configured to operate in a mode using virtual U-probe features, at 435, the U-probe model 230 creates different 2D virtual U-probe images relative to different orientations from the estimated 3D position of the U-probe. Then, at 445, the 2D features in the 2D virtual U-probe images associated with the opening of the U-probe are compared with features extracted from the laparoscopic images.

[0055] If the 3D U-probe pose estimator is configured to operate in a mode using virtual ultrasound images, the 2D virtual U-image generator 430 can create 2D virtual ultrasound images (455) based on different orientations of both the 3D model 240 and the U-probe model 230 relative to the U-probe. Such 2D virtual ultrasound images can correspond to different slices of the 3D model determined based on a given 3D pose of the U-probe (i.e., the assumed orientation of the U-probe and the estimated 3D position) and the operating parameters of the U-probe (i.e., the depth of the signal detected by the ultrasound probe). At 465, the 2D structures captured in both the virtual and actual ultrasound images can be identified and compared, allowing for the identification of the best match. The orientation corresponding to the virtual ultrasound image that produces the best match can be estimated as the orientation of the U-probe.

[0056] If the operating mode is configured to estimate orientation based on a comparison of 2D features from both the virtual U-probe image and the virtual ultrasound image, the virtual U-probe image and virtual ultrasound image created at 435 and 455, respectively, are used to extract the 2D features discussed herein at 475, and compared at 485 with corresponding features from the laparoscopic image (for the U-probe) and the ultrasound image. The comparison result (from 445, 465, and 485 depending on the operating mode) is then used to determine the 3D orientation of the U-probe at 495 (e.g., based on best match) and thus the 3D pose of the U-probe. This estimated 3D U-probe pose can be applied in LUS procedures to further assist users, such as surgeons, in receiving additional guidance. As discussed herein, in some LUS procedures, surgeons may need to clamp the nearby portal vein to stop blood flow before cutting the tumor. In such scenarios, automated guidance regarding which of the visual 2D structures seen in the ultrasound image corresponds to the portal vein can be useful. Furthermore, the estimated 3D U-probe pose based on this teaching can also be used to determine, for example, whether a 2D structure observable from an ultrasound image is a blood vessel or the boundary of a tumor. When 3D model 240 and U-probe model 230 are available, the 3D U-probe pose can be applied to correspond the 2D structure observed in the ultrasound image to the 3D anatomical structure captured by 3D model 240.

[0057] As discussed herein, an exemplary application of the 3D U-probe pose estimated according to this teaching is the labeling of blood vessels in 2D ultrasound images. Reference Figure 1B A specific example application is discussed, in which 2D structures are labeled as different types of blood vessels, such as the portal vein and hepatic vein, when they appear in 2D ultrasound images. Figure 2A The illustration shows the use of a 3D U-probe pose in such applications, wherein a 2D vascular tag generator 260 utilizes the 3D U-probe pose estimated according to this teaching and generates vascular tags for 2D structures identified as vessels by a U-image structure detector 250 from an ultrasound image to provide guidance to the surgeon during LUS procedures.

[0058] Some anatomical structures, including blood vessels, may deform during surgery. Therefore, the shape and relative position of these anatomical structures can change accordingly. This presents a challenge in how to use ground truth labels of different anatomical structures present in a virtual ultrasound image to label the corresponding 2D structures detected from the actual ultrasound image. This includes the task of labeling blood vessels. This teaching discloses an exemplary method for labeling 2D structures in ultrasound images during LUS surgery, given a known 3D ultrasound probe pose. In some embodiments, labeling of 2D structures based on the estimated 3D U-probe pose can be implemented in a two-stage process according to this teaching. In the first stage, each pixel in the 2D structure detected from the actual ultrasound image can be assigned a label associated with a corresponding pixel in the virtual ultrasound image. The virtual ultrasound image can be generated based on a given 3D U-probe pose and operating parameters of the U-probe. The corresponding pixels in such virtual ultrasound images can be selected based on several criteria, as discussed below. In the second stage, where all pixels of the detected 2D structure have assigned labels, a uniform label for each coherent 2D structure (e.g., pixels roughly forming a circular shape) can be determined based on some specified criteria, using the labels of the pixels provided in the first stage. The labeling method discussed herein can be applied to labeling blood vessel types, and references... Figures 5A-5E Provide relevant details.

[0059] Figure 5A An exemplary high-level system diagram of a 2D vessel label generator 260 according to an embodiment of the present teachings is depicted. In the illustrated embodiment, the 2D vessel label generator 260 takes as input a 2D structure detected from an ultrasound image and an estimated 3D U-probe pose, and outputs a U-image with labels provided to the 2D structures identified as vessels. In this illustration, the 2D vessel label generator 260 includes a vessel structure extractor 510, a labeled vessel projection unit 530, a vessel pixel labeling unit 540, and a vessel labeling unit 550. The vessel structure extractor 510 may be provided to extract vessel structures from the input 2D structure detected by the ultrasound image and to store such detected vessel structures in 520. Based on the viewing angle determined according to the estimated 3D U-probe pose, the labeled vessel projection unit 530 may be invoked to generate a virtual ultrasound image corresponding to a slice of the 3D model 240 located at a depth determined, for example, by the operating parameters of the U-probe specified by the U-probe model 230. The slice can include a 2D structure with ground truth labels, including vessel type labels. For example, each pixel in each of the virtual vessels is assigned a label representing, for example, the vessel type.

[0060] Figure 5BThe illustration shows actual ultrasound images with some 2D structures detected from them, some of which can be identified by the vascular structure extractor 510 as corresponding to blood vessels (e.g., 2D structures 570 and 580). Accordingly, the boundaries of the extracted 2D vascular structures are shown in... Figure 5C The area shown is indicated by a dashed line and is enclosed. For example... Figures 5B-5C As can be seen in the ultrasound images, the detected 2D blood vessels 570 and 580 may be flattened or have altered shapes due to deformation or noise in the ultrasound images. Figure 5D Ground truth vessels 590-1 and 590-2 are shown in a virtual ultrasound slice image generated based on the estimated 3D U-probe pose. Each pixel within 590-1 and 590-2 can have a ground truth label indicating, for example, the corresponding vessel type (such as portal vein or hepatic vein). In this case, all pixels within 590-1 can be labeled as portal vein, and all pixels within 590-2 can be labeled as hepatic vein. Based on the label, all vessel pixels connected to the labeled pixels can be assigned the same label. As can be seen, both ground truth vessels 590-1 and 590-2 have a smooth and substantially circular shape.

[0061] Figure 5E Ground truth vessel labels 590-1 and 590-2 are shown superimposed on the detected 2D vascular structures 570 and 580, respectively. Although they differ in shape, size, and position, the ground truth vessels (590-1 and 59-2) and the detected vascular structures (570 and 580) do overlap, which can be used to derive labels for each of the pixels in the detected 2D vascular structures. Based on the ground truth vessel labels in the virtual ultrasound slice image, a vascular pixel labeling unit 540 can be provided to perform the first stage of labeling, i.e., assigning labels to each pixel in each of the vascular structures 520. In some embodiments, the label assigned to each pixel in the detected blood vessel can be determined based on the labels of one or more corresponding pixels in the virtual ultrasound slice image. For example, such one or more corresponding pixels in the virtual ultrasound image can come from a neighborhood defined relative to each pixel to be labeled in the blood vessel detected from the ultrasound image. In some embodiments, the neighborhood can be defined as all pixels directly connected to the pixel to be labeled. Another definition of the neighborhood can be the nearest neighbor of the pixel to be labeled.

[0062] Corresponding neighborhoods in the virtual ultrasound slice image can be accordingly identified as neighborhoods at the same pixel location. Vessel labels associated with pixels in these corresponding neighborhoods in the virtual ultrasound image can be used to determine the vessel label to be assigned to the pixel to be labeled. For example, in some embodiments, a majority rule can be used to determine the label to be assigned to the pixel to be labeled, such as the label associated with a majority of pixels in the corresponding neighborhood in the virtual ultrasound image. Other means for determining the label can also be deployed. Through this first stage of the labeling process, all pixels within the 2D vessels in 520 are assigned labels indicating the vessel type.

[0063] Through the first stage of pixel-level labeling, it is possible for pixels belonging to the same blood vessel to be assigned different labels. For example, pixels within the illustrated blood vessel 570 could be assigned different labels. Figure 5E As shown, while a large portion of vessel 570 overlaps with ground truth vessel 590-1, which has a ground truth label (e.g., portal vein), a small portion of 570 may overlap with 590-2, which has a different label (e.g., hepatic vein). The second phase of vessel labeling can be directed to perform vessel-level labeling, i.e., assigning a uniform label to each of the detected vessels in 520. A vessel labeling unit 550 may be provided for this purpose. In some embodiments, a uniform label identifying each detected vessel can be implemented based on the labels assigned to all pixels within the vessel (in the first phase). For example, the labels assigned to pixels in detected vessel 570 can form the basis for determining a uniform vessel label for 570. In some embodiments, a majority rule can be employed, i.e., the label typically assigned to the majority of pixels in the vessel can be used as the uniform label for the vessel. In the example of vessel 570, although some of its pixels could be assigned the label portal vein and some pixels could be assigned the label hepatic vein, vessel 570 is assigned a uniform label as portal vein because the majority of pixels have already been labeled portal vein. This is in Figure 5F As shown in the figure, the entire 2D vascular structure is labeled as the portal vein, and correspondingly, another 2D vascular structure (580) is labeled as the hepatic vein.

[0064] Figure 5GThis is a flowchart of an exemplary process of a 2D vessel label generator 260 according to an embodiment of this teaching. At 505, a 2D structure detected from an ultrasound image and a 3D U-probe pose estimated according to this teaching may be received as input. To label the vessels appearing in the actual ultrasound image, a vessel structure extractor 510 processes the 2D structure and extracts the vessel structure 520 at 515. To facilitate pixel labeling in the first stage, a labeled vessel projection unit 530 retrieves the 3D model 240 and the U-probe model 230 at 525 and calculates an angle or viewpoint relative to the 3D model 240 based on the input 3D U-probe pose. From this viewpoint, at 535, a slice of the 3D model 240 perpendicular to the viewpoint is determined based on operating parameters associated with the ultrasound probe as specified by the U-probe model 230. For each of the vessels visible in the slice (e.g., a cross-section of the vessel), a label is provided to such pixels according to the 3D model 240 at 545. Then, such created virtual ultrasound slice images, in which blood vessels are marked, can be used to label 2D vascular structures detected from ultrasound images obtained during LUS surgery.

[0065] For each of the vascular structures in 520, the vascular pixel labeling unit 540 can perform a first stage of pixel labeling by assigning a label at 555 to each of the pixels within the vascular vessel based on a provided vascular pixel label identified in a virtual ultrasound slice image according to this teaching. Then, the ultrasound image with the vascular pixels assigned pixel labels can be processed by the vascular labeling unit 550 for a second stage, which labels each detected vascular vessel. To do this, as discussed herein, a uniform label can be identified for each vascular structure at 565, and then that uniform label is assigned to all pixels of the vascular vessel at 575. At this point, as... Figure 1B As shown, ultrasound images that reveal partial and noisy visual information at a certain depth below the surface of an organ can have different sections labeled to correspond to different blood vessel types. This can provide effective guidance for surgeons during surgical procedures.

[0066] While exemplary applications of 3D U-probe pose estimated from laparoscopic images are provided via vascular labeling, these are for illustrative purposes only and not as limitations regarding potential applications of 3D U-probe pose estimated from laparoscopic images according to exemplary embodiments of this teaching. Other applications, such as labeling other anatomical structures (such as tumors or organs), can also be achieved via labeling schemes as discussed herein.

[0067] Figure 6This is a schematic diagram of an exemplary mobile device architecture that can be used to implement a dedicated system that implements the teachings according to various embodiments. In this example, a user device on which the teachings can be implemented corresponds to mobile device 600, including but not limited to smartphones, tablets, music players, handheld game consoles, GPS receivers, and wearable computing devices, or any other form of factor. Mobile device 600 may include one or more central processing units (“CPU”) 640, one or more graphics processing units (“GPU”) 630, a display 620, memory 660, a communication platform 610 (such as a wireless communication module), a storage device 690, and one or more input / output (I / O) devices 650. Any other suitable components (including but not limited to a system bus or controller (not shown)) may also be included in mobile device 600. Figure 6 As shown, a mobile operating system 670 (e.g., iOS, Android, Windows Phone, etc.) and one or more applications 680 may be loaded from storage device 690 into memory 660 for execution by CPU 640. Application 680 may include, at least partially, a user interface according to this teaching or any other suitable mobile application for information analysis and management on mobile device 600. User interaction (if any) may be implemented via I / O device 650 and provided to various components connected via one or more networks.

[0068] To implement the various modules, units, and functions described herein, a computer hardware platform can be used as the hardware platform for one or more of the elements described herein. The hardware components, operating systems, and programming languages ​​of such computers are conventional in nature, and it is assumed that those skilled in the art are fully familiar with adapting these technologies to the appropriate settings described herein. A computer with user interface components can be used to implement a personal computer (PC) or other type of workstation or terminal device, but if properly programmed, the computer can also act as a server. It is believed that those skilled in the art are familiar with the structure, programming, and general operation of such computer equipment, and therefore the accompanying drawings should be self-evident.

[0069] Figure 7This is a schematic diagram of an exemplary computing device architecture that can be used to implement a dedicated system according to various embodiments of the present teachings. Such a dedicated system in conjunction with the present teachings has a functional block diagram of a hardware platform including user interface elements. The computer can be a general-purpose computer or a dedicated computer. Both can be used to implement a dedicated system for the present teachings. The computer 700 can be used to implement any component or aspect of the framework disclosed herein. For example, the information analysis and management methods and systems disclosed herein can be implemented on a computer such as computer 700 via the computer's hardware, software programs, firmware, or a combination thereof. Although only one such computer is shown for convenience, the computer functions described herein related to the present teachings can be implemented in a distributed manner on several similar platforms to distribute the processing load.

[0070] Computer 700 includes, for example, a COM port 750 connected to and from a network connected to the COM port 750 to facilitate data communication. Computer 700 also includes a central processing unit (CPU) 720 (in the form of one or more processors) for executing program instructions. An exemplary computer platform includes an internal communication bus 710, various forms of program storage devices and data storage devices (e.g., disk 770, read-only memory (ROM) 730, or random access memory (RAM) 740) for various data files to be processed and / or transferred by computer 700, and possibly program instructions to be executed by CPU 720. Computer 700 also includes an I / O component 760 that supports input / output flows between the computer and other components within the computer, such as user interface element 780. Computer 700 can also receive programming and data via network communication.

[0071] Therefore, as mentioned above, aspects of information analysis and management methods and / or other processes can be embodied in programming. The programmatic aspect of a technology can be considered generally as a "product" or "artifact" in the form of executable code and / or associated data executed on or implemented on a type of machine-readable medium. Tangible, non-transient "storage" type media includes any or all of memory or other storage devices for computers, processors, or the like, or their associated modules (such as various semiconductor memories, tape drives, disk drives, etc.) that can be provided for software programming at any time.

[0072] Software, in whole or in part, can sometimes be transmitted over a network, such as the Internet or various other telecommunications networks. Such communication, for example, can enable software to be loaded from one computer or processor to another, for instance, in relation to information analysis and management. Therefore, another type of medium that can carry software elements includes light waves, radio waves, and electromagnetic waves, such as those used through physical interfaces between local devices, through wired and optical ground networks, and through various air links. Physical elements carrying such waves (such as wired or wireless links, optical links, or the like) can also be considered as media carrying software. As used herein, unless limited to tangible “storage” media, terms such as “computer or machine-readable medium” refer to any medium involved in providing instructions to a processor for execution.

[0073] Therefore, machine-readable media can take many forms, including but not limited to tangible storage media, carrier media, or physical transmission media. Non-volatile storage media include, for example, optical discs or magnetic disks, any or similar storage devices in any computer(s) that can be used to implement a system as shown in the accompanying drawings or any component of that system. Volatile storage media include dynamic memory, such as the main memory of a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, which include lines forming a bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves (such as those generated during radio frequency (RF) and infrared (IR) data communications). Therefore, common forms of computer-readable media include, for example: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves for transmitting data or instructions, cables or links for transmitting such carrier waves, or any other media from which a computer can read programming code and / or data. Many of these forms of computer-readable media may be involved when carrying one or more sequences of one or more instructions to a physical processor for execution.

[0074] Those skilled in the art will recognize that this teaching is adaptable to various modifications and / or enhancements. For example, while the implementations of the various components described above may be embodied in a hardware device, they may also be implemented as a software-only solution, for example, installed on an existing server. Furthermore, the techniques disclosed herein may be implemented as firmware, a firmware / software combination, a firmware / hardware combination, or a hardware / firmware / software combination.

[0075] Although the foregoing has described what is considered to constitute this teaching and / or other examples, it should be understood that various modifications may be made thereto, and the subject matter disclosed herein may be implemented in various forms and examples, and the teachings may be applied to many applications, only some of which have been described herein. The appended claims are intended to claim protection for any and all applications, modifications, and variations that fall within the true scope of this teaching.

Claims

1. A method comprising: The two-dimensional (2D) position of an ultrasound probe visible in 2D laparoscopic (LP) images acquired by an LP camera previously calibrated in three-dimensional (3D) space and inserted into the patient during laparoscopic ultrasound (LPUS) surgery is detected; and The 3D pose of the ultrasound probe deployed during the LPUS procedure is estimated based on the detected 2D position of the ultrasound probe and the ultrasound model used for the ultrasound probe.

2. The method according to claim 1, wherein, The step of detecting the 2D position of the ultrasound image includes: Extract the 2D region corresponding to the ultrasound probe from the 2D LP image; The 2D position of the ultrasound probe is determined based on the extracted 2D region.

3. The method according to claim 2, wherein, The extraction step is based on a segmentation model for identifying ultrasound probes, obtained via machine learning from training data; and The 2D position of the ultrasonic probe is defined as the centroid of the 2D region.

4. The method according to claim 1, wherein, The steps for estimating the 3D pose of the ultrasound probe include: Based on the transformation matrix obtained during the calibration of the LP camera, the 2D position is transformed into the 3D coordinates of the ultrasound probe in the 3D space; Estimate the 3D orientation of the ultrasonic probe based on the ultrasonic model used for the ultrasonic probe; and The estimated 3D pose of the ultrasound probe is generated based on the 3D coordinates and the 3D orientation.

5. The method according to claim 4, wherein, The steps for estimating the 3D orientation include: Multiple virtual ultrasound probe images are generated by projecting the ultrasound model from the 3D coordinates using different 3D orientations, wherein each of the multiple virtual ultrasound probe images includes features defined by the ultrasound probe; Detect the corresponding defined 2D features of the ultrasound probe that can be observed in the 2D LP image; The defined features in each of the plurality of virtual ultrasound probe images are compared with the corresponding defined 2D features detected from the 2D LP image to obtain a comparison result; and Based on the comparison results, one of the 3D orientations is selected as the estimated 3D orientation.

6. The method according to claim 4, wherein, The steps for estimating the 3D orientation include: Multiple virtual ultrasound images are generated based on slices of a 3D model, the slices being obtained based on estimated 3D coordinates, different 3D orientations of the ultrasound probe, and the ultrasound model. Each of the plurality of virtual ultrasound images is compared with a 2D ultrasound image acquired by the ultrasound probe to obtain a comparison result; and Based on the comparison results, one of the different possible 3D orientations is selected as the estimated 3D orientation.

7. The method of claim 1, further comprising: labeling at least one blood vessel in a 2D ultrasound image acquired by the ultrasound probe based on the estimated 3D pose of the ultrasound probe.

8. The method according to claim 7, wherein, The step of marking the at least one blood vessel includes: Detecting 2D structures in the 2D ultrasound images; Access a 3D model of the target, which represents the target organ and related anatomical structures, including blood vessels; The slices of the 3D model are determined based on the 3D pose of the ultrasound probe and the ultrasound model; A virtual 2D blood vessel image is generated based on each blood vessel present in the 2D slice; and Labels are provided to 2D structures that correspond to blood vessels.

9. A machine-readable and non-transient medium having information recorded thereon, wherein, When the information is read by the machine, the machine performs the following steps: The two-dimensional (2D) position of an ultrasound probe visible in 2D laparoscopic (LP) images acquired by an LP camera previously calibrated in three-dimensional (3D) space and inserted into the patient during laparoscopic ultrasound (LPUS) surgery is detected; and The 3D pose of the ultrasound probe deployed during the LPUS procedure is estimated based on the detected 2D position of the ultrasound probe and the ultrasound model used for the ultrasound probe.

10. The medium according to claim 9, wherein, The step of detecting the 2D position of the ultrasound image includes: Extract the 2D region corresponding to the ultrasound probe from the 2D LP image; The 2D position of the ultrasound probe is determined based on the extracted 2D region.

11. The medium according to claim 10, wherein, The extraction step is based on a segmentation model for identifying ultrasound probes, obtained via machine learning from training data; and The 2D position of the ultrasonic probe is defined as the centroid of the 2D region.

12. The medium according to claim 9, wherein, The steps for estimating the 3D pose of the ultrasound probe include: Based on the transformation matrix obtained during the calibration of the LP camera, the 2D position is transformed into the 3D coordinates of the ultrasound probe in the 3D space; Estimate the 3D orientation of the ultrasonic probe based on the ultrasonic model used for the ultrasonic probe; and The estimated 3D pose of the ultrasound probe is generated based on the 3D coordinates and the 3D orientation.

13. The medium according to claim 12, wherein, The steps for estimating the 3D orientation include: Multiple virtual ultrasound probe images are generated by projecting the ultrasound model from the 3D coordinates using different 3D orientations, wherein each of the multiple virtual ultrasound probe images includes features defined by the ultrasound probe; Detect the corresponding defined 2D features of the ultrasound probe that can be observed in the 2D LP image; The defined features in each of the plurality of virtual ultrasound probe images are compared with the corresponding defined 2D features detected from the 2D LP image to obtain a comparison result; and Based on the comparison results, one of the 3D orientations is selected as the estimated 3D orientation.

14. The medium according to claim 12, wherein, The steps for estimating the 3D orientation include: Multiple virtual ultrasound images are generated based on slices of a 3D model, the slices being obtained based on estimated 3D coordinates, different 3D orientations of the ultrasound probe, and the ultrasound model. Each of the plurality of virtual ultrasound images is compared with a 2D ultrasound image acquired by the ultrasound probe to obtain a comparison result; and Based on the comparison results, one of the different possible 3D orientations is selected as the estimated 3D orientation.

15. The medium according to claim 9, wherein, When the information is read by the machine, the machine further performs the following steps: marking at least one blood vessel in the 2D ultrasound image acquired by the ultrasound probe based on the estimated 3D pose of the ultrasound probe.

16. The medium according to claim 15, wherein, The step of marking the at least one blood vessel includes: Detecting 2D structures in the 2D ultrasound images; Access a 3D model of the target, which represents the target organ and related anatomical structures, including blood vessels; The slices of the 3D model are determined based on the 3D pose of the ultrasound probe and the ultrasound model; A virtual 2D blood vessel image is generated based on each blood vessel present in the 2D slice; and Labels are provided to 2D structures that correspond to blood vessels.

17. A system comprising: An LP U-probe position detector, implemented by a processor, is configured to detect the two-dimensional (2D) position of an ultrasound probe visible in 2D laparoscopic (LP) images acquired by an LP camera previously calibrated in three-dimensional (3D) space and inserted into the patient during laparoscopic ultrasound (LPUS) surgery; and A 3D U-probe pose estimator, implemented by a processor and configured to estimate the 3D pose of the ultrasound probe deployed in the LPUS procedure based on the detected 2D position of the ultrasound probe and an ultrasound model for the ultrasound probe.

18. The system according to claim 17, wherein, The step of detecting the 2D position of the ultrasound image includes: Extract the 2D region corresponding to the ultrasound probe from the 2D LP image; The 2D position of the ultrasound probe is determined based on the extracted 2D region.

19. The system according to claim 18, wherein, The extraction step is based on a segmentation model for identifying ultrasound probes, obtained via machine learning from training data; and The 2D position of the ultrasonic probe is defined as the centroid of the 2D region.

20. The system according to claim 17, wherein, The steps for estimating the 3D pose of the ultrasound probe include: Based on the transformation matrix obtained during the calibration of the LP camera, the 2D position is transformed into the 3D coordinates of the ultrasound probe in the 3D space; Estimate the 3D orientation of the ultrasonic probe based on the ultrasonic model used for the ultrasonic probe; and The estimated 3D pose of the ultrasound probe is generated based on the 3D coordinates and the 3D orientation.

21. The system according to claim 20, wherein, The steps for estimating the 3D orientation include: Multiple virtual ultrasound probe images are generated by projecting the ultrasound model from the 3D coordinates using different 3D orientations, wherein each of the multiple virtual ultrasound probe images includes features defined by the ultrasound probe; Detect the corresponding defined 2D features of the ultrasound probe that can be observed in the 2D LP image; The defined features in each of the plurality of virtual ultrasound probe images are compared with the corresponding defined 2D features detected from the 2D LP image to obtain a comparison result; and Based on the comparison results, one of the 3D orientations is selected as the estimated 3D orientation.

22. The system according to claim 20, wherein, The steps for estimating the 3D orientation include: Multiple virtual ultrasound images are generated based on slices of a 3D model, the slices being obtained based on estimated 3D coordinates, different 3D orientations of the ultrasound probe, and the ultrasound model. Each of the plurality of virtual ultrasound images is compared with a 2D ultrasound image acquired by the ultrasound probe to obtain a comparison result; and Based on the comparison results, one of the different possible 3D orientations is selected as the estimated 3D orientation.

23. The system of claim 17, further comprising a 2D vessel tagger generator, the 2D vessel tagger generator being implemented by a processor and configured to tag at least one vessel in a 2D ultrasound image acquired by the ultrasound probe based on the estimated 3D pose of the ultrasound probe.

24. The system according to claim 23, wherein, The step of marking the at least one blood vessel includes: Detecting 2D structures in the 2D ultrasound images; Access a 3D model of the target, which represents the target organ and related anatomical structures, including blood vessels; The slices of the 3D model are determined based on the 3D pose of the ultrasound probe and the ultrasound model; A virtual 2D blood vessel image is generated based on each blood vessel present in the 2D slice; and Labels are provided to 2D structures that correspond to blood vessels.

Images