Method and system for estimating ultrasound probe pose in laparoscopic ultrasound procedures

Abstract

A method and system for estimating a 3D pose of an ultrasound probe based on acquired 2D ultrasound images. During surgery, the 3D probe pose is projected to a corresponding 2D location on the 2D laparoscopic image. An ultrasound probe is placed at / near a selected one of the 3D probe poses to acquire a 2D ultrasound image from which an anatomical structure (AS) is detected and used to generate an AS mask (ASM). The 3D probe pose of the ultrasound probe is estimated based on the ASM using an ASM pose mapping model and used to generate an optimal 3D probe pose estimate.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202480030545.9 (22) Application Date 2024.05.02 (30) Priority Data 63 / 464,282 2023.05.05 US (85) PCT International Application Entering National Phase Date 2025.11.05 (86) PCT International Application Application Data PCT / US2024 / 027475 2024.05.02 (87) PCT International Application Publication Data WO2024 / 233263 EN 2024.11.14 (71) Applicant: Medtronic Technology Co., Ltd. Address: New Jersey, USA (72) Inventors: Qi Xiao, Zang Xiaonan, Wei Guoqing, Fan Li, Zeng Xiaolan, Qian Jianzhong (74) Patent Agency: Shanghai Patent & Trademark Agency Co., Ltd. 31100 Patent Attorney Xu Qian Zhou Quan (51) Int.Cl. A61B 1 / 313 (2006.01) G06T 11 / 60 (2006.01) G06T 7 / 11 (2006.01) G06T 7 / 33 (2006.01) G06T 7 / 73 (2006.01) (54) Invention Title: Method and System for Estimating Ultrasonic Probe Pose in Laparoscopic Ultrasonic Surgery (57) Abstract: A method and system for estimating the 3D pose of an ultrasound probe based on acquired 2D ultrasound images. During surgery, the 3D probe pose is projected onto a corresponding 2D location on a 2D laparoscopic image. The ultrasound probe is placed at / near a selected 3D probe pose to acquire a 2D ultrasound image, from which anatomical structures (AS) are detected and the anatomical structures (AS) are used to generate an AS mask (ASM). The 3D probe pose of the ultrasound probe is estimated based on the ASM pose mapping model and used to generate the optimal 3D probe pose estimate. Claims: 5 pages; Description: 12 pages; Drawings: 20 pages. CN 121079021 A 2025.12.05 CN 1 21 07 90 21 A 1. A method implemented on at least one processor, memory, and communication platform, the method comprising: during surgery, projecting one or more three-dimensional (3D) probe poses onto corresponding 2D probe positions on a two-dimensional (2D) image, wherein the 2D image is acquired by a laparoscopic camera inserted into the patient during the surgery and pointing towards a target organ; receiving 2D ultrasound images acquired by an ultrasound probe placed at or near one of the 2D probe positions; and detecting anatomical structures from the 2D ultrasound images with respect to each of at least some of the 2D ultrasound images.1. The method of claim 1, wherein the anatomical structure mask (ASM) is generated for the 2D ultrasound image based on the detected anatomical structure; one or more 3D probe pose estimates of the ultrasound probe are predicted based on the ASM via an ASM pose mapping model; and an optimal 3D probe pose estimate is generated based on the 3D probe pose estimate, the 3D probe pose estimate being based on the ASM prediction obtained from the 2D ultrasound image acquired by the ultrasound probe in varying 3D probe poses. 2. The method of claim 1, wherein the one or more 3D probe poses are specified by a user for a 3D model of the target organ presented to the user; and are used to create the ASM pose mapping model based on the ASM, the ASM being obtained with respect to the one or more 3D probe poses. 3. The method of claim 1, wherein each of the one or more 3D probe pose estimates is predicted using a metric indicating the confidence level of the 3D probe pose estimate. 4. The method of claim 3, wherein the changing 3D probe pose is caused by motion of the ultrasound probe about a selected 2D probe position, wherein the motion includes changes in the 3D coordinates or 3D orientation of the ultrasound probe. 5. The method of claim 4, wherein the step of generating an optimal 3D probe pose estimate comprises: determining a trajectory of the ultrasound probe based on the changing 3D probe pose; identifying 3D probe pose estimates predicted for each of the changing 3D probe poses along the trajectory; and determining the optimal 3D probe pose estimate based on the 3D probe pose estimates predicted for the changing 3D probe poses along the trajectory. 6. The method of claim 5, wherein the step of determining the optimal 3D probe pose estimate comprises: selecting one of the 3D probe pose estimates for each of the changing probe poses along the trajectory, such that the metric of the selected predicted 3D probe pose is maximized; and determining the optimal 3D probe pose estimate based on the selected predicted 3D probe pose along the trajectory. 7. The method according to claim 1, wherein the ASM pose mapping model is obtained by: presenting a three-dimensional (3D) model constructed to model the target organ to the user before the surgery; receiving instructions from the user specifying the poses of the plurality of 3D probes; obtaining a simulated 3D probe pose including the plurality of 3D probe poses specified by the user and additional 3D probe poses automatically generated with respect to the plurality of 3D probe poses; and creating the ASM pose mapping model based on the simulated 3D probe poses. Claims 1 / 5 pages 2 CN 121079021 A8. The method of claim 7, wherein the additional 3D probe pose is randomly generated within a 3D region defined based on one or more of the plurality of 3D probe poses. 9. The method of claim 7, wherein the step of creating the ASM pose mapping model comprises: with respect to each of the simulated 3D probe poses: determining a viewpoint based on the simulated 3D probe pose relative to a presented 3D model; obtaining a simulated two-dimensional (2D) ultrasound image acquired from the viewpoint by a simulated ultrasound probe placed at the simulated 3D probe pose; deriving an anatomical structure mask (ASM) based on the anatomical structures in the simulated 2D ultrasound image; and creating a simulated mapping pair involving the ASM and the simulated 3D probe pose; and obtaining the ASM pose mapping model based on the simulated mapping pair. 10. The method of claim 9, wherein the step of obtaining the ASM pose mapping model comprises: generating training data for machine learning based on the simulated mapping pair; and training the ASM pose mapping model using the training data. 11. A machine-readable and non-transient medium having information recorded thereon, wherein the information, when read by the machine, causes the machine to perform the following steps: during surgery, projecting one or more three-dimensional (3D) probe poses onto corresponding 2D probe positions on a two-dimensional (2D) image, wherein the 2D image is acquired by a laparoscopic camera directed at a target organ and inserted into the patient during the surgery; receiving 2D ultrasound images acquired by an ultrasound probe placed at or near one of the 2D probe positions; for each of at least some of the 2D ultrasound images, detecting anatomical structures from the 2D ultrasound images; generating an anatomical structure mask (ASM) for the 2D ultrasound images based on the detected anatomical structures; predicting one or more 3D probe pose estimates for the ultrasound probe based on the ASM via an ASM pose mapping model; and generating an optimal 3D probe pose estimate based on the 3D probe pose estimate, the 3D probe pose estimate being based on the ASM prediction obtained from the 2D ultrasound images acquired by the ultrasound probe in varying 3D probe poses. 12. The medium of claim 11, wherein the one or more 3D probe poses are specified by a user for a 3D model of the target organ presented to the user; and are used to create the ASM pose mapping model based on the ASM, the ASM being obtained with respect to the one or more 3D probe poses. 13. The medium of claim 11, wherein each of the one or more 3D probe pose estimations is predicted using a metric indicating the confidence level of the 3D probe pose estimation.14. The medium according to claim 13, wherein the changing 3D probe pose is caused by the movement of the ultrasound probe about a selected 2D probe position, wherein the movement includes changes in the 3D coordinates or 3D orientation of the ultrasound probe. 15. The medium according to claim 14, wherein the step of generating an optimal 3D probe pose estimate comprises: determining a trajectory of the ultrasound probe based on the changing 3D probe pose; identifying a 3D probe pose estimate predicted for each of the changing 3D probe poses along the trajectory; and determining the optimal 3D probe pose estimate based on the 3D probe pose estimate predicted for the changing 3D probe pose along the trajectory. 16. The medium of claim 15, wherein the step of determining the optimal 3D probe pose estimation comprises: selecting one of the 3D probe pose estimates for each of the changing probe poses along the trajectory, such that the metric of the selected predicted 3D probe pose is maximized; and determining the optimal 3D probe pose estimation based on the selected predicted 3D probe pose along the trajectory. 17. The medium of claim 11, wherein the ASM pose mapping model is obtained by: presenting a three-dimensional (3D) model constructed to model the target organ to a user prior to the surgery; receiving instructions from the user specifying the plurality of 3D probe poses; obtaining a simulated 3D probe pose including the plurality of 3D probe poses specified by the user and additional 3D probe poses automatically generated with respect to the plurality of 3D probe poses; and creating the ASM pose mapping model based on the simulated 3D probe pose. 18. The medium of claim 17, wherein the additional 3D probe pose is randomly generated within a 3D region defined based on one or more of the plurality of 3D probe poses. 19. The medium of claim 17, wherein the step of creating the ASM pose mapping model comprises: with respect to each of the simulated 3D probe poses: determining a viewing angle based on the simulated 3D probe pose relative to a presented 3D model; obtaining a simulated two-dimensional (2D) ultrasound image acquired from the viewing angle by a simulated ultrasound probe placed at the simulated 3D probe pose; deriving an anatomical structure mask (ASM) based on the anatomical structures in the simulated 2D ultrasound image; and creating a simulated mapping pair involving the ASM and the simulated 3D probe pose; and obtaining the ASM pose mapping model based on the simulated mapping pair.20. The medium according to claim 19, wherein the step of obtaining the ASM pose mapping model comprises: generating training data for machine learning based on the simulated mapping pair; and training the ASM pose mapping model using the training data. 21. A system comprising: an ultrasound probe pose projector, the ultrasound probe pose projector being implemented by a processor and configured to project one or more three-dimensional (3D) probe poses onto corresponding 2D probe positions on a two-dimensional (2D) image during surgery, wherein the 2D image is acquired by a laparoscopic camera inserted into a patient during the surgery and pointing towards a target organ; and an ultrasound-based 3D anatomical structure visualizer, the ultrasound-based 3D anatomical structure visualizer being implemented by the processor and configured to receive 2D ultrasound images acquired by an ultrasound probe placed at or near one of the 2D probe positions, and, with respect to each of at least some of the 2D ultrasound images, detect anatomical structures from the 2D ultrasound images, and generate an anatomical structure mask (ASM) for the 2D ultrasound images based on the detected anatomical structures. (Claims 3 / 5, Page 4, CN 121079021 A) The system comprises: predicting one or more 3D probe pose estimates of the ultrasound probe based on the ASM pose mapping model; and generating an optimal 3D probe pose estimate based on the 3D probe pose estimates, wherein the 3D probe pose estimates are based on ASM predictions obtained from 2D ultrasound images acquired by the ultrasound probe in varying 3D probe poses. 22. The system of claim 21, wherein the one or more 3D probe poses are specified by a user for a 3D model of the target organ presented to the user; and used to create the ASM pose mapping model based on the ASM obtained with respect to the one or more 3D probe poses. 23. The system of claim 21, wherein each of the one or more 3D probe pose estimates is predicted using a metric indicating the confidence level of the 3D probe pose estimate. 24. The system of claim 23, wherein the varying 3D probe pose is caused by movement of the ultrasound probe around a selected 2D probe position, wherein the movement includes changes in the 3D coordinates or 3D orientation of the ultrasound probe. 25. The system of claim 24, wherein the step of generating the optimal 3D probe pose estimate comprises: determining the trajectory of the ultrasound probe based on the changing 3D probe pose; identifying a 3D probe pose estimate predicted for each of the changing 3D probe poses along the trajectory.Based on the 3D probe pose estimation predicted with respect to the changes along the trajectory, the optimal 3D probe pose estimation is determined. 26. The system of claim 25, wherein the step of determining the optimal 3D probe pose estimation comprises: selecting one of the 3D probe pose estimations for each of the changing probe poses along the trajectory, such that the metric of the selected predicted 3D probe pose is maximized; and determining the optimal 3D probe pose estimation based on the selected predicted 3D probe pose along the trajectory. 27. The system of claim 21, further comprising an ASM pose mapping model generator, the ASM pose mapping model generator being implemented by a processor and configured to obtain the ASM pose mapping model by: presenting a three-dimensional (3D) model constructed to model the target organ to a user prior to the surgery; receiving instructions from the user specifying the plurality of 3D probe poses; obtaining a simulated 3D probe pose including the plurality of 3D probe poses specified by the user and additional 3D probe poses automatically generated with respect to the plurality of 3D probe poses; and creating the ASM pose mapping model based on the simulated 3D probe poses. 28. The system of claim 27, wherein the additional 3D probe poses are randomly generated within a 3D region defined based on one or more of the plurality of 3D probe poses. 29. The system of claim 27, wherein the step of creating the ASM pose mapping model comprises: for each of the simulated 3D probe poses: determining a viewpoint based on the simulated 3D probe pose relative to a presented 3D model; obtaining a simulated two-dimensional (2D) ultrasound image acquired from the viewpoint by a simulated ultrasound probe placed at the simulated 3D probe pose; deriving an anatomical structure mask (ASM) based on the anatomical structures in the simulated 2D ultrasound image; and creating a simulated mapping pair relating to the ASM and the simulated 3D probe pose; and obtaining the ASM pose mapping model based on the simulated mapping pair. 30. The system of claim 29, wherein the step of obtaining the ASM pose mapping model comprises: generating training data for machine learning based on the simulated mapping pair; and training the ASM pose mapping model using the training data. Claims 5 / 5, page 6, CN 121079021 A Method and System for Estimating Ultrasound Probe Orientation in Laparoscopic Ultrasound Surgery. Cross-Reference to Related Applications

[0001] This application claims the filing date of U.S. Provisional Patent Application No. 63 / 464,282, filed May 5, 2023.The priority interest of this U.S. provisional patent application is incorporated herein by reference in its entirety. Background Art

[0002] This teaching generally relates to computers. More specifically, this teaching relates to signal processing.

[0003] With advancements in technology, an increasing number of tasks are now performed with the aid of computers. Various industries benefit from such technological advancements, including the medical industry, where large amounts of image data can be processed by computers to capture anatomical information of patients to identify anatomical structures of interest (e.g., organs, bones, blood vessels, or anomalous nodules), obtain measurements of each object of interest (e.g., the size of nodules growing in organs), and visualize related features (e.g., three-dimensional (3D) visualization of anomalous nodules). Such information can be used for a variety of purposes. For example, 3D models can be constructed for organs (e.g., the liver along different dimensions according to its volume, shape, or size) or different parts of an organ (e.g., nodules growing inside the liver, vascular structures inside and near the organ). An example of a 3D liver model 100 is shown in Figure 1A, and a 3D vascular tree model 110 constructed based on blood vessels detected inside the liver is shown in Figure 1B. The 3D model constructed in this way can also be used for various purposes, such as diagnosis, preoperative planning, and intraoperative navigation.

[0004] Figure 1C shows a laparoscopic ultrasound (LUS) surgical setup in which a patient 130 is undergoing laparoscopic surgery at the top of an operating table 120. A laparoscopic camera 140 is inserted into the patient and can be controlled to reach a position where the laparoscopic camera 140 can capture video images 150 of the observable scene near surgical instruments (not shown). The video images of the visible content near the surgical instruments acquired by the laparoscopic camera 140 can provide visual guidance to the surgeon as the user, such as a surgeon, manipulates the surgical instruments to perform surgical actions. In addition, during the operation, an ultrasound probe 160 can also be inserted into the patient and used to scan to “see” what is under the organs. For example, in liver resection surgery, an ultrasound probe can be used to help “see” what is inside the liver before the surgeon cuts it open to help the surgeon determine how to cut the liver. An exemplary ultrasound image formed based on the information acquired by the ultrasound probe 160 is shown at 170 in Figure 1C.

[0005] An ultrasound probe operates by emitting a sound beam that travels independently along a path to a plane and is reflected back by an object at the plane, generating an ultrasound image based on the reflected sound waves. Generally, what the ultrasound probe "sees" corresponds to information from a slice in space or a plane, as shown in Figure 1D, where the ultrasound probe scans organ 180, and the "observable" content is a slice of information from plane 190. Therefore, the content captured by the ultrasound probe may be limited or incomplete.An example is shown in Figure 1E, where an ultrasound image 195 generated based on information acquired from plane 190 shows a vascular structure 197 within an organ, which is visible because the vascular structure 197 is located on plane 190 where the sound beam emitted by the ultrasound probe reaches. Compared to the complete 3D vascular tree 110 of the organ shown in Figure 1B, the vascular structure 197 corresponds only to a portion of the 3D vascular tree 110 in a spatial slice corresponding to plane 190 marked in Figure 1B.

[0006] Such partial information can make it difficult for surgeons to determine what corresponds to the blood vessels observed in the ultrasound image and thus to make operational decisions. For example, in some surgeries on organs such as the liver, some blood vessels may need to be pushed aside or clamped to temporarily stop blood flow. In these cases, if what is seen does not inform the surgeon of the corresponding blood vessels, the detected ultrasound image may be confusing rather than helpful.

[0007] Therefore, a solution capable of addressing the above challenges is needed.

[0008] The teachings disclosed herein relate to methods, systems, and programming for information management. More specifically, the teachings relate to methods, systems, and programming related to hash tables and storage management using hash tables.

[0009] In one example, a method, implemented on a machine having at least one processor, storage device, and a communication platform capable of connecting to a network, is used to estimate the 3D pose of an ultrasound probe based on acquired 2D ultrasound images. During surgery, the 3D probe pose is projected onto a corresponding 2D location on a 2D laparoscopic image. The ultrasound probe is placed at / near a selected 3D probe pose to acquire a 2D ultrasound image, from which anatomical structures (AS) are detected and used to generate an AS mask (ASM). The 3D probe pose of the ultrasound probe is estimated based on the ASM using an ASM pose mapping model and used to generate an optimal 3D probe pose estimate.

[0010] In various examples, systems for estimating the 3D pose of an ultrasound probe based on acquired 2D ultrasound images are disclosed. The system includes an ultrasound probe pose projector and an ultrasound-based 3D anatomical structure visualizer. The former is used to project one or more three-dimensional (3D) probe poses onto corresponding 2D probe locations on a 2D laparoscopic image during surgery. The ultrasound probe is placed at / near a selected 3D probe pose to acquire a 2D ultrasound image. The ultrasound-based 3D anatomical structure visualizer is used to detect anatomical structures (AS) from the 2D ultrasound image to generate an AS mask (ASM), and to estimate the 3D probe pose of the ultrasound probe using an ASM pose mapping model based on the ASM. The estimated 3D probe pose is used to generate an optimal 3D probe pose estimate.

[0011] Other concepts relate to software used to implement the present teachings. Software products according to this concept include at least one machine-readable, non-transient medium and information carried by that medium. The information carried by the medium may be executable program code data, parameters associated with the executable program code, and / or information related to a user, request, content, or other additional information.

[0012] Another example is a machine-readable, non-transient, and tangible medium on which information for estimating the 3D pose of an ultrasound probe based on acquired 2D ultrasound images is recorded. The information on the medium, when read by a machine, causes the machine to perform various steps. The 3D probe pose is projected onto a corresponding 2D location on a 2D laparoscopic image during the procedure. The ultrasound probe is placed at / near a selected 3D probe pose to acquire a 2D ultrasound image, from which anatomical structures (AS) are detected, and these anatomical structures (AS) are used to generate an AS mask (ASM). The 3D probe pose of the ultrasound probe is estimated based on the ASM pose mapping model and used to generate the optimal 3D probe pose estimate.

[0013] Additional advantages and novel features will be set forth in part in the description below, and in part will be apparent to those skilled in the art upon review of the following and the accompanying drawings, or may be learned by production or operation of examples. The advantages of this teaching can be realized and obtained by practice or by using various aspects of the methods, tools and combinations set forth in the detailed examples discussed below. Brief Description of the Drawings

[0014] The methods, systems and / or programming described herein will be further described by way of exemplary embodiments. These exemplary embodiments will be detailed with reference to the accompanying drawings. These embodiments are non-limiting exemplary embodiments, wherein the same reference numerals denote similar structures throughout several views of the accompanying drawings, and wherein: Specification 2 / 12 Page 8 CN 121079021 A

[0015] Figures 1A–1B show exemplary 3D models of organs and 3D vascular tree models of blood vessels within the organs;

[0016] Figure 1C shows a typical LUS surgical setup;

[0017] Figures 1D–1E illustrate the characteristics of an ultrasound probe and the information acquired by that ultrasound probe;

[0018] Figure 2 depicts an exemplary high-level system diagram of a system for estimating 3D ultrasound probe pose (USPP) and visualizing anatomical structures (AS) detected based on 3D USPP according to embodiments of the present teachings;

[0019] Figure 3A shows a user-selected 3D USPP according to an embodiment of the present teachings;

[0020] Figures 3B–3C illustrate a discrete mapping from an anatomical structure mask (ASM) to a 3D USPP automatically generated as training data for machine learning according to embodiments of the present teachings;

[0021] Figure 3D shows an exemplary ASM obtained from blood vessels detected from 2D ultrasound images according to an embodiment of the present teaching;

[0022] Figure 3E illustrates an ASM pose mapping model created via machine learning based on discrete ASM pose mapping data according to an embodiment of the present teaching;

[0023] Figure 4A is a flowchart of the preoperative process of an AS visualization system based on 3D USPP according to an embodiment of the present teaching, generating an ASM pose mapping model via ultrasound probe pose selected with respect to a 3D organ model;

[0024] Figure 4B is a flowchart of the intraoperative process of an AS visualization system based on 3D USPP according to an embodiment of the present teaching, estimating the 3D ultrasound probe pose during LUS surgery based on the ASM pose mapping model;

[0025] Figure 5A depicts an exemplary high-level system diagram of an ASM pose mapping model generator according to an embodiment of the present teaching;

[0026] Figure 5B is a flowchart of an exemplary process of an ASM pose mapping model generator according to an embodiment of the present teaching;

[0027] Figure 6A depicts an exemplary high-level system diagram of a pose-based 2D ASM determiner according to an embodiment of the present teaching;

[0028] Figure 6B is a flowchart of an exemplary process of a pose-based 2D ASM determiner according to an embodiment of the present teaching;

[0029] Figure 7A illustrates a 2D projection of an exemplary 3D ultrasound probe pose selected relative to a 3D model according to an embodiment of the present teaching;

[0030] Figure 7B depicts an exemplary high-level system diagram of an ultrasound-based 3D anatomical structure visualizer according to an embodiment of the present teaching;

[0031] Figure 7C is a flowchart of an exemplary process of an ultrasound-based 3D anatomical structure visualizer according to an embodiment of the present teaching;

[0032] Figure 7D illustrates an exemplary operation of estimating 3D USPP based on ASM extracted from a 2D ultrasound image according to an embodiment of the present teaching;

[0033] Figure 8 is a schematic diagram of an exemplary mobile device architecture that can be used to implement a dedicated system implementing the present teaching according to various embodiments; and

[0034] Figure 9 is a schematic diagram of an exemplary computing device architecture that can be used to implement a dedicated system implementing the present teaching according to various embodiments. Detailed Description

[0035] In the following detailed description, numerous specific details are set forth by way of example in order to facilitate a thorough understanding of the relevant teachings. However, it should be apparent to those skilled in the art that this teaching can be practiced without these details. In other instances, well-known methods, processes, components, and / or systems have been described at a relatively high level without detail in order to avoid unnecessarily obscuring various aspects of this teaching.

[0036] This teaching discloses exemplary methods, systems, and specifications for an AS visualization system based on 3D USPP in LUS surgery. 3 / 12 pages 9 CN 121079021 AImplementation. LUS surgery can be used for various surgeries, such as laparoscopic liver resection. Using an ultrasound probe can provide a view of the anatomical structures beneath the surface of an organ, such as blood vessels and / or tumors. Prior to surgery, a 3D organ model can be constructed based on 3D medical images obtained via scanning, such as computed tomography (CT) or magnetic resonance imaging (MRI). Ultrasound images acquired by an ultrasound probe are typically noisy and contain only partial information. Therefore, ultrasound images often do not provide sufficient information to reveal anatomical structures to provide effective visual guidance during surgery.

[0037] This teaching discloses a method and system for estimating the 3D probe pose (USPP) of an ultrasound probe based on partial information observed in laparoscopic and ultrasound images. The estimated 3D USPP can then be used to determine how to visualize 3D anatomical structures corresponding to what is observed in 2D images. Specifically, the estimated 3D USPP is used to determine an appropriate viewpoint about a 3D organ model such that the 3D anatomical structures modeled in the 3D organ model as viewed from that viewpoint can be visualized to allow the user to see content corresponding to the limited content observed from 2D ultrasound images. To estimate 3D USPPs, the methods and systems according to this teaching involve preoperative and intraoperative procedures. During the preoperative planning phase, individual 3D USPPs can be specified, each of which can be used to determine the viewpoint of viewing a 3D organ model via a simulated ultrasound probe to create a 2D simulated ultrasound image with anatomical structures captured by the ultrasound probe.

[0038] Anatomical structures in each simulated ultrasound image can be identified, enabling the generation of an anatomical structure mask (ASM). The ASM can correspond to a binarized 2D image where pixels on the anatomical structure are in one state (e.g., state 1), while any other location is in another state (e.g., state 0). An ASM pose mapping model can be created via machine learning using a specified set of 3D probe poses and their corresponding ASMs to predict 3D probe poses based on 2D ASMs obtained from 2D ultrasound images. The ASM pose mapping model created in this manner can then be used during surgery by obtaining 2D ASMs based on real-time acquired ultrasound images and predicting candidate 3D USPPs using, for example, corresponding probabilities or confidence scores.

[0039] During LUS surgery, 2D ultrasound images can be acquired by a user-operated ultrasound probe. These 2D ultrasound images can be processed to identify anatomical structures, thereby obtaining the corresponding ASM. Using an ASM pose mapping model created preoperatively, for each ASM derived from the 2D ultrasound image, the ASM pose mapping model predicts multiple candidate 3D probe poses with probabilities. These multiple candidate 3D probe poses can then be used for optimization to identify the best-estimated 3D USPP. The estimated 3D...USPP can then be used to determine the viewpoint relative to the 3D organ model, such that the 3D anatomical structures in the 3D organ model viewed from this viewpoint correspond to the partially visible anatomical structures in the 2D ultrasound image. The 3D anatomical structures thus identified can be presented to the user from a viewpoint consistent with what is partially seen in the 2D ultrasound image, but with a 3D view of the complete anatomical structure. Detailed information regarding preoperative and intraoperative procedures is provided herein with reference to Figures 2A-7C to enable estimation of the 3D ultrasound probe pose and visualization of the 3D anatomical structures determined based on the estimated 3D probe pose.

[0040] Figure 2 depicts an exemplary high-level system diagram of an AS visualization system 200 based on 3D USPP according to an embodiment of this teaching. System 200 includes two parts, a preoperative part and an intraoperative part. The preoperative part includes presenting a 3D model 250 of the target organ to a user 210 and an ASM pose mapping model generator 220 that interacts with the user 210 to receive a user-specified probe pose 230 defined with respect to the 3D model 250. Based on these user-specified probe poses 230, the ASM pose mapping model generator 220 creates an ASM pose mapping model 240. In some embodiments, the ASM pose mapping model 240 is obtained via machine learning based on training data created based on the user-specified probe poses 230.

[0041] The intraoperative portion includes an ultrasound (US) probe pose projector 260, one or more intraoperative displays 270, and an ultrasound-based 3D anatomical (AS) visualizer 290. During LUS surgery, a laparoscopic camera acquires 2D laparoscopic images. The US probe pose projector 260 is provided to project the user-specified 3D probe pose 230 onto the 2D laparoscopic images as a 2D projection position to guide the user about where to place the ultrasound probe during surgery. Based on the projected 2D probe position, the user can select to place the ultrasound probe at or near one of these 2D probe positions. Ultrasonic Probe - Instruction Manual 4 / 12 Pages 10 CN 121079021 A Once placed, a real-time ultrasound (US) image 280 is acquired. The selected 2D probe position and the real-time US image 280 are then sent to an ultrasound-based 3D AS visualizer 290, which uses an ASM pose mapping model 240 to estimate the 3D ultrasound probe pose or 3D USPP corresponding to the selected 2D probe position. The 3D USPP needs to be estimated for the following reasons: Due to distortions and / or inaccuracies in registration, the 2D probe position projected from the user-specified probe pose 230 onto the 2D image may not accurately correspond to the 3D user-specified probe pose 230. Therefore, it is necessary to estimate the actual 3D USPP corresponding to the selected 2D probe position.USPP. Details relating to the ASM pose mapping model generator 220 and the ultrasound-based 3D anatomical structure visualizer 290 will be provided with reference to Figures 3A–7C.

[0042] As discussed herein, an ASM pose mapping model 240 is obtained prior to surgery. For this purpose, a 3D model 250 of the target organ can first be visualized to the user. The visualized 3D model can be manipulated by the user, for example, by scrolling, shifting, etc., to view the 3D model from different perspectives. The user can select any 3D position in the visualized 3D model as a simulated ultrasound probe pose, which defines the perspective from which the 3D model is viewed. Figure 3A illustrates an exemplary user-specified 3D US probe pose and its representation according to an embodiment of this teaching. As shown in Figure 3A, there are k user-specified 3D probe poses, namely pose 1 230-1, pose 2 230-2, ..., pose i 230-3, ..., and pose k 230-5. Such user-specified probe poses can be represented based on parameters, for example, in their respective six degrees of freedom, including, for example, 3D coordinates defined in the X-axis 310, Y-axis 320, and Z-axis 330, as well as pitch (PI) 340, roll (RO) 350, and yaw (YA) 360.

[0043] Each user-specified probe pose defines a viewpoint about the visualized 3D model and can be applied to generate simulated ultrasound images that can be captured by the ultrasound probe. An example is illustrated in FIG1D, where the ultrasound probe 160, once positioned at a specific location with a viewpoint relative to the organ 180, produces a view 190 of the organ determined via that viewpoint. That is, each user-specified probe pose can be used to determine a slice 190 in the 3D model, which can be used to generate a 2D ultrasound image with information about the slice 190. For each such 2D slice image, the anatomical structures seen therein can be detected, and a corresponding ASM can be generated. In this way, the ASM obtained from the 2D slice image corresponds to a user-specified probe pose, which generates an ASM-3D USPP pair or a mapping from the ASM to the 3D probe pose.

[0044] To construct the ASM pose mapping model 240, the mapping from the ASM generated based on the 2D slice image to the corresponding user-specified probe pose can be used to map the ASM obtained based on real-time ultrasound images to the estimated 3D USPP. In some embodiments, the pair can be used as a seed to generate additional mappings as training data to machine learn the ASM pose mapping model 240. In some embodiments, the additional mappings can be created in the following manner. With respect to each user-specified probe pose, a series of probe poses can be randomly generated in a predetermined vicinity around each user-specified pose. ExampleFor example, for a probe pose specified by a user, 10,000 probe poses can be randomly generated within a sphere centered on the probe pose specified by the user and having a given diameter. FIG. 3B shows an example, in which for the probe pose 1 specified by the user, a first set of random poses is generated, for the probe pose 2 specified by the user, a second set of random poses is generated, and so on, for the probe pose k specified by the user, a kth set of random poses is generated. For each of the 3D probe poses specified by the user or randomly generated, a 2D slice image can be obtained via the 3D model, and an ASM can be correspondingly generated. In this way, many ASM pose maps can be created, as shown in FIG. 3B. For example, ASMij->US pose ij, where 1, 1 < i, j < k, Mk. This operation produces discrete ASM pose map pairs, which can be used as a discrete mapping model 240 or as training data (e.g., via machine learning) for training an ASM pose mapping model 240, which is capable of extrapolating the training data by learning mappings based on any ASM (including those not included in the training data).

[0045] In different embodiments, additional random probe poses generated based on the probe pose 230 specified by the user can be implemented according to different schemes. FIG. 3C shows a scheme for generating additional random poses within a 3D region defined based on the probe pose specified by the user according to different embodiments of the present teachings. In this scheme, based on multiple probe poses specified by the user (the black dots in FIG. 3C), a 3D region (e.g., a 3D region covering all the probe poses specified by the user) is obtained according to some specified criteria. Then additional random poses are randomly generated within this 3D region (see the smaller dots within the sphere). In some cases, a density criterion can also be specified so that the number of additional probe poses meets the density criterion. Once the additional random probe poses are generated, each additional random probe pose is used to generate a 2D slice image and an ASM to generate a mapping from the ASM to the probe pose. In this way, a training data set with an ASM->probe pose mapping can be generated and used to train the ASM pose mapping model 240.

[0046] In generating the ASM pose map as training data, since the ASM is obtained based on the 2D slice image, and each 2D slice image may only include a partial view of certain anatomical structures, the detected ASM thus simulates what the ultrasound probe will see during the operation. FIG. 3D shows an exemplary ASM based on a blood vessel partially visible in a 2D ultrasound image. In FIG. 3D, 370 represents the 2D image, and 380 represents the partial anatomical structure (e.g., a blood vessel) visible in the 2D image. BecauseA 2D image represents a slice of an organ (either created during the simulation process discussed herein or an actual 2D ultrasound image acquired during surgery), and is therefore partial. With this in mind, the ASM obtained from the 2D image 370 (e.g., a binarized image of 370 where pixels on vascular structures have 1s and everywhere else has 0s) can reveal what is visible on that slice given an ultrasound probe at the 3D USPP. Thus, each 3D probe pose (simulated or actual) generates an ASM generated from a 2D image having a partial view of certain anatomical structures. The mapping can be used to derive an ASM pose mapping model 240, which can be used during surgery to predict the 3D USPP based on a given ASM obtained from a real-time ultrasound image. That is, the predicted 3D USPP is the 3D position of the ultrasound probe that allows the user to see what is visible in the real-time ultrasound image.

[0047] Figure 3E illustrates an ASM pose mapping model 240 generated via machine learning from discrete ASM pose mapping training data according to an embodiment of this teaching. In some embodiments, the ASM pose mapping model 240 may be implemented using an artificial neural network, which may be trained via, for example, deep learning, to predict candidate 3D USPPs based on a given ASM generated from a 2D ultrasound image. As shown, in some embodiments, each of the predicted 3D USPPs may be output with a corresponding probability to indicate, for example, the confidence level of the prediction.

[0048] FIG4A is a flowchart of an exemplary preoperative procedure for generating the ASM pose mapping model 240 according to an embodiment of the present teachings. To allow a user to select a user-specified probe pose, the ASM pose mapping model generator 220 presents a 3D model 250 of the target organ at 400. As discussed herein, the user can manipulate the presentation and determine the selected simulated probe pose. When information about the user-specified probe pose is received at 410, the selected probe pose 230 is archived at 420. In some embodiments, at 430, additional random probe poses may be generated based on the user-specified probe pose, according to some generation schemes disclosed herein. Based on a specified or randomly generated simulated probe pose, at 440, the ASM of the simulated probe pose is obtained from a 2D slice image created based on 3D model 250 according to the simulated probe pose. The mapping from ASM to simulated probe pose is then used at 450 to generate training data, which is then used at 460 to train the ASM pose mapping model 240 via machine learning.

[0049] FIG4B is a flowchart of an exemplary intraoperative procedure according to an embodiment of the present teaching, which is used to visualize 3D anatomical structures captured by an ultrasound probe and determined based on a predicted 3D USPP, the predicted 3DUSPP is based on the ASM pose mapping model. During the operation, at 405, the US probe pose projector 260 retrieves the archived user-specified probe pose 230 and projects it onto the 2D laparoscopic image at 415. The projection may be based on registration between the 3D model 250 and the 2D laparoscopic image. As described herein, such registration is often inaccurate and therefore unreliable (due to various reasons, such as distortion of the laparoscopic image or a limited field of view). However, such a user-specified probe pose projected in the laparoscopic image may help guide the user in determining where to position the ultrasound probe. Accuracy is based on the ASP pose mapping model obtained according to this teaching, achieved by estimating the actual 3D USPP.

[0050] Based on the user-specified probe pose projected in the 2D laparoscopic image, the user can select one of these poses (e.g., by clicking on a projected pose) to place the ultrasound probe at or near the selected user-specified 3D probe pose. Information about the selected 2D projection point can be received from the user and used to deploy the ultrasound probe in the selected 3D pose. With the ultrasound probe deployed in this way, the ultrasound-based 3D anatomical structure visualizer 290 receives real-time US images at 425. The user-selected probe pose projected and appearing in the 2D laparoscopic images can be used as an initial estimate of the actual 3D USPP of the ultrasound probe. To improve the accuracy of the estimate, the user can be instructed to introduce movement into the probe via manipulation, such as rolling the probe clockwise or counterclockwise. Such movement can generate a series of 2D ultrasound images, each of which can reveal information about the changes. Based on such a series of 2D ultrasound images, at 435, the corresponding ASM can be generated by detecting the anatomical structures observed in the ultrasound images and creating a binary representation of the detected anatomical structures. Obtaining the ASM generated based on the 2D ultrasound images, at 445, the candidate 3D USPP can be estimated using the ASM pose mapping model 240. Then, at 455, the optimal 3D USPP estimate can be determined via some optimization schemes explained below. The optimized 3D USPP estimation can then be used to determine the viewing angle for observing the 3D model 240, so that the anatomical structures on the slice (determined based on the viewing angle and the operating parameters of the ultrasound probe) can be visualized to the user at 465, providing improved visual guidance on what corresponds to information in the ultrasound image.

[0051] Details of the ASM pose mapping model generator 220 and the ultrasound-based 3D anatomical structure visualizer 290 are disclosed below. Figure 5A depicts an exemplary high-level system of the ASM pose mapping model generator 220 according to an embodiment of this teaching.Figure. As discussed herein, the ASM pose mapping model 240 is used to map the ASM obtained from a 2D ultrasound image (simulated or actual) to a 3D USPP. The ASM pose mapping model 240 may be a discrete model as a lookup table (as shown in Figure 3B), or it may be obtained via machine learning based on training data created from discrete mapping pairs shown in Figure 3B. In some exemplary embodiments, the ASM pose mapping model generator 220 includes a user interface 500, a random US pose generator 510, a pose-based 2D ASM determiner 520, an ASM pose mapping training data generator 540, and a machine learning engine 530.

[0052] Figure 5B is a flowchart of an exemplary process of the ASM pose mapping model generator 220 according to an embodiment of the present teachings. In operation, at 505, the user interface 500 presents a 3D model 240 of the target organ to the user, the 3D model 240 possibly corresponding to a manipulable 3D representation of the modeled target organ. As discussed herein, a presentation 230 is provided to facilitate user specification of a user-specified probe pose. For example, when a user determines where to define the user-specified probe pose, the user can rotate and translate the presented 3D organ model. During communication with the user, when the user interface 500 receives information about the user-specified 3D probe pose defined on the presented 3D model 250 at 515, the user interface 500 archives the user-specified probe pose at 230. In some embodiments, based on the user-specified probe pose, at 525, a pose-based 2D ASM determiner 520 can create an additional random 3D probe pose for that user-specified 3D probe pose. In other embodiments, additional 3D probe poses can be generated after all user-specified probe poses have been specified. Regarding the user-specified and additionally generated probe poses, corresponding 2D slice images can be obtained based on 2D slices (determined based on the viewpoint associated with each user-specified probe pose from the 3D model 240), and an ASM for such 2D slice images is generated at 535. Each ASM and its corresponding (user-specified or randomly generated) probe pose form a pair or mapping, i.e., ASM --> pose.

[0053] The process of generating discrete mappings (pairs) for each user-specified probe pose continues until the user specifies all probe poses. Using N discrete mappings created in this way (e.g., N = k x M, where k is the number of user-specified probe poses and M is the number of random probe poses generated for each user-specified probe pose), the ASM pose mapping training data generator 540 creates training data 550 at 555, which is then used by the machine learning engine 530. (See specification 7 / 12 pages 13 CN 121079021 A)The ASM pose mapping model is trained at position 565, and the trained ASM pose mapping model 240 is derived at position 575. As discussed herein, the trained ASM pose mapping model 240 learns the relationship between 2D ASMs of the 3D anatomy of the 3D USPP through machine learning. Creating additional probe poses for each user-specified probe pose can be presented herein as an exemplary manner to create more training data. This is for illustration only and not as a limitation. Other schemes for generating random probe poses based on user-specified probe poses can also be used. For example, as shown in FIG3C, random probe poses can also be generated within a 3D region (e.g., a sphere) defined for all user-specified probe poses according to some preset criteria. As another example, a specific density can be specified as a criterion to ensure that a sufficient number of additional probe poses are provided. Any other generation schemes can also be implemented, and they are all within the scope of this teaching.

[0054] FIG6A depicts an exemplary high-level system diagram of a pose-based 2D ASM determiner 530 according to an embodiment of this teaching. As discussed herein, a pose-based 2D ASM determiner 530 is provided to transform each given probe pose (user-specified or randomly generated) into an ASM; that is, taking a 3D probe pose as input and producing an ASM as the output of an ASM-->3D USPP mapping. In the illustrated embodiment, the pose-based ASM determiner 530 includes a pose-based slice generator 600, a 2D anatomical structure detector 620, and a binary ASM generator 640. The pose-based slice generator 600 is provided to simulate the process of an ultrasound probe acquiring ultrasound images in a given pose. Given an input probe pose, the pose-based slice generator 600 determines a plane in the 3D model 240 that is sliced ​​according to the input probe pose and some operating parameters of the ultrasound probe (e.g., the distance at which it can emit sound waves). In some embodiments, the input probe pose can be used to determine a viewpoint about the 3D model, and the slice is perpendicular to that viewpoint and at a distance determined based on the operating parameters of the ultrasound probe. A slice-based, pose-based slice generator 600 simulates an ultrasound sensing process to acquire 2D ultrasound images containing anatomical structures, as if an actual ultrasound probe were sensed from a 3D model. Such 2D simulated ultrasound images created based on a given probe pose are stored in 610. A 2D anatomical structure detector 620 is provided for detecting anatomical structures in each of the 2D slice images. Such anatomical structures may include tumors(multiple) and blood vessels. Detection may be performed based on an associated AS detection model 630. Based on the detected anatomical structures, a binary ASM generator 640 creates a corresponding ASM as output.

[0055] Figure 6B is a flow chart of an exemplary process of a pose-based 2D ASM determiner 530 according to an embodiment of the present teachings.Figure. In operation, when the pose-based slice generator 600 receives the input probe pose (user-specified and additionally generated) at 650 and obtains a 3D slice of each of the input poses at 655, a simulated 2D ultrasound image is created at 660 by determining what is visible on the 3D slices. For each of the simulated 2D ultrasound images thus created, the 2D anatomical structure detector 620 detects the anatomical structures appearing in the simulated 2D image at 665. The anatomical structures thus detected in the 2D image are used by the binary ASM generator 640 to generate an ASM at 670, and at 675, this ASM is sent as a pair with the input probe pose as output. The process of generating simulated 2D ultrasound images, detecting anatomical structures from them, and creating ASMs continues until at 680 all ASM pose mappings are determined for the input probe poses (both user-specified and additionally generated).

[0056] As discussed herein, the preoperative portion of the 3D USPP-based AS visualization system 200 is used to create a user-specified probe pose 230 and a trained ASM pose mapping model 240, which are used during the surgical phase to predict the accurate 3D USPP of the ultrasound probe. The accurately predicted 3D USPP enables improved visualization of anatomical structures captured in real-time ultrasound images. In some embodiments, the 3D USPP may be determined based on multiple candidate 3D USPPs predicted using the ASM pose mapping model 240. As disclosed herein, during surgery, the US probe pose projector 260 first retrieves the user-specified probe pose and projects it onto the laparoscopic image to allow the user to select an initial pose for placing the ultrasound probe. Figure 7A illustrates an exemplary projection of the user-specified probe pose 230. As shown, 701 may correspond to an organ captured by a laparoscopic camera. The previously user-specified probe pose 230 can then be projected onto it, such as 230-1, 230-2, ..., 230-5. These projected probe poses in the 2D image can provide alternative poses, which the user can select to place the ultrasound probe during surgery. For example, the user may prefer to place the ultrasound probe in a pose that shows the tumor and associated vascular structures beneath the surface of, for example, a portion of organ 701. As the user guides the ultrasound probe to be deployed in one of the projection positions, the ultrasound probe can acquire real-time ultrasound images and send them to an ultrasound-based 3D anatomical visualization device 290. The ultrasound-based 3D anatomical visualization device 290 can process the real-time ultrasound images and estimate the actual 3D USPP of the ultrasound probe, so that the 3D anatomical structure corresponding to what is seen in the 2D ultrasound image can be visualized to the user with enhanced quality.

[0057] FIG7B depicts an exemplary high-level system diagram of an ultrasound-based 3D anatomical structure visualizer 290 according to an embodiment of the present teachings. Inputs to the ultrasound-based 3D anatomical structure visualizer 290 include ultrasound images 380 acquired by an ultrasound probe and the position and motion of the ultrasound probe. Its outputs include 3D anatomical structure visualization results and, optionally, instructions provided to the user regarding the induction of specific motions of the ultrasound probe. In the illustrated embodiment, the ultrasound-based 3D anatomical structure visualizer 290 includes a 2D ASM generator 700, an ASM-based 3D pose candidate selector 720, an optimal 3D US pose estimator 740, a 3D pose-based anatomical structure retrieval unit 750, and a 3D anatomical structure presenter 760.

[0058] The 2D ASM generator 700 is provided for generating a binarized image based on anatomical structures(s) detected from real-time ultrasound images based on, for example, an AS detection model 710. The generated ASM is sent as input to the ASM pose mapping model 240, which predicts multiple candidate 3D USPPs with corresponding probabilities as confidence scores. The output of the ASM pose mapping model 240 is provided to the ASM-based 3D pose candidate selector 720 to select a set of appropriate, for example, promising, candidate ASM pose maps 730. When the ultrasound probe performs certain movements, it continuously generates a series of 2D ultrasound images. When such images are processed, they generate a series of ASMs, each of which is used by the ASM pose mapping model 240 to generate a set of candidate ASM pose maps, and in 730 (by the ASM-based 3D pose candidate selector 720) some candidate ASM pose maps are selected for further processing.

[0059] These multiple sets of candidate USPPs stored in 730 are used by the optimal 3D US pose estimator 740 to select the optimal 3D USPP estimate. Each group of candidate poses is estimated relative to a probe pose (with a probability of change), and multiple groups of candidate poses correspond to multiple probe poses due to, for example, slight movements of the probe (e.g., probe rotation). Different optimization schemes can be employed to determine the optimal USPP estimate based on multiple groups of candidate poses. In some embodiments, the trajectory generated by the probe due to its motion can be used to determine the optimal USPP estimate. For example, if the probe moves clockwise, the candidate poses selected from the associated groups of candidate poses can be chosen in a manner that maximizes the total confidence score of the selected candidate poses while also forming a clockwise trajectory. In this case, the trajectories formed by the candidate poses selected from different groups are consistent with the actual motion trajectory of the probe. The optimal USPP estimate can then be determined based on the candidate poses selected in this way. For example, one of the selected candidate poses along a consistent trajectory can be selected as the optimal one.Pose. In another example, the selected candidate pose corresponding to the final probe position can be used as the best estimate. Other means of determining the best estimated pose are also possible. For example, the median or centroid of all candidate poses along a consistent trajectory can be used as the best estimated pose.

[0060] Using the best 3D USPP estimate from the best 3D US probe estimator 740, the 3D pose-based anatomical structure retrieval device 750 can identify 3D anatomical structures from the 3D model 250 according to the viewpoint defined by the best 3D USPP estimate. Since the 3D USPP is obtained in an optimal manner, the correspondingly determined anatomical structure can also be closest to what is observed in the real-time ultrasound image. Accordingly, the 3D anatomical structure presenter 760 can visualize the determined 3D anatomical structure to provide the user with more effective visual guidance, i.e., allowing the user to see the 3D anatomical structure corresponding to the partially visible anatomical structure in the real-time ultrasound image. This visualization not only helps users see what lies beneath the surface of organs, but also provides a complete presentation of the corresponding anatomical structures registered with partial information revealed in the real-time ultrasound images (pages 9 / 12, CN 121079021 A). Therefore, this visualization provides users with effective visual guidance on what is where and, accordingly, determines what to do next during surgery.

[0061] Figure 7C is a flowchart of an exemplary process of an ultrasound-based 3D anatomical structure visualizer 290 according to an embodiment of this teaching. Upon receiving a real-time ultrasound image acquired by an ultrasound probe, a 2D ASM generator 700 detects visible anatomical structures in the ultrasound image and generates an ASM of the input ultrasound image at 705. Each of the ASMs is provided to an ASM pose mapping model 240, which predicts or estimates multiple candidate 3D USPPs at 715, where the corresponding probability indicates the confidence level of the estimate. The probabilistic candidate 3D USPPs can be output to an ASM-based 3D pose candidate selector 720, which can select some candidate 3D USPPs at 725 based on, for example, the probabilities of some candidate 3D USPPs. For example, the ASM-based 3D pose candidate selector 720 can select only candidate USPP estimates with probabilities higher than a predetermined threshold (e.g., >0.5). The selected candidate USPPs for different ASMs are stored in 730. The processing from steps 705 to 725 can be continuous as long as the ultrasound pose is still moving or until the user signals to stop. This is determined at 735.

[0062] The selected candidate USPP estimates stored in 730 can then be used for optimization to determine the optimal 3D USPP. As discussed herein, multiple groups of candidate USPP estimates may be stored in 730 and can be optimized by utilizing probe-related parameters.The optimal 3D US pose estimator 740 selects the best one based on the probe motion information. In some embodiments, the optimal 3D US pose estimator 740 receives probe motion information at 745 and selects the best USPP estimate based on the probe motion information at 755. As described above, this can be performed in two stages. In the first stage, candidate USPP estimates can be selected from each group of candidates corresponding to each probe pose, such that the trajectory formed by these selected candidate USPP estimates forms a motion trajectory consistent with the probe's motion trajectory, thereby maximizing the overall probability along that trajectory. The second stage determines the best USPP estimate based on the selected candidate USPP estimates. In some embodiments, one of the candidate USPP estimates can be selected as the best USPP estimate, for example, the estimated USPP corresponding to the last ASM. In other embodiments, the best USPP estimate can be derived based on all selected candidate USPP estimates, for example, by combining them in some way to produce the best estimated pose; for example, the centroids of all selected candidate USPP estimates can be calculated as the best estimated pose.

[0063] Once the optimal USPP estimate is derived, the 3D pose-based anatomical structure retrieval unit 750 obtains a 3D sub-model from the 3D model 250 at 765 based on the viewpoint determined based on the optimal 3D USPP estimate. The 3D sub-model corresponds to a portion of the 3D model 250 and includes relevant 3D anatomical structures as seen (e.g., partially) in at least some of the real-time ultrasound images. The relevant 3D anatomical structures thus determined can then be provided to the 3D anatomical structure presenter 760 so that the relevant 3D anatomical structures can be visualized to the user at 775. The visualization provides a view of the 3D anatomical structures by registering the 3D anatomical structures with what is displayed in the real-time ultrasound images and allowing the user to see the contents inside the target organ, thereby helping the surgeon understand the local internal environment in order to make decisions about what to do and how to do the surgical procedure.

[0064] Figure 7D provides an exemplary flow of intraoperative procedures during LUS surgery to estimate 3D USPP based on 2D ultrasound images according to an embodiment of this teaching. This exemplary procedure begins in the upper left corner, with step 702 projecting a user-specified probe pose (230) onto a 2D laparoscopic image (at the projected 2D position). Based on the projected 2D probe position, the user (e.g., a surgeon) can select one of the projected 2D positions to place the ultrasound probe at or near the selected probe position (704). The user can also move the probe around according to instructions such as “rotate the probe clockwise” (706). The placement of the ultrasound probe and subsequent movement of the probe generate a series of real-time ultrasound images from this series of real-time ultrasound...A corresponding ASM (708) is generated in the image. Each of the ASMs can then be sent to the SM pose mapping model 240, and multiple candidate 3D USPPs are estimated with corresponding probabilities. Optionally, the output from the ASM pose mapping model 240 can be filtered to retain those candidate 3D USPPs that meet a certain predetermined criterion (712). In order to determine an optimized 3D USPP estimate (712) based on the candidate 3D USPPs, motion information associated with the ultrasound probe is used to identify from 712 those estimated 3D USPPs that form a trajectory with the highest probability that is consistent with the motion of the ultrasound probe (714) (716). That is, the candidate 3D USPP estimates form a trajectory consistent with the motion trajectory of the ultrasound probe. Based on the candidate 3D USPP estimates in 716, an optimal 3D USPP estimate is generated (718) by, for example, selecting or combining to create the optimal estimated pose.

[0065] FIG8 is a schematic diagram of an exemplary mobile device architecture that can be used to implement a dedicated system that implements the present teachings according to various embodiments. In this example, a user device on which the present teachings can be implemented corresponds to mobile device 800, including but not limited to smartphones, tablets, music players, handheld game consoles, global positioning system (GPS) receivers, and wearable computing devices, or any other factor in any other form. Mobile device 800 may include one or more central processing units (“CPU”) 840, one or more graphics processing units (“GPU”) 830, a display 820, memory 860, a communication platform 810 (such as a wireless communication module), a storage device 890, and one or more input / output (I / O) devices 850. Any other suitable components, including but not limited to a system bus or controller (not shown), may also be included in mobile device 800. As shown in FIG8, a mobile operating system 870 (e.g., iOS, Android, Windows Phone, etc.) and one or more applications 880 may be loaded from storage device 890 into memory 860 for execution by CPU 840. Application 880 may include, at least in part, a user interface for information analysis and management, or any other suitable mobile application, in accordance with this teaching, on mobile device 800. User interaction (if any) may be implemented via I / O device 850 and provided to various components connected via one or more networks.

[0066] To implement the various modules, units, and functions described herein, a computer hardware platform may be used as one or more hardware platforms for one or more elements described herein. The hardware elements, operating system, and programming of such computers...The programming languages ​​are conventional in nature, and it is assumed that those skilled in the art are fully familiar with these technologies to adapt them to the appropriate settings described herein. A computer with user interface elements can be used to implement a personal computer (PC) or other type of workstation or terminal device, but if properly programmed, the computer can also act as a server. Those skilled in the art are believed to be familiar with the structure, programming, and general operation of such computer equipment, and therefore the accompanying drawings should be self-evident.

[0067] FIG9 is a schematic diagram of an exemplary computing device architecture that can be used to implement a dedicated system according to various embodiments of the present teachings. Such a dedicated system in conjunction with the present teachings has a functional block diagram of a hardware platform including user interface elements. The computer can be a general-purpose computer or a dedicated computer. Both can be used to implement a dedicated system for the present teachings. The computer 800 can be used to implement any component or aspect of the framework disclosed herein. For example, the information analysis and management methods and systems disclosed herein can be implemented on a computer such as computer 900 via the computer's hardware, software programs, firmware, or a combination thereof. Although only one such computer is shown for convenience, the computer functions described herein related to the present teachings can be implemented in a distributed manner on several similar platforms to distribute the processing load.

[0068] Computer 900 includes, for example, a COM port 950, which is connected to and from a network connected to the COM port 950 to facilitate data communication. Computer 900 also includes a central processing unit (CPU) 920 in the form of one or more processors for executing program instructions. Exemplary computer platforms include an internal communication bus 910, various forms of program storage and data storage (e.g., disk 970, read-only memory (ROM) 930, or random access memory (RAM) 940) for various data files to be processed and / or transferred by computer 900 and possible program instructions to be executed by CPU 920. Computer 900 also includes an I / O component 960 that supports input / output flow between the computer and other components in the computer, such as user interface element 980. Computer 900 may also receive programming and data via network communication.

[0069] Thus, as described above, aspects of information analysis and management methods and / or other processes can be embodied in programming. The technical program aspect can be considered as a "product" or "article of art" generally in the form of executable code and / or associated data that is executed on or implemented on a machine-readable medium. Tangible, non-transient "storage" type media includes memory that can be provided for software programming at any time or for use in computers, processors, or the like.Any or all of other storage or its associated modules (such as various semiconductor memories, tape drives, disk drives, etc.).

[0070] All or part of the software can sometimes be transmitted via a network (such as the Internet or various other telecommunications networks). Such communication, for example, can enable software to be loaded from one computer or processor to another computer or processor, for example, in connection with information analysis and management. Thus, another type of medium that can carry software elements includes light waves, radio waves, and electromagnetic waves, such as physical interfaces between local devices, via wired and optical ground networks, and via various air links. Physical elements carrying such waves (such as wired or wireless links, optical links, or the like) can also be considered as media carrying software. As used herein, unless limited to tangible “storage” media, terms such as “computer or machine-readable medium” refer to any medium that participates in providing instructions to a processor for execution.

[0071] Thus, machine-readable media can take many forms, including but not limited to tangible storage media, carrier media, or physical transmission media. Non-volatile storage media include, for example, optical discs or disks that can be used to implement a system as shown in the accompanying drawings or any of the components of that system, such as any storage device or the like in any computer(s). Volatile storage media include dynamic memory, such as the main memory of a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, which include lines forming a bus within a computer system. Carrier transmission media may take the form of electrical or electromagnetic signals, or sound or light waves (such as those generated during radio frequency (RF) and infrared (IR) data communications). Therefore, common forms of computer-readable media include, for example: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carriers for transmitting data or instructions, cables or links for transmitting such carriers, or any other media from which a computer can read programming code and / or data. Many forms of computer-readable media of these forms may be involved when carrying one or more sequences of one or more instructions to a physical processor for execution.

[0072] Those skilled in the art will recognize that the teachings herein are adaptable to various modifications and / or enhancements. For example, while implementations of the various components described above may be embodied in a hardware device, they may also be implemented as a software-only solution, for example, installed on an existing server. Additionally, the techniques disclosed herein may be implemented as firmware, a firmware / software combination, a firmware / hardware combination, or a hardware / firmware / software combination.

[0073] Although the foregoing descriptions have been intended to constitute this teaching and / or other examples, it should be understood that various modifications may be made thereto, and the subject matter disclosed herein may be implemented in various forms and examples, and the teachings may be applied to many applications, only some of which have been described herein. The appended claims are intended to claim protection for any and all applications, modifications, and variations that fall within the true scope of this teaching. Instruction Manual 12 / 12 Page 18 CN 121079021 A Figure 1A Figure 1B Instruction Manual Drawings 1 / 20 Page 19 CN 121079021 A Figure 1C Instruction Manual Drawings 2 / 20 Page 20 CN 121079021 A Figure 1D Figure 1E Instruction Manual Drawings 3 / 20 Page 21 CN 121079021 A Figure 2 Instruction Manual Drawings 4 / 20 Page 22 CN 121079021 A Figure 3A Instruction Manual Drawings 5 / 20 Page 23 CN 121079021 A Figure 3B Instruction Manual Drawings 6 / 20 Page 24 CN 121079021 A Figure 3C Instruction Manual Drawings 7 / 20 Page 25 CN 121079021 A Figure 3D Instruction Manual Drawings 8 / 20 Page 26 CN 121079021 A Figure 3E Instruction Manual Drawings 9 / 20 Page 27 CN 121079021 A Figure 4A Figure 4B Instruction Manual Drawings 10 / 20 Page 28 CN 121079021 A Figure 5A Instruction Manual Drawings 11 / 20 Page 29 CN 121079021 A Figure 5B Instruction Manual Drawings 12 / 20 Page 30 CN 121079021 A Figure 6A Instruction Manual Drawings 13 / 20 Page 31 CN 121079021 A Figure 6B Instruction Manual Drawings 14 / 20 Page 32 CN 121079021 A Figure 7A Instruction Manual Drawings 15 / 20 Page 33 CN 121079021 A Figure 7B Instruction Manual Drawings 16 / 20 Page 34 CN 121079021 A Figure 7C Instruction Manual Drawings 17 / 20 Page 35 CN 121079021 A Figure 7D Instruction Manual Drawings 18 / 20 Page 36 CN 121079021 A Figure 8 Instruction Manual Drawings, Page 19 / 20, 37 CN 121079021 A Figure 9 Instruction Manual Drawings, Page 20 / 20, 38 CN 121079021 A

Claims

1. A method implemented on at least one processor, memory, and communication platform, the method comprising: projecting one or more three-dimensional (3D) probe poses onto corresponding two- dimensional (2D) probe locations on 2D images acquired by a laparoscopic camera directed at a target organ inserted into a patient during a surgery; receiving 2D ultrasound images acquired by an ultrasound probe placed at or near one of the 2D probe locations; for each of at least some of the 2D ultrasound images, detecting anatomical structures from the 2D ultrasound images, generating anatomical structure masks (ASMs) for the 2D ultrasound images based on the detected anatomical structures, predicting one or more 3D probe pose estimates for the ultrasound probe based on the ASMs via an ASM pose mapping model; and generating a best 3D probe pose estimate based on the 3D probe pose estimates, the 3D probe pose estimate being predicted from ASMs obtained from the 2D ultrasound images acquired by the ultrasound probe at varied 3D probe poses.

2. The method of claim 1, wherein, the one or more 3D probe poses are specified by a user for a 3D model of the target organ presented to the user; and used to create the ASM pose mapping model based on ASMs obtained with respect to the one or more 3D probe poses.

3. The method of claim 1, wherein, each of the one or more 3D probe pose estimates is predicted with a metric indicative of a confidence level of the 3D probe pose estimate.

4. The method of claim 3, wherein, the varied 3D probe poses are due to motion of the ultrasound probe about a selected 2D probe location, wherein the motion includes a change in 3D coordinates or 3D orientation of the ultrasound probe.

5. The method of claim 4, wherein, the step of generating a best 3D probe pose estimate includes: determining a trajectory of the ultrasound probe from the varied 3D probe poses; identifying a 3D probe pose estimate predicted with respect to each of the varied 3D probe poses along the trajectory; determining the best 3D probe pose estimate based on the 3D probe pose estimates predicted with respect to the varied 3D probe poses along the trajectory.

6. The method of claim 5, wherein, the step of determining the best 3D probe pose estimate includes: selecting one of the 3D probe pose estimates with respect to each of the varied probe poses along the trajectory such that the metric of the selected predicted 3D probe pose is maximized; and determining the best 3D probe pose estimate based on the selected predicted 3D probe pose along the trajectory.

7. The method of claim 1, wherein, the ASM pose mapping model is obtained by: presenting a three-dimensional (3D) model configured to model the target organ to a user prior to the surgery; receiving instructions from the user specifying the plurality of 3D probe poses; obtaining simulated 3D probe poses including the plurality of 3D probe poses specified by the user and additional 3D probe poses automatically generated with respect to the plurality of 3D probe poses; and creating the ASM pose mapping model based on the simulated 3D probe poses.

8. The method of claim 7, wherein, the additional 3D probe poses are randomly generated within a 3D region defined based on one or more of the plurality of 3D probe poses.

9. The method of claim 7, wherein, the step of creating the ASM pose mapping model comprises: with respect to each of the simulated 3D probe poses: determining a view angle based on the simulated 3D probe pose with respect to a rendered 3D model, obtaining simulated two-dimensional (2D) ultrasound images acquired from the view angle by a simulated ultrasound probe placed at the simulated 3D probe pose, deriving an anatomical structure mask (ASM) based on anatomical structures in the simulated 2D ultrasound images, and creating a simulated mapping pair involving the ASM and the simulated 3D probe pose; and obtaining the ASM pose mapping model based on the simulated mapping pairs.

10. The method of claim 9, wherein, the step of obtaining the ASM pose mapping model comprises: generating training data for machine learning based on the simulated mapping pairs; training the ASM pose mapping model using the training data.

11. A machine-readable and non-transitory medium having information recorded thereon, wherein, the information, when read by the machine, causes the machine to carry out the steps of: projecting one or more three-dimensional (3D) probe poses to corresponding 2D probe locations on a two-dimensional (2D) image during a surgery, wherein the 2D image is acquired by a laparoscopic camera inserted into a patient's body during the surgery and pointing at a target organ; receiving 2D ultrasound images acquired by an ultrasound probe placed at or near one of the 2D probe locations; with respect to each of at least some of the 2D ultrasound images, detecting anatomical structures from the 2D ultrasound images, generating anatomical structure masks (ASMs) for the 2D ultrasound images based on the detected anatomical structures, predicting one or more 3D probe pose estimates for the ultrasound probe based on the ASMs via an ASM pose mapping model; and generating an optimal 3D probe pose estimate based on the 3D probe pose estimates, the 3D probe pose estimate being based on ASM predictions obtained from the 2D ultrasound images acquired by the ultrasound probe at varied 3D probe poses.

12. The medium of claim 11, wherein, the one or more 3D probe poses are specified by a user with respect to a 3D model of the target organ rendered to the user; and used to create the ASM pose mapping model based on ASMs obtained with respect to the one or more 3D probe poses.

13. The medium of claim 11, wherein, each of the one or more 3D probe pose estimates is predicted with a measure indicative of a confidence level of the 3D probe pose estimate.

14. The medium of claim 13, wherein, the varied 3D probe poses are due to a motion of the ultrasound probe around a selected 2D probe location, wherein the motion includes a change in 3D coordinates or 3D orientation of the ultrasound probe.

15. The medium of claim 14, wherein, the step of generating an optimal 3D probe pose estimate comprises: determining a trajectory of the ultrasound probe from the varied 3D probe poses; identifying a 3D probe pose estimate predicted with respect to each of the varied 3D probe poses along the trajectory; determining the optimal 3D probe pose estimate based on the 3D probe pose estimates predicted with respect to the varying 3D probe poses along the trajectory.

16. The medium of claim 15, wherein, The step of determining the optimal 3D probe pose estimate comprises: selecting one of the 3D probe pose estimates with respect to each of the varying probe poses along the trajectory such that the metric of the selected predicted 3D probe pose is maximized; and determining the optimal 3D probe pose estimate based on the selected predicted 3D probe pose along the trajectory.

17. The medium of claim 11, wherein, The ASM pose mapping model is obtained by: presenting a three-dimensional (3D) model configured to model the target organ to a user prior to the surgery; receiving instructions from the user specifying the plurality of 3D probe poses; obtaining simulated 3D probe poses comprising the plurality of 3D probe poses specified by the user and additional 3D probe poses automatically generated with respect to the plurality of 3D probe poses; and creating the ASM pose mapping model based on the simulated 3D probe poses.

18. The medium of claim 17, wherein, The additional 3D probe poses are randomly generated within a 3D region defined based on one or more of the plurality of 3D probe poses.

19. The medium of claim 17, wherein, The step of creating the ASM pose mapping model comprises: with respect to each of the simulated 3D probe poses: determining a view angle based on the simulated 3D probe pose with respect to the presented 3D model, obtaining a simulated two-dimensional (2D) ultrasound image acquired from the view angle by a simulated ultrasound probe placed at the simulated 3D probe pose, deriving an anatomical structure mask (ASM) based on anatomical structures in the simulated 2D ultrasound image, and creating a simulated mapping pair involving the ASM and the simulated 3D probe pose; and obtaining the ASM pose mapping model based on the simulated mapping pairs.

20. The medium of claim 19, wherein, The step of obtaining the ASM pose mapping model comprises: generating training data for machine learning based on the simulated mapping pairs; training the ASM pose mapping model using the training data.

21. A system, the system comprising: an ultrasound probe pose projector implemented by a processor and configured to project one or more three-dimensional (3D) probe poses onto a two-dimensional (2D) image at corresponding 2D probe locations during a surgery, wherein the 2D image is acquired by a laparoscopic camera pointing at a target organ inserted into a patient’s body in the surgery; an ultrasound-based 3D anatomical structure visualizer implemented by a processor and configured to receive 2D ultrasound images acquired by an ultrasound probe placed at or near one of the 2D probe locations, with respect to each of at least some of the 2D ultrasound images, detect anatomical structures from the 2D ultrasound images, generate anatomical structure masks (ASMs) for the 2D ultrasound images based on the detected anatomical structures, one or more 3D probe poses of the ultrasound probe based on the ASM, and generating an optimal 3D probe pose estimate based on the 3D probe pose estimates, the 3D probe pose estimate being based on ASM predictions obtained from the 2D ultrasound images acquired by the ultrasound probe at varied 3D probe poses.

22. The system of claim 21, wherein, the one or more 3D probe poses are specified by a user for a 3D model of the target organ presented to the user; and creating the ASM pose mapping model based on ASM, the ASM being obtained with respect to the one or more 3D probe poses.

23. The system of claim 21, wherein, each of the one or more 3D probe pose estimates is predicted with a measure indicative of a confidence level of the 3D probe pose estimate.

24. The system of claim 23, wherein, the varied 3D probe poses are due to motion of the ultrasound probe about a selected 2D probe location, wherein the motion includes a change in 3D coordinates or 3D orientation of the ultrasound probe.

25. The system of claim 24, wherein, the step of generating an optimal 3D probe pose estimate includes: determining a trajectory of the ultrasound probe from the varied 3D probe poses; identifying a 3D probe pose estimate predicted with respect to each of the varied 3D probe poses along the trajectory; determining the optimal 3D probe pose estimate based on the 3D probe pose estimates predicted with respect to the varied 3D probe poses along the trajectory.

26. The system of claim 25, wherein, the step of determining the optimal 3D probe pose estimate includes: selecting one of the 3D probe pose estimates with respect to each of the varied probe poses along the trajectory such that the measure of the selected predicted 3D probe pose is maximized; and determining the optimal 3D probe pose estimate based on the selected predicted 3D probe pose along the trajectory.

27. The system of claim 21, further comprising an ASM pose mapping model generator implemented by a processor and configured to obtain the ASM pose mapping model by: presenting a three-dimensional (3D) model configured to model the target organ to a user prior to the procedure; receiving instructions from the user specifying the plurality of 3D probe poses; obtaining simulated 3D probe poses including the plurality of 3D probe poses specified by the user and additional 3D probe poses automatically generated with respect to the plurality of 3D probe poses; and creating the ASM pose mapping model based on the simulated 3D probe poses.

28. The system of claim 27, wherein, the additional 3D probe poses are randomly generated within a 3D region defined based on one or more of the plurality of 3D probe poses.

29. The system of claim 27, wherein, the step of creating the ASM pose mapping model includes: with respect to each of the simulated 3D probe poses: determining a perspective based on the simulated 3D probe pose with respect to the presented 3D model, obtaining a simulated two-dimensional (2D) ultrasound image acquired from the perspective by a simulated ultrasound probe placed at the simulated 3D probe pose, deriving an anatomical structure mask (ASM) based on anatomical structures in the simulated 2D ultrasound images, and creating simulated mapping pairs involving the ASM and the simulated 3D probe poses; and obtaining the ASM pose mapping model based on the simulated mapping pairs.

30. The system of claim 29, wherein, The step of obtaining the ASM pose mapping model comprises: generating training data for machine learning based on the simulated mapping pairs; training the ASM pose mapping model using the training data. The step of obtaining the ASM pose mapping model comprises: generating training data for machine learning based on the simulated mapping pairs; training the ASM pose mapping model using the training data.

Images