Registration device and registration method for registering a model of a body part and a real world image of the body part

Abstract

To provide an enhanced apparatus and method for registering a model of a body part with a real world image.SOLUTION: The device 20 comprises an input-interface 21 for receiving 3D data-of the model, 2D data-of the real-world image, a display unit 24 for visualizing the model and the body part on the 2D display, and a user-input device 22 for manipulating a mouse pointer on the 2D display based on the received user-input data-of the model. The device 20 is further enhanced by a registering unit 30 for receiving user input data indicative of a register between a reference point in the real world image and in the 3D model by application of a mouse pointer, an augmenting unit 36 for augmenting the 3D datum of the reference point in the real world image to a synthetic 2D datum by adding depth information derived from the 2D datum of the model, the 3D datum of the real world image, and a transformation unit 38 for determining a registration matrix of the 3D datum of the model, the synthetic 3D datum of the real world image using the 3D coordinates of the reference point.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention provides a method for registering a model of a body part and a real-world image of the body part. Registration About the device. Registration The apparatus comprises an input interface configured to receive 3D data of a model of a body part and 2D data of a real-world image of the body part, a display unit configured to visualize the model and the body part on a 2D display of the display unit, and a user input device configured to receive user input data and communicate the user input data to the display unit, the display unit configured to manipulate a mouse pointer on the 2D display based on the user input data. Further, the present invention provides a method for registering the model of the body part and the real-world image of the body part. Registration Regarding the method, Registration The method includes receiving 3D data of a model of a body part and visualizing the model on a 2D display, receiving 2D data of a real-world image of the body part and visualizing the body part on the 2D display, and receiving user input data from a user input device that operates as a mouse pointer on the 2D display. [Background technology]

[0002] Registering 3D models of body parts, such as internal organs, to real-world images can assist surgeons during surgical procedures. Intraoperative navigation, which generally involves complex cognitive processes, is required. This is particularly applicable to minimally invasive surgeries, such as laparoscopic surgery. Intraoperative navigation requires a combination of knowledge of the general anatomical structure of the target organ or body part, the patient-specific anatomical structure of the target organ (which may differ in the case of abnormal anatomical structures), and specific knowledge of the spatial relationships between anatomical structures within the patient's target organ. While knowledge of the first two points can generally be acquired during the preoperative phase (e.g., by preoperative imaging methods such as computed tomography (CT) and / or magnetic resonance imaging (MRI)), knowledge of the third point can only be acquired during the laparoscopic surgical intervention itself. Therefore, careful dissection of the target organ's tissue and avoiding unexpected damage to anatomical structures throughout the laparoscopic procedure are of utmost importance.

[0003] FIG. 1 shows a classic minimally invasive surgical system 2 known from the prior art. The system 2 includes a navigation system 4 for assisting the surgeon with spatial anatomical information throughout the procedure. The navigation system 4 uses a hardware setup comprising an optical tracking camera 6, a tracking marker 8 (e.g., a reflective ball) that must be attached to a surgical instrument 10 (partially shown) to be tracked, and a laparoscopic camera 11, which may be a separate device as shown in FIG. 1 or may be included in the surgical instrument 10. Additionally, there is a first monitor 12a showing images from a segmented 2D CT / MRI scan, a second monitor 12b displaying a reconstructed 3D model derived from the organ under treatment, and a third monitor 12c, currently turned off and configured to show laparoscopic camera images. The reconstructed 3D model is typically acquired during preoperative imaging. This device is not part of the hardware setup.

[0004] In a conventional navigation system 4 , the tracking camera 6 acquires the position of the tracking marker 8 and calculates the position and orientation of the surgical instrument 10 in the coordinate system of the navigation system 4 .

[0005] The navigation system 4 is intended to provide the surgeon with additional information about spatial anatomy. This can be done by displaying a reconstructed 3D model derived from preoperative imaging modalities on a second monitor 12b and intraoperative images provided by a laparoscopic camera on a third monitor 12c. To simplify the interpretation of the information provided, both the intraoperative images and the 3D model need to be registered with each other. Technically, this step is called registration.

[0006] A robust and reliable solution for automatically performing registration remains necessary. To achieve the desired results, it is necessary to incorporate human knowledge into the registration process. To this end, the user is required to select at least three points, and better still, four points, on the intraoperative target organ. In the system shown in FIG. 1, this can be done with the tip of the surgical instrument 10, which is tracked by a tracking camera 6. The tracking camera 6 determines the position and spatial orientation of the surgical instrument 10 in the coordinate system of the navigation system 4. After calibration with the surgical instrument 10, the system knows where the tip of the surgical instrument 10 is located in space. This allows the spatial points to be acquired.

[0007] The user is then prompted to select three or four corresponding points on the preoperative 3D model displayed on the second monitor 12b. The corresponding points can be used to estimate a transformation matrix that minimizes the difference between both point clouds (i.e., the point cloud of four points on the intraoperative target organ and the point cloud of four corresponding points on the preoperative 3D model). This transformation matrix can then be applied to the preoperative 3D model to reorient it to directly match the orientation of the target organ in the intraoperative images provided by the laparoscopic camera.

[0008] The system 2 shown in Figure 1 requires a lot of hardware equipment. This equipment, for example, three monitors 12a, 12b, and 12c required to display the reconstructed 3D model and real-world images from the laparoscopic camera, takes up space. This is a particular drawback in laparoscopic surgery, as operating rooms (ORs) are typically already equipped with many technical devices that limit the space required for OR staff. Furthermore, the hardware is specifically designed for navigation purposes and is therefore associated with a significant amount of cost for such systems.

[0009] Additionally, the navigation system 4 itself must be prepared for use prior to surgery. This primarily involves calibration of the tracking camera 6 and the tracking markers 8 attached to the surgical instrument 10. Because the tracking camera 6 can only track markers 8 that are within the line of sight of the tracking camera 6, the tracking markers 8 on the surgical instrument 10 must be attached to a piece of equipment outside the patient's body (e.g., to the grip of the surgical instrument). The same applies to the laparoscopic camera 11 if such equipment is applied and the surgical instrument 10 does not include a camera.

[0010] Since the tip of the surgical instrument 10 is used to select or mark any point within the field of view of the optical camera that provides a real-world view onto the target organ (i.e., correspondence selection), the position of the tracking marker 8 on the surgical instrument 10 needs to be calibrated in relation to its tip. For this purpose, a separate hardware device is used, namely a calibration unit, which needs to be specially manufactured for the surgical instrument to be used.

[0011] This configuration has several drawbacks, as the tracking markers 8 must be attached to the surgical instrument 10 and laparoscopic camera 11 (if applicable), which can adversely affect the ergonomic design of the surgical instrument 10 and laparoscopic camera 11 and thus disrupt the surgeon's work. The tracking markers 8 themselves must be attached to a marker shield that can be attached to the surgical device. The marker shield is tailored to the specific device and is therefore expensive. Furthermore, the reflective tracking balls (tracking markers 8) are disposable and need to be replaced after each procedure. They are also expensive.

[0012] Furthermore, the calibration process assumes that the distance or spatial relationship between the tracking marker 8 and the tip of the surgical instrument 10 always remains the same throughout the procedure. This may be a very simplifying assumption, especially since the surgical instrument 10 is subject to bending effects that constantly change the distance between the tracking marker 8 and the tip of the instrument 10. These effects directly affect the accuracy of the registration of the pre-operative and intra-operative images, thus resulting in significant registration errors.

[0013] Another drawback of the conventional system 2 is that the target organ (inside the patient's body) must be within the field of view of the tracking camera 6 at all times throughout the procedure. Furthermore, the provided registration results are accurate only under the assumption that the target organ does not change its shape or undergoes motion. However, both assumptions cannot be met because the patient is often turned and carried to different positions during laparoscopic surgery, and the target organ undergoes constant manipulation by the surgeon to facilitate improved access to different areas. In the former case, the calibration is no longer valid, and the entire calibration process must be repeated. In the latter case, the registration results are no longer valid, and re-registration based on corresponding point selection must be performed. Both of these steps are associated with delays in the procedure. [Prior art documents] [Patent documents]

[0014] [Patent Document 1] U.S. Patent Application Publication No. 2023 / 116388 [Patent Document 2] U.S. Patent No. 11,989,833 Summary of the Invention [Problem to be solved by the invention]

[0015] The object of the present invention is to provide an enhanced method for registering a model of a body part and a real-world image of the body part. Registration Equipment and Registration The purpose is to provide a method. [Means for solving the problem]

[0016] The objective is to develop a method for registering models of body parts and real-world images of body parts. Registration The device is solved by an input interface configured to receive 3D data of a model of a body part and 2D data of a real-world image of the body part; a display unit for visualizing the model and the body part on a 2D display of the display unit; configured to receive user input data and communicate the user input data to the display unit; At the same time a user input device, wherein the display unit is configured to manipulate a mouse pointer on the 2D display based on user input data; 、 base a registration unit configured to receive user input data indicating registration between the reference points in the real world image and the corresponding reference points in the 3D model by application of a mouse pointer for at least first to third pairs including a quasi-point and a corresponding reference point; an augmentation unit configured to augment 2D data of reference points in the real world image into synthetic 3D data by adding depth information derived from 3D data of a model of a body part and / or 2D data of the real world image; a transformation unit that determines a registration matrix for registering the 3D data of the model of the body part and the synthesized 3D data of the real-world image of the body part using the 3D coordinates of the first to third pairs of reference points; Equipped with .

[0017] The user input device functions similarly to a computer mouse and can be used similarly to a mouse pointer on a display unit. Compared to conventional systems, systems according to aspects of the present invention provide the advantage that both the real-world image and the 3D model of the body part are manipulated via the same software and the same user input device. The software provides a user interface for registering the model of the body part and its real-world image. Registration The system operates on the surgical device's processing unit. There is no need to transfer or convert the coordinates of corresponding reference points from a real-world coordinate system to the display's coordinate system. Furthermore, the system advantageously eliminates the need for expensive additional hardware, such as a tracking system. Conventional tracking systems typically include a tracking camera and a tracking marker on the surgical instrument. Both units require installation space, need to be maintained, and are costly. Advantageously, the registration process can be performed using a single user input device. In particular, the user input device can be a wireless device similar to a 3D mouse. Because the system has reduced hardware requirements, it also requires less floor space or installation space. This is particularly advantageous in an operating room environment. In summary, the system offers the following advantages: Except for the user input device, no extensive hardware setup is required. Furthermore, no extensive and time-consuming calibration procedures are required before use. Because there is no transfer of coordinates from a real-world coordinate system to a display's coordinate system, the system eliminates the calibration process. Advantageously, no modifications to the surgical instrument, such as a tracking ball or similar device, are required. These can adversely affect the ergonomic design of the surgical instrument. For example, there is no need to work within the constant field of view of a tracking camera. Furthermore, the operator / endoscopist does not need to look at a second monitor to allow registration, as everything is displayed on the main monitor. Therefore, there is no need to take his or her eye off the main monitor.

[0018] In particular, the registration unit is configured to register the reference point in the real world image and the corresponding reference point in the 3D model by applying a mouse pointer for at least first to fourth pairs including the reference point and the corresponding reference point. Dots and and receiving user input data indicative of registration between the first and second reference points. Including a fourth pair of reference points and corresponding reference points improves the accuracy of the registration.

[0019] According to an advantageous embodiment of the invention, Registration The apparatus is further enhanced by a display unit configured to visualize the registered model and the registered real-world image in an overlaid visualization, particularly a 2D display, on a single screen. Advantageously, the system eliminates the need for a second screen, which reduces costs and installation space.

[0020] The extension unit is further configured to perform adding depth information to the 2D data of at least one reference point in the real world image by assigning depth information of the 3D data of the corresponding reference point of the model to the reference point in the real world image to provide synthetic 3D data of the reference point, Registration This method can enhance the device. Real-world images contain only 2D information. To perform registration between a 2D real-world image and a 3D model of a body part, the 2D image data of the real-world image must be converted into a 3D data dataset; that is, the 2D reference points must be extended to 3D reference points. The reference points can be transferred by adding the spatial depth component of the 3D model to the 2D reference points of the real-world image. This method works under the assumption that the target body part shown in the 2D real-world image and its 3D model have approximately the same shape.

[0021] In a more sophisticated approach, according to one embodiment: Registration The apparatus is further configured such that the augmentation unit is configured to perform adding depth information to the 2D data of the reference points in the real world image, i.e. an extension unit performing monocular depth estimation on the 2D data of the real world image and assigning the estimated depth values ​​to 2D coordinates of the 2D data of the real world image to provide synthesized 3D data of the real world image; Depth estimation for synthetic 3D data value By assigning to the reference point and providing synthetic 3D data of the reference point, It will be further strengthened.

[0022] Monocular depth estimation (MDE) is an artificial intelligence-based task of estimating the depth coordinates of individual pixels in a 2D image from visual cues such as the objects in the image frame and their relationship to each other. For this purpose, monocular depth estimation algorithms are Registration It can be implemented in an expansion unit of the device. Regions or pixels of a body part in a real-world image that are farther from the camera plane tend to appear darker than regions or pixels in the real-world image that are closer to the camera plane. This effect can be enhanced by placing a light source near or to the side of the imaging device, for example, at the tip of a laparoscopic camera or at the tip of a surgical instrument where the camera is typically located.

[0023] In particular, the MDE analysis is performed on a complete 2D dataset of real-world images, meaning that the complete 2D picture is assigned depth information.

[0024] According to yet another advantageous embodiment of the present invention, Registration The apparatus is further enhanced by a topological pattern analysis unit, which performing a topological pattern analysis within a surrounding area of ​​at least one of the reference point and the corresponding reference point; Optimizing registration between at least one reference point in the real-world image and a corresponding reference point in the 3D model by selecting an optimized reference point pair, wherein a best match pattern analysis can be found within a surrounding area of ​​the optimized reference point pair, thereby selecting the optimized reference point and the optimized corresponding reference point. R, To optimize and The method is further configured to:

[0025] The initially selected reference point can be considered a starting point for triggering the optimization process. For example, if MDE is applied to estimate the depth values ​​of individual pixels in a real-world image, topological analysis can select all points in the surrounding area to search for the best match between the 3D coordinates of the 3D model and the augmented 3D dataset of the real-world image. Therefore, the uniqueness of the reference point can be explained by evaluating the relationship between the selected reference point and its neighbors in the surrounding area. This significantly improves the match, especially in application scenarios where the real-world image provides little texture information. The best match criterion can be applied using a threshold. The similarity between the topological information of the data points of the 3D model and the augmented 3D dataset of the real-world image can be described by a parameter, such as the least squares deviation. If this parameter is below a certain threshold, the similarity between the parts or regions can be considered to be a match.

[0026] In an embodiment, a topological pattern analysis unit performs monocular depth estimation on the 3D data of the model and the 2D data of the real world image, and assigns estimated depth values ​​to 2D coordinates of the 2D data of the real world image to provide the synthetic 3D data of the real world image. topological It can be further enhanced by being further configured to perform pattern analysis, in other words, extending the completed 2D dataset of real-world images into 3D data.

[0027] According to yet another advantageous embodiment of the present invention, Registration The apparatus further comprises an imaging unit configured to determine 3D data of a model of the body part during a preoperative imaging procedure, and further comprises a surgical instrument having a camera configured to acquire 2D data of a real-world image of the body part during a surgical procedure. The system provides an integrated solution for imaging and registration of the acquired data.

[0028] Furthermore, the objective is to register a model of a body part and a real-world image of the body part. Registration This method is used to solve the problem. Registration The method is: receiving 3D data of a model of a body part and visualizing the model on a 2D display; receiving 2D data of a real-world image of a body part and visualizing the body part on a 2D display; receiving user input data from a user input device operating as a mouse pointer on the 2D display; 、 base receiving, for at least first to third pairs including a quasi-point and a corresponding reference point, user input data indicating registration between the reference point in the real-world image and the corresponding reference point in the 3D model by application of a mouse pointer; augmenting the 2D data of the reference points in the real-world image into the synthetic 3D data by adding depth information derived from the 3D data of the model of the body part and / or the 2D data of the real-world image; determining a registration matrix for registering the 3D data of the model of the body part and the combined 3D data of the real-world image of the body part using the 3D coordinates of the first to third pairs of reference points; Contains .

[0029] Registration The same or similar advantages stated for the device may be realized in the same or similar manner. Registration It applies to the method and will not be repeated.

[0030] In particular, Registration The method includes receiving user input data indicating registration between a reference point in the real-world image and a corresponding reference point in the 3D model by application of a mouse pointer for at least first through fourth pairs including the reference point and the corresponding reference point, wherein including the fourth pair improves accuracy of the registration.

[0031] Advantageously, this RegistrationThe method further includes visualizing the registered model and the registered real-world image in an overlay visualization, in particular, the overlay visualization is displayed on a single screen.

[0032] In an advantageous embodiment, Registration The method further includes a step of adding depth information to the 2D data of at least one reference point in the real world image, which is performed by assigning depth information of the 3D data of the corresponding reference point of the same pair to the 2D data of the reference point to provide composite 3D data of the reference point.

[0033] Furthermore, Registration The method is: adding depth information to the 2D data of the reference points in the real world image, performing monocular depth estimation on the 2D data of the real world image and assigning estimated depth values ​​to 2D coordinates of the 2D data of the real world image to provide synthetic 3D data of the real world image; Depth estimation for synthetic 3D data value to the reference points to provide composite 3D data of the reference points; This may be advantageously enhanced by further including:

[0034] In another advantageous embodiment of the invention, the Registration The method is: performing a topological pattern analysis within a surrounding area of ​​at least one of the reference point and the corresponding reference point; optimizing registration between at least one reference point in the real-world image and a corresponding reference point in the 3D model by selecting an optimized reference point pair, wherein a best match pattern analysis can be found within a surrounding area of ​​the optimized reference point pair, thereby selecting an optimized reference point and an optimized corresponding reference point. R, an optimizing step; Further includes:

[0035] Book RegistrationThe method can be further enhanced by performing a topological pattern analysis on the synthetic 3D data of the real world image determined by performing monocular depth estimation on the 3D data of the model and the 2D data of the real world image and assigning estimated depth values ​​to 2D coordinates of the 2D data of the real world image to provide synthetic 3D data of the real world image.

[0036] In yet another advantageous embodiment, the Registration The method is particularly enhanced in that 3D data of a model of a body part is determined during preoperative imaging, and 2D data of a real-world image of the body part is acquired by a surgical instrument during a surgical procedure. The surgical instrument may be, for example, a surgical endoscope. The endoscope may have a rigid or flexible shaft. For example, the endoscope may be a laparoscope.

[0037] Further features of the invention will become apparent from the description of the embodiments according to the invention together with the claims and the included drawings. The embodiments according to the invention can feature individual features or a combination of several features.

[0038] The present invention is described below on the basis of exemplary embodiments, without limiting the general spirit of the invention, and explicit reference is made to the drawings for the disclosure of all details according to the invention not described in more detail in the text. [Brief explanation of the drawings]

[0039] [Figure 1] FIG. 1 is a simplified perspective view of a conventional surgical system including a navigation system for surgical instruments. [Figure 2] FIG. 1 is a simplified perspective view of a registration device for registering a model of a body part and a real-world image of the body part. [Figure 3] FIG. 2 is a simplified schematic diagram of a registration device for registration. [Figure 4] Comparison of a real-world image (Fig. 4a) and an overlay image (Fig. 4b). The real-world image is shown together with a 3D model representation of each body part. [Figure 5]5a) shows a real-world image and the corresponding depth map computed by performing MDE on the real-world image (FIG. 5b). [Figure 6] 1 is a flowchart illustrating the individual steps performed in a registration method for registering a model of a body part and a real-world image of the body part. DETAILED DESCRIPTION OF THE INVENTION

[0040] FIG. 1 is a simplified perspective view of a prior art surgical system that does not form part of the present invention.

[0041] FIG. 2 shows a system for registering a model 34 of a body part (see FIG. 4) and a real-world image 32 of this body part (see FIG. 4). Registration equipment 20 (Hereinafter referred to as device 20) 1 is a simplified perspective view of a body part 20. The body part may in particular be an internal organ of a patient. The device 20 comprises a main unit 28 mounted on a rack 27. Atop the rack 27 is a 2D display 26, e.g., a flat screen display. Further details of the main unit 28 will be described with reference to FIG. 3. The device 20 further comprises a user input device 22, in particular, wirelessly connected to the main unit 28. The user input device 22 is configured to receive user input data. Operation of the user input device 22 is similar to that of a wireless 3D mouse. A mouse pointer on the display 26 is configured to be manipulated, e.g., by tilting and twisting, and further actions or operations can be performed using buttons on the user input device 22.

[0042] FIG. 3 illustrates a schematic diagram of the functional units of the device 20. The main unit 28 includes an input interface 21 configured to receive 3D data of a model 34 of a body part. This 3D data can be acquired, for example, using an imaging unit 44, which can form part of the device 20. The imaging unit 44 can be, for example, a computed tomography (CT) or magnetic resonance imaging (MRI) unit, which can be used to examine a patient's internal organs. The data from the imaging unit 44 is used to calculate a model 34 of the body part, for example, the patient's internal organ. The 3D data of the model 34 can be transmitted from the imaging unit 44 to the input interface 21 of the main unit 28. Alternatively, the 3D data of the model 34 can be transmitted from another source, for example, a computer configured to calculate a 3D model. For example, the 3D data can be derived from acquiring a 2D CT / MRI scan, which is then converted into a 3D model. The input interface 21 also receives 2D data of a real-world image 32 of the body part. The real-world image 32 is acquired, for example, during a surgery performed with a surgical instrument 46 on a patient's internal organs previously examined with the imaging unit 44. The surgical instrument 46 is, for example, a laparoscope. The distal end of the shaft of the surgical instrument 46 has a camera 48 for capturing the real-world image. The real-world image data is transmitted to the main unit 28 of the device 20 via the input interface 21.

[0043] The main unit 28 of the device 20 further comprises a display unit 24, e.g., an image processor, configured to visualize the model 34 and the body part on a 2D display 26. The model 34 and the body part can be displayed on a single screen. For example, the output of the main unit 28 can be a two-dimensional image showing the intraoperative video and the 3D model superimposed on the intraoperative video at a fixed orientation and scale.

[0044] The user input device 22 is configured to receive user input data, for example, by manual operation thereof. The user input data is transmitted to the display unit 24, which is then configured to operate a mouse pointer on the 2D display 26 based on the user input data. That is, the user input device 22 functions as a mouse pointer on the 2D display 26.

[0045] The main unit 28 further includes a registration unit 30 configured to receive user input data. The user input indicates at least first to third pairs of reference points P, P*. Each pair of reference points includes a reference point P and a corresponding reference point P*. The reference points are registered in the real-world image 20 by operation of the user input device 22 and by respective selections on the 2D display 26. 32 and on the model 34. This selection indicates registration between a reference point P in the real world image 32 and a corresponding reference point P* in the 3D model 34. Further details are provided with reference to FIG.

[0046] The data of the real world image 32 is 2D data. In order to register it to the 3D model 34, the 2D data needs to be converted into 3D data. For this purpose, there is an augmentation unit 36, which also forms part of the main unit 28. The augmentation unit 36 ​​is configured to augment the 2D data of a plurality of reference points P in the real world image 32 into synthetic 3D data. This is done by adding depth information to the coordinates in the 2D data of the reference points P. This depth information is derived from the 3D data of the model 34 of the body part and / or the 2D data of the real world image 32, as will be explained in more detail below.

[0047] The main unit 28 further includes a transformation unit 38 configured to determine a registration matrix for registering the 3D data of the model 34 of the body part and the composite 3D data of the real-world image 32 of the body part. The transformation matrix is ​​calculated from the coordinates of the first to third reference points P and the corresponding reference points P*. User selection of the reference point P and the corresponding reference point P* indicates that the reference point P in the real-world image 32 and the corresponding reference point P* in the model correspond to each other, meaning that the reference points P and P* point to the same position on the displayed body part. In other words, the reference point P selected in the real-world image 32 can be found at the position of the corresponding reference point P* in the model 34. Based on the coordinates of the four pairs of reference points P and P*, a transformation matrix for matching the real-world image 32 and the model 34 can be calculated for each pixel or point of the model 34 or the real-world image 32.

[0048] As a result, overlay images can be displayed on the 2D display 26 to assist the surgeon during surgery on the body part.

[0049] Figure 4a shows a real-world image 32, captured by a camera 48, for example, on the tip of a surgical instrument 46. The real-world image 32 shows a patient's internal organ, for example, a portion of the liver. Figure 4b shows an overlay of the real-world image 32 with a model 34 of the internal organ. The data for the model 34 is acquired preoperatively and shows, for example, the location and path of blood vessels or other internal structures of the organ. This information can assist the surgeon during the performance of the surgery.

[0050] The determination of the depth information to be added to the 2D data of the real world image 32 to provide synthetic 3D data can be performed by applying different techniques.

[0051] According to a first approach, the augmentation unit 36 ​​is configured to perform the addition of depth information to the 2D data of the reference point P in the real world image 32 by simply assigning the depth information of the 3D data of the corresponding reference point P* of the model to the reference point P in the real world image 32.

[0052] According to another approach, the augmentation unit 36 ​​is further configured to perform adding depth information to the 2D data of the reference point P in the real world image 32 as follows: The augmentation unit 36 ​​performs monocular depth estimation (MDE) on the 2D data of the real world image 32. The augmentation unit 36 ​​then assigns estimated depth values ​​from the monocular depth estimation to the 2D data of the real world image 32 to provide synthesized 3D data of the real world image 32. The assignment of estimated depth information can be performed for the reference point P or for each pixel or point of the real world image 32.

[0053] Figure 5a is a real-world image acquired, for example, by camera 48 of surgical instrument 46. Figure 5b shows the corresponding depth map in grayscale encoding, which is computed by performing monocular depth estimation on the real-world image 32 shown in Figure 5a.

[0054] The device 20 implements, among other things, software known as AXR-7, available from the present applicant. In its current configuration, the AXR-7 software is a lightweight navigation system that enables a surgeon to perform laparoscopic procedures by accessing reconstructed 3D model data of the target organ and intraoperative images of the target organ provided by a laparoscopic camera on the same monitor—i.e., on the 3D model 34 and on the real-world image 32. The surgeon can manipulate the field of view on the 3D model 34 by changing its orientation and can translate the 3D model 34 on the 2D display 26 throughout the procedure to obtain additional information about the spatial anatomy within the target organ. Additionally, the surgeon can manually overlay the 3D model 34 on the intraoperative target organ, which mimics the registration process, which can be done using the user input device 22. The device 20 implements an enhanced version of the AXR-7 software aimed at introducing semi-automatic registration functionality, as described below.

[0055] The enhanced AXR-7 software operates in combination with a mouse pointer function controlled by a user input device 22. The enhanced AXR-7 software may include functionality that enables the user input device 22 to be used to control a virtual mouse pointer (VMP). The VMP functions similarly to a computer mouse (hardware) and can be used as a pointer (software) on a 2D display 26. The advantage of the AXR-7 software over traditional navigation systems is that both the intraoperative real-world image 32 of the target organ as a body part and the 3D representation, i.e., the 3D model 34 (particularly the preoperative 3D model), of the target organ are already visualized on the same operating room monitor and can be controlled via the same software (i.e., the AXR-7 software). Therefore, the transfer of corresponding reference points tracked in physical space to software space is no longer required in this setting.

[0056] The 3D model 34 is displayed in a Picture-in-Picture (PiP) layer and can be overlaid on the real-world image of the laparoscopic video 32. This is important because it technically makes it possible to separate the pixels provided by the 2D video from the voxels (the 3D equivalent of a pixel) provided by the PiP layer.

[0057] Here, a user can use VMP to select at least four reference points P on the target organ as shown in the 2D intraoperative real-world image 32. Subsequently, the user uses VMP to select four corresponding reference points on the preoperative 3D model 34 of the target organ. However, because the selected reference points P on the target organ in the image are only 2D points (i.e., points described by x and y coordinates), they need to be expanded to become 3D points. To add a spatial depth component (i.e., z coordinate) to the 2D points, the z coordinate given by the corresponding point on the 3D model 34 can be transferred to the 2D reference points P. This technique works under the assumption that the target organ as shown in the 2D intraoperative real-world image 32 and the target organ given by the 3D model 34 have approximately the same shape (i.e., no deformation). A more advanced technique for converting 2D points to 3D points is provided by the use of monocular depth estimation (MDE). MDE can be artificial intelligence-based. It aims to estimate the depth values ​​of individual pixels in a 2D real-world image 32, thus making it possible to convert 2D points into 3D points. The result of this expansion is a point cloud.

[0058] Another advantage of the apparatus 20 is that the initially selected correspondence of the reference points P, P* can be considered a starting point for triggering the optimization process. For example, if MDE is applied to estimate the depth values ​​for each individual pixel in the intraoperative 2D real-world image 32, the software can use all points in the real-world image 32 to search for the most unique point in this image. The uniqueness of a point can be explained, for example, by evaluating the relationship between the selected point and its neighbors. This is a topological pattern analysis that aims to identify the most unique point or cluster of most unique points. The same process can then be applied to the 3D model 34 using the selected corresponding reference point P* as an initial starting point to better search for an enhanced reference point P* (or cluster of points) that matches the reference point P (or cluster of points) on the 2D real-world image. In this approach, inaccurate user-based selection of the corresponding point pair P, P* can be compensated for in the initial automatic optimization process, ultimately resulting in a more accurate registration result.

[0059] FIG. 6 shows a flow chart of the individual steps performed during a method for registering a 3D model 34 of a body part and a real-world image 32 of the body part.

[0060] In step 1, a virtual mouse pointer is applied to select a reference point P on the 2D real-world image 32 by operating the user input device 22. The real-world image 32 is displayed on the 2D display 26 together with the 3D model 34. In step 2, the virtual mouse pointer is used to select a corresponding reference point P* in the 3D model 34. In step 3, the depth value of the selected corresponding reference point P* is transferred to the reference point P in the real-world image 32. Alternatively, the depth information can be derived from the MDE. In step 4, this process is repeated for three additional pairs of reference points P, P*. In step 5, a transformation matrix is ​​calculated from the coordinates of the reference point P and the corresponding reference point P*. Registration is applied in step 6. In step 7, the registered real-world image 32 and the 3D model 34 are displayed on the 2D screen 26. Steps 3 to 7 are performed fully automatically. Only steps 1 and 2, which imply the selection of a reference point P in the real-world image 32 and the selection of a corresponding reference point P* in the 3D model 34, have to be performed manually.

[0061] In the drawings, elements of the same or similar type or corresponding parts are given the same reference numerals so as not to have to reintroduce the items.

[0062] All specified features, including those obtained solely from the drawings, and individual features disclosed in combination with other features, are considered to be important to the invention both alone and in combination. Embodiments according to the invention can be realized by individual features or by a combination of several features. Features associated with the terms "in particular" or "especially" should be treated as preferred embodiments. [Explanation of symbols]

[0063] 2 Surgical Systems 4. Navigation system 6. Tracking Camera 8 Tracking Markers 10 Surgical instruments 11 Laparoscopic camera 12a First Monitor 12b Second Monitor 12c Third Monitor 20 equipment 21 Input Interface 22 User Input Devices 23 Display Processing Unit 24 display units 25 Wireless Data Link 26 2D displays 27 racks 28 Main unit 30 Registered Units 32 Real-world images 34 3D models 36 Expansion Unit 38 Conversion Unit 40 Topological Pattern Analysis Unit 42 Surrounding Area 44 Imaging unit 46 Surgical instruments 48 Camera P reference point P* Corresponding reference point

Claims

1. 1. An apparatus (20) for registering a model (34) of a body part and a real-world image (32) of said body part, said apparatus (20) comprising: an input interface (21) configured to receive 3D data of the model (34) of the body part and 2D data of the real-world image (32) of the body part; a display unit (24) for visualizing said model (34) and said body part on a 2D display (26) of said display unit (24); a user input device (22) configured to receive user input data and communicate the user input data to the display unit (24), the display unit (24) configured to manipulate a mouse pointer on the 2D display (26) based on the user input data; Equipped with a registration unit (30) configured to receive user input data indicating registration between a reference point (P) in the real world image (32) and the corresponding reference point (P*) in the 3D model (34) by application of the mouse pointer for at least first to third pairs including the reference point (P) and the corresponding reference point (P*); an augmentation unit (36) configured to augment the 2D data of the reference points (P) in the real world image (32) into synthetic 3D data by adding depth information derived from the 3D data of the model (34) of the body part and / or the 2D data of the real world image (32); a transformation unit (38) for determining a registration matrix for registering the 3D data of the model (34) of the body part and the composite 3D data of the real-world image (32) of the body part using the 3D coordinates of the first to third pairs of reference points (P, P*); characterized by, An apparatus (20) for registering a model (34) of a body part and a real-world image (32) of said body part.

2. 2. The device (20) of claim 1, wherein the display unit (23) is configured to visualize the registered model (34) and the registered real-world image (32) in an overlay visualization, in particular the 2D display (26) is a single screen.

3. 3. The apparatus (20) of claim 1 or 2, wherein the augmentation unit (36) is further configured to perform the addition of the depth information to the 2D data of the at least one reference point (P) in the real world image (32) by assigning the depth information of the 3D data of the corresponding reference point (P*) of the model (34) to the reference point (P) of the real world image (32) to provide the synthetic 3D data of the reference point (P).

4. The augmentation unit (36) is further configured to perform the addition of the depth information to the 2D data of the reference points (P) in the real world image (32), namely: the augmentation unit (36) performs monocular depth estimation on the 2D data of the real world image (32) and assigns the estimated depth values ​​to the 2D coordinates of the 2D data of the real world image (32) to provide synthesized 3D data of the real world image (32); assigning the estimated depth information of the composite 3D data to the reference point (P) to provide the composite 3D data of the reference point (P); 3. An apparatus (20) according to claim 1 or 2.

5. The method further comprises a topological pattern analysis unit (40), the topological pattern analysis unit (40) performing a topological pattern analysis within a surrounding area (40) of at least one of said reference point (P) and said corresponding reference point(s) (P*); optimizing the registration between the at least one reference point (P) in the real-world image (32) and the corresponding reference point (P*) in the 3D model (34) by selecting an optimized pair of reference points (P, P*), wherein the optimized reference point and the optimized corresponding reference point are selected by a best match pattern analysis that can be found within the surrounding area (42) of the optimized pair of reference points; The device (20) according to one of claims 1 to 4, configured to:

6. 6. The apparatus (20) of claim 5, wherein the topological pattern analysis unit (40) is further configured to perform the pattern analysis on the 3D data of the model (34) and the synthetic 3D data of the real world image (32), determined by performing the monocular depth estimation on the 2D data of the real world image (32) and assigning the estimated depth values ​​to the 2D coordinates of the 2D data of the real world image (32) to provide the synthetic 3D data of the real world image (32).

7. 7. The apparatus (20) according to claim 1, further comprising an imaging unit (44) configured to determine the 3D data of the model (34) of the body part during a preoperative imaging procedure, and further comprising a surgical instrument (46) having a camera (48) configured to acquire the 2D data of the real-world image (32) of the body part during a surgical procedure.

8. 1. A method of registering a model (34) of a body part and a real-world image (32) of said body part, said method comprising: receiving 3D data of the model (34) of the body part and visualizing the model (34) on a 2D display (26); receiving 2D data of the real-world image (32) of the body part and visualizing the body part on the 2D display (26); receiving user input data from a user input device (22) that operates as a mouse pointer on the 2D display (26); Including, receiving user input data indicating registration between the reference point (P) in the real-world image (32) and the corresponding reference point (P*) in the 3D model (34) by application of the mouse pointer for at least first to third pairs including the reference point (P) and the corresponding reference point (P*); augmenting the 2D data of the reference point (P) in the real world image (32) into synthetic 3D data by adding depth information derived from the 3D data of the model (34) of the body part and / or the 2D data of the real world image (32); determining a registration matrix for registering the 3D data of the model (34) of the body part and the combined 3D data of the real-world image (32) of the body part using the 3D coordinates of the first to third pairs of reference points (P, P*); characterized by, A method for registering a model (34) of a body part and a real-world image (32) of said body part.

9. 9. The method of claim 8, further comprising the step of visualizing the registered model (34) and the registered real-world image (32) in an overlay visualization, in particular, the overlay visualization being displayed on a single screen.

10. 10. The method according to claim 8 or 9, wherein the step of adding depth information to the 2D data of the at least one reference point (P) in the real world image (32) is performed by assigning the depth information of the 3D data of the corresponding reference point (P*) of the same pair to the 2D data of the reference point (P) to provide the composite 3D data of the reference point (P).

11. adding depth information to the 2D data of the reference point (P) in the real world image (32), performing monocular depth estimation on the 2D data of the real world image (32) and assigning the estimated depth values ​​to the 2D coordinates of the 2D data of the real world image (32) to provide synthesized 3D data of the real world image (32); assigning the estimated depth information of the composite 3D data to the reference point (P) to provide the composite 3D data of the reference point (P); 11. The method of any one of claims 8 to 10, further comprising:

12. performing a topological pattern analysis within a peripheral area (34) of at least one of said reference point (P) and said corresponding reference point(s) (P*); optimizing the registration between the at least one reference point (P) in the real-world image (32) and the corresponding reference point (P*) in the 3D model (34) by selecting an optimized reference point pair, wherein the optimized reference point and the optimized corresponding reference point are selected by a best match pattern analysis that can be found within the surrounding area (42) of the optimized reference point pair; 12. The method of any one of claims 8 to 11, further comprising:

13. 13. The method of claim 12, wherein the topological pattern analysis is performed on synthetic 3D data of the real world image (32) determined by performing monocular depth estimation on the 3D data of the model (34) and the 2D data of the real world image (32), and assigning the estimated depth values ​​to the 2D coordinates of the 2D data of the real world image (32) to provide the synthetic 3D data of the real world image (32).

14. 14. The method according to any one of claims 8 to 13, wherein the 3D data of the model (34) of the body part is determined during pre-operative imaging, in particular the 2D data of the real-world image (32) of the body part is acquired by a surgical instrument (46) during a surgical procedure.

Images