Laparoscopy system

Abstract

The present invention relates to a laparoscopy system (1) comprising a laparoscope (2) suitable for acquiring a laparoscopy image (12), a first laparoscopic instrument (3), a control unit (4), and a display unit (5). The control unit (4) is configured to provide a mobility model (22) of the first instrument (3), which determines the maximum mobility of the first instrument (3) in the body part (11) relative to its first pose (P1.3). Furthermore, the control unit is configured to receive control commands for a 3D model of the first instrument (3M) and to determine a virtual spatial movement of the 3D model of the first instrument (3M) using the received control commands and the mobility model (22) of the first instrument, wherein the virtual spatial movement of the 3D model (3M) of the first instrument is limited by the maximum mobility of the first instrument determined in the mobility model (22).The control unit is configured to generate a virtual 3D scene depicting the virtual spatial movement of the 3D model of the first instrument (3M), and based on this, to generate a virtual simulation representation showing a virtual camera view of the 3D scene from the perspective of the laparoscope (2). Furthermore, the control unit (4) is configured to generate a superimposed display (23) in which the laparoscopy image (12) is superimposed on the virtual simulation display. The display unit (5) is configured to display the superimposed display (23).

Description

[0001] The present invention relates to a laparoscopy system, a method for displaying a superimposed image using a laparoscopy system, and an associated computer program product.

[0002] Laparoscopic surgery, also known as minimally invasive surgery or "keyhole surgery," is a surgical technique in which surgical instruments and a camera (laparoscope) are inserted into a patient's body cavity through small incisions and trocars. This allows the surgeon to visualize the surgical area and instruments in a live video feed on a monitor and perform precise procedures.

[0003] Laparoscopic surgery requires considerable skill and experience from the surgeon to navigate complex anatomical structures. The surgeon may encounter challenges or uncertainties during the procedure, such as identifying specific anatomical landmarks or orienting themselves in difficult areas.

[0004] To support the surgeon in such situations, telemedicine systems are used, through which another expert can provide assistance and support to the surgeon.

[0005] In addition to direct communication, telestration is also used. Telestration is a term derived from the words "tele" and "demonstration" and refers to the real-time marking and annotation of laparoscopic images or videos during a surgical procedure.

[0006] The expert (remote expert, telemedicine doctor) has access to the live image (live video) of the procedure and annotates it using various input devices, for example by precisely marking and naming anatomical structures (anatomical objects), drawing attention to dangers with lines, arrows or symbols, or drawing a planned surgical procedure or action and thus suggesting a movement path of a surgical instrument.

[0007] To improve the spatial understanding of the surgical field for both the surgeon and the expert, it was proposed to generate and display a virtual 3D representation of the surgical field based on the live image (live video) of the surgical field. The expert's annotations could then also be made using this virtual 3D representation and presented to the surgeon.

[0008] Recent advances in computer vision and computer-aided photography have made it possible to create detailed 3D reconstructions of complex scenes from 2D image data. Techniques such as photogrammetry, neural radiance fields (NeRF), and Gaussian splatting are increasingly used to create highly accurate 3D models of environments such as surgical fields, including those encountered in laparoscopic surgery.

[0009] For example, US patent US 9 298 884 B1 describes a telestration system in which a 3D representation of the surgical area is generated based on the live video of a camera, on which an expert can make annotations in the form of markings, lines or arrows to give instructions and assistance to the local surgeon in carrying out a surgical procedure.

[0010] Against the background of this prior art, the present invention is based on the objective of providing a laparoscopy system which is characterized by improved practicality, in particular with regard to the improved support of the surgeon in carrying out the surgical procedure by avoiding surgical errors and enabling a more precise and improved execution of the surgical procedure by the surgeon.

[0011] This problem is solved by the laparoscopy system according to claim 1, the method for displaying a superimposed image according to claim 6 and the computer program product according to claim 7.

[0012] The laparoscopy system according to the invention comprises - a laparoscope that can be inserted into a part of a patient's body, whose spatial pose and viewing direction can be determined, suitable for capturing a laparoscopy image, - a first laparoscopic instrument that can be inserted into the patient's body part through an instrument pivot point (fulcrum point) and extends along a longitudinal instrument axis, - a control unit, and - a display unit, the control unit is designed to - to provide the specific spatial pose and viewing direction of the laparoscope in a (global) coordinate system, - to receive the captured laparoscopy image, wherein the laparoscopy image includes (at least a partial) representation of the first instrument, - to provide a virtual 3D model of the first instrument, - to determine a first spatial pose of the first instrument in the coordinate system and an instrument pivot point of the first instrument (fulcrum point) using the spatial pose and the viewing direction of the laparoscope and the representation of the first instrument in the laparoscopy image, - to create an initial virtual 3D scene in which the virtual 3D model of the first instrument is arranged (represented) in the coordinate system, taking into account its specific spatial initial poses, - to provide a mobility model of the first instrument (by defining three translational and three rotational degree-of-freedom parameters) that determines the maximum mobility of the first instrument in the body part relative to its specific first pose, in particular such that the first instrument has maximum mobility relative to its first pose - is translationally movable along the longitudinal axis of the instrument, and the longitudinal play is in particular between 5 and 25 cm, - is translationally movable perpendicular to the longitudinal axis of the instrument in order to allow a transverse movement, and the transverse movement is in particular 2 cm, 1 cm or 0.5 cm, - is rotatable by a roll angle around the instrument's longitudinal axis, and the roll angle is in particular 360°, - is rotatable about a first pivot angle about a first pivot axis and about a second pivot angle about a second pivot axis, wherein the two pivot axes are arranged perpendicular to each other and to the longitudinal axis of the instrument and intersect at the instrument pivot point (fulcrum point), and the first and second pivot angles are each in particular 90°, 60°, 45° or 30°, - To receive control commands for the 3D model of the first instrument, - to determine a virtual spatial movement of the 3D model of the first instrument using the received control commands and the mobility model of the first instrument, wherein the virtual spatial movement of the 3D model of the first instrument is limited by the maximum mobility of the first instrument determined in the mobility model, - to create a second virtual 3D scene in which the virtual spatial movement of the 3D model of the first instrument is represented in the (global) coordinate system, - to create a virtual simulation representation in which a virtual camera view of the first and / or the second 3D scene is shown from the perspective of the laparoscope using the spatial pose and the viewing direction of the laparoscope, - to create a superimposed display in which the laparoscopy image is superimposed with the virtual simulation display, and the display unit is set up to display the superimposed display.

[0013] Through the synergistic interplay of the essential features of the invention, a laparoscopy system can be realized that provides even better support to the surgeon during the execution of the surgical procedure (a medical procedure) and thus makes a technical contribution.

[0014] The virtual spatial movement of the 3D model of the first instrument, displayed in the overlay view, serves as a suggestion or guide for the operator on how to move or operate the first instrument. This suggestion can originate from a human expert or a computer-implemented planning unit.

[0015] By taking into account the maximum mobility of the first instrument relative to its initial position in the surgical field when displaying the virtual spatial movement of the 3D model to the surgeon, it can be ensured that the virtually simulated surgical procedure can actually be implemented in reality. This prevents the virtual simulation from suggesting a movement of the first instrument to the surgeon that is mechanically or kinematically impossible in reality, as this would require moving the first instrument beyond its maximum mobility. The mobility model determines the maximum (kinematic, mechanical) mobility of the first instrument relative to its initial position.

[0016] The mobility model thus refers to a computer-aided (geometric, kinematic, mechanical) description of the maximum permissible range of motion (maximum mobility) of the first instrument relative to its first pose. The mobility model therefore defines the achievable working range of the first instrument.

[0017] When determining the virtual spatial movement of the 3D model of the first instrument using control commands, the movement of the 3D model is limited by the maximum range of motion defined in the mobility model. For example, if the lateral play is limited to 1 cm according to the mobility model, this maximum lateral play value of 1 cm will not be exceeded in the virtual movement of the 3D model of the first instrument, even if the corresponding control command aims for a larger lateral displacement of the first instrument. The same applies, of course, to all other degrees of freedom. Thus, when determining the virtual movement of the 3D model of the first instrument, the movements of the 3D model are restricted to the extent that they lie within the working range defined by the mobility model.

[0018] Within the framework of the virtual spatial movement of the 3D model of the first instrument, the 3D model of the first instrument can therefore only assume poses that lie within the working area defined by the mobility model around the first pose of the first instrument.

[0019] The laparoscopy system according to the invention thus makes it possible for the surgeon to be shown only virtual simulations - i.e. suggestions - for the movement of the first instrument (and thus for surgical actions) that are also feasible in reality because they take into account the real conditions in the operating area and the mechanical, kinematic working range of the first instrument.

[0020] The quality of the virtual simulation of a planned surgical procedure—and thus of the telestration—can be significantly improved in this way. It can prevent the surgeon from being guided by the virtual movement of the 3D model of the first instrument to perform a surgical action that requires unacceptably large movements of the first instrument and could therefore lead to potential patient injuries and / or is simply mechanically impossible (faulty guidance).

[0021] The inventive visualization of the virtual movement of the first instrument, limited by the mobility model, thus functionally intervenes in the control process of the surgical procedure by the surgeon.

[0022] This representation thus results in an objective technical improvement of the interaction between the laparoscopy system and the surgeon (human-machine interaction), as the execution of the surgical procedure is not only facilitated but also made technically more precise by avoiding faulty instructions. Critical movements of the first instrument by the surgeon outside the permissible range of motion can therefore be avoided. Conversely, it can be ensured that the surgeon moves the first instrument only within the permissible range of motion (defined by the mobility model). This can translate into increased surgical safety.

[0023] The surgeon is thus objectively and reliably supported by the laparoscopy system according to the invention in carrying out the technical task of the surgical procedure.

[0024] Some aspects and features of the invention are explained in detail below.

[0025] In the context of the present invention, the term laparoscope refers to a medical instrument specifically designed for minimally invasive examination of body parts, in particular body cavities or the abdominal cavity, and used for visualizing internal anatomical structures.

[0026] A laparoscope comprises an optical system with a light source for illuminating the surgical area and a camera or imaging system for capturing and transmitting a real-time (live) laparoscopy image of the surgical area to a control unit, as well as an elongated shaft that houses the optical system and gives the laparoscope its mechanical structure.

[0027] Determining the spatial pose and viewing direction of a laparoscope is state of the art and can be done in various ways, particularly using laparoscope markers and associated tracking systems.

[0028] A laparoscope marker is a component that is permanently attached to the laparoscope and serves to determine its spatial pose. The marker thus enables precise tracking of the laparoscope within a defined coordinate system. Laparoscope markers can utilize different operating principles to ensure accurate and reliable pose determination and can be implemented as optical, electromagnetic, and / or inertial markers. An optical marker comprises reflective or active-illuminating elements that can be detected by optical tracking systems. An electromagnetic marker comprises sensors that detect the electromagnetic fields of an electromagnetic tracking system. An inertial marker uses accelerometers and / or gyroscopes to detect accelerations and angular velocities experienced by the laparoscope in order to determine its pose.

[0029] The tracking system is suitable for capturing the spatial pose of the laparoscope marker and is adapted to the marker's operating principle.

[0030] In summary, it can be said that the pose and direction of view of the laparoscope can be recorded using sensor data and reflect the real situation currently present in the surgical area.

[0031] The spatial pose of an object, in this context, refers to its complete spatial description, encompassing both its position and orientation within a coordinate system. Position describes the object's spatial location using Cartesian coordinates (x, y, z), also known as translation parameters, which specify the location of a reference point relative to a defined origin. Orientation, on the other hand, specifies the object's spatial orientation using angles, also called rotation parameters, which describe its rotation around the three axes (X, Y, Z). Thus, the pose is defined by six parameters—three for position and three for orientation—and provides a complete spatial description of an object.

[0032] The viewing direction of a laparoscope refers to the orientation of its optical axis in space. It describes the direction in which the laparoscope "looks" at a given pose, i.e., the orientation of the image acquisition relative to the coordinate system. The viewing direction is typically defined by a vector along the laparoscope's optical axis, originating from the image sensor of the camera or the shaft and pointing into the image acquisition area. The viewing direction, together with the laparoscope's pose, forms the basis for determining the field of view and the representation of virtual superimpositions in the laparoscopy image.

[0033] The laparoscope and the first surgical instrument are typically inserted into the body part or cavity through a tubular trocar. A trocar is a medical device used to create access to a body part, particularly a body cavity. The trocar consists of a tubular sleeve that is inserted into the body part and serves as an access channel for a laparoscope or other surgical instruments.

[0034] In this context, a laparoscopic instrument is understood to be a surgical tool suitable for performing laparoscopic procedures. A laparoscopic instrument has an elongated shaft that is inserted into the body part or cavity through a trocar, establishing a connection between the instrument's working element and its operating element. The working element is used to perform specific surgical tasks, such as grasping, cutting, or suturing. The operating element is used to control the working element and may include levers, buttons, and / or controls. The length of the laparoscopic instrument's shaft defines its longitudinal axis.

[0035] The control unit comprises a computer (computer) with a processor unit and a memory unit. The computer can be a single-board computer, embedded system, digital terminal, desktop computer, edge device, or server. The processor unit is capable of performing arithmetic operations and can be a CPU, GPU, or similar device with one or more processor cores. Specifically, the processor unit is configured to receive the laparoscopy image, determine spatial poses and instrument pivot points, generate virtual 3D scenes, receive control commands, determine virtual movements of 3D models, and generate virtual simulation and overlay displays. The memory unit is capable of storing and outputting data (to the processor unit) and can be configured as volatile or non-volatile memory.The storage unit is specifically designed to provide spatial poses of the laparoscope, 3D models of the first instrument, and a mobility model by making corresponding data sets available in the storage unit.

[0036] The simulation is presented in particular as a simulation video, which depicts the virtual spatial movement of the 3D model of the first instrument over time.

[0037] The spatial pose of the first instrument in the (global) coordinate system is determined, in simplified terms, based on the position and dimensions (i.e., the size) of the first instrument in the captured laparoscopy image, as well as the known pose and viewing direction of the laparoscope. The determination of the first instrument's pose is thus based on sensor data that reflects the actual situation currently present in the surgical field.

[0038] Approaches to determining or estimating the spatial pose of an instrument from an image within a surgical scene are already known in the art and are used outside of medical technology in the field of computer-aided image processing as so-called 6D pose estimation methods. This 6D pose estimation is made possible by analyzing an image depicting the instrument in conjunction with known (extrinsic and / or intrinsic) imaging parameters of the camera, the known spatial pose and viewing direction of the camera, and the known dimensions of the instrument.

[0039] Given that such 6D pose estimation methods can be assumed to be common knowledge, a detailed explanation is omitted here, and reference is made to the following publications, which address or describe such methods in more detail: Park, J., Hong, J., Yoon, J., Park, B., Choi, MK, & Jung, H. (2024, October). Towards Precise Pose Estimation in Robotic Surgery: Introducing Occlusion-Aware Loss. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 639-648). Cham: Springer Nature Switzerland; Rai, U., Xu, H., & Giannarou, S. (2025). SurgPose: Generalisable Surgical Instrument Pose Estimation using Zero-Shot Learning and Stereo Vision. arXiv preprint arXiv:2505.11439. Yoshimura, M., Marinho, MM, Harada, K., & Mitsuishi, M. (2020, May). Single-shot pose estimation of surgical robot instruments' shafts from monocular endoscopic images.In 2020 IEEE International Conference on Robotics and Automation (ICRA) (pp. 9960-9966). IEEE.; Li, X., Shen, L., Li, L., Mo, H., Wang, X., Ma, X., & Zheng, J. (2024, December). 3DGS-based Tracking of Articulated Instruments for Instruments-Tissue Interaction Estimation in Robot-assisted Minimally Invasive Surgery. In 2024 IEEE International Conference on Robotics and Biomimetics (ROBIO) (pp. 1670-1675). IEEE.; Li, X., Shen, L., Li, L., Mo, H., Wang, X., Ma, X., & Zheng, J. (2024). 3DGS-based Tracking of Articulated Instruments for Instruments-Tissue Interaction Estimation in Robot-assisted Minimally Invasive Surgery.; Barragan, J. A., Zhang, J., Zhou, H., Munawar, A., & Kazanzides, P. (2024, May). Realistic data generation for 6d pose estimation of surgical instruments. In 2024 IEEE International Conference on Robotics and Automation (ICRA) (pp. 13347-13353). IEEE.; EP3273854B1.

[0040] Determining (estimating) the instrument pivot point of the first instrument, also called the fulcrum point of the first instrument, is also part of the state of the art in laparoscopy systems.

[0041] The instrument pivot point or fulcrum point is the point around which a laparoscopic instrument inserted through an incision into a part of the body can be rotated.

[0042] This point typically corresponds to the point where the instrument passes through the body wall and represents a kinematic center of rotation that significantly determines the instrument's mobility within the body. In an idealized and simplified model, the instrument's pivot point is defined as a geometric point intersected by the instrument's longitudinal axis, allowing the instrument to pivot around two mutually perpendicular axes. The instrument's pivot point thus limits the instrument's range of motion to a cone-shaped working area.

[0043] Given that procedures for determining (estimating) the instrument's pivot point can be assumed to be common knowledge, a detailed explanation is omitted here, and reference is made to the following publications, which address or describe such procedures in more detail: Lin Dong and G. Morel, “Robust trocar detection and localization during robot-assisted endoscopic surgery,” 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, 2016, pp. 4109–4114, doi: 10.1109 / ICRA.2016.7487602. The instrument pivot point can also be determined by setting it in advance of the operation, as described in the following paper: M. Feuerstein, T. Mussack, SM Heining and N. Navab, “Intraoperative Laparoscope Augmentation for Port Placement and Resection Planning in Minimally Invasive Liver Resection,” in IEEE Transactions on Medical Imaging, vol.27, no. 3, pp. 355-369, March 2008, doi: 10.1109 / TMI.2007.907327.

[0044] A virtual 3D model of the first instrument is understood to be a computer-generated, three-dimensional representation of the geometric structure of the first surgical instrument. The model is provided (generated) in a digital environment within the control unit and is based on known design data of the first instrument.

[0045] A virtual 3D scene is a computer-generated, three-dimensional representation of one or more virtual 3D models in a (global) coordinate system. The 3D scene takes into account the pose (i.e., the position and orientation) of the objects it contains relative to this coordinate system, thus realistically depicting their spatial arrangement in virtual space.

[0046] In this context, the (global) coordinate system is understood to be a spatially fixed reference system that serves as a uniform reference frame for determining position and orientation (pose determination). The global coordinate system is, in particular, independent of the movement or position of the laparoscope and the first instrument.

[0047] The first instrument is typically inserted into the body part (or body cavity) through an incision via a trocar and is therefore only movable to a limited extent relative to the body part or the surgical area.

[0048] This limited mobility of the first instrument within the body part (or body cavity) is represented and accounted for by the mobility model. The mobility model describes the maximum range of motion of the first instrument relative to its initial spatial pose along three translational and three rotational degrees of freedom. For each degree of freedom, a maximum value is defined by which the instrument can be moved along that degree relative to its initial spatial pose. These maximum values ​​are referred to as longitudinal play, lateral play, roll angle, first swivel angle, and second swivel angle.

[0049] The longitudinal range refers to the maximum possible translational displacement of the first instrument along the instrument's longitudinal axis (in both directions) relative to the first pose of the first instrument, and takes into account that the first instrument can be inserted deeper into the incision (or trocar) or withdrawn further from it to a certain extent.

[0050] The transverse play refers to the maximum possible translational displacement perpendicular to the instrument's longitudinal axis relative to the first pose of the first instrument and takes into account that the first instrument (including the trocar and the instrument pivot point) can be displaced to a certain extent perpendicular to the instrument's longitudinal axis due to the compliance (deformability) of the patient's body part.

[0051] The roll angle describes the maximum possible rotation of the first instrument (in both directions) around the instrument's longitudinal axis relative to the first pose of the first instrument.

[0052] The first swivel angle and the second swivel angle each describe the maximum possible rotation of the first instrument around the first swivel axis and the second swivel axis (in both directions) relative to the first pose of the first instrument, wherein the two swivel axes are arranged perpendicular to each other and to the longitudinal axis of the instrument and intersect at the instrument fulcrum point.

[0053] Control commands are defined as digital signals used to selectively influence the pose (position and orientation) of the 3D model of the first instrument (or the function of the working element of the 3D model of the laparoscopic first instrument). For example, a position command aims to change the spatial position, while an orientation command aims to adjust the orientation (alignment) of the 3D model of the first instrument. A control command can comprise a single instruction or a sequence of instructions generated by an input unit and transmitted to the control unit.

[0054] The formulation that the virtual spatial movement of the 3D model of the first instrument is limited by the maximum mobility of the first instrument as represented in the mobility model expresses that only those virtual movements of the 3D model of the first instrument that can actually be implemented by the first instrument in reality are determined and form the basis for the simulation. By limiting the virtual movement of the 3D model of the first instrument to the maximum values ​​defined in the mobility model, it is therefore possible to prevent the surgeon from being presented with a surgical procedure suggestion through the simulation that cannot be implemented in reality.

[0055] A virtual simulation is a computer-generated visual representation of a virtual 3D scene from the perspective determined by the pose and viewing direction of the laparoscope. The representation is based on a global coordinate system. The virtual simulation depicts the surgical field as it appears from the laparoscope's point of view, thus simulating the surgeon's field of vision at the given camera position for the representation of the virtual spatial movement of the 3D model of the first instrument.

[0056] A superimposed image is a computer-generated representation created by combining the (real-time) laparoscopy image with the virtual simulation. The laparoscopy image and the simulation can, in particular, overlap or be displayed side by side.

[0057] The invention is claimed and described in connection with the use of a first instrument. It is immediately apparent to a person skilled in the art that the invention can also be realized using several such instruments by storing an associated mobility model for each instrument, creating a 3D model, and generating a virtual spatial movement of the 3D model of the respective instrument, which then becomes part of the simulation and overlay representation.

[0058] According to a preferred embodiment of the laparoscopy system according to the invention, the control unit is configured to - to receive the laparoscopy image captured by the laparoscope, wherein the laparoscopy image includes the (at least partial) representation of the first instrument and a (at least partial) representation of an anatomical structure of the body part, - to create a virtual 3D model of the anatomical object depicted in the laparoscopy image using the representation of the anatomical object in the laparoscopy image, - to determine a spatial first pose of the 3D model of the anatomical object using the spatial pose and viewing direction of the laparoscope and the representation of the anatomical object in the laparoscopy image, - to create the first virtual 3D scene in which the virtual 3D models of the first instrument and the anatomical object are arranged (represented) in the coordinate system, taking into account their respective spatial first poses, - to provide a deformation model of the anatomical object, which depicts the deformability of the 3D model of the anatomical object through a virtual interaction with the 3D model of the first instrument, - to determine a virtual deformation of the 3D model of the anatomical object using the specific virtual movement of the first instrument and the deformation model, and - to create the second virtual 3D scene, in which the virtual spatial movement of the 3D model of the first instrument and the virtual deformation of the 3D model of the anatomical object are depicted.

[0059] In this way, not only is the movement of the first instrument virtually simulated as part of the simulation display (as well as the superimposition display), but also the deformation of the anatomical object resulting from the movement of the first instrument or the interaction of the first instrument with the anatomical object.

[0060] An anatomical object refers to a biological structure present in a patient that is relevant in a medical or surgical context. This includes, in particular, organs, parts of organs, tissue sections, blood vessels, nerve pathways, cartilage or bone structures, as well as other morphologically distinguishable units of the patient's body.

[0061] The procedures for generating a virtual 3D model of an anatomical structure depicted in an image, the procedures for determining the spatial pose of the 3D model of the anatomical object, the procedures for providing a deformation model of the anatomical object, and the procedures for determining a virtual deformation of the 3D model of the anatomical object using the deformation model and the virtual movement of the first instrument are, considered individually, each already known from the prior art and belong to general technical knowledge. Various approaches, ranging from stereo reconstruction with dual-camera laparoscopes to monocular depth estimation with deep learning models, have been proposed for this purpose.

[0062] Against this background, a detailed explanation of the procedure is omitted here, and reference is made to the following publications, which address or describe such approaches and procedures in more detail: Yang, S., Li, Q., Shen, D., Gong, B., Dou, Q., & Jin, Y. (2024, October). Deform3dgs: Flexible deformation for fast surgical scene reconstruction with gaussian splatting. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 132-142). Cham: Springer Nature Switzerland; Huang, Y., Cui, B., Bai, L., Guo, Z., Xu, M., Islam, M., & Ren, H. (2024, October). Endo-4dgs: Endoscopic monocular scene reconstruction with 4d gaussian splatting. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 197-207). Cham: Springer Nature Switzerland.; Song, J., Wang, J., Zhao, L., Huang, S., & Dissanayake, G. (2018).Mis-slam: Real-time large-scale dense deformable slam system in minimal invasive surgery based on heterogeneous computing. IEEE Robotics and Automation Letters, 3(4), 4068-4075.; Wang, Y., Long, Y., Fan, S. H., & Dou, Q. (2022, September). Neural rendering for stereo 3d reconstruction of deformable tissues in robotic surgery. In International conference on medical image computing and computer-assisted intervention (pp. 431-441). Cham: Springer Nature Switzerland.; Zhu, L., Wang, Z., Cui, J., Jin, Z., Lin, G., & Yu, L. (2024, October). EndoGS: deformable endoscopic tissues reconstruction with gaussian splatting. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 135-145). Cham: Springer Nature Switzerland.; Liu, H., Liu, Y., Li, C., Li, W., & Yuan, Y. (2024, October). Lgs: A light-weight 4d gaussian splatting for efficient surgical scene reconstruction.In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 660-670). Cham: Springer Nature Switzerland.; Li, C., Feng, B. Y., Liu, Y., Liu, H., Wang, C., Yu, W., & Yuan, Y. (2024, October). Endosparse: Real-time sparse view synthesis of endoscopic scenes using gaussian splatting. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 252-262). Cham: Springer Nature Switzerland.; Hayoz, M., Hahne, C., Kurmann, T., Allan, M., Beldi, G., Candinas, D., ... & Sznitman, R. (2024, October). Online 3D reconstruction and dense tracking in endoscopic videos. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 444-454). Cham: Springer Nature Switzerland.; Shu, H., Liu, M., Seenivasan, L., Gu, S., Ku, P. C., Knopf, J., ... & Unberath, M. (2025). Seamless augmented reality integration in arthroscopy: a pipeline for articular reconstruction and guidance.Healthcare Technology Letters, 12(1), e12119.; Gerats, B. G., Wolterink, J. M., Mol, S. P., & Broeders, I. A. (2024). Neural fields for 3D tracking of anatomy and surgical instruments in monocular laparoscopic video clips. Healthcare Technology Letters, 11(6), 411-417.; Tang, X., Tao, H., Qian, Y., Yang, J., Feng, Z., & Wang, Q. (2024). Real-time deformable SLAM with geometrically adapted template for dynamic monocular laparoscopic scenes. International Journal of Computer Assisted Radiology and Surgery, 19(7), 1375-1383.; Wang, E., Liu, Y., Tu, P., Taylor, Z. A., & Chen, X. (2024). Video-based soft tissue deformation tracking for laparoscopic augmented reality-based navigation in kidney surgery. IEEE Transactions on Medical Imaging.

[0063] A deformation model is a computer-aided simulation of the deformability of an anatomical structure as a result of mechanical interaction with a surgical instrument. The deformation model depicts how the geometric shape and / or position of the structure changes under the influence of external forces—for example, pressure, tension, or displacement exerted by a laparoscopic surgical instrument. The model can be physically based (e.g., using elastic, viscoelastic, or plastic material properties), empirically approximated, and / or based on data from preoperative imaging and intraoperative sensors. It is particularly useful for simulating tissue reactions and visualizing dynamic anatomical changes within a virtual environment.

[0064] According to a further preferred embodiment of the laparoscopy system according to the invention, this comprises - a remote unit connected to the control unit, wherein - the remote unit includes a remote display unit and a remote control unit, - the remote display unit is set up to display the overlay, and - the remote control unit is set up to receive control movements of an operator relating to the first instrument and to transmit them as control commands to the control unit.

[0065] The operator, an expert, can thus follow the surgeon's intervention in real time via the remote display unit and input control commands via the remote control unit, which are then processed into the virtual movement of the 3D model of the first instrument. The virtual movement of the 3D model of the first instrument, as part of the overlay display, can now be shown simultaneously to both the expert at the remote unit and the surgeon operating the laparoscope and the first instrument, serving as a guide for the surgeon.

[0066] The surgeon and patient, the laparoscope and the first instrument on the one hand, and the expert and the remote unit on the other, can be located either in the same room or in separate rooms. The remote unit is connected to the control unit via a data connection, in particular via the internet.

[0067] The expert at the remote unit also receives real-time feedback on the virtual movement of the 3D model of the first instrument. This feedback indicates that the first instrument can only assume poses within the workspace defined by the mobility model, around its initial position. The expert thus receives real-time feedback that a desired movement sequence is not feasible because it would require moving the first instrument outside the workspace defined by the mobility model. Based on this feedback, the expert can initiate an alternative surgical procedure using appropriate control commands and present this to the surgeon as a guideline or suggestion.

[0068] According to a further preferred embodiment of the invention, the laparoscopy system comprises - a (computer-implemented) planning unit, where - the control unit is set up to receive the control commands for the 3D model of the first instrument from the planning unit, and - the planning unit includes a planning model which is set up to generate the control commands for the 3D model of the first instrument based on the provided first 3D scenery.

[0069] A planning model is a computer-implemented model that, based on a virtual 3D scene, proposes a movement for a surgical instrument, creates the associated control commands, and outputs them to the control unit.

[0070] The planning model can be designed, in particular, as an artificial intelligence (AI) model based on a trained algorithm that has been trained by analyzing and evaluating movement data of surgical instruments used by human surgeons in order to replicate typical, precise, and safe movement patterns. Such an AI model can, for example, include an artificial neural network (e.g., a convolutional neural network or a recurrent neural network) or a reinforcement learning-based network.

[0071] According to a further preferred embodiment of the invention, the longitudinal play in the mobility model is larger, in particular 2, 5 or 10 times larger, than the transverse play.

[0072] The invention is further manifested in the inventive method for displaying a superimposed representation, in which a laparoscopy image is superimposed with a virtual simulation representation, on a display unit of a laparoscopy system, wherein the laparoscopy system comprises - a laparoscope that can be inserted into a part of a patient's body, whose spatial pose and viewing direction can be determined, suitable for capturing the laparoscopy image, - a first laparoscopic instrument that can be inserted into the patient's body part through an instrument pivot point (fulcrum point) and extends along a longitudinal instrument axis, - the control unit, suitable for receiving the captured laparoscopy image, for providing the specific spatial pose and the viewing direction of the laparoscope, suitable for generating a virtual simulation display and suitable for outputting a superimposed display in which the laparoscopy image is superimposed with the simulation display, and - a display unit suitable for displaying the overlay representation, with the following steps: A) Providing the specific spatial pose and viewing direction of the laparoscope in a (global) coordinate system in the control unit, B) Receiving the laparoscopy image captured by the laparoscope in the control unit, wherein the laparoscopy image includes (at least a partial) representation of the first instrument, C) Providing a virtual 3D model of the first instrument in the control unit, D) Determining the first spatial pose of the first instrument in the coordinate system and the instrument's fulcrum point using the spatial pose and viewing direction of the laparoscope and the representation of the first instrument in the laparoscopy image by the control unit, E) The control unit generates a first virtual 3D scene in which the virtual 3D model of the first instrument is arranged (represented) in the coordinate system, taking into account its specific spatial first poses. F) Providing a mobility model of the first instrument (by defining three translational and three rotational degree-of-freedom parameters) in the control unit, which determines the maximum mobility of the first instrument in the body part relative to its specified first pose, in particular such that the first instrument has maximum mobility relative to its first pose - is translationally movable along the longitudinal axis of the instrument, and the longitudinal play is in particular between 5 and 25 cm, - to allow translational movement across the instrument's longitudinal axis, and the transverse movement is in particular 2 cm, 1 cm or 0.5 cm, - is rotatable by a roll angle around the instrument's longitudinal axis, and the roll angle is in particular 360°, - is rotatable about a first pivot angle about a first pivot axis and about a second pivot angle about a second pivot axis, wherein the two pivot axes are arranged perpendicular to each other and to the longitudinal axis of the instrument and intersect at the instrument pivot point (fulcrum point), and the first and second pivot angles are each in particular 90°, 60°, 45° or 30°, G) Receiving control commands for the 3D model of the first instrument in the control unit, H) Determining a virtual spatial movement of the 3D model of the first instrument using the received control commands and the mobility model of the first instrument by the control unit, wherein the virtual spatial movement of the 3D model of the first instrument is limited by the maximum mobility of the first instrument determined in the mobility model, I) The control unit generates a second virtual 3D scene in which the virtual spatial movement of the 3D model of the first instrument is represented. J) Generating the virtual simulation display by providing a virtual camera view of the first 3D scene and / or the second 3D scene from the perspective of the laparoscope using the specific spatial pose and viewing direction of the laparoscope by the control unit, K) Generating a superimposed display in which the laparoscopy image is superimposed with the virtual simulation display by the control unit, and L) Displaying the overlay on the display unit.

[0073] The individual procedural steps do not necessarily have to be carried out in the order implied by the alphabetical naming.

[0074] Furthermore, the invention manifests itself in a computer program product comprising commands, the execution of which in a control unit of a laparoscopy system according to one of claims 1 to 5 causes the control unit to execute steps A) to K) of the method according to the invention.

[0075] An embodiment of the invention will now be explained in more detail with reference to the drawing. The drawing shows Fig. 1 a schematic representation of a laparoscopy system according to the invention, Fig. 2 schematic representations of a laparoscopy image, a first virtual 3D scene, a virtual simulation representation and a superimposed representation, Fig. 3 a schematic representation of the process of the method according to the invention, and Fig. 4 A schematic illustration of the mobility model.

[0076] Fig. Figure 1 shows a laparoscopy system 1 with a laparoscope 2, a first laparoscopic instrument 3, a control unit 4, a display unit 5 and a remote unit 6.

[0077] The laparoscope 2, the first laparoscopic instrument 3, the control unit 4 and the display unit 5 are located in an operating room 8 together with a patient 7 and a (not shown) surgeon, the remote unit 6 is located in a remote room 9 spatially separate from the operating room 8 together with an operator, also called an expert, the control unit 4 and the remote unit 6 are interconnected via the Internet.

[0078] The laparoscope 2 is inserted through a trocar 10 into a body part 11 of the patient 7. The spatial pose and viewing direction of the laparoscope 2 can be determined, and the corresponding information can be provided to the control unit 4. The laparoscope 2 has a camera unit and is designed to capture a (real-time) laparoscopy image 12 and transmit it to the control unit 4.

[0079] The first laparoscopic instrument 3 is designed as a grasping instrument and is inserted through a further trocar 10 into the body part 11 of the patient 7. The laparoscopic instrument 3 has a working element 13 designed as a grasper, a control element 14, and an elongated shaft 15 that connects the working element 13 and the control element 14 and defines the longitudinal axis 16 of the instrument. The instrument pivot point 17 (fulcrum point) is located on the longitudinal axis of the instrument.

[0080] The surgeon can move, operate and / or guide the first instrument 3 and the laparoscope 2 directly, either manually or by means of a (not shown) surgical robot.

[0081] The control unit 4 is operatively connected to the laparoscope 2 and the display unit 5.

[0082] The remote unit 6 includes a remote control unit 18 and a remote display unit 19. The remote display unit 19, like the display unit 5 located in operating room 8, is designed as a screen and configured to display visual information.

[0083] The remote control unit 18 has control controllers 20, by means of which control movements of the operator (i.e. the expert) can be recorded and transmitted as control commands to the control unit 4.

[0084] The invention will be described below with reference to the schematic representations according to Fig. 2 and Fig. 4 based on the exemplary sequence of the inventive method according to Fig. 3 will be explained: First, according to step A, the specific spatial pose and viewing direction of the laparoscope 2 are provided in a global coordinate system in the control unit 4. For this purpose, the spatial pose and viewing direction of the laparoscope 2 can be determined, for example, as is known in the prior art, using a laparoscope marker and an associated tracking system.

[0085] According to step B, the laparoscopy image 12 of the surgical area, acquired by the laparoscope 2, is received in the control unit 4. The laparoscopy image 12 includes a (at least partial) representation of the first instrument 3 and a (at least partial) representation of an anatomical object 21 of body part 11.

[0086] In control unit 4, a virtual 3D model 3M of the first instrument 3 is stored and thus made available there according to step C. In addition, a virtual 3D model 21M of the anatomical object 21 depicted in the laparoscope image 12 is created. For this, as already mentioned above, methods described in the prior art can be used.

[0087] In step D, a first spatial pose P1.3 of the first instrument 3 and the associated instrument pivot point 17 in the global coordinate system are determined by the control unit 4 using known methods, based on the spatial pose and the viewing direction of the laparoscope 2 and the representation of the first instrument 3 in the laparoscopy image 12.

[0088] In addition, a first spatial pose P1.21 of the 3D model 21M of the anatomical object 21 is determined by the control unit 4 using known methods, taking into account the spatial pose and the viewing direction of the laparoscope 2 and the representation of the anatomical object 21 in the laparoscopy image 12.

[0089] Then, in step E, a first virtual 3D scene is created (see below). Fig. 2) generated by the control unit 4, in which the virtual 3D models of the first instrument and the anatomical object 3M, 21M are arranged in the global coordinate system taking into account their respective spatial first poses P1.3, P1.21.

[0090] According to step F, a mobility model 22 of the first instrument 3 is stored in the control unit (together with the virtual 3D model of the first instrument) and thus made available there. The mobility model 22 determines the maximum mobility of the first instrument 3 in body part 11 relative to the defined first pose of the first instrument 3. For this purpose, the mobility model 22 defines three translational and three rotational degree-of-freedom parameters by which the first instrument 3 can be moved to its maximum extent relative to its first pose.

[0091] Mobility model 22 defines that the first instrument 3 is maximally relative to its first pose. - is translationally movable by a longitudinal play L along the instrument's longitudinal axis 16, and the longitudinal play is in particular 10 cm, - so that a transverse play Q is transversely movable to the longitudinal axis of the instrument 16, and the transverse play Q is in particular 1cm, - is rotatable by a roll angle R about the instrument's longitudinal axis, and the roll angle is in particular 360°, - is rotatable about a first swivel angle WS1 about a first swivel axis S1 and about a second swivel angle WS2 about a second swivel axis S2, wherein the two swivel axes S1, S2 are arranged perpendicular to each other and to the longitudinal axis 16 of the instrument and intersect at the instrument pivot point 17 (fulcrum point), and the first and the second swivel angles WS1, WS2 are each in particular 60°.

[0092] Steps G to L essentially run in real time and continuously.

[0093] According to step J, the control unit 4 generates a virtual simulation representation by providing a virtual camera view of the first 3D scene from the perspective of the laparoscope 2 using the specific spatial pose and viewing direction of the laparoscope 3.

[0094] According to step K, the control unit 4 generates a superimposed display 23 in which the laparoscopy image 12 is superimposed with the virtual simulation display. The superimposed display 23 thus shows both the first instrument 3 and the anatomical structure 21 as depicted in the laparoscopy image 12, as well as the virtual 3D model of the first instrument and the anatomical structure 3M, 21M. According to step L, this superimposed display 23 is shown on the display unit 5 and the remote display unit 19.

[0095] Based on the overlay representation 23, the expert at the remote unit now moves the control controllers 20 of the remote control unit 18 to cause the virtual 3D model of the first instrument 3M to move as desired, which should serve as a suggestion or inspiration for the operator to proceed.

[0096] The corresponding control commands are received in control unit 4 according to step G.

[0097] According to step H, a virtual spatial movement of the 3D model of the first instrument 3M is now determined by the control unit 4 using the received control commands and the mobility model 22 of the first instrument 3. It is essential that the virtual spatial movement of the 3D model of the first instrument 3M is limited by the maximum mobility of the first instrument 3 determined in the mobility model 22.

[0098] If, for example, the lateral play is limited to 1 cm according to mobility model 22, then this maximum lateral play value of 1 cm will not be exceeded in the virtual movement of the 3D model of the first instrument, even if the corresponding control command aims for a larger lateral displacement of the first instrument 3 by, for example, 3 cm. The working range of the first instrument defined by mobility model 22 can therefore not be left within the scope of the virtual movement of the first instrument.

[0099] In real time, according to step I, a second virtual 3D scene is generated by control unit 4, in which the virtual spatial movement of the 3D model of the first instrument 3M is represented. During the movement of the 3D model of the first instrument, the 3D model of the first instrument assumes a second pose P2.3, which differs from the first pose P1.3.

[0100] This second virtual 3D scene is now incorporated in real time, according to steps J, K, and L, into the generated simulation display and the overlay display 23, which is shown on the display unit 5 and the remote display unit 19. The virtual simulation display can, for example, be designed as a virtual simulation video that first shows the first 3D scene and then the second 3D scene.

[0101] The expert at the remote unit receives real-time feedback via the overlay display 23 as to whether their proposed movement of the 3D model of the first instrument is feasible in practice, given the mechanical and kinematic constraints depicted in the mobility model 22. If not, the expert can immediately "try out" and propose an alternative movement. In this way, the surgeon only receives suggestions for the movement of the first instrument 3 that can be implemented in practice without injuring the patient 7.

[0102] In the illustrated embodiment, the generation of the second virtual 3D scene also takes into account a deformation of the 3D model of the anatomical object 21M. For this purpose, a deformation model of the anatomical object 21 is provided in the control unit, which depicts the deformability of the 3D model of the anatomical object 21M through a virtual interaction with the 3D model of the first instrument 3M. The creation of such a deformation model is part of the prior art.

[0103] Using the deformation model and the virtual movement of the first instrument, the virtual deformation of the organic object is determined and the correspondingly deformed virtual 3D model of the organic object 21M2 is displayed in the second 3D scene and thus in the overlay representation 23. Reference symbol list 1 Laparoscopy system 2 Laparoscopes 3. First instrument: virtual 3D model 3M of the first instrument 4 Control unit 5 Display unit 6 remote unit 7 patients 8 Operating room 9 Remote Room 10 trocars 11 Body part 12 Laparoscopy image 13 Working element 14 Control element 15 shaft 16 Instrument longitudinal axis 17 Instrument pivot point 18 Remote control unit 19 Remote display unit 20 control controllers 21 anatomical object 2M virtual 3D model of the anatomical object M 21M2 Deformation of the virtual 3D model of the organic object 22 Mobility model 23 Overlay representation P1.3 First pose of the first instrument P2.3 second pose of the first instrument L Longitudinal game Q Cross play R roll angle WS1 swivel angle S1 first pivot axis WS2 second swivel angle S2 second pivot axis QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 9 298 884 B1

[0009] EP 3273854B1

[0039] Cited non-patent literature

[0000] Park, J., Hong, J., Yoon, J., Park, B., Choi, M. K., & Jung, H. (2024, October). Towards Precise Pose Estimation in Robotic Surgery: Introducing Occlusion-Aware Loss. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 639-648). Cham: Springer Nature Switzerland; Rai, U., Xu, H., & Giannarou, S. (2025). SurgPose: Generalisable Surgical Instrument Pose Estimation using Zero-Shot Learning and Stereo Vision arXiv preprint arXiv:2505.11439.; Yoshimura, M., Marinho, M. M., Harada, K., & Mitsuishi, M. (2020, May). Single-shot pose estimation of surgical robot instruments' shafts from monocular endoscopic images. In 2020 IEEE International Conference on Robotics and Automation (ICRA) (pp. 9960-9966). IEEE.; Li, X., Shen, L., Li, L., Mo, H., Wang, X., Ma, X., & Zheng, J. (2024, December). 3DGS-based Tracking of Articulated Instruments for Instruments-Tissue Interaction Estimation in Robot-assisted Minimally Invasive Surgery.In 2024 IEEE International Conference on Robotics and Biomimetics (ROBIO) (pp. 1670-1675). IEEE.; Li, X., Shen, L., Li, L., Mo, H., Wang, X., Ma, X., & Zheng, J. (2024). 3DGS-based Tracking of Articulated Instruments for Instruments-Tissue Interaction Estimation in Robot-assisted Minimally Invasive Surgery.

[0039] . Barragan, J. A., Zhang, J., Zhou, H., Munawar, A., & Kazanzides, P. (2024, May). Realistic data generation for 6d pose estimation of surgical instruments. In 2024 IEEE International Conference on Robotics and Automation (ICRA) (pp. 13347-13353). IEEE.

[0039] Lin Dong and G. Morel, „Robust trocar detection and localization during robot-assisted endoscopic surgery,“ 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, 2016, pp. 4109-4114, doi: 10.1109 / ICRA.2016.7487602

[0043] M. Feuerstein, T. Mussack, S. M. Heining and N. Navab, „Intraoperative Laparoscope Augmentation for Port Placement and Resection Planning in Minimally Invasive Liver Resection,“ in IEEE Transactions on Medical Imaging, vol. 27, no. 3, pp. 355-369, March 2008, doi: 10.1109 / TMI.2007.90732

[0043] Yang, S., Li, Q., Shen, D., Gong, B., Dou, Q., & Jin, Y. (2024, October). Deform3dgs: Flexible deformation for fast surgical scene reconstruction with gaussian splatting. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 132-142). Cham: Springer Nature Switzerland

[0062] Huang, Y., Cui, B., Bai, L., Guo, Z., Xu, M., Islam, M., & Ren, H. (2024, October). Endo-4dgs: Endoscopic monocular scene reconstruction with 4d gaussian splatting. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 197-207). Cham: Springer Nature Switzerland

[0062] Song, J., Wang, J., Zhao, L., Huang, S., & Dissanayake, G. (2018). Mis-slam: Real-time large-scale dense deformable slam system in minimal invasive surgery based on heterogeneous computing. IEEE Robotics and Automation Letters, 3(4), 4068-4075

[0062] Wang, Y., Long, Y., Fan, S. H., & Dou, Q. (2022, September). Neural rendering for stereo 3d reconstruction of deformable tissues in robotic surgery. In International conference on medical image computing and computer-assisted intervention (pp. 431-441). Cham: Springer Nature Switzerland.; Zhu, L., Wang, Z., Cui, J., Jin, Z., Lin, G., & Yu, L. (2024, October). EndoGS: deformable endoscopic tissues reconstruction with gaussian splatting. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 135-145). Cham: Springer Nature Switzerland

[0062] Liu, H., Liu, Y., Li, C., Li, W., & Yuan, Y. (2024, October). Lgs: A light-weight 4d gaussian splatting for efficient surgical scene reconstruction. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 660-670). Cham: Springer Nature Switzerland

[0062] Li, C., Feng, B. Y., Liu, Y., Liu, H., Wang, C., Yu, W., & Yuan, Y. (2024, October). Endosparse: Real-time sparse view synthesis of endoscopic scenes using gaussian splatting. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 252-262

[0062] . Cham: Springer Nature Switzerland

[0062] Hayoz, M., Hahne, C., Kurmann, T., Allan, M., Beldi, G., Candinas, D., ... & Sznitman, R. (2024, October). Online 3D reconstruction and dense tracking in endoscopic videos. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 444-454

[0062] Cham: Springer Nature Switzerland

[0062] Shu, H., Liu, M., Seenivasan, L., Gu, S., Ku, P. C., Knopf, J., ... & Unberath, M. (2025). Seamless augmented reality integration in arthroscopy: a pipeline for articular reconstruction and guidance. Healthcare Technology Letters, 12(1), e12119

[0062] Gerats, B. G., Wolterink, J. M., Mol, S. P., & Broeders, I. A. (2024). Neural fields for 3D tracking of anatomy and surgical instruments in monocular laparoscopic video clips. Healthcare Technology Letters, 11(6), 411-417

[0062] Tang, X., Tao, H., Qian, Y., Yang, J., Feng, Z., & Wang, Q. (2024). Real-time deformable SLAM with geometrically adapted template for dynamic monocular laparoscopic scenes. International Journal of Computer Assisted Radiology and Surgery, 19(7), 1375-1383

[0062] Wang, E., Liu, Y., Tu, P., Taylor, Z. A., & Chen, X. (2024). Video-based soft tissue deformation tracking for laparoscopic augmented reality-based navigation in kidney

[0062]

Claims

[1] Laparoscopy system (1) comprising - a laparoscope (2) that can be inserted into a body part (11) of a patient (7), the spatial pose and direction of view of which can be determined, suitable for capturing a laparoscopy image (12), - a first laparoscopic instrument (3) that can be inserted into the body part (11) of the patient (7) through an instrument pivot point (17) and extends along an instrument longitudinal axis (16), - a control unit (4), and - a display unit (5), the control unit (4) is designed to - to provide the specific spatial pose of the laparoscope (2) and the specific viewing direction of the laparoscope (2) in a coordinate system, - to receive the captured laparoscopy image (12), wherein the laparoscopy image (12) includes at least a partial representation of the first instrument (3), - to provide a virtual 3D model of the first instrument (3M), - to determine a first spatial pose (P1.3) of the first instrument (3) in the coordinate system and the instrument pivot point (17) of the first instrument (3) using the spatial pose and the viewing direction of the laparoscope (2) and the representation of the first instrument (3) in the laparoscopy image (12), - to create a first virtual 3D scene in which the virtual 3D model of the first instrument (3M) is arranged in the coordinate system taking into account its specific spatial first poses (P1.3), - to provide a mobility model (22) of the first instrument (3) which determines the maximum mobility of the first instrument (3) in the body part (11) relative to its first pose (P1.3), in particular such that the first instrument (3) is maximally mobile relative to its first pose (P1.3). - is translationally movable by a longitudinal play (L) along the instrument's longitudinal axis (16), and the longitudinal play (L) is in particular between 5 and 25 cm, - is translationally movable by a transverse clearance (Q) perpendicular to the longitudinal axis (16) of the instrument, and the transverse clearance (Q) is in particular 2 cm, 1 cm or 0.5 cm, - is rotatable about a roll angle (R) about the instrument's longitudinal axis (16), and the roll angle (R) is in particular 360°, - is rotatable about a first swivel angle (WS1) about a first swivel axis (S1) and about a second swivel angle (WS2) about a second swivel axis (S2), wherein the two swivel axes (S1, S2) are arranged perpendicular to each other and to the longitudinal axis (16) of the instrument and intersect at the instrument pivot point (17), and the first and second swivel angles (WS1, WS2) are each in particular 90°, 60°, 45° or 30°, - To receive control commands for the 3D model of the first instrument (3M), - to determine a virtual spatial movement of the 3D model of the first instrument (3M) using the received control commands and the mobility model (22) of the first instrument, wherein the virtual spatial movement of the 3D model (3M) of the first instrument is limited by the maximum mobility of the first instrument determined in the mobility model (22), - to create a second virtual 3D scene in which the virtual spatial movement of the 3D model of the first instrument (3M) is represented in the coordinate system, - to create a virtual simulation representation in which a virtual camera view of the first and / or the second 3D scene is shown from the perspective of the laparoscope (2) using the spatial pose and the viewing direction of the laparoscope (2), - to generate a superimposed display (23) in which the laparoscopy image (12) is superimposed with the virtual simulation display, and the display unit (5) is set up to display the superimposed display (23). [2] Laparoscopy system (1) according to claim 1, wherein the control unit (4) is configured to - to receive the laparoscopy image (12) captured by the laparoscope (2), wherein the laparoscopy image (12) includes at least a partial representation of the first instrument (3) and at least a partial representation of an anatomical object (21) of the body part (11), - to create a virtual 3D model of the anatomical object (21M) shown in the laparoscopy image using the representation of the anatomical object (21) in the laparoscopy image (12), - to determine a spatial first pose (P1.3) of the 3D model of the anatomical object (21M) using the spatial pose and viewing direction of the laparoscope (2) and the representation of the anatomical object (21) in the laparoscopy image (12), - to create the first virtual 3D scene in which the virtual 3D models of the first instrument (3M) and the anatomical object (21M) are arranged in the coordinate system taking into account their respective spatial first poses, - to provide a deformation model of the anatomical object that depicts the deformability of the 3D model of the anatomical object (21M) through a virtual interaction with the 3D model of the first instrument (3M), - to determine a virtual deformation of the 3D model of the anatomical object (21M) using the specified virtual movement of the first instrument and the deformation model, and - to create the second virtual 3D scene in which the virtual spatial movement of the 3D model of the first instrument (3M) and the virtual deformation of the 3D model of the anatomical object (21M2) are represented. [3] Laparoscopy system (1) according to any one of the preceding claims, comprising - a remote unit (6) connected to the control unit (4), where - the remote unit (6) comprises a remote display unit (19) and a remote control unit (18), - the remote display unit (19) is configured to display the overlay display (23), and - the remote control unit (18) is configured to receive control movements of an operator relating to the first instrument (3) and to transmit them as control commands to the control unit (4). [4] Laparoscopy system (1) according to any one of the preceding claims, comprising - a planning unit, where - the control unit is set up to receive the control commands for the 3D model of the first instrument from the planning unit, and - the planning unit includes a planning model which is set up to generate the control commands for the 3D model of the first instrument based on the provided first 3D scenery. [5] Laparoscopy system (1) according to one of the preceding claims, wherein the longitudinal play (L) in the mobility model (22) is larger, in particular 2, 5 or 10 times larger, than the transverse play (Q). [6] Method for displaying a superimposed representation (23) in which a laparoscopy image (12) is superimposed with a virtual simulation representation on a display unit (5) of a laparoscopy system (1), wherein the laparoscopy system (1) comprises - a laparoscope (2) that can be inserted into a body part (11) of a patient (7), the spatial pose and direction of view of which can be determined, suitable for capturing the laparoscopy image (12), - a first laparoscopic instrument (3) that can be inserted into the body part (11) of the patient (7) through an instrument pivot point (17) and extends along an instrument longitudinal axis (16), - the control unit (4), suitable for receiving the captured laparoscopy image (12), for providing the specific spatial pose and the viewing direction of the laparoscope (2), suitable for generating a virtual simulation display and suitable for outputting a superimposed display (23) in which the laparoscopy image (12) is superimposed with the simulation display, and - a display unit (5), suitable for displaying the overlay representation (23), comprising the following steps: A) Providing the specific spatial pose of the laparoscope (2) and the specific viewing direction of the laparoscope (2) in a coordinate system in the control unit (4), B) Receiving the laparoscopy image (12) captured by the laparoscope (2) in the control unit (4), wherein the laparoscopy image (12) includes at least a partial representation of the first instrument (3), C) Providing a virtual 3D model of the first instrument (3M) in the control unit (4), D) Determining the first spatial pose (P1.3) of the first instrument (3) in the coordinate system and the instrument pivot point (17) using the spatial pose and the viewing direction of the laparoscope (2) and the representation of the first instrument (3) in the laparoscopy image (12) by the control unit (4), E) The control unit (4) generates a first virtual 3D scene in which the virtual 3D model of the first instrument (3M) is arranged in the coordinate system taking into account its specific spatial first poses (P1.3), F) Providing a mobility model (22) of the first instrument (3) in the control unit which determines the maximum mobility of the first instrument (3) in the body part (11) relative to its specified first pose (P1.3), in particular such that the first instrument (3) has maximum mobility relative to its first pose - is translationally movable by a longitudinal play (L) along the longitudinal axis of the instrument, and the longitudinal play (L) is in particular between 5 and 25 cm, - is translationally movable by a transverse play (Q) transverse to the longitudinal axis of the instrument, and the transverse play (Q) is in particular 2 cm, 1 cm or 0.5 cm, - is rotatable about a roll angle (R) about the instrument's longitudinal axis (16), and the roll angle (R) is in particular 360°, - is rotatable about a first swivel angle (WS1) about a first swivel axis (S1) and about a second swivel angle (WS2) about a second swivel axis (S2), wherein the two swivel axes (S1, S2) are arranged perpendicular to each other and to the longitudinal axis (16) of the instrument and intersect at the instrument pivot point (17), and the first and second swivel angles (WS1, WS2) are each in particular 90°, 60°, 45° or 30°, G) Receiving control commands for the 3D model of the first instrument (3M) in the control unit (4), H) Determining a virtual spatial movement of the 3D model of the first instrument (3M) using the received control commands and the mobility model (22) of the first instrument (3) by the control unit (4), wherein the virtual spatial movement of the 3D model of the first instrument (3M) is limited by the maximum mobility of the first instrument (3) determined in the mobility model (22), I) The control unit (4) generates a second virtual 3D scene in which the virtual spatial movement of the 3D model of the first instrument (3M) is represented. J) Generating the virtual simulation display by providing a virtual camera view of the first 3D scene and / or the second 3D scene from the perspective of the laparoscope (2) using the specific spatial pose and viewing direction of the laparoscope (2) by the control unit (4), K) Generating the overlay display (23) in which the laparoscopy image (12) is superimposed with the virtual simulation display by the control unit (4), and L) Displaying the overlay representation (23) on the display unit (5). [7] Computer program product comprising instructions whose execution in a control unit (4) of a laparoscopy system (1) according to any one of claims 1 to 5 causes the control unit (4) to perform steps A) to K) of claim 6.

Images