A computer-implemented method for surgical assistance

Abstract

The present disclosure generally relates to a computer-implemented method for surgical assistance, specifically adapted to allow display of a captured image that is augmented with an overlayed visualization of an internal body portion (302) of a patient The method is performed by a computing system (100) comprising a processing unit (110), and uses data acquired by an imaging modality (130) and an image capturing device (140), in conjunction with a marker unit (200) that is both optically detectable and radiopaque, and physically related to the internal body portion (302). The present disclosure also relates to a corresponding computer system (100) and computer program product.

Description

[0001] A COMPUTER-IMPLEMENTED METHOD FOR SURGICAL ASSISTANCE

[0002] TECHNICAL FIELD

[0003] The present disclosure generally relates to a computer-implemented method for surgical assistance, specifically adapted to allow display of a captured image that is augmented with an overlayed visualization of an internal body portion of a patient. The present disclosure also relates to a corresponding computer system and computer program product.

[0004] BACKGROUND

[0005] Minimally invasive surgery (MIS) has emerged as a preferred technique due to its advantages in reducing patient recovery time, minimizing surgical trauma, and lowering the risk of infection. Among the various technologies enabling MIS, laparoscopic surgery stands out, offering the ability to perform complex procedures through small incisions with the assistance of a camera and specialized instruments. Despite its advantages, laparoscopic surgery presents unique challenges, primarily due to the surgeon’s reliance on two- dimensional video feeds to navigate the three-dimensional internal anatomy of the patient. Such a limitation may result in difficulties of spatial orientation, identification of specific anatomical structures, and the precise manipulation of surgical tools.

[0006] To address these challenges, augmented reality (AR) technologies have been introduced to the surgical field, aiming to enhance the surgeon’s visual field by overlaying critical virtual information onto the real-world image. AR in surgery promises to merge the physical and digital worlds, providing real-time, three-dimensional navigational aids directly within the surgeon’s field of view. However, the practical application of AR in surgical procedures demands high accuracy in the alignment and tracking of virtual overlays with the patient’s anatomy, which remains a significant technical hurdle.

[0007] One exemplary system attempting to solve the challenges associated with AR in surgery is presented in US9646423B1, which introduces an AR system for MIS that leverages pre-operative image data. The system presented in US9646423B1 seeks to enhance surgical precision by merging pre-operative images with intra-operative views captured by an endoscope, aiming to provide real-time augmented reality overlays during surgical procedures. The solution as presented in US9646423B1 represents a great step towards integrating AR technology into MIS, facilitating a more informed surgical process by allowing surgeons to visualize structures that are not immediately apparent during surgery. However, the solution proposed in US9646423B1, necessitates significant computational resources for achieving the desired merging effect, and may encounter reliability issues in dynamically changing surgical environments. The reliance on the processing and registration of complex image data sets may also introduce latency and will as such not always offer the real-time accuracy required for certain surgical interventions. Furthermore, the method as presented in US9646423B1 relies on blood vessel -based feature detection for tracking the position and orientation of the endoscope, which may encounter difficulties in consistently identifying reliable anatomical landmarks. Accordingly, there is an apparent risk that such a dependency could affect the ability of the system to accurately correlate between pre-operative and intra-operative images, thus impacting the overall reliability of the augmented reality overlays during critical surgical tasks.

[0008] A further enhanced solution for associated with AR in surgery is presented in US2023355315A1, focusing on a multifunctional platform that is configured for surgical navigation, wherein input data from one or more hardware components is received and converted into an augmented reality (AR) or virtual reality (VR) experience that a remote user can utilize for performing at least some portions of a surgical procedure.

[0009] US2023355315A1 further presents the use of a reference marker as part of a real-time capture process, providing a spatial reference. However, also the solution presented in US2023355315A1 relies heavily on significant computational resources, specifically making use of a software system configured to orient medical image data to real-time image data using image data about the reference marker.

[0010] With the above in mind, there is an apparent desire to provide further enhancements for surgical assistance, specifically focusing on user-friendly AR-assisted navigation in surgical settings and for enhancing the capabilities of surgeons to perform complex operations with greater reliability and accuracy, without the limitations observed in currently available prior art solutions, such as exemplified by US9646423B1 and US2023355315A1.

[0011] SUMMARY

[0012] According to an aspect of the present disclosure, the above is at least partly alleviated by a computer-implemented method performed by a computing system comprising a processing unit, for generating an augmented visualization for surgical assistance, wherein the method comprises the steps of acquiring, using an imaging modality, data representing a three-dimensional model of an internal body portion of a patient, wherein the internal body portion is physically related with a marker unit, the data includes a representation of the marker unit, and the marker unit is optically detectable and radiopaque, generating, using the processing unit, a point cloud of the three-dimensional model representing a geometry of the internal body portion in coordinates defined for the computing system, correlating, using the processing unit and the positioned marker unit, the point cloud with a coordinate system of the imaging modality to align with the computing system’s coordinates, overlaying, using the processing unit, a visualization of the internal body portion over an image captured by an image capturing device in which the marker unit is visible, based on an estimation of a position of the image capturing device relative to the marker unit, and displaying, on at least one display device arranged in communication with the computing system, the captured image augmented with the overlayed visualization of the internal body portion of the patient.

[0013] The expression ’’three-dimensional model” should within the context of the present disclosure be understood to refer to a digital representation of the internal body portion of a patient, constructed using data acquired from an imaging modality. Such a model is a digital replica that embodies the precise geometrical and spatial characteristics of the anatomical area under examination. It serves as the basis for generating a ’’point cloud“, which is a collection of data points in space produced by the processing unit. These points represent the surface geometry of the internal body portion, converting the continuous form of human tissue into a three-dimensional model using an imaging modality, and then further into a discrete set of points. The transformation of coordinates from the imaging modality’s three-dimensional model into a point cloud in the computing system’s coordinate system based on the marker unit is used in subsequent computational steps, enabling the correlation with the computing system’s coordinates, and facilitating accurate augmented reality overlays.

[0014] The term “marker unit” as used within the context of the present disclosure refers to a device or marker designed to be physically related with an internal body portion of a patient, facilitating localization and tracking within the surgical field. Such a marker is defined by its dual properties, namely being optically detectable and radiopaque. The optically detectable feature enables the marker to be easily identified by imaging devices that rely on visual data, such as cameras used for augmented reality overlays, for example by providing the marker with geometrical figures comprising a plurality of machine-readable codes. Conversely, its radiopaque nature ensures that the marker is clearly visible under imaging modalities that employ radiographic techniques, such as CT scans. Such a dual functionality allows for use across various imaging platforms (i.e. the imaging modality and the image capturing device), enhancing the computer system’s ability to accurately correlate the physical position of the internal body portion with its digital representation.

[0015] By means of the present disclosure, it is made possible to enhance the accuracy and efficiency of minimally invasive surgeries (MIS), particularly the reliance on two-dimensional video feeds for navigating three-dimensional internal anatomy. Such an enhancement directly correlates to improvements in spatial orientation, the identification of specific anatomical structures, and the precise manipulation of surgical tools, which are critical challenges in current surgical practices.

[0016] These advancements are made possible by a novel method of acquiring a three-dimensional model of an internal body portion of a patient, physically related with an optically detectable and radiopaque marker unit. Such an approach not only streamlines the acquisition process but significantly improves the reliability and accuracy of the surgical assistance provided. Such an implementation also addresses the computational and real-time accuracy challenges associated with the alignment and tracking in currently available prior art, offering a practical and efficient alternative. The generation of a point cloud representing the geometry of the internal body portion in coordinates defined for the computing system, may further refine surgical precision, enabling a detailed and accurate representation of the surgical site.

[0017] Moreover, the technique of correlating the point cloud with the imaging modality’s coordinate system, facilitated by the physically related marker unit to align with the computing system’s coordinates, overcomes the above mentioned technical hurdles present with currently available prior art. Specifically, the mentioned correlation ensures a highly accurate alignment and tracking of virtual overlays with the patient’s anatomy, crucial for the practical application of augmented reality in surgical procedures. The use of the marker unit may as such circumvent the latency and reliability issues observed in the augmentation process of existing systems, enhancing the ability to perform complex procedures with heightened precision and confidence. Overlaying of a visualization of the internal body portion over an image captured by an image capturing device, based on the estimated position of the device relative to the marker unit, represents an advancement towards providing real-time, three-dimensional navigational aids directly within an operator’s field of view.

[0018] Displaying the captured image augmented with the visualization of the internal body portion on a display device in communication with the computing system further solidifies the method’s contribution to surgical assistance. Such a display mechanism may allow for an intuitive interaction with the augmented reality interface, significantly enhancing the surgical process’s overall efficiency and outcome.

[0019] Furthermore, it should be stressed that the solution according to the present disclosure offers technical advantages that go beyond the general use of augmented reality in surgical navigation, specifically addressing the computational constraints typically imposed by prior art systems relying on intensive post-processing or continuous image registration. By embedding the physical marker unit directly in both the imaging data and the live visual feed, and by using the marker unit to generate and align the point cloud in system coordinates, the present system achieves real-time overlay with reduced latency, improved robustness to patient or instrument movement, and minimal reliance on external tracking infrastructure. Such a solution accordingly contributes to a more efficient, reliable, and clinically viable surgical assistance system that performs well even in dynamic and resource-constrained environments, in contrast to available prior art solutions where the reliance on software-based matching places heavy demands on computational resources.

[0020] In some embodiments it may be desirable to allow the imaging modality to apply at least one of cone beam computed tomography (CBCT), ultrasound (US), computed tomography (CT), magnetic resonance imaging (MRI) technology adapted for acquiring the three-dimensional model of the internal body portion. Other similar technologies are of course possible and within the scope of the present disclosure.

[0021] The possibility of using different types of technologies allows for selecting the most suitable imaging technique based on the specific surgical needs, the anatomical area of interest, and the distinct characteristics of the tissue or structure being examined. For instance, MRI offers unparalleled soft tissue contrast, ideal for visualizing organs and neural structures, whereas CT and CBCT provide high-resolution images of bone and dense tissues.

[0022] The technical advantages of incorporating such a diverse range of imaging modalities are manifold. Firstly, such an approach enhances the precision of the three- dimensional model creation, directly influencing the accuracy of subsequent surgical planning and guidance. It further allows for tailoring the imaging process to the specific requirements of each surgery, optimizing visualization of the target area, and thereby facilitating more precise interventions. Still further, by providing alternatives that can circumvent limitations associated with a particular imaging technology the scheme according to the present disclosure enhances patient safety and improves the overall quality of surgical outcomes. In accordance with the present disclosure, it may in some embodiments be desirable to arrange the marker unit to comprise optically detectable markers and radiopaque markers. These markers are provided in a geometrical pattern that does not exhibit rotational symmetry, which allows for the rotational position of the marker unit to be determined. Additionally, the geometrical patterns of the optically detectable markers and the radiopaque markers have a fixed, predetermined correlation. Such a configuration permits an accurate identification of the marker unit’s position from all directions, using both medical imaging data acquired by the imaging modality and optical data captured by the image capturing device, since they use the same transformation between medical imaging modality three- dimensional model data and the suggested product’s coordinate system.

[0023] The use of radiopaque markers, in some embodiments defined as cylinders made of e.g. metal, may further enhance acquiring of the three-dimensional model of the internal body portion of the patient. One example of an exemplary metal is gold, where gold’s high visibility under radiographic imaging techniques ensures that these markers stand out in the imaging data.

[0024] During the acquisition process, specific voxels associated with the radiopaque markers are identified within the three-dimensional model. These voxels, as the smallest unit of digital imaging data generated by the imaging modality, are used for creating a binary volume that simplifies the three-dimensional model, specifically by emphasizing areas marked by the radiopaque markers. The creation of such a binary volume, by focusing on areas delineated by the radiopaque markers, provides a structured basis for transforming the continuous spatial data into the above discussed point cloud. The found radiopaque markers are shape matched with their expected pattern since their positions relative to each other are uniquely identifiable. The strength of such an implementation is that the shape matching results in a precise transformation between coordinates in the three-dimensional model of the imaging modality and the computing system’s coordinate system based on the marker unit’s visual markers.

[0025] Such a transformation facilitates the subsequent generation of the three- dimensional model of the segmented tissue, the one that the operator wants to have a virtual overlay of in the endoscopic camera view, into the point cloud. Specifically, identifying and classifying the voxels according to their associated radiopaque markers allows for an accurate representation of the internal body portion’s geometry relative to the positioned marker unit. The point cloud, derived from the binary volume, effectively captures the geometry of the internal body portion, facilitating its accurate alignment in relation to the computing system’s coordinates.

[0026] In addition to the above, the scheme according to the present disclosure encompasses the use of the mentioned image capturing device, for example implemented by means of a laparoscopic or endoscopic camera, which is inserted into the patient’s body through minimally invasive incisions and / or ports. The process is enhanced by utilizing the marker unit to ensure the alignment of the camera’s captured images with the pre-acquired three-dimensional model of the internal body portion. Thus, the integration of the camera within the surgical procedure is seamlessly achieved, enhancing the visualization of the internal anatomy. It should be stressed that other forms of image capturing devices may be used within the scope of the present disclosure, such as for example a bronchoscopic camera. Similar to a laparoscopic camera, a bronchoscopic camera is designed for internal examination but is specifically arranged for accessing and visualizing the airways and lungs.

[0027] Preferably, the scheme according to the present disclosure further comprises the step of applying, using the processing unit, a predefined algorithm for noise reduction on the three-dimensional model data before generating the point cloud, wherein the predefined algorithm is adapted for enhancing an accuracy of a visualization of the internal body portion. The noise reduction algorithm is preferably designed to address and mitigate the noise present in the data, which may originate from various sources, including the imaging modality itself or external environmental factors that impact the quality of imaging. By applying the selected algorithm, the data underlying the three-dimensional model is refined, laying a more accurate foundation for subsequent processes such as the identification of the correct data of the point cloud and the visualization of the internal body portion.

[0028] The integration of a noise reduction step into the process brings technical advantages to the present disclosure, for example providing for an improvement in the precision of the internal body portion’s visualization by ensuring that the generated point cloud is a more faithful representation of the actual anatomical structure, devoid of artifacts that might otherwise obscure critical details or mislead a team of operators, such as for example a surgical team. Consequently, operators may be provided with clearer, more accurate navigational aids, particularly vital in intricate surgical procedures where distinguishing between different tissues or structures is crucial. Furthermore, the improved data quality from such a step contributes to a more accurate alignment and tracking of augmented reality overlays with the patient’s anatomy, thus elevating the effectiveness and reliability of the surgical assistance offered by the system. Additionally, it may in some embodiments be desirable to arrange the scheme to comprise the step of applying, using the processing unit, a predefined augmented reality (AR) technique adapted to utilize the correlation between the marker unit’s position and the computing system’s coordinates to overlay the visualization accurately on the image captured using the image capturing device. Such an implementation is designed to make use of the established correlation between the marker unit’s position and the computing system’s coordinates. Its primary function is to ensure the accurate overlay of the internal body portion’s visualization onto the real-time image captured by the image capturing device. By aligning the virtual data (the visualization) with the real-world data (the captured image), the technique ensures that the augmented overlay precisely matches the actual location and orientation of the internal body portion as viewed through the image capturing device.

[0029] Incorporating such a predefined AR technique into the process provides substantial technical advantages. First, it significantly improves the ability to understand the spatial relationship between the surgical instruments and the internal body portion being operated on, thereby enhancing surgical precision, and reducing the likelihood of errors. The accurate overlay of virtual data on real-world images allows operators to navigate more confidently during procedures, especially when performing complex maneuvers or when visibility is limited. Furthermore, the technique’s dependency of the marker unit’s position for overlay accuracy simplifies the alignment process, making the AR assistance more robust and reliable even in dynamic surgical environments where the patient’s positioning might change, which in the end contributes to a more efficient surgical workflow and can potentially reduce operation times and improve patient outcomes.

[0030] Preferably, the step of displaying the captured image augmented with the overlayed visualization comprises adjusting, using the processing unit, a visualization opacity based on the estimated depth of the internal body portion from an external surface of the patient’s body. The adjustment may for example be based on the estimated depth of the internal body portion relative to the patient’s external surface. Such depth estimation allows for an improved representation of the visualization, wherein deeper structures may appear more translucent, enhancing the perception of depth in the augmented image. Such a technique may aid in providing a more intuitive understanding of the spatial relationships within the patient’s body by simulating a more realistic three-dimensional view on a two- dimensional display.

[0031] For operators, such as for example surgeons, such an inclusion may be seen as enriching the perceived image quality, offering a deeper insight into the anatomical layers during a procedure. The depth-enhanced visualization may also facilitate a more precise identification of structures, aiding in the avoidance of critical anatomy and in the execution of surgical tasks with higher precision. Moreover, by providing a depth-adjusted view, the system minimizes the cognitive load on operators, allowing them to focus more on the surgical technique rather than interpreting flat images.

[0032] Furthermore, on some embodiments of the present disclosure it may be desirable to additionally include the step of calculating, using the processing unit, a three- dimensional distance map between geometry of the internal body portion and the marker unit to facilitate AR alignment. Such a calculation may generally be performed by the processing unit and serves to accurately align augmented reality (AR) with the actual anatomical structures. The distance map may effectively quantify the spatial discrepancies between the virtual model (derived from the point cloud) and the physical location of the marker unit, providing a reference for adjusting AR overlays to match “real” coordinates.

[0033] The inclusion of a three-dimensional distance map calculation offers substantial technical advantages, mainly by elevating the precision of AR overlay alignment within the surgical field. Such precision is suitable for ensuring that virtual information accurately corresponds to the physical anatomy, thereby enhancing the operator’s spatial awareness and the accuracy of surgical interventions. Additionally, a more accurate AR alignment reduces the likelihood of surgical errors by ensuring that virtual guides or annotations are precisely superimposed over the relevant anatomical landmarks.

[0034] Still further, the scheme according to the present disclosure may also comprise the step of generating, using the processing unit, a virtual surgical plan that overlays navigational cues on the at least one display device, based on the augmented visualization of the internal body portion, for guiding surgical instruments during an associated surgical procedure. The cues may for example be dynamically aligned with the augmented visualization of the patient’s internal anatomy, providing a real-time guide for the precise manipulation and placement of surgical instruments. The process leverages augmented reality to transform abstract surgical planning into an intuitive, visually guided experience, directly overlaying crucial navigational information onto the operator’s view of the surgical field.

[0035] The inclusion of a virtual surgical plan presents numerous technical advantages, and foremost, it substantially reduces the cognitive load on operators by providing an intuitive, visual guide that aligns with their natural visual processing capabilities. Such an inclusion may not only accelerate the decision-making process during surgeries but also enhances precision by providing clear, contextual cues for instrument navigation. As a result, the risk of navigational errors may possibly be reduced, and the likelihood of achieving surgical objectives with greater accuracy and efficiency is increased.

[0036] Preferably, the overlaying of the visualization over the image captured by the image capturing device includes dynamically adjusting, using the processing unit, the visualization in real-time based on changes in the position and orientation of the image capturing device relative to the marker unit, for ensuring consistent alignment of the augmented reality overlay with the internal body portion. The processing unit may as such modify the visualization in real-time to reflect any changes in the position or orientation of the image capturing device relative to the marker unit, thereby ensuring that the AR overlay remains accurately aligned with the internal body portion being visualized, regardless of movements made by the operator / surgeon or adjustments to the camera’s position. The adaptation may generally be seamless, providing continuous and accurate alignment between the virtual and real-world views without manual recalibration.

[0037] The capacity for real-time dynamic adjustment of the AR overlay offers significant technical advantages, for example by ensuring that the augmented visualization remains precisely aligned with the actual anatomy of the patient, which is desirable for maintaining the accuracy of surgical guidance. Such an implementation may also contribute to reducing the likelihood of surgical errors by providing operators with consistently accurate visual cues.

[0038] It may in some embodiments be desirable to additionally (or optionally) implement a user interface on the display device for allowing an operator to interactively adjust parameters of the overlayed visualization to tailor the augmented reality view. Operators, such as surgeons, may as such be provided with the option to modify aspects such as the scale, orientation, and possibly the transparency of the virtual overlay directly from the interface. The inclusion of such a functionality may also grant operators more control over the visualization tools at their disposal, making the AR system not just a passive display tool but an interactive aid that can be fine-tuned in real-time during surgeries.

[0039] It may of course be possible, and within the scope of the present disclosure, to provide a separate user device, such as a tablet or a laptop, for presenting the user interface and / or a portion of the user interface. Such an implementation may allow for further improvements in the interaction between the present system and users of the system.

[0040] The implementation of an interactive user interface may allow for operators to modify the AR overlay in ways that might be most conducive to the task at hand, enhancing the practical utility of the AR system as a surgical guide. For instance, adjusting the opacity or contrast of the overlay could help in highlighting specific anatomical features or surgical landmarks, thus aiding in precision tasks such as the removal of a tumor or the navigation around critical structures. Moreover, the operator may add additional virtual objects to the set of visuals, both segmentations of patient tissues, e.g., blood vessels, bile ducts, and nearby organs, but also prerendered geometries of known sizes, e.g., rulers, cubes, and spheres. The operator may even draw virtual lines on specific planes correlated to the marker unit using a laparoscopic tool that is instantly added to the set of virtual objects on display, to further enhance the practical utility of the AR system. Furthermore, the ability to interact with the AR visualization on-the-fly promotes a more intuitive surgical process. Operators may adapt the visualization based on their real-time assessment of the surgical field, potentially reducing the cognitive load, and allowing for a more focused approach to the procedure.

[0041] According to another aspect of the present disclosure, there is provided a computer system for surgical assistance, the computer system comprising a processing unit, wherein the processing unit is adapted to acquire, using an imaging modality, data representing a three-dimensional model of an internal body portion of a patient, wherein the internal body portion is physically related with a marker unit, the data includes a representation of the marker unit, and the marker unit is optically detectable and radiopaque, generate a point cloud of the three-dimensional model representing a geometry of the internal body portion in coordinates defined for the computing system, correlate, using the positioned marker unit, the point cloud with a coordinate system of the imaging modality to align with the computing system’s coordinates, overlay a visualization of the internal body portion over an image captured by an image capturing device in which the marker unit is visible, based on an estimation of a position of the image capturing device relative to the marker unit, and display, on at least one display device arranged in communication with the computing system, the captured image augmented with the overlayed visualization of the internal body portion of the patient. Such an aspect of the present disclosure provides similar advantages as discussed above in relation to the previous aspects of the present disclosure.

[0042] According to a further aspect of the present disclosure, there is provided a computer program product comprising a non-transitory computer readable medium having stored thereon computer program means for operating a computing system designed for surgical assistance, the computing system comprising a processing unit, wherein the computer program product comprises code for acquiring, using an imaging modality, data representing a three-dimensional model of an internal body portion of a patient, wherein the internal body portion is physically related with a marker unit, the data includes a representation of the marker unit, and the marker unit is optically detectable and radiopaque, code for generating, using the processing unit, a point cloud of the three-dimensional model representing a geometry of the internal body portion in coordinates defined for the computing system, code for correlating, using the processing unit and the positioned marker unit, the point cloud with a coordinate system of the imaging modality to align with the computing system’s coordinates, code for overlaying, using the processing unit, a visualization of the internal body portion over an image captured by an image capturing device in which the marker unit is visible, based on an estimation of a position of the image capturing device relative to the marker unit, and code for displaying, on at least one display device arranged in communication with the computing system, the captured image augmented with the overlayed visualization of the internal body portion of the patient. Also, such an aspect of the present disclosure provides similar advantages as discussed above in relation to the previous aspects of the present disclosure.

[0043] A software executed by the processing unit for operation in accordance to the present disclosure may be stored on a computer readable medium, being any type of memory device, including one of a removable nonvolatile random access memory, a hard disk drive, a floppy disk, a CD-ROM, a DVD-ROM, a USB memory, an SD memory card, a solid state drive, other non-volatile flash based storage mediums, or a similar computer readable medium known in the art.

[0044] Further features of, and advantages with, the present disclosure will become apparent when studying the appended claims and the following description. The skilled addressee realizes that different features of the present disclosure may be combined to create embodiments other than those described in the following, without departing from the scope of the present disclosure.

[0045] BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The various aspects of the present disclosure, including its particular features and advantages, will be readily understood from the following detailed description and the accompanying drawings, in which:

[0047] Fig. 1 shows an example of a computer system for surgical assistance for use in accordance with a presently preferred embodiment of the present disclosure,

[0048] Fig. 2 provides an exemplary embodiment of a marker unit to be arranged in physical relation to an internal body portion, Figs. 3 A - 3E presents conceptual illustrations of the method steps according to the present disclosure, and

[0049] Fig. 4 is a flow chart illustrating the exemplary steps of the method according to the present disclosure, aligning with Figs. 3 A - 3E.

[0050] DETAILED DESCRIPTION

[0051] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which currently preferred embodiments of the present disclosure are shown. Such a present disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided for thoroughness and completeness, and fully convey the scope of the present disclosure to the skilled person. Like reference characters refer to like elements throughout. The following examples illustrate the present disclosure and are not intended to limit the same.

[0052] Turning now to the drawings, and to Fig. 1 in particular, there is conceptually illustrated a computer system 100 for surgical assistance, comprising a processing unit 110. The computer system 100 is specifically arranged to facilitate augmented reality (AR) overlays during surgical procedures, particularly improving spatial perception and surgical precision through enhanced visualization.

[0053] For reference, the processing unit 110 (and / or processing functionality) may for example be manifested as a general-purpose processor, a graphics processing unit, an application specific processor, a circuit containing processing components, a group of distributed processing components, a group of distributed computers configured for processing, a field programmable gate array (FPGA), etc. The processor may be or include any number of hardware components for conducting data, signal and / or image processing or for executing computer code stored in memory 120. It may also be possible and within the scope to make use of System-on-Chip (SOC) implementations. The memory may be one or more devices for storing data and / or computer code for completing or facilitating the various methods described in the present description. The memory may include volatile memory or non-volatile memory. The memory may include database components, object code components, script components, or any other type of information structure for supporting the various activities of the present description. According to an exemplary embodiment, any distributed or local memory device may be utilized with the systems and methods of such a description. According to an exemplary embodiment the memory is communicably connected to the processor (e.g., via a circuit or any other wired, wireless, or network connection) and includes computer code for executing one or more processes described herein.

[0054] The processing unit 110 is communicatively connected to a plurality of separate components, including an imaging modality 130, an image capturing device 140, and at least one display device 150. Such connections may for example be wired or wireless and are designed to allow seamless data transfer between the components.

[0055] The imaging modality 130 is used to acquire a three-dimensional model of an internal body portion of a patient. Such an imaging modality 130 may include, but is not limited to, technologies such as cone beam computed tomography (CBCT), ultrasound (US), computed tomography (CT), magnetic resonance imaging (MRI), and other suitable imaging techniques capable of providing detailed internal anatomical views. Other types of imaging modalities are exemplified above.

[0056] The image capturing device 140, which for example may be a laparoscopic camera or any other suitable camera used in surgical settings, is preferably designed to capture real-time images of the surgical site. Such images are received by the processing unit 110 and are in line with the present disclosure used for overlay AR visualizations created based on the three-dimensional model acquired by the imaging modality 130, in a general sense in line with the present disclosure.

[0057] The display device 150 is preferably arranged to communicate with the processing unit 110 to show the captured images augmented with the AR overlays. The display device 150 may be part of a surgical navigation system, standalone monitors, or integrated within surgical instruments or head-mounted displays, providing surgeons with real-time, enhanced visual information.

[0058] In operation, as will be further elaborated in relation to Figs. 3 A - 3E and 4, the processing unit 110 generates a point cloud of a three-dimensional model representing a geometry of an internal body portion in coordinates defined for the computing system. Such a process is typically achieved by processing the data received from the imaging modality 130 and utilizing algorithms stored in the memory 120 to convert the detailed geometrical form into a discrete point cloud, as will be further elaborated below.

[0059] The capability of the computer system 100 to overlay visualizations accurately onto real-time images captured by the image capturing device 140 is applied for enhancing surgical precision and orientation. Such an overlay process is supported by the correlation between the position of a marker unit (as shown in Fig. 2) associated with the internal body portion and coordinates defined for the computing system 100, facilitating the precise alignment of AR visualizations.

[0060] Turning further to Fig. 2, there is depicted an exemplary embodiment of the marker unit 200, generally designed for integration with the surgical assistance computer system 100 as presented in relation to Fig. 1. The marker unit 200 comprises a flat body with two main surfaces, 202 and 204 opposite each other, each tailored for specific functional roles within augmented reality (AR)-aided surgery.

[0061] The surface 202 of the marker unit 200 is preferably provided with a set of optically detectable markers 210. These markers are typically arranged in a non-rotationally symmetric geometrical pattern, enabling for a determination of the rotational position of the marker unit 200 when viewed by the image capturing device 140 during surgery. The optically detectable markers 210 are preferably designed to be (preferably highly) visible under the operational lighting conditions typically found in surgical environments. Such markers 210 may include a variety of shapes such as QR codes, geometric figures, or custom- designed icons that may be easily recognizable by the processing unit’s 110 imaging algorithms, generally known within the technical field. Furthermore, the optically detectable markers 210 on surface 202 may include hydrophobic and non-reflective coatings to prevent occlusion by bodily fluids and to minimize light reflections that could impair visibility, enhancing the reliability of optical detection during the procedure.

[0062] Conversely, the opposite surface 204 is preferably designed to interface directly with the internal body portion of a patient during surgery. Possibly, the surface 204 may be coated with a biocompatible adhesive that enables firm attachment to the organ of interest without necessitating invasive procedures. Such an adhesive ensures that the marker unit 200 remains attached throughout the duration of the surgical procedure, even considering mechanical bending stress in relation to minimally invasive surgery.

[0063] Embedded between surfaces 202 and 204 are typically radiopaque markers 220, visible by the medical imaging modality 130. The radiopaque markers 220 may generally be composed of high-density materials such as gold, bismuth, barium sulfate, steel and alloys, or tungsten, which provide excellent visibility under X-ray and CT imaging. The placement of the radiopaque markers 220 are typically designed to align with the optically detectable markers 210, creating a fixed, predetermined geometrical correlation between the two types of markers. Such a dual -marker setup enhances the system’s capability to merge AR visualizations with real-time surgical images, offering a more comprehensive view of the surgical site. The (internal) body of the marker unit 200 itself may typically be made from durable, medical-grade polymers or composite materials that offer high rigidity and resistance to deformation under typical surgical manipulations. Such materials ensure that the marker unit maintains its integrity and dimensional stability, crucial for maintaining accurate geometrical alignment of the markers.

[0064] Additionally, the design of the marker unit 200 is preferably arranged to facilitate secure placement on the targeted internal body portion. For example, the flat body of the marker unit is preferably optimized for introduction through standard laparoscopic ports, with typical dimensions ranging from 5 to 20 mm in length and width, ensuring a slim profile to avoid obstructing surgical tools or views.

[0065] Furthermore, to additionally enhance surgical utility, the marker unit 200 may include features such as handles or tabs (not explicitly shown in Fig. 2) that facilitate manipulation using surgical instruments (again not explicitly presented). Such features may be designed to protrude from the main body without compromising the sterility or functionality of the marker unit 200 and are typically manufactured from the same medicalgrade materials as the body of the marker unit to ensure uniformity in mechanical and sterilization properties, providing surgeons with enhanced control during placement and adjustment. Such features may also be of single-use characteristics, which are detached from the main body of the marker unit after application. Lastly, such features may also consist of a separate tool specially designed for the purpose of transporting and precisely manipulating the marker unit into a desired position and rotation in the patient.

[0066] Turning now to Figs 3 A - 3E in conjunction with Fig. 4, generally presenting an embodiment of operating the computer system 100 in line with the scheme according to the present disclosure. The Figs. 3 A - 3E and 4 illustrate the methodological steps involved in enhancing surgical procedures through augmented reality, facilitated by the computer system 100.

[0067] Firstly, Fig. 3 A illustrates the first step of acquiring, SI, a three-dimensional model of an internal body portion of a patient, in Fig. 3B exemplified by the patient’s liver 302, using an imaging modality such as elaborated above. In relation to Fig. 3B, the marker unit 200 is affixed to the liver 302 of the patient, ensuring that the anatomical and positional data gathered includes the reference points provided by the radiopaque markers of the marker unit. The acquisition process may generally involve scanning techniques that generate high- resolution images, capturing minute anatomical details essential for accurate surgical intervention. Each scan slice acquired is preferably aligned and stacked, using algorithms to reconstruct a full three-dimensional volume of the liver of the patient. Furthermore, during the present step, the computer system 100 may possibly introduce contrast-enhancement protocols to improve the visibility of soft tissues and vascular structures, enhancing the differentiation between various tissue types and pathological conditions, such as for identifying undesired elements within the internal body portion of a patient (such as for example cancer).

[0068] Fig. 3B depicts the generation, S2, of a point cloud of the three-dimensional model representing the geometry of the liver, incorporating the spatial information from the marker unit 200. At this part of the scheme according to the present disclosure, the processing unit 110 utilizes segmentation algorithms to differentiate the liver tissue from surrounding anatomical structures based on contrasts in the imaging data. Such a segmentation is desirable for creating a focused point cloud that precisely represents the target organ's geometry, including the positions of the marker unit's radiopaque markers. The algorithms are designed to handle the variability in tissue densities and interfaces, adjusting parameters dynamically to optimize the accuracy and resolution of the point cloud. The data conversion may generally involve mathematical modeling techniques, such as surface reconstruction and mesh generation, to transform segmented image data into structured spatial representations suitable for real-time processing and visualization. It should be understood that the segmentation could be automatic or semi-automatic.

[0069] Additionally, voxelization algorithms may possibly be employed to discretize the volume captured by the imaging modality 130 into a grid of cubes (voxels), which are then analyzed for their proximity to the surface of the liver. Each voxel at the (liver) surface boundary is transformed into a point in the point cloud, capturing geometric detail necessary for achieving high quality AR overlays.

[0070] The construction of the point cloud may also incorporate error correction mechanisms to address potential anomalies in imaging data, such as noise and artifacts introduced during the image acquisition phase. Such corrections may possibly ensure that the point cloud does not incorporate misleading data that could affect the surgical outcome. Further filtering techniques, such as Gaussian smoothing and median filtering, may possibly be applied to the raw data to refine the point cloud's accuracy, providing a cleaner and more precise anatomical representation for further processing and AR overlay generation.

[0071] Fig. 3C shows the correlation, S3, of the point cloud with the coordinate system of the imaging modality to align with coordinates of the computing system 100. This specific part of the scheme according to the present disclosure involves the implementation of coordinate transformation algorithms that translate the point cloud data into the coordinate system used by the surgical navigation system. The transformation generally takes into account the position and orientation of the marker unit 200, utilizing the radiopaque markers 220 as reference points to ensure that the virtual and real-world align precisely. Techniques such as rigid body transformations, including translations and rotations, may possibly be employed to map the data points accurately onto the patient's anatomy as visualized in the surgical field.

[0072] The correlation process additionally involves calibration procedures that align the imaging systems with the actual position and orientation of the surgical tools and the image capturing device 140. Calibration ensures that any movement of the patient or changes in the imaging perspective are accounted for, maintaining the accuracy of the AR overlays. The dynamic alignment is desirable for adapting to changes in the surgical field, allowing the AR system to update visualizations directly to reflect the current state of the surgical site.

[0073] Additionally, further refinement may possibly be achieved using machine learning algorithms that predict and compensate for any potential misalignment caused by patient movement or variability in the imaging procedure. Such machine learning algorithms may for example be trained on a dataset of prior surgeries to learn variability patterns, thus enhancing the computer system’s ability to maintain accurate alignment in a dynamic surgical environment.

[0074] Fig. 3D illustrates the overlaying, S4, of a visualization of the liver 302 of the patient over an image captured by an image capturing device, such as the laparoscopic camera 140 as discussed above. Here, the liver is also illustrated as comprising a cancer section 304, that is intended to be removed by surgery, where the cancer section 304 has been identified from the three-dimensional model of the liver 302 as captured using the imaging modality 130. Such a phase of the scheme according to the present disclosure may make use of the aligned point cloud to generate an augmented reality layer that maps onto the live video feed from the operating field. Real-time rendering techniques may be applied to blend the virtual and real images seamlessly. GPU-accelerated rendering processes are utilized to project the point cloud data onto the camera’s images, adjusting for perspective, lighting, and shading to create a cohesive and intuitive visual experience. Moreover, the computer system 100 may incorporate dynamic adjustment protocols that recalibrate the overlay in real-time as the camera moves or as the surgical scene changes, ensuring that the virtual augmentations remain precisely aligned with the actual anatomical landmarks. Fig. 3E demonstrates the display, S5, of the captured image augmented with the AR visualization on at least one display device 150. The display device 150 could for example be a high-resolution monitor that provides the surgical team with a combined view of real and augmented visuals, enhancing depth perception and surgical accuracy.

[0075] Furthermore, the control functionality of the present disclosure may be implemented using existing computer processors, or by a special purpose computer processor for an appropriate system, incorporated for this or another purpose, or by a hardwire system. Embodiments within the scope of the present disclosure include program products comprising machine-readable medium for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor. By way of example, such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, solid state drives or other non-volatile flash based storage devices, or any other medium which can be used to carry or store desired program code in the form of machine-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer or other machine with a processor. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a machine, the machine properly views the connection as a machine-readable medium. Thus, any such connection is properly termed a machine-readable medium. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data which cause a general-purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.

[0076] Although the figures may show a sequence, the order of the steps may differ from what is depicted. For example, two or more steps may be performed concurrently or with partial concurrence. Such variation will depend on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations could be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps and decision steps. Additionally, even though the present disclosure has been described with reference to specific exemplifying embodiments thereof, many different alterations, modifications and the like will become apparent for those skilled in the art.

[0077] In addition, variations to the disclosed embodiments can be understood and effected by the skilled addressee in practicing the claimed present disclosure, from a study of the drawings, the disclosure, and the appended claims. Furthermore, in the claims, the word ’’comprising” does not exclude other elements or steps, and the indefinite article ”a” or ”an” does not exclude a plurality.

Claims

CLAIMS1. A computer-implemented method performed by a computing system (100) comprising a processing unit (110), for generating an augmented visualization for surgical assistance using a computing system comprising a processing unit, wherein the method comprises the steps of: acquiring (SI), using an imaging modality (130), data representing a three- dimensional model of an internal body portion (302) of a patient, wherein the internal body portion (302) is physically related with a marker unit (200), the data includes a representation of the marker unit (200), and the marker unit (200) is optically detectable and radiopaque, generating (S2), using the processing unit (110), a point cloud of the three- dimensional model representing a geometry of the internal body portion (302) in coordinates defined for the computing system (100), correlating (S3), using the processing unit (110) and the positioned marker unit (200), the point cloud with a coordinate system of the imaging modality (130) to align with the computing system’s coordinates, overlaying (S4), using the processing unit (110), a visualization of the internal body portion (302) over an image captured by an image capturing device (140) in which the marker unit (200) is visible, based on an estimation of a position of the image capturing device (140) relative to the marker unit (200), and displaying (S5), on at least one display device (150) arranged in communication with the computing system (100), the captured image augmented with the overlay ed visualization of the internal body portion (302) of the patient.

2. The method according to claim 1, wherein the imaging modality (130) is arranged to apply at least one of cone beam computed tomography (CBCT), ultrasound (US), computed tomography (CT), magnetic resonance imaging (MRI), C-arm fluoroscopy (CARM), X-ray imaging (XRAY), and fluoroscopy technology adapted for acquiring the three-dimensional model of the internal body portion (302).

3. The method according to any one of claims 1 and 2, further comprising the step of:applying, using the processing unit (110), a predefined algorithm for noise reduction on the three-dimensional model data before generating the point cloud, wherein the predefined algorithm is adapted for enhancing an accuracy of a visualization of the internal body portion (302).

4. The method according to any one of the preceding claims, further comprising the step of applying, using the processing unit (110), a predefined augmented reality (AR) technique adapted to utilize the correlation between the marker unit’s position and the computing system’s coordinates to overlay the visualization accurately on the image captured using the image capturing device (140).

5. The method according to any one of the preceding claims, wherein the step of displaying the captured image augmented with the overlayed visualization comprises: adjusting, using the processing unit (110), a visualization opacity based on an estimated depth of the internal body portion (302) from an external surface of the patient’s body.

6. The method according to any one of the preceding claims, further comprising the step of calculating, using the processing unit (110), a three-dimensional distance map between geometry of the internal body portion (302) and the marker unit (200) to facilitate AR alignment.

7. The method according to any one of the preceding claims, further comprising the step of generating, using the processing unit (110), a virtual surgical plan that overlays navigational cues on the at least one display device (150), based on the augmented visualization of the internal body portion (302), for guiding surgical instruments during an associated surgical procedure.

8. The method according to any one of the preceding claims, wherein the overlaying of the visualization over the image captured by the image capturing device (140) includes:dynamically adjusting, using the processing unit (110), the visualization in real-time based on changes in the position and orientation of the image capturing device (140) relative to the marker unit (200), for ensuring consistent alignment of the augmented reality overlay with the internal body portion (302).

9. The method according to any one of the preceding claims, further comprising the step of: implementing, using the processing unit (110), a user interface on the display device (150) for interactively adjusting parameters of the overlayed visualization to tailor the augmented reality view.

10. A computer system (100) for surgical assistance, the computer system comprising a processing unit (110), wherein the processing unit (110) is adapted to: acquire, using an imaging modality (130), data representing a three- dimensional model of an internal body portion (302) of a patient, wherein the internal body portion (302) is physically related associated with a marker unit (200), the data includes a representation of the marker unit (200), and the marker unit (200) is optically detectable and radiopaque, generate a point cloud of the three-dimensional model representing a geometry of the internal body portion (302) in coordinates defined for the computing system (100), correlate, using the positioned marker unit (200), the point cloud with a coordinate system of the imaging modality (130) to align with the computing system’s coordinates, overlay a visualization of the internal body portion (302) over an image captured by an image capturing device (140) in which the marker unit (200) is visible, based on an estimation of a position of the image capturing device (140) relative to the marker unit (200), and display, on at least one display device (150) arranged in communication with the computing system (100), the captured image augmented with the overlayed visualization of the internal body portion (302) of the patient.

11. The computer system (100) according to claim 10, wherein the imaging modality (130) is arranged to apply at least one of cone beam computed tomography (CBCT), ultrasound (US), computed tomography (CT), magnetic resonance imaging (MRI),C-arm fluoroscopy (CARM), X-ray imaging (XRAY), and fluoroscopy technology adapted for acquiring the three-dimensional model of the internal body portion (302).

12. The computer system (100) according to any one of claims 10 and 11, wherein the processing unit (110) is further adapted to: apply a predefined algorithm for noise reduction on the three-dimensional model data before generating the point cloud, wherein the predefined algorithm is adapted for enhancing an accuracy of a visualization of the internal body portion (302).

13. The computer system (100) according to any one of claims 10 - 12, wherein the processing unit (110) is further adapted to: apply a predefined augmented reality (AR) technique adapted to utilize the correlation between the marker unit’s position and the computing system’s coordinates to overlay the visualization accurately on the image captured using the image capturing device (140).

14. The computer system (100) according to any one of claims 10 - 12, wherein the processing unit (110) when displaying the captured image augmented with the overlayed visualization if further adapted to: adjust a visualization opacity based on the estimated depth of the internal body portion (302) from an external surface of the patient’s body.

15. A computer program product comprising a non-transitory computer readable medium having stored thereon computer program means for operating a computing system (100) designed for surgical assistance, the computing system comprising a processing unit (110), wherein the computer program product comprises: code for acquiring, using an imaging modality (130), data representing a three- dimensional model of an internal body portion (302) of a patient, wherein the internal body portion (302) is physically related associated with a marker unit (200), the data includes a representation of the marker unit (200), and the marker unit (200) is optically detectable and radiopaque, code for generating, using the processing unit (110), a point cloud of the three- dimensional model representing a geometry of the internal body portion (302) in coordinates defined for the computing system (100),code for correlating, using the processing unit (110) and the positioned marker unit (200), the point cloud with a coordinate system of the imaging modality (130) to align with the computing system’s coordinates, code for overlaying, using the processing unit (110), a visualization of the internal body portion (302) over an image captured by an image capturing device (140) in which the marker unit (200) is visible, based on an estimation of a position of the image capturing device (1 0) relative to the marker unit (200), and code for displaying, on at least one display device (150) arranged in communication with the computing system (100), the captured image augmented with the overlay ed visualization of the internal body portion (302) of the patient.

Images