Marker unit for use in AR-assisted surgery - Patent application

Abstract

A marker unit for use in augmented reality assisted surgery, in particular laparoscopic liver surgery, is disclosed, comprising a flat body having two main surfaces, one of which is provided with a set of optically detectable markers and the opposite main surface is configured to be coupled to a body organ, preferably provided with an adhesive. The marker unit further comprises a set of radiopaque markers that are visible by a medical imaging system. Both the optically detectable markers and the radiopaque markers are provided in a geometric pattern that does not have rotational symmetry, thereby enabling the rotational position of the marker unit to be determinable, and the geometric patterns of the optically detectable markers and the radiopaque markers have a fixed, pre-determined correlation.

Description

[Technical field]

[0001] The present invention relates to a marker unit for use in augmented reality (AR) assisted surgery. The present invention also relates to a system for augmented reality (AR) assisted surgery including such a marker unit. [Background technology]

[0002] Surgical removal of cancer tumors in the liver was traditionally performed by open surgery. Today, there has been a major shift towards minimally invasive surgery using laparoscopy. However, laparoscopic surgery is a relatively complex procedure that requires great skill and a lot of training. In particular, it is often difficult for surgeons to find the way and direction based on the limited visual input provided by the laparoscopic camera. Liver surgery is particularly challenging due to the uniformity of the liver's shape and surface. Moreover, liver tumors are often located deep inside the liver and are not visible by the thin laparoscopic light camera.

[0003] In laparoscopic surgery, several ports are provided for access to the surgical site by tools, cameras, etc. The surgeon orients from the outside and perceives the inside through 2D images from the cameras plus pre-operative 3D images as a map.

[0004] Laparoscopic surgery has many advantages, but it is a difficult technique to master and requires a lot of practice. Liver tumors, for example, are particularly complex because the liver is a very homogeneous organ with a smooth surface, making orientation in the 2D camera view very difficult for the surgeon.

[0005] In recent years, many attempts have been made to overcome these problems. "Radiopaque fiducials guiding laparoscopic resection of liver tumours," M. Falkenberg et al., Surg. Laparosc. Endosc. Percutan Tech, September 28, 2021, explores how radiopaque fiducials and fluoroscopy can complement ultrasound during liver tumour resection, providing extended 3D information of the liver combined with information from a camera view. The overall aim of the study was to develop a workflow whereby the location of the tumour could be tracked during the procedure as a complement to the use of ultrasound.

[0006] "The effect of intraoperative imaging on surgical navigation for laparoscopic liver resection surgery", A. Teatini et al., Scientific Reports, (2019) 9:18687, relates to the so-called CAScination AR solution. It is based on a camera that detects markers outside the body. For this to work, the body and the liver must be immobile. Using this, information from a previously taken CT scan gives the opportunity to display (in theory) hidden parts. But in practice, it is difficult to maintain perfect stability between all parts during the operation.

[0007] Additionally, U.S. Patent Application Publication No. 20200005473 discloses an alignment system for liver surgery, WO2016 / 170372 discloses an apparatus and method for registering pre-operative image data with intraoperative laparoscopic ultrasound images, and WO2016 / 012556 discloses an image generating apparatus and method using a combination of functional imaging and ultrasound images.

[0008] Despite these efforts, a need remains for more efficient systems for guiding surgeons during laparoscopic procedures, particularly for liver surgery, and in particular for systems that facilitate orientation at an internal surgical site that can be implemented and used in a relatively fast and cost-effective manner. Summary of the Invention [Problem to be solved by the invention]

[0009] It is therefore an object of the present invention to provide a marker unit for use in augmented reality assisted surgery that mitigates all or at least some of the shortcomings of currently known systems. It is another object of the present invention to provide a system for AR assisted surgery that includes such a marker unit. [Means for solving the problem]

[0010] This object is achieved by a marker unit and system for AR assisted surgery as defined in the appended claims.

[0011] According to a first aspect of the present invention there is provided a marker unit for use in augmented reality assisted surgery, the marker unit comprising a flat body having two main surfaces, one of the main surfaces being provided with a set of optically detectable markers and the opposite main surface being configured to be coupled to a body organ and preferably provided with an adhesive, the marker unit further comprising a set of radiopaque markers that are visible by a medical imaging system, both the optically detectable markers and the radiopaque markers being provided in a geometric pattern that does not have rotational symmetry thereby enabling a rotational position of the marker unit to be determined, and the geometric patterns of the optically detectable markers and the radiopaque markers have a fixed, predetermined geometric correlation.

[0012] By medical imaging system is meant here a system for imaging the inside of a body. The medical imaging data is preferably 3D data and is preferably obtained through a non-invasive procedure. In particular, the medical imaging system may be a medical X-ray or ultrasound system.

[0013] Optically detectable markers are detectable by detection of light from the visible spectrum, such as by a camera, and preferably provide as 2D data.

[0014] The invention is based on the realization that a single marker unit, designed and structured in a particular way, allows the correlation between medical image data and received optical data, such as from a laparoscopic camera. This correlation can be done in real time and can be presented to the user as an augmented reality view showing both images in combination.

[0015] The marker unit can be easily and reliably placed, resulting in fast and cost-effective treatment. Furthermore, the set of markers can be directly correlated from images received from medical imaging systems and optical detection systems such as cameras, without the need to know the exact location of the medical imaging device or camera or any other external device.

[0016] Although two or more marker units may be used, a single marker unit is usually sufficient to provide the required correlation between images. It is therefore sufficient to place a single marker unit on the anatomical part and surface where the operation will be performed, such as the surface of the liver. The marker unit includes a set of optically detectable markers, which are detectable, for example, with an optical camera. Furthermore, the marker unit includes a set of radiopaque markers, which are visible by medical imaging systems, such as computed tomography (CT), cone-beam computed tomography (CBCT), magnetic resonance imaging (MRI), and ultrasound (US) imaging. However, other medical imaging systems and techniques may also be used, as known per se in the art.

[0017] One side of the marker unit may be provided with an adhesive, making it very easy to fix it to the anatomical structure where it is to be located, such as the surface of the liver, etc. Such fixation is non-invasive and can be done quickly and efficiently through a laparoscopic port.

[0018] Both the optically detectable markers and the radiopaque markers are provided in a geometric pattern that does not have rotational symmetry, thereby allowing the rotational position of the marker unit to be determinable, and furthermore the geometric patterns of the optically detectable markers and the radiopaque markers have a fixed, pre-determined correlation. This allows the exact position of the marker unit to be uniquely identified in all directions from both the medical imaging data and the received light data. Due to the known correlation between the marker sets, this allows the images to be correlated and they can be presented in a combined manner as an augmented reality image.

[0019] Thus, the invention provides a marker unit that can be used both as a reference point for pre-operative three-dimensional (3D) image data of an organ and as an optical reference point for, for example, a camera.

[0020] The flat body of the marker unit is preferably made from a rigid material. As a result, the marker unit is preferably rigid. By rigid, it is meant here that the flat body and the marker unit do not deform in normal use and handling and are not prone to bending under normal finger pressure. The rigidity of the flat body and the marker unit ensures that the marker unit does not deform during use, especially when applied to a body organ, thereby ensuring that the reading and correlation of the set of markers remains unaffected.

[0021] In an embodiment, the flat body includes two layers fixedly bonded to one another, and the set of radiopaque markers is disposed sandwiched between the two layers. However, the flat body may alternatively include only a single layer, or three or more layers. Furthermore, the radiopaque markers may alternatively be disposed on the outer surface of the flat body.

[0022] In an embodiment, the flat body has a rectangular shape, preferably a square shape, but other shapes are possible, such as an approximately rectangular shape but with rounded or chamfered corners, etc. In other embodiments, the flat body may also be circular, elliptical, hexagonal, octagonal, etc.

[0023] In one embodiment, the flat body may include at least one layer made of a radiopaque material, such as a radiopaque plastic material, so-called X-ray plastic. For example, the flat body may include a single layer made of such a radiopaque material. In such an embodiment, recognizable features, for example at the edges, may form the set of radiopaque markers. For example, corners may serve as such recognizable features, thereby forming the set of radiopaque markers. In that case, at least one corner may be shaped differently from the others in order to make the pattern non-rotationally symmetric. For example, the differently shaped corners may be truncated corners, corners with cut-out recesses, rounded corners, etc.

[0024] The main faces may have dimensions in two orthogonal directions in the range 5 to 20 mm, preferably in the range 5 to 15 mm, most preferably in the range 7 to 12 mm. In the case of a rectangle, the dimensions may be seen along the orthogonal sides.

[0025] In an embodiment, the marker unit may have a maximum extension length in any direction that is less than 12 mm, preferably less than 10 mm. The maximum extension length refers to the longest distance between any two positions of the marker unit. In a flattened circular configuration, the maximum extension length will substantially correspond to the diameter, and in a square or rectangular configuration, the maximum extension length will substantially correspond to the extension length of the diagonal between two oppositely disposed corners. A marker unit with a maximum extension length of less than 12 mm makes the marker unit highly advantageous for use in many laparoscopic surgical procedures, since laparoscopic ports often have an inner diameter of 12 mm. A marker unit with a maximum extension length of less than 10 mm makes the marker unit more suitable for laparoscopic surgical procedures, since it can then be used for both laparoscopic ports with an inner diameter of 12 mm and laparoscopic ports with an inner diameter of 10 mm, which is also commonly used.

[0026] The set of optically detectable markers preferably includes a plurality of geometric shapes distributed throughout the major surface. The geometric shapes may be, for example, rectangular. In an embodiment, the plurality of geometric shapes may include a plurality of QR Codes. The QR Codes may include black squares arranged in a square grid on a white background. The color scheme may also be reversed, white squares on a black background. However, other color combinations are possible, such as black squares on a red background. The squares may include relatively large position squares at three of the four corners and, optionally, a relatively small alignment square at the fourth corner. Additional alignment squares may also be provided, such as at the sides, between the corner squares, and / or in the center. Timing lines may also be provided between the corner squares. The QR Code may also include a data carrying pattern, although this is optional in this context and may be omitted.

[0027] However, other types of geometric shapes may also be used, such as polygons, for example triangles, pentagons, and hexagons, as well as ellipses, circles, semicircles, and other forms of sectors.

[0028] The radiopaque markers may be formed from metal etc. as known per se in the art. The set of radiopaque markers may be arranged, for example, as a grid of elliptical or rectangular dots, such that one or more positions are empty, thereby making the pattern non-rotationally symmetric.

[0029] However, the radiopaque marker may also be formed by a detectable feature on a layer or body formed by a radiopaque material, such as a radiopaque plastic material. Such a detectable feature may, for example, be provided in the form of a detectable and identifiable corner of a generally rectangular layer / body.

[0030] The major surface containing the set of optically detectable markers is preferably hydrophobic, which avoids water, body fluids, etc. from accumulating on the surface. This ensures that the optically detectable markers are not obscured during use. The major surface containing the set of optically detectable markers is further preferably non-reflective, which also makes optical detection easier.

[0031] The hydrophobic surface may be obtained by the use of a coating or layer of a hydrophobic material disposed on the main surface of the marker unit. Alternatively, however, the marker unit may itself be made of a hydrophobic material. The hydrophobic material preferably comprises mainly non-polar molecules. The hydrophobic material is preferably such that an aqueous liquid in contact with the hydrophobic material group packs together to form micelles and has a large contact angle with respect to the surface of the hydrophobic material, preferably greater than 90 degrees, more preferably greater than 100 degrees, most preferably greater than 110 degrees. The contact angle may also be greater than 145 degrees, in which case the hydrophobic material may be called superhydrophobic.

[0032] The marker unit is particularly suitable for laparoscopic surgery, in particular laparoscopic liver surgery, where the laparoscopic surgery may be a conventional manually performed laparoscopic surgery or a robotically assisted laparoscopic surgery, but the marker unit may also be used for other types of laparoscopic and endoscopic surgery, and for other types of surgery where direct optical visibility is limited.

[0033] The marker unit may include a handle. Such a handle may be provided in the form of a gripping or maneuvering area formed on the marker unit. The gripping or maneuvering area is preferably a portion of the marker unit that is not provided with adhesive. The gripping or maneuvering area may preferably be located outside the perimeter of the optically detectable marker. This allows the marker unit to be gripped and manipulated by conventional laparoscopic instruments and tools, such as laparoscopic graspers, graspers, etc., without risk of damaging the marker unit or otherwise adversely affecting its properties, functionality and performance.

[0034] Additionally or alternatively, the handle may be configured as a protrusion configured to protrude laterally or transversely with respect to the major surface of the marker unit. In one embodiment, the protrusion may extend laterally, preferably in the plane of the major surface of the marker unit. In another embodiment, the protrusion may protrude transversely with respect to the major surface, such as orthogonally with respect to the major surface. For example, the protrusion may have a width that is less than the width and length of the marker unit.

[0035] The handle may be provided with a friction-increasing configuration to increase friction, thereby making it easier to grasp and hold the marker unit, for example with a laparoscopic instrument. For example, the friction-increasing configuration may include a surface texture on the surface of the handle, for example in the form of corrugations. Additionally or alternatively, the friction-increasing configuration may include a coating of a high friction material or the like.

[0036] According to another aspect of the present invention, there is provided a system for augmented reality assisted surgery comprising a marker unit as discussed above, and further comprising a display and a controller, the controller comprising: receiving medical imaging data indicative of a marker unit in a position intended for surgery; receiving received light image data indicative of a marker at a location intended for surgery; correlating the medical imaging data and the received light image data based on locations of the set of radiopaque markers in the medical imaging data and the set of optically detectable markers in the received light image data; providing an augmented reality image on the display, the augmented reality image including, at least in part, image data from the medical imaging data and received image data; The present invention is configured to perform the following steps.

[0037] The medical imaging data may be received from at least one of a computed tomography scan (CT), a cone-beam computed tomography (CBCT), a magnetic resonance imaging (MRI), and an ultrasound imaging (US).

[0038] The received light image data is preferably received from a camera, preferably a laparoscopic camera.

[0039] In this aspect of the invention, the same advantages and preferred features exist as in the first aspect of the invention discussed above, and vice versa.

[0040] These and other features and advantages of the invention will be apparent from the following with reference to the embodiment(s) described hereinafter.

[0041] For illustrative purposes, the invention will now be described in more detail with reference to embodiments thereof illustrated in the accompanying drawings. [Brief description of the drawings]

[0042] [Figure 1] 1 is a schematic diagram in a top plan view of a main surface of a marker unit having a set of optically detectable markers according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a schematic diagram in a top plan view of the lower layer of the marker unit of FIG. 1 having a set of radiopaque markers with the upper layer removed or made invisible, according to an embodiment of the present invention. [Diagram 3] 1 is a schematic perspective view of a marker unit according to the present invention; [Figure 4] FIG. 1 is a schematic diagram of a system for augmented reality assisted surgery according to an embodiment of the present invention. [Diagram 5] FIG. 13 is a schematic diagram in a top plan view of a major surface of a marker unit having a set of optically detectable markers and radiopaque markers according to another embodiment of the present invention. [Figure 6] FIG. 13 is a schematic diagram in a top plan view of a major surface of a marker unit having a set of radiopaque markers according to another embodiment of the present invention. [Figure 7] FIG. 13 is a schematic diagram in a top plan view of a major surface of a marker unit having a set of radiopaque markers according to another embodiment of the present invention. [Figure 8] 1 is a schematic perspective view of a marker unit having a handle according to an embodiment of the present invention. [Figure 9] FIG. 13 is a schematic diagram in a top plan view of a marker unit having a laterally protruding handle according to another embodiment of the present invention. [Figure 10] FIG. 13 is a schematic diagram in a top plan view of a marker unit having a laterally protruding handle according to another embodiment of the present invention. [Figure 11] FIG. 13 is a schematic perspective view of a marker unit having a handle protruding in a transverse direction according to yet another embodiment of the present invention. [Figure 12] FIG. 13 is a schematic perspective view of a marker unit having a handle protruding in a transverse direction according to yet another embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0043] In the following detailed description, preferred embodiments of the present invention are described. However, it should be understood that the features of different embodiments can be exchanged between the embodiments and can be combined in different ways, unless otherwise specified. In the following description, many specific details are described to provide a more thorough understanding of the present invention, but it will be apparent to those skilled in the art that the present invention can be practiced without these specific details. In other cases, well-known structures or functions are not described in detail so as not to obscure the present invention. In the following examples, embodiments related to laparoscopic liver surgery are disclosed. However, it should be appreciated by those skilled in the art that the method and system can be used correspondingly to many other types of surgical procedures.

[0044] First, the marker unit 1 will be discussed with reference to Figures 1 to 3. The marker unit includes a flat body 11, preferably formed from a relatively rigid plastic material. The material is preferably biocompatible and heat resistant up to at least 100°C. The flat body may include two layers 11a, 11b fixedly bonded to each other, for example by an adhesive.

[0045] The flat body has two main faces 12, 13 on either side and a surrounding side wall. Here, the flat body, and thus the main faces, have a rectangular shape, in particular an approximately square shape. However, other shapes are possible, such as an approximately rectangular shape but with rounded or chamfered corners. In other embodiments, the flat body may also be circular, elliptical, hexagonal, octagonal or other polygonal form, etc. In one embodiment, the flat body has a polygonal shape.

[0046] The major faces may have dimensions in two orthogonal directions in the range 5 to 20 mm, preferably in the range 5 to 15 mm, and most preferably in the range 7 to 12 mm. If rectangular, the dimensions may be seen along the orthogonal sides. For example, the major faces may be about 10 mm on each side and about 1 cm2 The marker unit may form a square that defines an area of ​​10 mm. The height of the marker unit is sufficiently smaller than the dimensions of the sides. The height may be, for example, in the range of 1-5 mm, preferably in the range of 1-3 mm. The limited size allows the marker unit to be easily introduced into the body and to the surgical site through a laparoscopic port, which will be discussed further below, without the need to fold or compress the marker unit for insertion. In other words, therefore, the marker unit is preferably dimensioned so that it can be introduced into a standard laparoscopic port.

[0047] On one of the major surfaces 12 there is a set of optically detectable markers as discussed in more detail below.

[0048] On the other, opposing main surface 13, an adhesive may be provided. The adhesive may be of any type suitable for attachment to an anatomical body part, as known per se in the art. For example, the adhesive may be a biodegradable adhesive, such as a fibrin adhesive. The adhesive may be initially covered by a cover liner, which may be removed before use to expose the adhesive. However, the adhesive may also be provided in other ways, such as initially provided separately from the marker unit and disposed on the main surface of the marker unit prior to insertion and fixation of the marker unit to a body organ.

[0049] As will also be discussed in more detail below, a set of radiopaque markers are provided that are integrally configured into the marker unit, for example disposed between layers 11a and 11b.

[0050] Both the optically detectable markers and the radiopaque markers are provided in a geometric pattern that does not have rotational symmetry, thereby allowing the rotational position of the marker unit to be determinable, and furthermore the geometric patterns of the optically detectable markers and the radiopaque markers have a fixed, pre-determined correlation. This allows the exact position of the marker unit to be uniquely identified in all directions from both the medical imaging data and the received light data. Due to the known correlation between the marker sets, this allows the images to be correlated and they can be presented in a combined manner as an augmented reality image.

[0051] The set of optically detectable markers preferably comprises a plurality of geometric shapes distributed across a major surface. The geometric shapes may be, for example, rectangles.

[0052] In this embodiment, the plurality of geometric figures includes a plurality of QR codes 2. The QR code may include black squares arranged in a square grid on a white background. The squares may include relatively large position squares 21a-c at three of the four corners and, optionally, a relatively small alignment square 22a at the fourth corner. An additional alignment square 22b may also be provided in the center. Further, an additional alignment square 22c may be provided between the corner squares at the sides. Also, so-called timing lines 23a, 23b may be provided between the corner squares, in particular between the position squares 21a-c. The QR code may also include a data carrying pattern, although this is optional in this context and may be omitted.

[0053] However, it is also possible to use position squares 21a-c alone, or alignment square 22a alone, or additionally or alternatively in combination with one or more other alignment squares 22b-c.

[0054] The face 12 of the marker unit may include a plurality of such QR codes. In the illustrated example, four QR codes 2 are provided, arranged in the four quadrants of the main face. As each marker unit includes at least 3-4 QR codes, and as each QR code typically includes four detectable squares, e.g. one at each corner of each QR code, this provides a total of at least 12-16 detectable markers on the main face of the marker unit. Additionally or alternatively, the corners of the QR codes may themselves serve as optically detectable markers. Also, other geometrically distinct patterns or features of the main face may serve as optically detectable markers.

[0055] The radiopaque markers may be formed from metal or the like, as known per se in the art. The set of radiopaque markers may be arranged, for example as a grid of circular or rectangular dots 3, such that one or more positions are empty, thereby making the pattern non-rotationally symmetric. In the illustrated example, the markers 3 are provided in a 3x3 grid, with the marker in the lower right corner omitted. However, other non-rotationally symmetric patterns are also possible.

[0056] The two sets of markers, the optically detectable markers 2 and the radiopaque markers 3, have a fixed, pre-determined geometric correlation. For example, the outer corner elements of both patterns may overlap each other. For example, the location square 21a in the top left QR code 2 may overlap the top left marker 3, the location square 21b in the top right QR code may overlap the top right marker 3, and the location square 21c in the bottom left QR code may overlap the bottom left marker 3. However, other ways of correlating the sets of markers with each other are also possible.

[0057] The major surface containing the set of optically detectable markers is preferably hydrophobic and non-reflective.

[0058] Other implementations of optically detectable and / or radiopaque markers are possible.

[0059] In one embodiment shown in FIG. 5, the flat body may include at least one layer made of a radiopaque material, such as a radiopaque plastic material, so-called X-ray plastic. For example, the flat body may include a single layer made of such a radiopaque material. In such an embodiment, recognizable features, for example at the edges, may form the set of radiopaque markers. For example, corners may serve as such recognizable features, thereby forming the set of radiopaque markers 3a-b. In that case, at least one corner 3b may be shaped differently from the others in order to make the pattern non-rotationally symmetric. For example, the differently shaped corners 3b may be truncated corners.

[0060] Here, the optically detectable marker 2 may be provided as a QR code, similar to the previously discussed embodiment. Alternatively, the optically detectable marker 2 may also be other types of geometrically distinct shapes, such as a rectangle. In one embodiment, the optically detectable marker 2 may be provided by the shape of one or more layers of a flat body, such as a corner or other distinct geometric feature of a shape.

[0061] In the illustrated embodiment of FIG. 5, one of the corners is cut and chamfered to form an obtuse angle, thereby forming a distinct marker 3b. However, other ways of forming a distinct corner marker are also possible. For example, as shown in FIG. 6, a distinct marker 3b' may be formed by cutting a rectangular piece out of the corner, thereby forming an inward corner as opposed to the outward corners of the other corner positions. It is also possible to provide a cutout with a hypotenuse, as in the illustrated example of FIG. 7, thereby forming a distinct marker 3b'' with an inward corner having an obtuse or acute angle.

[0062] However, other distinct markers are possible, such as rounded cutouts, etc. Also, cutouts etc. may be provided in other locations besides corners, such as along one or more sides or within the body / layer.

[0063] In the illustrated examples of Figures 5-7, only one of the markers 3b, 2b', 3b'' is formed distinct from the remaining marker 3a. However, it is also possible to use two or more distinct markers, for example provided with differently shaped cut-outs. Thus, two, three, four or more distinct radiopaque markers may be provided.

[0064] Next, with reference to FIG. 4, a system for augmented reality assisted surgery will be discussed that includes the marker unit discussed above.

[0065] The system includes at least one marker unit 1, here placed on the liver 3. Placement of the marker unit 1 may be performed through a laparoscopic port 5 placed through the patient's skin 4 when inflated for surgery.

[0066] The system further comprises a controller 6 connected to an optical display 7. The controller is further connected to a medical imaging device / system 9 for receiving medical imaging data. The medical imaging device may for example be a device / system for computed tomography (CT), cone-beam computed tomography (CBCT), magnetic resonance imaging (MRI) or ultrasound (US) imaging. However, also other medical imaging systems and techniques may be used, as known per se in the art. The controller 6 is further connected to an optical imaging device / system, such as a laparoscopic camera 8.

[0067] Controller 6 is receiving medical imaging data indicative of a marker unit in a position intended for surgery; receiving received light image data indicative of a marker at a location intended for surgery; correlating the medical imaging data and the received light image data based on locations of the set of radiopaque markers in the medical imaging data and the set of optically detectable markers in the received light image data; providing an augmented reality image on the display, the augmented reality image including, at least in part, image data from the medical imaging data and received image data; The present invention is configured to:

[0068] The controller may be implemented on a personal computer or other type of computer, such as a tablet.

[0069] The display may be a conventional display screen for the optical camera 8. However, more than one display may be provided. Additionally or alternatively, the display may be provided in the form of a wearable display, such as one integrated into glasses.

[0070] Methods of using the system are now discussed.

[0071] First, the patient is prepared for surgery, which typically means the surgeon makes a small incision, attaches a laparoscopic port, and the abdominal cavity is inflated with carbon dioxide to give the surgeon more room to work.

[0072] Secondly, the marker unit is inserted through the laparoscopic port and placed on the liver surface. Before introducing the marker unit, the adhesive is exposed by removing any liner or the like. Upon insertion, for example with laparoscopic forceps, it is sufficient to gently press the adhesive side of the marker unit towards the liver surface to obtain fixation. The marker unit is preferably placed over or near the location where a portion of the liver is surgically removed. However, the marker unit may also be fixed to the body organ in other ways, such as by an externally applied adhesive or by other fixing members known per se.

[0073] The marker unit is fixed to the body organ in such a way that the main surface having the optically detectable marker faces away from the body organ, thereby exposing the optically detectable marker.

[0074] The marker units are then detected with respect to the anatomical geometry of the patient, this can be done by medical imaging such as CT, CBCT, MRI, US etc.

[0075] A laparoscopic camera and tools are then inserted through the laparoscopic port.

[0076] When the liver and marker unit are located within the field of view of the camera, the marker unit provides a reference point for the software and an augmented reality projection is shown on the surgeon's display. The displayed augmented reality image may, for example, include 2D live image data received from the camera, with additional image information based on the medical image data, for example showing the interior of the liver and in particular the tumor, which may be 3D but projected onto the 2D image.

[0077] In particular, the augmented reality view may complement the optical 2D image with features such as structures of interest derived from medical imaging data, e.g., structures at risk such as tumors, important blood vessels or nerves, specific recognizable features, structures within organs that are not visible because they are covered by tissue, etc.

[0078] The surgeon now performs a tumor resection and removes the tumor together with the marker unit, which may remain attached to the tissue / tumor piece that is removed if the marker unit is placed substantially over the tumor.

[0079] To facilitate grasping and handling of the marker unit by laparoscopic instruments, such as laparoscopic graspers and graspers, the marker unit may be provided with a handle. Such a handle may be realized in a variety of ways. Several exemplary embodiments of marker units having such handles are discussed below with reference to Figures 8-12.

[0080] The marker unit shown in FIG. 8 includes a handle 14. Here, the handle 14 is provided in the form of a gripping or maneuvering area formed on the marker unit. Here, the gripping or maneuvering area is that portion of the marker unit that is not provided with adhesive 13'. The gripping or maneuvering area may preferably be located outside the confines of the optically detectable marker. This allows the marker unit to be gripped and manipulated by conventional laparoscopic instruments such as laparoscopic graspers, graspers, and the like, without risk of damaging the marker unit or otherwise adversely affecting its properties, functionality and performance.

[0081] In the embodiment of Figures 9 and 10, the handle 14' is instead configured as a protrusion configured to protrude laterally from the flat body 11, generally within the main plane of the marker unit. The protrusion may be formed as a tab or tongue and may have a width and length significantly shorter than the width and length of the flat body 11. The handle may be generally solid, as shown diagrammatically in Figure 9, or may be provided with an opening 14a, as shown diagrammatically in Figure 10. Such an opening 14a may facilitate gripping of the handle.

[0082] In the illustrated embodiment, the handle is located on a side of the flat body, preferably substantially in the center of this side, but other locations are possible, for example the handle may instead be located at or near a corner of the flat body 11.

[0083] The handle may alternatively be oriented in a direction other than transversely, in the plane of the main surface. Such an embodiment is shown diagrammatically in Figures 11 and 12. Here, the handle 14'' is formed as a protrusion, which protrudes transversely with respect to the flat body 11 and the main surface. In the illustrated example, the handle protrudes substantially perpendicularly with respect to the flat body 11 and the main surface. However, the handle may alternatively protrude in other directions, such as in various oblique directions.

[0084] In the illustrated embodiment, the handle is configured as a transversely projecting flat tab extending along a side of the flat body 11. However, the handle may also be provided in other ways, such as being non-flat, extending non-linearly, extending in a direction other than along a side, etc.

[0085] The protrusion may, for example, have a width and length that are shorter than the width and length of the marker unit.

[0086] The handle may further be provided with a friction-increasing configuration to increase friction, thereby simplifying grasping and holding the marker unit, for example with a laparoscopic instrument. Such a handle 14'' is shown diagrammatically in FIG. 12. For example, the friction-increasing configuration may include a surface texture on a surface of the handle, for example in the form of corrugations, dimples, etc. Additionally or alternatively, the friction-increasing configuration may include a coating, such as a high friction material.

[0087] In the embodiments discussed above, the marker unit is provided with a single handle. However, it is also possible to use more than two handles, such as two, three or more handles. For example, the two handles may be located on two opposite sides, or alternatively on two adjacent sides.

[0088] Herein, the invention has been described with reference to specific embodiments. However, several variations of the marker unit and the surgical system are possible. For example, one or both of the sets of radiopaque markers and optically detectable markers may be realized in other ways to form other forms of non-rotationally symmetric patterns. Furthermore, the correlation between the marker sets may be by placing the corner markers so that they overlap each other, but other ways of geometrically correlating the sets are also possible. The system may use other types of medical imaging, other types of displays, etc. Such and other obvious modifications must be considered to be within the scope of the invention as defined by the appended claims. It should be noted that the above-described embodiments illustrate rather than limit the invention, and that a person skilled in the art could design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed in parentheses shall not be interpreted as limiting the claims. The word "comprising" does not exclude the presence of other elements or steps than those recited in the claims. The word "a" preceding an element does not exclude the presence of a plurality of such elements.

Claims

1. 1. A marker unit for use in augmented reality assisted surgery, comprising: The marker unit comprises a flat body having two main surfaces, one of which is provided with a set of optically detectable markers and the opposite main surface is configured to be attached to a body organ and is preferably provided with an adhesive, the marker unit further comprising a set of radiopaque markers that are visible by a medical imaging system, both the optically detectable markers and the radiopaque markers being provided in a geometric pattern that does not have rotational symmetry, thereby enabling the rotational position of the marker unit to be determined, and the geometric patterns of the optically detectable markers and the radiopaque markers having a fixed and predetermined correlation.

2. The marker unit of claim 1 , wherein the flat body is made from a rigid material.

3. The marker unit of claim 1 or 2, wherein the flat body includes two layers fixedly bonded to each other, and the set of radiopaque markers is sandwiched between the two layers.

4. The marker unit according to claim 1 or 2, wherein the flat body has a polygonal shape, preferably a substantially rectangular shape, more preferably a square shape.

5. A marker unit according to claim 1 or 2, wherein the main faces have dimensions in two orthogonal directions in the range of 5 to 20 mm, preferably in the range of 5 to 15 mm, most preferably in the range of 7 to 12 mm.

6. The marker unit of claim 1 or 2, wherein the set of optically detectable markers comprises a plurality of geometric shapes distributed across the main surface.

7. The marker unit of claim 6 , wherein the plurality of geometric shapes comprises a plurality of QR codes.

8. The marker unit according to claim 1 or 2, wherein the main surface containing the set of optically detectable markers is hydrophobic and non-reflective.

9. 3. A marker unit according to claim 1 or 2, wherein the maximum extension length of the marker unit in any direction is less than 12 mm, preferably less than 10 mm.

10. The marker unit according to claim 1 or 2, wherein the marker unit is adapted for use in augmented reality assisted laparoscopic surgery.

11. The marker unit of claim 1 or 2, wherein the marker unit is rigid.

12. The marker unit of claim 1 or 2, further comprising a handle for a laparoscopic instrument.

13. The marker unit of claim 12 , wherein the handle is located on a portion of the flat body that is free of the adhesive and / or the optically detectable marker.

14. The marker unit of claim 12 , wherein the handle includes a protrusion that protrudes laterally and / or transversely relative to the main surface of the flat body.

15. 10. A system for augmented reality assisted surgery comprising the marker unit of claim 1, and further comprising a display and a controller, the controller: receiving medical imaging data indicative of the marker unit in a position intended for surgery; receiving received light image data indicative of the marker at the location intended for the procedure; correlating the medical imaging data and the received light image data based on the positions of the set of radiopaque markers in the medical imaging data and the set of optically detectable markers in the received light image data; providing an augmented reality image on the display, the augmented reality image including, at least in part, image data from the medical imaging data and the received light image data; A system configured to:

16. 16. The system of claim 15, wherein the medical imaging data is received from at least one of a computed tomography scan, a cone beam computed tomography scan, magnetic resonance imaging, and ultrasound imaging.

17. 17. The system according to claim 15 or 16, wherein the received light image data is received from a camera, preferably a laparoscopic camera.

Images