Surgical imaging systems, as well as corresponding systems, methods, and computer programs.

Abstract

Some aspects of this disclosure relate to a system (110) for a surgical imaging system (100), the system being configured to: obtain sensor data from sensors (150, 122) of the surgical imaging system, the sensor data representing distances between an objective lens (124) of the surgical imaging device (120) of the surgical imaging system and a plurality of points on the surface (10) of a surgical site imaged by the surgical imaging system; determine a three-dimensional contour of the surface of the surgical site based on the sensor data; determine the intersection between the surface of the surgical site and a three-dimensional preoperative scan of the surgical site based on the three-dimensional contour of the surface of the surgical site; and generate a visual overlay of at least a portion of the three-dimensional preoperative scan based on the intersection between the surface of the surgical site and the three-dimensional preoperative scan of the surgical site.

Description

Technical Field

[0001] This disclosure relates to medical imaging and visualization, particularly in the field of image-guided surgery (IGS). IGS utilizes various imaging techniques to provide real-time 3D visualization of patient anatomy and pathology during surgical procedures. Background Technology

[0002] Image-guided surgery (IGS) systems typically use the working distance (WD) of a surgical microscope as input, visualizing cross-sections of 3D preoperative data based on the assumption of a planar tissue surface. Because of this reliance on the planar tissue surface assumption, this approach has limitations when visualizing non-planar, irregular tissue surfaces, which can lead to structures located in different tissue layers (higher or lower) being incorrectly represented as if they were on that surface. This can result in an inaccurate representation of the surgical field, where structures located in different tissue layers may be visualized as if they were on that surface. Such inaccuracies can mislead the surgeon during the surgical procedure.

[0003] Furthermore, such systems cannot adapt their visualization to irregular tissue surfaces, preventing them from fully utilizing available 3D preoperative data and thus missing the potential to provide a more accurate, detailed, and realistic representation of the surgical field. Consequently, these systems may fail to provide comprehensive surgical guidance, potentially compromising the precision of surgical procedures, especially in complex microsurgical settings.

[0004] For example, IGS is an important tool for assisting surgeons by overlaying preoperative 3D imaging data (e.g., MRI scans) onto a live surgical view provided by a microscope. Current visualization methods typically rely on alpha blending techniques to overlay subsurface anatomical and pathological structures onto a live image. However, these methods fail to effectively convey crucial information about the depth of these subsurface structures relative to the tissue surface. The lack of readily available, intuitive depth information can hinder surgical precision and decision-making for surgeons.

[0005] It may be desirable to provide an improved concept for image-guided surgery. Summary of the Invention

[0006] This expectation is addressed by the subject matter of the independent claims. For example, an improved visualization technique may be needed that clearly shows the distance of subsurface structures from the tissue surface during surgery.

[0007] The various examples disclosed herein are based on the finding that the assumption of a planar tissue surface has limitations when visualizing non-planar, irregular tissue surfaces, which may lead to structures located in different tissue layers (higher or lower) being incorrectly represented on that surface. To overcome these limitations, in the proposed concept, instead of using a single distance value, multiple distance values ​​are determined for multiple points on the surface of the surgical site. Based on these distances, a three-dimensional profile of the surface is generated and used to generate a visual overlay. To generate the visual overlay, preoperative data is obtained. The preoperative data indicates a three-dimensional preoperative scan including the structure. The distance from the surface to the structure is determined. Based on this distance between the three-dimensional profile of the surface and the structure in the three-dimensional preoperative scan, a more realistic and accurate visual overlay can be provided for image-guided surgery.

[0008] Some aspects of this disclosure relate to a system for a surgical imaging system. The system includes one or more processors and one or more storage devices. The system is configured to acquire sensor data from sensors of the surgical imaging system. The sensor data represents distances between the objective lens of the surgical imaging device of the surgical imaging system and multiple points on the surface of a surgical site imaged by the surgical imaging system. The system is configured to determine a three-dimensional profile of the surface of the surgical site based on the sensor data. The system is configured to acquire preoperative data indicating a three-dimensional preoperative scan including a structure. The system is configured to determine the distance between the surface of the surgical site and the structure of the three-dimensional preoperative scan of the surgical site based on the three-dimensional profile of the surface of the surgical site and the preoperative data. The system is configured to generate a visual overlay of at least a portion of the structure of the three-dimensional preoperative scan based on the distance between the surface of the surgical site and the structure of the three-dimensional preoperative scan of the surface of the surgical site. This allows for the generation of a more realistic and accurate visual overlay that accurately distinguishes features that will be shown (or hidden) at or above the surface from features that will be shown (or hidden) below the surface.

[0009] Generally, the three-dimensional contour of the surgical site surface can be used to determine which features included in the three-dimensional preoperative scan are on or above the surgical site surface, and which features are below the surface. Therefore, the cross-section of the three-dimensional preoperative scan can be determined based on the three-dimensional contour of the surgical site surface. In other words, the system can be configured to generate a cross-section of the three-dimensional preoperative scan based on the intersection between the surface of the surgical site and the three-dimensional preoperative scan of the surgical site surface, and to generate the visual overlay based on the generated cross-section. The cross-section can show the features included in the three-dimensional preoperative scan according to the current surface of the surgical site.

[0010] The primary reason for using a three-dimensional profile of a surface instead of a single working distance is the ability to more accurately determine which features are at or above the surface and which features are below it. For example, the system can be configured to determine a first portion of the three-dimensional preoperative scan that will be visualized at or above the surface of the surgical site, and a second portion that will be visualized below the surface of the surgical site, based on the distance between the surface of the surgical site and the structure of the three-dimensional preoperative scan of the surface of the surgical site. This can improve the accuracy of the resulting visual overlay.

[0011] To assist surgeons during surgical procedures, a clear visual distinction can be made between features at or above the surface and features below the surface. The system can be configured to determine a first portion (e.g., features at or above the surface) and a second portion (e.g., features below the surface) of the three-dimensional preoperative scan based on the distance between the surface of the surgical site and the structure of the surface of the surgical site in the three-dimensional preoperative scan. The system can be configured to include the first portion in the visual overlay in a first visualization style. The system can be configured to perform one of the following: a) include the second portion in the visual overlay in a second visualization style, or b) omit the second portion from the visual overlay. In this way, features at or above the surface and features below the surface can be clearly distinguished by the surgeon.

[0012] Generally, the goal is to represent the surface of the surgical site more accurately. This can be achieved by creating a three-dimensional model of the surface of the surgical site. For example, the three-dimensional contour of the surface of the surgical site can include a three-dimensional model of the surface of the surgical site.

[0013] For example, various options exist for generating three-dimensional surface profiles based on sensor data. For instance, sensor data can include a three-dimensional surface scan of the surface of the surgical site. A three-dimensional surface scan can be used to accurately model the surface of the surgical site. Generally, various techniques exist for performing such three-dimensional surface scans. For example, sensor data can include a representation of a structured light pattern emitted by a structured light emitter of a surgical imaging system. Structured light-based techniques have the advantage that the main optical imaging sensor of the corresponding surgical imaging device can be combined with the structured light emitter for this purpose without requiring additional sensors. This can lead to a low-cost, low-workload approach. Alternatively, sensor data can be sensor data from a time-of-flight sensor of the surgical imaging system. Time-of-flight sensors tend to require more hardware (i.e., emitter and sensor) but can be used independently of the surgical imaging device, for example, without interrupting the operation of the surgical imaging device. Finally, sensor data can include multiple image frames representing multiple different focal lengths or working distances. For each focal plane or working distance, the system can determine which portions of the image frame are in focus or out of focus to determine the three-dimensional profile of the surface of the surgical site. For example, the system can be configured to determine the three-dimensional contour of the surgical site surface based on contrast and / or on the presence of spatial frequencies above a predefined spatial frequency threshold in corresponding image frames across multiple image frames. Using these techniques (contrast, presence of high-frequency spatial frequencies), it is possible to classify portions of image frames as being in focus or out of focus. This may eliminate the need for additional hardware. However, the surgical imaging device may be unavailable to surgeons when capturing image frames at different focal lengths or working distances.

[0014] In the example, the system can be configured to generate the visual overlay such that the structure from the 3D preoperative scan is visualized differently for different defined distances. That is, the distance of the (subsurface) structure from the surface of the surgical site (e.g., the tissue surface) can be visualized during surgery. Visualizing the structure at different distances enables 3D perception of the structure. In this way, the usability of preoperative scans can be improved.

[0015] In the example, the structure from the 3D preoperative scan can be visualized differently using color coding. This means that visualization of the structure at different distances can be achieved using color coding. Color coding is a visualization technique used to distinguish different parts of a structure or different structures. Color coding can enhance the interpretability of complex scans by assigning different colors to individual regions or features. Therefore, color coding can allow for improved depth perception of structures, such as the distance of different parts of the structure from the tissue surface during surgery.

[0016] In the example, the system can be configured to automatically adjust the color scale based on the depth range of the 3D contour. In this way, visualization can be adjusted for the current depth range present in the current field of view.

[0017] In the example, the system can be configured to visualize, for example, the structure from a three-dimensional preoperative scan at or above the surface of the surgical site in a partially transparent manner. This means that the transparency of the structure can be adjusted to improve the visibility of overlaid data, such as the three-dimensional contour of the surgical site surface. For example, the structure can be overlaid on the three-dimensional contour of the surgical site surface, partially obscuring information provided by a microscope. By adjusting the transparency, the overlay can be adjusted so that the perception of the three-dimensional contour of the surgical site surface and the preoperative scan is sufficient for the user.

[0018] In this example, the system can be configured to obtain user data indicating the transparency of the structure in the 3D preoperative scan, and to adjust the transparency of the structure based on the user data. The user data can be received from an input device, allowing the user to provide information about the target transparency. In this way, the transparency of the structure can be adjusted based on user input.

[0019] In the example, the structure in the 3D preoperative scan can be visualized differently by applying a depth threshold, allowing specific portions of the structure to be focused within a certain depth range. This emphasizes key parts of the structure, such as those near the surgical site.

[0020] One aspect of this disclosure relates to a surgical imaging system comprising the system, the surgical imaging apparatus, and the sensor for providing the sensor data. For example, the surgical imaging apparatus may be a microscope or an excimer. Surgical microscopes and surgical excimers are highly relevant for use in surgical procedures.

[0021] For example, the sensor used to provide sensor data could be an imaging sensor in a surgical imaging device. This is likely the case if structured light-based or focal plane / working distance-based techniques are used.

[0022] Alternatively, the sensor used to provide sensor data can be separate from the surgical imaging device. This may be the case if structured light-based techniques with separate sensors or time-of-flight techniques are used.

[0023] Some aspects of this disclosure relate to a corresponding method for a surgical imaging system. The method includes obtaining sensor data from sensors of the surgical imaging system. The sensor data represents distances between an objective lens of the surgical imaging device of the surgical imaging system and multiple points on the surface of a surgical site imaged by the surgical imaging system. The method includes determining a three-dimensional profile of the surface of the surgical site based on the sensor data. The method includes obtaining preoperative data indicating a three-dimensional preoperative scan including a structure. The method includes determining a distance between the surface of the surgical site and the structure of the three-dimensional preoperative scan of the surgical site based on the three-dimensional profile of the surface of the surgical site and the preoperative data. The method includes generating a visual overlay of at least a portion of the structure of the three-dimensional preoperative scan based on the distance between the surface of the surgical site and the structure of the three-dimensional preoperative scan of the surface of the surgical site.

[0024] One aspect of this disclosure relates to a computer program having program code for performing the methods described above when the computer program is run on a processor. Attached Figure Description

[0025] The following will describe some examples of devices and / or methods by way of example and with reference to the accompanying drawings, in which:

[0026] Figure 1a A block diagram of an example system for surgical imaging is shown;

[0027] Figure 1b A schematic diagram illustrating an example of a surgical imaging system is shown;

[0028] Figure 1c A schematic diagram illustrating the determination of a planar tissue surface is shown;

[0029] Figure 1d and 1e A schematic diagram illustrating the determination of the three-dimensional profile of a tissue surface is shown;

[0030] Figure 2 A flowchart illustrating an example of a method for a surgical imaging system is shown;

[0031] Figure 3a and 3b An illustration of IGS operating under the assumption of a planar tissue surface is shown;

[0032] Figure 4a and 4b An illustration of IGS based on three-dimensional tissue scanning is shown; and

[0033] Figure 5A block diagram of an example system including an imaging device and a computer system is shown. Detailed Implementation

[0034] Various examples will now be described more fully with reference to the accompanying drawings, some of which illustrate certain aspects. For clarity, the thickness of lines, layers, and / or regions may be exaggerated in the drawings.

[0035] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items and may be abbreviated to “ / ”.

[0036] Figure 1a A surgical imaging system 100 is shown. Figure 1b A block diagram of an example of system 110 (shown in the diagram). System 110 is a component of surgical imaging system 100 and can be used to control various aspects of surgical imaging system 100. Specifically, it can be used to acquire and / or process imaging sensor data or other sensor data from the optical imaging sensor of imaging device 120 of surgical imaging system by various means, which will be described in more detail below. In addition, system 110 can be configured to control additional aspects of surgical imaging system, such as providing display signals to various displays of surgical imaging system.

[0037] Generally, system 110 can be considered a computer system. System 110 includes one or more processors 114 and one or more storage devices 116. Optionally, system 110 also includes one or more interfaces 112. The one or more processors 114 are coupled to one or more storage devices 116 and one or more interfaces 112. Generally, the functionality of system 110 can be achieved by combining one or more processors 114 with one or more interfaces 112 (for communication with one or more other components of surgical imaging system 100 and external to surgical imaging system 100, such as optical imaging sensor 122 of imaging device 120, another sensor 150, or transmitter 155, all within...). Figure 1b The system 110 may include, for example, machine-readable instructions within one or more storage devices 116 (as shown in the diagram) for exchanging data / information. Generally, the functionality of one or more processors 114 can be implemented by one or more processors 114 executing machine-readable instructions. Therefore, any feature attributable to one or more processors 114 can be defined by one or more instructions from a plurality of machine-readable instructions. The system 110 may include, for example, machine-readable instructions within one or more storage devices 116.

[0038] As outlined above, system 110 is part of surgical imaging system 100, which includes various other components besides system 110. For example, surgical imaging system 100 includes imaging device 120 and may include one or more additional components, such as robotic arm 130. Figure 1b A schematic diagram illustrating an example of such a surgical imaging system 100, particularly a surgical microscope system 100, is shown. Hereinafter, the surgical imaging system 100 may also be referred to as a surgical microscope system 100, which includes a (surgical) microscope as an imaging device 120. However, the proposed concept is not limited to such embodiments. The surgical imaging system 100 can be based on various (single or multiple) imaging devices, such as one or more microscopes, one or more endoscopes, or one or more exoscopy devices (sometimes also called external telescopes). Exoscopy is a camera-based imaging system, particularly a camera-based 3D imaging system, suitable for providing images of surgical sites with high magnification and large depth of field. Compared to a microscope, which can be used via an eyepiece, an exoscopy is used only via a display modality (such as a monitor or head-mounted display). Therefore, the surgical imaging system can alternatively be a surgical endoscopy system or a surgical exoscopy system. The following illustrations assume that the imaging device 120 is a surgical microscope for use in neurosurgery and that the surgical imaging system 100 is a surgical microscope system 100 for use in neurosurgery.

[0039] Therefore, the surgical imaging system or surgical microscope system 100 may include imaging devices such as microscope 120, endoscope, or exoscope. Generally, a microscope, such as microscope 120, is an optical instrument suitable for examining objects too small to be examined (alone) by the human eye. For example, a microscope can provide optical magnification of a sample. In this concept, optical magnification is also provided for the optical imaging sensor. Therefore, microscope 120 includes an optical imaging sensor coupled to system 110. Microscope 120 may also include one or more optical magnifying elements, such as objectives (i.e., lenses), for magnifying views on the sample. For example, the surgical imaging device or microscope 120 is often also referred to as the "optical device carrier" of the surgical imaging system.

[0040] Various types of surgical imaging devices exist. If the surgical imaging device is used in the medical or biological field, the object observed through the surgical imaging device can be, for example, a sample of organic tissue arranged in a petri dish or present in a part of a patient's body. In the various examples presented herein, the imaging device 120 can be a microscope of a surgical microscope system, i.e., a microscope used during surgical procedures such as neurosurgical procedures or ophthalmic procedures (i.e., eye surgery). Therefore, the object observed through the surgical imaging device and shown in a digital view generated by imaging sensor data provided by (optionally) optical imaging sensors (which is shown in the field of view of the surgical imaging device) can be a sample of the patient's organic tissue, and in particular, can be a surgical site operated on by the surgeon during the surgical procedure, such as the brain or the eye. However, the proposed concept is also applicable to other types of surgery, such as cardiac surgery.

[0041] Figure 1b A schematic diagram is shown of a surgical imaging system 100 including system 110 and microscope 120, and in particular an example of a surgical microscope system 100 for use in neurosurgery. Figure 1b The surgical microscope system 100 shown includes several optional components, such as a base unit 105 (including system 110) with a (rolling) support, an eyepiece display 140a disposed at the microscope 120, an auxiliary display 140b disposed at the base unit, and a robotic arm 130 that holds the microscope 120 in place and is coupled to the base unit 105 and the microscope 120. Generally, these optional and non-optional components can be coupled to system 110, which can be configured to control and / or interact with the respective components.

[0042] In the proposed concept, the system is used for the purpose of performing image-guided surgery. Image-guided surgery (IGS) is a surgical tool in which preoperative and intraoperative imaging is used to guide the surgeon and track the precise position of the surgical tools relative to the patient's anatomy during the procedure. This technology is designed to help surgeons navigate more precisely within the body, particularly in areas that are difficult to observe or too risky to explore using conventional surgical methods. Several types of imaging modalities exist that are frequently used in image-guided surgery, including computed tomography (CT) scans and magnetic resonance imaging (MRI). By integrating images generated from preoperative scans with a digital view of the surgical site, continuous visual guidance is provided to the surgeon on screen, showing them the location of three-dimensional anatomical features of interest.

[0043] However, this disclosure does not relate to techniques for performing preoperative scans or registering preoperative scans with surgical sites, but rather to techniques for overlaying a visual representation of a preoperative scan onto a digital view of the surgical site. Therefore, system 110 is configured to acquire sensor data from sensors of a surgical imaging system. The sensor data represents the distance between the objective lens 124 of the surgical imaging device 120 of the surgical imaging system and multiple points on the surface 10 of the surgical site imaged by the surgical imaging system. System 110 is configured to determine a three-dimensional contour 10a of the surface of the surgical site based on the sensor data. Figure 1e and 4b (As shown in the diagram). System 110 is configured to acquire preoperative data indicating a three-dimensional preoperative scan including structures (e.g., structures 32 or 34 as described below). System 110 is configured to determine the distance between the surface of the surgical site and the structures of the three-dimensional preoperative scan of the surgical site based on the three-dimensional contour of the surgical site surface and the preoperative data. System 110 is configured to generate a visual overlay of at least a portion of the structures of the three-dimensional preoperative scan based on the distance between the surface of the surgical site and the structures of the three-dimensional preoperative scan of the surgical site surface. For example, surgical imaging system 100 may include system 110, surgical imaging device 120, and sensors for providing sensor data. For example, as shown in the diagram. Figure 1b As shown, the sensor used to provide sensor data can be the optical imaging sensor 122 of the surgical imaging device 120. Alternatively, as shown... Figure 1b As shown, the sensor 150, which provides sensor data, can be separated from the surgical imaging device.

[0044] The action process taken in the proposed concept is Figures 1c to 1e Example in. exist Figure 1c The text illustrates commonly used methods. Figure 1c A schematic diagram illustrating the determination of a planar tissue surface is shown. (e.g.) Figure 1c As shown, in other surgical imaging systems, for the purpose of image-guided surgery, the following assumption is made: the surgical site is substantially flat and can therefore be represented by plane 10b, which is based on the working distance between the objective lens 124 and the portion of the surgical site in focus. (As shown from...) Figure 1c Obviously, this assumption is not always useful, because many surgical sites are surgical cavities with a high degree of unevenness.

[0045] exist Figure 1d and 1e , illustrate the proposed method. Figure 1d and 1e A schematic diagram illustrating the determination of the three-dimensional profile of a tissue surface is shown. Note that... Figure 1d and 1eThe proposed method for generating 3D contours is illustrated only in two dimensions to simplify the figures. However, the proposed method extends the methods illustrated in these figures to three dimensions. For example, a 3D illustration of the concept is shown below. Figure 4a and 4b As shown. Figure 1d and 1e As shown in 4a and 4b, instead of the assumption of a planar surface, the surface of the surgical site is accurately modeled in three dimensions. Figure 1d The image illustrates multiple distances between the objective lens 124 and the surface 10 of the surgical site. Figure 1e The diagram illustrates a obtained three-dimensional (or, in this illustration, more precisely, two-dimensional) contour 10a of the surface of the surgical site. In practice, the three-dimensional contour of the surgical site surface can include a three-dimensional model of the surface. A three-dimensional model of a surface is a mathematical or numerical representation of the surface's morphology in three dimensions. Unlike a 2D model that only includes width and height, a three-dimensional model also includes depth. The three-dimensional contour of the surgical site surface can include the three-dimensional topology of the surgical site. It can represent the distances between multiple points on the surface of the surgical site and the objective lens 124 of the surgical imaging device. For example, the three-dimensional contour of the surgical site surface can be a three-dimensional point cloud of the surgical site.

[0046] In the proposed concept, the three-dimensional contour of the surgical site surface is determined based on sensor data, where the sensor data represents the distances between the objective lens 124 of the surgical imaging device 120 of the surgical imaging system and multiple points on the surface 10 of the surgical site imaged by the surgical imaging system. This can be accomplished by scanning the surface of the surgical site using 3D scanning technology. In other words, the sensor data can include a three-dimensional surface scan of the surgical site. Many different 3D scanning techniques can be used for this purpose.

[0047] For example, structured light-based scanning can be used. Structured light 3D scanning is an optical method for capturing the three-dimensional contours of an object using projected light patterns and a camera, such as an optical imaging sensor 122 for a surgical imaging device 120 or another camera sensor 150. The process involves projecting a known light pattern (e.g., stripes or grids) onto a surface and using a camera to capture images of the distorted pattern from different angles. For example, a surgical imaging system may include a structured light emitter 155 (… Figure 1bA known pattern (typically a digital projector or laser) is projected onto the surface of the surgical site. The light pattern becomes distorted when it conforms to the geometry of the surgical site. These distortions provide clues about the depth and contours of the surface. An optical imaging sensor 122 or another camera sensor 150 can then be used to capture image frames of the distorted light pattern appearing on the surface of the surgical site. For example, sensor data may include a representation of a structured light pattern emitted by a structured light emitter 155 of the surgical imaging system. The system can be configured to process the representation of the structured light pattern to calculate the positions of said plurality of points on the surface of the surgical site in three-dimensional space based on the known geometry of the pattern and based on the angles of the camera and projector and the way the pattern is distorted on the object.

[0048] Alternatively, a time-of-flight (ToF) sensor can be used to determine the three-dimensional contour of the surgical site. Therefore, the sensor data can be sensor data from the time-of-flight sensor 150 of the surgical imaging system. Thus, the surgical imaging system can include a corresponding transmitter 155 for the time-of-flight sensor 150. The time-of-flight sensor can similarly provide a point cloud of the surgical site, which the system uses to determine the three-dimensional contour of the surgical site.

[0049] Another technique involves capturing multiple image frames at multiple different focal planes or multiple different working distances. In other words, sensor data can include multiple image frames representing multiple different focal lengths or working distances. It is assumed that any portion of a corresponding image frame that is in focus at a given focal plane or working distance has its surface (in the depth direction) located at that focal plane or working distance. When using this technique, image sharpness is used to determine which portions of the image frame are in focus at a given focal length or working distance. The sharpness of different portions can be determined based on the contrast of the corresponding image and / or based on the proportion of high spatial frequencies in the corresponding image frame. For example, the system can be configured to determine the three-dimensional contour of the surface of the surgical site based on contrast and / or based on the presence of spatial frequencies above a predefined spatial frequency threshold in the corresponding image frames among multiple image frames, for example, by determining that the surface of a portion of the surgical site is located at a focal plane or working distance (at which the corresponding portion of the surgical site appears sharp in the corresponding image frame representing that focal plane or working distance).

[0050] For example, the system can be configured to determine the contrast of a portion (e.g., a block of pixels) of a given image frame, for instance, by determining the ratio of the standard deviation to the mean of the pixels in the image, or by performing a kernel-based comparison between a single pixel and its neighbors (i.e., adjacent pixels). The sharper a portion of an image appears, the higher the contrast of the image is generally.

[0051] The system can also be configured, for example, to determine the spatial frequency distribution of a corresponding portion (e.g., pixel block) of the imaging sensor data by performing a 2D Fourier transform on the corresponding image frame. The higher the proportion of high spatial frequencies in a corresponding portion of the image frame, the more visible fine-grained structures are in that portion of the image frame; this is the case if the corresponding portion of the image frame containing fine-grained structures is perceived as sharp. By determining the integral of the portion of the spatial frequency distribution above a predefined frequency, a quantitative measure can be determined, which can be used to compare the presence of fine-grained structures between different image frames, for example, comparing the sharpness of two corresponding portions of the field of view in different image frames.

[0052] Once the three-dimensional contour of the surgical site surface is determined, it can be used to determine the distance between the surgical site surface and the structures in the three-dimensional preoperative scan of the surgical site. This is particularly relevant to distinguishing between three-dimensional features / structures that will be shown on or above the surface of the surgical site and those that will be shown below the surface (or not shown at all). In effect, the distance between the surgical site surface and the structures in the three-dimensional preoperative scan of the surgical site can be used to "cut open" the three-dimensional preoperative scan to visualize which anatomical features are on or below the surface of the surgical site.

[0053] For example, the system can be configured to generate visual overlays such that the structure of a 3D preoperative scan is visualized differently for different defined distances. That is, the structure of a 3D preoperative scan can be visualized differently depending on its distance from the surface of the surgical site. For example, when the structure intersects the surface of the surgical site (i.e., the distance is 0), the structure can be included in the visual overlay in a first visualization style, such as a first thicker (i.e., stronger lines) or brighter (i.e., lines with brighter or more prominent colors). For example, when the structure does not intersect the surface of the surgical site (i.e., the distance is greater than 0), the structure can be included in the visual overlay in a second visualization style (e.g., a second softer visualization style, with thinner lines, lines with darker colors, less prominent colors, or dashed lines).

[0054] For example, three-dimensional preoperative scans can be visualized differently using color coding of structures (see also...). Figures 6a-6bIn other words, the color of a structure can be based on its distance from the surface of the surgical site. That is, the color can vary with depth, with lighter shading used for closer structures and darker or cooler shading (such as blue) used for darker structures. For example, when a part of a structure intersects the surface of the surgical site (i.e., the distance is 0), that part of the structure can be included in the visual overlay with a first color (e.g., red). Conversely, when a structure does not intersect the surface of the surgical site (i.e., the distance is greater than 0), a part of the structure can be included in the visual overlay with a second color (e.g., a bluish hue). For example, the greater the distance between a part of the structure and the surface of the surgical site, the bluer that part of the structure can be. This means that color coding can improve depth perception by enhancing spatial understanding, and thus depth perception, which is achieved by using color gradients and / or different hues to represent different distances of a structure from the surface of the surgical site.

[0055] In the example, the system can be configured to visualize the structure of a 3D preoperative scan at or above the surface of the surgical site in a partially transparent manner. Partially transparent visualization allows deeper structures to remain visible while preserving the clarity of foreground elements. In other words, partially transparent visualization can allow for maintaining the visibility of the surface of the surgical site (e.g., a biopsy).

[0056] In this example, the system can be configured to acquire user data indicating the transparency of the structures in the 3D preoperative scan, and to adjust the transparency of the structures based on the user data. For example, the user data can be provided by the user, such as received from a user input device. In this way, the user can adjust the transparency level of the structures. Alternatively, the user data can indicate a target color scale range and / or a specific structure to be displayed. In this way, the user can adjust the visualization of color coding and / or distinguish different anatomical structures or tissue types.

[0057] In the example, the system can be configured to automatically adjust the color scale based on the depth range of the three-dimensional contour. By automatically adjusting the color scale, the system can highlight structures within a specific depth range, thereby improving clarity and focus during navigation. For example, in surgery, the system can adjust the color scale to assign bright colors (e.g., red) to target depth ranges, such as tumors located below the surface, while using softer hues (e.g., blue or green) to represent deeper or less relevant areas. This dynamic adjustment of the color scale can enhance visualization, reduce cognitive load, improve spatial awareness, and / or help users avoid critical areas such as blood vessels or functional tissues, thereby ensuring safer and more efficient procedures.

[0058] In the example, the structure in the 3D preoperative scan can be visualized differently by applying a depth threshold, allowing portions of the structure within a certain depth range to be focused. By applying a depth threshold to the 3D preoperative scan, the system can focus on specific portions of the structure within a defined depth range, allowing for targeted visualization of critical areas. This approach improves clarity by reducing visual clutter from irrelevant deeper structures and achieving precise emphasis on critical areas, such as those near the surgical site. For example, highlighting superficial areas near the surface can help surgeons identify and navigate critical structures, such as blood vessels or nerves, with greater accuracy. This focused visualization not only enhances spatial awareness but also minimizes cognitive load, thus supporting safer and more efficient surgical planning and execution.

[0059] For example, the system can be configured to generate a cross-section of a three-dimensional preoperative scan based on the intersection between the surface of the surgical site and a three-dimensional preoperative scan of the surgical site. In other words, the system can be configured to determine where the surface of the surgical site intersects with the three-dimensional preoperative scan of the surgical site. A visual overlay can then be generated based on the generated cross-section, displaying three-dimensional features included in the three-dimensional preoperative scan according to the cross-section, for example, by “cutting through” the three-dimensional features included in the three-dimensional preoperative scan at the cross-section.

[0060] To aid in interpreting the three-dimensional preoperative scan, a distinction can be made between features located at or above the surface and features located below the surface. Therefore, the system can be configured to determine a first portion 42 of the three-dimensional preoperative scan that will be visualized at or above the surface of the surgical site based on the distance between the surface of the surgical site and the structures of the three-dimensional preoperative scan of the surgical site surface. Figure 4b (shown in the image) and the second part 44, which visualizes the three-dimensional preoperative scan below the surface of the surgical site (also shown in the image). Figure 4b(As shown in the diagram). For example, the first portion of a three-dimensional preoperative scan may be a three-dimensional feature / structure included in the three-dimensional preoperative scan that is located at the intersection defined by the three-dimensional contour of the surface of the surgical site and thus intersects with the cross section of the three-dimensional preoperative scan. The second portion of a three-dimensional preoperative scan may be a three-dimensional feature / structure included in the three-dimensional preoperative scan that is below the intersection defined by the three-dimensional contour of the surface of the surgical site (i.e., further away from the surface of the surgical site than objective lens 124). The system may be configured to determine such a first portion 42 and a second portion 44 of the three-dimensional preoperative scan based on the intersection between the surface of the surgical site and the three-dimensional preoperative scan of the surface of the surgical site. These portions may be visualized differently in a visual overlay. For example, the first portion in the visual overlay may be included in a first visualization style, such as a first thicker (i.e., stronger lines) or brighter (i.e., lines with brighter or more prominent colors) visualization style. The second part can be included in a second visualization style (e.g., a second, softer visualization style with thinner lines, darker colors, or less prominent colors), or omitted from the visual overlay.

[0061] For example, as described above, the system generates a visual overlay of at least a portion of the three-dimensional preoperative scan (e.g., based on a portion of the current field of view, a portion at or above the surface, and / or a portion at or below the surface) based on the distance between the surface of the surgical site and a three-dimensional preoperative scan of the surface of the surgical site. For example, at least some aspects of the generation of the visual overlay can be performed by a separate image-guided surgical system (not shown). If such a separate image-guided surgical system is used, the three-dimensional contour of the surface of the surgical site can be provided as input to the separate image-guided surgical system, and the visual overlay can be obtained from the image-guided surgical system. In this case, some of the operations described above related to the cross-section of the three-dimensional preoperative scan and the generation of the visual overlay can be performed by the image-guided surgical system. The system can be configured to combine the visual overlay with a digital view of the surgical site based on imaging sensor data from the optical imaging sensor 122 of the surgical imaging device 120 (e.g., overlaying the visual overlay on the digital view). The system can be configured, for example, to provide a display signal having the combined digital view and visual overlay to at least one of the multiple display devices 140a, 140b of the surgical imaging system 100.

[0062] For example, the display signal may be a signal used to drive (e.g., control) the corresponding display devices 140a, 140b. For example, the display signal may include video data and / or control commands for driving the display devices. For example, the display signal may be provided via one of one or more interfaces 112 of the system. Therefore, the system 110 may include a video interface 112 adapted to provide display signals to the display devices 140a, 140b of the surgical imaging system 100.

[0063] In the proposed surgical imaging system, an optical imaging sensor is used to provide imaging sensor data and optionally provides sensor data. Therefore, an optical imaging sensor, which may be part of the surgical imaging apparatus 120 (e.g., a microscope), can be configured to generate imaging sensor data and / or sensor data. For example, the optical imaging sensor of the surgical imaging apparatus 120 may include or may be an APS (Active Pixel Sensor) based imaging sensor or a CCD (Charge-Coupled Device) based imaging sensor. For example, in an APS-based imaging sensor, light at each pixel is recorded using a photodetector and an active amplifier of the pixel. APS-based imaging sensors are typically based on CMOS (Complementary Metal-Oxide-Semiconductor) or S-CMOS (Scientific CMOS) technology. In a CCD-based imaging sensor, incident photons are converted into electronic charges at the semiconductor-oxide interface, and these electronic charges then move between capacitor cells in the imaging sensor through the imaging sensor's circuitry to perform imaging. System 110 can be configured to acquire (i.e., receive or read out) imaging sensor data and / or sensor data from the optical imaging sensor. Imaging sensor data and / or sensor data can be obtained by receiving imaging sensor data and / or sensor data from an optical imaging sensor (e.g., via interface 112), by reading imaging sensor data and / or sensor data from the memory of the optical imaging sensor (e.g., via interface 112), or by reading imaging sensor data and / or sensor data from the storage device 116 of system 110, for example, after the imaging sensor data and / or sensor data has been written to storage device 116 by the optical imaging sensor or another system or processor. System 110 can be configured to obtain (i.e., receive or read out) preoperative data from an optical imaging sensor and / or external storage device. Preoperative data can be obtained by receiving preoperative data from an optical imaging sensor (e.g., via interface 112), by reading imaging sensor data from the memory of the optical imaging sensor (e.g., via interface 112), or by reading imaging sensor data from storage device 116 of system 110, for example, after the preoperative data has been written to storage device 116 by the optical imaging sensor or another system or processor, such as including external storage device.

[0064] One or more interfaces 112 of system 110 may correspond to one or more inputs and / or outputs for receiving and / or sending information within a module, between modules, or between modules of different entities. This information may be a digital (bit) value based on a specified code. For example, one or more interfaces 112 may include an interface circuitry configured to receive and / or send information. One or more processors 114 of system 110 may be implemented using one or more processing units, one or more processing devices, or any means for processing, such as a processor, computer, or programmable hardware components that can operate with appropriately adapted software. In other words, the described functionality of one or more processors 114 may also be implemented in software, which then executes on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. One or more storage devices 116 of system 110 may include at least one element from the group of computer-readable storage media, such as magnetic or optical storage media, such as hard disk drives, flash memory, floppy disks, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or network storage devices.

[0065] In conjunction with the proposed concept or the preceding or following text (e.g., Figures 2 to 6b The one or more examples described herein refer to further details and aspects of system 110 and surgical imaging system 100. System 110 and surgical imaging system 100 may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0066] Figure 2 A surgical imaging system (e.g., for use in conjunction with) is shown. Figures 1a to 1eA flowchart illustrating an example of a method corresponding to the surgical imaging system 100 described herein. The method includes obtaining 210 sensor data from sensors of the surgical imaging system, the sensor data representing distances between the objective lens of the surgical imaging device of the surgical imaging system and multiple points on the surface of the surgical site imaged by the surgical imaging system. The method includes determining 220 a three-dimensional contour of the surface of the surgical site based on the sensor data. Method 200 includes determining 230 the distance between the surface of the surgical site and a three-dimensional preoperative scan of the structure of the surgical site based on the three-dimensional contour of the surface of the surgical site and preoperative data. Method 200 includes generating 240 a visual overlay of at least a portion of the structure of the three-dimensional preoperative scan based on the distance between the surface of the surgical site and the structure of the three-dimensional preoperative scan of the surface of the surgical site. Method 200 includes generating 250 a visual overlay of at least a portion of the structure of the three-dimensional preoperative scan based on the distance between the surface of the surgical site and the structure of the three-dimensional preoperative scan of the surface of the surgical site.

[0067] For example, method 200 can be derived from combining Figures 1a to 1e One of the surgical imaging systems described herein is used for implementation. Combined with... Figures 1a to 1e The features described in the (surgical) imaging system can also be included in the corresponding method 200.

[0068] In conjunction with the proposed concept or the preceding or following text (e.g., Figures 1a to 1e , Figures 3a to 6b The examples described herein refer to further details and aspects of the method. The method may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples described above or below.

[0069] Various examples of this disclosure relate to a concept for accurate 3D surface adaptation visualization in image-guided surgery, referred to as 3D Adaptive IGS.

[0070] The proposed concept specifically addresses the use of IGS in microsurgical procedures where accuracy is paramount. By improving the accuracy of visualized data, the proposed concept can significantly enhance surgical outcomes in these high-precision environments.

[0071] A key insight of the proposed concept is the integration of 3D surface scanning capabilities into image-guided surgery (IGS) systems to allow for more accurate visualization of a patient's anatomy during surgery. The proposed concept addresses the problems associated with the planar tissue surface assumption commonly used in IGS systems by including a method that scans the actual non-planar tissue surface and generates its accurate 3D representation. This surface data can then be used to instruct the IGS system to present surface-fitted cross-sections of the preoperative 3D structures. Regardless of whether these structures are located in layers with higher or lower specific surface areas, they are accurately represented in their actual locations, rather than being incorrectly shown as being on that surface. By providing a more accurate visualization that more closely conforms to the actual tissue topology, the system provides surgeons with more precise guidance during the procedure. This, in turn, helps reduce the risk of surgical inaccuracies, which enhances patient safety and potentially leads to improved surgical outcomes.

[0072] The proposed concept directly addresses a potential pain point experienced by surgeons relying on image-guided surgery (IGS) systems for accurate visualization during surgical procedures: structures located in different tissue layers are inaccurately depicted due to the assumption of planar tissue surfaces. By providing more accurate, surface-fitting visualization of the surgical field, this invention allows surgeons to operate with greater precision and confidence. This not only enhances the surgeon's ability to perform safe and efficient procedures but also reduces the risk of misinterpretation of IGS data of the surgical field, thereby directly addressing the aforementioned pain point.

[0073] Figure 3a and 3b An illustration of IGS operating under the assumption of a planar tissue surface is shown. Figure 3a The surgical microscope objective 124, tissue surface 10, and segmented preoperative 3D structure 30 from a three-dimensional preoperative scan are shown. Figure 3a The working distance 300 between the objective lens 124 and the center of the tissue surface 10 is also shown. Figure 3a An illustration of IGS is shown to address surgeons' need for 3D preoperative anatomical and pathological information. Figure 3b This demonstrates IGS visualization of segmented preoperative 3D structures based on a flat planar cross-section 10b at the microscope's working distance. The non-planar nature of actual tissue surfaces can lead to inaccurate visualization depth of these preoperative 3D structures. For example, in Figure 3b The upper right side shows a schematic diagram of the tissue image obtained after preoperative 3D structure enhancement. The 3D structure 32 of the preoperative scan 30 is correctly drawn because it is drawn at the tissue surface. The 3D structure 34 is incorrectly represented as being at the surface level, when its actual location is in a deeper tissue layer.

[0074] Figure 4a and4b An illustration of IGS based on three-dimensional tissue scanning is shown. Figure 4a An operating microscope with 3D surface scanning capability (i.e., its objective lens 124) is illustrated, which provides the IGS with accurate tissue surface information of the surface 10 of the surgical site for correct cross-sectional calculation. Figure 4a The segmented preoperative 3D structure of the 3D preoperative scan 40 is also shown. Figure 4b This demonstrates how visualized data can accurately match actual tissue surfaces, resulting in more precise visualizations. Figure 4b The image shows the surface adaptation segment 10a (i.e., the 3D contour of the surgical site), as well as the segmented preoperative 3D structure from the 3D preoperative scan 40. Figure 4b The upper right side shows a schematic diagram of the tissue image enhanced with preoperative 3D structural analysis. Figure 4b In the diagram, 3D structure 42 is correctly depicted on the tissue surface, and 3D structure 44 (shown in dashed outline) accurately indicates that the structure is located in a deeper layer rather than on the surface.

[0075] 3D surface scanning of tissues can be achieved through a variety of different imaging techniques, each tailored to the unique requirements of surgical microscopes. One common method is structured light scanning, where a known pattern of light is projected onto a surface and its deformation is analyzed to generate a 3D model. Time-of-flight scanning is another method, measuring the time it takes for a light pulse to travel to and return from the surface. For surgical microscopes specifically, a technique known as depth of focus or focus stacking can be used. This involves sequentially adjusting the microscope's working distance, capturing images at each step, and analyzing each image to identify the optimal focusing distance for each region. This information can then be used to create a detailed 3D map of the tissue surface. The choice of scanning method can depend on a number of factors, including the required resolution, the nature of the tissue being scanned, and the specific needs and constraints of the surgical procedure. Each of these methods can be integrated into a surgical microscope (or more generally, a surgical imaging device) to provide the 3D surface scanning capabilities of this invention.

[0076] Although some aspects are described in the context of the device, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of the corresponding device.

[0077] Some embodiments relate to an imaging device such as a microscope or exoskeleton, the imaging device comprising, as in combination with Figures 1a to 4b The system described in one or more of these. Alternatively, the imaging device, such as a microscope or exoskeleton, can be a combination of... Figures 1a to 4bOne or more of the ones described are part of or connected to the system. Figure 5 A schematic illustration of a system 500 configured to perform the methods described herein is shown. System 500 includes an imaging device 510 and a computer system 520. The imaging device 510 is configured to capture images and is connected to the computer system 520. The computer system 520 is configured to perform at least a portion of the methods described herein. The computer system 520 may be configured to execute machine learning algorithms. The computer system 520 and the imaging device 510 may be separate entities, but may also be integrated together in a common housing. The computer system 520 may be part of the central processing system of the imaging device 510, and / or the computer system 520 may be part of a sub-component of the imaging device 510, such as a sensor, actuator, camera, or illumination unit of the imaging device 510.

[0078] Computer system 520 may be a local computer device (e.g., a personal computer, laptop computer, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in various locations (e.g., distributed at local clients and / or one or more remote server farms and / or data centers)). Computer system 520 may include any circuitry or combination of circuitry. In one embodiment, computer system 520 may include one or more processors of any type. As used herein, a processor may mean any type of computing circuitry, such as, but not limited to, an imaging device or an imaging device component (e.g., a camera), such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field-programmable gate array (FPGA), or any other type of processor or processing circuitry. Other types of circuitry that may be included in computer system 520 may be custom circuitry, application-specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 520 may include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for disposing of removable media (such as optical discs (CDs), flash memory cards, digital video discs (DVDs), etc.). Computer system 520 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that allows a system user to input information into and receive information from computer system 520.

[0079] Figures 6a-6b A visualization of the subsurface structure is shown. Figure 6a Examples of existing technologies are shown, and Figure 6b An example using depth encoding overlay is shown. Figure 6a The prior art visualization shown does not allow for depth perception of structure 610. In fact, structure 610 is simply superimposed on tissue surface 620. Structure 610 is subsurface tissue. For example, this structure could be a critical anatomical structure.

[0080] Figure 6aThe current visualization method shown relies on alpha blending to reveal subsurface structures. This current visualization provides very little depth information, showing key anatomical structures beneath the tissue surface but lacking clear depth cues.

[0081] In comparison, Figure 6b The visualization shown enhances depth perception by applying a color-coded scale to represent the distance of the subsurface structure 640 from the tissue surface 650. For example, warmer colors indicate shallower structures (i.e., structures 650 are less distant from the surgical site surface / tissue surface 650), and cooler colors indicate deeper areas (i.e., structures 650 are more distant from the surgical site surface / tissue surface 650). In this way, intraoperative accuracy can be improved by providing intuitive and immediate depth feedback to the user (e.g., a surgeon). This means that using color-coded depth overlay can improve the depth perception of structure 640.

[0082] Generally speaking, Figure 6b The visualization shown can enhance intraoperative visualization by combining color-coded depth visualization to capture the tissue surface in 3D. Capturing the tissue surface in 3D can be achieved using the stereoscopic imaging capabilities of a microscope to generate an accurate depth map of the tissue surface at 650° during surgery (e.g., based on references above). Figure 1a The sensor data determines the three-dimensional contour of the surgical site surface. Color-coded depth visualization may include displaying subsurface anatomical and / or pathological structures 640 using a color scale representing their distance from the tissue surface 650. By combining real-time 3D surface data with preoperative 3D imaging, the system can provide users (e.g., surgeons) with intuitive, color-coded depth information about the subsurface structures 650. This visualization addresses the need for readily available depth cues, improving surgical accuracy and / or outcomes without significantly altering existing workflows or requiring substantial additional hardware.

[0083] For example, a method can be provided for visualizing subsurface structures during surgery by using a color-coded scale to represent the depth of subsurface structures relative to the tissue surface. This method leverages the existing capabilities of microscopy to capture 3D surface data and integrates it with preoperative imaging to provide the user with immediate and intuitive depth information. An exemplary workflow may be shown below.

[0084] Data indicating the 3D tissue surface during surgery can be obtained. For example, stereoscopic images can be captured using the left and right imaging channels of a surgical microscope. Based on the stereoscopic images from the left and right imaging channels, a depth map can be generated. For example, parallax analysis of the stereoscopic images can produce a real-time depth map of the tissue surface. Alternatively, structured light scanning can be used. For example, a pattern can be projected onto the tissue surface, and analysis of the deformation can be performed to create a 3D model. Alternatively, a time-of-flight sensor can be used. For example, the time it takes for light to reflect from the tissue surface can be measured to determine the depth and thus create a 3D model.

[0085] Furthermore, data indicating the intraoperative 3D tissue surface (e.g., a 3D model) can be integrated with preoperative 3D data. For example, alignment can be performed. The intraoperative 3D model can be registered with preoperative 3D imaging data to ensure spatial alignment. Registration can be achieved using anatomical reference points or benchmarks. Additionally, depth calculations can be performed. Depth calculations can calculate the distances between multiple points on subsurface structures and corresponding points on the tissue surface of the 3D model (e.g., as mentioned above in the reference section). Figure 1a Described as determining the distance between the surface of the surgical site and the structure in a three-dimensional preoperative scan.

[0086] Furthermore, a color-coded depth representation can be performed. For example, a color scale definition can be utilized. A color spectrum can be defined, where each color corresponds to a specific depth range (e.g., light depths are red, gradually transitioning to deeper areas of blue). Visualizations of subsurface structures can be generated by rendering the depth based on the color scale. The color-coded structure can then be overlaid on a live surgical view, optionally with adjustable transparency to maintain visibility of the living tissue.

[0087] Figure 6b The visualization shown provides real-time visualization. Users can view a live surgical field with depth-enhanced overlays, providing immediate depth cues. Additionally, optional customization options are available. For example, users can adjust the color scale range, transparency level, and / or select specific structures to display. Optionally, a depth threshold can be set to focus on structures within certain depth ranges.

[0088] The proposed visualization enhances depth perception by providing direct visual cues about the distance to subsurface structures, thus aiding surgical navigation and decision-making. Furthermore, it offers improved accuracy by combining real-time 3D surface data with preoperative imaging for precise structural localization. Additionally, it provides intuitive visualization using color-coded depth information that is readily interpretable within the time constraints of surgery. Moreover, it offers reduced workflow disruptions without requiring significant modifications by integrating with existing microscope systems and workflows. Furthermore, it provides alternative designs and methods for dynamic depth adjustment. For example, the system can automatically adjust the color scale based on the depth range present in the current field of view.

[0089] Some or all of the method steps may be performed by (or using) hardware devices (e.g., processors, microprocessors, programmable computers, or electronic circuits). In some embodiments, one or more of the most important method steps may be performed by such devices.

[0090] Depending on certain specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. This implementation can be executed using a non-transitory storage medium (such as a digital storage medium, e.g., floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory) that stores electronically readable control signals thereon, which cooperate with (or are capable of cooperating with) a programmable computer system to cause the corresponding method to be executed. Therefore, the digital storage medium can be computer-readable.

[0091] Some embodiments of the invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system to enable the execution of one of the methods described herein.

[0092] Typically, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, is operable to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0093] Other embodiments include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0094] Therefore, in other words, an embodiment of the present invention is a computer program having program code for executing one of the methods described herein when the computer program is run on a computer.

[0095] Therefore, another embodiment of the invention is a storage medium (or data carrier, or computer-readable medium) including a computer program stored thereon for performing one of the methods described herein when executed by a processor. Data carriers, digital storage media, or recording media are generally tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, including a processor and a storage medium.

[0096] Therefore, another embodiment of the invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0097] Another embodiment includes a processing component (e.g., a computer or programmable logic device) configured or adapted to perform one of the methods described herein.

[0098] Another embodiment includes a computer having a computer program installed thereon for performing one of the methods described herein.

[0099] Another embodiment of the invention includes an apparatus or system configured to transmit, for example, electronically or optically to a receiver, a computer program for performing one of the methods described herein. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transmitting the computer program to the receiver.

[0100] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device. List of reference numerals

Claims

1. A system (110) for a surgical imaging system (100), the system comprising one or more processors and one or more storage devices, wherein the system is configured to: Sensor data is obtained from the sensors (150; 122) of the surgical imaging system, the sensor data representing the distance between the objective lens (124) of the surgical imaging device (120) of the surgical imaging system and multiple points on the surface (10) of the surgical site imaged by the surgical imaging system; The three-dimensional contour of the surface of the surgical site is determined based on the sensor data (10a). Obtain preoperative data including three-dimensional preoperative scans of the structure; The distance between the surface of the surgical site and the structure of the three-dimensional preoperative scan (40) of the surgical site is determined based on the three-dimensional contour of the surface of the surgical site and the preoperative data; as well as A visual overlay of at least a portion of the structure of the three-dimensional preoperative scan is generated based on the distance between the surface of the surgical site and the structure of the surface of the surgical site.

2. The system of claim 1, wherein the system is configured to generate a cross section of the three-dimensional preoperative scan based on the intersection between the surface of the surgical site and the three-dimensional preoperative scan of the surface of the surgical site, and to generate the visual overlay based on the generated cross section.

3. The system according to any one of claims 1 or 2, wherein the system is configured to determine a first portion (42) of the three-dimensional preoperative scan to be visualized at or above the surface of the surgical site and a second portion (44) of the three-dimensional preoperative scan to be visualized below the surface of the surgical site based on the distance between the surface of the surgical site and the structure of the three-dimensional preoperative scan of the surface of the surgical site.

4. The system according to any one of claims 1 to 3, wherein the system is configured to determine a first portion (42) and a second portion (44) of the three-dimensional preoperative scan based on the distance between the surface of the surgical site and the structure of the three-dimensional preoperative scan of the surface of the surgical site, and to include the first portion in the visual overlay in a first visualization style, and a) include the second portion in the visual overlay in a second visualization style, or b) omit the second portion from the visual overlay.

5. The system according to any one of claims 1 to 3, wherein the three-dimensional contour of the surface of the surgical site includes a three-dimensional model of the surface of the surgical site.

6. The system according to any one of claims 1 to 5, wherein the sensor data includes a three-dimensional surface scan of the surface of the surgical site.

7. The system according to any one of claims 1 to 6, wherein the sensor data includes a representation of a structured light pattern emitted by the structured light emitter of the surgical imaging system.

8. The system according to any one of claims 1 to 6, wherein the sensor data is sensor data of the time-of-flight sensor of the surgical imaging system.

9. The system according to any one of claims 1 to 6, wherein the sensor data comprises a plurality of image frames representing a plurality of different focal lengths or working distances.

10. The system of claim 9, wherein the system is configured to determine the three-dimensional contour of the surface of the surgical site based on contrast and / or based on the presence of spatial frequencies above a predefined spatial frequency threshold in the respective image frames of the plurality of image frames.

11. The system according to any one of claims 1 to 10, wherein the system is configured to generate the visual overlay such that the structure of the three-dimensional preoperative scan is visualized differently for different defined distances.

12. The system of claim 11, wherein the structure of the three-dimensional preoperative scan is visualized differently by color coding of the structure.

13. The system of claim 12, wherein the system is configured to automatically adjust the color scale based on the depth range of the three-dimensional contour.

14. The system of claim 11, 12 or 13, wherein the system is configured to visualize the structure of the three-dimensional preoperative scan in a partially transparent manner.

15. The system of claim 14, wherein the system is configured to obtain user data indicative of the transparency of the structure in the three-dimensional preoperative scan; and The transparency of the structure in the three-dimensional preoperative scan is adjusted based on the user data.

16. The system according to any one of claims 11 to 15, wherein the structure of the three-dimensional preoperative scan is visualized differently by applying a depth threshold so that portions of the structure within a certain depth range are focused.

17. A surgical imaging system (100) comprising a system according to any one of claims 1 to 10, the surgical imaging device (120), and the sensors (150; 122) for providing the sensor data.

18. The surgical imaging system of claim 17, wherein the sensor for providing the sensor data is an imaging sensor (122) of the surgical imaging device (120), or wherein the sensor (150) for providing the sensor data is separate from the surgical imaging device.

19. The surgical imaging system according to any one of claims 17 or 18, wherein the surgical imaging device is a microscope and an exoscopy.

20. A method for use in a surgical imaging system, the method comprising: (210) Sensor data is obtained from the sensors of the surgical imaging system, the sensor data representing the distance between the objective lens of the surgical imaging device of the surgical imaging system and multiple points on the surface of the surgical site imaged by the surgical imaging system; (220) The three-dimensional contour of the surface of the surgical site is determined based on the sensor data; Obtain (230) preoperative data including three-dimensional preoperative scans of the structure; Based on the three-dimensional contour of the surface of the surgical site and the preoperative data, determine (240) the distance between the surface of the surgical site and the structure of the three-dimensional preoperative scan of the surgical site; as well as (250) A visual superposition of at least a portion of the structure of the three-dimensional preoperative scan is generated based on the distance between the surface of the surgical site and the structure of the surface of the surgical site.

21. A computer program having program code for executing the method of claim 20 when the computer program is run on a processor.

Images