Surgical microscope system, and system, method and computer program for a surgical microscope system

Abstract

The present example relates to a surgical microscope system (100) and a system (110) for a surgical microscope system, a method, and a computer system. The system (110) is configured to acquire first imaging sensor data of a view on a surgical site from a first optical imaging sensor (122) of a microscope (120) of the surgical microscope system. The system is configured to acquire second imaging sensor data of a view on the surgical site from a second sensor (124) of the microscope. The system is configured to determine an extent of one or more anatomical features of the surgical site based on the second imaging sensor data. The system is configured to apply spatially varying noise filtering to the first imaging sensor data based on the extent of the one or more anatomical features of the surgical site.

Description

[Technical field]

[0001] The present examples relate to surgical microscope systems, as well as systems, methods and computer systems for surgical microscope systems. [Background technology]

[0002] Digital microscopes, particularly surgical digital microscopes, often have multiple imaging modes, such as a reflectance imaging mode in which light reflected by a sample being imaged is used to generate a digital image of the sample, and a fluorescence imaging mode in which fluorescent emissions emitted by a fluorescent dye applied to the sample are used to generate a digital image of the sample.

[0003] Since the light intensity of fluorescent emission is considerably lower compared to the reflection used in reflectance imaging, the respective optical imaging sensor usually operates at a higher sensitivity. As a result, fluorescent images often exhibit more noise. Although there are several methods for noise filtering, noise filtering of fluorescent images remains a challenging task and filtered images often have a lower than optimal quality. In general, existing noise filtering methods are applied uniformly to the whole image.

[0004] More effective filtering of noise can improve image quality, especially increasing contrast. Summary of the Invention [Problem to be solved by the invention]

[0005] Improved concepts for performing noise filtering on microscopic images of low intensity light sources are desired. [Means for solving the problem]

[0006] This need is addressed by the subject matter of the independent claims.

[0007] Various examples of the present disclosure are based on the finding that different types of anatomical features may appear very differently in imaging sensor data generated by a surgical digital microscope. For example, in a reflectance image of the brain, vessels (blood vessels) may appear very clearly, e.g., as red or purple features, while other types of tissue may be shown as a low-intensity flesh-colored background. Similarly, in a fluorescence image of the brain, blood vessels containing fluorescent dyes appear as relatively bright parts of the fluorescence image, while blood vessels that do not contain fluorescent dyes or other types of tissues remain dark. To improve noise reduction, these different parts of each image may be processed differently, so that the noise filtering applied to the imaging sensor data may be spatially varied. To improve the accuracy of the spatially varied noise filtering, imaging sensor data that clearly shows the range and / or type of each feature is used to determine the range of each anatomical feature to be processed differently, and apply the actual spatially varying noise filter to another set of imaging sensor data where the range may be less accurate or more difficult to determine.

[0008] Various examples of the present disclosure relate to a system for a surgical microscope system. The system includes one or more processors and one or more storage devices. The system is configured to acquire first imaging sensor data of a view on the surgical site from a first optical imaging sensor of a microscope of the surgical microscope system. The system is configured to acquire second imaging sensor data of a view on the surgical site from a second sensor of the microscope. The system is configured to determine a range of one or more anatomical features of the surgical site based on the second imaging sensor data. The system is configured to apply spatially varying noise filtering to the first imaging sensor data based on a range of the one or more anatomical features of the surgical site. By varying the spatially varying noise filtering based on a range of the one or more anatomical features, the noise filtering can be tailored to the anatomical features shown in the first and second imaging sensor data. By determining a range of the one or more anatomical features using the second imaging sensor data, the accuracy of the range determination can be increased.

[0009] For example, the first imaging sensor data may be based on fluorescence imaging. In this case, spatially varying noise filtering may be particularly effective, where a more aggressive noise filter may be used in portions of the imaging sensor data that do not exhibit fluorescence emission, and a specialized noise filter may be used in portions of the imaging sensor data that do exhibit fluorescence emission. Furthermore, because fluorescence imaging sensor data typically has low contrast, a separate set of imaging sensor data may improve the determination of the extent of one or more anatomical features (i.e., a second imaging sensor data is used).

[0010] For example, the second imaging sensor data may be based on reflectance imaging. Alternatively or additionally, the second imaging sensor data may be based on multispectral, hyperspectral and derivatives, reflectance and / or fluorescence, Raman imaging and derivatives, laser speckle imaging and derivatives, confocal images and derivatives, optical property imaging, i.e. μ a (absorption coefficient) and μ s Other imaging techniques may be used, such as (scattering coefficient), ultrasound imaging, photoacoustic imaging and derivatives, 3D surface scanning, kinetic mapping imaging (e.g., indocyanine green (ICG) bolus kinetics), pre-operative, or intra-operatively, prior or in real-time functional imaging with any modality, or anatomical estimate imaging derived from comparison of tissue imaging combined with an anatomical database. Depending on the type of anatomical feature and the type of imaging modality available, different types of imaging may be suitable to determine the extent of the anatomical feature.

[0011] In general, not all anatomical features are relevant for spatial variation filtering. For example, in fluorescence imaging, only some blood vessels may contain fluorescent dye, while other blood vessels and other types of tissue may not emit light. Thus, a subset of anatomical features (i.e., anatomical features of interest) may be selected from the determined one or more anatomical features. The system may be configured to determine at least one feature of interest among the one or more anatomical features of the surgical site and apply spatial variation noise filtering based on the extent of the at least one feature of interest. For example, as outlined above, the one or more anatomical features may be one or more blood vessels, and the at least one feature of interest may be at least one blood vessel that emits fluorescent light.

[0012] Whether an anatomical feature is an anatomical feature of interest can be determined based on different criteria, for example, based on a classification of the respective anatomical feature. In some examples, the first imaging sensor data can be used to classify anatomical features of interest and anatomical features of no interest. For example, the system can be configured to determine at least one feature of interest from among the one or more anatomical features based on the first imaging sensor data.

[0013] In some cases, whether an anatomical feature is an anatomical feature of interest may not be identifiable in all frames of the first imaging sensor data. Thus, multiple frames of the first imaging sensor data may be processed to determine whether an anatomical feature is of interest. For example, the system may be configured to determine at least one feature of interest from among the one or more anatomical features based on multiple frames of the first imaging sensor data covering a predefined time interval or two or more predefined time points.

[0014] In many cases, it may be sufficient to apply two different types of noise filtering: a first type for anatomical features (of interest) and a second type for background (e.g., tissue that is not or rarely of interest to the surgical procedure). For example, the system may be configured to subdivide the first imaging sensor data into a first portion and a second portion, the first portion being based on the extent of at least a subset of one or more anatomical features of the surgical site. The system may be configured to apply a first noise filter to the first portion and a different second noise filter to the second portion.

[0015] For example, as outlined above, the first imaging sensor data may be subdivided into a first portion including anatomical features (of interest) and a second portion including the remainder, such as anatomical features not of interest and other types of tissue. For example, the system may be configured to subdivide the first imaging sensor data such that the first portion includes the extent of at least a subset of one or more anatomical features of the surgical site and the second portion includes the remainder of the first imaging sensor data.

[0016] In fluorescence imaging, several assumptions can be made regarding the desired appearance of the anatomical features that contain the fluorescent dye. One assumption is that the intensity of the fluorescent emission mostly follows a known intensity pattern. Thus, the first noise filter can be configured to apply a predefined intensity pattern to the first portion of the first imaging sensor data. In particular, the intensity pattern of the fluorescent emission is approximately uniform within the blood vessel. Thus, the predefined intensity pattern may be configured to apply a uniform intensity distribution to the first portion or a coherent sub-portion of the first portion. However, some gradual changes may be exhibited, e.g., a gradient along the vertical axis of the respective feature. For example, the predefined intensity pattern is configured to apply a uniform intensity distribution along the horizontal axis of the coherent portion of the first portion and an intensity gradient along the vertical axis of the coherent portion of the first portion.

[0017] The proposed concept is particularly suitable for performing noise filtering on blood vessels. Blood vessels, especially cerebral blood vessels, may be tree-like structures with many branches. The distribution of fluorescent dyes in blood vessels may be affected by the branching of blood vessels. Thus, blood vessels may be subdivided into smaller sub-parts and noise filtering may be applied separately to each smaller sub-part. For example, the system may be configured to subdivide the first part into multiple coherent sub-parts based on the extent of at least a subset of one or more anatomical features and to apply the first noise filter separately to the coherent sub-parts. For example, the one or more anatomical features may be one or more blood vessels. The system may be configured to subdivide the first part into multiple coherent sub-parts based on the branching points of one or more blood vessels. This may improve the quality of noise filtering since subtle gradients between the sub-parts may be modeled more accurately.

[0018] Coarser and / or more aggressive noise filtering may be applied to background, or less significant anatomical features. For example, the second noise filter may be configured to apply a spatial low-pass filter to a second portion of the first imaging sensor data.

[0019] In general, at least two techniques can be used to determine the extent of the surgical site: a machine learning based approach and a color / shape based approach. For example, the system may be configured to determine the extent of one or more anatomical features of the surgical site using a machine learning model that is trained to perform image segmentation and / or object detection. Alternatively or additionally, the system may be configured to determine the extent of one or more anatomical features of the surgical site based on at least one of a characteristic color spectrum and a characteristic shape of the one or more anatomical features of the surgical site. Both techniques have advantages and disadvantages, for example, with respect to computational effort, implementation complexity and / or traceability.

[0020] The results of the noise filtering may be displayed via a display device of the surgical microscope system. For example, the system may be configured to generate a display signal for the display device of the surgical microscope system based on at least the filtered first imaging sensor data. For example, the system may be configured to generate a composite digital image of the surgical site based on the filtered first imaging sensor data and the second imaging sensor data, and generate a display signal based on the composite digital image. In other words, the first and second imaging sensor data may be combined into a single image and shown together on the display device.

[0021] Various examples of the present disclosure relate to a corresponding method for a surgical microscope system. The method includes acquiring first imaging sensor data of a view on a surgical site from a first optical imaging sensor of a microscope of the surgical microscope system. The method includes acquiring second imaging sensor data of a field of view on the surgical site from a second sensor of the microscope. The method includes determining an extent of one or more anatomical features of the surgical site based on the second imaging sensor data. The method includes applying spatially varying noise filtering to the first imaging sensor data based on the extent of the one or more anatomical features of the surgical site.

[0022] Various examples of the present disclosure relate to corresponding computer programs having program codes for performing the above methods, when the computer program runs on a processor.

[0023] Some examples of apparatus and / or methods are described below, by way of example only, and with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0024] [Figure 1a] FIG. 1 is a schematic diagram showing an example of a system for a surgical microscope system. [Figure 1b] FIG. 1 is a schematic diagram showing an example of a surgical microscope system. [Figure 1c] FIG. 1 is a schematic diagram illustrating an example of a noisy fluorescent image of a surgical site. [Figure 1d] FIG. 2 is a schematic diagram showing an example of a reflected image of a surgical site. [Diagram 2] 1 is a flow chart illustrating an example of a method for a surgical microscope system. [Diagram 3] FIG. 1 is a schematic diagram showing noise filtering applied to a noisy fluorescent image. [Figure 4] FIG. 1 is a schematic diagram illustrating an example of a system including a microscope and a computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Various examples will now be described more fully with reference to the accompanying drawings, in which several examples are shown, in which the thicknesses of lines, layers and / or regions may be exaggerated for clarity.

[0026] FIG. 1a is a schematic diagram showing an example of a system 110 for a surgical microscope system 100. The surgical microscope system 100 comprises a microscope 120 that is a digital microscope, i.e., it comprises at least one optical imaging sensor 122; 124 connected to the system 110. In general, a microscope, such as the microscope 120, is an optical instrument suitable for inspecting objects that are too small to be inspected by the human eye (alone). For example, the microscope can provide optical magnification of the sample. In modern microscopes, the optical magnification is often provided to an imaging sensor or camera, such as the first optical imaging sensor 122 and / or the second sensor 124 of the microscope 120. In FIG. 1a, the microscope 120 is shown with two optical imaging sensors 122; 124. However, the second sensor 124 does not necessarily have to be an optical imaging sensor, but may be any kind of sensor capable of providing an image representation of the sample 10. The microscope 120 may further comprise one or more optical magnification components, such as an objective lens (i.e., a lens), used to magnify the view on the sample.

[0027] Besides the optical components that are part of the microscope 120, the surgical microscope system 100 further comprises a system 110, which is a computer system. The system 110 comprises one or more processors 114 and one or more storage devices 116. Optionally, the system further comprises one or more interfaces 112. The one or more processors 114 are connected to the one or more storage devices 116 and to any one or more interfaces 112. In general, the functionality of the system is provided by the one or more processors 114 in cooperation with the one or more interfaces 112 (e.g., for exchanging information with the first optical imaging sensor 122, the second sensor 124, and / or a display device such as the eyepiece display 130a or the auxiliary display 130b) and / or in cooperation with the one or more storage devices 116 (for storing and / or retrieving information).

[0028] The system 110 is configured to acquire first imaging sensor data of a view on the surgical site 10 from a first optical imaging sensor 122 of a microscope 120 of the surgical microscope system 100. The system 110 is configured to acquire second imaging sensor data of a view on the surgical site 10 from a second sensor 124 of the microscope 120. The system 110 is configured to determine an extent of one or more anatomical features of the surgical site based on the second imaging sensor data. The system 110 is configured to apply spatially varying noise filtering to the first imaging sensor data based on an extent of one or more anatomical features of the surgical site. As will be apparent, the system 110 is a system for processing imaging sensor data within the surgical microscope system 100 and / or for controlling the microscope 120 and / or other components of the surgical microscope system 100.

[0029] In general, a microscope system such as surgical microscope system 100 is a system that includes a microscope 120 and additional components that operate in conjunction with the microscope, such as system 110 (which is a computer system configured to control the surgical microscope system and, for example, to process the microscope's imaging sensor data), additional sensors, a display, etc.

[0030] There are a wide variety of microscopes. When microscopes are used in the medical or biological field, the object 10 seen through the microscope can be a sample of organic tissue, for example arranged in a Petri dish or present in a part of a patient's body. In this case, the microscope 120 is a microscope of a surgical microscope system, i.e. a microscope used during a surgical procedure such as neurosurgery, tumor surgery or during tumor surgery. The object seen through the microscope can therefore be a sample of the patient's organic tissue or it can be a surgical site, in particular where the surgeon operates during a surgical procedure. In the following, it is assumed that the object 10 to be imaged, i.e. the surgical site, is a surgical site of the brain in the course of a neurosurgery operation. However, the proposed concept is also suitable for other types of surgery, such as cardiac surgery or ophthalmology.

[0031] 1b is a schematic diagram showing an example of a surgical microscope system 100 comprising a system 110 and a microscope 120 (having a first optical imaging sensor 122 and a second optical imaging sensor 124). The surgical microscope system 100 shown in FIG. 1b comprises a base unit 105 (containing the system 110) with a (rotating) stand, an eyepiece display 130a arranged on the microscope 120, an auxiliary display 130b arranged on the base unit 105, and some optional components such as a (robotic or manual) arm 140 that holds the microscope 120 in place and is connected to the base unit 105 and the microscope 120. In general, these optional and non-optional components may be connected to a system 110 that may be configured to control and / or interact with each of the components.

[0032] The proposed concept is based on processing two types of imaging sensor data: first imaging sensor data, which is imaging sensor data to which spatial variation noise filtering is applied, and second imaging sensor data, which is imaging sensor data used to determine the extent of one or more anatomical features. Figure 1c and Figure 1d show two examples of such imaging sensor data. Figure 1c is a schematic diagram showing an example of a noisy fluorescence image of a surgical site. In other words, the first imaging sensor data may be based on fluorescence imaging. The first optical imaging sensor may therefore be an optical imaging sensor for performing fluorescence imaging. Alternatively, the first imaging sensor data may be based on other types of imaging based on sensing low light intensity. Figure 1d is a schematic diagram showing an example of a corresponding reflection image at the surgical site. In other words, the second imaging sensor data may be based on reflection imaging. For example, the second imaging sensor data may be based on hyperspectral reflection imaging. The second sensor may therefore be an optical imaging sensor for performing reflection imaging, for example hyperspectral reflection imaging. For example, the first and second imaging sensor data may have the same field of view, or the fields of view of the sets of the two imaging sensor data may have a known spatial relationship. In Fig. 1d, different anatomical features, namely the two blood vessels 14;16, can be clearly distinguished. Between the anatomical features 14;16, tissues 12a;12b;12c are shown, which are considered to be the "background" behind the anatomical features of interest. As is evident from the comparison between both figures, the blood vessels 14;16 have different colors (visualized by two different line types) due to different oxygenation levels of the blood. Similarly, as shown in Fig. 1c, the fluorescent emission emanating from the two blood vessels has different intensities due to the concentration of the fluorescent dye in the two blood vessels. Within the blood vessels, the intensity of the fluorescent emission can be considered to be uniform or at least uniform along the horizontal axis and to have a gradient along the vertical axis.

[0033] In the above example, the second imaging sensor data is based on reflectance imaging, for example hyperspectral reflectance imaging. However, different types of sensors imaging sensor data may be used. In this context, the term "second imaging sensor data" indicates that the second imaging sensor data is acquired in image format, i.e. as an image. In general, the second imaging sensor data may be images acquired from an optical imaging sensor, or images derived from other types of optical or non-optical sensor data, or computer-generated images derived from preoperative scans or a database of anatomical features. For example, the second imaging sensor data may be images acquired from imaging spectroscopy (including multispectral, hyperspectral and derivatives thereof, reflectance and / or fluorescence), Raman imaging (and derivatives thereof), laser speckle imaging (and derivatives thereof), confocal imaging (and derivatives thereof), optical property images (in particular μ a (absorption coefficient) and / or μ sThe second sensor may be based on or include at least one of: an optical property image representing a scattering coefficient (scattering coefficient) or a derivative thereof), ultrasound imaging, photoacoustic imaging (and derivatives thereof), 3D surface scanning (e.g., using a depth sensor), kinetic mapping imaging (e.g., ICG bolus dynamics), functional images captured with any modality (pre-operative or real-time before or during surgery), and an anatomical estimation image derived from a comparison of tissue imaging combined with an anatomical database (e.g., showing where anatomical or functional tissue regions are located within the surgical site). Thus, the second sensor may be one of an optical imaging sensor (such as a hyperspectral optical imaging sensor, a multispectral optical imaging sensor, an optical imaging sensor for fluorescence imaging (for dynamic imaging and / or imaging spectroscopy), a spectroscopic optical imaging sensor, a Raman spectroscopic optical imaging sensor, an optical imaging sensor for laser speckle imaging, a confocal optical imaging sensor, or an optical imaging sensor for determining optical properties), an ultrasound sensor, a photoacoustic imaging sensor, and a depth sensor.

[0034] Here, the second imaging sensor data is used to determine the extent of one or more anatomical features of the surgical site. To determine the extent of the one or more anatomical features, the one or more anatomical features may be determined (e.g., detected, identified) in the second imaging sensor data. For example, the second imaging sensor data may be analyzed to determine and distinguish anatomical features, e.g., blood vessels, tumors, branches, portions of tissue, etc., within the imaging sensor data. In general, two types of techniques may be used to perform the extent determination.

[0035] In the first technique, a color-based approach and / or a shape-based approach can be used. For example, the system can be configured to determine the extent of one or more anatomical features of the surgical site based on at least one of a characteristic color spectrum and a characteristic shape of the one or more anatomical features of the surgical site. For example, each anatomical feature (of interest) can have a known characteristic color spectrum and / or a characteristic shape. For example, depending on the oxygenation of the blood, blood vessels can have a color that is part of a known color spectrum that is characteristic of blood vessels, ranging from bright red (for arterial blood) to blue-violet (for venous blood). For example, separate characteristic color spectra can be considered for blood vessels carrying arterial blood and blood vessels carrying venous blood. Furthermore, blood vessels usually have a characteristic shape with sharp edges that are recognizable in the second imaging sensor data. The system may be configured to identify portions (e.g., pixels) of the second imaging sensor data that have a color that is part of a characteristic spectrum of the (interesting) anatomical feature and / or portions (e.g., pixels) of the second imaging sensor data that are part of a structure having a characteristic shape, and to determine the extent of one or more anatomical features based on the portions (e.g., pixels) of the second imaging sensor data that have a color that is part of a characteristic spectrum of the (interesting) anatomical feature and / or portions (e.g., pixels) of the second imaging sensor data that are part of a structure having a characteristic shape.

[0036] Alternatively (or additionally), as a second technique, machine learning can be used to determine the extent of one or more anatomical features. For this purpose, one or both of the following machine learning based techniques can be used: image segmentation and object detection. In other words, the system can be configured to determine the extent of one or more anatomical features of the surgical site using a machine learning model trained to perform image segmentation and / or object detection. The system can be configured to perform image segmentation and / or object detection to determine the presence and / or location of one or more anatomical features in the imaging sensor data. In object detection, the location of one or more predefined objects (i.e., the object for which the respective machine learning model is trained) in the imaging sensor data is output by the machine learning model, together with a classification of the object (if the machine learning model is trained to detect multiple different types of objects). In general, the location of the one or more predefined objects is provided as a bounding box, i.e., a set of locations forming a rectangular shape surrounding the respective object to be detected. In image segmentation, the location of features (i.e., portions of the second imaging sensor data with similar attributes, e.g., belonging to the same object) is output by the machine learning model. Typically, the locations of the features are provided as a pixel mask, i.e. the locations of the pixels that belong to the feature are output for each feature.

[0037] For both object detection and image segmentation, a machine learning model trained to perform the respective task is used. For example, to train a machine learning model trained to perform object detection, a plurality of samples of imaging sensor data may be provided as training input samples, and a corresponding list of bounding box coordinates may be provided as a desired output of the training, and a training algorithm based on supervised learning is used to perform the training using the plurality of training input samples and the corresponding desired output. For example, to train a machine learning model trained to perform image segmentation, a plurality of samples of imaging sensor data may be provided as training input samples, and a corresponding pixel mask may be provided as a desired output of the training, and a training algorithm based on supervised learning is used to perform the training using the plurality of training input samples and the corresponding desired output. In some examples, the same machine learning model may be used to perform both object detection and image segmentation. In this case, the two types of desired output described above may be used in parallel during training, and the machine learning model is trained to output both bounding boxes and pixel masks.

[0038] In general, both object detection and image segmentation may be used. The pixel mask output by the image segmentation machine learning model may be used to determine the extent of one or more anatomical features, and the classification provided by the object detection may be used to determine whether the anatomical features are of interest. Thus, at least one of the two techniques "object detection" and "image segmentation" may be used to analyze the second imaging sensor data and determine the anatomical features in the second imaging sensor data. For example, the system may be configured to perform image segmentation on the second imaging sensor data and determine the extent of the one or more anatomical features based on the one or more pixel masks output by the image segmentation. Alternatively or additionally, the system may be configured to perform object detection on the second imaging sensor data to identify at least one anatomical feature and determine one or more features of interest from among the identified anatomical features.

[0039] In some examples, the features used to determine the one or more anatomical features (e.g., at least one feature of interest) may be limited to a particular group of features. For example, the system may be configured to determine at least one feature of interest from among one or more anatomical features of the surgical site. For example, the system may be configured to perform object detection to identify at least one of a blood vessel, a blood vessel branch point, and a tumor in the second imaging sensor data as an anatomical feature (of interest). Thus, the machine learning model trained to perform object detection may be trained to detect at least one of a blood vessel, a blood vessel branch point, and a tumor in the imaging sensor data. Similarly, the machine learning model trained to perform image segmentation may be trained to perform image segmentation on at least one of a blood vessel, a blood vessel branch point, and a tumor in the imaging sensor data. Thus, the one or more anatomical features (or features of interest) may be determined based on the output of one or more machine learning models trained to output information such as a bounding box or pixel mask representing a particular feature such as a blood vessel, a branch point, or a tumor.

[0040] In various examples of the present disclosure, the first imaging sensor data is based on fluorescence imaging, and (only) anatomical features carrying a fluorescent dye (and thus emitting fluorescent emission) are of interest with respect to spatial variation noise filtering. In other words, the one or more anatomical features may be one or more blood vessels, or one or more other types of tissue (such as tissue of a tumor). Thus, the at least one feature of interest may be at least one blood vessel or other type of tissue that emits fluorescent emission. To determine whether an anatomical feature such as a blood vessel is of interest, the determined range of the anatomical feature may be cross-referenced with the first imaging sensor data, for example, to determine a blood vessel or tissue that carries a fluorescent dye and thus is of interest with respect to the proposed noise filtering approach. In other words, the system may be configured to determine at least one feature of interest from among the one or more anatomical features based on the first imaging sensor data. For example, the system may be configured to compare the determined ranges of one or more anatomical features to the presence of fluorescent emissions in the first imaging sensor data and determine an anatomical feature of interest when the fluorescent emissions (e.g., a sufficient amount in the case of fluorescent emissions) intersect the range of the respective anatomical feature.

[0041] In some cases, the fluorescent emission may not be visible in all frames of the first imaging sensor data. Thus, multiple frames of the first imaging sensor data may be used when determining whether an anatomical feature is of interest. For example, the system may be configured to determine at least one feature of interest from among the one or more anatomical features based on multiple frames of the first imaging sensor data covering a predefined time interval or two or more predefined time points. For example, the predefined time interval may be a time interval that extends into the past (relative to the time at which the determination is made), e.g., at least 5 seconds into the past, or at least 10 seconds, or at least 30 seconds into the past. Similarly, the two or more predefined time points may include a current time point and a past time point, e.g., at least 5 seconds, or at least 10 seconds, or at least 30 seconds into the past.

[0042] Once the one or more anatomical features, or at least one feature of interest, are determined, they may be used as a basis for spatial variation noise filtering. In other words, the system is configured to apply spatial variation noise filtering to the first imaging sensor data based on the extent of one or more anatomical features of the surgical site, for example, to apply spatial variation noise filtering based on the extent of the at least one feature of interest. In practice, the first imaging sensor data may be subdivided into different portions, with different noise filtering applied to the different portions. For example, pixels of the first imaging sensor data may be assigned to one of two or more different portions. Noise filtering may be applied to each pixel based on which portion the respective pixel is assigned to.

[0043] In a basic example, two different portions are used: a first portion including at least one feature of interest (one or more anatomical features, if the concept of feature of interest is not used) and a second portion including the remainder of the first imaging sensor data. In other words, the system may be configured to subdivide the first imaging sensor data into a first portion and a second portion. The first portion may be based on the extent of at least a subset of one or more anatomical features of the surgical site (e.g., based on the extent of at least one feature of interest). For example, the first portion may include the extent of at least a subset of one or more anatomical features of the surgical site (e.g., the extent of at least one feature of interest). The second portion may include the remainder of the first imaging sensor data. For example, pixels that are part of the extent of at least a subset of one or more anatomical features of the surgical site, e.g., pixels that are part of at least one feature of interest, may be assigned to the first portion and the remainder of the pixels may be assigned to the second portion. Within the first portion, pixels may be assigned to different coherent sub-portions (as will become clear below). The system may be configured to apply a first noise filter to a first portion and a different second noise filter to a second portion, in other words, the noise filtering may vary between the first and second portions, i.e. the noise filtering applied to the first portion may be different from the noise filtering applied to the second portion.

[0044] The second portion may be considered "background" and therefore of less interest to the surgeon, and the at least one feature of interest defining the first portion may be considered "foreground." In general, the second portion, i.e., the background, may be expected to be mostly static and include mostly soft transitions between different types of tissue (due to background composition and / or due to lack of focus). Thus, a spatial low-pass filter may be applied to the second portion of the first imaging sensor data. In other words, the second noise filter may be configured to apply a spatial low-pass filter to the second portion of the first imaging sensor data. For example, a 2D Fourier transform may be used to transform each pixel into frequency space, and a spatial low-pass filter may be applied.

[0045] In various examples of the present disclosure, the first imaging sensor data is assumed to be based on fluorescence imaging. As described in connection with FIG. 1c, the fluorescence images tend to have a uniform intensity profile within each blood vessel, with little or no variation along the horizontal axis (e.g., an axis perpendicular to the blood flow) and a gradient along the vertical axis (e.g., an axis along the blood flow). Similar assumptions can be made for other types of first imaging sensor data, and often a known intensity pattern can be assumed. Such a known intensity pattern can be used to define a noise filter to be used for the first portion. For example, the first noise filter may be configured to apply a predefined intensity pattern to the first portion of the first imaging sensor data. For example, the predefined intensity pattern may be characteristic of an anatomical feature (of interest) included in the first portion.

[0046] In general, a microscope has a narrow field of view due to the magnification it provides. Within this field of view, in a basic approximation, the intensity of the fluorescent emission emitted by the fluorescent dyes present in the blood vessel can be considered to be uniform throughout the blood vessel. The predefined intensity pattern may therefore be configured to apply a uniform intensity distribution to the first portion or to the coherent sub-portions of the first portion. For example, a uniform intensity distribution (i.e., the same intensity) may be applied to the entire first portion. Alternatively, a separate uniform intensity distribution may be applied to each anatomical feature (of interest). Furthermore, separate uniform intensity distributions may be applied to the portions of the anatomical feature. Alternatively, the predefined intensity pattern may be configured to apply a uniform intensity distribution along the horizontal axis of the coherent sub-portion of the first portion and an intensity gradient along the vertical axis of the coherent sub-portion of the first portion. Depending on the granularity used, the first filter may be applied to the entire first portion or to the coherent sub-portions of the first portion. Thus, if a finer granularity is used, the first portion may be subdivided into coherent sub-portions. In other words, the system may be configured to subdivide the first portion into a plurality of coherent sub-portions based on the extent of at least a subset of one or more anatomical features. Such a subdivision is illustrated with respect to FIG. 1d, which represents second imaging sensor data but is used to illustrate the concept, since the coherent sub-portions may be more clearly delimited in FIG. 1d.

[0047] As outlined above, different granularity levels of the coherent sub-portions may be used. For example, each blood vessel may be considered to be a separate coherent sub-portion. For example, blood vessel 14 may be considered to be a coherent sub-portion and blood vessel 16 may be considered to be another coherent sub-portion. Since blood vessels are often interconnected structures with many branches, a finer granularity may be selected for the subdivision. For example, the coherent sub-portions of a blood vessel may be defined by the branching points of the blood vessel. For example, blood vessel 14 may be subdivided into coherent sub-portions 14a; 14b and 14c at the branching points between sub-portion 14a and sub-portions 14b and 14c. In other words, the system may be configured to subdivide the first portion into multiple coherent sub-portions based on the branching points of one or more blood vessels. Depending on the selected granularity, the first filter may be applied separately to each sub-portion. For example, the first filter may be applied separately to each blood vessel, or to each coherent sub-portion of a blood vessel. In practice, the system may be configured to apply the first noise filter separately to the coherent sub-portions.

[0048] Depending on which type of predefined intensity distribution is used, different measurements may be used as a criterion for determining the intensity criterion of each predefined intensity distribution. For example, the system may be configured to determine the average or median intensity of the pixels of the first imaging sensor data belonging to the first portion or a coherent sub-portion of the first portion, and determine the intensity of the uniform intensity distribution based on the average or median intensity of the pixels belonging to the first portion or a coherent sub-portion of the first portion. If a gradient along the vertical axis is used, the intensity of the pixels at each end of the vessel or at the coherent sub-portion of the vessel may be determined, and the intensity value of the gradient may be determined based on the intensity of the pixels at each end of the vessel or at the coherent sub-portion of the vessel.

[0049] The result of the noise removal may be displayed on a display device of the surgical microscope system, for example, the eyepiece display 130a and / or the auxiliary display 130b. For example, the system may be configured to generate a display signal of the display device 130 of the surgical microscope system based on at least the filtered first imaging sensor data. For example, the system may be configured to generate a digital image of the surgical site based (only) on the filtered first imaging sensor data, for example, a digital image in which the fluorescent emission is shown in isolation. Alternatively, the system may be configured to generate a digital image of the surgical site, for example, a composite digital image, based on the filtered first imaging sensor data and based on the (filtered or unfiltered) second imaging sensor data. For example, the system may be configured to generate a composite digital image of the surgical site based on the filtered first imaging sensor data and the second imaging sensor data. For example, the fluorescent emission may be included in the composite digital image as a pseudo-color representation (e.g., using a single color that stands out against the second imaging sensor data). The system may be configured to generate a display signal based on the digital image, for example, the composite digital image. For example, the display signal may be a signal for driving (e.g., controlling) the display device 130. For example, the display signal may include video data and / or control instructions for driving the display. For example, the display signal may be provided via one of the one or more interfaces 112 of the system. Thus, the system 110 may include a video interface 112 suitable for providing the display signal to the display 130 of the microscope system 100.

[0050] In the proposed microscope system, at least one optical imaging sensor is used to provide imaging sensor data. Thus, the first optical imaging sensor 122 is configured to generate the first imaging sensor data. Optionally, the second optical imaging sensor 124 or another type of sensor may be configured to generate the second imaging sensor data. For example, the optical imaging sensor 122;124 of the microscope 120 may comprise or be an APS (Active Pixel Sensor) (or CCD (Charge Coupled Device)) type imaging sensor. For example, in an APS type imaging sensor, light is recorded at each pixel using a photodetector and an active amplifier of the pixel. APS type imaging sensors are often based on CMOS (Complementary Metal-Oxide-Semiconductor) or S-CMOS (Scientific CMOS) technology. In a CCD type imaging sensor, the incident photons are converted into electronic charges at the semiconductor-oxide interface and then transferred between the capacitive bins of the imaging sensor by the imaging sensor's circuitry to perform imaging. The processing system 110 may be configured to obtain (i.e., receive or read) the respective imaging sensor data from the respective sensors 122; 124. The imaging sensor data may be obtained by receiving the imaging sensor data from the respective sensors (e.g., via the interface 112), by reading the imaging sensor data from the respective sensor's memory (e.g., via the interface 112), or by reading the imaging sensor data from the storage device 116 of the system 110, for example, after the imaging sensor data has been written to the storage device 116 by the sensor or another system or processor.

[0051] The one or more interfaces 112 of the system 110 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be digital (bit) values ​​according to a specified code, within a module, between modules, or between modules of different entities. For example, the one or more interfaces 112 may comprise an interface circuit configured to receive and / or transmit information. The one or more processors 114 of the system 110 may be implemented using any means for processing, such as one or more processing units, one or more processing devices, processors, computers, or programmable hardware components operable with software adapted accordingly. In other words, the described functions of the one or more processors 114 may also be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. The one or more storage devices 116 of the system 110 may comprise at least one element of a group of computer-readable storage media, such as a magnetic or optical storage medium, e.g., a hard disk drive, a flash memory, a floppy disk, a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electronically erasable programmable read-only memory (EEPROM), or network storage.

[0052] Further details and aspects of the system and surgical microscope system are described in relation to the proposed concept or one or more examples above or below (e.g., Figures 2-4). The system and surgical microscope system may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples above or below.

[0053] FIG. 2 is a flow chart illustrating an example of a corresponding method for a surgical microscope system. For example, the method may be performed by the system illustrated in connection with FIGS. 1a-1d and / or a surgical microscope system. The method includes acquiring 210 first imaging sensor data of a view on the surgical site from a first optical imaging sensor of a microscope of the surgical microscope system. The method includes acquiring 220 second imaging sensor data of a view on the surgical site from a second sensor of the microscope. The method includes determining 230 an extent of one or more anatomical features of the surgical site based on the second imaging sensor data. The method includes applying 240 spatially varying noise filtering to the first imaging sensor data based on an extent of the one or more anatomical features of the surgical site.

[0054] For example, features implemented in the system of FIGS. 1a-1d and / or the surgical microscope system may be implemented in the corresponding method of FIG. 2 as well.

[0055] Further details and aspects of the method are described in relation to the proposed concept or one or more examples above or below (e.g., Figures 1a-1d, 3-4). The method may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples above or below.

[0056] Various examples of the present disclosure relate to the concept of spatially selective (i.e., spatially varied) noise filtering on fluorescence images using information collected from, for example, white light images, i.e., the proposed concept may be said to relate to selective noise filtering.

[0057] In particular, the proposed concept can be used to filter noise from a fluorescent image using information from a white light image. By evaluating the white light image, the fluorescent image can be subdivided into background and fluorescent area portions. In practice, different filters can be applied to different parts of the image. For example, a high pass filter can be applied to the background and a low pass filter to the fluorescent area. This can reduce the noise on the fluorescent image and improve the image quality of the fluorescent image.

[0058] In particular, in some fluorescence imaging modes, the fluorescence emission is known to be restricted to blood vessels, and therefore (aggressive) filters can be applied to reduce background noise. Noise filtering can be made more efficient if additional information about the imaging sensor data is available. For example, if the image can be segmented into regions with uniform fluorescence intensity (e.g., coherent subparts), it may be more efficient to filter each segment separately, while preserving the sharpness of the segment boundaries. In more complex cases, it is possible to adapt / filter the signal if it is known that the intensity within a segment follows a certain pattern. In practice, the proposed concept is based on using additional information for the filtering process, which is extracted from a simultaneously captured white light image, or any other image. Furthermore, prior knowledge about the expected spatial pattern of the fluorescence signal can be used for a more effective filtering process.

[0059] FIG. 3 shows a schematic diagram of noise filtering applied to a noisy fluorescence image 310 (e.g., first imaging sensor data). In some systems, a universal noise filter 320 is applied to the noisy fluorescence image, resulting in a blurry filtered image 330, i.e., a filtered image with low contrast and soft edges. In the proposed concept, a white-light image 340 (e.g., second imaging sensor data) is segmented 350 into different parts 360 (e.g., coherent subparts) and a segment-wise noise filter 370 is applied, resulting in a higher-contrast filtered image 380 with sharper edges.

[0060] For example, ICG (Indo-Cyanine Green) is typically used for imaging blood vessels. The fluorescence intensity within each blood vessel can be considered constant within the vessel, while the shape of the vessel can be extracted from the white light image. A simple filtering model can be implemented by calculating the average fluorescence intensity within the vessel and assigning that value to every vessel pixel. A more complex model can assume a smooth variation (e.g., gradient) along the length (vertical axis) of the vessel while being constant along the width (horizontal axis). A 2D fitting model can then be applied to assign values ​​within the vessel pixels.

[0061] For background tissue, it may not be possible to assume a constant value for all pixels, but it can be assumed that there are no high spatial frequencies due to the typical smooth appearance of tissue without sharp edges. Therefore, a spatial low-pass filter may be applied. Such spatially selective / varied filtering models may result in a significant increase in sensitivity, since fluorescence values ​​may be calculated from hundreds or thousands of pixels. As a result, it may be possible to effectively detect signals that are one or two orders of magnitude weaker than in unfiltered images. As a result, it may be possible to administer 1 / 10 or 1 / 100 of the fluorescent dye dose.

[0062] A more general implementation of the proposed concept concerns the analysis of white-light images to calculate 2D spatial spectra representing the anatomical properties of tissues and, by extension, to gain knowledge about spatial patterns that are expected not to appear in fluorescence images and, conversely, that may potentially appear in fluorescence images.

[0063] The concept of spatially selective filtering may be relevant for example for image enhancement for use in microsurgery (Vascular & Oncology), which may be based on the use of software and / or machine learning.

[0064] Further details and aspects of the concept for spatially selective noise filtering are described in relation to the proposed concept or one or more examples above or below (e.g., Figures 1a-2). The concept for spatially selective noise filtering may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples above or below.

[0065] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0066] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block, item or feature of a corresponding apparatus.

[0067] Some embodiments relate to a microscope comprising a system as described in relation to one or more of Figures 1-3. Alternatively, the microscope may be part of or connected to the system as described in relation to one or more of Figures 1-3. Figure 4 is a schematic diagram illustrating a system 400 configured to perform the methods described herein. The system 400 comprises a microscope 410 and a computer system 420. The microscope 410 is configured to take images and is connected to the computer system 420. The computer system 420 is configured to perform at least some of the methods described herein. The computer system 420 may be configured to perform machine learning algorithms. The computer system 420 and the microscope 410 may be separate entities, but may be integrated in one common housing. The computer system 420 may be part of a central processing system of the microscope 410 and / or the computer system 420 may be part of a subcomponent of the microscope 410, such as a sensor, actor, camera or lighting unit of the microscope 410.

[0068] The computer system 420 may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed across various locations, e.g., local clients and / or one or more remote server farms and / or data centers). The computer system 420 may include any circuit or combination of circuits. In one embodiment, the computer system 420 may include one or more processors, which may be of any type. As used herein, a processor may mean any type of computational circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), for example, in a microscope or a microscope component (e.g., a camera), or any other type of processor or processing circuit. Other types of circuitry that may be included in computer system 420 may be custom circuitry, application specific integrated circuits (ASlCs), etc., such as one or more circuits (e.g., communications circuits) for use in wireless devices such as cell phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 420 may include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as a main memory in the form of random access memory (RAM), one or more hard drives, and / or one or more drives that handle removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), and the like.Computer system 420 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that enables a system user to input information to and receive information from computer system 420.

[0069] Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0070] Depending on the specific implementation requirements, the embodiments of the present invention can be implemented in hardware or software. The implementation can be performed using a digital storage medium, such as a non-transitory storage medium, such as a floppy disk, DVD, Blu-Ray, CD, ROM, PROM, and EPROM, EEPROM or FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system so that the respective methods are executed. Thus, the digital storage medium can be computer readable.

[0071] Some embodiments according to the invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0072] Generally, embodiments of the present invention can be implemented as a computer program product having program code which operates to perform one of the methods when the computer program product is run on a computer. The program code may for example be stored on a machine readable carrier.

[0073] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0074] In other words, an embodiment of the invention is therefore a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0075] A further embodiment of the invention is therefore a storage medium (or data carrier, or computer readable medium) storing a computer program for performing one of the methods described herein when executed by a processor. The data carrier, digital storage medium or recorded medium is typically tangible and / or non-transitory. A further embodiment of the invention is an apparatus as described herein, comprising a processor and a storage medium.

[0076] A further embodiment of the present invention is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals may for example be adapted to be transmitted via a data communication connection, for example via the Internet.

[0077] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0078] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0079] Further embodiments according to the invention comprise an apparatus or a system configured to transmit (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may for example be a computer, a mobile device, a memory device, etc. The apparatus or system may for example comprise a file server for transmitting the computer program to the receiver.

[0080] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0081] The embodiments may be based on the use of machine learning models or algorithms. Machine learning may refer to algorithms and statistical models that a computer system can use to perform a particular task without explicit instructions, relying instead on models and inferences. For example, in machine learning, instead of rule-based transformation of data, transformation of data inferred from analysis of historical data and / or training data may be used. For example, the content of an image may be analyzed using a machine learning model or using a machine learning algorithm. For a machine learning model to analyze the content of an image, the machine learning model may be trained using training images as input and training content information as output. By training the machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize the content of the image, so that content of the image not included in the training data can be recognized using the machine learning model. The same principle can be used for other types of sensor data, i.e., by training the machine learning model using training sensor data and a desired output, the machine learning model "learns" the transformation between sensor data and output, which can be used to provide an output based on the non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata and / or image data) may be pre-processed to obtain feature vectors that are used as input to a machine learning model.

[0082] The machine learning model may be trained using training input data. The above identified examples use a training method called "supervised learning". In supervised learning, the machine learning model is trained using multiple training samples, each of which may include multiple input data values ​​and multiple desired output values, i.e., each training sample is associated with a desired output value. By specifying both the training samples and the desired output value, the machine learning model "learns" and outputs values ​​to provide based on input samples similar to the samples provided during training. Apart from supervised learning, semi-supervised learning may be used. In semi-supervised learning, some of the training samples lack a corresponding desired output value. Supervised learning may be based on a supervised learning algorithm (e.g., a classification algorithm, a regression algorithm or a similarity learning algorithm). If the output is restricted to a set of limited values ​​(categorical variables), i.e., the input is classified into one of a set of limited values, a classification algorithm may be used. If the output can have any numerical value (within a range), a regression algorithm may be used. Similarity learning algorithms may be similar to both classification and regression algorithms, but are based on learning from examples using a similarity function that determines how similar or related two objects are. Apart from supervised or semi-supervised learning, unsupervised learning may also be used to train a machine learning model. In unsupervised learning, input data (only) may be provided, and unsupervised learning algorithms may be used to find structure in the input data (e.g., by grouping or clustering the input data and finding commonalities in the data). Clustering is the assignment of input data containing multiple input values ​​into subsets (clusters) such that input values ​​in the same cluster are similar according to one or more (predefined) similarity criteria, while dissimilar to input values ​​contained in other clusters.

[0083] Reinforcement learning is a third group of machine learning algorithms. In other words, reinforcement learning can be used to train machine learning models. In reinforcement learning, one or more software actors (so-called "software agents") are trained to perform actions in an environment. Based on the actions taken, a reward is calculated. Reinforcement learning is based on training one or more software agents to select actions such that the cumulative reward increases, leading to the software agent becoming better at a given task (as evidenced by an increase in reward).

[0084] Additionally, some techniques may be applied to some of the machine learning algorithms. For example, feature learning may be used. In other words, the machine learning model may be trained at least in part using feature learning, and / or the machine learning algorithm may comprise a feature learning component. Feature learning algorithms, sometimes called representation learning algorithms, can preserve information in the input, but also transform it to make it useful, often as a pre-processing step before performing classification or prediction. Feature learning may be based, for example, on principal component analysis or cluster analysis.

[0085] In some examples, anomaly detection (i.e., outlier detection) may be used, which aims to identify input values ​​that are significantly different from the majority of the input or training data and thus raise doubts. In other words, machine learning models may be at least partially trained using anomaly detection and / or machine learning algorithms may comprise an anomaly detection component.

[0086] In some examples, the machine learning algorithm may use a decision tree as a predictive model. In other words, the machine learning model may be based on a decision tree. In a decision tree, observations about an item (e.g., a set of input values) may be represented by a branch of the decision tree, and output values ​​corresponding to the item may be represented by a leaf of the decision tree. The decision tree may support both discrete and continuous values ​​as output values. If discrete values ​​are used, the decision tree may be represented as a classification tree, and if continuous values ​​are used, the decision tree may be represented as a regression tree.

[0087] Association rules are a further technique that can be used in machine learning algorithms. In other words, machine learning models can be based on one or more association rules. Association rules are created by identifying relationships between variables in large amounts of data. Machine learning algorithms can identify and / or utilize one or more association rules that represent knowledge derived from data. Rules can be used, for example, to store, manipulate, or apply knowledge.

[0088] A machine learning algorithm is usually based on a machine learning model. In other words, the term "machine learning algorithm" may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" may refer to a set of data structures and / or rules that represent learned knowledge (e.g., knowledge based on training performed by a machine learning algorithm). In an embodiment, the use of a machine learning algorithm may refer to the use of an underlying machine learning model (or multiple underlying machine learning models). The use of a machine learning model may refer to a machine learning model and / or a set of data structures / rules that are machine learning models being trained by a machine learning algorithm.

[0089] For example, the machine learning model may be an artificial neural network (ANN). An ANN is a system inspired by biological neural networks, such as those that can be found in the retina or the brain. An ANN comprises a number of interconnected nodes and a number of connections between the nodes, so-called edges. Typically, there are three types of nodes: input nodes, which receive input values, hidden nodes, which are connected (only) to other nodes, and output nodes, which provide output values. Each node may represent an artificial neuron. Each edge may transmit information from one node to another. The output of a node may be defined as a (non-linear) function of its inputs (e.g., the sum of its inputs). The inputs of a node may be used in a function based on the "weights" of the edges or nodes that provide the inputs. The weights of the nodes and / or edges may be adjusted in a learning process. In other words, training an artificial neural network may include adjusting the weights of the nodes and / or edges of the artificial neural network, i.e., to achieve a desired output for a given input.

[0090] Alternatively, the machine learning model may be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with an associated learning algorithm that can be used to analyze data (e.g., in classification or regression analysis). A support vector machine may be trained by providing inputs with multiple training input values ​​that belong to one of two categories. A support vector machine may be trained to assign new input values ​​to one of two categories. Alternatively, the machine learning model may be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network can use a directed acyclic graph to represent a set of random variables and their conditional dependencies. Alternatively, the machine learning model may be based on a genetic algorithm, which is a search algorithm and heuristic technique that mimics the process of natural selection. [Explanation of symbols]

[0091] 10 Samples, surgical site 12 Background Organization 14,16 blood vessels 14a, 14b, 14c Coherent subdivisions of blood vessels 100 Surgical Microscope System 105 Base Unit 110 System 112 Interface 114 processors 116 Storage device 120 Microscope 122 Optical Imaging Sensor 124 Sensor, Optical Imaging Sensor 130a Eyepiece Display 130b Auxiliary Display 140 Arm 210 Acquiring first imaging sensor data 220 Acquiring Second Imaging Sensor Data 230 Determining the extent of one or more anatomical features 240 Applying spatially-varying noise filtering 310 Noisy Fluorescence Images 320 Universal Noise Filter 330 Blurred Filtered Images 340 White Light Images 350 Segmentation 360 Segmented Part 370 Segment-based noise filter 380 Filtered Images 400 Systems 410 Microscope 420 Computer Systems

Claims

1. A system (110; 420) for a surgical microscope system (100; 400), said system comprising one or more processors (114) and one or more memory devices (116), The system comprises: acquiring first imaging sensor data of a view on a surgical site (10) from a first optical imaging sensor (122) of a microscope (120; 410) of said surgical microscope system; acquiring second imaging sensor data of the view on the surgical site from a second sensor of the microscope; determining an extent of one or more anatomical features of the surgical site based on the second imaging sensor data; configured to apply spatially varying noise filtering to the first imaging sensor data based on the extent of the one or more anatomical features of the surgical site. system.

2. the first imaging sensor data is based on fluorescence imaging; The system of claim 1 .

3. the second imaging sensor data is based on reflectance imaging; 3. A system according to claim 1 or 2.

4. the system is configured to determine at least one feature of interest from among the one or more anatomical features of the surgical site and apply the spatially varying noise filtering based on the range of the at least one feature of interest. A system according to any one of claims 1 to 3.

5. the one or more anatomical features are one or more blood vessels; the at least one feature of interest is at least one blood vessel that emits fluorescent light; The system of claim 4.

6. the system is configured to determine the at least one feature of interest from among the one or more anatomical features based on the first imaging sensor data.

6. A system according to claim 4 or 5.

7. the system is configured to determine the at least one feature of interest from among the one or more anatomical features based on a plurality of frames of the first imaging sensor data covering a predefined time interval or two or more predefined time points.

6. A system according to claim 4 or 5.

8. the system is configured to subdivide the first imaging sensor data into a first portion and a second portion, the first portion based on an extent of at least a subset of the one or more anatomical features of the surgical site; the system is configured to apply a first noise filter to the first portion and a different second noise filter to the second portion. A system according to any one of claims 1 to 7.

9. the system is configured to subdivide the first imaging sensor data such that the first portion includes the extent of at least the subset of the one or more anatomical features of the surgical site and the second portion includes a remaining portion of the first imaging sensor data. The system of claim 8.

10. the first noise filter is configured to apply a predefined intensity pattern to the first portion of the first imaging sensor data.

10. A system according to claim 8 or 9.

11. the predefined intensity pattern is configured to apply a uniform intensity distribution to the first portion or a coherent sub-portion of the first portion, or the predefined intensity pattern is configured to apply a uniform intensity distribution along a horizontal axis of the coherent sub-portion of the first portion and to apply an intensity gradient along a vertical axis of the coherent sub-portion of the first portion. The system of claim 10.

12. the system is configured to subdivide the first portion into a plurality of coherent sub-portions based on the extent of at least the subset of the one or more anatomical features and to apply the first noise filter separately to the coherent sub-portions. A system according to any one of claims 8 to 11.

13. the one or more anatomical features are one or more blood vessels, and the system is configured to subdivide the first portion into a plurality of coherent sub-portions based on branching points of the one or more blood vessels. The system of claim 12.

14. the second noise filter configured to apply a spatial low pass filter to the second portion of the first imaging sensor data. A system according to any one of claims 8 to 13.

15. The system is configured to determine the extent of one or more anatomical features of the surgical site using a machine learning model trained to perform image segmentation and / or object detection. A system according to any one of claims 1 to 14.

16. the system is configured to determine the extent of the one or more anatomical features of the surgical site based on at least one of a characteristic color spectrum and a characteristic shape of the one or more anatomical features of the surgical site.

16. A system according to any one of claims 1 to 15.

17. the system is configured to generate a display signal for a display device (130) of the surgical microscope system based on at least the filtered first imaging sensor data.

17. A system according to any one of claims 1 to 16.

18. the system is configured to generate a composite digital image of the surgical site based on the filtered first imaging sensor data and the second imaging sensor data, and generate the display signal based on the composite digital image.

20. The system of claim 17.

19. 1. A method for a surgical microscope system, the method comprising: acquiring (210) first imaging sensor data of a view on a surgical site from a first optical imaging sensor of a microscope of the surgical microscope system; acquiring (220) second imaging sensor data of the view on the surgical site from a second sensor of the microscope; determining (230) the extent of one or more anatomical features of the surgical site based on the second imaging sensor data; applying spatially varying noise filtering to the first imaging sensor data based on the extent of the one or more anatomical features of the surgical site (240); The method includes:

20. 20. A computer program having a program code for performing the method according to claim 19, when the computer program runs on a processor.

Images