Optical imaging system and related apparatus, method and computer program

Abstract

Examples relate to optical imaging systems and devices, methods, and computer programs for optical imaging systems. The optical imaging system includes a camera for providing a camera image of a surface of an object. The optical imaging system also includes one or more measurement modules for performing one or more measurements at one or more points on the surface of the object. The optical imaging system also includes a display module. The optical imaging system also includes a processing module configured to obtain a camera image of the surface of the object from the camera. The processing module is configured to obtain one or more measurement values ​​at one or more points on the surface of the object from the one or more measurement modules. The processing module is configured to determine the spatial position of one or more points of the surface relative to the camera image. The processing module is configured to provide the camera image with an overlay to the display module, and the overlay identifies the measurement values ​​at one or more points on the surface of the object.

Description

Technical Field

[0001] Examples relate to optical imaging systems and apparatus, methods and computer programs for optical imaging systems. Background Art

[0002] In the medical field, there are a multitude of technologies that can be used to assist surgeons during surgery on a patient. For example, a surgeon may use a surgical microscope to examine the tissue he or she is operating on. Additionally, other equipment may be used to examine the patient, such as a point probe or confocal microscope, which may be used to gain further insight into the patient's tissue. Unfortunately, in many cases, using multiple technologies simultaneously may result in diminishing returns, as the surgeon may have to consider and remember the details of each measurement or test he or she performs using different technologies. Summary of the invention

[0003] There may be a need for an improved concept that can increase the utility of an optical imaging device to a user, such as a surgeon using a surgical microscope, and that can gain benefits from using an optical imaging system with one or more other measurement devices.

[0004] Embodiments of the present disclosure relate to an optical imaging system. The optical imaging system includes a camera for providing a camera image of a surface of an object. The optical imaging system also includes one or more measurement modules, which are used to perform one or more measurements at one or more points on the surface of the object. The optical imaging system also includes a display module. The optical imaging system also includes a processing module, which is configured to obtain a camera image of the surface of the object from the camera. The processing module is configured to obtain one or more measurement values ​​at one or more points on the surface of the object from the one or more measurement modules. The processing module is configured to determine the spatial position of one or more points on the surface relative to the camera image. The processing module is configured to provide a camera image with an overlay to the display module, and the overlay represents the measurement values ​​at one or more points on the surface of the object.

[0005] By determining the spatial position of one or more points of the surface at which one or more measurements were made relative to the camera image, a spatial relationship between the one or more measurements and the camera image can be derived, and then an overlay showing the one or more measurements relative to the camera image can be generated. By creating an overlay, the camera image can be enhanced with the measurements, thereby enabling a user to consider information from different sources.

[0006] Embodiments of the present disclosure relate to a device for an optical imaging system. The device can be used to control the optical imaging device, such as processing of a camera and measurement data. The device includes an interface for communicating with a camera of the optical imaging system, one or more measurement modules, and a display module. The device includes a processing module, which is configured to obtain a camera image of a surface of an object from the camera of the optical imaging system. The processing module is configured to obtain one or more measurement values ​​at one or more points on the surface of the object from the one or more measurement modules. The processing module is configured to determine the spatial position of one or more points on the surface relative to the camera image. The processing module is configured to provide a camera image with an overlay to a display module, and the overlay represents the measurement values ​​at one or more points on the surface of the object.

[0007] Embodiments of the present disclosure relate to a corresponding method for an optical imaging system. The method includes obtaining a camera image of a surface of an object from a camera of the optical imaging system. The method includes obtaining one or more measurement values ​​at one or more points on the surface of the object from one or more measurement modules of the optical imaging system. The method includes determining the spatial position of one or more points of the surface relative to the camera image. The method includes providing a camera image with an overlay to a display module, the overlay representing the measurement values ​​at one or more points on the surface of the object. Embodiments of the present disclosure relate to a computer program with a program code, and when the computer program is executed on a processor, the program code is used to execute the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Some examples of devices and / or methods will be described below, by way of example only, and with reference to the accompanying drawings, in which

[0009] Figure 1 A block diagram illustrating embodiments of an optical imaging system and an apparatus for an optical imaging system;

[0010] Figure 2 A flow chart illustrating an embodiment of a method for an optical imaging system;

[0011] Figure 3 A block diagram illustrating generation of an overlay by a processing unit according to an embodiment;

[0012] Figure 4 A schematic diagram of a system including a microscope or optical imaging system and a computer system is shown. DETAILED DESCRIPTION

[0013] Various examples will now be described more fully with reference to the accompanying drawings showing some examples. In the accompanying drawings, the thickness of lines, layers and / or regions may be exaggerated for clarity.

[0014] Therefore, although further examples can have various modifications and alternative forms, some specific examples thereof are shown in the accompanying drawings and will be described in detail later. However, these detailed descriptions will not limit further examples to the specific forms described. Further examples may cover all modifications, equivalents and substitutes falling within the scope of the present disclosure. Throughout the description of the accompanying drawings, the same or similar numbers represent the same or similar elements, which, when compared to each other, can be implemented in the same or modified form while providing the same or similar functions.

[0015] It should be understood that when an element is referred to as being "coupled" or "connected" to another element, these elements may be coupled or connected directly or via one or more intermediate elements. If two elements A and B are combined together using "or", this should be understood to disclose all possible combinations, i.e. only A, only B, and A and B, if not explicitly or implicitly defined. Another way of saying the same combination is "at least one of A and B" or "A and / or B". The same applies to combinations of more than two elements, mutatis mutandis.

[0016] The terms used to describe specific examples herein are not intended to limit further examples. Whenever singular forms such as "one", "an" and "the" are used, and only using a single element is neither explicitly nor implicitly defined as mandatory, further examples can also use plural elements to implement the same function. Similarly, when a function is subsequently described as being implemented using multiple elements, further examples can use a single element or processing entity to implement the same function. It is also understood that the terms "include", "contain", "includes" and / or "have" when used, specify the existence of stated features, integers, steps, operations, processes, actions, elements and / or components, but do not exclude the existence or addition of one or more other features, integers, steps, operations, processes, actions, elements, components and / or any combination thereof.

[0017] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same common meaning as those in the art to which these examples belong.

[0018] Figure 1 A block diagram of an embodiment of an optical imaging system 100, such as a microscope or surgical microscope 100, is shown. The optical imaging system includes a camera 120 for providing a camera image of a surface of an object. The optical imaging system includes one or more measurement modules 130 for performing one or more measurements at one or more points on the surface of the object. The optical imaging system includes a display module 140. The optical imaging system includes a processing module 114, which is coupled to the camera, the one or more measurement modules, and the display module 140, such as via an interface 112. Figure 1Also shown is a device 110, which includes a processing module 114 and an interface 112 coupled to the processing module 114. The processing module 114 is configured to obtain a camera image of a surface of an object from a camera. The processing module 114 is configured to obtain one or more measurements at one or more points of the surface of the object from one or more measurement modules 130. The processing module 114 is configured to determine the spatial position of the one or more points of the surface relative to the camera image. The processing module 114 is configured to provide the camera image with an overlay to a display module 140, the overlay representing the measurements at one or more points of the surface of the object.

[0019] Embodiments relate to an optical imaging system, or an apparatus, method, or computer program for an optical imaging system. For example, the optical imaging system may be or may include a microscope. For example, the optical imaging system may be a system including a microscope (e.g., a surgical microscope), a camera 120 as a camera for the microscope, and one or more measurement modules. In other words, the one or more measurement modules may be external to the microscope and be part of the system including the microscope. For example, the optical imaging system may be a surgical microscope system or an image guidance system. In general, a microscope is an optical instrument suitable for examining objects that are too small to be examined by the human eye (alone). For example, a microscope may provide optical magnification of an object (e.g., a sample of organic tissue). In modern microscopes, optical magnification is typically provided to a camera or imaging sensor, e.g. Figure 1 camera 120 of the optical imaging system. In other words, the optical imaging system 100 may also include one or more optical magnification components that are used to magnify the view of the object. If the optical imaging system is used in the medical field, the object may be a sample of organic tissue, such as organic tissue arranged in a culture dish or present in a part of the body. For example, the microscope 100 may be a microscope used in a laboratory, such as a microscope that can be used to examine samples of organic tissue in a culture dish. Alternatively, the microscope 100 may be part of a surgical microscope system, such as a microscope used during surgery. Although the embodiments are described in conjunction with a microscope, they may also be applied in a more general manner to any optical device. For example, the optical imaging system may be a system for performing material testing or integrity testing of a material, such as a metal or a composite material. In this case, one or more measurements may be performed on the material or on the material.

[0020] The optical imaging system includes a camera 120. Typically, the camera 120 may include an imaging sensor, such as an APS (active pixel sensor) or CCD (charge coupled device) based imaging sensor. For example, in an APS based camera, light is recorded at each pixel using a photodetector and an active amplifier of the pixel. APS based imaging sensors are typically based on CMOS (complementary metal oxide semiconductor) or S-CMOS (scientific grade complementary metal oxide semiconductor) technology. In a CCD based imaging sensor, incident photons are converted into electron charges at the semiconductor-oxide interface, which are then moved between capacitor boxes in the imaging sensor module by the control circuit of the sensor imaging module for imaging.

[0021] The camera 120 is adapted to or configured to provide a camera image of the surface of an object. In general, the camera image may be an analog or digital camera image, i.e., the camera image may be provided as an analog camera signal or a digital video signal or a digital still image. The processing module 114 is configured to obtain a camera image of the surface of the object from the camera. For example, the processing module 114 may be configured to obtain a camera image as an analog camera signal or as a digital video signal or a digital still image from the camera, for example, via a wireless or wired connection or via an intermediate storage or memory, for example, via an interface 112. For example, the camera image may be a two-dimensional camera image based on pixels. In some embodiments, the camera image may be a single image, i.e., a static image. Alternatively, the camera image may be an image of a camera image stream (i.e., a video) that is periodically updated. In some embodiments, the term "camera image" may be used for multiple (substantially) simultaneously recorded images. For example, a camera image may include a "white light" camera image (captured across the entire visible spectrum) and one or more reflectance images and / or fluorescence images (e.g., images in which one or more wavelength bands are isolated and which include the results of reflectance or fluorescence images recorded in one or more wavelength bands). Generally speaking, in reflectance imaging, an object is illuminated in one wavelength band and an image of the object is recorded in the same wavelength band. In fluorescence imaging, an object is illuminated in a first wavelength band and an image is recorded in a second, different wavelength band.

[0022] As previously mentioned, the object may be a sample of organic tissue, such as a sample of organic tissue consisting of a part of the body, or a sample of organic tissue arranged on a culture dish or a microscope slide. Therefore, the camera image of the surface of the object may be a camera image showing a top view of the surface of the sample of organic tissue, such as a view of the outside of the sample of organic tissue seen from a camera of a microscope. Alternatively, the object may be any other object observable by an optical imaging system, such as an object to be inspected in a materials inspection. In this case, the camera image of the surface of the object may show a top view or a side view of the object to be inspected.

[0023] The optical imaging system includes one or more measurement modules 130, which are used to perform one or more measurements at one or more points on the surface of the object. Generally, the one or more measurement modules 130 can be one or more measurement modules suitable for measuring at a point on the surface of the object, that is, indicating a certain property of the surface of the object at this point. In some embodiments, this does not exclude that the measurement module measures an area that is larger than the contrast point but smaller than the surface covered by the camera image, such as an area of ​​at most 5% (or at most 2%, at most 1%, at most 0.5%, at most 0.2% or at most 0.1%) of the surface covered by the camera image. In other words, one or more measurements can be performed on an area of ​​at most 5% (or at most 2%, at most 1%, at most 0.5%, at most 0.2% or at most 0.1%) of the surface covered by the camera image.

[0024] Various measurement means can be used to perform the measurement. For example, one or more measurements may include at least one of the following: one or more optical measurements, one or more electrical measurements, one or more acoustic measurements, one or more mechanical measurements, one or more thermal measurements, one or more tissue sampling measurements, and one or more measurements based on human sensory reception. For example, one or more optical measurements may be performed by one or more optical measurement modules, such as by one or more of a confocal microscope, a fluorescence imaging module, a reflection imaging module, a spectral module, and a laser ablation module. Therefore, one or more optical measurements may include one or more of a fluorescence image, a reflection image, a spectral measurement (e.g., a Raman spectroscopy measurement), and a laser ablation measurement at at least a subset of one or more points. For example, one or more electrical measurements may be performed by one or more electrical measurement modules, such as by a neural stimulation module and / or by an electric potential measurement module. Therefore, one or more optical measurements may include one or more of a neural stimulation measurement value and an electric potential measurement value at at least a subset of one or more points. For example, one or more acoustic measurements may be performed using, for example, an ultrasonic sensor. Therefore, one or more acoustic measurements may include one or more ultrasonic measurements at at least a subset of one or more points and / or one or more measurements related to tissue elasticity. For example, one or more mechanical measurements may be performed by a tactile measurement device. One or more mechanical measurements may include one or more tissue elasticity measurements at at least a subset of one or more points. For example, one or more thermal measurements may be performed by one or more thermal measurement modules, such as using one or more probes or using one or more optical thermal measurements. One or more thermal measurements may represent the temperature at at least a subset of one or more points. For example, one or more tissue sampling measurements may be performed using a biopsy or mass spectrometry. One or more tissue sampling measurements may include one or more measurements based on a biopsy at at least a subset of one or more points, or one or more mass spectrometry measurements at at least a subset of one or more points. Finally, one or more measurements received based on human senses, such as palpation measurements, may be manually input by a person, such as classified by a person and input using a human input interface. For example, the human input interface may be a touch interface or a voice control or a spoken interface. In an embodiment, a combination of the above measurements may be used for one or more measurements at one or more points on the surface of an object.

[0025] The processing module is configured to obtain one or more measurement values ​​at one or more points of the surface of the object from the one or more measurement modules 130. For example, the one or more measurement values ​​can be obtained (i.e., received) from the one or more measurement modules as analog or digital sensor data. If received using human senses, at least a subset of the one or more measurement values ​​can be obtained via a human-machine interface. In this case, the human-machine interface can be regarded as one of the one or more measurement modules.

[0026] The optical imaging system includes a display module 140. For example, the display module can be one of a projection-based display module and a screen-based display module, such as a liquid crystal display (LCD)-based display module or an organic light emitting diode (OLED)-based display module. The display module 140 can be controlled by the processing module 114, such as by a video or control signal provided by the display module. For example, the video or control signal can be based on a camera image and an overlay. In some embodiments, the camera image (or at least a processed version of the camera image) and the overlay can be provided to the display module 140 separately, such as as two different video or control signals. Optionally, the camera image and the overlay can be provided to the display module 140 in a combined form, such as as a video or control signal including a combination of both the camera image (or at least a processed version of the camera image) and the overlay.

[0027] The processing module 114 is configured to determine the spatial position of one or more points of the surface relative to the camera image. For example, the camera image may represent a portion (i.e., a section) of the surface of an object. The spatial position of a point of the surface relative to the camera image may represent the position of the point within the camera image (i.e., a portion of the surface of the object), i.e., within a portion of the surface of the object, at a position where a measurement has been performed at the point. In other words, one or more points of the surface may be projected onto the camera image, for example, so that one or more points are assigned to corresponding positions within the camera image. This assignment may be achieved in various ways. For example, at least some of the one or more measurements may be performed using a measurement probe. The processing module 114 may be configured to identify the position of one or more probes used to perform one or more measurements within the camera image to determine the spatial position of one or more points of the surface relative to the camera image. For example, the processing module 114 may be configured to detect the measurement points of one or more probes within the camera image (e.g., based on the optical markers of the corresponding probes), and determine the spatial position of one or more points based on the position of the measurement points within the camera image (e.g., based on the previous camera image). This may enable positioning of the measurements performed using the probe.

[0028] In some embodiments, at least some of the probes may be equipped with a light source that can be used to track the probe in two-dimensional or three-dimensional space. Light can therefore help find the probe in a camera image. For example, the light source of the corresponding probe can be used as an optical marker of the probe to achieve probe positioning as described above. The processing module 114 can be configured to use the light emitted by one or more light sources attached to the one or more probes to identify the position of one or more probes used to perform one or more measurements to determine the spatial position of one or more points on the surface relative to the camera image. For example, the processing module 114 can be configured to detect light emitted by one or more light sources attached to the one or more probes in the camera image (e.g., in a previous camera image) to determine the position of one or more probes based on the detected light, and determine the spatial position of one or more points on the surface relative to the camera image based on the determined position of the one or more probes. Additionally or optionally, the optical imaging system may include one or more additional cameras for tracking the light source of the corresponding probe. For example, one or more light sources attached to one or more probes may be configured to emit infrared light (e.g., near-infrared light), and the processing module 114 may be configured to use one or more additional cameras to detect light emitted by the one or more light sources attached to the one or more probes, determine the positions of the one or more probes based on the detected light, and determine the spatial positions of one or more points on the surface relative to the camera image based on the determined positions of the one or more probes, wherein the one or more additional cameras are cameras for sensing infrared light.

[0029] In some embodiments, a method independent of the camera image may be selected. For example, at least a subset of the measurements may be performed using a robotic arm. The orientation of the robotic arm may be known to or controlled by the processing module. Based on the orientation of the robotic arm, the spatial position of at least a subset of the one or more measurements may be obtained. In other words, the processing module 114 may be configured to determine the spatial position of one or more points of the surface relative to the camera image based on the orientation of the robotic arm when performing the subset of one or more measurements.

[0030] As described above, at least some of the measurements may be made based on human sensory perception. In this case, the spatial location of one or more points of the surface of one or more corresponding measurements relative to the camera image may be manually input, or may be detected within the camera image by the processing module 114, for example, in response to voice control by a user / surgeon.

[0031] The processing module is configured to provide a camera image with an overlay to the display module 140, the overlay representing the measurement values ​​at one or more points on the surface of the object. As described above, the processing module can be configured to generate a video or control signal for the display module based on the camera image and the overlay, such as a video or control signal comprising a combination of both the camera image and the overlay, or two separate video or control signals, one comprising the camera image and one comprising the overlay. In the case of using a combined video or control signal, the processing module can be configured to merge the camera image and the overlay, that is, the processing module can be configured to add the overlay to the camera image, or generate a new video signal comprising both the camera image and the overlay. For example, the processing module 114 can be configured to provide the video or control signal to the display module via the interface 112. In any case, the camera image provided to the display module can be or can include a white light image, or one of one or more reflectance / fluorescence images, or a combination thereof.

[0032] The processing module 114 may also be configured to generate an overlay based on one or more measurements at one or more points on the surface of the object and based on the spatial position of the one or more points on the surface relative to the camera image. For example, in a simple embodiment, the processing module 114 may be configured to generate an overlay by generating one or more indicator symbols, indicator colors, or numbers that represent the one or more measurements at the spatial position of the one or more points on the surface relative to the camera image. For example, thermal readings may be represented by indicator colors or numbers, while pathological tissue may be represented by a "negative" symbol and healthy tissue may be represented by a "positive" symbol (or red and green). One or more indicator symbols, indicator colors, or numbers may be placed in the overlay at the spatial position of one or more points on the surface relative to the camera image. Such an overlay may be seen. Figure 3 , reference numeral 330.

[0033] In some embodiments, additional processing may be performed to increase the utility of the overlay. For example, the processing module may be configured to generate an overlay to represent one or more coherent spatial regions in which similar measurements are grouped. The processing module 114 may be configured to group one or more measurements into one or more coherent spatial regions. In this case, a "coherent spatial region" may represent a coherent (i.e., tightly packed, uninterrupted) region within a camera image and which includes similar measurements from one or more measurements. For example, the processing module 114 may be configured to classify one or more measurements into one or more groups of similar measurements, or one or more groups of measurements indicating a predefined condition, and generate one or more coherent spatial regions based on the one or more groups of similar measurements, or one or more groups of measurements indicating a predefined condition. The processing module 114 may be configured to provide a camera image with an overlay to the display module 140, the overlay representing one or more coherent spatial regions. Such an overlay may be viewed Figure 3 , reference numeral 360.

[0034] For example, one or more coherent spatial regions can be determined for a sample of organic tissue. In this case, for example, healthy tissue and pathological tissue (i.e., unhealthy tissue) can be distinguished. In other words, the processing module 114 can be configured to classify the surface of the sample of organic tissue into healthy tissue and pathological tissue at one or more point-measured spatial positions, for example, based on one or more measurement values ​​and / or based on the camera image. For example, multiple categories can be used depending on the capabilities of the measurement module. Some modes have a fairly simple classification process, for example, a fluorescence signal above a certain threshold means that the disease is positive, while other modes require more complex processing, such as confocal images that require complex image analysis or AI (artificial intelligence, such as machine learning models). For example, the processing module 114 can be configured to classify the surface of the sample of organic tissue into healthy tissue and pathological tissue at one or more point-measured spatial positions based on the camera image, for example, based on the fluorescence image included in the camera image. For example, if the brightness within the fluorescence image is above a threshold, the diseased / pathological tissue can be determined. Alternatively or additionally, the processing module 114 can be configured to classify the surface of the sample of organic tissue into healthy tissue and pathological tissue at one or more point-measured spatial positions based on one or more measurement values. For example, the processing module 114 may be configured to classify, for each of the one or more points, whether the tissue at the point is considered healthy or pathological. The result may be a cloud of classified points scattered in 2D space.

[0035] The processing module 114 can be configured to group one or more measurements into two or more coherent spatial regions. The two or more coherent spatial regions can include at least one healthy tissue spatial region and at least one pathological tissue spatial region. For example, the processing module 114 can be configured to use the spatial position of one or more points of the surface relative to the camera image for different categories (e.g., healthy or pathological tissue). In addition, the processing module 114 can also be configured to, for example, determine a probability or certainty value or graph for each of the one or more points or for each coherent spatial region, for example, to enable the surgeon to identify how much they can trust the classification, or whether additional measurements are needed. The processing module can be configured to use multiple measurements, or a camera image together with one or more measurements, in order to perform classification. For example, fluorescence imaging that can be included in the camera image, such as fluorescence induced by 5-ALA (5-aminolevulinic acid), can be a marker for brain tumors. In addition, color images can be analyzed using image analysis algorithms or machine learning models. More generally, the processing module can be configured to use a machine learning model to classify the surface of a sample of organic tissue into healthy tissue and pathological tissue at spatial locations measured at one or more points, for example using a camera image and / or one or more measurement values ​​as input to the machine learning model, and obtaining an output indicating whether the surface at one of the one or more points is considered to be healthy or pathological.

[0036] For example, if the object is a sample of organic tissue, the one or more measurements may include at least one of: one or more measurements related to blood oxygen content, one or more measurements related to tissue perfusion, and one or more measurements related to the temperature of the organic tissue. The processing module 114 can be configured to classify the surface of the sample of organic tissue into healthy tissue and pathological tissue at one or more spatial locations measured at the one or more points based on at least one of the blood oxygen content, tissue perfusion, and temperature at the one or more spatial locations measured at the one or more points. The classification method can then be used to determine two or more coherent spatial regions. For example, at least one of the two or more spatial regions can indicate healthy tissue and / or at least one of the two or more spatial regions can indicate pathological tissue.

[0037] In at least some embodiments, the camera image and / or one or more measurements can be used to create and visualize a map that can be used as an overlay. For example, the map can include points where measurements were taken using one or more measurement modules. Location markers can also indicate the classification result, i.e., healthy tissue or diseased tissue. The classification can be indicated by symbols, colors, or time-modulated symbols. Additionally or alternatively, the map can include a classification of one or more coherent spatial regions. One approach is to use a "color-heat map" similar to that used for weather / temperature forecasts, i.e., it can use green and red for healthy tissue and diseased tissue accordingly, and the level of transparency can vary depending on the confidence, with transparency indicating low certainty and dark opaque colors indicating high certainty.

[0038] In organic tissue, anatomical features such as blood vessels, adipose tissue, bones, tumors, etc. can be found. Typically, measurements within anatomical features can be similar - for example, blood oxygen content can be similar at different points in a blood vessel, and temperature can also be similar. Therefore, one or more coherent regions can be mapped to corresponding coherent regions within the camera image (which represent anatomical features). For example, the processing module 114 can be configured to determine one or more coherent regions within the camera image. For example, one or more coherent regions can represent one or more anatomical features of a sample of organic tissue. The processing module 114 can be configured to further group one or more measurements into one or more coherent spatial regions based on the one or more coherent regions within the camera image. For example, a coherent spatial region to be overlaid on the camera image can be aligned with one or more coherent regions within the camera image. This can provide additional content for the measurement.

[0039] In some embodiments, the measurement is performed within a numerical scale, such as a scale based on numerical values. For example, blood oxygen content can be measured as a percentage of a percentage scale, and temperature can be measured in a Celsius (or Fahrenheit) scale. Such values ​​can be represented by colors, for example, in an overlay, where the colors represent values ​​within a numerical scale. In some embodiments, the colors can be placed as color dots at the spatial positions of one or more points relative to the camera image. In order to provide a better overview, the values ​​between one or more points can be interpolated so that the surgeon can grasp the transition in the measurement more quickly. For example, the processing module 114 can be configured to determine (i.e., estimate and / or insert) one or more intermediate values ​​of the spatial position between one or more points on the surface based on one or more measured values. For example, the processing module 114 can be configured to determine one or more intermediate values ​​of the spatial position between one or more points on the surface based on a grid across the camera image, i.e., points on the grid. The processing module 114 can be configured to determine one or more intermediate values ​​by interpolating or estimating one or more intermediate values ​​(linearly) based on one or more measurements, for example, based on the spatial position of one or more points where one or more measurements have been performed. The processing module 114 may be configured to provide the camera image with an overlay representing the one or more measured values ​​and the one or more intermediate values ​​within a continuous spectrum (eg, within a continuous spectrum of colors representing the one or more measured values ​​and the one or more intermediate values).

[0040] In some embodiments, the distribution of one or more points at the surface of the object may be analyzed to determine how one or more measurements may be enhanced with additional measurements. For example, it may be useful to perform additional measurements at anatomical features that have not been previously probed, or at "holes" in one or more measurements, such as between measurements that indicate healthy and pathological tissue, respectively. The processing module 114 may be configured to determine the distribution of one or more points at the surface of the object. For example, the processing module 114 may be configured to determine at least one of the following: a local density of the distribution, a uniformity of the distribution, and one or more gaps within the distribution of one or more points at the surface of the object. The processing module 114 may be configured to determine one or more additional points (e.g., a scan path) of the surface of the object at which one or more additional measurements are required based on the distribution of the one or more points at the surface of the object. For example, the processing module 114 may be configured to determine one or more additional points of the surface of the object at which one or more additional measurements are required, such that the local density of the distribution is improved, the uniformity of the distribution is improved, and / or one or more gaps within the distribution are reduced or closed. For example, the one or more additional points of the surface of the object at which one or more additional measurements are required may be points at which measurements may be performed to improve at least one of the above criteria.

[0041] In some embodiments, one or more additional points on the surface of an object where one or more additional measurements need to be made at the surface of the object can be determined based on one or more coherent regions within the camera image. For example, if there are enough measurements in a coherent region, the coherent region may not produce additional points that require additional measurements. On the other hand, if another coherent region includes very few or no measurements, additional measurements can be performed in that region. Additionally or optionally, additional measurements may be required in the region where the disease boundary is to be determined. In other words, the processing module 114 can be configured to further determine one or more additional points at the surface of the object where one or more additional measurements need to be made based on one or more coherent regions within the camera image, for example, so that one or more additional points are determined for regions in one or more coherent regions where the number or distribution of measurements is insufficient. This can improve the effectiveness of additional measurements because measurements can be made where they are most needed.

[0042] The interface 112 may correspond to one or more inputs and / or outputs for receiving and / or sending information, which may be digital (bit) values ​​according to a specified code within a module, between modules, or between modules of different entities. For example, the interface 112 may include an interface circuit configured to receive and / or send information. In an embodiment, the processing module 114 may be implemented using one or more processing units, one or more processing devices, or any device for processing, such as a processor, a computer, or a programmable hardware component that can operate with corresponding adapted software. In other words, the functions of the processing module 114 described may also be implemented in software, which is then executed on one or more programmable hardware components. Such software components may include a general-purpose processor, a digital signal processor (DSP), a microcontroller, etc.

[0043] In combination with the proposed concept or one or more examples described above or below (e.g. Figure 2 or Figure 3 ), mentions more details and aspects of the optical imaging system or the apparatus for the optical imaging system. The optical imaging system or the apparatus for the optical imaging system may include one or more additional optional features, which correspond to one or more aspects of the proposed concept or one or more examples described above or below.

[0044] Figure 2 An optical imaging system (eg Figure 1Flowchart of an embodiment of a corresponding method (e.g., a computer-implemented method) for an optical imaging system 100). Although aspects of the optical imaging system have been described in the context of a device, it is apparent that these aspects also represent a description of a corresponding method, where properties or functional features of the device correspond to method steps or features of method steps. The method includes 210, obtaining a camera image of a surface of an object from a camera of the optical imaging system 100. The method includes 220, obtaining one or more measurement values ​​at one or more points on the surface of the object from one or more measurement modules 130 of the optical imaging system 100. The method includes 230, determining the spatial position of one or more points of the surface relative to the camera image. The method includes 240, providing a camera image with an overlay to a display module 140, the overlay representing the measurement values ​​at one or more points on the surface of the object.

[0045] As mentioned above, combined Figure 1 The features described in the optical imaging system 100 can also be applied to Figure 2 method.

[0046] In combination with the proposed concept or one or more examples described above or below (e.g. Figure 1 or Figure 3 ), mentions more details and aspects of the method. The method may include one or more additional optional features, which correspond to one or more aspects of the proposed concept or one or more examples described above or below.

[0047] At least some embodiments of the present disclosure relate to concepts for tracking point measurements and providing corresponding visual mappings.Embodiments may be based on collecting and visualizing different measurement data (eg, one or more measurements), in particular point measurements such as confocal microscopy results.

[0048] Various diagnostic modes can be used in parallel with the surgical microscope to provide additional information. For example, one or more measurement modules may include one or more of the following diagnostic modes. These modes may be point measurements or imaging compared to the field of view of the microscope. For example, both ultrasound and confocal microscopes can provide images, but the size of the sampling / imaging area is significantly different. For example, the image size provided by ultrasound (US) can be compared with the field of view (FOV) of the microscope, so even if the size is different, the US image can be overlaid on the optical image. However, the confocal microscope (CM) may be intended to observe cell morphology and may therefore have a higher magnification than the surgical microscope. Therefore, typically, a CM image (typically 0.5 mm in diameter) only covers a few pixels of a surgical microscope image (typically 20 mm in diameter). In this example, the CM image may be a measurement of one or more measurements, and the surgical microscope image may be a camera image. In essence, although the CM measurement is technically an image, it is only used as a point measurement in the context of the surgical microscope image. The CM image can report the pathology of the sampling area, i.e., healthy or diseased, without providing any spatial information about how the disease is distributed within the microscope image. Therefore, multiple CM measurements can be used to gather some information about the healthy and / or disease spatial distribution, i.e., disease boundaries. For example, other modalities may include various spectroscopy types, such as fluorescence, reflectance, Raman spectroscopy, laser ablation, and polarization or neural stimulation electrodes.

[0049] In some systems, point measurements are used as a random sampling tool without any record of the location and corresponding findings. This process is similar to a metal detector that beeps when there is metal on the ground, but does not provide a map. Therefore, the surgeon can try to remember the measurements so that he can create some sense of the spatial distribution in his mind. This may not be ideal because the virtual map in the surgeon's mind may be very rough and prone to errors. In addition, this information may not be saved and may be difficult to remember even during surgery. In addition, the spatial sampling of tissue may be quite random, so it may be difficult to remember where a previous measurement was taken. In addition, the association between different modes may be very difficult. For example, it may be difficult to cross-check fluorescence images and multiple CM results.

[0050] Embodiments of the present disclosure provide a method and system for collecting, visualizing, and storing point measurements. The system may include at least some of the following components:

[0051] First, a system, such as an optical imaging system, may include one or more measurement modes, such as a camera and one or more measurement modules. In this case, the system may include a camera, typically a main camera of a microscope. In addition, the system may further include one or more measurement modes, typically one or more point measurement devices. In other words, at least a subset of the one or more measurement modules may be point measurement modules. The measurement modes may include one or more of the following:

[0052] -Optical modes such as fluorescence, reflectance, spectroscopy, laser ablation

[0053] - Electrical modalities, such as nerve stimulation and potential measurement

[0054] -Acoustic modes, such as ultrasound

[0055] - Mechanical modes, such as tissue elasticity (this can also be measured with acoustic measurements)

[0056] - Thermal mode (thermal measurements can also be performed optically)

[0057] -Tissue sampling modalities, e.g. biopsy, mass spectrometry

[0058] - Human sensory modalities such as palpation and manual classification of each point

[0059] The system may also include a processing unit, such as a processing module or device 110, which can collect data from the measurement mode. For example, the processing unit can include one or more of the following features:

[0060] Point measurements can be performed together with a simultaneous measurement of the spatial coordinates relative to the image captured by the microscope camera. This can be achieved, for example, by:

[0061] - Uses a microscope camera and point measurement probe that can see the tissue, so the tip position at the time of measurement can be recorded

[0062] - Use an external position measurement system that provides information about the measuring tip. This can be easily achieved with three or more NIR (near infrared) LEDs and a NIR camera attached to the probe

[0063] -By using a robotic arm that scans the tissue surface with a probe tip. The robot can provide the coordinates of the tissue in space.

[0064] Point measurements can be processed to produce a classification of tissue, such as healthy and diseased (i.e., pathological), for example as a precursor to determining one or more coherent spatial regions. Depending on the capabilities of the measurement modality, multiple categories can be used. Some modalities have a fairly simple classification process, for example, a fluorescence signal above a certain threshold means positive for disease, while other modalities require more complex processing, such as confocal images requiring complex image analysis or AI (artificial intelligence). The result may be a cloud of classified points scattered in 2D space.

[0065] The processing unit (i.e., processor) can use the spatial distribution of the measurements to segment the 2D space into different categories used, such as into one or more coherent spatial regions. In the simplest case, the segmentation is into healthy regions and diseased regions. This segmentation can also include probabilistic or deterministic maps so that the surgeon can know how much they can trust the calculated map, or whether additional measurements are needed.

[0066] In some embodiments, the processing unit can also suggest sampling locations or scanning paths in order to clarify the most ambiguous areas. Thus, it can reduce the workload of the surgeon and increase the efficiency of the procedure.

[0067] The processing unit can use multiple diagnostic data in order to perform the classification. For example, the following data can be used for comprehensive evaluation. For example, fluorescence imaging, such as 5-ALA-induced fluorescence, can be a marker for brain tumors. In addition, color images can be analyzed using image analysis algorithms or AI (e.g., machine learning models).

[0068] In at least some embodiments, the captured data can be used to create and visualize a graph that can be used as an overlay. For example, the graph can include any of the following data. For example, the graph can include an image of the tissue, such as a camera image. The image can typically be a color image showing the anatomy of the tissue. However, it can also be any combination of color, fluorescence, and multispectral images. Additionally (or optionally), the graph can include points measured using a point measurement mode. Position markers can also indicate the diagnosis, i.e., healthy tissue or diseased tissue. It can be implemented with symbols, colors, or time-modulated symbols. Additionally or optionally, the graph can include a classification of image regions, such as one or more coherent spatial regions. One approach is to use a "color-heat map" similar to that used for weather / temperature forecasts, i.e., it can use green and red for healthy tissue and diseased tissue accordingly, and the level of transparency can vary depending on the confidence, with transparency meaning low confidence and dark color opacity meaning high certainty.

[0069] Figure 3 A block diagram of an overlay generated by a processing unit according to an embodiment is shown. Figure 3In a system, a camera 120 of a surgical microscope is used to generate a camera image 320 of a sample surface of an organic tissue. At the same time, one or more measurements 330 are performed at the tissue 310 using a point measurement mode 130 (e.g., a measurement module). In the figure shown in the reference numeral 330, disease-positive measurements are represented by a plus sign and disease-negative measurements are represented by a minus sign. Both the camera image and the measurements are provided to a processing unit 110 (e.g., a device 110 including a processing module 114). Based on user settings 340 and artificial intelligence 350 (e.g., a machine learning model), the processing unit can generate a combined image 360 ​​based on the camera image 320 and the map of measurements 330, and wherein coherent spatial regions 362, 364, 366 are displayed as overlaid on the camera image, the coherent spatial regions indicating healthy tissue (362, 366) and diseased tissue (364).

[0070] In combination with the proposed concept or one or more examples described above or below (e.g. Figure 1 and / or Figure 2 ), mentioned the combination Figure 3 More details and aspects of the concepts shown. Figure 3 The illustrated concepts may include one or more additional optional features corresponding to one or more aspects of the presented concepts or one or more examples described above or below.

[0071] Embodiments may be based on the use of machine learning models or machine learning algorithms. Machine learning may refer to algorithms and statistical models that a computer system can use to perform specific tasks without using explicit instructions, but instead relying on models and inferences. For example, in machine learning, transformations of data inferred from analysis of historical and / or training data may be used instead of rule-based transformations of data. For example, a machine learning model may be used or a machine learning algorithm may be used to analyze the content of an image. In order for a machine learning model to analyze the content of an image, a machine learning model may be trained using training images as input and training content information as output. By training a machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize the content of the image, so the machine learning model can be used to recognize the content of the image not included in the training data. The same principle can also be used for other types of sensor data: by training a machine learning model using training sensor data and desired outputs, the machine learning model "learns" the transformation between sensor data and output, which can be used to provide output based on non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) may be preprocessed to obtain a feature vector, which is used as input to the machine learning model.

[0072] The machine learning model can be trained using training input data. The examples detailed above use a training method called "supervised learning". In supervised learning, a machine learning model is trained using multiple training samples, each of which may include multiple input data values ​​and multiple expected output values, i.e., each training sample is associated with an expected output value. By specifying both the training samples and the expected output values, the machine learning model "learns" which output value to provide based on input samples similar to the samples provided during training. In addition to supervised learning, semi-supervised learning can also be used. In semi-supervised learning, some training samples lack corresponding expected output values. Supervised learning can be based on supervised learning algorithms, such as classification algorithms, regression algorithms, or similarity learning algorithms. When the output is restricted to a finite set of values ​​(categorical variables), i.e., when the input is classified as one of a finite set of values, a classification algorithm can be used. When the output can have any numerical value (within a certain range), a regression algorithm can be used. A similarity learning algorithm may be similar to both a classification algorithm and a regression algorithm, but is based on learning from examples using a similarity function that measures the degree of similarity or correlation between two objects. In addition to supervised learning and semi-supervised learning, unsupervised learning can be used to train machine learning models. In unsupervised learning, (only) input data can be provided, and an unsupervised learning algorithm can be used to discover structure in the input data (e.g., by grouping or clustering the input data to discover commonalities in the data). Clustering is the assignment of input data comprising multiple input values ​​into subsets (clusters) such that input values ​​in the same cluster are similar and different from input values ​​included in other clusters according to one or more (predefined) similarity criteria.

[0073] Reinforcement learning is the third group of machine learning algorithms. In other words, reinforcement learning can be used to train machine learning models. In reinforcement learning, one or more software actors (called "software agents") are trained to perform actions in an environment. Based on the actions performed, a reward can be calculated. Reinforcement learning is based on training one or more software agents to choose actions such that the cumulative reward increases, which in turn causes the software agents to become better at completing the tasks they are given (as evidenced by increasing rewards).

[0074] In addition, some techniques can be applied to some machine learning algorithms. For example, feature learning can be used. In other words, a machine learning model can be trained at least in part using feature learning, and / or a machine learning algorithm can include a feature learning component. Feature learning algorithms can be referred to as representation learning algorithms, which can preserve the information in their input but can also transform it in a way that makes it useful, often as a preprocessing step before performing classification or prediction. For example, feature learning can be based on principal component analysis or cluster analysis.

[0075] In some examples, anomaly detection (i.e., outlier detection) can be used, which aims to provide identification of input values ​​that are suspicious because they are significantly different from the majority of inputs or training data. In other words, a machine learning model can be trained at least in part using anomaly detection, and / or a machine learning algorithm can include an anomaly detection component.

[0076] In some examples, a machine learning algorithm may use a decision tree as a prediction model. In other words, a machine learning model may be based on a decision tree. In a decision tree, observations about an item (e.g., a set of input values) may be represented by branches of the decision tree, and output values ​​corresponding to the items may be represented by leaves of the decision tree. A decision tree may support both discrete and continuous values ​​as output values. If discrete values ​​are used, the decision tree may be represented as a classification tree, and if continuous values ​​are used, the decision tree may be represented as a regression tree.

[0077] Association rules are another technique that can be used with machine learning algorithms. In other words, a machine learning model can be based on one or more association rules. Association rules are created by identifying relationships between variables in a large amount of data. A machine learning algorithm can identify and / or utilize one or more relationship rules that represent knowledge learned from the data. Rules can be used, for example, to store, manipulate, or apply knowledge.

[0078] Machine learning algorithms are typically based on machine learning models. In other words, the term "machine learning algorithm" may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" may refer to a data structure and / or a rule set that represents learned knowledge (e.g., based on the training performed by a machine learning algorithm). In an embodiment, the use of a machine learning algorithm may imply the use of an underlying machine learning model (or multiple underlying machine learning models). The use of a machine learning model may imply a machine learning model trained by a machine learning algorithm and / or a data structure / rule set that is a machine learning model.

[0079] For example, a machine learning model can be an artificial neural network (ANN). ANN is a system inspired by biological neural networks, such as those found in the retina or the brain. ANN comprises a plurality of interconnected nodes and a plurality of connections between the nodes, the so-called edges. There are usually three types of nodes, input nodes that receive input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node can represent an artificial neuron. Each edge can transfer information from one node to another. The output of a node can be defined as a (non-linear) function of its input (e.g., the sum of its inputs). The input of a node can be used in a function based on the "weight" of the edge or node that provides the input. The weights of nodes and / or edges can be adjusted during the learning process. In other words, the training of an artificial neural network can include adjusting the weights of the nodes and / or edges of the artificial neural network, i.e., to achieve the desired output for a given input.

[0080] Alternatively, the machine learning model can be a support vector machine, a random forest model or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with a related learning algorithm, which can be used to analyze data (e.g., in classification or regression analysis). A support vector machine can be trained by providing an input with multiple training input values ​​(which belong to one of the two categories). A support vector machine can be trained to assign new input values ​​to one of the two categories. Alternatively, the machine learning model can be a Bayesian network, which is a probabilistic directed acyclic graph model. A Bayesian network can use a directed acyclic graph to represent a set of random variables and their conditional dependencies. Alternatively, the machine learning model can be based on a genetic algorithm, which is a search algorithm and heuristic technique that simulates a natural selection process.

[0081] Some embodiments relate to a microscope or optical imaging system comprising a combination of Figures 1 to 3 Alternatively, a microscope or optical imaging system may be combined with Figures 1 to 3 A portion of or connected to the system described in one or more of the figures. Figure 4A schematic diagram of a system 400 configured to perform the methods described herein is shown. System 400 includes a microscope or optical imaging system 410 and a computer system 420. Microscope or optical imaging system 410 is configured to capture images and is connected to computer system 420. Computer system 420 is configured to perform at least a portion of the methods described herein. Computer system 420 can be configured to execute a machine learning algorithm. Computer system 420 and microscope or optical imaging system 410 can be separate entities, but can also be integrated together in a common housing. Computer system 420 can be part of a central processing system of microscope or optical imaging system 410, and / or computer system 420 can be part of a subcomponent of microscope or optical imaging system 410, such as a sensor, actuator, camera or lighting unit of microscope or optical imaging system 410, etc.

[0082] The computer system 420 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in different locations, such as at a local client and / or one or more remote server farms and / or data centers). The computer system 420 may include any circuit or combination of circuits. In one embodiment, the computer system 420 may include one or more processors that may be of any type. As used herein, a processor may refer to any type of computing circuit, such as, but not limited to, a microprocessor, microcontroller, complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, graphics processor, digital signal processor (DSP), multi-core processor, field programmable gate array (FPGA), or any other type of processor or processing circuit of a microscope or optical imaging system or a component of a microscope or optical imaging system (e.g., camera). Other types of circuits that may be included in the computer system 420 may be custom circuits, application specific integrated circuits (ASICs), etc., such as one or more circuits used in wireless devices such as mobile phones, tablet computers, laptops, two-way radios, and similar electronic systems. The computer system 420 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for handling removable media, such as compact disks (CDs), flash memory cards, digital video disks (DVDs), etc. The computer system 420 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system 420.

[0083] In combination with the proposed concept or one or more examples described above or below (e.g. Figures 1 to 3 ), mentions more details and aspects of the system of microscope or optical imaging system and / or computer system. The system, microscope or optical imaging system or computer system may include one or more additional optional features, which correspond to one or more aspects of the proposed concept or one or more examples described above or below.

[0084] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0085] Although some aspects have been described in the context of equipment, it is clear that these aspects also represent the description of corresponding methods, wherein the blocks or devices correspond to method steps or the features of method steps. Similarly, the aspects described in the context of method steps also represent the description of the corresponding blocks or items or the description of the features of corresponding equipment. Some or all of the method steps can be performed by (or using) hardware devices, such as processors, microprocessors, programmable computers or electronic circuits. In some embodiments, some of one or more important method steps can be performed by such devices.

[0086] According to certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed using a non-transitory storage medium such as a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, read-only memory (ROM), programmable read-only memory (PROM) and programmable read-only memory (EPROM), electrically programmable read-only memory (EEPROM) or flash memory having electronically readable control signals stored thereon, which cooperate (or can cooperate) with a programmable computer system to perform the corresponding method. Therefore, the digital storage medium may be computer readable.

[0087] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0088] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.The program code may, for example, be stored on a machine readable carrier.

[0089] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0090] In other words, therefore, an embodiment of the inventive method is a computer program with a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0091] Therefore, another embodiment of the present invention is a storage medium (or a data carrier, or a computer-readable medium) comprising a computer program stored thereon, which, when executed by a processor, is for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium is typically tangible and / or non-transitory. Another embodiment of the present invention is a device as described herein, comprising a processor and a storage medium.

[0092] Another embodiment of the invention is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.For example, the data stream or the sequence of signals may be configured to be transmitted via a data communication connection (eg via the Internet).

[0093] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0094] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0095] Another embodiment according to the invention includes a device or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, a mobile device, a memory device, etc. For example, the device or system may include a file server for transmitting the computer program to the receiver.

[0096] In some embodiments, a programmable logic device (e.g., a field programmable logic gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, the field programmable logic gate array can collaborate with a microprocessor to perform one of the methods described herein. Typically, these methods are preferably performed by any hardware device.

[0097] Description of Reference Numerals

[0098] 100 Optical Imaging System

[0099] 110 Equipment for optical imaging systems

[0100] 112 Interface

[0101] 114 Processing Module

[0102] 120 Cameras

[0103] 130 One or more measurement modules

[0104] 140 Display module

[0105] 210 Obtaining a camera image

[0106] 220 Get one or more measurements

[0107] 230 Determine spatial position

[0108] 240 Provide camera images

[0109] 310 Organization

[0110] 320 camera images

[0111] 330 Plot of one or more measurements

[0112] 340 User Settings

[0113] 350 Artificial Intelligence

[0114] 360 composite images

[0115] 362 Health Organization

[0116] 364 Pathological tissue

[0117] 366 Health Organization

[0118] 400 System

[0119] 410 Microscope or optical imaging system

[0120] 420 Computer Systems

Claims

1. An optical imaging system (100; 410), which includes: A camera (120) for providing a camera image of a surface of an object; one or more measurement modules (130) for performing one or more measurements at one or more points on the surface of the object; a display module (140); and The processing module (114) is configured to: obtaining the camera image of the surface of the object from the camera, obtaining one or more measurement values ​​at the one or more points of the surface of the object from the one or more measurement modules (130), determining the spatial position of the one or more points of the surface relative to the camera image, grouping the one or more measurements into one or more coherent spatial regions in which similar measurements are grouped; The camera image is provided to the display module (140) with an overlay representing the measured values ​​at the one or more points on the surface of the object, the overlay representing the one or more coherent spatial regions.

2. An optical imaging system according to claim 1, wherein the object is a sample of organic tissue, wherein the processing module (114) is configured to classify the surface of the sample of organic tissue into healthy tissue and pathological tissue at the spatial positions measured at the one or more points, and group the one or more measurement values ​​into two or more coherent spatial regions, the two or more coherent regions including at least one healthy tissue spatial region and at least one pathological tissue spatial region.

3. An optical imaging system according to claim 2, wherein the processing module (114) is configured to determine one or more coherent regions within the camera image, and also group the one or more measurement values ​​into one or more coherent spatial regions based on the one or more coherent regions within the camera image.

4. An optical imaging system according to any one of claims 1 to 3, wherein the processing module (114) is configured to determine one or more intermediate values ​​of spatial positions between the one or more points on the surface based on the one or more measured values, and provide a camera image with an overlay map representing the one or more measured values ​​and the one or more intermediate values ​​within a continuous spectrum.

5. An optical imaging system according to any one of claims 1 to 3, wherein the processing module (114) is configured to identify the positions of one or more probes used to perform the one or more measurements within the camera image to determine the spatial positions of the one or more points of the surface relative to the camera image.

6. An optical imaging system according to any one of claims 1 to 3, wherein the processing module (114) is configured to use light emitted by one or more light sources attached to the one or more probes to identify the position of one or more probes used to perform the one or more measurements to determine the spatial position of the one or more points of the surface relative to the camera image.

7. An optical imaging system according to any one of claims 1 to 3, wherein at least a subset of the one or more measurements are performed using a robotic arm, and wherein the processing module (114) is configured to determine the spatial position of the one or more points of the surface relative to the camera image based on the orientation of the robotic arm when performing the subset of the one or more measurements.

8. An optical imaging system according to any one of claims 1 to 3, wherein the processing module (114) is configured to determine the distribution of the one or more points on the surface of the object, and determine one or more additional points on the surface of the object requiring one or more additional measurements on the surface of the object based on the distribution of the one or more points on the surface of the object.

9. An optical imaging system according to claim 8, wherein the processing module (114) is configured to determine one or more coherent regions within the camera image, and also determine one or more additional points on the surface of the object requiring one or more additional measurements at the surface of the object based on the one or more coherent regions within the camera image.

10. The optical imaging system of any one of claims 1 to 3, wherein the one or more measurements include at least one of: one or more optical measurements, one or more electrical measurements, one or more acoustic measurements, one or more mechanical measurements, one or more thermal measurements, one or more tissue sampling measurements, and one or more measurements based on human sensory reception.

11. An optical imaging system according to any one of claims 1 to 3, wherein the object is a sample of organic tissue, and wherein the one or more measurement values ​​include at least one of the following: one or more measurement values ​​related to blood oxygen content, one or more measurement values ​​related to tissue perfusion, and one or more measurement values ​​related to the temperature of the organic tissue.

12. A device (110) for an optical imaging system (100; 410), the device (110) comprising: An interface (112) for communicating with a camera, one or more measurement modules, and a display module (140) of the optical imaging system; as well as The processing module (114) is configured to: obtaining a camera image of a surface of an object from the camera of the optical imaging system, obtaining one or more measurement values ​​at one or more points on the surface of the object from the one or more measurement modules (130), determining the spatial position of the one or more points of the surface relative to the camera image, grouping the one or more measurements into one or more coherent spatial regions in which similar measurements are grouped, and The camera image is provided to the display module (140) with an overlay representing the measured values ​​at the one or more points of the surface of the object, the overlay representing the one or more coherent spatial regions.

13. A method for an optical imaging system (100; 410), the method comprising: (210) obtaining a camera image of a surface of an object from a camera of the optical imaging system; (220) obtaining one or more measurements at the one or more points on the surface of the object from one or more measurement modules (130) of the optical imaging system; (230) determining the spatial position of the one or more points of the surface relative to the camera image; grouping the one or more measurements into one or more coherent spatial regions, wherein similar measurements are grouped; and (240) providing a camera image with an overlay to a display module (140), the overlay representing the measurements at the one or more points on the surface of the object, the overlay representing the one or more coherent spatial regions.

14. A computer program product having a program code, which, when the program code in the computer program product is executed on a processor, performs the method according to claim 13.

Images