Medical observation device and method for temporally and / or spatially modulated pseudo-color patterns
By using an image processor in a medical observation device to generate and combine pseudo-color patterns of time and/or spatial modulation, the problem that visible light and fluorescent image superposition in the prior art is difficult to combine time and space information, and more efficient diagnosis is achieved and surgical effects.
Patent Information
- Application Number
- CN202110516737.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2016-05-23
- Filing Date
- 2017-05-19
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2037-05-19
AI Technical Summary
When existing medical observation equipment superimposes visible and fluorescent images, it is difficult to effectively combine temporal and spatial information, limiting the accuracy and efficiency of diagnosis and surgical procedures.
By introducing an image processor in a medical observation device, a pseudo-color pattern of time and/or spatial modulation is generated and combined with input image data and auxiliary input data, an output image data containing a pseudo-color pattern field is generated, and a pseudo-color pattern field is used to generate a pseudo-color pattern field. Visualize the data of the additional data source and represent imperceptible information in the output image.
It realizes the effective merger of information from multiple data sources, improves the accuracy and efficiency of diagnosis and surgical procedures, provides automatic identification and visualization of pseudo-color images, and enhances visibility of biological tissue structure and movement direction.
Smart Images

Figure CN113256545B_ABST
Abstract
Description
[0001] This application is a divisional application of Leica Instruments (Singapore) Pte. Ltd., with an application date of May 19, 2017, an application number of 201780032076.4, and an invention name of “Medical observation device and method for pseudo-color patterns modulated in time and / or space”. Technical Field
[0002] The present invention relates to medical observation equipment and methods. Typical medical observation equipment used to implement the present invention is a microscope or an endoscope. Background Art
[0003] It is known in the prior art to inject fluorophores into tissues observed by optical imaging devices such as microscopes or endoscopes. Fluorophores are configured to mark only specific tissues. For example, a fluorophore may be adapted to remain in the bloodstream only during the observation time so that only blood vessels are marked by the fluorophore. Another fluorophore may be provided in the form of a precursor material that reacts only with tumor cells to produce a fluorophore that marks only cancer. Therefore, fluorescence imaging can provide useful diagnostic information. However, fluorescence generally does not provide anatomical information, i.e., the appearance of tissue color. In contrast, visible light images provide anatomical information but do not provide fluorescence diagnostic information. Pseudocolor is a convenient way to present information of these two different image types (visible light and fluorescence) in a single image. It is known in the prior art to superimpose or merge visible light images and fluorescence images.
[0004] The use of such image superposition devices and methods greatly facilitates diagnosis and surgery. However, further improvements in this technology are needed to further facilitate diagnosis and surgery. Summary of the invention
[0005] It is therefore an object of the present invention to provide an apparatus and a method which further improve the existing apparatus and methods.
[0006] For a medical observation device, this object is achieved according to the present invention, wherein the medical observation device includes an input interface, an image processor and an output interface, the input interface is configured to receive a sequence set of input image data and a sequence set of auxiliary input data, the output interface is configured to output a sequence set (subsequent set) of output image data at an output data rate, the image processor is connected to the input interface so as to receive the set of input image data and auxiliary input data and is connected to the output interface so as to output the output image data, the image processor is also configured to generate a sequence set of pseudo-color image data depending on the content of at least one of the auxiliary input data and the input image data and to generate at least one synthetic pseudo-color pattern field within the pseudo-color image data depending on the content of at least one of the input image data, the auxiliary input data and the pseudo-color image data, the pseudo-color pattern field including at least one of the temporally and spatially varying patterns, and the image processor is configured to merge the pseudo-color image data with at least one portion of the input image data to generate output image data containing the pseudo-color pattern field.
[0007] The object of the present invention is also achieved by a method for displaying a medical image, the method comprising the following steps: receiving a sequence set of input image data, receiving a sequence set of auxiliary input data, generating a sequence set of pseudo-color image data depending on the content of at least one of the auxiliary input data and the input image data, generating at least one coherent pattern field within the pseudo-color image data depending on the content of at least one of the input image data, the auxiliary input data and the pseudo-color image data, the pattern field having at least one of spatial and temporal modulation, merging the pseudo-color image data with the input image data to obtain output image data, and displaying the output image data with the pseudo-color pattern field.
[0008] The use of temporally and / or spatially modulated pseudo-color patterns allows further data, particularly representing additional data sources, to be incorporated into the output image. Additionally, the use of pseudo-color patterns allows visualization of data in the output image data that would otherwise be difficult to perceive in the pseudo-color image.
[0009] Since the pattern field is both pseudo-colored and synthetic, it is immediately distinct from the input image, which is preferably used as a background in the output image data and is based on visible light image data.
[0010] In the context of the present invention, a pattern represents a discernible regularity in which elements repeat in a predictable manner. Regularity and predictability mean that the pattern does not rely on human cognitive processes. Instead, patterns can be automatically recognized and discovered using pattern recognition algorithms. Pseudo-colors are colors that do not occur naturally in that context, such as neon colors in biological tissue. Colors in the context of the present disclosure are labeled as specific combinations of wavelengths of light and are therefore independent of human cognitive processes.
[0011] In the following, other aspects of the present invention are described. Each other aspect described below has its own advantages and technical effects, and thus can be arbitrarily combined with any other aspect.
[0012] For example, the pattern field can be filled with a spatial pattern (such as hatching) or with a regular repetition in time and / or space of the template. In a pattern with spatial modulation, the template can be composed of geometric shapes such as points, tiles, symbols and / or pictures. In a pattern with temporal modulation, the pattern is composed of a regular repetition of a collection of different images over a period of time. Typical examples of such patterns are bitmap templates of hatching or simple geometric shapes (such as circles, polygons and / or waveforms).
[0013] In a pattern having both temporal and spatial modulation, different patterns having spatial modulation are repeatedly displayed. Sequential patterns in a temporally modulated pattern may be similar to one another, e.g., spatially displaced relative to one another but otherwise identical, to indicate the direction of motion. For example, sequential shadings of a temporally modulated pattern within one cycle of a rate of change may all be geometrically displaced the same amount relative to one another. This may be used to produce a temporally modulated pattern that indicates the direction of motion. A propagating wave pattern may, for example, be used to indicate the direction of fluid flow, such as blood or lymph flow.
[0014] The false color pattern preferably extends on at least one coherent or continuous part of the false color image data, thereby forming a false color pattern field. In the false color pattern field, the content of at least one of the image input data and the auxiliary input data preferably meets the same condition. For example, if the content of the auxiliary input data and / or the false color image data exceeds the upper threshold, this can represent the saturation in the data. Then, these data can be represented by the pattern field, so that they are marked as unreliable. The same method can be used in conjunction with the lower threshold, and the lower threshold can represent the data below the sensitivity threshold and therefore can be considered to be unreliable. The example of the pattern of time modulation can be a false color image or its coherent part, which is filled with false color and wherein the false color is turned on and off, or changes at least one of brightness, hue and saturation according to the rate of change.
[0015] The temporally modulated pattern may have a variation rate, wherein a cyclic repetition of the alternating pattern occurs. For clear visibility, the variation rate should be smaller than the flicker fusion rate, ie, in particular smaller than 25 Hz.
[0016] The image processor may be configured to determine the rate of change in dependence on the content of at least one of the input image data and the auxiliary input data. The rate of change may be changed automatically over time by the image processor. In particular, the change over time may depend on the content of at least one of the input image data, the auxiliary input data and the pseudo-color image data. For example, if the auxiliary data is ultrasound data, the rate of change may depend on the instantaneous flow velocity, or on the frequency and / or amplitude of the velocity changes (e.g. caused by pulses in the blood flow).
[0017] If the pseudo color pattern field is a temporal pattern, where the image processor is configured to generate a temporal pattern with a rate of change, then it is advantageous that the output data rate is greater than the rate of change. Thus, temporal changes in the pattern can be displayed smoothly. The output data rate typically corresponds to the frame rate and should therefore be higher than the flicker fusion rate in order to produce a smooth sequence of images. Conversely, the rate of change is configured to produce perceptible motion.
[0018] Pseudo-color image data may also include symbols, such as arrows, letters, and numbers.
[0019] According to another aspect of the invention, the input interface may include an input portion configured to receive at least one of ultrasound data and fluorescence image data as auxiliary input data. The ultrasound data may be a one-dimensional data field generated by a recording of an ultrasound microphone. The ultrasound data may additionally or alternatively be multi-dimensional, such as generated by an ultrasound sensor head or scanner. In particular, the ultrasound data may be two-dimensional or three-dimensional (in spatial dimensions) and contain additional data at each data point representing a spatial position, such as speed and direction of flow or movement.
[0020] The image processor may comprise a spatial alignment module adapted to spatially align the input image data and any auxiliary input image data such that spatial features in the two data are spatially consistent.
[0021] For example, if the input image data is image data received from a visible light camera and the auxiliary input data is fluorescence image data received from a fluorescence camera, where both the visible light camera and the fluorescence camera have overlapping fields of view, then the spatial alignment module corrects at least one of the images so that spatial features have the same size and orientation in both image data. This allows accurate merging of the various image data in the output image data. In this case, the auxiliary input data may alternatively or additionally include data from sensors that detect other physical quantities (preferably invisible quantities). These data can be corrected using the spatial alignment module in the same manner as the fluorescence image data mentioned above.
[0022] If the fluorescence occurs in the NIR range, the fluorescence camera may be a NIR camera and the fluorescence image data may be NIR image data.
[0023] If the auxiliary data are image data from a fluorescence camera, in particular fluorescence image data, the temporal and / or spatial pattern may depend on the saturation and / or brightness of the fluorescence image data. It is also possible to generate a pseudo-color pattern field in which two different colors or wavelengths in the fluorescence image data overlap and, as an optional further criterion, each color or wavelength exceeds a minimum intensity.
[0024] If the auxiliary input image data comprises a three-dimensional data set, the temporal and / or spatial pattern may depend on the location, in particular the depth, at which the predetermined condition for switching the pattern is detected. This allows including in the output image data data representing structures and / or physical features in a plane underlying the plane represented in the input image data, such as blood vessels or bone underlying a tumor marked by a fluorophore.
[0025] According to another aspect, the image processor may be configured to generate a sequence of pseudo-color image data according to a temporal modulation scheme. The temporal modulation scheme is preferably stored in the image processor. The modulation scheme may include at least one time-varying change between different pseudo-color image data representing different pseudo-colors, pseudo-color image data of different saturation and / or brightness, and sets of different pseudo-color image data representing a cyclically changing pattern.
[0026] The modulation scheme may further include a propagation wave pattern, wherein in the propagation wave pattern, at least one of speed, direction, and wavelength depends on the content of the auxiliary input data.
[0027] According to another aspect of the medical observation device, the output image data may include three-dimensional image data. The image processor may be configured to generate and / or process the three-dimensional output image data, wherein pseudo-color image data is generated at different depths or planes of the three-dimensional image data depending on the content of the auxiliary data. This aspect may be implemented independently of the generation of the pseudo-color pattern field.
[0028] For example, three-dimensional input image data can be generated by Z stacking using either a visible light camera or a fluorescence camera. Thus, pseudo-color image data can be used to present three-dimensional features. As explained above, the three-dimensionality of image data can also be generated by three-dimensional ultrasound images.
[0029] According to another aspect of the invention, the output image data may be two-dimensional image data, and at least one of the input image data and the auxiliary input data may include three-dimensional data. The image processor is adapted to calculate the two-dimensional output image data from the three-dimensional input data using pseudo-colors of features that are not in the plane of the two-dimensional output image data. The pseudo-color image data, in particular its pseudo-color pattern field, may be used to display data content of a portion of the three-dimensional input image and / or the three-dimensional auxiliary input data that is not in the plane presented in the two-dimensional output data.
[0030] It is advantageous to use the pseudo-color image itself in conjunction with the three-dimensional input image or the auxiliary input data, ie without using a pseudo-color pattern field.
[0031] The medical observation device may include at least one visible light camera connected to the input interface to provide input image data. The visible light camera may provide three-dimensional input image data or two-dimensional input image data. The medical observation device may also include at least one sensor device, at least one sensor device configured to sense a physical quantity of electromagnetic radiation that is not within the visible wavelength range, such as at least one fluorescence camera, at least one ultrasound sensor, at least one micro-radiation camera, at least one gamma ray camera, at least one X-ray camera and / or at least one thermal imaging camera. These devices may be connected to the input interface to provide auxiliary input data. The auxiliary input data may be one-dimensional, two-dimensional, three-dimensional, or a collection of auxiliary input data of different dimensions.
[0032] The medical observation device is preferably configured to discriminate, process and display more than one fluorophore, i.e. more than one different fluorescence spectrum. To this end, at least one of the fluorescence camera, the light source and the visible light camera may be provided with a filter arrangement for selectively blocking and / or transmitting the fluorescence spectrum. In this case, the pseudo-color image preferably comprises at least two different pseudo-colors, wherein each pseudo-color is assigned to a different fluorescence spectrum, i.e. to a different fluorophore.
[0033] The image processor and its components may be hardware devices or may be represented at least in part by software modules executing in a multi-function computer or processor.
[0034] The present invention may employ the merging of false color image data with the input image data as described in parallel application EP 16155 625.3, which is incorporated herein by reference in its entirety. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In the following, the present invention is exemplarily described with reference to the accompanying drawings. In the drawings, elements corresponding to each other with respect to at least one of function and design are given the same reference numerals.
[0036] The combination of features shown in the figures and described below is only an example. If the technical effect of the above-mentioned specific features is required or not required for a specific application, each feature can be added or omitted.
[0037] In the accompanying drawings,
[0038] Figure 1 Schematically illustrating an embodiment of a medical observation device according to the present invention;
[0039] Figure 2 A schematic diagram showing an operation method for superimposing medical images according to the present invention;
[0040] Figure 3 schematically illustrates an example where a set of input image data and auxiliary input data are merged into output image data;
[0041] Figure 4 schematically illustrates another example where input image data and auxiliary input data are merged into an output image;
[0042] Figure 5 schematically illustrating a pseudo color pattern field having spatial modulation in pseudo color image data;
[0043] Figures 6A-6C schematically showing a temporally modulated pattern;
[0044] Figures 7A-7I The temporal modulation pattern is schematically shown. DETAILED DESCRIPTION
[0045] First, refer to Figure 1 The structure of the medical observation apparatus 1 including the present invention is described.
[0046] As an example only, the medical observation device 1 is Figure 1 The medical observation device 1 may also be an endoscope.
[0047] The medical observation device 1 is used for examining an object 4, such as living tissue 6, for diagnostic or surgical purposes. In the object 4, one or more fluorophores may be present.
[0048] When excited by a light source 8 of certain wavelengths, the fluorophores emit fluorescence at different peak emission wavelengths, for example, in the NIR range. If two or more fluorophores are used, they may emit light at different peak emission wavelengths, or simply have different spectra. Different fluorophores may be used to label different types 10, 12 of living tissue 6, so that these types 10, 12 can be distinguished by the different fluorescence wavelengths they emit. The light source 8 may also be used as illumination for observing objects in the visible range.
[0049] For example, the first fluorophore 14 (in Figure 1 The second fluorophore 16 (schematically indicated by a type of hatching in FIG. 1 ) can be used to specifically mark a type 12 of living tissue 6, such as a blood vessel. Figure 1 Another type of hatching (indicated in the figure) can, for example, specifically mark living tissue 6 of another type 10, such as tumor cells.
[0050] The medical viewing device 1 may include an optical system 18, such as a zoom magnifier. In the case of an endoscope, the optical system 18 may also include optical fibers. In operation of the medical viewing device 1 and its medical image superposition device 49, the optical system 18 is directed onto at least a portion of the living tissue 6. The visible portion of the object 4 is determined by the field of view 20 of the optical system 18. Light reflected from the living tissue 6 is collected by the optical system 8 together with any fluorescence emitted from at least one fluorophore 14, 16 and directed to one or more beam splitter systems 24 including at least one beam splitter.
[0051] A portion 26 of light 22 is directed to a visible light camera 28 through at least one beam splitter system 24. Another portion 30 of light 22 may be directed to a fluorescence camera 32. Another portion 34 of light 22 may be directed to an eyepiece 36, which may be a monocular or binocular.
[0052] The eyepiece 36 may be directed onto a transmissive display 38, which is arranged between the eyepiece 36 and the at least one beam splitter system 24. The transmissive display 38 allows a portion 34 of the light 22 to pass through and superimposes the picture currently displayed on the display 38 onto the image provided by the optical system 18.
[0053] Alternatively, the display 38 may not be transmissive. In this case, the display 38 may render a real-time view of the object 4 with additional information, or display any other information requested by the operator of the medical image overlay device 49 or the medical viewing device 1. In particular, the background image that would be provided directly by the optical system 18 in the case of a transmissive display 38 may instead be provided in real time by the visible light camera 28.
[0054] The medical observation device 1 may further comprise at least one sensor 40 , for example an ultrasound or blood flow sensor, which may be in contact with the object 4 .
[0055] In addition to the at least one sensor 40, at least one further sensor device 42 may be provided. The sensor device 42 is preferably a non-optical sensing device configured to sense invisible physical quantities such as microradio, gamma, X-ray, infrared or sound data. The non-optical sensing device 42 provides two-dimensional or three-dimensional data. Examples of non-optical sensing devices 42 are microradio imaging cameras, ultrasound sensor heads, thermal imaging cameras, gamma ray cameras and X-ray cameras.
[0056] The non-optical sensing device preferably has a sensor field 43 that overlaps the field of view 20 , so that the data captured by the optical system 28 and the non-optical sensing device 42 represent different physical quantities from the same area of the object 4 .
[0057] In order to separate light in the visible range from light of fluorescent wavelengths and to avoid reflections, a filter arrangement 44 may preferably be used immediately in front of at least one of the light source 8, the beam splitter system 24, the visible light camera 28, the fluorescent camera 32, the eyepiece 36 and the display 38.
[0058] Visible light camera 28 is connected to input interface 48 of medical image overlay device 49 via data connection 46. The medical overlay device may be implemented by hardware dedicated to performing specific functions (such as an ASIC), software executed by a general purpose computing device, or a combination of both.
[0059] In operation of the medical image overlay device 49 and the medical viewing device 1, the input interface 48 receives a sequence set 50 of input image data 52. The input image data 52 is two-dimensional and is organized to contain pixels 54. Figure 1 In the embodiment of FIG. 5 , the sequence set 50 of input image data 52 corresponds to the sequence of visible light image data 53 provided by the camera 28 .
[0060] The input interface 48 is also configured to receive auxiliary input data 58, which may be a set 62 of one-dimensional auxiliary input data 60, two-dimensional auxiliary input data 64, or a set 62 of three-dimensional auxiliary input data 66 composed of several planes 67 of two-dimensional data. Of course, the auxiliary input data 58 may also include higher dimensional data, for example if a hyperspectral camera or ultrasound sensor head is used that outputs a three-dimensional picture containing additional physical data, such as blood flow velocity and direction for each pixel or voxel.
[0061] Fluorescence camera 32 may send via data connection 56 a sequence set 62 of two-dimensional auxiliary input data 64, which may specifically represent fluorescence image data 65 from fluorescence camera 32. Fluorescence image data 65 may have the same format as input image data 52, but with a different resolution.
[0062] The sensor 40 may provide one-dimensional auxiliary input data, for example, in the form of a time series of blood flow velocity 68 in the vessel 12 .
[0063] The sensing device 42 may provide a sequence set 62 of two-dimensional or three-dimensional auxiliary input data 64 , for example in a plane 67 parallel to a depth direction 70 of the object 4 , ie parallel to a viewing direction 72 of the medical viewing device 1 .
[0064] The sensing device 42 may be connected to the input interface 48 of the medical overlay device 1 via a data connection 74. Another data connection, not shown for simplicity, may exist between the ultrasound sensor 42 and the input interface 48. All data connections may comprise wired and / or wireless parts.
[0065] It should be understood that the auxiliary input data 58 may be received by the medical image overlay device 49 from any of the described devices, for example, only from the fluorescence camera 38 or only from the ultrasound sensor 40, as well as from any combination of these devices. Figure 1 , but other devices may also be used to provide auxiliary input data 58. For example, a thermal imaging camera, a micro-radiography camera, an X-ray camera, and / or a gamma-ray camera may be used instead of or in addition to an ultrasound sensor and / or a fluorescence camera.
[0066] The medical image overlay device 49 comprises an image processor 80 configured to mix the input image data 52 from the visible light camera 28 with the auxiliary input data 58 and to provide output image data 81 containing at least part of the input image data 52 and at least part of the auxiliary input data 58. The input image data 52 is used as background data, while the auxiliary input data 58 are assigned at least one pseudo color and / or pseudo color pattern before they are merged with the input image data 52. The pseudo color is selected manually or automatically by the image processor 80 from a list of colors that are not present in the image input data 58 and preferably are not present in the tissue 6, such as neon colors. Preferably, a different pseudo color and / or pseudo color pattern is assigned to each different physical quantity.
[0067] To this end, the image processor 80 includes a pseudo-color image generator 82. The pseudo-color image generator 82 assigns a pseudo-color to the auxiliary input data 58 depending on the content and / or source of the auxiliary input image data 58. Preferably, for each set 50 of image data 52, the pseudo-color image generator 82 is configured to generate in real time a corresponding set 84 of pseudo-color image data 86. The pseudo-color image data 86 is then mixed with the input image data 52.
[0068] The image processor 80, in particular the pseudo-color image generator 82, is further configured to generate at least one pseudo-color pattern 87 within the pseudo-color image data 86. The pattern 87 extends over a coherent portion of the pseudo-color image data 86 and thus forms a pseudo-color pattern field 88. The pattern 87 may include temporal modulation and / or spatial modulation of pseudo-color regions within the pseudo-color image data 65. A spatially modulated pattern has regularity in space, and a temporally modulated pattern has regularity over time. Regularity means that the pattern repeats in a predictable manner.
[0069] If the pattern 87 has a time modification, it is generated by the false color image generator 82 to have a rate of change which represents the time between successive repetitions of the pattern 87 .
[0070] The pattern 87 is generated depending on the content of the auxiliary input data 58. For example, the pseudo-color image generator 82 may generate a pattern 87 instead of a solid pseudo-color in an area of the auxiliary input data 58, in particular the fluorescence image data 65, where a threshold, such as a brightness threshold, is not met. Different patterns 87 may be generated depending on the extent to which the content is below the threshold. This is described below with reference to Figure 5 , 6A 6C and 7A to 7I are described by way of example. Figure 5 shows a pseudo color pattern 87' with spatial modulation, Figures 6A to 6C shows a time-modulated pattern 87", while Figures 7A to 7IA pattern 87'" is shown with a temporal and spatial modulation, the temporal modulation being based on a temporal modulation scheme 87".
[0071] exist Figure 5 In, a schematic representation of a set of pseudo-color image data 86 is given. The pseudo-color image data 86 includes at least one pseudo-color 89 (including two different pseudo-colors 89a and 89b in this example), and two different pseudo-color pattern fields 88 including different pseudo-color patterns 87, in which the patterns have only spatial modulation in this example. For example, a pseudo-color pattern field 88a is composed of a repeated waveform template and can be used to specify an area in the auxiliary input data 58 (for example, from an ultrasonic sensor) where an increased liquid content has been found by automatically comparing the auxiliary input data 58 with a database. Another pseudo-color pattern field 88b can be used to specify an area in the auxiliary input data 58 where the intensity is lower or higher than a threshold value. Since the pattern 87 has only spatial modulation, they do not change over time. However, the range of the pseudo-color pattern field 88 can change over time because it depends on the content of the input image data 52 and / or the auxiliary input data 58.
[0072] Figures 6A to 6C A pseudo-color pattern 87 with temporal modification is shown. In this example, the temporal modification is a simple blinking operation over time. Fig. 6A In FIG. 8 , the pseudo color pattern field 88 contains the first pseudo color 89a. Figure 6B , the same pseudo-color pattern field 88 is shown as including a second pseudo-color 89b (or no pseudo-color) at a later point in time, which has replaced the first pseudo-color 89a. Again at a later point in time, as Figure 7C As shown in FIG. 8 , the pseudo color pattern field 88 is restored to Fig. 6A . Thus, a repetitive pattern is displayed over time. The time period between successive repetitions is determined by the rate of change of the time-modulated pseudo-color pattern 87. The rate of change is determined in the pseudo-color image generator 82 depending on the content of the input image data 52 and / or the auxiliary input data 58. In order to have a clearly visible temporal modulation, the rate of change is less than the flicker fusion rate, in particular less than half the flicker fusion rate.
[0073] Figures 6A to 6C The temporal modulation shown in can, for example, be assigned to auxiliary input data, where the intensity is above a threshold, and if assigned to a static pseudo color, it is not discernible in the pseudo color image data 86, for example, because saturation has been reached. In this case, instead of displaying the second pseudo color 98 b, the pseudo color can be turned off in the pseudo color image data 86.
[0074] Can be used Figures 6A to 6CAnother example of temporal modulation shown in is an area that has otherwise been assigned two static pseudo colors. Using temporal modification, it can be clearly indicated that the pseudo color pattern field 88 comes from auxiliary input data whose content meets the criteria for assigning more than one pseudo color 89. Of course, Figures 6A to 6C The temporal modulation shown in can be extended to include more than two pseudo colors 89a, 89b in sequence.
[0075] The pseudo color pattern field 88 may include a pseudo color pattern field 88 having both temporal modulation and spatial modulation. Figures 7A to 7I , these figures depict a pattern 87 comprising at least one solid pseudo-color pattern region 87a which assumes different positions in the sequence set 84 of pseudo-color image data 86 to produce a progressive wave pattern 87b that advances in the direction of arrow 87c. Figures 7A to 7I Each of the sequences in FIG. 8 shows one of the sequence sets 84 during a change period. At the end of the change period, Fig.7I In, through Fig. 7A The pattern begins to repeat the cycle.
[0076] The pseudo-color pattern field 88 shows the regularity of its change over time and space, wherein at least one solid pseudo-color pattern area 87a regularly alternates with a transparent area or an area filled with another pseudo-color in space, and after each pass through the change rate, the pseudo-color pattern field 88 contains the same spatial pattern. It should be noted that the change rate itself can change over time depending on the content of the auxiliary input data 58.
[0077] Figures 7A to 7I The pseudo color pattern 87 shown in the figure can be used, for example, to represent auxiliary input data 58 of speed, flow rate and / or direction, such as blood flow data. The speed represented in the auxiliary input data 59 can be represented by the rate of change, and the flow rate is represented by the spatial rate of change or intensity of the pseudo color 89 in the pattern 87. For each set 52, the range of the pseudo color pattern field 88 is automatically determined from the auxiliary input data 58.
[0078] The image processor 80 may also include a spatial alignment module 90. The spatial alignment module 90 is configured to spatially correct the input image data 52 and the auxiliary input data 58 so that each set 52, 62 is spatially consistent with each other. This can be done, for example, by algorithmically correlating spatial features present in the input image data 52 and the auxiliary input image data 58. The spatial alignment module 90 is configured to correct at least one of the image input data 52 and the spatial input data 58 by morphing the image so that related structures are aligned. The morphing operation may include stretching, rotating, resolving, and / or distorting at least one of the input image data and the auxiliary input data.
[0079] For example, the fluorescence image data from the fluorescence camera 32 may be slightly rotated, distorted, and shifted relative to the visible light image data from the visible light camera 82. Additionally, the two-dimensional data 64 from the ultrasound sensor 40 may have a different scale and resolution and be distorted and rotated relative to the input image data 50 and / or the fluorescence image data 65 from the fluorescence camera. In order to properly overlay these data, the spatial alignment module 90 is configured to execute a correlation algorithm to identify common structures in the corresponding data 52, 58 and calculate their respective orientations and scales. Spatial correction is performed depending on the results of this algorithm for each synchronized set of input image data and auxiliary input data.
[0080] Additionally, the spatial alignment module 90 may include a transfer function, which has been obtained through an initial calibration process, of the device generating the auxiliary input data 58. The transfer function may be used to correct errors in the optical system, such as vignetting, distortion, and / or aberrations.
[0081] The medical image overlay apparatus 49 further comprises an output interface 91 configured to output a sequence set 92 of output image data 94. The output image data 94 is generated by a merging of the pseudo color image data 86 and the input image data 50.
[0082] An example of such merging is given in the applicant's application EP 16 15 5625.3, which is incorporated herein by reference in its entirety. Although merging is described in this reference only with respect to fluorescence image data, merging can be used for any other type of two-dimensional auxiliary input data 58, 64, 67 without any further changes.
[0083] The output image data 94 is output at an output data rate that is preferably one order of magnitude greater than the highest rate of change of the temporally modulated pattern 87 in the output image data 94. In particular, the output data rate is higher than the flicker fusion rate, in particular higher than 25 Hz. Conversely, the rate of change is lower than the flicker fusion rate so that the regular changes of the pseudo-color pattern 87 are clearly visible.
[0084] It should be understood that the recognition of pattern 87 does not depend on the cognitive process of human thinking, but due to its regularity in at least one of the spatial domain and the temporal domain, pattern 87 can be recognized by any automatic pattern recognition process.
[0085] The display 38 is connected to the output interface 91 via a data connection 96. In the display 38, the output image data 94 is displayed. If the display 38 is transmissive, then the input image data 52 can be omitted from the output image data 92, and only the pseudo-color image data 86 can be displayed. In this case, the input image data 50 is only used to align the pseudo-color image data 86 and the pattern 87 with the input image data 52. When the input image data 50 gives an accurate reproduction of what is seen through the display 38, the effect is the same as using a non-transmissive display and displaying the combined input image data 52 and the pseudo-color image 86.
[0086] exist Figure 1 A schematic example of what is seen through the eyepiece 36 is given in Detail I of FIG.
[0087] The fluorophores 14, 16 are each assigned a different pseudo color by the pseudo color image generator 82. The output image data 94 includes at least one pseudo color pattern field 88, which is generated in real time by the pseudo color image generator 82 and has a temporal and / or spatial modulation. The pattern field 88 includes a pseudo color pattern 87 and is generated depending on the content of the input image data 52 and / or the auxiliary input data 58.
[0088] For example, if in the fluorescence image data 65 of the fluorescence camera 32 there are areas within the fluorescence image data 65 with fluorescence intensities below a predetermined but preferably adjustable threshold, they can be marked with a pseudo-color pattern 87 in which the pseudo-color 89 can have at least half of its full brightness, so that this area is still visible to the observer despite the low intensity in the fluorescence image data 65. Alternatively or additionally, a different pattern 87 can be used in areas where the fluorescence image data has intensities above a predetermined but preferably adjustable threshold. This can indicate to the observer that these data are not reliable or that he has to adjust the camera sensitivity.
[0089] The medical image overlay device 49 and the medical observation device 1 may provide two-dimensional and / or three-dimensional images in the display 38 , and thus the display 38 may be a 2D or 3D display.
[0090] If auxiliary input data 58 is used that includes data from locations in the depth direction 70 that are located below the plane of the field of view 20 of the optical system 18 and are therefore not contained in the visible light image data 53 from the visible light camera 28 or in the fluorescence image data 65 from the fluorescence camera 32, then they can be added to the pseudo color image data 86 using the pseudo color pattern 87. For example, if the ultrasound sensor 40 or 42 has detected a coherent structure 100 in the object 4, such as a large blood vessel or nerve below the field of view 20, then the structure 100 can be rendered by using the pseudo color pattern field 88 in the pseudo color image data and ultimately in the output image data.
[0091] Various auxiliary input data 58 from different devices can be displayed simultaneously by using different pseudo colors and different patterns 87.
[0092] Next, refer to Figure 2 The process of generating output image data from input image data and auxiliary input data via pseudo color image data is explained. Figure 2 Optional steps are indicated by dashed lines.
[0093] Figure 2 It is shown that the input image data 52 is pre-processed in a first step 110, which may be performed by the medical image superposition apparatus 1. The pre-processing step 110 may include any one or any combination of histogram equalization, automatic contrast adjustment, compensation for aberrations, vignetting and distortion errors caused by the optical system 18, and noise reduction, but may not be limited to these.
[0094] The auxiliary input data 58, such as the fluorescence image data 65 or other and additional auxiliary input data 58, such as ultrasound data, microradiography image data, thermography image data, and / or X-ray image data, may also undergo a preprocessing step 112 to compensate for at least one error, noise, and distortion. The preprocessing algorithm performed in step 112 may generally be the same as that performed in step 110 for the input image data 52, but using different parameters to account for different physical parameters and systems. For example, compensation for errors introduced by the optical system 18 may be different for the fluorescence image data 63 than for the visible light image data 53.
[0095] In particular, for the auxiliary input data 58, the pre-processing 112 may also include a threshold comparison, wherein those auxiliary input data having content below a certain sensitivity threshold are, for example, blanked or deleted, so that they will not be included in the output image data. For example, if the intensity of a pixel is below the sensitivity threshold, then that pixel may be blanked or set to zero or set to transparent in the auxiliary input data 58. The sensitivity threshold may be determined by a user of the medical image overlay device 1 or the medical viewing device 1. Thus, pixels that only represent very low signal intensities (i.e., fluorescence levels or ultrasound reflectivity) may therefore not be considered in the pseudo-color image generation.
[0096] The next step 114 comprises spatially aligning at least one of the input image data 52 and the auxiliary input data 58, in particular the two-dimensional and / or three-dimensional auxiliary input data, so that the spatial features are located at the same position and have the same size and orientation in the input image data 52 and the auxiliary input data 58. At the end of the spatial alignment 114, the input image data and the auxiliary input data are at least substantially spatially consistent. Preferably, the input image data 52 is used for reference in the spatial alignment step 114. The auxiliary input data 58 are thus modified in the spatial alignment step 114, while the input image data 53 remain unchanged. This is particularly advantageous if the auxiliary input data 58 contains less data than the input image data, for example because pixels have been blanked and / or because the resolution measured with the pixels of the fluorescence camera 32 and / or the micro-radiography, thermal imaging or gamma-ray camera and / or the ultrasound sensor is less than the resolution of the visible light camera 28. However, keeping the input image data unchanged will maintain the field of view 20 of the optical system 18 and the visible light camera 28, so that the field of view 20 will be faithfully rendered on the display 38.
[0097] In a spatial alignment step 114, the auxiliary input data 58 of the set 62 or plane 67 may be distorted, rotated and altered in terms of its resolution so that it is spatially correlated with the input image data 52. The input image data 52 and the auxiliary input data 58 may be time stamped so that a set of input image data sets 52 is processed together with a set of auxiliary input data 58 that was sampled at the same time as the input image data.
[0098] Next, in step 116 , false color image data is generated from the auxiliary input data 58 .
[0099] For example, different fluorescent colors in the fluorescent image data 63 (which are in the NIR range and therefore invisible to humans) can be assigned different pseudo colors in the visible range. In addition, auxiliary input data 58 representing invisible physical quantities (i.e., physical quantities of electromagnetic waves that are not in the visible light range, such as ultrasound, micro-radiation, gamma rays or thermal imaging sensing) can also be assigned pseudo colors. Preferably, different pseudo colors and / or different patterns are assigned for each sensing modality.
[0100] Using many different types of auxiliary input data 58 in a single set of output image data may result in the use of too many pseudo colors. Therefore, patterns using the same pseudo color can be used to keep the number of different pseudo colors small. In addition, using patterns 87 such as hatching only obscures a portion of the underlying input image data and allows a better view of the underlying input image data.
[0101] In addition, if the content of the auxiliary input data 58 specifies low intensities of the signals recorded in the auxiliary input data 58, then they will be visualized only with a subtle pseudo-coloring. Such a subtle pseudo-coloring may not be sufficiently separated from the non-pseudo-colored areas. Therefore, these areas should not be filled with a solid pseudo-color, but with a pattern in which the pseudo-color has a high brightness (e.g., at least 50% of the maximum brightness value).
[0102] Preferably, different patterns and pseudo colors are automatically selected from a storage device 120 maintained in the medical image overlay apparatus 2. The user may be able to preset certain assignments, i.e. assigning, for example, specific pseudo colors and / or patterns to particular data sources, particular contents of data from a data source, such as particular fluorophores, pixel intensities, or patterns of changes in the data, such as rates of change.
[0103] In step 116 , a sequence set 84 of pseudo-color image data 86 is obtained, wherein at least one pseudo-color pattern field 88 is filled with a pattern 87 depending on the content of at least one of the input image data 52 and the auxiliary input data 58 .
[0104] In the next step 124, the pseudo color image and the input image are merged. It will be appreciated that this merging occurs for each data set in the sequential set of auxiliary input data 58 and input image data 52 to allow real time processing.
[0105] The merging process is described in detail in the parallel application EP 16 155 625.3, which is incorporated herein by reference. As a result of the image merging step 124, output image data 94 are obtained and finally displayed on the display 38.
[0106] exist Figure 3 In further explanation Figure 2The processing depicted in .
[0107] The image input data 52 represents visible light image data 53 from the visible light camera 28. The fluorescence image data 63 is obtained as auxiliary input data 58 from the fluorescence camera 32 and contains emission spectra of at least two fluorophores 14, 16, Figure 3 The two fluorophores are assigned different pseudo colors designated as FL800 and FL400.
[0108] Depending on the content of the fluorescent image data 63, at least one pseudo-color pattern field 88 is generated in the pseudo-color image data 86. The pseudo-color pattern field 88a specifies the area in which the two fluorophores 14, 16 overlap. The pseudo-color pattern field 88a can contain one or two pseudo-colors that have been assigned to the corresponding fluorophores 14, 16 or the corresponding emission spectra, or different pseudo-colors. Another synthetically generated pseudo-color pattern field 88c specifies an area in which one of the pseudo-colors has a very low intensity. This pseudo-color pattern field 98 can use the same pseudo-color assigned to the corresponding fluorophore or fluorescence emission spectrum.
[0109] The two-dimensional auxiliary input data 58 from the ultrasound sensor 42 is also transformed into pseudo-color image data 86. In areas where the pixel intensity in the auxiliary input data 58 from ultrasound is above a threshold, a pattern field 88 is synthetically generated. The pseudo-color image is then merged with the image input data 52 to obtain output image data 94 containing at least one pseudo-color pattern field 88.
[0110] exist Figure 4 In FIG. 9 , a set 92 of output image data 94 has been generated by merging fluorescence image data 65 containing fluorescence from tumor marker fluorophores 16 with ultrasound image data and visible light image data 53 .
[0111] At least one pseudo color pattern field 88 in the output image data 94 may have a temporal modulation as indicated by arrow 134. The area occupied by the pseudo color pattern field 88 corresponds to the area of the blood vessel automatically identified as having the blood flow velocity and direction in each sequence set. The pseudo color pattern field 88 is a wave pattern that is modified in the sequence set of the output image data 94 to produce a wave pattern 87b that propagates in the direction of arrow 134.
[0112] The speed and direction of wave propagation 34 depends on the blood flow direction and blood flow velocity contained in the auxiliary input data 58 from the ultrasound sensor 40 or 42. The wave length can be selected to depend on the distance of the blood vessel from the surface. Information about the location and extent of the blood vessels and the speed and direction of the blood flow can be extracted from the ultrasound data by algorithms executed by the image processor 80.
[0113] Reference numerals
[0114] 1 Medical observation equipment
[0115] 4 Objects
[0116] 6 Living tissue
[0117] 8 Light Source
[0118] 10. Tissue type, tumor tissue
[0119] 12 Tissue types, blood vessels
[0120] 14 First Fluorophore
[0121] 16 Second fluorophore
[0122] 18 Optical System
[0123] 20 Field of view
[0124] 22 Light collected by the optical system
[0125] 24 Beam Splitter System
[0126] 26 Branching to visible light camera
[0127] 28 Visible light camera
[0128] 30 Branches of light to the fluorescence camera
[0129] 32 Fluorescence Camera
[0130] 34 Light branching to the eyepiece or binocular eyepiece
[0131] 36 Eyepiece
[0132] 38 Display
[0133] 40 Ultrasonic sensor
[0134] 42 Non-optical sensing devices
[0135] 43 Field of view of non-optical sensing devices
[0136] 44 Filter Arrangement
[0137] 46 Data Connections
[0138] 48 Input Interface
[0139] 49 Medical image overlay equipment
[0140] 50 Input image data set
[0141] 52 Input image data
[0142] 53 Visible light image data
[0143] 54 pixels
[0144] 56 Data Connections
[0145] 58 Auxiliary input data
[0146] 60 One-dimensional auxiliary input data
[0147] 62 A collection of two-dimensional auxiliary input data
[0148] 64 2D auxiliary input data
[0149] 65 Fluorescence image data
[0150] 66 3D auxiliary input data
[0151] 67 Plane of 3D input data
[0152] 68 Blood flow speed / direction
[0153] 70 Depth direction
[0154] 72 Check direction
[0155] 74 Data Connections
[0156] 80 Image Processor
[0157] 82 Pseudo-color image generator
[0158] 81 Output image data
[0159] A collection of 84 pseudo-color image data
[0160] 86 Pseudo-color image data
[0161] 87 Pseudo-color patterns
[0162] 87' False color pattern with spatial modulation
[0163] 87” False color pattern with time modulation
[0164] 87”’ False color pattern with temporal and spatial modulation
[0165] 87”” Time Modulation Scheme
[0166] 87a Solid pseudo-color area within pseudo-color pattern
[0167] 87b Advancing Wave Pattern
[0168] 87c The way forward
[0169] 88 pseudo color pattern fields
[0170] 88a Pseudo-color pattern field indicating data overlap
[0171] 88b Indicates low intensity pseudo color pattern field
[0172] 89 Pseudo Color
[0173] 89a A pseudo color
[0174] 89b Different pseudo colors
[0175] 90 Space Image Alignment Module
[0176] 91 Output interface
[0177] 92 Output image data set
[0178] 94 Output image data
[0179] 96 Data Connections
[0180] 98 pseudo color pattern fields
[0181] 100 Large Blood Vessels
[0182] 110 Steps to preprocess input image data
[0183] 112 Steps for Preprocessing Auxiliary Input Data
[0184] 114 Space Image Alignment
[0185] 116 Pseudo-color image generation
[0186] 120 Pseudo-color and / or pseudo-color pattern storage
[0187] 124 Image Merger
[0188] 134 Arrow
Claims
1. A microscope comprising an input interface (48) and an output interface (91), The input interface (48) is configured to receive a sequence set (50) of input image data (52) and auxiliary input data (58), The output interface (91) is configured to output a sequence set (92) of output image data (94) at an output data rate, The microscope also includes an image processor (80), wherein the image processor is connected to the input interface (48) to receive a sequence set (50) of input image data (52) and auxiliary input data (58), and is connected to the output interface (91) to output a sequence set (92) of output image data (94), The image processor is further configured to generate a sequence set (84) of pseudo-color image data (86) in real time depending on the content of at least one of the auxiliary input data (58) and the input image data (52), wherein the output image data (94) is three-dimensional, and wherein the image processor (80) is configured to generate pseudo-color image data (86) at different depth layers of the three-dimensional output image data depending on the content of the auxiliary input data (58); in, The image processor (80) is configured to generate a synthesized pseudo-color pattern field (88) within the pseudo-color image data (86) depending on the content of at least one of the input image data (52), the auxiliary input data (58) and the pseudo-color image data (86), the pseudo-color pattern field (88) comprising a time-modulated pattern (87'', 87''') having a varying rate; and wherein the image processor is configured to calculate the rate of change of the temporally modulated pattern (87) depending on the content of at least one of the input image data (52) and the auxiliary input data (58); wherein the rate of change depends on the instantaneous flow velocity or on the frequency and / or amplitude of the velocity changes.
2. The microscope according to claim 1, in, The image processor (80) is configured to merge the false color image data (86) with at least a portion of the input image data (52) to generate output image data (94) including a false color pattern field (88).
3. The microscope of claim 2, wherein the false color pattern field (88) further comprises a spatially varying false color pattern (87, 87'').
4. A microscope as claimed in claim 3, wherein the sequence set (50) of output image data (94) includes a pseudo-color pattern field (88), and wherein the time-modulated pattern includes a propagating wave pattern (87b) whose speed, direction and wavelength depend on the content of the auxiliary input data (58).
5. The microscope of any one of claims 2 to 4, wherein the output image data (94) contains a pseudo-color pattern field (88) in an area where the content of the auxiliary input data (58) is at least one of below a threshold value and above a threshold value.
6. The microscope as claimed in any one of claims 2 to 4, wherein the auxiliary input data (58) comprises three-dimensional auxiliary input data (66), and wherein the image processor (80) is adapted to generate a pseudo-color pattern field (88) depending on the content of the three-dimensional auxiliary input data (66).
7. The microscope of claim 6, wherein the image processor is adapted to generate the false color pattern field (88) depending on the depth at which the predetermined condition for switching the false color pattern field (88) is detected.
8. The microscope of claim 1 , comprising a visible light camera (28) connected to the input interface (48) for providing a sequence set (50) of input image data (52), and comprising at least one of a fluorescence camera (32) and an ultrasound sensor (40, 42) connected to the input interface (48) for providing auxiliary input data (58).
9. A microscope as claimed in claim 8, comprising a fluorescence camera (32) and a light source (8), at least one of the fluorescence camera (32), the light source (8) and the visible light camera (28) being provided with a filter arrangement (44) for at least two different fluorophores (14, 16), each fluorophore (14, 16) having a different peak emission wavelength, the output image data (94) comprising at least two pseudo colors (89), and the image processor (80) being configured to assign a different pseudo color (89) to each different fluorescence peak emission wavelength.
10. The microscope of claim 8 or 9, further comprising an ultrasonic sensor (40, 42), wherein the image processor (80) is configured as in any one of claims 2 to 7 and is further configured to generate a pseudo-color pattern field (88) in the output image data (94) depending on the content of the auxiliary input data (58) from the ultrasonic sensor (40, 42).
11. The microscope according to claim 1 , comprising an optical system (18), a transmission display (38), an eyepiece (36) and at least one beam splitter system (24), the transmission display (38) being arranged between the eyepiece (36) and the at least one beam splitter system (24), the eyepiece (36) being directed onto the transmission display (38), The transmissive display (38) is configured to allow a portion (34) of the light (22) from the inspected object (4) to pass therethrough and to superimpose a picture currently displayed on the transmissive display (38) onto an image provided by an optical system (18) which, when in operation, is directed onto at least a portion of the object (4).
12. The microscope according to claim 11, include: A visible light camera (28) to which another portion (26) of the light (22) is directed via at least one beam splitter system (24).
13. A method for displaying a microscope image, The following steps are involved: receiving a sequence set (50) of input image data (52), Receive auxiliary input data (58), generating a sequence set of pseudo-color image data (86) in real time depending on the content of at least one of the auxiliary input data (58) and the input image data (52), combining each set (84) of pseudo-color image data (86) with each set of input image data (52) to obtain a set (92) of three-dimensional output image data (94), wherein the pseudo-color image data (86) is generated at different depth levels of the three-dimensional output image data depending on the content of the auxiliary input data (58), and displaying the output image data at the output data rate (94); Wherein, depending on the content of at least one of the input image data (52), the auxiliary input data (58) and the pseudo-color image data (86), a pseudo-color pattern field (88) is generated in the pseudo-color image data (86), and the pseudo-color pattern field (88) includes a time-modulated pattern (87'', 87''') having a change rate; the change rate is calculated depending on the content of at least one of the input image data (52) and the auxiliary input data (58); wherein the change rate depends on the instantaneous flow velocity or on the frequency and / or amplitude of the velocity change.
14. The method of claim 13, wherein at least one set of auxiliary input data is received from an ultrasound sensor, a micro-ray imaging camera, a thermal imaging camera, a fluorescence camera, a multi-spectral camera, and a gamma-ray camera.
15. A non-transitory computer storage medium storing a program for causing a computer to execute the method according to claim 13 or 14.
