Method, computer program and microscope system for processing a microscope image

Abstract

This invention provides a method, computer program, and microscope system for processing microscope images. In the method, at least one microscope image (B) is provided as an input image (1) for an image processing algorithm (10). An output image (3) is created from the input image (1) by the image processing algorithm (10). The creation of the output image (3) includes adding (S3) low-frequency components (N) representing the stereoscopic nature of the image structure of the input image (1) to the input image (1), wherein the low-frequency components (N) depend at least on the high-frequency components of these image structures, and wherein the high-frequency components are defined by spatial frequencies higher than the low-frequency components (N). The corresponding computer program and microscope system are also described.

Description

Technical Field

[0001] The present invention relates to methods, computer programs and microscope systems for processing microscope images, particularly contrast images recorded using an optical microscope. Background Technology

[0002] While the objects themselves may not exhibit significant absorption differences in the incident light, they do affect the phase of the transmitted light. Various contrast methods are known for converting the phase differences generated by such samples into brightness differences in the recorded microscopic images. One established procedure involves recording differential interference contrast (DIC) images, in which a stereoscopic effect is produced, meaning that objects in a two-dimensional image appear to have depth or height. This aids users in visually evaluating or assessing microscopic images, which is one reason for the widespread use of DIC techniques. However, DIC microscopy is demanding to operate, the required components are relatively expensive, and combinations with other measurement techniques, such as fluorescence image recording, may only be feasible to a limited extent.

[0003] Differential phase contrast (DPC) has advantages over DIC techniques, as described by Tian et al. in Optics Letters, 2015, Vol. 23, No. 9, under the category of "quantitative differential contrast." A phase image can be reconstructed by processing two oriented DPC images. Only two opposing half-pupil illumination records are needed to calculate a DPC image; typically, no further component settings are required. However, a drawback of the DPC method is that, when the objective aperture is larger than the illumination observation aperture, the DPC contrast cannot measure small phase gradients in the sample due to the condenser (typically with a NA of approximately 0.55 at larger working distances). The resulting microscope image loses stereoscopic quality, making optical evaluation by the user more difficult, for example, in the case of a single unit. However, in order to present users with a typical image, such as that known from DIC methods, the present invention should be able to produce this stereoscopic image effect from the measured image. For better understanding, refer to... Figure 1 , Figure 1 Images similar to DPC and images similar to DIC are shown, which produce better spatial effects than images similar to DPC.

[0004] In principle, one could conceive of modifying the instrument; for example, by limiting the objective aperture to the illumination aperture, a scene that can also measure small phase gradients could be recreated. However, this limits the resolution. Moreover, due to the limitation of the objective aperture, the detection efficiency of fluorescence recordings with the same optical configuration is reduced.

[0005] Therefore, this disclosure focuses on a computational method for improving the quality of recorded microscope images. In a general method for processing microscope images, at least one microscope image is input as an image processing algorithm. The image processing algorithm creates an output image based on the input image. Thus, a general microscope system includes a microscope for recording microscope images and a computing device configured to execute an image processing algorithm, wherein the image processing algorithm creates an output image based on the microscope image as an input image.

[0006] For example, such a method is described in "Label-less Prediction of 3D Fluorescence Images from Transmitted Light Microscopy," Chawin Ounkomol et al., bioRxiv 289504; doi: https: / / doi.org / 10.1101 / 289504. Here, a CNN-based machine learning algorithm is used to estimate the fluorescence image as the output image, for example, from a recorded transmitted light image as the input image. Furthermore, a similar method is known from "Deep Learning for Super-Resolution in Fluorescence Microscopy," Hongda Wang et al., bioRxiv 309641; doi: https: / / doi.org / 10.1101 / 309641. Here, a machine learning algorithm is trained based on a generative adversarial network (GAN) to subsequently compute the output image, in this case, a super-resolution fluorescence image. In this case, a lower-resolution image is used as the input image, particularly a wide-field fluorescence image or a diffraction-limited confocal image. Therefore, by leveraging the learned relationships about how image structures appear in the super-resolution reference image, the aforementioned machine learning algorithm qualitatively generates higher-quality images. Thus, the machine learning algorithm is trained to supplement low-resolution images with details not present in the input image. In this process, it is impossible to rule out the possibility that image structures may appear "semi-transparent" or be invented within the output image.

[0007] However, typically, image processing algorithms are needed to evaluate recorded microscope images without increasing the risk of falsifying the representation of sample structure. Figure 1 An example application involves the DPC images mentioned above. Summary of the Invention

[0008] The object of the present invention can be considered as providing a method, computer program and microscope system for processing microscope images, which improves image performance without the risk of falsifying the image structure shown.

[0009] This objective is achieved through methods, computer programs, and a microscope system.

[0010] In the methods and microscope systems of the above type, according to the present invention, the creation of the output image includes adding low-frequency components of a stereoscopic (three-dimensional) image structure representing the input image to the input image, wherein the low-frequency components depend at least on the high-frequency components of these image structures, and wherein the high-frequency components are defined by spatial frequencies higher than the low-frequency components.

[0011] The computer program described in this invention includes commands that, when executed by a computer, prompt the user to perform the method described in this invention.

[0012] Higher-frequency image components essentially carry information about the edges and position of objects in a microscope image. Therefore, no or almost no higher-frequency image components are added to the input image to prevent this information from being falsified. Specifically, it is the low-frequency components that contribute to the stereoscopic impression and may be suppressed or completely absent in microscope images due to certain recording techniques (e.g., differential phase contrast (DPC) mentioned earlier). Therefore, since the added image components are essentially limited to low spatial frequencies, an improved spatial impression can be obtained without erroneously adding new objects or object edges.

[0013] Optional design

[0014] Advantageous variations of the embodiments of the present invention are the subject of the dependent claims and will be discussed in the following description.

[0015] Spatial frequency limit

[0016] Image processing algorithms can be designed to define a spatial frequency upper limit based on the image content or the frequency distribution of the input image. Therefore, this upper limit is not strict but a variable frequency limit, depending on the individual input images. The image processing algorithm then adds only low-frequency components below the spatial frequency upper limit to the input image. This ensures that the frequency spacing of the added low-frequency components adapts to each input image. The spatial frequency upper limit can be represented by a precise frequency value or a decrease across the entire frequency range. This decrease indicates that the amplitude of the added low-frequency components decreases as the frequency increases.

[0017] Generate low-frequency information not present in the input image

[0018] For clarity, consider the following facts: compared to conventional, known low-pass filters in image processing, new low-frequency information is generated, and this is not merely a case of amplifying low-frequency components already present in the microscope image. Therefore, the low-frequency components supplemented solely by the low-pass filter are amplifications of low-frequency components already present in the input image, calculated independently of the high-frequency components. Even if the gain factor of the existing low-frequency components were not described by a constant factor, but rather by, for example, a monotonic frequency correlation function in the low-pass filter, this would not achieve the purpose of this invention in representing the three-dimensionality of image structure.

[0019] Conversely, the image processing algorithm of this invention variant can be designed such that it adds low-frequency components to the input image, wherein the added low-frequency components no longer represent an amplification or multiplication of the low-frequency components already present in the input image. Since optical recording techniques often lack low spatial frequencies, amplification / multiplication is also ineffective. Therefore, the new added low-frequency components are not, or not only, calculated based on the available low-frequency components of the input image, but are calculated based on the image structure represented by the high-frequency components.

[0020] Context information

[0021] Furthermore, the low-frequency components to be added can be defined based on contextual information related to the microscope images. For example, contextual information can involve the microscope parameters used when recording each microscope image. Microscope parameters can provide information about which frequency range is missing from the microscope image, thus defining the low-frequency components or spatial frequency upper limits based on the missing frequency range. For example, microscope parameters can be the aperture (illumination aperture or objective aperture) used to record the microscope images.

[0022] For the following variants with machine learning algorithms, contextual information can also be stored along with the training data and taken into account when training the machine learning algorithm. If the machine learning algorithm has already learned an image-to-image mapping with contextual information for the training data, the input data of the image processing algorithm may optionally include contextual information in addition to the actual microscope images.

[0023] As a supplement or alternative to microscope parameters, contextual information can also be related to image content. For example, contextual information can represent the type and / or relative position of the imaged sample and / or sample container. Contextual information can be specified or preliminarily determined from the input image using contextual information machine learning algorithms. For instance, a segmentation mask can be output as contextual information, specifying the relative position and size of multiple samples (e.g., biological units) or multiple sample components in the input image. Such contextual information can be helpful in adding those low-frequency components that are particularly suitable for producing a stereoscopic (i.e., spatial / three-dimensional) impression.

[0024] Image processing using machine learning algorithms

[0025] Optionally, the image processing algorithm may include a machine learning algorithm trained to add low-frequency components to the input image.

[0026] Specifically, machine learning algorithms can be trained using supervised learning, where a microscope image is used as input and a target image is spatially registered to the microscope image. Local registration should be understood as recording the same sample in the microscope image and in its spatially corresponding target image. Specifically, the target image can be a differential interferometric contrast image (DIC image), or more generally, an image dependent on the DIC data, such as a further processed original DIC image or a difference image made from the DIC image and the input image. The input image or microscope image for training data can be a contrast image, particularly where phase information is converted to brightness information. In this way, DIC-like images can be computed without specific DIC hardware.

[0027] Machine learning algorithms can include neural networks, particularly neural networks with encoder-decoder structures. The decoder determines the low-frequency components provided for, and particularly added to, the input image.

[0028] Compared to known uses of machine learning algorithms for processing microscope images, the determination of low-frequency components in this invention is particularly advantageous, as it can be achieved during the training of the machine learning algorithm using a specific loss function. This loss function can also be called a cost function or reward function. During the training of the machine learning algorithm, the input image is mapped to the output image using parameters (weights), whose precise values ​​should be determined during the learning process. Target / reference images can be stored for the input image. The loss function now specifies the deviation between the output image calculated from the input image using the current (not yet fully trained) parameters and the associated specified target / reference image. Therefore, the greater the deviation between the current output image and the specified target image, the more the loss function can reward or represent a larger penalty integral. Depending on the penalty integral output by the loss function, the optimization function can then determine the gradient for changing the parameters / weights, through which the output image can be sequentially calculated from the input image. This process is repeated iteratively. Compared to conventional loss functions, this invention can include a loss function that not only considers the deviation between the output image calculated from the training data and the associated target image, but also additionally penalizes or prevents the spatial frequency components from increasing or exceeding the spatial frequency limit. If the output image corresponds very well to the associated specified target image, but differs from the associated input image in high-frequency image components, the loss function will output a high penalty value. Due to this specific loss function, machine learning algorithms can be trained to compute an output image from an input image that minimizes the difference between the output image and the associated specified target image, without supplementing high-frequency image components or only supplementing them to a small extent.

[0029] For example, the frequency-dependent expression for the loss function can be determined as follows: First, a difference image is calculated from the FFT (Fourier Transform) of the input image and the FFT of the output image. This difference image specifies the amplitudes of various frequencies that are added to the input image to produce the output image. The relative position of a pixel in the difference image corresponds to its spatial frequency, and the pixel value specifies the amplitude of that spatial frequency. The horizontal and vertical axes (which typically both pass through the center of the difference image) specify zero frequencies in the horizontal and vertical directions, respectively. The farther a pixel is from the image center, the higher its frequency. The frequency-dependent expression for the loss function can now depend on, for example, the distance of each pixel from the image center, each of these distance values ​​being weighted by the pixel value (i.e., the brightness value). Thus, all the weighted distances of pixels from the image center are summed or combined (summed) in any other way. The magnitude of the expression becomes increasingly larger as the frequency of the image components added to the input image increases. In an improvement to this embodiment, the distance to the nearest horizontal or vertical axis (whose vertical or horizontal frequency is zero) can also be used instead of the distance of the pixel from the image center. Within the scope of this invention, a specific distance in frequency space can be considered as the frequency of the low-frequency component to be added. Any other distance measure characterizing frequency may also be used.

[0030] Optionally, the aforementioned spatial frequency limit or the spatial frequency dependence of the aforementioned added spatial frequency components can be defined in the loss function based on the image content of the input images. Therefore, the spatial frequency limit of the penalty integral output by the loss function is not the same for all input images; instead, it is determined based on (or according to) the image content of each input image. Contextual information related to each input image can also be included at the parameter level of the loss function (e.g., for selecting the spatial frequency limit).

[0031] In addition to a specific loss function that penalizes high-frequency image components, specific training data can be used during training: for example, prepared target images can be used, in which, by exceeding a specified image frequency limit, it can be ensured that they are indistinguishable from the associated input images.

[0032] Image processing with verification algorithms

[0033] The validation algorithm can complement variations of the machine learning algorithm as well as the conventional design of image processing algorithms without machine learning components. The image processing algorithm generates an output image in a first working step and provides the output image to the validation algorithm for evaluation in a second working step.

[0034] The verification algorithm aims to evaluate the output image in relation to image processing artifacts, which may occur within the range of added low-frequency components.

[0035] Based on the evaluation results, the first working step can optionally be repeated to produce a new output image, where the spatial frequency limit or upper limit is reduced to a lower frequency value. Multiple sets of weights for the neural network of the machine learning algorithm can also be pre-learned, where these different sets of weights have been determined using different loss functions. Different loss functions may differ in the degree of penalty applied to higher frequency components (especially as the frequency increases or above the spatial frequency limit). If the verification algorithm has now detected image processing artifacts, a set of weights determined with a stronger penalty can be used for higher frequency components when image processing is performed again. This reduces the likelihood of image processing artifacts recurring.

[0036] If the verification algorithm subsequently identifies image processing artifacts even in the new output image, the above steps can be repeated. The output image is only used, stored, or output to the user for further processing if the evaluation result indicates that there are no image processing artifacts.

[0037] The verification algorithm can also be designed to analyze and evaluate the frequency domain of the output image relative to the frequency domain of the relevant input image. In some variations, the verification algorithm compares the output image with the input image and evaluates whether the added low-frequency components conform to a specified upper frequency limit. This upper frequency limit can be the aforementioned spatial frequency limit.

[0038] Validation algorithms can also include machine learning validation algorithms. Specifically, the latter may have been trained using training data, which includes the aforementioned microscope images as input images and relevant output images computed by image processing algorithms as target images. Here, it may also be ensured beforehand that the target images used in the training data are free of image artifacts. Machine learning validation algorithms can also be evaluated in the frequency domain in a similar manner: during training, appropriate frequency transforms, such as Fourier transforms, are used instead of the microscope images and output images.

[0039] General properties

[0040] The microscope image used as input can be a contrast image, specifically, in which the phase information of the object being examined is represented by brightness values. For example, this could be a DPC image, phase-contrast image, DIC image, Huffman-modulated contrast image, or Dodt contrast image, recorded through specific adjustments of the illumination aperture. More generally, in principle, this can be any image recorded by an optical microscope or any other microscope.

[0041] Image processing algorithms can be understood as programs that can be executed on computing devices and can compute output images based on input images. These programs can be written entirely in software, or at least partially in hardware. In principle, the input and / or output images can also be used in the frequency domain.

[0042] The frequency domain of an image can be calculated from the image, for example, using a Fourier transform, particularly a discrete Fourier transform. In the frequency domain, an image is represented by different frequency components / spatial frequencies (e.g., cosine waves or sawtooth waves) with their own amplitudes and optional phases. All of the above frequencies are spatial frequencies, not, for example, time frequencies.

[0043] Therefore, low-frequency components, or components with low spatial frequencies, are related to spatial frequencies. This can be achieved, for example, after the input image has been converted to the frequency domain, or by first converting the low-frequency components to the spatial domain and then providing them, particularly superimposing or adding them, to the input image. As a general definition, low-frequency components can be understood as frequency components having spatial frequencies lower than some frequency components appearing in the input image, for example, less than 50% of the frequency components that can be represented in the frequency distribution of the input image. Therefore, high-frequency components of the input image can be understood as frequency components having frequencies higher than the low-frequency components.

[0044] A three-dimensional representation refers to the impression of a three-dimensional (i.e., a three-dimensional object). In the calculations of DIC images and the images shown here, this impression does not necessarily correspond to the actual shape of the three-dimensional sample structure. Rather, what matters is a representation that is easy for the user to understand.

[0045] The image structure of the input image is a non-random structure, which may be caused by the sample being examined or the sample environment. For example, the image structure may represent multiple biological units within the input image. Image content refers to multiple or all of the image structures of the (input) image.

[0046] The added low-frequency components can be supplemented based on the high-frequency components of the entire image or corresponding local image regions. For example, a low-frequency component can be defined whose contribution belongs to an image region of a specific image structure (e.g., one of multiple biological units). This low-frequency component is based at least on the high-frequency components of the same image region and optionally also on the high-frequency components of other image regions or the entire input image. Exclusivity to the high-frequency components is not required; more precisely, the low-frequency components can also be determined based on the entire spectrum of the input image, which may only partially contain image frequencies higher than the low-frequency components.

[0047] In some variations of the embodiments, there are separate steps for determining the low-frequency components and for adding them to the input image. Therefore, the image processing algorithm can initially determine the low-frequency components representing the stereoscopic nature of the image structure of the input image, at least based on the high-frequency components of these image structures. The determined low-frequency components are then added to the input image to create an output image. Alternatively, these two steps can be performed in a single process. Specifically, the image processing algorithm can be a procedure for image-to-image mapping, where the parameters of the mapping may have been determined through a machine learning process. In this image-to-image mapping, it is not necessary to explicitly output the low-frequency components; instead, the output image is directly computed, where the difference between the output image and the input image (which does not need to be explicitly computed or output) can correspond to the low-frequency components. In a neural network, the low-frequency components can correspond to a layer that is not necessarily the output layer of the neural network.

[0048] The steps of the image processing algorithm described herein should not be construed as a complete list of working steps. Rather, image processing algorithms can, in principle, perform other steps as well.

[0049] In this regard, the creation of the output image involves the addition or superposition of low-frequency components onto the input image; however, further computational operations may be performed before the output image is generated. For example, contrast stretching or brightness normalization may be performed simultaneously. In particular, if the image processing algorithm is designed as a machine learning algorithm, this operation can be implemented over the same range of image-to-image mappings as a general process of superposition with the low-frequency components.

[0050] Various variations of how to train machine algorithms are described. Further inventive variations are provided by performing the corresponding training steps.

[0051] A microscope system should be understood as an instrument that includes at least a microscope and a computing device. The computing device may be physically integrated into the microscope or may be placed separately in the environment surrounding the microscope. Alternatively, the computing device may also have a distributed design and communicate with the microscope via a data link. The microscope may be an optical microscope, and particularly, or generally, any magnifying image recording device.

[0052] When the microscope system according to the invention is used as intended, optional features of the microscope system can also produce variations of the method according to the invention. Conversely, the microscope system can also be configured to perform variations of the method. Attached Figure Description

[0053] Other advantages and features of the invention will now be described with reference to the accompanying schematic diagrams.

[0054] Figure 1The microscopic image and output image that can be calculated by the present invention are illustrated schematically.

[0055] Figure 2 An exemplary embodiment of the method according to the present invention is shown;

[0056] Figure 3 Another exemplary embodiment of the method according to the present invention is shown;

[0057] Figure 4 Another exemplary embodiment of the method according to the invention is shown; and

[0058] Figure 5 An exemplary embodiment of the microscope system according to the present invention is illustrated schematically. Detailed Implementation

[0059] Various exemplary embodiments are described below with reference to the accompanying drawings. Identical and functionally equivalent components are generally identified by the same reference numerals.

[0060] Figure 1

[0061] Figure 1 Microscopic image B is shown, which is a contrast image in this example. The phase variation of transmitted light can be traced back to the sample, represented by differences in brightness.

[0062] also, Figure 1 Output image 3 is shown, which, by way of example, can be calculated from microscope image B. Output image 3 contains image information from microscope image B and differs from the latter due to its stereoscopic nature. In this example, the image content includes multiple biological units. These image structures are presented in three dimensions in output image 3, thus facilitating visual evaluation by the observer compared to the original representation in microscope image B.

[0063] In terms of stereoscopic impression, the output image 3 is similar to a DIC image. In the latter, the spatial offset between interfering beams undergoing different phase changes results in bright and dark areas at the edges of objects. This produces a three-dimensional impression that does not necessarily correspond to the actual 3D contours of the sample, but rather helps microscope users perceive and evaluate it more quickly and easily.

[0064] An exemplary embodiment of the present invention can be described with reference to the following drawings, which shows how an output image 3 can be calculated from a microscope image B. As an important aspect of the invention, the forgery of image content during the process is eliminated. In particular, it is ensured that no new image structures, such as new cells or cell details, are added due to image processing. In principle, this problem exists in known machine learning algorithms, as specified in the section relating to the prior art.

[0065] This invention utilizes the discovery that a spatial impression of the output image 3 can be achieved by supplementing the microscope image B with a superposition determined by low spatial frequencies. Conversely, higher spatial frequencies are decisively responsible for the relative position and orientation of the visible edges (i.e., unit edges) of the image structure. By not adding or adding almost no higher frequency components, the forgery or addition of image structure can be avoided.

[0066] Figure 2 Exemplary embodiments

[0067] Figure 2 An exemplary embodiment of the method according to the invention is described. This example should aid in better understanding. Conversely, in practical implementations, multiple steps can be performed in a computationally efficient manner through a single operation, or modifications may be made, as will also be explained below.

[0068] Figure 2 The example uses image processing algorithm 10 based on machine learning algorithm M. The learning process is illustrated. Training data is formed from multiple microscope images B, which are provided as input images 1 to the machine learning algorithm M in step S1. An associated target image 5 is provided for each microscope image B in the training data. The microscope image B and the associated target image 5 can be images of the same sample region recorded using different microscopy techniques. For example, the target image 5 can be a DIC image, while the microscope image B is a DPC image.

[0069] The machine learning algorithm M comprises a neural network P, in this case formed by an encoder-decoder structure. The encoder E generates a feature vector from the input microscope image B, which can theoretically have any size. This feature vector is the input to the decoder D, which, along with the input image B, outputs an image in step S2. The latter should be composed of low spatial frequencies and is therefore referred to as the low-frequency component N. In step S3, the low-frequency component N is added to the input image B, thereby producing the output image B. Once the machine learning algorithm has been trained, the output image B corresponds to... Figure 1 The output image shown is 3.

[0070] For clarity, it should be noted that this processing can be performed in the spatial domain, the frequency domain, or partly in both. In the spatial domain, the image (i.e., input image 1, output image 3, and low-frequency component N) can be represented as a 2D matrix of brightness values. The representation in the spatial domain can be converted to a representation in the frequency domain through a frequency transformation, such as a Fourier transform. In a variation of the illustrated embodiment, the frequency transformation of the microscope image B can also be provided as the input image (now represented in the frequency domain) to the machine learning algorithm. Similarly, or alternatively, the low-frequency component N calculated based on input image 1 can be output in the frequency domain and added to the associated input image 1 only after being transformed into the spatial domain.

[0071] During the learning process, the calculated output image 3 is provided to the loss function L. The loss function L calculates a measure of the correspondence between the output image 3 and the associated target image 5, both belonging to the same input image 1. This measure can also be considered as a penalty number as its size increases; the smaller the correspondence between the output image 3 and the target image 5, the smaller the difference. A conventional loss function L calculates the deviation or distance R between pixels, for example, by summing the squared deviations between locally corresponding pixels in the output image 3 and the target image 5. However, the loss function L according to an exemplary embodiment of the present invention is not only a function of this distance R. Instead, the loss function L also depends on the added low-frequency component N (in Figure 2 The symbol is indicated by f. N The frequency value f. N The higher the value, the higher the value (penalty number) of the loss function L. For example, the low-frequency component N can contain different frequencies with different amplitudes. These frequencies, weighted by their respective amplitudes, can now be summed and combined into the loss function L. Therefore, the loss function L is not only a measure of how well the calculated output image 3 corresponds to the relevant target image 5, but also a measure of whether low or high spatial frequencies were added in the calculation of the output image 3. The frequency dependence in the loss function can be represented by the parameter f0, which represents the spatial frequency limit. The added frequency components f0 below the spatial frequency limit f0... N In contrast, more penalty points are awarded to the added frequency components f above the spatial frequency limit f0. N Optionally, the penalty point can be higher, which is the added frequency component f above the spatial frequency limit f0. N More.

[0072] The spatial frequency limit f0 can be a specified constant or a variable in the loss function. For example, the variable can depend on contextual information, such as the illumination aperture of the input image.

[0073] Based on the results of the loss function L, the optimization function O calculates how the parameters / weights to be learned by the neural network P should be changed. The updated parameter values ​​are then used to calculate the updated low-frequency component N, and the above steps are repeated until the parameters that minimize the loss function L are determined.

[0074] As input, the loss function L does not necessarily need to obtain the output image 3, the low-frequency component N, and the input image 1; rather, as shown by the dashed arrow, two of these three are sufficient.

[0075] The summation in step S3 illustrates a simple example of the superposition of the low-frequency component N on the input image 1. However, other computational operations can also be used to combine the low-frequency component N and the input image 1.

[0076] In all current descriptions, the loss function L can also be replaced by a reward function that, inversely to the loss function L, should be maximized. In the reward function, the spatial frequency limit f0 and the bias R are inversely related to the frequency fN; that is, the reward function decreases as the bias R decreases, and increases when f0 is insufficient.

[0077] The neural network P can be formed in a different way compared to the encoder-decoder model.

[0078] refer to Figure 3 Further modifications are described below.

[0079] Figure 3 Exemplary embodiments

[0080] Figure 3 An exemplary embodiment of the method according to the present invention is shown, which differs from... Figure 2 The neural network P directly calculates the output image 3 from the input image 1, as shown in step S3. Therefore, in this case, the neural network P does not necessarily need to explicitly calculate or output the low-frequency component N.

[0081] An exemplary design of the neural network P includes a residual skipping structure: here, the low-frequency component N initially skips to a frequency similar to... Figure 2 The calculation is performed in a manner where the low-frequency component N and the input image 1 are both fed into subsequent layers of the neural network. Therefore, the input image 1 skips all layers of the neural network P.

[0082] In the loss function L, the implicit added low-frequency component N can be reconstructed by comparing the input image 1 and the associated output image 3.

[0083] Figure 4 Exemplary embodiments

[0084] Figure 4 The progress of an exemplary method of the present invention for processing microscope image B is shown. Figure 4 The use of the pre-trained image processing algorithm 10 is shown.

[0085] As illustrated in the previous figures, the trained machine learning algorithm M can be limited to the neural network P, where the functionality used to train the neural network P, i.e., the functionality used to define its weights, is not required.

[0086] In step S1, the microscope image B is also provided as input image 1 to the image processing algorithm 10. The image processing algorithm 10 calculates the output image 3 in step S3, as follows: Figure 1 As shown, the training process has been described above.

[0087] refer to Figure 4 There is now an optional verification process designed to ensure that no image processing artifacts are added to the input image 1. Such a security step is particularly convenient if the image processing algorithm 10 is based on a machine learning algorithm.

[0088] In step S4, the output image 3 is provided to the verification algorithm V. The latter (verification algorithm V) compares the output image 3 with the input image 1 and evaluates the frequency distribution of the differences between these images. These differences, i.e., the frequencies f of the added low-frequency components, can be considered as... N Comparison, for example, with the upper frequency limit f G The specified values ​​are compared. For comparison, values ​​from the low-frequency component f can be used. N It can be variably derived or accumulated, such as its average.

[0089] If f N Less than f G This ensures that no high-frequency components have been added, which could potentially add or remove image structures from the input image 1, or cause "halogenation" of newly added structures. Therefore, the image processing is evaluated as correct, and the output image 3 is output in step S5.

[0090] Conversely, if f N Greater than f G If the verification algorithm V determines that the output image 3 is potentially forged, it will prompt the image processing algorithm 10 to perform reprocessing. During this process, image processing parameters are modified to suppress increased high-frequency image components. For example, Figure 2 and Figure 3The machine learning algorithm M shown can be pre-trained using parameters f0 with different values ​​in multiple training iterations. Therefore, different loss functions L are used, which impart different penalty integrals at different frequencies, or the reward level of the penalty integral increases with the frequency of the added component N. Thus, multiple neural networks P are determined, differing in their basic parameters f0. If... Figure 4 The verification algorithm V in the image now reprocesses the image, changing the parameter value f0 to a smaller value f0' and selecting the corresponding associated neural network P. This reduces the likelihood of image processing artifacts reappearing in the output image 3.

[0091] The verification algorithm V can optionally also be formed using a machine learning algorithm.

[0092] Figure 5 Exemplary embodiments

[0093] Figure 5 An exemplary embodiment of a microscope system 40 according to the present invention is illustrated schematically. The microscope system 40 includes a (optical) microscope 20 by means of which at least one microscope image B is recorded. The latter (microscope image B) is provided to an image processing algorithm 10 and optionally to a verification algorithm V.

[0094] Image processing algorithm 10 and optional verification algorithm V are configured as a computer program. The above exemplary design of image processing algorithm 10 and verification algorithm V provides an exemplary embodiment of a computer program according to the present invention.

[0095] Figure 5 The illustrated microscope system 40 includes a computing device 30 configured to execute computer programs, namely image processing algorithm 10 and verification algorithm V. For example, the computing device 30 may be a server-based computer system or a (personal) computer. Here, the machine learning algorithm may, in particular, be trained using the graphics processing unit (GPU) of the computing device 30.

[0096] Through the various exemplary embodiments explained, an output image that the user finds visually easier to understand can be calculated from an input image by means of the generated stereoscopic impression, without the risk of falsifying the relevant sample structure. The described exemplary embodiments are merely illustrative and can be modified within the scope of the appended claims.

[0097] Reference Mark List

[0098] B. Microscope image

[0099] D decoder

[0100] E encoder

[0101] f Spatial frequency of the image component to be added

[0102] f0 Spatial Frequency Limit

[0103] f G Frequency limit

[0104] f N Frequency value

[0105] L loss function

[0106] M Machine Learning Algorithm

[0107] N should be added to the low-frequency components of the input image.

[0108] O Optimization function

[0109] P Neural Network

[0110] R is a term used in loss functions to specify the deviation between the output image and the target image.

[0111] S1 uses the microscope image as input to the image processing algorithm.

[0112] S2 calculates and outputs the low-frequency components that should be added to the input image.

[0113] S3 creates the output image by adding low-frequency components to the input image.

[0114] S4 provides the output image to the verification algorithm.

[0115] S5 outputs the image through the verification algorithm.

[0116] V Verification Algorithm

[0117] 1. Input Image

[0118] 3. Output Image

[0119] 5. Target Image

[0120] 10 Image Processing Algorithms

[0121] 20 microscopes

[0122] 30 Computer equipment

[0123] 40. Microscope system.

Claims

1. A method for processing microscope images, including At least one microscope image (B) is input as input image (1) into image processing algorithm (10); An output image (3) is created from the input image (1) using an image processing algorithm (10). Its features The creation of the output image (3) includes adding (S3) low-frequency components (N) representing the stereoscopic nature of the image structure of the input image (1) to the input image (1), wherein the low-frequency components (N) depend at least on the high-frequency components of these image structures, and wherein the high-frequency components are defined by spatial frequencies higher than the low-frequency components (N).

2. The method according to claim 1, in, The image processing algorithm (10) includes a machine learning algorithm (M) that is trained to add low-frequency components (N) to the input image (1).

3. The method according to claim 2, in, The machine learning algorithm (M) includes a neural network (P) with an encoder-decoder structure, whose decoder (D) determines the low-frequency components (N) added to the input image (1).

4. The method according to claim 2, in, The machine learning algorithm (M) is trained using a loss function (L) that not only considers the deviation (R) between the output image (3) calculated from the training data and the relevant target image (5), but also penalizes or prevents the increase of spatial frequency components as the spatial frequency (f) increases or exceeds the spatial frequency limit (f0).

5. The method according to claim 4, in, Based on the image content of the input image (1), the spatial frequency limit (f0) or the spatial frequency dependence of the aforementioned added spatial frequency components is defined in the loss function (L).

6. The method according to claim 2, The training of the machine learning algorithm (M) is carried out within this range, with the microscope image (B) used as the input image (1) and the differential interference contrast image registered with the microscope image (B) in space used as the target image (5).

7. The method according to claim 4, in, The image processing algorithm (10) generates the output image (3) in the first working step and provides the output image (3) to the verification algorithm (V) in the second working step. Among them, the verification algorithm (V) evaluates the output image (3) based on image processing artifacts. Based on the evaluation results, the first working step is repeated to generate a new output image (3), wherein the spatial frequency limit (f0) or spatial frequency upper limit is reduced, and then the second working step is performed for the new output image (3).

8. The method according to claim 7, in, The verification algorithm (V) compares the output image (3) with the input image (1) and evaluates whether the added low-frequency component (N) conforms to the predetermined frequency upper limit (f). G ).

9. The method according to claim 7, in, The verification algorithm (V) includes a machine learning verification algorithm, which is trained using training data, which includes an input image (1) and an associated output image (3) created by the image processing algorithm (10).

10. The method according to claim 1, in, Image processing algorithm (10) defines the upper limit of spatial frequency based on the image content or frequency distribution of the input image (1), and The image processing algorithm (10) adds only the low-frequency components (N) below the upper limit of spatial frequency to the input image (1).

11. The method according to claim 1, The added low-frequency component (N) is defined based on contextual information from the microscope image (B).

12. Microscope system, including The microscope (20) used to record microscope images (B) and A computing device (30) configured to execute an image processing algorithm (10), wherein the image processing algorithm (10) creates an output image (3) based on a microscope image (B) input as an input image (1) to the image processing algorithm (10). Its features The creation of the output image (3) involves adding (S3) low-frequency components (N) representing the stereoscopic nature of the image structure of the input image (1) to the input image (1), wherein the low-frequency components (N) depend at least on the high-frequency components of these image structures, and wherein the high-frequency components are defined by spatial frequencies higher than the low-frequency components (N).

Images