Training denoising model for microscope

Abstract

To provide a computer-implemented method, program and data processing device for training a denoising model for a microscope.SOLUTION: A method (100) disclosed herein comprises obtaining (106) a plurality of training images with different image acquisition settings captured using a microscope (104), the plurality of training images containing noise caused by the microscope hardware. The method also comprises training (108) a denoising model (102) using the plurality of training images obtained with different image acquisition settings, thereby making the denoising model (102) specific to the microscope hardware.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention generally relates to the field of data processing in microscopy applications, and more particularly to improved techniques for removing noise from microscopy images using machine learning.

Background Art

[0002] In certain fields of microscopy, such as fluorescence microscopy for studying cell structure or biological samples, it is often necessary to make a trade-off between the exposure amount and the quality of the acquired images in order to minimize undesirable effects such as photobleaching and phototoxicity.

[0003] Typically, researchers attempt to minimize photobleaching and phototoxicity by minimizing the energy input, which can be achieved by reducing the exposure time or gain, but at the cost of an increase in the amount of noise. In experiments where many objects are evaluated (e.g., high-throughput screening in cell culture), fluorescence imaging needs to be as fast as possible and retain sufficient quality for post-analysis.

[0004] From such a background, the importance of image noise removal is increasing. Image noise removal generally refers to an attempt to restore an image contaminated with additive noise. Such noise can originate from electronic sources, such as Gaussian noise, salt-and-pepper noise, shot noise, or sample-dependent high-frequency noise due to quantization, sensitivity, or heat from a camera sensor.

[0005] The topic of denoising camera images is a subject of scientific research. For example, the paper “Fast and accurate sCMOS noise correction for fluorescence microscopy” by Mandracchia, B., Hua, X., Guo, C. et al. (Nat Commun 11, 94 (2020). https: / / doi.org / 10.1038 / s41467-019-13841-8) discloses a content-adaptive algorithm for the automatic correction of sCMOS-related noise for fluorescence microscopy. The technique disclosed therein combines camera physics and hierarchical sparse filtering to reduce noise sources within the sCMOS sensor while preserving fine details of the signal.

[0006] In the papers “Probabilistic Noise2Void: Unsupervised Content-Aware Denoising” (arXiv:1906.00651) by Alexander Krull et al. and “Noise2Void - Learning Denoising From Single Noisy Images” (IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019), methods are disclosed for training a convolutional neural network to predict the intensity distribution per pixel in order to obtain a complete probabilistic model for noisy observations and the true signal at every pixel.

[0007] In addition to these scientific approaches, conventional microscopes typically use standard noise reduction devices, either within the camera or as part of post-processing routines after image acquisition. Therefore, the effectiveness of the noise reduction techniques actually employed is limited. [Overview of the project] [Problems that the invention aims to solve]

[0008] Therefore, the underlying problem of the embodiments of the present invention is to provide an improved microscopic image noise reduction technique, thereby overcoming, at least partially, the shortcomings of the prior art described above. [Means for solving the problem]

[0009] One embodiment of the present invention provides a computer-implemented method for training a denoising model for a microscope. The method may include acquiring a plurality of training images with different image acquisition settings taken using a microscope, wherein the plurality of training images include noise caused by the microscope hardware. The method may further include training a denoising model using the plurality of training images acquired with different image acquisition settings, thereby creating a denoising model specific to the microscope hardware.

[0010] Therefore, unlike general noise reducers commonly used in conventional cameras, which are not specific to the sample being imaged or the image acquisition settings used, the above embodiments of the present invention enable the noise reduction model (also referred to herein as the “image noise reduction model”) to be tailored to the microscope hardware and further adapted to new experimental environments. In other words, conventional solutions typically rely on a noise reduction model that works for any microscope, i.e., any microscope, for any image captured by such microscope, whereas the above embodiments of the present invention generate a noise reduction model that is highly specific to particular microscope hardware.

[0011] In one embodiment, different image acquisition settings can be selected so that multiple training images have different signal-to-noise ratios. This ensures that when the model is trained using the training images, it will very accurately reflect the nature of the noise generated by the microscope hardware.

[0012] The step of acquiring multiple training images with different image acquisition settings may be performed according to a predetermined protocol, which defines different image acquisition settings. Thus, this aspect of the method ensures that a sufficient number of training images are acquired and that the set of training images covers the image acquisition settings necessary to train the denoising model sufficiently well.

[0013] A given protocol can define multiple exposure times, preferably a gradient of exposure times. Optionally, the protocol can define multiple illumination settings for each exposure time. This, in turn, provides a set of training images that particularly well cover the range of noise factors in the microscope.

[0014] Preferably, the training images are images taken using a microscope, either without a sample or with only a sample carrier (also referred to herein as “blank images,” “dark images,” or “open shutter images,” the purpose of which is to capture the optical path without the presence of a sample). Thus, the training images directly reflect any noise induced by the hardware of the microscope in question, regardless of any additional noise that may be induced by the content of the images.

[0015] In another aspect of the present invention, the denoising model can be trained to take into account each camera, objective lens, filter, optical path, and / or modality of the microscope. For example, the denoising model may include separate models for at least some of the microscope's cameras, objective lenses, filters, optical paths, and / or modalities, and / or combined models for at least some of the microscope's cameras, objective lenses, filters, optical paths, and / or modalities. This thus makes it possible to obtain a denoising model specific to the particular hardware of the microscope.

[0016] Furthermore, a computer-implemented method is provided for training a denoising model for a microscope, which may include acquiring multiple images of a sample with different image acquisition settings taken using the microscope. The multiple images of the sample are denoised using a denoising model trained using one of the methods disclosed herein, and the multiple images of the sample include noise caused by the sample. The method may further include training the denoising model or a copy thereof using the multiple images of the sample, thereby creating a denoising model or a copy thereof specific to the microscope hardware and the sample.

[0017] Therefore, in this embodiment, an attempt is made not only to learn the noise and adapt the model to the specific hardware used, but also to remove specific noise inherent to the sample in the microscope. Then, if the noise is absent, an image can be acquired with a higher signal-to-noise ratio (SNR). Using this method, a denoising model highly specialized for specific microscope hardware is generated (through initial training), and after a second training process, the embodiment produces a specialized synergistic effect for the sample in question as well.

[0018] In one embodiment, the step of acquiring multiple images of a sample with different image acquisition settings may be performed according to a predetermined protocol, which defines different image acquisition settings. The predetermined protocol may be the same protocol used to acquire the multiple training images. In this way, the predetermined protocol can be efficiently reused to train the denoising model in both phases of training.

[0019] In another embodiment, the method may include storing trained denoising models specific to the microscope hardware and sample for use with other samples, particularly similar types of samples. Thus, the trained denoising models can be used more synergistically with other, particularly similar types of samples, without the need to train the models specifically for these types of samples.

[0020] Image acquisition settings used to acquire multiple training images and / or multiple images of a sample may include at least one of gain settings, exposure time, or illumination. More generally, embodiments of the present invention support any type of image acquisition settings related to training a denoising model.

[0021] In certain implementations, the denoising model is or includes machine learning models, artificial neural networks, convolutional neural networks, particularly U-Nets and / or generative adversarial networks. However, no particular type of denoising model or algorithm is important to certain embodiments of the invention, insofar as such models or algorithms are useful in the training process disclosed herein.

[0022] A data processing device comprising means for performing any of the methods disclosed herein, as well as a computer program having program code for performing any of the methods disclosed herein when the computer program is executed on the processor, is also provided. Finally, a trained denoising model of a microscope is also provided, which is trained according to any of the methods disclosed herein.

[0023] This disclosure can be better understood by referring to the following drawings. [Brief explanation of the drawing]

[0024] [Figure 1]It is a flowchart diagram showing a method for training a hardware-specific noise removal model of a microscope according to an embodiment of the present invention. [Figure 2] It is a flowchart diagram showing a method for training a noise removal model specific to the hardware and sample of a microscope according to an embodiment of the present invention. [Figure 3] It is a diagram showing an exemplary use of a noise removal model for various types of materials according to an embodiment of the present invention. [Figure 4] It is a block diagram of a system capable of executing an embodiment of the present invention.

Mode for Carrying Out the Invention

[0025] Embodiments of the present invention provide an improved technique for training a noise removal model of a microscope. In particular, embodiments regarding a workflow that combines the microscope to be used and a noise removal algorithm to finally reduce the amount of adjustment usually required by the user will be described.

[0026] The inventors have found that, as a way to understand the problem of noise removal, an ideal picture (having a high signal-to-noise ratio (SNR)) and a mathematical distribution that adds some random values to each pixel of the acquired image are considered together. This function can be modeled, for example, in the case of fluorescence, using a Poisson-Gaussian distribution in which Poisson noise (also called shot noise) is dominant.

[0027] More specifically, the image is F = M×X + S + N This can be thought of as follows: In the equation, X is a clear image, M is multiplicative noise such as speckle, S is additive sparse noise such as fringing or illumination changes caused by different sensitivities in the camera region, which typically follows a Laplace distribution and is mainly caused by artifacts in the imaging equipment (camera, etc.), and N is additive Gaussian noise. This additive Gaussian noise is usually sample-independent if enough photons are captured by the camera, but otherwise it follows a Poisson distribution and its variance depends on the sample being imaged.

[0028] The inventors have further found that the noise can be decomposed into two components: noise related to the camera / hardware electronics, namely, noise that is affected by distortion in the lens system (objective lens, filter, etc.), and noise inherent to the sample (typically related to fluorescence or transmitted light).

[0029] Embodiments of the present invention provide a workflow that independently learns each of these noise components and combines them into a single workflow for sample acquisition in a microscope.

[0030] Figure 1 shows a flowchart of Method 100 for training a denoising model for a microscope, according to one embodiment of the present invention.

[0031] The denoising model to be trained is a machine learning model, in particular an artificial neural network such as a convolutional neural network (CNN), or includes such a model. In certain embodiments, the model may be a U-Net and / or a generative adversarial network (GAN). Further embodiments may use a general denoising model as described in the introduction above. In yet another embodiment, the denoising model may be a model disclosed in European Patent Application No. 20206032.3 entitled “Methods and systems for training convolutional neural networks”.

[0032] The denoising model may include a single machine learning model for the associated microscope 104, or a set of machine learning models. For example, a specific noise model can be associated with each camera and / or optical path. It is also possible to generate a model for each objective lens and / or filter. In yet another embodiment, a denoising model is provided for each microscope modality, e.g., confocal, multiphoton, widefield.

[0033] Referring back to method 100 in Figure 1, in step 106 (labeled “Acquire training data”), multiple training images with different image acquisition settings are acquired. In the illustrated embodiment, this includes taking images using the microscope 104, adjusting the image acquisition settings of the microscope 104, taking other images, etc., until sufficient training data is available (see the “Yes” branch in Figure 1). During this process, the microscope shutter can be opened and blank images (also referred to as “dark” images) can be acquired for training purposes, each with different acquisition settings (e.g., gain, exposure time, and / or objective lens). When dark images are taken, the background is expected to be homogeneous, so baseline data for the model can be obtained.

[0034] The image acquisition process can be performed according to a predetermined routine, protocol, or script. In one embodiment, an expert can create a configurable routine that lists the settings (e.g., automatic changes to illumination, exposure, gain, etc.) necessary to optimally train a noise model for a particular microscope, in order to provide comprehensive training data (acquisition of low- and high-quality data for validation of optimal conditions).

[0035] In another embodiment, an automated, configurable routine, protocol, or script is provided to systematically collect training images under different conditions. One embodiment may include using a gradient of exposure time and further varying the light intensity for each exposure time. In this way, it is possible to obtain different images, each with a different SNR. Sets of images obtained with different SNRs can be used as training material.

[0036] Step 106 can be performed by saving the settings at the customer site during service, for example, immediately after manufacturing, or when operating conditions change. This procedure can be applied to any type of optical microscope (e.g., bright-field, dark-field, fluorescence) inspection.

[0037] After sufficient training data has been acquired, in step 108, a denoising model is trained using the training images, resulting in a denoising model 102 that is specifically fitted to the noise of a particular microscope 104 (since microscope 104 is the microscope used to acquire the training images).

[0038] Some microscopes may have template matrices or the like to correct fixed errors, which are commonly referred to as flat-field corrections (see, for example, https: / / www.adimec.com / which-types-of-flat-field-corrections-exist-and-why-it-matters-for-high-resolution-cameras / ). However, these methods only resolve static errors. In contrast, embodiments of the present invention capture the dynamics of noise, i.e., how errors caused primarily by thermal noise and certain electronic equipment are statistically distributed when the sample is absent.

[0039] In fluorescence microscopy, using probe samples can be useful to model more complex interactions. Here, completely modeling noise is extremely difficult. Noise may depend on differences in refractive index, which, among other factors, include the concentration and type of fluorophores, the inclusion of cross-excitation and cross-emission in different channels (autofluorescence in some cases), and / or complex scattering effects. From the microscope side, there may be effects from the impulse response (point spreading function; PSF), the type of illumination beam, energy (laser intensity and / or gain), and / or exposure time. Furthermore, there may be other factors that affect the final image. Thus, once an image is generated, the noise will have a sample-dependent distribution. Therefore, one embodiment of the present invention, particularly in the case of fluorescence microscopy, includes a second model, or additional training of the model described above, which will be discussed below.

[0040] Figure 2 shows a flowchart of another method 200 for training a denoising model for a microscope, according to one embodiment of the present invention. In the illustrated embodiment, the hardware-specific denoising model 102, which was a result of method 100, serves as input to method 200 and is used as a basis for further training. Here, training is performed using images of sample 204, rather than using blank training images. These are denoised using the hardware-specific denoising model 102 in step 206. Similar to step 106 in Figure 1, in method 200 in Figure 2, multiple training images are acquired using different image acquisition settings, as shown in step 202 in Figure 2. Once a sufficient number of training images have been acquired, the denoising model 102 is trained in step 208, resulting in a denoising model 102 that is specific to both hardware and sample, i.e., takes into account both hardware-related noise of the particular microscope and noise induced by sample 204 itself.

[0041] It should be understood that methods 100 and 200 described may be performed in different combinations depending on the specific embodiment. For example, method 100 may be performed by the manufacturer as initial training of a denoising model, regardless of the type of microscope used. In the case of fluorescence microscopy described above, both methods 100 and 200 may be performed by the microscope manufacturer, as described. In both cases, method 200 may be repeated by the customer to obtain a more refined denoising model, as will be described in more detail below.

[0042] Regarding different image acquisition settings, in Method 200 as well, these settings can be modified according to a predetermined script or routine, which may be the same routine used in Method 100. This is especially true when there are some samples 204 available and the user is able to acquire a sufficient number of training images.

[0043] On the other hand, acquiring a second set of training images (containing sample 204) is generally more sensitive than acquiring a first set of training images (blank images) because taking too many images could damage or destroy sample 204. Therefore, acquiring a second set of training images may involve a user-defined routine. However, in certain embodiments, it is still possible to automate this routine at least partially by providing the user with specific guidelines, for example, "If using different exposure times, you can proceed to exposure settings..."

[0044] In one embodiment, the overall workflow for training a denoising model is as follows: -In the factory, the models are trained for each common configuration. The resulting set of models is called the "raw" model. - In the case of fluorescence, several fixed probes (e.g., beads, cells, or model organisms such as zebrafish), i.e., multiple random samples, are inserted. The resulting set of models is called a "fluo" model. In the case of neural networks, the raw model serves as the basis for training (a pre-trained model). - Upon receiving the microscope, the user can use the "Fluoro" model (in one embodiment, the user only knows there is a "Denoising" button, without being aware of the associated complexity). For other optical microscopy applications such as bright-field microscopy, the user uses the "Raw" model. -The user may determine that the current noise reduction settings are not satisfactory. For example, the user may be experimenting with some delicate stem cells and require better quality noise reduction. In one embodiment, the user uses another button, for example, labeled "Noise Reduction Training". The user determines and clicks several positions within the sample (positions where images are acquired) and general configuration (objective lens and channel corresponding to the camera) so that images are acquired with different SNRs, similar to the factory protocol. - The "Fluoro" model is used as a pre-trained model, and the new model is then adjusted on top of it (saving the user time). The user is only interested in reusing this new model.

[0045] It should be understood that other embodiments of the present invention may comprise only a subset of the steps described above. Furthermore, it should be understood that the buttons on the microscope mentioned are for illustrative purposes in the above description, and the concepts of the embodiments described can be equally realized using other types of user input mechanisms.

[0046] The following describes one embodiment of a predetermined protocol for acquiring images. Some embodiments may include using an automated illumination model, such as the one disclosed in European Patent Application Amendment 2020 / 072057, entitled “Steuerverfahren fuer ein Mikroskop”. For example, the model may enable the acquisition of an image having a specific SNR. Generally, SNR is related to the number of photons per pixel. The number of photons per pixel can be estimated using a mathematical formula after the image has been obtained.

[0047] One exemplary embodiment includes the following instruction set: 1) Prepare a microscope job by selecting a specific group of general settings. - Cameras exist simultaneously, and each camera requires its own model. - For objective lenses, especially in the case of fluorescence, a separate model is required for each objective lens (the image at 10x magnification will look different from the image at 63x magnification, the model will depend on the sample, and each objective lens will have a different SNA; considering immersion in oil / water may also be useful). - Other elements For simplicity, embodiments of the present invention have been described in relation to one model. However, this model may include multiple models. For example, if the cameras are from the same manufacturer and the differences between them are minimal, each model may be compressed into one model per camera. The total number of models could be, for example, the number of cameras multiplied by the number of objective lenses (this is commonly referred to as the "noise" configuration). 2) Prepare a group of specific settings for each job. The automated illumination model fixes the gain and other complex factors related to illumination. Therefore, the image gradient only needs to be specified with different SNRs. For example, it can be specified that 20 images are needed, ranging from "very noisy" to "noisy". If automated illumination is not used in a particular embodiment, this means fixing the exposure time gradient and increasing the light intensity and adjusting the laser gain for each exposure time. 3) Next, the images are fed into the CNN, which trains the model until it converges. If the results are not satisfactory, steps 2 and 3 can be repeated iteratively until the images in the output are satisfactory.

[0048] Figure 3 shows a process 300 according to an embodiment of the present invention in which the noise reduction model 102 is used in multiple scenarios. In particular, there are four possible paths through the flow in Figure 3, as illustrated. 1. Blank images are taken using in-house protocols and can be used to train Model A1. Model A1 can then be used to denoise any type of sample, removing some of the independent additive noise originating from electronic equipment. 2. Images taken by the probe at the factory using factory or user protocols (labeled "sample probe" in Figure 3) can be used together with the pre-trained Model A1 as a basis for training Model A2. Model A2 can then be used with fluorescent samples, and noise arising from the fluorescence is (at least partially) removed. 3. Images taken from the same sample in the lab using a user protocol can be used together with the pre-trained model A1 as the basis for training model B1. Model B1 can achieve the best quality for the corresponding sample type. 4. Images taken from the same sample in the lab using the user protocol can be used together with the pre-trained Model A2 as the basis for training Model B2. Model B2 can achieve the best quality for the corresponding fluorescent sample.

[0049] It is also possible to use a hardware-specific denoising model as a basis for continuous learning of sample-specific noise, while reusing the same training method as before and adapting it to specific experiments.

[0050] As used herein, the term "and / or" includes all possible combinations of one or more of the items listed herein and may be abbreviated as " / ".

[0051] While several embodiments have been described in the context of the apparatus, it is clear that these embodiments also represent descriptions of the corresponding methods, where blocks or apparatus correspond to steps or features of steps. Similarly, embodiments described in the context of steps also represent descriptions of the corresponding blocks, items, or features of the corresponding apparatus.

[0052] Some embodiments relate to microscopes that include systems such as those described in relation to one or more of Figures 1 to 3. Alternatively, the microscope may be part of a system such as those described in relation to one or more of Figures 1 to 3, or may be connected to a system such as those described in relation to one or more of Figures 1 to 3. Figure 4 shows a schematic diagram of a system 400 configured to carry out the methods described herein. The system 400 includes a microscope 410 and a computer system 420. The microscope 410 is configured to take images and is connected to the computer system 420. The computer system 420 is configured to carry out at least some of the methods described herein. The computer system 420 may be configured to run machine learning algorithms. The computer system 420 and the microscope 410 may be separate entities or may be integrated within a common housing. The computer system 420 may be part of the central processing system of the microscope 410, and / or the computer system 420 may be part of a dependent component of the microscope 410, such as a sensor, actor, camera, or illumination unit of the microscope 410.

[0053] The computer system 420 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) comprising one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system comprising one or more processors and one or more storage devices distributed to various locations such as local clients and / or one or more remote server farms and / or data centers). The computer system 420 may include any circuit or combination of circuits. In one embodiment, the computer system 420 may include one or more processors, which can be of any kind. As used herein, the processor may be intended to be any kind of computing circuit, such as a microprocessor for a microscope or microscopic component (e.g., a camera), a microcontroller, a composite instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multicore processor, a field-programmable gate array (FPGA), or any other kind of processor or processing circuit. Other types of circuits that may be included in the computer system 420 may be custom circuits, application-specific integrated circuits (ASICs), etc., such as one or more circuits (communication circuits, etc.) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 420 may also include one or more storage devices that may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives that handle removable media such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.The computer system 420 may also include a display device, one or more speakers and a controller which may include a keyboard and / or mouse, trackball, touchscreen, voice recognition device, or any other device which enables a user of the system to input information into and receive information from the computer system 420.

[0054] Some or all of the steps may be performed by a hardware device (or by using a hardware device), such as a processor, microprocessor, programmable computer, or electronic circuit. In some embodiments, one or more of the most critical steps may be performed by such a device.

[0055] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation is feasible using a non-transient recording medium, which is a digital recording medium, etc., that stores electronically readable control signals and cooperates (or can cooperate) with a programmable computer system to carry out each method. Examples include floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs and EPROMs, EEPROMs, or FLASH memory. Thus, the digital recording medium may be computer-readable.

[0056] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system so as to carry out any of the methods described herein.

[0057] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code, which operates to perform one of the methods when the computer program product is executed on a computer. This program code may be stored, for example, on a machine-readable carrier.

[0058] Another embodiment includes a computer program stored in a machine-readable carrier for carrying out any of the methods described herein.

[0059] Therefore, in other words, embodiments of the present invention are computer programs having program code for carrying out any of the methods described herein when the computer program is executed on a computer.

[0060] Accordingly, another embodiment of the present invention is a recording medium (or data carrier or computer-readable medium) containing a stored computer program for carrying out any of the methods described herein when executed by a processor. The data carrier, digital recording medium, or recording medium is typically tangible and / or non-transient. Another embodiment of the present invention is an apparatus, such as those described herein, comprising a processor and a recording medium.

[0061] Therefore, another embodiment of the present invention is a data stream or signal sequence representing a computer program for carrying out any of the methods described herein. The data stream or signal sequence may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0062] Another embodiment includes processing means, for example, a computer or programmable logic device configured or adapted to carry out any of the methods described herein.

[0063] Another embodiment includes a computer having an installed computer program for carrying out any of the methods described herein.

[0064] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for carrying out any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0065] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to carry out any of the methods described herein. Generally, the methods are advantageously carried out by any hardware device.

[0066] Embodiments may be based on the use of machine learning models or machine learning algorithms. Instead of relying on models and inference, machine learning may refer to algorithms and statistical models that a computer system can use to perform a particular task without using explicit instructions. For example, machine learning may use data transformations that are inferred from the analysis of historical data and / or training data, instead of rule-based data transformations. For example, image content may be analyzed using a machine learning model or a machine learning algorithm. For a machine learning model to analyze image content, the machine learning model may be trained with training images as input and training content information as output. By training a machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model “learns” to recognize image content so that image content not included in the training data becomes recognizable using the machine learning model. The same principle may be used in the same way for other types of sensor data: by training a machine learning model with training sensor data and a desired output, the machine learning model “learns” the transformation between sensor data and output, which can then be used to provide output based on non-trained sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) may be preprocessed to obtain feature vectors that can be used as input to a machine learning model.

[0067] A machine learning model may be trained using training input data. The example above uses a training method called "supervised learning." In supervised learning, a machine learning model is trained using multiple training samples, each of which may contain multiple input data values ​​and multiple desired output values; that is, each training sample is associated with a desired output value. By specifying both the training samples and the desired output values, the machine learning model "learns" during training which output values ​​to provide based on input samples similar to the provided samples. In addition to supervised learning, semi-supervised learning may be used. In semi-supervised learning, some of the training samples lack corresponding desired output values. Supervised learning may be based on a supervised learning algorithm (e.g., a classification algorithm, a regression algorithm, or a similarity learning algorithm). A classification algorithm may be used if the output is limited to a limited set of values ​​(categorical variables), i.e., the input is classified into one of a limited set of values. A regression algorithm may be used if the output may have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms, but are based on learning from examples using a similarity function that measures how similar or related two objects are. In addition to supervised or semi-supervised learning, unsupervised learning may be used to train machine learning models. In unsupervised learning, input data (only) may be provided, and unsupervised learning algorithms may be used to find structure in the input data (for example, by grouping or clustering the input data, or by finding commonalities in the data). Clustering is the process of assigning input data containing multiple input values ​​into multiple subsets (clusters), so that input values ​​within the same cluster are similar according to one or more (predefined) similarity criteria, but are not similar to input values ​​in another cluster.

[0068] Reinforcement learning is a third group of machine learning algorithms. In other words, reinforcement learning may be used to train machine learning models. In reinforcement learning, one or more software actors (referred to as “software agents”) are trained to take actions in their surroundings. A reward is calculated based on the actions taken. Reinforcement learning is based on training one or more software agents to choose actions that result in software agents that perform better on a given task, with cumulative rewards increasing (as revealed by the increase in rewards).

[0069] Furthermore, several techniques may be applied as part of a machine learning algorithm. For example, feature representation learning may be used. In other words, a machine learning model may be trained at least partially using feature representation learning, and / or a machine learning algorithm may include feature representation learning components. A feature representation learning algorithm, which may be called a representation learning algorithm, may not only store information in its own input but may also transform the information to make it useful, often as a preprocessing step before performing classification or prediction. Feature representation learning may be based, for example, on principal component analysis or cluster analysis.

[0070] In some examples, anomaly detection (i.e., outlier detection) may be used, which aims to provide the identification of input values ​​that raise suspicion by being significantly different from the majority of the input or training data. In other words, a machine learning model may be trained with anomaly detection, at least in part, and / or a machine learning algorithm may include anomaly detection components.

[0071] In some examples, a machine learning algorithm may use a decision tree as its predictive model. In other words, a machine learning model may be based on a decision tree. In a decision tree, observations about an item (e.g., a set of input values) may be represented by branches of the decision tree, and the output values ​​corresponding to these items may be represented by leaves of the decision tree. A decision tree may support both discrete and continuous values ​​as output values. When discrete values ​​are used, the decision tree may be represented as a classification tree, and when continuous values ​​are used, the decision tree may be represented as a regression tree.

[0072] Correlation rules are another technique that can be used in machine learning algorithms. In other words, a machine learning model may be based on one or more correlation rules. Correlation rules are created by identifying relationships between variables in a large amount of data. A machine learning algorithm may identify and / or utilize one or more correlational rules that represent knowledge derived from the data. These rules may be used, for example, to store, manipulate, or apply knowledge.

[0073] Machine learning algorithms are typically based on machine learning models. In other words, the term “machine learning algorithm” may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term “machine learning model” may refer to a set of data structures and / or rules that represent learned knowledge (for example, based on training performed by a machine learning algorithm). In embodiments, usage of a machine learning algorithm may mean usage of one underlying machine learning model (or multiple underlying machine learning models). Usage of a machine learning model may mean that a machine learning model and / or a set of data structures / rules that are a machine learning model are trained by a machine learning algorithm.

[0074] For example, a machine learning model may be an artificial neural network (ANN). An ANN is a system influenced by biological neural networks, such as those found in the retina or brain. An ANN consists of multiple interconnected nodes and multiple junctions between nodes, so-called edges. Typically, there are three types of nodes: input nodes that receive input values, hidden nodes that are (simply) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transmit information from one node to another. The output of a node may be defined as a (nonlinear) function of its input (e.g., the sum of its inputs). The input of a node may be used in a function based on the "weights" of the edges or nodes that provide the input. The weights of nodes and / or edges may be adjusted during the learning process. In other words, training an artificial neural network may involve adjusting the weights of the nodes and / or edges of the artificial neural network to obtain a desired output for a given input.

[0075] Alternatively, a machine learning model may be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with a relevant learning algorithm that can be used to analyze data (e.g., in classification or regression analysis). A support vector machine may be trained by providing inputs with multiple training input values ​​belonging to one of two categories. A support vector machine may be trained to assign new input values ​​to one of two categories. Alternatively, a machine learning model may be a Bayesian network, which is a stochastic directed acyclic graphical model. A Bayesian network may use a directed acyclic graph to represent a set of random variables and their conditional dependencies. Alternatively, a machine learning model may be based on a search algorithm and a genetic algorithm, which is a heuristic method that mimics the process of natural selection. [Explanation of symbols]

[0076] The process of training a hardware-specific denoising model. 102 Noise Reduction Models 104 Microscope 106 Steps to obtain training data 108 Training The process of training a noise reduction model specific to 200 samples. 202 Steps to obtain training data 204 samples 206 Noise Reduction Step 208 Training Steps 300 Process for reusing noise reduction models for similar sample types 400 System 410 Microscope 420 Computers

Claims

1. A computer-implemented method (100) for training a denoising model (102) of a microscope (104), the method comprising: acquiring (106) a plurality of training images taken with the microscope (104), each having a different image acquisition setting, the plurality of training images including noise caused by the microscope hardware; training (108) the denoising model (102) using the plurality of training images, each acquired with a different image acquisition setting, thereby creating a denoising model (102) that is specific to the microscope hardware; A method comprising:

2. the different image acquisition settings are selected such that the plurality of training images have different signal-to-noise ratios. The method of claim 1.

3. acquiring (106) the plurality of training images at different image acquisition settings is performed according to a predetermined protocol that defines the different image acquisition settings; The method of claim 1.

4. the predetermined protocol defines a plurality of exposure times, preferably a gradient of exposure times; Optionally, the predetermined protocol defines multiple illumination settings for each exposure time. The method of claim 3.

5. the plurality of training images are images taken with the microscope (104) without a sample or with only a sample carrier; The method of claim 1.

6. the denoising model (102) is trained to take into account each camera, objective, filter, optical path and / or modality of the microscope (104); The method of claim 1.

7. the denoising model (102) includes separate models for at least some of the cameras, objectives, filters, light paths and / or modalities of the microscope (104) and / or a combined model for at least some of the cameras, objectives, filters, light paths and / or modalities of the microscope (104); The method of claim 6.

8. A computer-implemented method (200) for training a denoising model (102) of a microscope (104), the method comprising: acquiring (202) a plurality of images of a sample (204) taken using the microscope (104), each having a different image acquisition setting, wherein the plurality of images of the sample (204) are denoised (206) using a denoising model (102) trained using the method of claim 1, so that the plurality of images of the sample (204) contain noise caused by the sample (204); training (208) the denoising model (102) or a copy thereof using the plurality of images of the sample (204), thereby making the denoising model (102) or a copy thereof specific to the microscope hardware and the sample (204); A method comprising:

9. acquiring (202) the plurality of images of the sample (204) at different image acquisition settings is performed according to a predetermined protocol, the predetermined protocol defining the different image acquisition settings; Optionally, the predetermined protocol is the same protocol used to acquire the plurality of training images. The method of claim 8.

10. The method comprises: storing the trained denoising model (102) specific to the microscope hardware and the sample for use with other samples, particularly samples of similar types. The method of claim 8.

11. the image acquisition settings used to acquire the plurality of training images and / or the plurality of images of the sample (204) include at least one of a gain setting, an exposure time, or an illumination; The method of claim 8.

12. the denoising model (102) is or includes a machine learning model, an artificial neural network, a convolutional neural network, in particular a U-Net and / or a generative adversarial network; The method of claim 1.

13. Data processing device comprising means for carrying out a method according to any one of claims 1 to 12.

14. A trained denoising model (102) for a microscope (104), trained according to the method of any one of claims 1 to 12.

15. 13. A computer program having a program code for performing the method according to any one of claims 1 to 12, when the computer program is run on a processor.

Images