OPTICAL CORRECTION THROUGH MACHINE LEARNING

Description

[0001] The invention relates to a method and a device for determining a neural network for correcting optical imaging errors.

[0002] An optical system can be thought of as an imaging function, also called an optical transfer function (OTF), such that a waveform leaving the optical system at the output differs from that at the input. Some of these changes are undesirable and lead to well-known optical aberrations, such as aberrations, astigmatism, vignetting, and / or coma. Optical aberrations can be related to the manufacture and operation of optical systems, such as objectives or individual lenses. These optical aberrations can impair the quality of images to such an extent that certain information in the images is not immediately apparent.

[0003] The state of the art demonstrates various methods for correcting optical aberrations. One possible way to avoid these optical aberrations is to optimize the optical systems during their manufacture. For example, coating lenses to reduce chromatic aberrations or using specific glasses or minerals with strictly defined physical properties can reduce optical aberrations. Alternatively, the use of additional optical components in certain configurations can also improve the optical properties of an optical system.

[0004] However, this prior art method has several disadvantages and problems, which are discussed below. Optical aberrations can be reduced by using higher quality materials or additional optical elements in an optical system, but they cannot be completely eliminated. For example, the correction of aberrations in lenses can decrease towards the edges. The determination of which numerical aperture and image field a lens is specified for is largely at the discretion of the developer and depends on the drop in certain quality parameters as well as the expected manufacturing tolerances. However, optical aberrations related to manufacturing tolerances cannot be eliminated by optimizing the optics. Therefore, only very small manufacturing tolerances can be tolerated for high-quality lenses.Furthermore, this method of reducing imaging errors can result in high costs (material costs or costs for additional components).

[0005] Another option for correcting optical aberrations is to further process an image captured by an image recording system using electronic data processing (EDP). Typically, aberration correction is carried out using mathematical models. The individual components of the aberration can be described using Zernike polynomials. Aberrations are broken down into components such as spherical aberrations, chromatic aberrations, astigmatism, coma, etc., which in turn are assigned to specific Zernike polynomials. If an image of a point source is captured using a lens and the point spread function (PSF) of the lens is determined, the contributions of the individual aberrations and thus the amplitudes of the Zernike polynomials can be deduced by comparing the measured point image with the ideal point image. However, in this case, an exact determination of the PSF of the lens is necessary.This is not always possible or sometimes too imprecise.

[0006] Other methods, such as vignetting correction or deconvolution, can also be used to correct optical errors. Existing vignetting correction methods typically involve taking a reference image of a homogeneous sample, which is then combined with the acquired image of the sample using a linear method. However, this method has the disadvantage of only acting on one sample plane and ignoring other errors related to vignetting.

[0007] A disadvantage of these state-of-the-art mathematical methods for correcting aberrations is that they cannot be applied, or can only be applied to a limited extent, to old data acquired with unknown lenses, as they rely on measurements of specific samples or on the measurement of the lens and its optical properties. Therefore, these state-of-the-art methods are not applicable to old data (images) acquired with unknown lenses, or they produce inaccurate results.

[0008] Hongda Wang ET AL, "Deep learning achieves super-resolution in fluorescence microscopy", bioRxiv, doi:10.1101 / 309641, 27.04.2018, discloses a deep learning-based method for achieving super-resolution in fluorescence microscopy.

[0009] HAO ZHANG ET AL, "High-throughput, high-resolution registration-free generated adversarial network microscopy", CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 07.01.2018 discloses a method in which a generative adversarial network (GAN) is combined with light microscopy to achieve deep learning super-resolution under a large field of view (FOV).

[0010] The object of the present invention is therefore to provide improved means for correcting optical imaging errors.

[0011] The present invention is defined in the independent claims and solves the problems and the object addressed by a method for determining a neural network for correcting optical aberrations. The method comprises the steps of determining one or more images, wherein the one or more images are at least partially related to an optical system or the design of an optical system, and determining a neural network as a function of the determined one or more images such that the determined neural network, applied to an image acquired by means of the optical system, outputs an image corrected with respect to one or more optical aberrations.

[0012] The device according to the invention comprises one or more processors and one or more computer-readable storage media, wherein computer-executable instructions are stored on the one or more computer-readable storage media, which, when executed by the one or more processors, cause one or more images to be captured by means of an imaging and / or image recording system, wherein one or more optical aberrations in the one or more captured images are associated with at least a part of the imaging and / or image recording system, and a neural network is applied to the one or more captured images, wherein the neural network is designed to generate one or more corresponding corrected images from the one or more captured images in such a way that the one or more optical aberrations are corrected or reduced in the one or more corrected images.

[0013] The method and device according to the invention have the advantage that neural networks, e.g., in the sense of deep learning, can be determined or applied to correct optical aberrations. This makes it possible to improve the imaging performance of existing optical systems. Furthermore, the method and device according to the invention enable costs to be saved in the manufacture of optical systems. The manufacturing costs of optical systems, such as high-performance systems, depend heavily on the measures that must be taken to prevent optical aberrations (such as the selection of materials, the application of coatings, the insertion of additional lenses and lens groups). Another cost factor in the production of optical systems is the permissible variance of the optics. The method according to the invention allows higher tolerances for the variance of optical parameters.This can reduce costs. By using neural networks to correct optical aberrations, certain manufacturing steps in the production of optical systems can be eliminated, more cost-effective materials can be used, and / or the optical design can be simplified (e.g., by eliminating lenses). This allows image recording systems or optical systems that work in combination with a neural network to correct optical aberrations to be manufactured more cost-effectively. Furthermore, neural networks enable better generalizability for previously unknown applications.

[0014] A further advantage of the method and device according to the invention is that optical systems can be adapted to expand their field of application without affecting image quality. For example, lower demands can be placed on certain parameters of optical systems, thereby increasing the degrees of freedom in other optical properties and opening up new areas of application. For example, maximum planarity and maximum working distance are mutually exclusive. If optical errors related to the planarity of an optical system can be neglected by determining them using a neural network to correct these errors, a larger working distance is possible and thus more flexibility in sample selection. Furthermore, the method according to the invention can provide a means for simultaneously correcting multiple, even non-linear, optical errors.

[0015] The method and device according to the invention can each be further improved by specific embodiments. Individual technical features of the embodiments of the invention described below can be combined with one another as desired and / or omitted, provided that the technical effect achieved by the omitted technical feature is not important.

[0016] In one embodiment, determining the neural network comprises training the neural network. Determining the one or more images may comprise determining training images, wherein training the neural network comprises training on the training images, which comprise a plurality of image pairs. Each image pair of the plurality of image pairs for training may each comprise an input image for the neural network and a target output image. This enables the correction of aberrations in old data recorded with unknown lenses and is possible because the neural network has learned to correct optical aberrations based on training data related to an optical system or its design.Therefore, it's not necessarily necessary to perform reference measurements for each existing lens in individual cases, as the neural network learns what the image data and image content should look like. Conventional methods can't do this.

[0017] In embodiments, the training images can be generated from acquired images that, for example, depict one or more sample types, wherein the acquired images were acquired using one or more optical systems and / or with the optical system, and wherein none, a subset, or all of the one or more optical systems are of the same design as the optical system. In further embodiments, which can be combined with the previous one, the training images can be generated by simulating optical aberrations. For the simulation, an optical transfer function (OTF) or a point spread function (PSF) can be determined for the optical system or for a design of optical systems. Alternatively, the OTF or PSF can also be determined independently of the simulation.Using the PSF or the OTF, which can be converted into one another, input images of the plurality of image pairs can be generated by convolving defect-free images with the PSF. This allows for the generation of nearly "perfect" training data (image pairs), since the defect-free images can be used as the corresponding target output images of the plurality of image pairs. The optical system can be part of an imaging and / or image recording system of a microscope, a microscope system, a camera, a smartphone, a telescope, a computer (which can advantageously also be portable), or a measuring device, and can comprise one or more optical components (e.g., lenses, mirrors, and / or other optics).

[0018] In one embodiment, a further (second) neural network can be used to train the neural network (hereinafter also referred to as the first neural network). The further neural network can be used as a loss function for training the first neural network. This enables improved training of the first neural network, since a neural network as a loss function enables accurate training, ensuring that the output of the neural network resembles a desired image. This cannot be guaranteed if an error is only calculated for each pixel in the output image. Thus, the output images are not treated as a set of independent pixels, but are placed in a semantic context.

[0019] In advantageous embodiments, the specific neural network can be further trained (fine-tuned). The fine-tuning (hereinafter also referred to as fine-tuning) can comprise training only a part of the specific (first) neural network. In this case, one or more parameters of the specific neural network can remain unchanged during the fine-tuning. Furthermore, the fine-tuning can comprise training specifically for the optical system. Thus, the neural network can be adapted to correct aberrations that are specific to the optical system. The fine-tuned neural network can be uniquely assigned to the optical system. The fine-tuning can be carried out using individual training data. The individual training data can be generated based on optical properties and / or measurement samples related to the optical system.The optical properties of the optical system can be determined in a single step and / or the measurement samples can be acquired using the optical system. At least one of the measurement samples, the optical properties, and the individual training data can be stored, for example, in a database, with the measurement samples, the optical properties, and / or the individual training data being uniquely assigned to the optical system.

[0020] In embodiments of the method according to the invention, the optical system can be uniquely and / or automatically identifiable. The first neural network, as well as the adapted / finely adjusted neural network, can be uniquely assigned to the optical system. For example, the optical system can be identifiable by means of electromagnetic identification, optical identification, mechanical identification, or magnetic identification.

[0021] In one embodiment that can be combined with the previous ones, the inventive method for correcting optical aberrations comprises applying the specific neural network or the finely tuned neural network to acquired data. The acquired data (e.g., images) can have been acquired using the optical system or an optical system of the same type (and a photodetector or, in scanning systems, with a point photodetector). The optical aberrations can include astigmatism, vignetting, coma, chromatic aberration, spherical aberration, or defocus.

[0022] In embodiments, determining the neural network may include training the neural network based on the one or more specific images. During training, the neural network learns what objects and / or structures in the one or more images ideally look like and corrects deviations from this. This makes it possible to correct imaging errors when the OTF is not precisely known. In this case, the neural network can automatically detect the type of error and the type of object depicted in a captured image. Neither the type of error nor the object type need to be explicitly specified. The neural network is trained to transfer objects in a captured image into a corrected image in such a way that the object is correctly reconstructed "implicitly," i.e., as part of the learned parameters, in the network. This requires that the neural network has seen similar objects during training.Similar here means that the same image features were present in the training images as in the images to be corrected, i.e. the training images are in a context with the acquired images.

[0023] In one embodiment, the device according to the invention is configured to store the one or more captured images and to apply the neural network to the one or more stored captured images. Alternatively, the neural network can be applied directly to the one or more captured images, and only the one or more corrected images are stored. Thus, the correction of the optical aberrations can take place during the acquisition of the images, i.e., before the image is saved ("in real time"), or only after it has already been saved. The at least one part of the imaging and / or image recording system can comprise an optical system, a photographic layer, an sCMOS or CCD sensor, or one or more diffusion screens.

[0024] The present invention will be described in more detail below with reference to exemplary drawings. The drawings show examples of advantageous embodiments of the invention.

[0025] They show: Figure 1 a schematic representation of a method according to the invention for determining a neural network for correcting optical imaging errors according to one embodiment, Figure 2 a schematic representation of a method according to the invention for correcting optical imaging errors by means of a neural network according to one embodiment, Figure 3 a schematic representation of a device according to the invention according to one embodiment, and Figure 4 a schematic flow diagram of an embodiment of the method according to the invention.

[0026] Figure 1shows a schematic representation of a method for determining a neural network 130. The neural network 130 for correcting optical aberrations, also referred to below as a "correction network," can be determined and / or trained in several steps 110 and 120. In addition, the determined neural network 130 can be fine-tuned in a further training step 140. The neural network 130 for correcting image aberrations can, for example, comprise an autoencoder, a U-Net, or a "Generative Adversarial Network" (GAN) and is designed to output images. The neural network 130 belongs to the class of "Convolutional Neural Networks," also called CNNs.

[0027] In a first step 110, which is optional, the training of steps 120 and / or 140 can be prepared. One possible preparation for training includes determining a loss function 136 for training in training step 120. For example, a second neural network 116 can be determined or trained and used at least partially as the basis for the loss function 136. For example, the second neural network 116 or parts of the second neural network 116, preferably the part of the second neural network 116 that extracts image parameters or image properties, can serve as the loss function 136 or target function for training the neural network 130. In this case, the loss function 136 is a loss network.

[0028] This second neural network 116, also referred to below as the "master model," can be selected from a plurality of neural networks stored in a data storage 114 and does not have to be a neural network that outputs images itself. For example, the master model 116 can be a pre-trained neural network that solves a classification problem.

[0029] Alternatively, the master model can also be found or determined using machine learning in a so-called "training" 112. The master model can then be trained using a large number of samples from different application areas (e.g., single-cell culture, three-dimensional cell culture, tissue sections, organoids, spheroids, native organs, or living organisms). So-called "supervised training" can be used for this purpose. For training, a scenario can be used in which the model, for example, solves a classification problem, i.e., does not output any images itself. The master model can be trained in a context related to the later application area of ​​the neural network 130. For example, the master model can be trained using a dataset of microscope images.

[0030] The master model can be used as a "feature extractor" to calculate so-called "activations." These activations indicate which neurons respond to specific image components (features), for example, which have recognized faces or (in the context of microscopy) cell organelles. In aggregate form, the activations of all neurons in the entire or part of the master model can be viewed as a measure of the "imageability" and thus the correctness of the corrected image and can therefore be used as a loss function.

[0031] In embodiments, the master model or parts of the master model may be used to make a prediction with the neural network 130 or to accelerate the convergence of the neural network 130 through transfer learning.

[0032] The correction network 130 can be configured to convert input images X i into output images ŷ i or to map input images X i to output images ŷ i . The loss function 136 can output a numerical measure I feat of how well the prediction ŷ i of the correction network 130 matches the target output image y 0 (an error-free image). The variables X i , ŷ i , and y 0 are vectors or matrices whose elements are assigned to the pixels of images. The correction network 130 to be trained and the loss function 136 form a system for training the correction network 130.

[0033] A loss network or loss function can be used to define one or more loss functions that measure perceptual differences in content between images, output images ŷ i , and the corresponding target output image y 0 . During training of the correction network 130, the loss network is not changed or co-trained.

[0034] In embodiments, the correction network 130 may be a neural network, for example, a residual convolutional neural network, parameterized by weights W. The correction network 130 converts input images Xi into output images ŷi via the mapping ŷi = fw(Xi). Each loss function of the one or more loss functions may calculate a scalar value representing the difference between the output image ŷi and the desired output image or the target image y0. The correction network may be trained using deep learning methods. For example, the correction network may be trained using stochastic gradient descent to minimize a weighted combination of loss functions 136. For example, the weights W are adjusted such that a feature recovery loss Ifeat is minimal.

[0035] Loss functions based on per-pixel loss are error-prone and can produce imprecise training results. To counteract these disadvantages, embodiments may use loss functions that determine perceptual and semantic differences between images. A master model 116 trained for image classification already has the ability to express features or semantic information of the input images in hidden representations or hidden features. These hidden representations can be used to make a statement about the similarity of images. Thus, a neural network 116 can define a loss function 136 in which hidden representations and thus semantic information are compared with each other.The loss network 136 may define a feature recovery loss I feat that indicates a measure of the differences in the content of the images between the target output image y 0 and the output image ŷ i of the correction network 130.

[0036] The loss network 136 can thus determine a perceptual loss. Thus, the loss network 136 ensures that the output ŷ i of the neural network 130 resembles an expected image. This is not the case with loss functions that only calculate an error per pixel of an image. Thus, the output images ŷ i are not treated as a set of independent pixels, but rather placed in a semantic context.

[0037] In embodiments, a training data set is generated. The training data set may comprise image pairs 126 (training images), wherein the image pairs 126 are characterized in that one image of an image pair is an input image X i for the neural network 130 and the other image of the image pair corresponds to a target output image y 0 of the neural network 130. The input image X i can be viewed as an error-prone image or a measured or acquired image, and the target output image y 0 can be viewed as the desired error-free or corrected image. Based on the image pairs 126 and in particular on the target output images, the neural network 130 can be trained to correct one or more of the plurality of possible aberrations. Using the image pairs 126, a neural network can be trained that corrects aberrations that are present in the input image and are missing or attenuated in the target output image.Alternatively or additionally, the training data set for training the neural network 130 may include further data. The further data may include at least one of the following: parameter data associated with the image pairs 126 or training images, validation data, measurement data related to the manufacture of an optical system, data on the procedure of an experiment or measurement, information on reagents and materials, information on an object or sample, information on an optical system, user-related data, user inputs, and information on an image acquisition system.

[0038] The training dataset can be generated using different methods or combinations of these methods. Figure 1A distinction is made between two steps 122 and 124 for producing training data, which can be combined. In step 122, the training data can be produced artificially. For example, aberrations can be artificially generated through image processing. In step 124, the training data can be produced from measured or acquired data, wherein the measured or acquired data is acquired using one or more optical systems and the optical aberrations are generated by the one or more optical systems. The produced training data can be stored in data memory 128. Some possible methods for producing training data are described in more detail below.

[0039] A first method (zoom-in strategy) for generating training data for training neural networks is based on capturing images of homogeneous samples using one or more optical systems. Images of these homogeneous samples are created at different zoom levels. For example, first images are taken with a maximum zoom of an optical system, and second images are taken with a minimum zoom of the same optical system of a homogeneous sample. When capturing a homogeneous sample with maximum or large zoom in the image center, an (almost) defect-free image can be created, while a minimal or small zoom can be created with a defect-affected image. This method takes advantage of the fact that some aberrations have less of an impact in the image center than at the edges.

[0040] A second method (shift strategy) for generating training data makes it possible to generate image pairs 126 from images of homogeneous and structured samples. As in the first method, this method also exploits the fact that certain optical aberrations have a lesser effect in the center of the image than at the edge of the image. In this second method, at least one image is acquired with any zoom, for example a medium or a high zoom. The target output image is created from the center of a captured image of the at least one captured image, and the input image is generated from the captured image at a defined position at the edge or in a corner of the captured image. The image pairs 126 for the training data set can include many different defined positions and / or different zooms.To create the training dataset, images can be taken on one or more homogeneous and / or structured samples using one or more optical systems of one or more types.

[0041] A third method (simulation strategy) for generating training data is based on the simulation of optical aberrations. If the optical transfer function (OTF), which is equal to the Fourier-transformed point spread function (PSF) of an optical system, or the PSF of an optical system is known or can be measured or simulated, image pairs 126 can be artificially generated. A defect-free image, which can serve as the target output image, can be transformed into a defect-affected image, which can serve as the input image, using mathematical methods such as convolving the defect-free image with the PSF of an optical system.

[0042] The training dataset can be created using a variety of optical systems, e.g., a variety of lenses of different designs. This can include all optical systems or types of optical systems in connection with which a neural network trained on this training data is used. For example, training data can be generated using different lenses, and a neural network can be trained using this training data. This neural network can be used to correct aberrations in measurement images acquired through one or more of the different lenses.

[0043] The training dataset can be stored in a cloud, on a data storage 128, a computer, such as a standalone computer, or a server suitable for training neural networks. The server or computer can then perform the training and save the training results (for example, on the data storage 128 or on another storage).

[0044] Neural networks can be trained using deep learning methods. This involves the systematic application of at least one deep learning method, but preferably several deep learning methods, to achieve a specific goal. The goal can include image processing (e.g., correcting one or more optical defects, generating an image from another image where at least one feature differs between the images, etc.). Deep learning methods can comprise a sequence of process steps that divide a process into comprehensible steps, in such a way that this process is repeatable. The process steps can be specific deep learning algorithms. They can also be methods with which a network learns (e.g., backpropagation), the type of data collection, the way data is processed via hardware, etc.

[0045] The training dataset generated as in one or more of the methods described above or provided by third parties can be used to train the neural network 130. In one embodiment, the goal of the training is to generate a neural network 130 that transforms input images, for example images captured with an optical system, into corrected images, wherein the transformed images correct or reduce at least one aberration caused by the optical system in the input images. This means that the image quality, such as the contrast or sharpness, in the output image (in one or more areas) has been increased compared to the input image. Ideally, this makes it possible to generate error-free images from error-prone images.

[0046] In a further phase 140, the so-called "fine-tuning," the correction network 130 can be further optimized. Optical systems are typically manufactured with certain tolerances during production. Therefore, optical systems manufactured using the same manufacturing process may exhibit deviations within the range of the manufacturing tolerances. These deviations can affect optical aberrations caused by the optical systems. Optical systems of the same design may therefore be associated with different OTFs.

[0047] To compensate for these tolerances in manufacturing, properties of optical systems can be determined in a step 142. For example, the optical properties of an individual optical system 152 can be measured. The measurement of the optical system 152 can comprise capturing one or more measurement samples. These measurement samples and / or the properties of the optical system can be stored on a data storage 144. In embodiments, determining the properties of an optical system 152 comprises determining an OTF and / or PSF for this manufactured optical system 152. The data set with the measurement samples and / or the properties of the optical system can be available over the entire life cycle of the optical system. In one embodiment, the data set with the measurement samples and / or the properties of the optical system can be stored in a cloud.

[0048] In order to be able to assign the measurement samples and / or the properties of an optical system 152, such as a lens, to the optical system 152, the data comprising the measurement samples and / or the properties of the optical system are assigned to an identifier 154 of the optical system and can be stored depending on this identifier 154 or together with an identification number corresponding to the identifier 154. The optical system 152 can receive its identifier 154 during production, wherein the identifier 152 can be unique for each optical system and permanently attached to the optical system 152. For example, the identifier 154 can be an optical coding (such asa barcode, a quick response code, a mark, or a specific color), a coding of one or more threads and / or one or more mechanical parts, a specific shape, a specific weight, a sound stripe, a steel engraving, radio frequency transponder, magnetic stripe chip card or magnetic paint.

[0049] Thus, the optical system 152 may be identified by means of electromagnetic identification, optical identification, mechanical identification, magnetic identification, or a combination thereof.

[0050] Based on the measurement samples and / or the optical properties, training data 156 can be generated. This training data can be generated like the training data, with the training data 156 being related only or specifically to the optical system 152. The training data 156 can be generated based on the measurement samples, which can include images captured by the optical system 152, or based on a simulation using the OTF or the associated PSF of the optical system 152.

[0051] Since the measurement samples and the optical properties of an optical system can be assigned to this optical system 152, the training data 156 can also be assigned to this optical system 152. Therefore, the training data 156 can be stored depending on the identifier 154. This data 156 can, as in Figure 1 indicated, depending on the identifier 154, are stored in data memory 144.

[0052] In embodiments, one or more of the data stores 114, 128, and 144 may be identical. For example, all training steps 110, 120, and 130 may be performed by a manufacturer of the optical system 152. Alternatively, individual training steps 110, 120, and 130, such as fine-tuning 130, may be performed by the user of the optical system 152.

[0053] The data set containing the measurement samples and / or the properties of the optical system 152, or the data set containing the training data 156 associated with an optical system, can be accessed using the identifier 154 of the optical system 152 or the identification number corresponding to the identifier 154. In embodiments, these data sets can be stored on a manufacturer's server or in a cloud. For example, a user can access these data sets using an authorization credential, such as a user name and password or the identification number of the optical system 152.

[0054] In training step 140, the neural network 130 from training step 120 can be fine-tuned. The following describes the fine-tuning using neural network 130 as an example. Alternatively, however, neural networks from other sources, such as a user of a neural network, can also be further fine-tuned.

[0055] Fine-tuning refers to adapting a previously trained neural network, which can already recognize essential image features, to a new, unknown data set. A neural network, such as neural network 130, can be pre-trained to correct certain image aberrations. The largest possible data set can be used for this purpose. For a specific optics or optical system, such as a specific lens, the prediction accuracy, i.e., the image quality of the output images, could then be improved by fine-tuning by creating a (smaller) training data set 156 with this lens and "fine-tuning" the pre-trained network 130. This type of fine-tuning can take place at the manufacturer of the optical system. Alternatively, the fine-tuning can also take place at the customer's or a third party's site.Fine-tuning can be performed with respect to a specific optical system or one or more specific samples. These one or more specific samples may, for example, have been missing or underrepresented in the original training dataset. Thus, fine-tuning can improve the prediction accuracy of a neural network. During fine-tuning, a pre-trained neural network can be trained on new data at a learning rate 1-2 orders of magnitude lower. "Learning rate" in this sense is a scaling factor for numerical optimization that determines the step size for changing the learned model parameters.

[0056] If this is large, the model can converge in fewer steps, but there is also a risk that the model parameters will move away from the optimum again. When fine-tuning, one can assume that the pre-trained parameters are already close to the optimum. Therefore, the step size or learning rate can be reduced compared to the training in step 120 in order to find the global minimum of the loss function. This can prevent the existing "knowledge" of the model, represented by the model parameters, from being destroyed by excessively large steps.

[0057] The training data set 156 can be used for fine-tuning the neural network 130. During fine-tuning, an existing neural network, e.g., the neural network 130, is used as the basis for further training. Further training involves training only a portion of the neural network. Some parameters of the neural network are fixed or unchangeable, while the remaining parameters can be influenced or changed through further training. This enables rapid training. Further training can be performed with different learning rates depending on the position of the parameters in the neural network.

[0058] For further training on the training data 156, various deep learning methods can be used. In embodiments, the neural network 150 is trained using a second neural network as a loss function. This can be done as in the training step 120 in Figure 1happen.

[0059] Further training creates an adapted neural network 150 that has been specifically trained for the optical system 152. Thus, an individual adaptation of the neural network 150 to the specific optical system 152 (such as a lens) can be achieved, while the fundamental properties of the correction network 130, which can correct certain optical errors, remain intact.

[0060] The adapted neural network 150 can be stored in a data storage device, a server, or a cloud in conjunction with the identifier 154 of the optical system 152 or together with an identification number corresponding to the identifier 154. In embodiments, the adapted neural network 150 is implemented and used on a server or a cloud. The server or the cloud is thus configured to correct optical aberrations in images with the aid of the adapted neural network 150. For example, defective images can be uploaded to the cloud or to the server. Error-free or error-reduced images can then be generated and made available using the adapted neural network 150. The adapted neural network 150 can be accessed using the identifier 154 or the corresponding identification number of the optical system 152.For example, a user may access the adapted neural network 150 using a credential, such as a username and password. The user's credential may be obtained by purchasing or receiving the optical system. Alternatively, the lens or optical system 152 may be part of an image recording system, and the neural network 150 may be implemented on the image recording system.

[0061] Thus, differences in manufacturing tolerances occurring during the manufacture of optical systems can be individually taken into account and the optical performance of the optical system 152 can be improved in conjunction with applying the adapted neural network 150 to the images captured by the optical system 152.

[0062] In embodiments, the fine-tuning may be performed for a type or design of optical systems rather than for individual optical systems. In this case, a correction network 130 trained using training data generated using different types of optical systems may be fine-tuned, with the training data 156 being created in association with the one type of optical system. In this case, the adapted neural network 150 may be stored in association with an identifier for the type of optical system.

[0063] Figure 2 shows a schematic representation of the operation of a neural network 200 for correcting optical aberrations. The neural network 200 may include the correction network 130 or the adapted neural network 150, which, as in Figure 1were trained. The neural network is configured to reduce or remove optical aberrations from an image 210 by the neural network 200 generating an error-free or error-reduced image 220. The input image 210 may contain one or more optical aberrations, such as vignetting. Alternatively or additionally, the image 210 may also contain one or more optical aberrations, such as astigmatism, coma, chromatic aberration, spherical aberration, field curvature, distortion, Gaussian error, or defocus. Aberrations may be due to technical reasons.For example, dimensional and shape deviations of the elements (lenses and mirrors) of an optical system, deviations of the elements of the optical system from their intended positions, a manufacturing-related deviating refractive index of lenses of the optical system, inhomogeneities (streaks) in the glass of an optic or the optical system, or residual stresses of the optics and stresses through the mount, which can lead to stress birefringence, can be related to aberrations in a captured image.

[0064] When the neural network 200 is applied to an error-prone image 210, or when this error-prone image 210 is input to the neural network, the neural network 200 can generate an output image 220 from the image 210. In the output image 220, the one or more optical aberrations can be reduced or eliminated. This can be expressed by the image quality (e.g., the detail contrast) being greater in the output image 220 or in regions of the output image 220 than in the input image 210 or the corresponding regions of the input image.

[0065] The application of the neural network 200 differs from the training of the neural network 200 in the data sets used. During training, one or more faulty images are input into the neural network, and the internal parameters of the neural network are adjusted so that the output images of the neural network best match the target output images. During application of the neural network, the image data passes through the neural network once, and the neural network generates an output image as a prediction.

[0066] Neural networks can represent results learned through at least one deep learning process and / or at least one deep learning method. These neural networks condense the knowledge gathered for a specific task ensemble in a suitable manner through automated learning, so that a specific task can then be performed automatically and with the highest quality.

[0067] An imaging and / or image recording system may be configured to capture one or more images, wherein one or more optical aberrations in the one or more captured images are associated with at least a portion of the imaging and / or image recording system. The one or more images may include image 210, which may be processed by neural network 200. The imaging and / or image recording system may include an optical system, such as a lens, optics or individual lenses, a photographic layer, an sCMOS ("scientific complementary metal-oxide-semiconductor") or CCD ("charge-coupled device") sensor, or one or more diffusion screens. In embodiments, the one or more captured images may be stored, and the one or more stored images may be input to neural network 200.Alternatively, the one or more acquired images may be fed directly into the neural network 200 and only the one or more corrected images may be stored.

[0068] Figure 3 shows a device 300 comprising one or more processors 310 and one or more storage media 320. The device 300 may comprise an imaging and / or image recording system. Alternatively, the device 300 may be spatially separated from an imaging and / or image recording system and connected to the imaging and / or image recording system via a network, for example, a wireless network. In this case, the device may comprise a workstation, a server, a microcomputer, a computer, or an embedded computer.

[0069] The one or more processors 310 can include computing accelerators such as graphical processing units (GPUs), tensor processing units (TPUs), application-specific integrated circuits (ASICs) or field-programmable gated arrays (FPGAs) specialized for machine learning (ML) and / or deep learning (DL), or at least one central processing unit (CPU). An application-specific integrated circuit (ASIC, also known as a custom chip) is an electronic circuit that can be implemented as an integrated circuit. Because their architecture is adapted to a specific problem, ASICs operate very efficiently and considerably faster than a functionally equivalent implementation in software in a microcontroller. Tensor processing units (TPUs), also known as tensor processors, are application-specific chips and can accelerate machine learning applications compared to CPUs.This or similar specialized hardware can be used to optimally solve deep learning tasks. In particular, the application of a neural network, which requires orders of magnitude less computing power than training, i.e., the development of a model, also works on conventional CPUs.

[0070] Furthermore, in embodiments, the device may comprise one or more neural networks 330. With the help of the one or more neural networks 330, the device 300 may be enabled to correct or minimize optical aberrations in images using artificial intelligence (AI). The one or more neural networks 330 may be executed by the one or more processors 310. Executing neural networks 330 requires orders of magnitude less computing power than training or developing a neural network.

[0071] By implementing the neural network 330 on the device 300, the device gains additional "intelligence." The device 300 can thus be enabled to solve a desired task independently. This results in a cognitively enhanced device 300. Cognitively enhanced means that the device can be enabled to semantically recognize and process image content or other data through the use of neural networks (or deep learning models) or other machine learning methods.

[0072] Furthermore, the device 300 may include one or more components 340. For example, the one or more components 340 may include a user interface and / or an interface for downloading neural networks to the device 300. In one embodiment, the one or more components 340 include an image recording system for capturing images.

[0073] In embodiments, the device 300 can be used to train a neural network 330. For this purpose, the device 300 can comprise a device for determining a neural network for correcting optical aberrations. Computer-executable instructions stored on the one or more computer-readable storage media 320, when executed by the one or more processors 310, can cause one or parts of the methods according to the Figure 1 and / or 4.

[0074] Figure 4shows a schematic flowchart according to an embodiment of a (computer-implemented) method 400 according to the invention for determining a first neural network for correcting optical aberrations. The method 400 comprises a step 418 in which one or more images are determined in order to determine a first neural network based on these images (step 420). The determination 418 of the one or more images can comprise determining training images for training a first neural network. The determination of the training data or training images can comprise one or more measurements and / or one or more simulations in connection with one or more optical systems or types (designs) of optical systems and / or one or more sample types in order to generate the training data.Alternatively, training data stored in a database or provided by third parties can be used for training. The training data comprises one or more image pairs, where the image pairs contain error-prone input images and error-free or error-reduced target output images.

[0075] In an optional step 410, training can be prepared. In step 410, the training conditions can be specified. This can include determining a second neural network as a loss function for the first neural network (the neural network to be trained). For example, this can be done by training the second neural network. Alternatively, the determination can include selecting the second neural network from a plurality of neural networks, wherein the plurality of neural networks have been trained on different sample types and the second neural network is selected depending on the sample type. The second neural network can have been trained or be trained on a sample type that is related to the later application of the first neural network.The second neural network may be configured to make predictions, such as classification, based on images as input to the second neural network.

[0076] In step 420, the first neural network is trained using the training data from step 418. During training in step 420, internal parameters (e.g., weights "W" and threshold values ​​"B") of the first neural network are found that optimally or best map a plurality of input images input to the first neural network to the target output images. Thus, the first neural network is capable of generating new images from images and solving a task related to the training data. The first neural network can be trained to remove or reduce one or more optical aberrations in images such that, for example, input images have a lower detail contrast or lower sharpness than corresponding output images of the first neural network.

[0077] In step 426, an optical system is determined. To uniquely identify this optical system, the optical system is assigned an identifier according to the invention. For this purpose, the optical system can be provided with an individual code. This can be done during the optical system's manufacture.

[0078] In step 428, individual training data is determined. In embodiments, determining the individual training data comprises determining optical properties of the individual optical system determined in step 426. This can include measuring and / or simulating the optical system. For example, an optical transfer function OTF can be determined for the individual optical system. This can occur during or after the manufacture of the optical system. The optical system can be uniquely assigned to the OTF using the identifier of the optical system. In this case, the individual training data can be generated based on the optical properties of the optical system for fine-tuning. With the help of knowledge of the OTF, image pairs can be generated as individual training data, since the OTF describes the optical aberrations of the associated optical system.

[0079] In a step 430, the first neural network determined in step 420 can be fine-tuned. Here, the first neural network is further trained to obtain an adapted (third) neural network. The adapted neural network can be trained for a specific application. For example, the first neural network can be further trained using the individual training data generated in step 428. Thus, the adapted (third) neural network can be trained specifically for an individual optical system. Alternatively, the first neural network can also be further trained for a specific application purpose by using training data of a specific sample type (which is in a context with the specific application purpose) for fine-tuning.

[0080] During further training (fine-tuning) in step 430, these individual training data can be used to further train at least a portion of the pre-trained (first) neural network. For example, only a portion of the internal parameters of the neural network can be changed depending on the further training, while the remaining internal parameters cannot be changed by the further training. This enables rapid individual adaptation of the neural network to a specific application, such as the correction of optical aberrations in images acquired by a specific optical system.

[0081] By fine-tuning neural networks, they can be continuously improved and / or the application range of the neural networks can be specified. This can advantageously be achieved by training only a few nodes in a neural network. For example, a slightly damaged optical system can impair the quality of images captured with this damaged optical system. By fine-tuning and / or training a neural network according to the method according to the invention, a neural network can be determined to correct these errors.

[0082] In a further step 440, the adapted neural network can be made available to third parties, applied to a device, the device being associated with the optical system, or stored in a cloud, on a server or other data storage.

[0083] Various optical aberrations associated with the manufacture and operation of optical systems can be corrected using machine learning (ML) methods. The ML methods described include algorithms that allow machines to learn from experience and can be derived from so-called "deep learning" (DL), a specific type of neural network. Effective correction of aberrations enables the cost-effective production of high-performance lenses, as aberrations in lenses with significantly larger aberrations, especially larger variations between individual lenses of the same series, can be corrected. This allows, on the one hand, tolerances between lenses to be compensated by allowing a greater spread in the tolerances. Furthermore, corresponding lenses can be designed to optimize their imaging quality along the optical axis.The optical correction at the edge of the image field doesn't need to be as high, as this can be corrected by the neural networks. For example, during lens design, aberrations can be evaluated based on the complexity of the correction provided by the lens or software post-processing (application of the neural network). Errors that are particularly easy to correct using the ML / DL network can be given less priority in the design / concept of the lens. Reference symbols:

[0084] 110, 120, 140 training steps 112 process step 114, 128, 144 data storage 116, 130, 150 neural network 122, 124 process steps 126 image pairs 136 loss function 142 process step 152 optical system 154 identifier 156 individual training data 200 neural network 210 image with optical aberrations 220 corrected image 300 device 310 processor 320 storage medium 330 neural network 340 components 400 process 410 - 440 process steps

Claims

1. Method for determining a neural network (130; 200; 330) for correcting optical imaging errors, having the steps of: determining (418) a plurality of images, wherein the plurality of images are at least partially related to an optical system (152) or the type of an optical system (152), wherein determining (418) the plurality of images comprises determining training images comprising a plurality of image pairs (126); and determining (420) a neural network (130; 200; 330) based on the determined plurality of images such that the determined neural network (130; 200; 330) applied to an image (210) acquired by means of the optical system (152) outputs an image (220) which has been corrected with respect to one or more optical imaging errors, wherein determining (418) the neural network (130; 200; 330) comprises training the neural network (130; 200; 330) on the training images, wherein each image pair of the plurality of image pairs (126) comprises an input image (Xi) for the neural network (130; 200; 330) and a target output image (y0); characterized in that the method further comprises the step (140) of fine-tuning (430) the determined neural network (130; 200; 330), wherein the fine-tuning (430) comprises training specifically for the optical system (152), wherein the one or more optical imaging errors comprise at least one of the following optical imaging errors: astigmatism, vignetting, coma, chromatic aberration, spherical aberration, and defocusing, wherein the fine-tuned neural network can be assigned to the optical system via a unique identifier of the optical system, preferably an identification number, such that the fine-tuned neural network can be accessed using the identifier of the optical system.

2. The method according to claim 1, characterized in that the training images are generated from acquired images, wherein the acquired images were acquired by means of one or more optical systems and / or with the optical system (152), and wherein none, a subset, or all of the one or more optical systems are of the same type as the optical system (152), and / or wherein the acquired images depict one or more sample types.

3. The method according to claim 2, characterized in that the training images are generated by a simulation of optical imaging errors, or the method further comprises the step of determining at least one optical transfer function, OTF, or at least one point spread function, PSF, for the optical system (152) or for optical systems that are at least partially identical to the optical system (152), wherein input images of the plurality of image pairs (126) are convolution of error-free images with one of the at least one point spread function, PSF, and wherein the error-free images are used as corresponding target output images of the plurality of image pairs (126), and / or training the neural network (130; 200; 330) comprises training on the training images and on further data, wherein the further data comprises at least one of the following: parameter data related to the training images, validation data, measurement data related to the production of the optical system (152), data on the course of an experiment or a measurement, information about reagents and materials, information about an object or a sample, information about the optical system (152), user-related data, user inputs, and information about an image acquisition system.

4. The method according to any one of claims 1 to 3, characterized in that the neural network (130; 200; 330) is a first neural network (130; 200; 330) and the training of the first neural network (130; 200; 330) comprises applying a second neural network (136), wherein the second neural network (136) is applied as a loss function for the training of the first neural network (130; 200; 330).

5. The method according to any one of claims 1 to 4, characterized in that the fine-tuning (430) comprises training only a part of the determined neural network (130; 200; 330), wherein one or more parameters of the determined neural network (130; 200; 330) remain unchangeable during the fine-tuning (430).

6. The method according to any one of claims 1 to 5, characterized in that the method further comprises the step of generating (428) individual training data (156) for the fine-tuning (430), wherein generating (428) the individual training data (156) comprises determining (142) optical properties of the optical system and / or detecting measurement samples by means of the optical system (152), wherein the individual training data (156) are generated on the basis of the optical properties and / or the measurement samples.

7. The method according to claim 6, characterized in that the method comprises the step of storing the measurement samples, the optical properties and / or the individual training data (156), and / or wherein the measurement samples, the optical properties and / or the individual training data (156) are uniquely assigned to the optical system (152).

8. The method according to any one of claims 1 to 7, characterized in that the optical system (152) is uniquely and / or automatically identifiable, and / or the neural network (130; 200; 330) can be uniquely assigned to the optical system (152).

9. The method according to claim 8, characterized in that the optical system (152) is identifiable by means of electromagnetic identification, optical identification (154), mechanical identification or magnetic identification.

10. The method according to any one of claims 1 to 9, characterized in that the method comprises the step of applying the determined neural network (130; 200; 330) to acquired data (210), wherein the acquired data (210) are acquired by means of the optical system (152) or an optical system of the same type.

11. The method according to claim 1, characterized in that determining (420) the neural network (130; 200; 330) comprises training the neural network (130; 200; 330) based on the plurality of images, wherein the neural network (130; 200; 330) during the training learns how objects and / or structures in the plurality of images ideally look and corrects deviations therefrom.

12. The method according to any one of claims 1 to 11, characterized in that the optical system (152) is part of an imaging and / or image recording system of a microscope, a microscope system, a camera, a smartphone, a telescope, a mobile computer, a stationary computer or a measuring appliance.

13. Device (300) for correcting optical imaging errors, comprising: one or more processors (310); one or more computer-readable storage media (320) having computer-executable instructions stored thereon that, when executed by the one or more processors (310), cause one or more images (210) to be acquired by means of an imaging and / or image recording system, wherein one or more optical imaging errors in the one or more acquired images (210) are associated with at least a part of the imaging and / or image recording system, wherein the at least one part of the imaging and / or image recording system comprises an optical system; and a fine-tuned neural network (130, 150; 200; 330) is applied to the one or more acquired images (210), wherein the fine-tuned neural network (130, 150; 200; 330) is configured to generate one or more corresponding corrected images (220) from the one or more acquired images (210) such that the one or more optical imaging errors are corrected or reduced in the one or more corrected images (220), wherein the fine-tuned neural network (130, 150; 200; 330) is determined by fine-tuning a neural network (130; 200; 330), wherein the fine adjustment (430) comprises training only a part of the neural network (130; 200; 330), and wherein the fine adjustment (430) comprises training specifically for the optical system (152), wherein the one or more optical imaging errors comprise at least one of the following optical imaging errors: astigmatism, vignetting, coma, chromatic aberration, spherical aberration and defocusing, and wherein the fine-tuned neural network can be assigned to the optical system via a unique identifier of the optical system, preferably an identification number, such that the fine-tuned neural network can be accessed using the identifier of the optical system.

14. The device (300) according to claim 13, characterized in that the one or more acquired images (210) are stored and the neural network (130, 150; 200; 330) is applied to the one or more stored acquired images (210), or in that the neural network (130, 150; 200; 330) is applied directly to the one or more acquired images (210) and only the one or more corrected images (220) are stored.

15. The device (300) according to any one of claims 13 and 14, characterized in that the at least one part of the imaging and / or image recording system comprises a photographic layer, an sCMOS- or CCD-sensor, or one or more diffuser plates.