Optical correction through machine learning

Abstract

The invention relates to a method for determining a neural network (130; 200; 330) for correcting optical imaging errors, comprising the following steps: determining (418) one or more images, wherein the one or more images (126) are at least partially related to an optical system (152) or a configuration thereof; determining (420) a neural network (130; 200; 330) based on the determined one or more images, in such a way that the determined neural network (130; 200; 330) is applied to an image (210) acquired by the optical system (152) to output an image (220) corrected for one or more optical imaging errors.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for determining a neural network for correcting optical imaging errors. Background Art

[0002] An optical system can be understood as a mapping function, also called optical transfer function (OTF), in such a way that the waveform leaving the optical system at the output is different from the waveform at the input of the optical system. Some of these variations are undesirable and lead to known optical imaging errors, such as aberrations, astigmatism, vignetting and / or coma. Optical imaging errors may be related to the way an optical system, such as a lens or individual lenses, is manufactured and operates. These optical imaging errors affect the image quality, resulting in some information in the image not being directly visible.

[0003] The prior art discloses various methods for correcting optical imaging errors. One possible solution to avoid these optical errors in the prior art is to optimize the optical system during its manufacture. For example, in order to reduce chromatic aberration, the lens is tempered (coated), or certain glasses or minerals with strictly defined physical properties are used to reduce optical imaging errors. Alternatively, in some cases, the use of additional optical devices can also lead to improved optical characteristics of the optical system.

[0004] However, this method in the prior art exhibits some disadvantages and problems that will be examined below. Optical errors can be reduced by using higher quality materials or using other optical elements in the optical system, but they cannot be completely avoided. For example, the correction of image errors in the lens can be reduced towards the edge. The numerical aperture for which image field the lens is specified is to a certain extent determined by the developer and is related to the decline in certain quality parameters and the expected manufacturing tolerances. However, optical imaging errors related to manufacturing tolerances cannot be eliminated by optimizing the optical structure. Therefore, only very small manufacturing tolerances can be allowed for high-quality lenses. In addition, high costs (material costs or costs due to additional components) are generated by this method for reducing imaging errors.

[0005] Another possibility for correcting optical imaging errors is to further process the images acquired by the image recording system by means of electronic data processing (EDV). Usually, the correction of image errors is carried out using mathematical models. Here, the individual components of the errors can be described using Zernike polynomials. The image errors are decomposed into components such as spherical aberration, chromatic aberration, astigmatism, coma, etc., which in turn are assigned to certain Zernike polynomials. If an image of a point source is now acquired using a lens and the point spread function (PSF) of the lens is determined from this, the contribution of the individual image errors and thus the amplitude of the Zernike polynomials can be inferred by comparing the measured point image with the ideal point image. In this case, however, the PSF of the lens needs to be determined exactly. This is not always possible or is sometimes too imprecise.

[0006] Other methods, such as vignetting correction or deconvolution, can also be used to correct optical errors. In existing methods for vignetting correction, a reference image is usually acquired using a uniform sample, which is then settled with the acquired image of the sample using a linear method. However, the disadvantage of this method is that it only works at a sample level and ignores other errors related to vignetting.

[0007] The disadvantage of these mathematical methods used in the prior art to correct imaging errors is that these mathematical methods cannot or can only be used to a limited extent for old data acquired with unknown lenses, because these mathematical methods rely on measuring specific samples or measuring lenses and their optical properties. Therefore, these methods in the prior art cannot be applied to old data (images) acquired with unknown lenses, or provide inaccurate results. Summary of the invention

[0008] It is therefore an object of the present invention to provide an improved mechanism for correcting optical imaging errors.

[0009] The invention solves the above-mentioned problems and objects by a method for determining a neural network for correcting optical imaging errors. The method comprises the following steps: determining one or more images, wherein the one or more images are at least partially related to the optical system or the configuration of the optical system; determining a neural network based on the determined one or more images, in such a way that the determined neural network is applied to the images acquired by the optical system and an image corrected for one or more optical imaging errors is output.

[0010] The apparatus according to the present invention comprises one or more processors and one or more computer-readable storage media, wherein computer-executable instructions are stored on the one or more computer-readable storage media, which instructions, when executed by the one or more processors, cause one or more images to be acquired through an imaging and / or image recording system, wherein one or more optical imaging errors in the one or more acquired images are associated with at least a part of the imaging and / or image recording system, and a neural network is applied to the one or more acquired images, wherein the neural network is designed to generate one or more corresponding corrected images from the one or more acquired images, thereby correcting or reducing one or more optical imaging errors in the one or more corrected images.

[0011] The method according to the invention and the device according to the invention have the following advantages: the neural network, for example in the sense of deep learning, is specified or applied to the correction of optical imaging errors. This can achieve an improvement in the imaging performance of existing optical systems. In addition, the method according to the invention and the device according to the invention can achieve the ability to save manufacturing costs for optical systems. The manufacturing costs of optical systems (for example high-performance systems) depend to a large extent on what measures must be taken to avoid optical imaging errors (such as the choice of materials, the application of coatings, the connection of other lenses and lens groups). Another cost factor in the production of optical systems is the permissible deviation of the optical structure. The method according to the invention allows a higher tolerance for changes in optical parameters. This can reduce costs. By determining a neural network for correcting optical imaging errors, for example, certain manufacturing steps in the manufacture of the optical system can be saved, more cost-effective materials can be used, and / or the optical structure can be simplified (for example by eliminating lenses). As a result, image recording systems or optical systems that work in combination with neural networks for correcting optical imaging errors can be manufactured more cost-effectively. In addition, neural networks can achieve better popularity for previously unknown applications.

[0012] A further advantage of the method according to the invention and of the device according to the invention is that optical systems can be adjusted so as to expand their fields of application without affecting the image quality. For example, lower requirements can be placed on certain parameters of the optical system, thereby increasing the degrees of freedom of other optical properties, which opens up new fields of application. For example, maximum flatness and maximum working distance are mutually exclusive. If optical errors related to the flatness of the optical system can be ignored by determining a neural network for correcting these errors, a larger working distance can be achieved, thereby providing greater flexibility in sample selection. In addition, the method according to the invention can provide means for simultaneously correcting multiple optical errors that are also nonlinear.

[0013] The method according to the invention and the device according to the invention can be further improved by specific design. The various technical features of the invention described below can be arbitrarily combined with each other and / or omitted, as long as the technical effects achieved by the omitted technical features are not important.

[0014] In one embodiment, determining the neural network includes: training the neural network. Determining one or more images may include: determining training images, wherein training the neural network includes: training on training images including a plurality of image pairs. Each image pair of the plurality of image pairs used for training may respectively include an input image and a given output image for the neural network. This may enable correction of imaging errors in old data acquired with an unknown lens, and it is thereby possible that the neural network has learned to correct optical imaging errors with the aid of training data related to the optical system or the way it is constructed. Therefore, in individual cases it is not absolutely necessary to perform a reference measurement for each existing lens, because the neural network will learn what the image data and the image content should look like. Conventional methods cannot do this.

[0015] In some embodiments, training images can be generated from acquired images, for example, depicting one or more types of samples, wherein the acquired images are acquired by means of one or more optical systems and / or using the optical system, and wherein none, part or all of the one or more optical systems are of the same type as the optical system. In other embodiments that can be combined with the previous embodiments, training images can be generated by simulating optical imaging errors. For simulation, an optical transfer function (OTF) or point spread function (PSF) of the optical system or a type of optical system can be determined. Alternatively, the OTF or PSF can also be determined independently of the simulation. With the help of PSF or OTF that can be converted to each other, input images of multiple image pairs can be generated by convolving error-free images with PSF. This allows the generation of almost "perfect" training data (image pairs) because error-free images can be used as corresponding given output images of multiple image pairs. The optical system can be part of an imaging and / or image recording system of a microscope, a microscope system, a camera, a smart phone, a telescope, a computer (which can also be portable) or a measuring device, and can include one or more optical components (such as lenses, mirrors and / or other optical mechanisms).

[0016] In one embodiment, for training the neural network (hereinafter also referred to as the first neural network), another (second) neural network may be used. The other neural network may be used as a loss function for training the first neural network. This enables improved training of the first neural network, since the neural network as a loss function enables accurate training, wherein it is ensured that the output of the neural network is equivalent to the desired image. This cannot be guaranteed if the error in the output image is calculated only pixel by pixel. Therefore, the output image is not processed as a set of independent pixels, but is placed in a semantic association.

[0017] In an advantageous embodiment, the determined neural network can be further trained (fine-tuned). Fine-tuning (hereinafter also referred to as fine tuning) can include training only a part of the determined (first) neural network. Here, one or more parameters of the determined neural network can remain unchanged during fine-tuning. In addition, fine-tuning can include training specifically for the optical system. The neural network can thus be adjusted to correct imaging errors specific to the optical system. The fine-tuned neural network can be explicitly assigned to the optical system. Fine-tuning can be performed with the aid of individual training data. Individual training data can be generated based on optical properties and / or measurement samples related to the optical system. The optical properties of the optical system can be determined in one step, and / or measurement samples can be acquired by the optical system. At least one of the measurement samples, optical properties and individual training data can be stored, for example, in a database, wherein the measurement samples, optical properties and / or individual training data are explicitly assigned to the optical system.

[0018] In the design of the method according to the invention, the optical system can be identified explicitly and / or automatically. The first neural network and the adaptive / fine-tuned neural network can be assigned explicitly to the optical system. For example, the optical system can be identified by electromagnetic recognition, optical recognition, mechanical recognition or magnetic recognition.

[0019] In a design that may be combined with the preceding design, a method for correcting optical imaging errors according to the present invention comprises applying the determined neural network or the fine-tuned neural network to acquired data. The acquired data (e.g., an image) may be obtained by an optical system or an optical system of the same type (and a photodetector, or in the case of a scanning system, using a point photodetector). The optical imaging errors may include astigmatism, vignetting, coma, chromatic aberration, spherical aberration, or defocus.

[0020] In some embodiments, the determination of the neural network may include: training the neural network with the aid of one or more specific images, wherein the neural network learns during training how objects and / or structures ideally appear in one or more images and corrects deviations therefrom. This makes it possible to correct imaging errors when the OTF is not completely known. In this case, the neural network can automatically identify what type of error exists and what type of object is imaged in the acquired image. Here, neither the error type nor the object type needs to be explicitly specified. The neural network is trained to transform the object in the acquired image into the corrected image so that the object is correctly reconstructed "implicitly" in the network, i.e. as part of the learning parameters. This presupposes that the neural network has seen similar objects during training. "Similar" here means that the same image features are present in the training image as in the image to be corrected, that is, the training image is associated with the acquired image.

[0021] In one embodiment, the device according to the invention is designed to store one or more acquired images and to apply the neural network to one or more stored acquired images. Alternatively, the neural network can be applied directly to one or more acquired images and only one or more corrected images can be stored. The correction of optical imaging errors can thus be performed already during image acquisition, i.e. before the image is stored ("in real time"), or only after it has been stored. At least part of the imaging and / or image recording system can include the optical system, the photographic layer, the sCMOS or CCD sensor or one or more scattering slides. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The subject matter of the invention is explained in more detail below with reference to exemplary drawings. An example of an advantageous embodiment of the invention is shown in the drawing.

[0023] Figure 1 A schematic diagram of a method for determining a neural network for correcting optical imaging errors according to one embodiment of the present invention;

[0024] Figure 2 A schematic diagram of a method for correcting optical imaging errors by means of a neural network according to one embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of a device according to one embodiment of the present invention;

[0026] Figure 4 The present invention is a schematic flow chart of one embodiment of the method of the present invention. DETAILED DESCRIPTION

[0027] Figure 1A schematic diagram of a method for determining a neural network 130 is shown. This neural network 130 for correcting optical imaging errors - also referred to below as a "correction network" - can be determined and / or trained in a plurality of steps 110 and 120. Additionally, the determined neural network 130 can be fine-tuned in a further training step 140. The neural network 130 for correcting aberrations can, for example, comprise an autoencoder, a U-Net or a "generative adversarial network" (GAN) and is designed to output an image. The neural network 130 belongs to the class of "convolutional neural networks", also referred to as CNNs.

[0028] In an optional first step 110, preparations can be made for training of steps 120 and / or 140. Possible preparations for training include determining a loss function 136 for training in the training step 120. For example, the second neural network 116 can be determined or trained and used at least partially as a basis for the loss function 136. For example, the second neural network 116 or parts of the second neural network 116, preferably parts of the second neural network 116 extracted from image parameters or image characteristics, can be used as loss function 136 for training the neural network 130 or can also be used as an objective function. In this case, the loss function 136 is a loss network.

[0029] The second neural network 116, also referred to below as the "main model", can be selected from a plurality of neural networks stored in the data storage 114, and need not be a neural network that outputs images itself. For example, the main model 116 can be a pre-trained neural network that solves a classification problem.

[0030] Alternatively, the master model can also be found or determined by machine learning in so-called "training" 112. The master model can then be trained with the aid of a plurality of samples from different fields of application (e.g. single cell cultures, three-dimensional cell cultures, tissue slices, organoids, spheroids, natural organs or living organisms). So-called "supervised training" can be used here. For training, the following scenario can be used: the model solves, for example, a classification problem, i.e. it does not even output images itself. The master model can be trained in connection with the subsequent field of application of the neural network 130. For example, a data set of microscope images can be used to train the master model.

[0031] The master model can be used as a "feature extractor" in order to calculate so-called "activations". Activations indicate which "neurons" respond to certain image parts (features), i.e., for example, recognized faces or (in the case of microscopy) cell organelles. In aggregate form, the activations of all neurons of the whole or part of the master model can be considered as a measure of the "image quality" and thus of the correctness of the rectified image, and therefore used as a loss function.

[0032] In some embodiments, the main model or parts of the main model can be used to make predictions through the neural network 130, or to accelerate the convergence of the neural network 130 through "transfer learning".

[0033] The rectification network 130 can be designed to transform the input image X i Convert to output image Or the input image X i Mapping to output image The loss function 136 can output the prediction of the correction network 130 A numerical measure of the degree of match with a given output image y0 (the error-free image) l feat .VARIABLE X i , and y0 are vectors or matrices whose elements are assigned to pixels of the image. The rectification network 130 to be trained and the loss function 136 form a system for training the rectification network 130.

[0034] The loss network or loss function can be used to define one or more loss functions that measure the and the corresponding given output image y0. During training the rectification network 130, the loss network is not changed or trained together.

[0035] In some embodiments, the correction network 130 can be a neural network, such as a "residual convolutional neural network", which is parameterized by weights W. The correction network 130 maps The input image X i Convert to output image Each of the one or more loss functions can compute a scalar value that represents the value of The difference between the given output image or target image y0. A deep learning method can be used to train the correction network. For example, the correction network can be trained by stochastic gradient descent to minimize the weighted combination of the loss function 136. For example, the weight W is adjusted so that the feature recovery loss l feat Minimum.

[0036] Loss functions based on per-pixel losses are prone to errors and may provide inaccurate results for training. To eliminate these shortcomings, in some embodiments, a loss function that determines the perceptual differences and semantic differences between images can be used. The main model 116, which has been trained for image classification, already has the property of expressing the features or semantic information of the input image in hidden representations ("hiddenrepresentations") or hidden features. These hidden representations can be used to make decisions about the similarity of each image. The neural network 116 can therefore define a loss function 136, in which each hidden representation and thus each semantic information is compared with each other. The loss network 136 can define a feature recovery loss l feat , which indicates that given the output image y0 and the output image of the rectification network 130 A measure of the difference in image content between two images.

[0037] Therefore, the perceptual loss can be determined by the loss network 136. Therefore, the output of the neural network 130 can be ensured by the loss network 136. looks like the expected image. This is not the case for the loss function which only calculates the error for each pixel of the image. Therefore, the output image Rather than being treated as a set of independent pixels, they are placed in a semantic context.

[0038] In some embodiments, a training data set is generated. The training data set may include image pairs 126 (training images), wherein the image pairs 126 are characterized in that one image of the image pair is an input image X of the neural network 130. i , and the other image of the image pair corresponds to a given output image y0 of the neural network 130. Input image X i It can be regarded as an image with errors, or an image to be measured or detected, while the given output image y0 can be regarded as the desired error-free or corrected image. Based on the image pair 126, in particular based on the given output image, the neural network 130 can be trained to correct one or more imaging errors among a large number of possible imaging errors. Through the image pair 126, the neural network can be trained to correct imaging errors that exist in the input image and are missing or weakened in the given output image. Alternatively or additionally, the training data set for training the neural network 130 may include other data. These other data may include at least one of the following: parameter data associated with the image pair 126 or the training image, verification data, measurement data related to the manufacture of the optical system, data about the experiment or measurement process, information about reagents and materials, information about the object or sample, information about the optical system, user-related data, user input and information about the image acquisition system.

[0039] The training data set can be generated using various methods or a combination of these methods. Figure 1 In the embodiment of the present invention, a distinction is made between two steps 122 and 124 for generating combinable training data. In step 122, the training data can be generated artificially. For example, the aberrations can be artificially generated by image processing. In step 124, the training data can be generated from measured or detected data, wherein the measured or detected data are obtained by one or more optical systems and the optical imaging errors are generated by one or more optical systems. The generated training data can be stored in a data storage 128. Some possible methods for generating training data are described in more detail below.

[0040] The first method (magnification strategy) for generating training data for training a neural network is based on acquiring images of uniform samples through one or more optical systems. Images with different zooms are created from these uniform samples. For example, a first recording of the maximum zoom of an optical system and a second recording of the minimum zoom of the same optical system are generated from the uniform sample. When recording with maximum or large zoom in the center of the image of the uniform sample, a (nearly) error-free image can be generated, and an image with errors can be generated with minimum or small zoom. This method exploits the fact that certain imaging errors have less impact in the center of the image than at the edges.

[0041] A second method for generating training data (shift strategy) enables the generation of image pairs 126 from recordings of uniform structured samples. Like the first method, this method also exploits the fact that certain optical imaging errors have a weaker effect in the center of the image than at the edge of the image. In this second method, at least one image is recorded with an arbitrary zoom, such as medium zoom or high zoom. A given output image is created from the image center of the recorded image of at least one recorded image, and the input image is generated from the recorded image at a specified position at an edge or corner of the recorded image. The image pairs 126 of the training data set can include many different specified positions and / or different zooms. In order to create the training data set, one or more uniform and / or structured samples can be recorded using one or more optical systems of one or more types.

[0042] The third method (simulation strategy) for generating training data is based on the simulation of optical imaging errors. If the optical transfer function (OTF) equivalent to the Fourier transform point spread function (PSF) of the optical system or the PSF of the optical system is known or can be measured or simulated, the image pair 126 can be artificially generated. The error-free image that can be used as a given output image can be converted into an error-containing image that can be used as an input image by mathematical methods, such as convolution of the error-free image with the PSF of the optical system.

[0043] The training data set may be generated using multiple optical systems of different structural types, such as multiple lenses. This may include all optical systems or all types of optical systems associated with which a neural network trained with the training data is employed. For example, training data may be generated using a variety of different lenses, and the training data may be used to train a neural network. The neural network may be used to correct imaging errors in measurement images acquired by one or more different lenses.

[0044] The training data set may be stored in the cloud, on data storage 128, on a computer such as a stand-alone work computer, or on a server suitable for training a neural network. The server or computer may then perform the training and then store the results from the training (e.g., on data storage 128 or another memory).

[0045] Deep learning methods can be used to train neural networks. This includes the planned use of at least one deep learning method, but preferably a plurality of deep learning methods for achieving a specific goal. The goals here can include image processing (e.g. correcting one or more optical errors, generating an image from another image, wherein at least one feature in the images is different, etc.). The deep learning method can include a series of method steps, which divide a process into a plurality of steps that are traceable, that is, repeatable. These method steps can be certain deep learning algorithms. But they can also be methods for learning networks (e.g. back propagation), which can be, for example, the type of data collection, or the way how to process data about hardware, etc.

[0046] To train the neural network 130, a training data set may be used, which may be generated, for example, according to one or more of the above methods, or provided by a third party. In one embodiment, the goal for training is to generate a neural network 130 that converts input images, such as those detected by an optical system, into corrected images, wherein the converted image corrects or reduces at least one imaging error generated by the optical system in the input image. This means that the image quality, such as contrast or sharpness, is improved in the output image (in one or more regions) compared to the input image. Ideally, this allows an error-free image to be generated from an image with errors.

[0047] In a further stage 140, the so-called "fine tuning", the correction network 130 can be further optimized. Optical systems are usually produced with certain tolerances during manufacturing. Therefore, optical systems produced by the same manufacturing process can have deviations from each other within the range of the manufacturing tolerances. These deviations affect the optical imaging errors generated by the optical system. Therefore, the same type of optical systems can be associated with different OTFs.

[0048] To compensate for these tolerances in manufacturing, characteristics of the optical system may be determined in step 142. For example, optical characteristics of a single optical system 152 may be measured. Measuring the optical system 152 may include detecting one or more measurement samples. The measurement samples and / or characteristics of the optical system may be stored on the data storage 144. In some embodiments, determining the characteristics of the optical system 152 includes determining the OTF and / or PSF of the manufactured optical system 152. A data set with the measurement samples and / or characteristics of the optical system may be available throughout the life cycle of the optical system. In one embodiment, a data set with the measurement samples and / or characteristics of the optical system may be stored in the cloud.

[0049] In order to be able to assign the properties of the measurement sample and / or the optical system 152 to the optical system 152, such as a lens, the data comprising the properties of the measurement sample and / or the optical system are assigned to an identification 154 of the optical system and can be stored according to the identification 154 or together with an identification number corresponding to the identification 154. The optical system 152 can receive its identification 154 during manufacture, wherein the identification 152 can be unique for each optical system and can be permanently attached to the optical system 152. For example, the identification 154 can be an optical code (e.g. a barcode, a quick response code, a mark or a certain color), a code of one or more threads and / or one or more mechanical parts, a certain shape, a certain weight, a sound cord, a steel engraving, a radio frequency transponder, a magnetic tape chip card or a magnetic color.

[0050] Thus, the optical system 152 may be identified by electromagnetic identification, optical identification, mechanical identification, magnetic identification, or a combination thereof.

[0051] Based on the measurement samples and / or the optical properties, training data 156 may be generated. These training data may be generated as training data, wherein the training data 156 is only or specifically associated with the optical system 152. The training data 156 may be generated based on the measurement samples, which may contain images acquired by the optical system 152, or based on simulations with the aid of the OTF or the associated PSF of the optical system 152.

[0052] Since the optical characteristics of the measurement sample and the optical system can be assigned to the optical system 152, the training data 156 can also be assigned to the optical system 152. Therefore, the training data 156 can be stored according to the identification 154. Figure 1 As shown, these data 156 can be stored in the data storage 144 according to the identification 154. In some embodiments, one or more of the data storages 114, 128 and 144 can be the same. For example, all training steps 110, 120 and 130 can be performed at the manufacturer of the optical system 152. Alternatively, individual training steps 110, 120 and 130, such as fine-tuning 130, can also be performed at the user of the optical system 152.

[0053] The identification 154 of the optical system 152 or an identification number corresponding to the identification 154 can be used to access a data set having characteristics of a measured sample and / or the optical system 152, or to access a data set having training data 156 associated with the optical system. In some embodiments, these data sets can be stored on a manufacturer's server or in the cloud. For example, a user can access these data sets with authorization credentials such as a username and password or an identification number of the optical system 152.

[0054] In a training step 140, neural network 130 from training step 120 can be fine-tuned. The fine-tuning is explained below by way of example with reference to neural network 130. Alternatively, however, neural networks from other sources, such as neural network users, can also be further fine-tuned.

[0055] Fine-tuning means adapting a previously trained neural network that can already recognize basic image features to a new unknown data set. A neural network, such as neural network 130, can be pre-trained in a way that corrects certain image errors. For this purpose, the most extensive data set can be used. For a specific optical mechanism or a specific optical system, such as a certain lens, the prediction accuracy, that is, the image quality of the output image, can then be improved by fine-tuning, in that a (smaller) training data set 156 is generated using the lens and the pre-trained network 130 is "fine-tuned", that is, adjusted. This fine-tuning can be done at the manufacturer of the optical system. However, it can also be done at the customer or a third party. Fine-tuning can be done for a specific optical system or one or more specific samples. For example, these one or more specific samples may not be included in the original training data set or are not representative enough. Therefore, the prediction accuracy of the neural network can be improved by fine-tuning. During fine-tuning, the pre-trained neural network can be trained on new data, and its learning speed ("learning rate") is 1-2 orders of magnitude lower. "Learning rate" in this sense is a scaling factor for numerical optimization, which determines the step size of changing the learned model parameters. If this value is large, the model can converge in fewer steps, but there is also a risk that the model parameters will deviate from the optimal value again. When fine-tuning, it can be assumed that the pre-trained parameters are already close to the optimal value. Therefore, compared with the training in step 120, the step size or learning rate can be reduced to find the global minimum of the loss function. This prevents the existing "knowledge" of the model represented by the model parameters from being destroyed by steps that are too large.

[0056] Training data set 156 can be used to fine-tune neural network 130. When fine-tuning, an existing neural network such as neural network 130 can be used as a basis for further training. Further training includes: training only a portion of the neural network. Here, some parameters of the neural network are fixed or unchangeable, while the remaining parameters can be influenced or changed by further training. This allows for fast training. Further training can be performed with different learning rates depending on the position of the parameters in the neural network.

[0057] To further train the training data 156, various deep learning methods may be used. In some embodiments, the neural network 150 is trained with a second neural network as a loss function. This may be done as in Figure 1 This is done as in the training step 120 in .

[0058] Through further training, an adaptive neural network 150 is generated, which is specially trained for the optical system 152. Therefore, it is possible to make the neural network 150 individually adaptable to the specific optical system 152 (such as a lens), while the basic characteristics of the correction network 130 that can correct certain optical errors are retained.

[0059] The adaptive neural network 150 may be stored in a data storage, a server or a cloud together with the identification 154 of the optical system 152 or with an identification number corresponding to the identification 154. In some embodiments, the adaptive neural network 150 is implemented and used on a server or a cloud. Therefore, the server or the cloud is designed to correct optical imaging errors in the image with the help of the adaptive neural network 150. For example, an image with errors may be uploaded to the cloud or the server. With the help of the adaptive neural network 150, an error-free or error-reduced image may then be generated and provided. The adaptive neural network 150 may be accessed with the help of the identification 154 or the corresponding identification number of the optical system 152. For example, a user may access the adaptive neural network 150 with the help of an authorization credential such as a username and a password. The user's authorization credential may be obtained by purchasing or receiving the optical system. Alternatively, the lens or the optical system 152 may be part of an image recording system and the neural network 150 may be implemented on the image recording system.

[0060] Thus, variations within manufacturing tolerances that occur in the manufacture of the optical system may be considered separately and may be combined with applying adaptive neural network 150 to images acquired by optical system 152 to improve the optical performance of optical system 152.

[0061] In some embodiments, fine-tuning can be performed for one type or configuration of optical systems rather than for individual optical systems. Here, a correction network 130 that was trained using training data generated using different types of optical systems can be fine-tuned, wherein the training data 156 is created in conjunction with one type of optical system. In this case, the adaptive neural network 150 can be stored with an identification of the type of optical system.

[0062] Figure 2 FIG. 2 is a schematic diagram showing the working mode of a neural network 200 for correcting optical imaging errors. The neural network 200 may include: Figure 1The neural network 200 is designed to reduce or remove optical imaging errors in the image 210 in such a way that the neural network 200 generates an error-free or error-reduced image 220. The input image 210 may contain one or more optical imaging errors, such as vignetting. Alternatively or additionally, the image 210 may also contain one or more optical imaging errors, such as astigmatism, coma, chromatic aberration, spherical aberration, image field curvature, distortion, Gaussian error or defocus. Aberrations may be due to technical reasons. For example, size deviations and shape deviations of optical system elements (lenses and mirrors), deviations of optical system elements relative to their specified positions, manufacturing-induced deviations in the refractive index of optical system lenses, inhomogeneities in the optical structure or in the glass of the optical system (streaks, or inherent stresses of the optical structure and stresses caused by the frame - which may lead to stress birefringence, are related to aberrations in the acquired image.

[0063] If the neural network 200 is applied to the image 210 with errors or the image 210 with errors is input into the neural network, the neural network 200 can generate an output image 220 from the image 210. In the output image 220, one or more optical imaging errors can be reduced or eliminated. This can be expressed as the image quality (e.g., detail contrast) in the output image 220 or in a region of the output image 220 is greater than the image quality in the input image 210 or in a corresponding region of the input image.

[0064] The difference between the application of neural network 200 and the training of neural network 200 lies in the data set used. In training, one or more images with errors are input to the neural network, and the internal parameters of the neural network are adjusted so that the output image of the neural network matches the given output image as closely as possible. When using the neural network, the image data passes through the neural network once, and the neural network generates an output image as a prediction.

[0065] Neural networks can represent results, wherein these results are learned by at least one deep learning process and / or at least one deep learning method. These neural networks condense the collected knowledge into a specific set of tasks in a suitable way by automatic learning, so that the specific tasks can be automated and performed with the highest quality in the future.

[0066] The imaging and / or image recording system may be designed to acquire one or more images, wherein one or more optical imaging errors in the one or more acquired images are associated with at least a portion of the imaging and / or image recording system. The one or more images may include an image 210 that can be processed by the neural network 200. The imaging and / or image recording system may include an optical system, such as a lens, an optical mechanism or lenses, a photographic layer, a sCMOS (“scientific complementary metal oxide semiconductor”) or CCD (“charge coupled device”) sensor, or one or more scattering slides. In some embodiments, one or more acquired images may be stored, and the one or more stored images may be input into the neural network 200. Alternatively, one or more acquired images may be directly input into the neural network 200, and only one or more corrected images may be stored.

[0067] Figure 3 An apparatus 300 is shown, which includes one or more processors 310 and one or more storage media 320. The apparatus 300 may include an imaging and / or image recording system. Alternatively, the apparatus 300 may also be spatially separated from the imaging and / or image recording system and connected to the imaging and / or image recording system via a network (e.g., a radio network). In this case, the apparatus may include a workstation computer, a server, a microcomputer, a computer, or an embedded computer.

[0068] One or more processors 310 may include a computing accelerator, such as a graphics processing unit (GPU), a tensor processing unit (TPU), or a field programmable gate array (FPGA) or at least one central processing unit (CPU) on a machine learning (ML) and / or deep learning (DL) application specific integrated circuit (ASIC). An application specific integrated circuit (ASIC, also known as a custom chip) is an electronic circuit that can be implemented as an integrated circuit. Because its architecture is suitable for specific problems, ASICs are very efficient and much faster than functional equivalents implemented in software in a microcontroller. A tensor processing unit (TPU), also known as a tensor processor, is an application-specific chip and can accelerate applications in the field of machine learning compared to a CPU. This or similar specialized hardware can be used to optimally solve deep learning tasks. In particular, the use of neural networks requires several orders of magnitude less computing power than training, i.e., model development, and is also suitable for common CPUs.

[0069] In addition, in some embodiments, the device may include one or more neural networks 330. With the one or more neural networks 330, the device 300 can correct or minimize optical imaging errors in the image with the help of artificial intelligence (AI). The one or more neural networks 330 can be run by one or more processors 310. The computing power required to run the neural network 330 is several orders of magnitude less than that required to train or develop the neural network.

[0070] By implementing a neural network 330 on the device 300, the device gains additional "intelligence". The device 300 is thus able to independently solve the desired task. This results in a cognitively extended device 300. Cognitively extended means that the device can semantically recognize and process image content or other data by using a neural network (or deep learning model) or other machine learning methods.

[0071] Additionally, the device 300 may include one or more components 340. For example, the one or more components 340 may include a user interface and / or an interface for downloading a neural network onto the device 300. In one embodiment, the one or more components 340 include an image recording system for acquiring images.

[0072] In some embodiments, the apparatus 300 can be used to train a neural network 330. To this end, the apparatus 300 can include means for determining a neural network for correcting optical imaging errors. The computer executable instructions stored on one or more computer readable storage media 320, when executed by one or more processors 310, can cause execution according to Figure 1 and / or Figure 4 one or some parts of a method.

[0073] Figure 4 A schematic flow chart of an embodiment of a (computer-implemented) method 400 according to the invention for determining a first neural network for correcting optical imaging errors is shown. The method 400 comprises a step 418 in which one or more images are determined in order to determine the first neural network with the aid of these images (step 420). Determining 418 the one or more images may comprise determining training images for training the first neural network. The determination of the training data or training images may comprise carrying out one or more measurements and / or one or more simulations in connection with one or more optical systems or one or more types (configurations) of optical systems and / or one or more samples in order to generate the training data. Alternatively, training data stored in a database or provided by a third party may be determined for the training. The training data comprises one or more image pairs, wherein the image pairs contain an input image with errors and a given output image without errors or with reduced errors.

[0074] In optional step 410, training may be prepared. In step 410, training conditions may be determined. This may include determining a second neural network as a loss function for a first neural network (the neural network to be trained). For example, this may be done by training the second neural network. Alternatively, the determination may include selecting a second neural network from a plurality of neural networks, wherein the plurality of neural networks have been trained for different sample types, and the second neural network is selected based on the sample type. The second neural network may have been trained or trained on a sample type that is relevant to a subsequent application purpose of the first neural network. The second neural network may be designed to make predictions, such as classification, based on images that are input values ​​to the second neural network.

[0075] In step 420, the first neural network is trained with the aid of the training data from step 418. During the training in step 420, internal parameters of the first neural network (e.g., weights "W" and thresholds "B") are found that allow a plurality of input images input into the first neural network to be optimally or as closely mapped to a given output image. Thus, the first neural network is able to generate new images from images and solve tasks associated with the training data. The first neural network may be trained to remove or reduce one or more optical imaging errors in the image, such that the input image has, for example, lower detail contrast or lower sharpness than the corresponding output image of the first neural network.

[0076] In step 426, the optical system is determined. In order to be able to clearly identify the optical system, an identification can be assigned to the optical system. For this purpose, a separate coding can be set for the optical system. This can be done during the manufacture of the optical system.

[0077] In step 428, individual training data is determined. In some embodiments, determining the individual training data includes determining the optical properties of the individual optical systems determined in step 426. This may include measurements and / or simulations of the optical systems. For example, an optical transfer function OTF may be determined for an individual optical system. This may be done during or after manufacture of the optical system. An optical system may be clearly assigned to an OTF with the aid of an identification of the optical system. In this case, individual training data may be generated based on the optical properties of the optical system for fine tuning. With knowledge of the OTF, image pairs may be generated as individual training data, since the OTF describes the optical imaging errors of the associated optical system.

[0078] In step 430, the first neural network determined in step 420 can be fine-tuned. Here, the first neural network is further trained to obtain an adaptive (third) neural network. The adaptive neural network can be trained for a specific application. For example, the first neural network can be further trained with the aid of the individual training data generated in step 428. Thus, the adaptive (third) neural network can be trained specifically for a separate optical system. Alternatively, the first neural network can also be further trained for a specific application purpose by using training data of a specific sample type (related to the specific application purpose) for fine-tuning.

[0079] During further training (fine tuning) in step 430, these individual training data may be used to further train at least a portion of the pre-trained (first) neural network. For example, only a portion of the internal parameters of the neural network may be changed according to the further training, wherein the remaining internal parameters cannot be changed by further training. This enables the neural network to be quickly and individually adapted to a specific application purpose, such as correcting optical imaging errors in images acquired by a specific optical system.

[0080] By fine-tuning the neural networks, these neural networks can be continuously improved and / or the application fields of the neural networks can be specified. This can be done advantageously by training only a few nodes in the neural network. For example, a slightly impaired optical system can affect the quality of images acquired with the impaired optical system. By fine-tuning and / or training the neural networks according to the method of the invention, a neural network can be determined for correcting these errors.

[0081] In another step 440, the adaptive neural network may be provided to a third party for use on a device associated with the optical system or stored in the cloud, on a server or another data storage.

[0082] A wide variety of optical aberrations that are related to the way optical systems are manufactured and operate can be corrected with the aid of machine learning (ML) methods. The described ML methods include algorithms with which machines can learn from experience and which can come from so-called "deep learning" (DL), a specific type of neural network. The effective correction of imaging errors enables the production of high-performance lenses in a more advantageous manner, since the imaging errors of lenses with significantly larger image errors, in particular larger fluctuations between individual samples of the same series, can be corrected. As a result, on the one hand, tolerances between individual lenses can be compensated by allowing a larger tolerance distribution. In addition, the corresponding lenses can be designed so that their imaging quality is optimized on the optical axis. The optical correction at the edge of the image field does not have to be as high, since it can be corrected by the neural network. For example, the imaging errors can also be evaluated when designing the lens according to the complexity of the correction by the lens or software post-processing (application of the neural network). Errors that are particularly easy to correct by ML / DL networks can be less important in the construction / design of the lens.

[0083] List of Reference Numerals

[0084] 110, 120, 140 training steps

[0085] 112 Methods and Steps

[0086] 114, 128, 144 Data memory

[0087] 116, 130, 150 Neural Networks

[0088] 122, 124 Methods and Steps

[0089] 126 Image Pairs

[0090] 136 Loss Function

[0091] 142 Methods and Steps

[0092] 152 Optical System

[0093] 154 Logo

[0094] 156 training data

[0095] 200 Neural Networks

[0096] 210 Images with optical imaging errors

[0097] 220 Corrected Image

[0098] 300 Devices

[0099] 310 processor

[0100] 320 Storage Media

[0101] 330 Neural Networks

[0102] 340 Components

[0103] 400 Methods

[0104] 410–440 Methods and Steps

Claims

1. A method for determining a neural network for correcting optical imaging errors, comprising the following steps: determining one or more images, wherein the one or more images are at least partially related to the optical system or the configuration of the optical system; determining a neural network based on the determined one or more images by applying the determined neural network to an image acquired by the optical system to output an image corrected for one or more optical imaging errors; wherein determining the neural network comprises: training the neural network; the neural network is a first neural network, and training the first neural network comprises: using a second neural network, wherein the second neural network is used as a loss function for training the first neural network; and wherein the loss function defines a feature recovery loss, which indicates a measure of the difference in image content between a target output image and a first neural network output image.

2. The method according to claim 1, characterized in that Determining the one or more images includes: determining a training image, the training image including a plurality of image pairs, Wherein, training the neural network comprises: training the training images, wherein each image pair in the plurality of image pairs comprises an input image for the neural network and a given output image.

3. The method according to claim 2, characterized in that The training images are generated from acquired images, wherein the acquired images are acquired by means of and / or using one or more optical systems, and wherein none, part or all of the one or more optical systems are of the same type as the optical system, and / or, Therein, the acquired images depict one or more types of samples.

4. The method according to claim 2, characterized in that The training image is generated by simulating optical imaging errors.

5. The method according to claim 2, characterized in that The method further comprises the step of determining at least one optical transfer function OTF or at least one point spread function PSF for the optical system or for some optical systems of at least partially the same type as the optical system, wherein the input images of the plurality of image pairs are generated by convolving an error-free image with one of the at least one point spread function PSF, wherein the error-free image is used as the corresponding given output image of the plurality of image pairs.

6. The method according to claim 2, characterized in that Training the neural network includes: training the training images and other data, wherein the other data includes at least one of the following: parameter data associated with the training images, verification data, measurement data related to the manufacture of the optical system, data about the experiment or measurement process, information about reagents and materials, information about the object or sample, information about the optical system, user-related data, user input and information of the image acquisition system.

7. The method according to claim 1, characterized in that The method further comprises the step of fine-tuning the determined neural network.

8. The method according to claim 7, characterized in that The fine-tuning comprises training only a portion of the determined neural network, wherein one or more parameters of the determined neural network remain unchanged during the fine-tuning, and / or, Wherein, the fine-tuning includes training specifically for the optical system, and / or, The fine-tuned neural network can be explicitly assigned to the optical system.

9. The method according to claim 7, characterized in that The method also includes the following steps: generating individual training data for the fine-tuning, wherein generating the individual training data includes: determining optical properties of the optical system, and / or acquiring measurement samples with the aid of the optical system, wherein the individual training data is generated based on the optical properties and / or the measurement samples.

10. The method according to claim 9, characterized in that The method comprises the steps of storing the measurement samples, the optical properties and / or the individual training data, and / or, Therein, the measurement samples, the optical properties and / or the individual training data are unambiguously assigned to the optical system.

11. The method according to any one of claims 1 to 10, characterized in that: capable of unambiguously and / or automatically identifying the optical system, and / or, The neural network can be unambiguously assigned to the optical system.

12. The method according to claim 11, characterized in that The optical system can perform identification by electromagnetic identification, optical identification, mechanical identification or magnetic identification.

13. The method according to any one of claims 1 to 10, characterized in that: The method comprises the following steps: The determined neural network is applied to acquired data, wherein the acquired data is acquired by means of the optical system or an optical system of the same type.

14. The method according to any one of claims 1 to 10, characterized in that: The one or more optical imaging errors include at least one of the following optical imaging errors: astigmatism, vignetting, coma, chromatic aberration, spherical aberration, and defocus.

15. The method according to claim 1, wherein: The determination of the neural network includes training the neural network with the aid of one or more images, wherein during the training the neural network learns how objects and / or structures ideally appear in the one or more images and corrects deviations therefrom.

16. The method according to any one of claims 1 to 10, characterized in that: The optical system is part of an imaging and / or image recording system of a microscope, a microscope system, a camera, a smartphone, a telescope, a mobile computer, a stationary computer or a measuring device.

17. A device for correcting optical imaging errors, comprising: one or more processors; One or more computer-readable storage media having computer-executable instructions stored thereon, which when executed by the one or more processors cause: acquiring one or more images by an imaging and / or image recording system, wherein one or more optical imaging errors in the one or more acquired images are associated with at least a portion of the imaging and / or image recording system; and, applying a neural network to the one or more acquired images, wherein the neural network is configured to generate one or more corresponding corrected images from the one or more acquired images, thereby correcting or reducing one or more optical imaging errors in the one or more corrected images; Wherein, the neural network is a first neural network, and training the first neural network includes: using a second neural network, wherein the second neural network is used as a loss function for training the first neural network; and wherein the loss function defines a feature recovery loss, which indicates a measure of the difference in image content between a target output image and a first neural network output image.

18. The device according to claim 17, characterized in that storing one or more acquired images and applying the neural network to the one or more stored acquired images, or, The neural network is applied directly to one or more acquired images and only the one or more rectified images are stored.

19. The device according to any one of claims 17 and 18, characterized in that At least a part of the imaging and / or image recording system comprises an optical system, a camera layer, a sCMOS or CCD sensor or one or more scattering slides.

20. A computer program comprising instructions which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 10.

Images