Resizing images using a neural network

Description

[0001] The invention relates to a method and a device for providing a neural network for scaling images.

[0002] When an image capture system, such as a digital camera, captures an image, it can produce a digital image with a specific resolution or pixel resolution. The digital image can be thought of as a number of columns (width) and rows (height) of a raster graphic. The information content in images is limited by the resolution, or the total number of pixels in an image. For certain applications, it may be advantageous to adjust or change the size of an image.

[0003] Existing systems scale images or change the size of an image using interpolation. When digital images are enlarged, values ​​for the image brightness of new pixels that lie between existing pixels must be calculated. During interpolation, the existing pixels are used as boundary values, and the intermediate values ​​in the pixel grid are calculated according to a defined mathematical function. Well-known mathematical methods are bilinear, biquadratic, or bicubic interpolation. However, all of these methods for calculating new intermediate values ​​only use image information from the one existing image. The resulting image enlarges all image elements and objects in the image proportionally correctly, but does not reproduce any new image details. This means that no new image details are visible that might become visible in a higher-resolution image of an object.Therefore, when processes work with interpolation, the resulting image impression appears washed out.

[0004] To circumvent this problem, recent state-of-the-art methods use techniques based on internal image similarities, such as those presented by Freedman, G., and Fattal, R. in the journal ACM Transactions on Graphics (TOG) Volume 30 Issue 2, (April 2011) "Image and Video Upscaling from Local Self-Examples " described, or similarities with image elements of external image pairs from low and high resolution images, as for example by D. Dai, R. Timofte, and L. Van Gool in Computer Graphics Forum, Volume 34, pages 95-104 (2015) "Jointly Optimized Regressors for Image Super-resolution " described, to upscale images.

[0005] These state-of-the-art methods exhibit several disadvantages and problems, which will be discussed below. For example, these methods are often domain-specific, and the scaling cannot be easily transferred to new application areas involving previously unknown objects in images. Furthermore, existing microscopes or image acquisition systems use image upscaling methods only as a means of image post-processing. This limits the usability of the upscaled images. Furthermore, state-of-the-art methods rarely achieve scaling factors greater than 2.0.

[0006] VAIR RIVENSON ET AL: "Toward a Thinking Microscope: Deep Learning in Optical Microscopy and Image Reconstruction", CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, shows a deep learning method and its application for optical microscopy and microscopic image reconstruction.

[0007] VAIR RIVENSON ET AL: "Deep Learning Microscopy", CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, discloses a deep neural network for optical microscopy to improve spatial resolution over a large field of view and depth of field.

[0008] YAIR RIVENSON ET AL: "Deep learning enhanced mobile-phone microscopy", CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, discloses the use of deep learning to correct distortions caused by mobile phone-based microscopes to produce high-resolution, denoised, and color-corrected images.

[0009] US 2018 / 342044 A1 discloses a device that receives first image data with a first resolution and second image data with a resolution lower than the first resolution. The second image data can be scaled to the first resolution and compared with the first image data. The application of a neural network can scale the first image data to a resolution higher than the first resolution.

[0010] The object of the present invention is therefore to provide improved means for scaling images.

[0011] The present invention arises from the appended claims and solves the addressed problems and the object by means of a device for scaling images. The device according to the invention comprises one or more processors and one or more computer-readable storage media, wherein computer-executable instructions are stored on the one or more computer-readable storage media, which, when executed by the one or more processors, cause one or more first images to be acquired by means of an imaging and / or image recording system and a neural network to generate one or more corresponding second images depending on the one or more acquired first images. The one or more acquired first images are associated with a first resolution, and the one or more second images are associated with a second resolution that differs from the first resolution.

[0012] The method according to the invention comprises the steps of determining a plurality of image pairs, and training a neural network as a function of the determined plurality of image pairs such that the trained neural network, when applied to a first image, outputs a second image, wherein the resolution of the first image differs from the resolution of the second image, wherein input images of the plurality of image pairs were captured with a first imaging and / or image recording system and desired output images of the image pairs corresponding to the input images were captured with a second imaging and / or image recording system, and wherein the first imaging and / or image recording system differs from the second imaging and / or image recording system.

[0013] The method and device according to the invention have the advantage that, by means of the related neural networks, e.g., in the sense of deep learning, intelligent scaling of digital images in real time or near real time during image acquisition is enabled. Furthermore, the method and device according to the invention enable the pixel resolution of a captured image to be increased. The missing image information can be generated by a neural network pre-trained with a large number of images, for example, with many thousands or even millions of images. The missing image information is therefore not simply interpolated from the single available image, but rather generated based on learned image semantics.

[0014] By using a neural network, an image is generated that contains more detail and / or more image information than the original image. Thus, the image generated using a neural network is more detailed than the original image. This can facilitate the interpretation of the image by a human or an algorithm. Furthermore, neural networks enable better generalizability for previously unknown applications.

[0015] The method and device according to the invention can each be further improved by specific embodiments. Individual technical features of the embodiments of the invention described below can be combined with one another as desired and / or omitted, provided that the technical effect achieved by the omitted technical feature is not important.

[0016] In one embodiment, the device according to the invention is designed to apply the neural network directly to each of the one or more acquired first images and to save only the one or more second images. By reducing the first resolution of the one or more acquired first images before saving, storage space can be saved. Alternatively, the one or more first images acquired with an image acquisition system, such as a microscope with a camera, a video camera, a microscope system, a microscope with a point scanner, a digital camera, a smartphone, a telescope, a measuring device or a computer with a camera, can be saved and the neural network applied to the one or more stored first images. Neural networks have the advantage that images input to the neural network only pass through it once.No optimization problem needs to be solved to generate an upscaled image. Thus, the higher-resolution image can be calculated quickly and efficiently from the low-resolution images stored.

[0017] In embodiments, the one or more processors may include computational accelerators, such as graphical processing units (GPUs), field-programmable gated arrays (FPGAs), TensorFlow processing units, or tensor processing units (TPUs), application-specific integrated circuits (ASICs) specialized for machine learning (ML) and / or deep learning (DL), or at least one central processing unit (CPU). This allows images to be processed quickly, and image scaling may be part of the image acquisition process.

[0018] The device according to the invention can further be advantageously configured in that the computer-executable instructions, when executed by the one or more processors, further cause the imaging and / or image recording system to be adjusted depending on the first resolution. The adjustment can comprise adjusting at least one of the following settings: a frame rate for acquiring the one or more first images, a transmission rate of the one or more acquired first images from the imaging and / or image recording system to a workstation, the device, or a data storage device, binning, pixel selection from all pixels of the imaging and / or image recording system for acquiring the one or more first images, an exposure time, a scan format, or a pixel resolution for acquiring the one or more first images.wherein the one or more first images comprise one or more scanned images, a scanning frequency (e.g., a line scan frequency) of the imaging and / or image recording system, wherein the imaging and / or image recording system comprises a scanning imaging and / or image recording system, a numerical aperture, NA, of an optical system of the imaging and / or image recording system, and a light intensity of a light source of the imaging and / or image recording system. A further advantage is that images can be acquired and subsequently resized to a desired size with a smaller pixel resolution, for example, fewer image lines with a scanning point detector (scanner) or more binning with an area detector of the image acquisition system. This enables a high acquisition speed for a measurement and / or the use of a sample-protecting or small light dose during an acquisition.

[0019] In advantageous embodiments, the first resolution of the image acquisition system can be adjusted. For example, adjusting the first resolution can comprise adjusting a binning and / or selecting a subset of pixels of the imaging and / or image recording system for capturing the one or more first images. By reducing the first resolution of the imaging and / or image recording system, the imaging and / or image recording system is configured to reduce an exposure time for capturing the one or more first images by means of binning and / or selecting a subset of pixels, to reduce a light intensity of a light source of the imaging and / or image recording system by means of binning, and to increase a transmission rate of the one or more first images by means of binning and / or selecting a subset of pixels.To increase a frame rate for capturing the one or more first images by means of binning and / or selecting a subset of pixels, To increase a maximum number of storable images or to generate more storage space for the one or more captured images by means of binning and / or selecting a subset of pixels, To increase a recording speed for capturing the one or more first images by means of binning and / or selecting a subset of pixels, and / or To store a larger number of the one or more captured first images by means of binning and / or selecting a subset of pixels. For example, scanning imaging and / or image recording systems can reduce the exposure time by selecting a subset of pixels, as the number of sampling points is reduced, thus exposing the sample to light for a shorter period of time.

[0020] The device according to the invention, which can be part of an image recording system, such as a microscope, can be designed to shorten the recording time of a measurement if fewer image lines are scanned or pixels are recorded, to reduce the storage space for the image data if only smaller images are stored, since the high-resolution images can be reconstructed, to reduce the light dose for illuminating an object, and to increase the resolution of captured images.

[0021] In one embodiment of the method according to the invention, image pairs for training the neural network are acquired using a second and a first imaging and / or image recording system, wherein the second imaging and / or image recording system is designed to acquire images with a resolution that is higher than the resolution of the first imaging and / or image recording system. By training with image pairs created using different techniques, such as diffraction-limited microscopy and nanoscopy with super-high optical resolution, upscaled images with a higher detail contrast can be generated. If the neural network has been trained in a context with the one or more acquired first images, image information can be generated semantically correctly in the one or more second images.

[0022] In embodiments, training the (first) neural network may include applying a second neural network. The second neural network may be applied as a loss function for training the first neural network. This enables improved training of the first neural network, since a neural network as a loss function enables accurate training, ensuring that the output of the neural network resembles a desired image. This cannot be guaranteed if an error in the output image is only calculated per pixel. Thus, the output images are not treated as a set of independent pixels, but rather placed in a semantic context.

[0023] In advantageous embodiments, the first neural network can be further trained (fine-tuned). Fine-tuning (hereinafter also referred to as fine-tuning) can involve training only a portion of the specific (first) neural network. In this case, one or more parameters of the specific neural network can remain unchanged during fine-tuning. Fine-tuning can be performed using individual training data.

[0024] The present invention will be described in more detail below with reference to exemplary drawings. The drawings show examples of advantageous embodiments of the invention.

[0025] They show: Figure 1 Images with different resolutions, Figure 2 a schematic representation of a method according to the invention for scaling images by means of a neural network according to one embodiment, Figure 3a schematic representation of a method according to the invention for scaling images by means of a neural network according to one embodiment, Figure 4A a schematic representation of a device according to the invention according to one embodiment, Figure 4B a schematic representation of an inventive system for scaling images according to one embodiment, Figure 5A a schematic flow diagram of an embodiment of the method according to the invention for scaling images, and Figure 5B a schematic flow diagram of an embodiment of the inventive method for training a neural network.

[0026] Figure 1shows an image 100 with a first (low) resolution, for example, 73x75 pixels, and images 110, 120, and 130 with a second (high) resolution, for example, 250x300 pixels. Images 100, 110, 120, and 130 show the same object: stained vertebrate cells. Images 100 and 120 may have been acquired using the same imaging and / or image recording system, such as a microscope (Leica TCS SP 5). Alternatively, images 100 and 120 may have been acquired using different image recording systems. The different resolutions of images 100 and 120 result in a difference in the information content of the two images, with image 120 with the high resolution having more information content than image 100 with the low resolution. For example, more details of the same object can be seen in image 120 than in image 100.The low resolution at which the first image was captured may be the maximum resolution of an image acquisition system for a single shot, or the resolution of the image recording system set for a specific shot or specific conditions. For example, a higher maximum frame rate can be achieved with a lower resolution. However, the lower resolution results in the loss of image information.

[0027] In some applications, it may be advantageous to upscale image 100. Images 110 and 130 are upscaled images generated from image 100. Image 110 is an image upscaled using bicubic interpolation, and image 130 was generated using the inventive method for scaling images according to one embodiment. The resolution of image 100 limits the information content in image 110 upscaled using bicubic interpolation, since the bicubic interpolation merely smooths and does not add any new information. As shown in Figure 1 As can be seen, the image 130 upscaled using the method according to the invention provides significantly better results than the interpolated image 110 in comparison to the high-resolution measured image 120.

[0028] Figure 2shows a schematic representation of the functionality of a neural network 200 for scaling images. The neural network 200 is configured to change the resolution of an image 210 by generating an image 220 with a changed resolution. For example, the neural network 200 can output an image 220 with a resolution twice as large, four times as large, or m / n times as large as the input image 210 of the neural network, where m and n are natural or positive integers. In embodiments, individual dimensions of an image can be scaled.

[0029] The input image 210 for the neural network 200 may include a first resolution. When the neural network 200 is applied to image 210 or this image 210 is input into the neural network 200, the neural network 200 may map the image 210 onto the output image 220, wherein the number of pixels of the image 210 does not have to match the number of pixels of the image 220. Thus, the image 210 may be scaled up or down. Figure 2the neural network 200 is configured to generate image 220 from an image 210, wherein the image 220 has a higher resolution and / or contains more image information than the input image 210. The neural network can be configured to generate missing image information in image 210 semantically correctly from the neural network. The neural network 200 can have been trained on training data that is in context with the acquired images. The difference from conventional methods, such as bicubic interpolation, is that the generated image information does not originate only from the one available image, but is generated from a plurality of images taking into account learned image representations. In this way, semantically correct image information that is not contained in the available low-resolution image is supplemented.

[0030] The application of the neural network 200 differs from the training of the neural network 200 in the data sets used. During training, one or more initial images are input into the neural network 200, and internal parameters of the neural network are adjusted so that the output images of the neural network 200 match the target output images as closely as possible. During application of the neural network, the image data passes through the neural network 200 once, and the neural network 200 generates an output image as a prediction. Neural networks can represent results learned through at least one deep learning process and / or at least one deep learning method. These neural networks condense collected knowledge into a specific task ensemble in a suitable manner through automated learning, such that a specific task can then be performed automatically and with the highest quality.

[0031] Image 210 may be an image captured or recorded with an image acquisition system. In one embodiment, the captured image 210 may be preprocessed before being fed into the neural network 200. The preprocessing may include interpolating, such as bilinear, biquadratic, or bicubic interpolation, the image 210. For example, the image 210 may be interpolated to the desired output size of the neural network. This corresponds to the image size of image 220.

[0032] The neural network can then be applied to the interpolated image, producing an output image. The output image and the interpolated image differ in their information content, detail contrast, and / or sharpness. For example, the information content, detail contrast, and / or sharpness of the output image may be increased compared to the interpolated image. Alternatively, preprocessing can also be part of the neural network.

[0033] In Figure 3The principle of upscaling is shown schematically. A first image 300 with a first (low) resolution of 11x12 pixels comprises a pixel 330 with a value of 68. The first image 300 can be sampled and scaled to a desired size. For example, resulting gaps can be filled with the pixel value zero, resulting in a second image 310 with the same resolution or size as the desired image 320. In a next step, a convolution matrix 340 can be applied to the image 310. Figure 3For example, the convolution matrix 340 has a size of 3x3 pixels. Convolution of the convolution matrix 340 with the image 310 can result in the image 320. This corresponds to a step size of approximately 1 / 2 for upscaling. The neural network for scaling images can have a plurality of layers, with the step size of the convolutions in the layers being able to vary. For example, a step size of 1 for a convolution generates a new image with a size corresponding to the size of the input image, a step size greater than 1 generates a new image with a size smaller than the size of the input image, and a step size less than 1 generates a new image with a size larger than the size of the input image.

[0034] The parameters of the convolution matrix are learned by the neural network during training. This allows the neural network to generate higher-resolution output images when applied to input images, with missing image information from the input image being generated by the neural network. The neural network can learn hidden representations through training with a large number of training images. This allows semantically correct images with higher resolution to be generated from low-resolution images.

[0035] Figure 4Ashows a device 400 comprising one or more processors 410 and one or more storage media 420. The device 400 may comprise an imaging and / or image recording system. Alternatively, the device 400 may be physically separated from an imaging and / or image recording system and connected to the imaging and / or image recording system via a network, for example, a wireless network. In this case, the device may comprise a workstation, a server, a microcomputer, a computer, or an embedded computer.

[0036] The one or more processors 410 can include computing accelerators, such as graphical processing units (GPUs), TensorFlow processing units, or Tensor processing units (TPUs), application-specific integrated circuits (ASICs) or field-programmable gated arrays (FPGAs) specialized for machine learning (ML) and / or deep learning (DL), or at least one central processing unit (CPU). An application-specific integrated circuit (ASIC, also known as a custom chip) is an electronic circuit that can be implemented as an integrated circuit. Because their architecture is adapted to a specific problem, ASICs operate very efficiently and considerably faster than a functionally equivalent implementation in software in a microcontroller. TPUs, also known as TensorFlow processors, are application-specific chips and can accelerate machine learning applications compared to CPUs.This or similar specialized hardware can be used to optimally solve deep learning tasks. In particular, the application of a neural network, which requires orders of magnitude less computing power than training, i.e., the development of a model, also works on conventional CPUs.

[0037] Furthermore, in embodiments, the device may include one or more neural networks 430. With the help of the one or more neural networks 430, the device 400 may be enabled to scale images using artificial intelligence (AI). The one or more images may include single images, 3D image stacks, videos, images from multidimensional time series, and / or different scales. The one or more neural networks 430 may be executed by the one or more processors 410. Executing neural networks 430 requires orders of magnitude less computing power than training or developing a neural network.

[0038] By implementing the neural network 430 on the device 400, the device gains additional "intelligence." The device 400 can thus be enabled to solve a desired task independently. This results in a cognitively enhanced device 400. Cognitively enhanced means that the device can be enabled to semantically recognize and process image content or other data through the use of neural networks (e.g., deep learning models) or other machine learning methods.

[0039] Furthermore, the device 400 may include one or more components 440. For example, the one or more components 440 may include a user interface, an interface for downloading neural networks to the device 400, or an image acquisition system.

[0040] In embodiments, the device 400 may be used to train a neural network 430. Computer-executable instructions stored on the one or more computer-readable storage media 420, when executed by the one or more processors 410, may cause one or more portions of the methods of the Figure 5A and / or 5B.

[0041] Figure 4Bshows a system 450 comprising an imaging and / or image recording system 460 and a computer 470, for example, a workstation, connected via a network, for example, a wireless network or a fiber optic network. In embodiments, the imaging and / or image recording system 460 and / or the workstation 470 may comprise the device 400. Thus, the image recording system 460 and / or the workstation 470 may be configured with artificial intelligence (AI). Alternatively, the device 400 may be spatially separated from the imaging and / or image recording system 460 and / or the workstation 470 and connected to them via a network.

[0042] The imaging and / or image recording system 460 may include an optical system, such as an objective, optics, or individual lenses, and a detection system, such as a photographic layer, an sCMOS ("scientific complementary metal-oxide-semiconductor"), or CCD ("charge-coupled device") sensor. For example, the imaging and / or image recording system 460 may include a microscope with a camera, a video camera, a microscope system, a microscope with a point scanner, a digital camera, a smartphone, a telescope, a measuring device, or a computer with a camera. Figure 4B1 shows image recording system 460 as a light microscope only by way of example. In embodiments, image recording system 460 may include all types of microscopes. For example, a microscope may include one of the following: a light microscope, a stereo microscope, a confocal microscope, a multiphoton microscope, a STED microscope, a slit lamp microscope, a surgical microscope, a digital microscope, a USB microscope, an electron microscope, a scanning electron microscope, a specular microscope, a fluorescence microscope, a focused ion beam microscope (FIB), a helium-ion microscope, a magnetic resonance microscope, a neutron microscope, a scanning SQUID microscope, an X-ray microscope, an ultrasonic microscope, a light sheet microscope (SPIM), or an acoustic microscope, etc.

[0043] Figure 4BIn one embodiment, FIG. 1 shows communication between an AI-enabled microscope 460 and an AI-enabled computer 470. A single image recording system can itself include hardware acceleration and / or a microcomputer, such as device 400, that enables the execution of trained models (e.g., neural networks). Alternatively, only the image recording system 460 or only the computer 470 can be AI-enabled.

[0044] In embodiments, the image acquisition system 460 is configured to acquire one or more images. The acquired images may be stored locally on the image acquisition system 460 or physically separated from the image acquisition system 460. For example, the image acquisition system 460 may send the images to the workstation 470 or to the device 400 via a network, where the images are then stored. The stored images may then be fed into a neural network for image scaling. Alternatively, the acquired images may be fed directly into the neural network, and only the corrected images may be stored. This enables real-time scaling of the images.In one embodiment, for a large number of images or a very large batch of data, upscaling can also be performed asynchronously on a cluster in the network or in a cloud.

[0045] In one embodiment, the processing of the acquired images takes place on computer 470, which runs software implementing a neural network capable of intelligently scaling images. Missing image information can be semantically correctly generated from hidden representations present in the neural network through learning. Thus, system 450 can capture images or measurements at higher speeds and / or perform data reduction. The required storage space or memory bandwidth can be reduced.

[0046] The speed gain is based on the fact that fewer pixels need to be acquired than actually necessary to achieve the desired resolution. Depending on the light sensor of the image acquisition system, a distinction can be made between point detectors and area detectors. Point detectors are used in microscopy, primarily in confocal laser scanning microscopes. With laser scanning, the image is created by scanning a sample line by line. The image acquisition time depends on the number of scanned lines. If, for example, the number of lines is reduced to a quarter, the acquisition speed can be approximately quadrupled. Area detectors can be used in wide-field microscopy. Here, a speed advantage can be achieved through binning, i.e. grouping pixels together. This reduces the resolution and exposure time, which increases the acquisition speed.

[0047] Regardless of the acquisition technique used, the storage size of an uncompressed two-dimensional image is quadratically related to the resolution in each spatial direction. If the desired image content can be reconstructed through intelligent upscaling, it is possible to downscale the images accordingly before saving. In one embodiment, a neural network can be used to intelligently upscaling images in three spatial dimensions and the time dimension, thereby reducing the amount of data to be stored. The storage saving S is given by: S = ß d< . Here, S is the factor of storage savings compared to full resolution, ß is the isotropic binning in each spatial direction, and d is the dimension of the image. For a 3D stack with an isotropic binning of ß = 4, for example, this results in a storage reduction of S = 64.

[0048] Microscopy often involves the use of light-sensitive samples or dyes that can be photochemically decomposed by the application of high light doses. This is also known as photobleaching. Stronger binning in an area detector, or a reduction in pixel resolution by the binning factor in a point detector, exposes the sample to light for a shorter period of time. Binning thus has the effect of reducing the light dose. By reducing the light dose and increasing the acquisition speed, the effect of photobleaching can be reduced, expanding the range of applications of microscopes.

[0049] If the maximum useful resolution is present during the acquisition, i.e., the pixel size is approximately two to three times smaller than the corresponding point spread function, increasing the number of pixels no longer increases the image information. Intelligent upscaling can result in an image containing more image information in this limit range. For this purpose, the neural network or trained model used can be trained in such a way that the input images in the training dataset include diffraction-limited images, while the images available as the target value at the model's output (target output images) show the same location or object captured using an optical super-resolution method. Through training, the neural network can then simulate this optical super-resolution method.In embodiments, for example, wide-field microscopy can be used to generate a low-resolution image as the input image for training, and confocal microscopy can be used to generate a high-resolution image as the corresponding target output image. Alternatively, confocal microscopy could be used to generate a low-resolution image as the input image for training, and stimulated emission depletion (STED) nanoscopy could be used to generate a high-resolution image as the corresponding target output image.

[0050] Figure 5Ashows a schematic flowchart according to an embodiment of an inventive (computer-implemented) method 500 for scaling images. The method 500 may include a first step 510 in which an imaging and / or image recording system is set. For example, a resolution with which the imaging and / or image recording system captures one or more images may be set. This may involve binning (combining pixels) or selecting specific pixels for capture.Furthermore, a frame rate for capturing the one or more first images, a transmission rate of the one or more captured first images from the imaging and / or image recording system to a computer or a workstation, the device or a data storage device, an exposure time, a scan format or a pixel resolution for capturing the one or more first images, wherein the one or more first images comprise one or more scanned images, a scanning frequency (for example a line scanning frequency) of the imaging and / or image recording system, wherein the imaging and / or image recording system comprises a scanning imaging and / or image recording system, a numerical aperture, NA, of an optical system of the imaging and / or image recording system and / or a light intensity of a light source of the imaging and / or image recording system can be set.The sampling frequency can be related to the speed at which a light source (e.g., a laser) of the imaging and / or image recording system scans an object. The scan format can specify a spatial resolution (e.g., 1024 x 1024 or 512 x 512) during scanning. The settings of the imaging and / or image recording system can be related to the resolution. For example, a frame rate can be increased by reducing the resolution, or binning (reducing the resolution) can reduce the light intensity and / or exposure time of a light source of the imaging and / or image recording system.

[0051] In a second step 520, one or more first images can be captured using an imaging and / or image recording system. The one or more captured first images are associated with a first resolution.

[0052] In a third step 530, a neural network generates, for example, in real time, one or more corresponding second images depending on the one or more acquired first images. The one or more second images are associated with a second resolution, wherein the first resolution and the second resolution differ. The neural network can be applied directly to each of the one or more acquired first images, and only the one or more second images can be stored.

[0053] In one embodiment, the neural network upscales a low-resolution image, and the upscaled image is sent to a computer for further processing. Alternatively, the low-resolution image can be sent to a computer and upscaled on the computer using the neural network.

[0054] Neural networks can be trained using deep learning methods. This involves the systematic application of at least one deep learning method, but preferably several deep learning methods, to achieve a specific goal. The goal can include image processing (e.g., correcting one or more optical defects, generating an image from another image where at least one feature differs in the images, etc.). Deep learning methods can comprise a sequence of process steps that divide a process into comprehensible steps, in such a way that this process is repeatable. The process steps can be specific deep learning algorithms. They can also be methods with which a network learns (e.g., backpropagation), the type of data collection, the way data is processed via hardware, etc.

[0055] Figure 5Bshows a schematic flow diagram of a computer-implemented method 550 for training a neural network for scaling images. In a first step 560, image pairs are determined as training data in order to train a neural network based on these image pairs (step 570). Determining the training data or training images can include one or more measurements and / or one or more simulations related to one or more sample types to generate the training data. Alternatively, training data stored in a database or provided by third parties can be determined for training. The image pairs include input images and target output images.

[0056] Loss functions based on per-pixel loss are error-prone and can produce imprecise training results. To address these drawbacks, embodiments may use loss functions that determine perceptual and semantic differences between images. Therefore, a second neural network may be determined as the loss function for the first neural network (the neural network to be trained). For example, this may be done by training the second neural network. Alternatively, determining the second neural network may include selecting the second neural network from a plurality of neural networks, wherein the plurality of neural networks have been trained on different sample types, and the second neural network is selected depending on the sample type.The second neural network can be trained or trained on a sample type related to the intended use of the first neural network. The second neural network can be configured to make predictions, such as classification, based on images as input to the second neural network. The loss network can thus determine a perceptual loss, which can be used to ensure that the neural network's output resembles an expected image. This is not the case with loss functions that only calculate an error per pixel of an image. Thus, the output images are not treated as a set of independent pixels, but rather placed in a semantic context.

[0057] In step 520, the first neural network is trained using the training data from step 510. During training in step 520, internal parameters (e.g., weights / filters "W" and thresholds "B") of the first neural network are found that optimally or best map a plurality of input images fed into the first neural network to the target output images. Thus, the first neural network is capable of generating new images from images and solving a task related to the training data. The first neural network can be trained to scale images.

[0058] Furthermore, the first neural network trained in step 520 can be fine-tuned. The first neural network is further trained to obtain an adapted (third) neural network. The adapted neural network can be trained for a specific application. For example, the first neural network can be further trained using special training data. Thus, the adapted (third) neural network can be further trained specifically for a specific application by using training data of a specific sample type (which is in a context with the specific application) for fine-tuning. During further training (fine-tuning), this special training data can be used to further train at least part of the pre-trained (first) neural network.For example, only some of the neural network's internal parameters can be changed depending on further training, while the remaining internal parameters cannot be changed by further training. This allows for rapid individual adaptation of the neural network to a specific application, such as upscaling images depicting a specific sample type.

[0059] By fine-tuning neural networks, they can be continuously improved and / or the application area of ​​the neural networks can be specified. This can be advantageously achieved by training only a few nodes in a neural network.

[0060] The adapted neural network can then be made available to third parties, applied to a device associated with the optical system, or stored in a cloud, on a server, or other data storage device.

[0061] Images with a certain resolution can be scaled using machine learning (ML) methods. The ML methods described include algorithms that allow machines to learn from experience and can come from so-called "deep learning" (DL), a specific type of neural network.

[0062] The neural network is trained to transfer objects in a captured image to an upscaled image in such a way that the object is correctly reconstructed "implicitly"—that is, as part of the learned parameters—in the network. The neural network may have seen similar objects during training. "Similar" here means that the same image features were present in the training images as in the images to be scaled, thus the training images are in context with the captured images.

[0063] The device and method according to the invention provide means for scaling images. The device according to the invention is designed to scale one or more captured images using a neural network. This can advantageously be done during or directly after the images are captured, i.e., before the image is saved ("in real time"). Image scaling is thus part of the image acquisition process. Reference symbol:

[0064] 110, 120, 130, 140Images 210, 220Images 200Neural network 300, 310, 320Images 330, 350Pixels 340Convolutional matrix 400Device 410Processor 420Storage medium 430Neural network 440Components 450System 460Imaging and / or image recording system 470Computer 500, 550Procedure 510 - 530, 560, 570Procedure steps

Claims

1. Device (400) for scaling images, comprising: one or more processors (410); one or more computer-readable storage media (420) on which computer-executable instructions are stored, which, when executed by the one or more processors (410), lead to: setting a first resolution, wherein setting the first resolution comprises setting a binning and / or selecting a subset of pixels of a microscope for capturing the one or more first images (100; 210); adjusting the microscope based on the first resolution, wherein adjusting the microscope comprises adjusting at least one of the following: a numerical aperture, NA, of an optical system of the microscope, an exposure time, and a light intensity of a light source of the microscope; capturing one or more first images (100; 210) by means of the microscope, wherein the one or more captured first images (100; 210) comprise the first resolution; and applying a neural network (200; 430) to the one or more captured first images (100; 210) to generate one or more corresponding second images (130; 220), wherein the one or more second images (130; 220) comprise a second resolution, wherein the first resolution and the second resolution differ, wherein the one or more captured first images (100; 210) have a lower resolution than the one or more second images (130; 220).

2. Device (400) according to claim 1, characterized in that, by reducing the first resolution, the microscope is configured - to reduce an exposure time for capturing the one or more first images (100; 210) by means of binning and / or a selection of a subset of pixels, - to reduce a light intensity of a light source of the microscope by means of binning, - to increase a transmission rate of the one or more first images (100; 210) by means of binning and / or a selection of a subset of pixels, - to increase a frame rate for capturing the one or more first images (100; 210) by means of binning and / or a selection of a subset of pixels, - to increase a maximum number of storable images by means of binning and / or a selection of a subset of pixels, - to increase a recording speed for capturing the one or more first images (100; 210) by means of binning and / or a selection of a subset of pixels, and / or - to store a larger number of the one or more captured first images (100; 210) by means of binning and / or a selection of a subset of pixels.

3. Device (400) according to any one of claims 1 and 2, characterized in that the one or more captured first images (100; 210) are stored and the neural network (200; 430) is applied to the one or more stored first images (100; 210).

4. Device (400) according to claim 3, characterized in that the first resolution of the one or more captured first images (100; 210) is reduced before storing.

5. Device (400) according to any one of claims 1 to 4, characterized in that the microscope comprises a microscope with a camera, a video camera, a microscope system, a microscope with a point scanner.

6. Device (400) according to any one of claims 1 to 5, characterized in that the one or more processors (410) comprise computing accelerators, such as graphical processing units, GPUs, or field-programmable gated arrays, FPGAs, tensor processing units, TPUs, application-specific integrated circuits, ASICs, specialized in machine learning, ML, and / or deep learning, DL, or at least one central processing unit, CPU.

7. Device (400) according to any one of claims 1 to 6, characterized in that the neural network (200; 430) has been trained in a context with the one or more first images (100; 210) and the one or more second images contain image information that has been generated semantically correctly from the neural network (200; 430).

8. Device (400) according to any one of claims 1 to 7, characterized in that the neural network (200; 430) has been trained on image pairs, wherein input images of the image pairs have been captured with a first imaging and / or image recording system (460) and target output images of the image pairs, corresponding to the input images, have been captured with a second imaging and / or image recording system (460), wherein the first imaging and / or image recording system differs from the second imaging and / or image recording system.

9. Device (400) according to claim 8, characterized in that the second imaging and / or image recording system is configured to capture images with a higher resolution than the first imaging and / or image recording system.

10. Device (400) according to any one of claims 1 to 9, characterized in that the computer-executable instructions, when executed by the one or more processors (410), further lead to the one or more captured first images (100; 210) being pre-processed, wherein the preprocessing comprises interpolating the one or more captured first images (100; 210) to the second resolution and applying the neural network (200; 430) to the one or more interpolated first images (100; 210), or applying the neural network (200; 430) to the one or more captured first images (100; 210).

11. System 450 for scaling images, comprising: one or more devices (400) according to any one of claims 1 to 10; and the microscope, wherein the microscope is configured to capture the one or more first images (100; 210).