Method for training an image processing system with a machine learning model to perform a virtual multi-angle reconstruction of image stacks acquired with a light-sheet microscope.

Abstract

Method for training an image processing system with a machine learning model to perform a virtual multi-angle reconstruction of image stacks acquired with a light-sheet microscope, in particular of a sample of a sample type, comprising: - Taking at least one light sheet fine stack of a sample, wherein the light sheet fine stack comprises several image stacks and for different of the image stacks a light sheet illuminates the sample at a different angle, - Determining target outputs based on the light-sheet fine stack, the machine learning model, and a classical multi-angle reconstruction, - Determining training inputs based on a rough stack of light sheets, the machine learning model, and classical multi-angle reconstruction, where the rough stack of light sheets comprises one or more image stacks, and for different image stacks, the light sheet illuminates the sample at various angles. - Creating an annotated dataset encompassing the target outputs and learning inputs, - Optimizing the machine learning model to perform the virtual multi-angle reconstruction using the annotated dataset, characterized in that - the rough stack of light sheets contains fewer images than the fine stack of light sheets, in particular fewer image stacks, and a virtual reconstruction determined by means of virtual multi-angle reconstruction has fewer image artifacts than a reconstructed image stack determined by means of classical multi-angle reconstruction from the rough stack of light sheets.

Description

Technical background

[0001] The invention relates to a method for training an image processing system with a machine learning model to perform a virtual multi-angle reconstruction of image stacks recorded with a light-sheet microscope, a method for performing a virtual multi-angle reconstruction, an image processing system for performing the method, and a computer program product. State of the art

[0002] Light sheet fluorescence microscopy (LSFM) is an optical fluorescence microscopy technique used for fast, high-resolution imaging of biomedical samples. It reduces the negative effects of photobleaching and light-induced stress on the samples because only a thin layer of the sample is illuminated by a single light sheet. Because the light sheet is so thin, sample structures with high absorption or scattering, or low spectral transmittance, cause image artifacts in LSFM samples, particularly fringe artifacts (also called shadows). These artifacts degrade image quality, especially in sample areas located behind these structures relative to the light sheet's propagation direction.

[0003] Various methods have been developed in the prior art to eliminate stripe artifacts. These include both hardware-based and software-based solutions. For example, Dong, D. et al. describe a method in “Vertically scanned laser sheet microscopy”, J. Biomed. Opt. 19(10), 1 (2014) in which a light sheet of a unidirectional light-sheet microscope is shifted along a direction parallel to the light sheet and perpendicular to the propagation direction of the light sheet, thereby eliminating stripe artifacts stationary relative to the optics.

[0004] Dodt, HU et al describe in “Ultramicroscopy: three-dimensional visualization of neuronal networks in the whole mouse brain”, Nat. Methods 4(4), 331-336 (2007), a bidirectional light-sheet microscope in which two parallel light sheets illuminate the sample from opposite sides in order to reduce the fringes.

[0005] Huisken, J. et al. describe in “Even fluorescence excitation by multidirectional selective plane illumination microscopy (mSPIM)”, Opt. Lett. 32(17), 2608-2610 (2007) a method in which a sample is illuminated from several different directions and the resulting images are then averaged to eliminate fringes or shadows. The results are very promising; however, the multi-angle illumination of the sample also increases the light exposure, which is why this method cannot be applied to all samples, depending on their light sensitivity.

[0006] Although the methods described above can partially eliminate the resulting stripe artifacts, Tainaka, K. et al. in “Whole-body imaging with single-cell resolution by tissue decolorization”, Cell 159(4), 911-924 (2014) show, for example, that samples with low brightness and samples with very high density can still be affected by stripe artifacts, even when the methods described above for eliminating the stripe artifacts or modifications thereof are used.

[0007] Zechen, Wei et al. take a slightly different approach in “Elimination of stripe artifacts in light sheet fluorescence microscopy using an attention-based residual neural network” Biomed Opt Express. 2022 Mar 1; 13(3): 1292-1311, who train a modified U-Net to eliminate stripe artifacts. The training dataset for the U-Net consists of stripe-free training images recorded with a special light sheet microscope as target outputs. The training inputs include augmented training images, calculated from the stripe-free training images using a perturbation model, which exhibit various artificially generated stripe artifacts.

[0008] The various described methods for eliminating shadows or fringe artifacts deliver good results for their respective specific application scenarios. However, the current state of the art does not provide a method that can generate fringe- or shadow-free image data for any sample type with the lowest possible light exposure. Summary of the invention

[0009] The invention is based on the objective of providing a method, in particular an automated method, with which an image processing system can be trained to perform a virtual multi-angle reconstruction, enabling the creation of streak-free and shadow-free images, especially image stacks of high quality, while simultaneously minimizing sample exposure. Furthermore, the invention solves the objective of providing a method for creating streak-free and shadow-free images, especially image stacks of high quality, while simultaneously minimizing sample exposure. Finally, the invention solves the objective of providing an image processing system, in particular comprising an imaging device, a computer program, and a computer-readable storage medium, with which the methods can be executed.

[0010] One or more problems are solved by the subject matter of the independent claims. Advantageous further developments and preferred embodiments form the subject matter of the dependent claims.

[0011] One aspect of the invention relates to a method for training an image processing system with a machine learning model to perform a virtual multi-angle reconstruction of image stacks acquired with a light-sheet microscope, in particular a sample of a sample type, comprising: - Recording at least one light sheet fine stack of a sample, wherein the light sheet fine stack comprises several image stacks and for different image stacks a light sheet illuminates the sample at different angles, - Determining target outputs based on the light-sheet fine stack, the machine learning model, and a classical multi-angle reconstruction, - Determining training inputs based on a rough light sheet stack, the machine learning model and classical multi-angle reconstruction, where the rough light sheet stack comprises several image stacks and for different image stacks the light sheet illuminates the sample at various angles, - Creating an annotated dataset encompassing the target outputs and learning inputs, - Optimizing the machine learning model to perform the virtual multi-angle reconstruction using the annotated dataset, characterized in that - the rough stack of light sheets contains fewer images than the fine stack of light sheets, in particular fewer image stacks, and a virtual reconstruction determined by means of virtual multi-angle reconstruction has fewer image artifacts than a reconstructed image stack determined by means of classical multi-angle reconstruction from the rough stack of light sheets.

[0012] The samples can be any objects, fluids, or structures. Each sample is appropriately positioned and fixed in the beam path of a microscope using a sample holder.

[0013] In the following, "images" refers specifically to microscope images, particularly those acquired with a light-sheet microscope. Furthermore, "images" also includes processed images, such as those processed with a machine learning model. Specifically, "image data" encompasses single images, image stacks, and multiple image stacks acquired with different exposure angles, such as those obtained with light-sheet microscopes. Additionally, "images" can include image data and time series of images or image stacks. In particular, images can contain depth information in addition to color and brightness information.

[0014] In the following, a processing model refers to a model designed to process input data and output result data, sometimes also called output data. The processing model can be a classical model that, for example, applies or was created to apply classical optimization or analysis methods. Likewise, the processing model can be a model trained using a learning method; in this case, it is also called a machine learning model.

[0015] In the following, a virtual processing mapping refers to the processing of image data performed using a trained machine learning model. Each virtual processing mapping corresponds to a classical processing mapping implemented using optimization or analysis methods. In particular, a distinction is made between a virtual multi-angle reconstruction performed using a trained machine learning model and a classical multi-angle reconstruction performed using a classical processing model.

[0016] In the following, an image stack, sometimes also called a stack or Z-stack, refers to one or more images offset from each other in height, which are specifically registered or registerable with each other, i.e., identical points in the sample are mapped to identical points in the images of the image stack.

[0017] In the following, a fine light-sheet stack comprises several image stacks of a sample, wherein each image stack is acquired with a different illumination direction, or rather, each light sheet illuminates the sample from a different angle. A fine light-sheet stack contains more images than a coarse light-sheet stack; in particular, the spacing between the images in the coarse light-sheet stack is greater, specifically twice as large, three times as large, or four times as large as in the fine light-sheet stack.Alternatively, the angular offset between the different illumination angles in the coarse stack of light sheets can be larger than in the fine stack of light sheets, for example twice as large, three times as large, or even four times as large. In particular, the coarse stack of light sheets can be a true subset of the fine stack of light sheets; for example, some of the images or some of the image stacks of the fine stack of light sheets are not included in the coarse stack of light sheets when the coarse stack of light sheets is compiled.

[0018] In the following, learning inputs refer to data used in the training of a machine learning model that is entered into the machine learning model; a training dataset, also called an annotated dataset, includes target outputs in addition to the learning inputs.

[0019] In the following, target outputs, when training a machine learning model to execute a processing function, refer to data used to which the result outputs generated by the machine learning model based on the training inputs are to be adapted. This approximation is achieved using an objective function.

[0020] In the following, classical multi-angle reconstruction refers to classical image processing in which one or more images, in particular one or more image stacks, taken at different lighting angles are combined. Specifically, shadow effects occurring during classical multi-angle reconstruction are eliminated, for example, by averaging. Preferably, classical multi-angle reconstruction includes

[0021] In the following, image artifacts, especially fringe artifacts or shadow artifacts, refer to striped image areas with a reduced image signal compared to adjacent image areas. Such shadow artifacts occur in images taken with light-sheet microscopes behind opaque or partially opaque sample structures. They arise when sample structures absorb light from the light source so strongly that, extending from the light source, a strip with a reduced image signal forms behind it.

[0022] In the prior art, a training dataset is calculated from image data without shadow artifacts using augmentation to train a machine learning model. However, suitable augmentation is only possible if image data without shadow artifacts can be generated or acquired, and the described scenario uses a very specific type of sample. Other methods known in the prior art involve exposing samples to strong light levels from different directions using multiple exposures to obtain artifact-free or at least artifact-reduced image data.

[0023] The inventors of the present invention have recognized that light exposure can be significantly reduced by training specific machine learning models individually for different sample types, by first acquiring or generating shadow-free or artifact-free image data using multiple exposures, determining training datasets based on the artifact-free image data, and training a machine learning model on the basis of the determined training datasets so that it can generate shadow-free or artifact-free image data from image data with shadow artifacts, in particular for samples of the respective sample type.

[0024] For this purpose, coarse stacks of light sheets are determined or recorded for training. A machine learning model is then trained to perform a virtual multi-angle reconstruction based on these coarse stacks and the artifact-free target outputs determined from fine stacks. Such a trained virtual processing image or machine learning model can then calculate result data based on the coarse stacks of light sheets. These result data, like the virtually artifact-free target outputs, exhibit no or at least significantly fewer fringe artifacts than the images from the coarse stacks of light sheets.

[0025] Preferably, determining the learning inputs includes a - Recording the rough stack of light sheets, whereby fewer different image stacks, fewer different angles, or fewer images per image stack are recorded when recording the rough stack of light sheets compared to the fine stack of light sheets, - Determining the coarse stack of light sheets from the fine stack of light sheets, wherein the coarse stack of light sheets is a true subset of the fine stack of light sheets, preferably comprising only every second, third or fourth of the image stacks recorded when recording the fine stack of light sheets, or comprising only every second, third or fourth image of each image stack of the coarse stack of light sheets, particularly preferably comprising only a single image stack of the image stacks of the fine stack of light sheets.

[0026] If the coarse light sheet stack is a true subset of the fine light sheet stack, then the light exposure during the determination of the training data is particularly low, since the coarse light sheet stack does not need to be recorded separately, thus reducing the light exposure during training and protecting the sample.

[0027] Preferably, the method comprises determining one or more acquisition parameters prior to acquiring the light sheet fine stack, depending on the sample type for which the training is performed, such that areas in the sample located behind opaque or partially opaque sample structures are illuminated in at least one of the acquired image stacks or are partially illuminated in at least several of the image stacks of the light sheet fine stack. The acquisition parameters include one or more of the following parameters: - a vertical offset of vertically offset images in an image stack, - a number and angular distance of the different angles, - an exposure spectrum, - fluorophores used, - Contextual information; and furthermore In particular, the sample type is determined based on one or more of the following properties of opaque or partially opaque sample structures: - an expansion of the sample structures, - an opacity, - a spectral transmittance, - usable fluorophores, and - a density in the sample of the sample type.

[0028] By selecting the acquisition parameters based on the sample properties, optimal parameters can be chosen for each sample type. At the same time, the sample can be protected by avoiding unnecessary images. For example, if a sample contains a particularly large number of opaque structures, it must be illuminated with a very fine light, meaning that many different image stacks, each with a different illumination angle, must be acquired to ensure the sample is illuminated as comprehensively as possible.

[0029] Preferably, the machine learning model is a staged processing model or a total processing model, wherein the staged processing model comprises a detail enhancement model and a reconstruction model, the reconstruction model is configured to compute the classical multi-angle reconstruction, the detail enhancement model is trained using the annotated dataset to perform a detail enhancement mapping, and sequential execution of the classical multi-angle reconstruction and the detail enhancement mapping yields the virtual multi-angle reconstruction, depending on the order in which the detail enhancement model and the reconstruction model are applied, the detail enhancement model is either trained to calculate a reconstructed image stack, also called a reconstructed light-sheet rough stack, from the light-sheet coarse stack using classical multi-angle reconstruction,to map to a reconstructed image stack, also called a reconstructed light-sheet fine stack, calculated from the light-sheet coarse stack using classical multi-angle reconstruction, or to map the light-sheet coarse stack to the light-sheet fine stack, and to determine the training inputs and target outputs depending on the order in which the detail improvement model and the reconstruction model are applied, includes: selecting a reconstructed light-sheet coarse stack or selecting a reconstructed light-sheet fine stack as the training inputs and selecting a reconstructed light-sheet fine stack or selecting a light-sheet fine stack as the target outputs, and, if the machine learning model is the overall processing model, the overall processing model is trained directly to perform the virtual multi-angle reconstruction using the annotated dataset.where, in particular, the virtual multi-angle reconstruction includes the detail enhancement mapping and the classical multi-angle reconstruction, and the determination of the annotated dataset includes: selecting a rough light sheet stack comprising at least one image stack as the training inputs and selecting at least one reconstructed image stack determined from the fine light sheet stack using the classical multi-angle reconstruction as the target outputs.

[0030] If the machine learning model is implemented as a staged processing model, then a detail improvement mapping is trained during training, which is simpler and less complex than training a complete processing model. However, when the machine learning model is implemented as a complete processing model, the processing is more efficient than when implemented as a staged processing model.

[0031] Preferably, determining the target outputs from the lightsheet fine stack includes: - Calculating, using classical multi-angle reconstruction, several reconstructed candidate stacks, wherein a different set of reconstruction parameters of classical multi-angle reconstruction is used for each of the several reconstructed candidate stacks, wherein, for example, a reconstruction algorithm used, a number of iterations when applying the reconstruction algorithm, correction methods used, or correction parameters of the correction method used are selected using the parameters used. - Reviewing the candidate stacks, particularly regarding the quality of classical multi-angle reconstruction, and - Selecting one reconstructed light-sheet fine stack from the reconstructed candidate stacks, particularly with regard to the quality of the classical multi-angle reconstruction.

[0032] By selecting the reconstructed light sheet fine stack from the candidate stacks, it can be ensured that an optimally reconstructed image is used for training, depending on the sample type.

[0033] Preferably, optimizing the machine learning model includes augmenting the annotated data set or simulating additional data using a point spread function and the target outputs of the annotated data set, wherein the point spread function is, for example, a depth-variant point spread function.

[0034] Augmenting the data in the annotated dataset can increase the data basis for training and thus improve the training.

[0035] Preferably, the augmentation includes, in particular before calculating the target outputs, one or more of the following: - Transforming the images of the light-sheet fine stack or the light-sheet coarse stack or both stacks, wherein the transformation includes one or more of the following: - Noise reduction, - De-Blooming, - a reflection, - a turn, - a scaling, - deformation by means of an elastic lattice, - a brightening, - a darkening, - adjusting the gamma correction value, - a vignette, - an offset, - a color inversion, - an artificial noise effect, - an undersampling, - a masking, - a soft focus, - any filtering with a linear or non-linear filter, - sharpening, - an artifact distance mapping, - an unfolding, - a histogram spread, - a downsampling, and - an inpainting of the images, whereby the transformation is carried out in particular by means of a transformation machine learning model trained for the respective transformation.

[0036] Preferably, after optimizing the machine learning model, the method also includes checking whether the machine learning model is suitable for reconstructing the light-sheet rough stack using the learned virtual multi-angle reconstruction.

[0037] By verifying the processing with the machine learning model, it can be ensured that the virtual multi-angle reconstruction is performed with good quality by the machine learning model.

[0038] Preferably, the acquisition of at least one light sheet fine stack is carried out at a specific location on the sample; in particular, the specific location is not required for acquiring further images of the sample; and in particular, the specific location on the sample is automatically selected by the image evaluation system, for example, in a predetermined area of ​​the sample, whereby, for example, the specific location can be chosen so appropriately that, in areas that are particularly interesting, for example, in a later course of an experiment, there is no light exposure at the beginning of the experiment due to the acquisition of the light sheet fine stack for training.

[0039] Another aspect according to at least one embodiment of the present invention relates to a method for performing a virtual multi-angle reconstruction of an image stack recorded with a light microscope using an image processing system comprising a machine learning model, comprising: - Providing a machine learning model for performing virtual multi-angle reconstruction, wherein a machine learning model is used that has been trained according to the method for training an image processing system according to one of the preceding claims, - Recording, using a light-sheet microscope, a rough stack of light sheets to be processed, comprising at least one image stack of the sample of the sample type, - Calculating a virtual reconstruction from the rough stack of light sheets using virtual multi-angle reconstruction.

[0040] In conventional, classical multi-angle reconstructions, the light-sheet microscope must acquire numerous image stacks at different illumination angles, which can place a significant burden on the sample. By contrast, the method described above requires only coarse light-sheet stacks to generate a high-quality reconstructed image stack or a high-quality virtual reconstruction, thus reducing the light exposure during sample examination.

[0041] Preferably, the above procedure also includes the following before calculating a virtual reconstruction: - Check whether one or more machine learning models are suitable for reconstructing the rough stack of light sheets to be processed using the respective learned virtual multi-angle reconstruction, and, if no suitable machine learning model is available: - Performing the above-described method for training an image processing system with the sample of the sample type, wherein in particular the acquisition of the light sheet fine stack takes place in a predetermined area of ​​the sample and in particular the acquisition of the light sheet coarse stack to be processed takes place in an area of ​​the sample different from the predetermined area.

[0042] By first checking whether a suitable machine learning model has already been trained, an unnecessary exposure to a fine light stack can be avoided, thus protecting the sample from unnecessary light exposure. In particular, several machine learning models can be checked as described above; the machine learning models to be checked can be determined, for example, based on contextual information.

[0043] Preferably, checking a reconstructed fine stack output by the machine learning model includes: - a manual check, - a comparison with example target images, - Determining a quality metric, in particular based on image properties of the virtual reconstruction, for example image sharpness, edge sharpness, number of image artifacts such as stripe artifacts, in particular by means of a metric quality model, wherein the metric quality model is set up, in particular trained, to identify image properties, - input into a quality classification model that has been trained to identify well and poorly reconstructed image stacks, - Check based on image features such as image sharpness, noise level, blood-bleed, artifacts such as ring, stripe or shadow artifacts.

[0044] Another aspect according to one or more of the embodiments of the present invention relates to an image processing system comprising an evaluation device for carrying out the methods according to the aspects described above.

[0045] Preferably, the image processing system comprises an imaging device, in particular a light microscope.

[0046] Another aspect according to one or more embodiments of the present invention relates to a computer program product comprising instructions which, when the program is executed by one or more computers, cause them to execute the method according to one of the aspects described above; the computer program product is in particular a computer-readable storage medium. Brief summary of the characters

[0047] The invention is explained in more detail below with reference to the examples shown in the drawings. The drawings show: Fig. 1 schematically an image processing system for training or performing a virtual multi-angle reconstruction according to one embodiment; Fig. 2 schematically an evaluation device for performing the training or the virtual multi-angle reconstruction with a machine learning model and an imaging device according to an embodiment; Fig. 3 a schematic representation of a machine learning model according to one embodiment; Fig. 4 a schematic representation of a method according to one embodiment; Fig. 5 a schematic representation of a method according to one embodiment; Fig. 6 a schematic representation of a method according to a further embodiment; Fig. 7 schematically a picture processing system for use with the methods according to one or more embodiments; Fig. 8 A schematic representation for better understanding of a method according to one embodiment. Detailed description of the embodiments

[0048] One embodiment of an image processing system 1 comprises an imaging device 100 and a control and evaluation device 130, hereinafter referred to as the evaluation device 130. The evaluation device 130 is communicatively connected to the imaging device 100, for example, via a wired or wireless communication link. The evaluation device 130 can evaluate image data 200 acquired by the imaging device 100 and, for example, control the imaging device 100 based on the evaluated image data. The imaging device 100 according to the first embodiment is a light-sheet microscope. Fig. Figure 1 shows a schematic representation of the imaging device 100, illustrating the basic components of the light-sheet microscope, with the light-sheet microscope shown schematically in both a top and a side view. The imaging device 100 comprises a light source 101, a beam expander 102, a lens 103, an illumination objective 104, a sample chamber 105, a detection objective 106, and a camera 107.

[0049] The light source 101 is, in particular, a laser, for example, a solid-state or gas laser. A laser beam generated by the laser is collimated and expanded by the beam expander 102. A lens 103, in particular a cylindrical lens, forms a sheet of light 110, which is projected by the illumination objective 104 onto a sample 120 such that a focal point or the thinnest part of the sheet of light 110 is located in the center of the sample chamber 105. By using a cylindrical lens, the laser beam expanded by the beam expander 102 is focused, in particular, along one direction. A so-called sheet of light, also called a light sheet 110, is formed, which illuminates only a very thin layer in the sample 120. For this reason, light-sheet microscopes are considered particularly gentle with regard to the light exposure of samples 120.According to this embodiment, a light sheet is formed in the light-sheet microscope which is essentially parallel to the plane spanned by the x-axis and the y-axis.

[0050] The sample chamber 105 typically consists of optically clear glass walls and has an open top through which the sample 120 can be inserted into the sample chamber 105 on a sample holder 121. The sample chamber 105 is filled either with a heated physiological solution for imaging living cells or with a cleaning fluid for fixed and purified tissue. The sample 120 is attached to the rod-shaped sample holder 121. The sample 120 is moved into the light sheet 110 by means of the sample holder 121. Within the light sheet 110, the sample 120 is excited to fluoresce, with the excitation occurring only in the narrow area of ​​the light sheet 110. This is why background image signals in light-sheet microscopes are generally lower than in conventional fluorescence microscopes.

[0051] The fluorescent light 115 emitted by the sample is collected in the detection lens 106, which is arranged perpendicular to the plane of the light sheet 110, and recorded by the camera 107. The image data 200 recorded by the camera 107 are transmitted to the evaluation unit 130 and stored in a storage module 132 of the evaluation unit.

[0052] When small, thin samples and a relatively thick light sheet 110 are used, a light-sheet microscope can capture the sample 120 in real time. For samples larger than a local parameter of the light, the sample 120 is captured tile by tile in multiple images offset from each other along the Z-direction. While in conventional microscopes the objective is often moved along a Z-direction to create a Z-stack, in light-sheet microscopes the sample is moved accordingly. In a light-sheet microscope, the narrow light sheet 110 and the focus position of the detection objective 106 are aligned so that the focus lies precisely within the light sheet 110.If the focus position were adjusted by moving the detection lens 106, the position of the light sheet 110 would also have to be adjusted, thus requiring additional adjustment of the position of the light sheet 110 against the focus position of the detection lens, which would complicate the control.

[0053] The sample 120 can be moved within the sample chamber along all three spatial directions, and the sample can also be rotated around the y-axis using the sample holder 121.

[0054] According to this embodiment, the evaluation unit 130 can be connected to a monitor (not shown) on which the image data 200 can be displayed. The evaluation unit 130 is configured to control the imaging unit 100 for acquiring image data 200 with the camera 107, to evaluate the acquired image data 200 with an evaluation module 131, and to store the image data 200 on a storage module 132 (see Fig. 5) to store the evaluation unit 130. The recorded image data 200 can be displayed on the monitor if required. The evaluation unit 130 is configured to process and evaluate the recorded image data 200. The image data 200 includes, in particular, individual images 205, image stacks 210, light sheet fine stacks 220, light sheet coarse stacks 230, reconstructed image stacks 240, virtual reconstructions 250, and virtual light sheet fine stacks 260.

[0055] The evaluation unit 130 comprises the evaluation module 131, the storage module 132, and the control module 133. The modules of the evaluation unit 130 are interconnected via channels 134, enabling them to exchange data. These channels 134 are logical data connections between the individual modules. The modules can be implemented as either software or hardware modules.

[0056] The evaluation module 131 evaluates the entered image data 200 and, based on the evaluation, forwards information to the control module 133 or forwards the results of the evaluation to the storage module 132 for storage.

[0057] The storage module 132 stores the image data 200 acquired by the imaging device 100 and manages the data to be evaluated in the evaluation device 130.

[0058] The control module 133 can read the image data 200 from the storage module 132 and forward it to the evaluation module 131 for evaluation. Furthermore, the control module 133 can send control commands, also called control information, to the imaging device 100. In particular, the control module 133 can be configured to generate the control information based on the information received from the evaluation module 131. The control information can control the imaging device 100 as a whole or only specific parts of it.

[0059] According to the present embodiment, the evaluation unit is configured to read processing models from the storage module 132 and to process the image data 200 using the processing models in the evaluation module 141. The processing models include, in particular, a classical processing model trained to perform a classical multi-angle reconstruction 135, as well as machine learning models 140 that can be trained to perform a virtual multi-angle reconstruction, or machine learning models 140 that are already trained to perform the virtual multi-angle reconstruction.

[0060] In particular, the evaluation module 131 can comprise one or more machine learning models 140, each trained for different sample types to perform the virtual multi-angle reconstruction. Specifically, the machine learning models 140 are implemented as neural networks.

[0061] A machine learning model 140 (see Fig. 3) can, in particular, be a neural network with multiple layers. Specifically, the machine learning model 140 has an input layer 141, one or more intermediate layers 142, and an output layer 143. Input data 150 can be entered into the input layer 141 and processed by the input layer 141, the intermediate layers 142, and the output layer 143. The output layer 143 outputs result data 152. Depending on the type or implementation of the machine learning model 140 used, the form and scope of the input data 150 and the result data 152 vary. Some machine learning models 140 can also output intermediate outputs 151.

[0062] In the following, an input layer 141 denotes a first layer of a multi-layered machine learning model 140. In particular, a first layer of a neural network. According to the first embodiment, the machine learning model 140 is a U-Net, in particular a generative U-Net, which is implemented as a complete processing model. Light sheet rough stacks 230 are input into the machine learning model 140 as input data 150.

[0063] The machine learning model 140 is trained to perform virtual multi-angle reconstruction. During training, the evaluation module 131, controlled by the control module 133, reads a portion of the image data 200 of a training dataset (in particular, an annotated dataset) from the storage module 132 and inputs this training data into the respective machine learning model 140. Based on the output data 152 of the machine learning model 140 and target data contained in the annotated dataset, the evaluation module 131 determines an objective function and optimizes the objective function by adjusting the model parameters of the machine learning model 140 based on the optimization of the objective function.

[0064] In particular, the objective function is optimized using a stochastic gradient descent method. In this method, only a small subset of the training data from the annotated dataset, called a batch, is used at any given time. The control module 133 determines the objective function—here a loss function—for the input data of the batch, based on the output data 152 generated by the machine learning model 140 and the corresponding target data from the annotated dataset. This target function captures the difference between the output data 152 and the target data. The control module 133 then calculates a gradient for each of the calculated objective functions with respect to the model parameters of the machine learning model 140, sums the calculated gradients across the batch, and determines the mean.From the mean value, the control module 133 determines updated model parameters for the machine learning model 140 by means of so-called backpropagation. The control module 133 then initiates the machine learning model 140 with the updated model parameters in the evaluation module 131 and performs the next step of the stochastic gradient descent procedure.

[0065] The training of the machine learning model 140 terminates as soon as the optimization of the objective function achieves a predetermined limit.

[0066] Once the training is complete, the control module 133 stores the last used model parameters of the machine learning model 140 in the memory module 132, in particular together with context information, so that the newly trained machine learning model 140 can later be identified again and initialized, for example, for further training or inference.

[0067] As an alternative to the stochastic gradient descent method, other methods can also be used. In particular, any other training method can be used.

[0068] Once the training of a machine learning model 140 is complete, the corresponding model parameters are stored in the memory module 132 and can later be read out in the inference to execute the learned virtual multi-angle reconstruction.

[0069] The following describes a method for performing virtual multi-angle reconstruction with the image processing system 1 with a machine learning model 140 for images of a sample type according to the first embodiment ( Fig. 4).

[0070] The procedure comprises several steps. According to a first step S1 “acquisition of a rough stack of light sheets”, the camera 107 of the imaging device 100 acquires at least one rough stack of light sheets 230 in the sample under consideration, wherein the rough stack of light sheets 230 comprises at least one image stack 210. The acquired image data 200 are stored in the storage module 132.

[0071] According to some embodiments of the present invention, capturing a stack of light sheets with a light sheet microscope comprises capturing several image stacks 210, wherein each image stack 210 comprises several images 205 offset vertically from one another. When capturing the vertically offset images 205, not only is the focus position of the detection objective 106 shifted within the sample, but the position of the light sheet 110 within the sample is also shifted. This can be achieved, as described above, by shifting the sample 120, thus ensuring that the focus position and the position of the light sheet 110 within the sample 120 remain aligned. Alternatively, the light sheet 110 and the detection objective 106 can be shifted relative to each other such that they are shifted together within the sample 120 along the observation direction, so that the focus position of the detection objective 106 is always illuminated by the light sheet 110.

[0072] As with any image acquisition using a microscope, the light from the light sheet 110 is scattered by the various sample structures of the sample 120. Along the direction of propagation of the light sheet 110, shadows form behind each sample structure, depending on the transmittance of the illuminated sample structures. Since only the very narrow light sheet 110 is illuminated in the sample 120, and light detection occurs perpendicular to the light sheet 110, the light intensity in the shadows of opaque or partially opaque sample structures 125 (hereinafter referred to as opaque sample structures 125) is particularly reduced in light-sheet microscopes compared to the light intensity in the light sheet 110. This is illustrated schematically in Figure 1. Fig. 4 (a). In particular, compared to other types of microscopes, especially those that use large-area illumination, the light intensity behind opaque sample structures is considerably reduced.

[0073] Therefore, objects within the shadow 126, also called shadow objects 127, are particularly poorly illuminated, which is why image signals from shadow objects 127 in an image 205 or in a corresponding image stack 210 are only poorly visible and, in particular, only visible with a reduced image signal or reduced intensity. This is due to the Fig. 4 (a) the image stack 210 shown, in which in an uppermost image 205 a shadow 126 practically completely obscures a shadow object 127, is shown by way of example or schematic representation.

[0074] Therefore, when using light-sheet microscopes for samples 120 in which shadows 126 occur, several image stacks 210 are often taken, with the sample 120 being illuminated from a different angle for each of the different image stacks 210. This is in Fig. 4 (b) is shown schematically. Five different image stacks 210, 211, 212, 213, and 214 are acquired from sample 120, with the various lines and dashed lines indicating the focus positions for the different image stacks 210 to 214 of the imaging device 100 in sample 120 when acquiring the respective image stack 210 to 214. For the different image stacks 210 to 214, sample 210 is therefore illuminated at a different angle. This can be achieved, in particular, by rotating sample 120 about the y-axis. Fig. 4 (b) For the sake of clarity, the shadows are only indicated for image stacks 210, 211 and 213.

[0075] As in Fig. As can be seen in Figure 4(b), the shadow object 127 in image stack 210 is located precisely within the shadow 126 of the opaque sample structure 125, whereas, for example, image stacks 211 and 213 are no longer located within the shadow 126 when the image was taken. For the sake of clarity, the following is shown in the Fig. 4 (a) to (d) always show only a single opaque sample structure 125, likewise only one or a few shadows 126 and a shadow object 127.

[0076] In real samples 120, the number and density, as well as the geometric extents of the opaque sample structures 125, can differ significantly between different sample types. Accordingly, it may be necessary to acquire a different number of image stacks 210 with different illumination angles, and the vertical offset of the vertically offset images 205 of an image stack 210 can also be selected depending on the sample type.

[0077] According to a second step S2 “Check whether a suitable machine learning model is available”, the evaluation unit 130 checks whether the storage module 132 can provide a suitable, trained machine learning model 140 to perform the virtual multi-angle reconstruction for the sample type of the investigated sample 120.

[0078] If the evaluation unit 130 determines that no suitable machine learning model 140 is stored in the memory module 132, the imaging unit 100 is instructed by the control module 133 to take a light sheet fine stack 220.

[0079] In the third step S3, "Selecting a specific location in the sample," a selection machine learning model autonomously selects a specific location in sample 120 based on the sample type. The selection machine learning model was trained on specific locations previously selected by a user on comparable samples 120 for the acquisition of light sheet fine stacks 220. At the selected location, the imaging device 100 automatically acquires the light sheet fine stack 220, which, for example, comprises five image stacks 210 with different illumination angles. The acquired light sheet fine stack 220 is read from the camera 107 and stored in the memory module 132.

[0080] A fine light sheet stack 220 differs from a coarse light sheet stack 230 in that the fine light sheet stack 220 captures the sample 120 more finely than the coarse light sheet stack 230, and in particular, the coarse light sheet stack 230 contains fewer images 205 than the fine light sheet stack 220. Specifically, the fine light sheet stack 220 contains more image stacks 210 than the coarse light sheet stack 230; see, for example, here. Fig. 6 (a), according to which capturing the rough stack of light sheets 230 comprises capturing only three image stacks 210, 212, and 214. Alternatively, the rough stack of light sheets 230 can also, for example, contain fewer images 205 per image stack 210, i.e., the distance, also called the vertical offset, between the vertically offset images 205 of the image stack 210 is greater for the rough stack of light sheets 230 than for the fine stack of light sheets 220; see, for example, Fig. 6 (b).

[0081] Alternatively, a specific location can also be selected by a user, for example in an overview image.

[0082] According to step S4 “Determine target outputs”, the evaluation module 132 calculates the target outputs depending on the light sheet fine stack 220, the light sheet coarse stack 230 and the machine learning model 140.

[0083] According to the first embodiment, the machine learning model 140 is a total processing model, the total processing model is trained to calculate virtual light sheet fine stacks 260 from input light sheet coarse stacks 230.

[0084] To train the machine learning model 140, the target outputs are first determined. For the overall processing model, the target outputs are reconstructed using a classic multi-angle reconstruction 135 from the light-sheet fine stack 220 image stack 240, see Fig. 4 (c). In the classical multi-angle reconstruction 135, image points from images 205 in image stacks 210 to 214 of the light sheet fine stack 220, acquired at different exposure angles, are first determined. These image points must each capture the same point in the sample 120. For this purpose, all images 205 of image stacks 210 to 214 are registered in the three spatial directions, so that the image points of images 205 of image stack 210 are each assigned to voxels of the sample 120. Based on this registration, a new image value, a so-called reconstructed image value, is then assigned to each individual voxel from the image signals of the image points of the different images 205, for example by averaging. This is carried out for as many voxels of the sample 120 as possible.More detailed information on various methods for registering and subsequently combining the image stacks 210 to 214 is described, for example, in Temerinac-Ott, Maja, (2012), “Multiview reconstruction for 3D Images from light sheet based fluorescence microscopy”.

[0085] According to the exemplary illustration, the light sheet fine stack 220 comprises precisely the five image stacks 210 to 214. If we consider, for example, the shadow object 127, it is located in image stack 210 in the shadow 126 of the opaque sample structure and accordingly has a significantly reduced image signal. In image stack 210, however, the shadow object 127 is not shadowed by the opaque sample structure 125, which is why a considerably higher image signal is to be expected for the corresponding pixel of image stack 210 than for image stack 210, in which the shadow object 127 is located in the shadow 126 of the opaque sample structure 125.

[0086] If the different illumination angles and height offsets in the image stacks 210 are chosen such that shadow objects 127 are well illuminated in a majority of the image stacks 210 to 214, then, using the classical multi-angle reconstruction 135, a reconstructed image stack 240, or a reconstructed image, can be determined for the pixel of the shadow object 127 with a significantly higher image signal compared to image 205, in which the shadow object 127 lies precisely in the shadow 126 of the opaque structure, which is why the shadow object 127 is clearly visible in the reconstructed image stack 240, and the shadows 126 can be eliminated or at least reduced.

[0087] An illumination angle corresponding to the reconstructed image stack 240 corresponds precisely to the illumination angle of the rough light sheet stack 230. If the rough light sheet stack 230 comprises several image stacks 210, then a reconstructed image stack 240 is determined for each of the image stacks 210 of the rough light sheet stack 230. The multiple reconstructed image stacks 240 then constitute the target outputs.

[0088] According to one embodiment, the target outputs can also be a three-dimensional volume image in which each voxel is assigned a correspondingly reconstructed image value, which can also be called voxel value here.

[0089] In the following step S5, "Determining Training Inputs," the training inputs corresponding to the target outputs are determined. According to the first embodiment, the training inputs comprise a coarse stack of light sheets 230, distinct from the coarse stack 230 taken in step S1. According to step S5, the coarse stack 230 used for training is determined from the fine stack 220. According to the first embodiment, the coarse stack 230 used for training comprises precisely every second image stack 210 to 214 of the fine stack 220, see [reference]. Fig. 4 (d) and Fig. 5 (a). The light-leaf coarse stack 230 thus forms a true subset of the light-leaf fine stack 220.

[0090] Alternatively, every third or only every fourth of the image stacks 210 to 214 of the fine light sheet stack 220 acquired in step S3 could be used as the training inputs. According to another alternative, a coarse light sheet stack 230 can also be acquired separately from the fine light sheet stack 220; however, the image data 210 used for training must be acquired at the same location in the sample.

[0091] According to further alternatives, in each of the image stacks 210 of the light-sheet fine stack 220, every second image within an image stack 210 cannot be transferred to the light-sheet coarse stack 230, as in Fig. 5 (b) is shown. Even then, the light-sheet coarse stack 230 is a proper subset of the light-sheet fine stack 220.

[0092] In step S6, "Creating an annotated dataset comprising the target outputs and the learning inputs," pairs of corresponding learning inputs and target inputs are compiled in the annotated dataset. According to the first embodiment, this includes, in particular, combining the learning inputs and target outputs described above into learning pairs. According to one embodiment, further learning pairs can be created by means of suitable augmentation in order to write a larger amount of data into the annotated dataset. The augmentation includes, in particular, the transformations described above.

[0093] In step S7, "Optimizing the machine learning model to perform the virtual multi-angle reconstruction using the annotated dataset," the control module 133 and evaluation modules 131 use the machine learning model 140 to execute a stochastic gradient method to optimize the model parameters of the machine learning model 140. For this purpose, the control module 133 randomly selects a set of examples from the annotated dataset in several successive training steps and inputs the training inputs of the randomly selected set of examples into the machine learning model 140. The machine learning model 140 then computes the result data 152. The control module 133 determines the objective function for each training pair, sums these over the training pairs of the randomly selected set of examples (also called the batch), and optimizes the machine learning model 140 using a gradient determined from the objective function in a gradient descent procedure.

[0094] The optimization of the machine learning model is carried out with several successive optimization steps until the objective function reaches a termination condition. Once the termination condition is reached, step S7 of training the machine learning model 140 is completed.

[0095] According to step S8, the control module 133 reads the rough stack of light sheets 230, acquired in step S1, from the memory module 132 and inputs the rough stack of light sheets 230 into the fully trained machine learning model 140. The rough stack of light sheets 230 comprises only one image stack 210 from the sample 120. From the input image stack 210, the machine learning model 140 calculates a virtual reconstruction 250 and stores the virtual reconstruction 250 in the memory module 132.

[0096] According to the first embodiment, when determining the target outputs in step S4, one or more reconstructed image stacks 240 are determined, wherein the reconstructed image stacks 240 exhibit a significantly reduced number of shadow artifacts or fringe artifacts compared to the image stacks 210 of the light sheet fine stack 220 and the light sheet coarse stack 230. Ideally, the reconstructed image stacks 240 exhibit no shadow artifacts at all; however, this may not always be the case, especially for samples 120 with a high density of opaque sample structures 125. After training, the machine learning model 140 outputs the virtual reconstruction 250, which also exhibits a significantly reduced number of shadow artifacts 126 compared to the light sheet coarse stack 230 input as input data 150 to the machine learning model 140. Therefore, once a machine learning model 140 has been trained for the respective sample type, further analysis or...In further experiments with samples 120 of the respective sample type, significantly fewer images are taken in order to eliminate or at least reduce shadow artifacts 126 in the recorded light-sheet coarse stacks 230. Thus, the first embodiment provides a method for processing image data 200 of samples of a specific sample type taken with a light-sheet microscope, in which the sample burden can be significantly reduced.

[0097] Alternatively, step S3 can also be executed before step S1, in which case step S2 would be omitted. Steps S4 and S5 can also be executed after step S3 and before step S1.

[0098] If, for example, it is known that no machine learning model yet exists, it is not necessary to first record a rough stack of light sheets 230 according to step S1. Instead, the fine stacks of light sheets 220 are recorded directly according to step S3. According to step S4, the fine stack of light sheets 220 is processed using classical multi-angle reconstruction 135. According to step S5, the training inputs are determined, and according to step S6, the annotated data set is provided. According to step S7, the machine learning model 140 is then trained, and subsequently, according to step S1, the rough stack of light sheets 230 to be processed is recorded and processed using the fully trained machine learning model 140.

[0099] According to an alternative, the annotated data set stored on the memory module 132 consists of only a single learning pair, comprising a rough stack of light sheets 230 as the training inputs and the reconstructed image stack 240. During the training step S7, the control module 133 then performs the augmentation described with reference to the first embodiment randomly before the learning pair is input into the machine learning model 140. Alternatively, several rough stacks of light sheets 230 and several fine stacks of light sheets 220 can also be included for training.

[0100] According to one embodiment of the first configuration, several reconstructed candidate stacks are calculated when determining the target outputs. These stacks are then compared to identify the reconstruction with the highest reconstruction quality. The best reconstruction can be determined, for example, based on a metric that captures image properties such as the number of fringing artifacts, image sharpness, image contrast, brightness distribution, color distribution, or similar characteristics.When calculating the multiple reconstructed candidate stacks, a different set of parameters is used for each stack. This set of parameters specifies, in particular, the classical multi-angle reconstruction, specifically a reconstruction algorithm and parameters used in the algorithm, such as correction parameters, the number of iterations of the algorithm, the correction methods used, and correction parameters for each of these methods. The images from the candidate stack that yields the best reconstruction, according to the metric, are then selected as the target outputs.

[0101] According to one embodiment of the first model, a fully trained machine learning model 140 for a sample 120 of the sample type to be investigated is already stored in the memory module 132. As described above, the virtual multi-angle reconstruction is very sensitive to changes in the optical properties of the different sample types. The different sample types can be distinguished, for example, based on the optical properties of sample structures within a sample, in particular the optical properties of the opaque or partially opaque sample structures 125. The optical properties can include one or more of the following: for example, their extent, their opacity, their spectral transmittance, usable fluorophores, and the density of the opaque sample structures in the sample of the respective sample type.

[0102] For example, a rough stack of light sheets 230 to be processed can first be acquired. Then, the rough stack of light sheets 230 is virtually reconstructed using several potentially suitable, pre-trained machine learning models 140 stored in the memory module 132. The virtual reconstructions 250 created by the different machine learning models 140 are then checked using a suitable metric, such as the number of fringe artifacts in the virtual reconstructions 250 or similar image properties that capture the quality of the virtual reconstruction 250, to determine whether one of the machine learning models 140 produces the virtual reconstructions 250 with sufficient quality. If this is the case, no further fine stack of light sheets 220 needs to be acquired, and the sample can be evaluated with the respective machine learning model 140.This allows the light exposure of the respective sample to be further reduced, as no further light sheet fine stack 220 needs to be acquired. However, if the check reveals that the created virtual reconstructions 250 do not meet the quality requirements, then step S3 is carried out to acquire a light sheet fine stack 230 and the corresponding subsequent steps are executed to train a new machine learning model 140.

[0103] In principle, when conducting an experiment, the quality of a created virtual reconstruction 250 can be checked at regular intervals as described above. For example, with time-varying samples 120, the reconstruction quality may deteriorate during the course of an experiment. This is primarily due to changes in the optical properties of the time-varying sample 120. If such a deterioration in the reconstruction quality is detected, a new training is performed with a new machine learning model 140 using a reconstructed image stack 240 determined from a newly acquired light sheet fine stack 220.

[0104] According to a further embodiment, a tilting device can also be used when acquiring each image stack 210. This device tilts the light sheet 110 along the z-axis at a high frequency, for example 1 kHz, several kHz, 1-10 kHz, or 1-20 kHz, by up to 10°, so that when acquiring each of the images 205 of an image stack 210, the direction of the light sheet 110 oscillates by 10° around a rest position at 0°. This allows the shadow artifacts 126 to be reduced with the acquisition of a single image 205, but often not completely eliminated. Therefore, even for light-sheet microscopes with a tilting device, fine stacks of light sheets 220 are acquired in order to further reduce the shadow artifacts using the classic multi-angle reconstruction 135.

[0105] According to a further embodiment, the classic multi-angle reconstruction 135 also includes an unfolding with a point spreading function.

[0106] According to the first embodiment, the machine learning model 140 is a U-Net, in particular a U-Net implemented as a generative adversarial network, or GAN for short.

[0107] Generative adversarial networks comprise at least one generator and one discriminator. In this case, the generator receives the rough stack of light sheets as training input and generates the virtual reconstruction from it. The discriminator receives as input either the output of the generator or a reconstructed image stack 240 from the annotated dataset, derived from the fine stack of light sheets. The output of the discriminator is also called the discrimination result. The discrimination result indicates whether the input image is a virtual reconstruction 240 output by the generator or a reconstructed image stack 240 from the training dataset. The generator and the discriminator are trained together. Their outputs are captured in a common objective function.In the objective function, the generator is penalized if the discriminator recognizes result data output by the generator as such, and the discriminator is penalized if it incorrectly classifies virtual reconstructions 250 output by the generator as reconstructed image stacks 240.

[0108] Various objective functions are used when training GANs. Common objective functions include, for example, a minimax loss or a Wasserstein loss.

[0109] In one variation, the discriminator can be implemented as a semantic segmenter. For example, the discriminator can output pixel by pixel whether the respective pixel was generated by the generator or originates from the annotated dataset.

[0110] As another alternative, the discriminator can also be a so-called patch classifier. Unlike the semantic segmenter, the discriminator does not check pixel by pixel whether the input images are generated data or data from the training dataset, but rather based on image sections, or patches, and then sums the results of the patches to determine whether an input originates from the generator or from the training dataset. The size of the patches can be adjusted as needed.

[0111] The person skilled in the art is aware of further suitable configurations of both the discriminator and the objective functions from the prior art, which is why he has no difficulty implementing further configurations here.

[0112] According to a second embodiment, the sample 120 is illuminated from two sides by overlapping light sheets 110. This can be implemented, for example, by suitable mirrors and beam splitters that redirect the light sheet 110 so that the sample 120 is illuminated from both sides. Alternatively, a second light source 101 can be used, which illuminates the sample as shown in Fig. Figure 7 shows a schematic representation, illuminated from an opposite side. To form the light sheet 110, the imaging device 100 comprises, in addition to the light source 101, a beam expander 102, a lens 103, and an illumination objective 104.

[0113] Additionally, a second camera 107 with a corresponding detection lens 106 can also be provided.

[0114] While in the first embodiment the machine learning model 140 is a complete processing model, in a second embodiment the machine learning model 140 is designed as a staged processing model. As exemplified in the Fig. As shown in Figures 8 (a) and (b), the processing of the image data 200 takes place in several stages according to the staged processing model.

[0115] As shown, the machine learning model 140, implemented as a staged processing model, includes a detail improvement model that is trained to perform a detail improvement mapping, and a reconstruction model that performs the classic multi-angle reconstruction 135.

[0116] According to Fig. 8 (a) The staged processing model initially comprises a first detail improvement model 1401, which is configured or trained to map a recorded rough stack of light sheets 230 onto a virtual fine stack of light sheets 260. The output virtual fine stack of light sheets 260 is then processed using the classic multi-angle reconstruction 135. The output is again referred to as a virtual reconstruction 250, since it is based on a virtual fine stack of light sheets 260. Accordingly, in the second embodiment, the target data differ from the target data of the first embodiment in that the target data of the second embodiment according to the alternative according to Fig. 8 (a) just comprise the light-sheet fine stack 220.

[0117] According to the second embodiment, candidate stacks can again be calculated when performing the classical multi-angle reconstruction 135, as described in relation to the first embodiment. In this embodiment, these would then be called virtual candidate stacks, and a best virtual reconstruction 250 is selected based on the virtual candidate stacks. In contrast to the first embodiment, however, the best virtual reconstruction 250 is not part of the training dataset; instead, the best virtual reconstruction 250 is determined during each inference.

[0118] According to an alternative design, the order in which the reconstruction model and the detail improvement model are applied can be reversed in the staged processing model. As in Fig. Figure 8(b) shows that the classical multi-angle reconstruction 135 is first applied to the coarse light sheet stack 230. A resulting reconstructed image stack 240 is also called the reconstructed coarse light sheet stack. Compared to a reconstructed image stack 240 determined based on the fine light sheet stack 220, which is also called the reconstructed fine light sheet stack, the reconstructed coarse light sheet stack exhibits more shadow artifacts or fringe artifacts. A second detail enhancement model 1402 is trained according to this configuration to map the reconstructed coarse light sheet stack to a fine light sheet stack. The fully trained second detail enhancement model 1402 should then output a virtual reconstruction 250 that has approximately the same number of shadow artifacts as the reconstructed fine light sheet stack. According to the design according to Fig.8 (b) the learning inputs comprise the reconstructed rough stack of light sheets, i.e. the reconstructed image stack 240 calculated from the rough stack of light sheets 230 using the classical multi-angle reconstruction 135, and the target outputs again comprise the reconstructed image stacks 240, but the image stack 240 reconstructed from the fine stack of light sheets 220.

[0119] The rough stack of light sheets can, in particular, comprise only a single image stack. Accordingly, in the case of the staged processing model, where the reconstruction model is applied first, classical multi-angle reconstruction would be applied only to the single image stack, and then the detail enhancement model would be applied to the reconstructed rough stack of light sheets. When applying classical multi-angle reconstruction to the single image stack, for example, if the multi-angle reconstruction involves averaging over several image stacks, the averaging would only occur for the single image stack. Often, however, classical multi-angle reconstruction also includes other steps, such as unfolding the images of the respective image stack (e.g., a Richardson-Lucy unfolding), or other image processing algorithms.If the rough stack of light sheets now comprises only the single image stack, the reconstruction model would also perform the unfolding, thereby improving the image quality to a certain extent compared to the unfolded image stack. During the training of the detail improvement model, the unfolded image stacks would be used as training input, and the image stack 240 reconstructed from the fine stack of light sheets would be used as the target output. Thus, as described above, the detail improvement model will use the single image stack as training input and the image stacks of the fine stack as target output.

[0120] The variations and designs described for the different figures can be combined with one another. The designs shown and described are purely illustrative, and modifications are possible within the scope of the attached claims. Reference symbol list 1 Image processing system 100 Imaging equipment 101 Light source 102 Beam expanders 103 lens 104 Lighting lens 105 Sample chamber 106 Detection lens 107 Camera 110 light leaf 115 Fluorescence light 120 Sample 121 Sample holders 125 opaque sample structure 126 shadows 127 Shadow object 130 Evaluation unit 131 evaluation modules 132 memory module 133 Control module 134 channels 135 classic multi-angle reconstructions 140 Machine learning model 141 Input layer 142 Intermediate shift 143 Output layer 150 data entries 151 interim issues 152 results data 200 image data 205 images 210 image stacks 220 light leaf fine stacks 230 light leaf coarse stacks 240 reconstructed image stacks 250 virtual reconstruction 260 virtual light sheet fine stacks 1401 first detailed improvement model 1402 second detailed improvement model QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Dong, D. et al in “Vertically scanned laser sheet microscopy”, J. Biomed. Opt. 19(10), 1 (2014

[0003] Dodt, H. U. et al beschreiben in „Ultramicroscopy: three-dimensional visualization of neuronal networks in the whole mouse brain,“ Nat. Methods 4(4), 331-336 (2007

[0004] Huisken, J. et al beschreiben in „Even fluorescence excitation by multidirectional selective plane illumination microscopy (mSPIM),“ Opt. Lett. 32(17), 2608-2610 (2007

[0005] Tainaka, K. et al in „Whole-body imaging with single-cell resolution by tissue decolorization,“ Cell 159(4), 911-924 (2014

[0006] Wei et al in „Elimination of stripe artifacts in light sheet fluorescence microscopy using an attention-based residual neural network“ Biomed Opt Express. 2022 Mar 1; 13(3): 1292-1311

[0007] Temerinac-Ott, Maja, (2012), „Multiview reconstruction for 3D Images from light sheet based fluorescence microscopy

[0084]

Claims

[1] Method for training an image processing system (1) with a machine learning model (140) to perform a virtual multi-angle reconstruction of image stacks (210) recorded with a light-sheet microscope, in particular of a sample (120) of a sample type, comprising: - Taking at least one light sheet fine stack (220) of a sample (120), wherein the light sheet fine stack (220) comprises several image stacks (210) and for different image stacks (210) a light sheet (110) illuminates the sample (120) at a different angle, - Determining target outputs based on the light-sheet fine stack, the machine learning model (140) and a classical multi-angle reconstruction (135), - Determining learning inputs based on a rough light sheet stack (230), the machine learning model (140) and classical multi-angle reconstruction (135), wherein the rough light sheet stack (230) comprises one or more image stacks (210) and for different image stacks (210) the light sheet illuminates the sample (120) at various angles, - Creating an annotated dataset encompassing the target outputs and learning inputs, - Optimizing the machine learning model (140) to perform the virtual multi-angle reconstruction (135) using the annotated dataset, characterized by , that - the light sheet coarse stack (230) comprises fewer images (205) than the light sheet fine stack (220), in particular fewer image stacks (210), and a virtual reconstruction determined by means of virtual multi-angle reconstruction has fewer image artifacts than a reconstructed image stack (210) determined by means of classical multi-angle reconstruction (135) from the light sheet coarse stack (230). [2] Method according to claim 1, wherein determining the learning inputs comprises one or more of the following: - Taking the rough stack of light sheets (230), wherein when taking the rough stack of light sheets (230) fewer different image stacks (210), fewer different angles or fewer images (205) per image stack (210) are taken, - Determining the coarse stack of light sheets (230) from the fine stack of light sheets (220), wherein the coarse stack of light sheets (230) is a true subset of the fine stack of light sheets (220), preferably the coarse stack of light sheets (230) comprises only every second, third or fourth of the image stacks (210) recorded when the fine stack of light sheets (220) was recorded, or for each image stack (210) of the coarse stack of light sheets (230) comprises only every second, third or fourth image (205) of the respective image stack (210), particularly preferably the coarse stack of light sheets (230) comprises only a single image stack (210). [3] A method according to one of the preceding claims, wherein, prior to the acquisition of the light sheet fine stack (220), one or more acquisition parameters are determined depending on a sample type for which the training is performed, such that areas in the sample (120) which are arranged behind opaque or partially opaque sample structures in the sample (120) are illuminated in at least one of the acquired image stacks (210) or are partially illuminated in at least several of the image stacks (210) of the light sheet fine stack (220), the acquisition parameters comprising one or more of the following acquisition parameters: - a height offset of height-offset images (205) in an image stack (210), - a number and angular distance of the different angles, - an exposure spectrum, - used fluorophores, and - Contextual information; and furthermore, in particular, the sample type is determined based on one or more of the following properties of opaque or partially opaque sample structures: - an expansion, - an opacity, - a spectral transmittance, - usable fluorophores, and - a density in the sample of the sample type. [4] A method according to any of the preceding claims, wherein the machine learning model (140) is a staged processing model or an overall processing model, the staged processing model comprises a detail improvement model (1401, 1402) and a reconstruction model, the reconstruction model is configured to compute the classical multi-angle reconstruction (135), the detail improvement model (1401, 1402) is trained to perform a detail improvement mapping using the annotated data set, and sequential execution of the classical multi-angle reconstruction (135) and the detail improvement mapping results in the virtual multi-angle reconstruction, depending on the order in which the detail improvement model (1401, 1402) and the reconstruction model are applied, the detail improvement model (1401, 1402) is either trained to perform,mapping a reconstructed light-sheet coarse stack (230) calculated using classical multi-angle reconstruction (135) to a reconstructed light-sheet fine stack (220) or mapping the light-sheet coarse stack (230) to the light-sheet fine stack (220) and determining the learning inputs and target outputs depending on the order in which the detail improvement model (1401, 1402) and the reconstruction model are applied, includes: selecting a reconstructed light-sheet coarse stack (230) or selecting a reconstructed light-sheet fine stack (220) as the learning inputs and selecting a reconstructed light-sheet fine stack (220) or selecting a light-sheet fine stack (220) as the target outputs, and when the machine learning model (140) is the overall processing model is, the overall processing model is trained directly to execute the virtual multi-angle reconstruction using the annotated dataset,in particular the virtual multi-angle reconstruction, the detail enhancement mapping and the classical multi-angle reconstruction (135), and the determination of the annotated data set includes: selecting a rough light sheet stack (230) comprising at least one image stack as training inputs and selecting at least one reconstructed fine light sheet stack (220) determined from the fine light sheet stack (220) by means of the classical multi-angle reconstruction (135) as the target outputs. [5] Method according to any of the preceding claims, comprising determining the target outputs from the light sheet fine stack (220): - Computation, using classical multi-angle reconstruction (135), of several reconstructed candidate stacks, wherein for each of the several reconstructed candidate stacks a different set of reconstruction parameters of classical multi-angle reconstruction (135) is used, wherein, for example, a reconstruction algorithm used, a number of iterations in the application of the reconstruction algorithm, correction procedures used, or correction parameters of the correction procedure used are selected by means of the parameters used. - Reviewing the candidate stacks, and - Selecting target outputs from the reconstructed candidate stacks. [6] Method according to one of the preceding claims, wherein the optimization of the machine learning model (140) comprises augmenting the annotated data set or simulating further data using a point spread function and the target outputs of the annotated data set, wherein the point spread function is, for example, a depth-variant point spread function. [7] Method according to one of the preceding claims after optimizing the machine learning model (140) further comprising checking the machine learning model (140) to see if the machine learning model (140) is suitable to reconstruct the light sheet rough stack (230) using the learned virtual multi-angle reconstruction. [8] Method for performing a virtual multi-angle reconstruction of image stacks (210) acquired with a light-sheet microscope using an image processing system comprising a machine learning model (140), comprising: - Providing a machine learning model (140) for performing the virtual multi-angle reconstruction, wherein a machine learning model (140) is used which has been trained according to the method for training an image processing system according to one of the preceding claims, - Recording, using a light-sheet microscope, a rough stack of light sheets (230) to be processed, comprising at least one image stack (210) of the sample (120) of the sample type, - Calculating a virtual reconstruction from the light sheet rough stack (230) using virtual multi-angle reconstruction. [9] Method according to the preceding claim 8, wherein the method further comprises, prior to calculating a virtual reconstruction: - Checking whether one or more machine learning models (140) are suitable for reconstructing the rough stack of light sheets (230) to be processed using the respective learned virtual multi-angle reconstruction, and, if no suitable machine learning model (140) is available: - Performing the method for training an image processing system according to one of the preceding claims 1 to 6 with the sample (120), wherein in particular the picking up of the light sheet fine stack (220) takes place in a predetermined area of ​​the sample (120) and in particular the picking up of the light sheet coarse stack (230) to be processed takes place in an area of ​​the sample (120) different from the predetermined area. [10] Method according to any one of the preceding claims, comprising checking the one or more machine learning models (140): - Inputting the rough stack of light sheets (230) into the machine learning model (140) and checking the output virtual reconstruction, including: - a manual check, - a comparison with sample target outputs, - Determining a quality metric, in particular based on image properties of the virtual reconstruction, for example image sharpness, edge sharpness, number of image artifacts such as stripe artifacts, in particular by means of a metric quality model, wherein the metric quality model is set up, in particular trained, to identify image properties, - input into a quality classification model that has been trained to classify virtual reconstructions and / or classical multi-angle reconstructions (135), - Check based on image features such as image sharpness, noise level, blood-bleed, artifacts such as ring and stripe artifacts. [11] Image processing system comprising an evaluation unit (130) for performing a method according to any one of the preceding claims 1 to 10. [12] Image processing system according to claim 11, further comprising an imaging device, in particular a light-sheet microscope. [13] Computer program product comprising instructions which, when the program is executed by one or more computers, cause them to execute the method according to any one of the preceding claims 1 to 10, the computer program product being in particular a computer-readable storage medium.

Images