Method for operating light sheet microscope and apparatus therefor

Abstract

A first aspect of the present disclosure relates to a method for operating a light sheet microscope, comprising the steps of capturing an image of a body block by exciting the body block with one or more light sheets and by capturing a light sheet image of each of the one or more excitation light sheets, the one or more light sheets are inclined relative to the optical axis of the main lens of the microscope and are emitted at different positions in the body block; converting the captured image from the camera space to the user space by mapping information from the one or more light sheet images to one or more single view focal planes; displaying a volume block in the user space at the user interface; receiving, from the user interface, selection information about a portion of the volume block in the user space; reconverting the selection information from the user space to the camera space to control the microscope to capture an image based on the selection information; a selected image of a portion of the related volume block is captured.

Description

Technical Field

[0001] The present disclosure relates to methods for operating a light sheet microscope. Also disclosed are devices such as light sheet microscopes and devices configured to interact with light sheet microscopes. Background Art

[0002] Light sheet microscopy is a fluorescence microscopy technique that uses one or more light sheets that are oriented in a direction different from the observation direction (i.e., the optical axis of the main lens) to illuminate the sample. Compared with traditional point scanning methods, this method allows imaging of various samples (such as embryos or organoids) with minimal photodamage and photobleaching. By selectively illuminating thin sections of the sample each time, high-resolution images with improved contrast and depth can be obtained. However, when using oblique light sheets, the images generated are not intuitively understandable to the user. This affects the operation of such devices. Therefore, improvements are needed. Summary of the Invention

[0003] The object of the present disclosure is to improve work with light sheet microscopy.

[0004] This object is achieved by the disclosed embodiments, which are particularly defined by the subject matter of the independent claims. The dependent claims provide information on further embodiments. The following summary and detailed description also disclose various aspects and embodiments of these aspects, which provide additional features and advantages.

[0005] A first aspect of the present disclosure relates to a method for operating a light sheet microscope,

[0006] The following steps are involved:

[0007] - capturing an image of a volume in camera space by exciting the volume with one or more light sheets, which are tilted relative to the optical axis of a main lens of the microscope and emitted at different positions in the volume, and by capturing a light sheet image of each of the one or more excitation light sheets;

[0008] - transforming the captured image from the camera space to user space by mapping information from one or more light sheet images to one or more monoscopic focal planes;

[0009] - displaying the volume in the user space on a user interface;

[0010] - receiving, from the user interface, selection information about a portion of the volume in the user space;

[0011] - converting the selection information from the user space to the camera space to control the microscope to capture an image based on the selection information;

[0012] - capturing selected images of portions of the volume.

[0013] Light-sheet microscopy is configured to illuminate a sample from the side using a sheet of light (light sheet). In particular, one or more light sheets can be emitted so that they intersect the imaging plane of the detection lens. A light-sheet microscope can be a fluorescence microscope. Light-sheet microscopy allows for the selective illumination of thin sections of a sample, thereby minimizing damage and photobleaching of samples outside the focal plane. Furthermore, light-sheet microscopy enables high-resolution and / or three-dimensional imaging of sample bulk, particularly living samples.

[0014] A volume can be any sample or portion of a sample that can be analyzed using light-sheet microscopy. In particular, a volume can be a portion of a sample specified in user space. The volume is sampled by exciting the volume using multiple light sheets at different locations or by moving a light sheet and then capturing the light emitted from the volume with a sensor (such as a camera). The sampled volume generates a volume dataset in camera space.

[0015] The camera space is the space in which a camera (or multiple cameras) captures the sample. The camera space represents "what the system sees" or "what the system can see". The image in the camera space may comprise multiple image parts, where each part is associated with a specific light sheet and / or a specific light sheet position (e.g. in the case of a moving light sheet). An image generated using excitation with a specific light sheet is also referred to as a light sheet image. Each light sheet image represents a sampling point of the entire image. For example, an image may be sampled by capturing ten consecutive light sheet images of a sample via a microscope objective, where the light sheet is moved continuously over the sample volume. In this case, the moving light sheet illuminates a different part of the sample volume for each light sheet image. This results in a different light sheet detection plane for each light sheet image. In order to capture multiple light sheet images, a rolling shutter may also be used.

[0016] User space is a space in which a user (or another entity) can perceive or receive sampled data in a more natural manner. A camera "sees" an image in camera space, and therefore samples the image through a light sheet that is non-parallel (e.g., tilted) to the optical axis of the camera lens. That is, by capturing the image using multiple light sheet images, the camera "sees" the volume from multiple perspectives. Therefore, depicting the entire image in camera space is not suitable for human users or entities seeking to further process the image in a user-based, single-perspective reference frame. Therefore, user space is a transformation of camera space (or of an image captured in camera space) that is more intuitive to the user. To transform an image from camera space to user space, that is, to determine the image of a sampled volume / sample in user space, information from different sampling points (i.e., from different light sheet images) is transformed into one or more single-perspective focal planes. By transforming information from multiple sampling points into a single-perspective focal plane, a two-dimensional image can be generated. By transforming information from multiple sampling points into multiple single-perspective focal planes, a three-dimensional image can be generated.

[0017] The user interface or manual system interface may be an interface at the microscope and / or another device that communicates with the microscope or at least can receive data from the microscope.

[0018] The selection information is related to information in the user space (e.g., regarding the sample volume being analyzed). The selection information can be two-dimensional information and / or three-dimensional information. The selection information can be obtained through a user interface, for example, provided by the user. Additionally or alternatively, the selection information can be provided automatically, for example, by a feature recognition algorithm. For example, the feature recognition algorithm can provide the selection information, and the information can be displayed to the user within an image of the sampled volume in the user space. In this case, the user can confirm, modify, and / or delete the suggested selection information.

[0019] Based on this selection information, the parameters of the microscope (i.e., parameters in camera space) can be obtained to sample the corresponding image (in camera space). These parameters can be based on further parameters such as sampling resolution, exposure time, sampling width, etc. The parameters for the microscope can be obtained by the inverse transformation of the transformation from camera space to user space. Based on the information in camera space, the microscope can be controlled so that the selected information can be sampled from the volume.

[0020] By providing the sampled information to the user in a transformed manner (i.e., in user space), the microscope user can operate it similarly to a normal widefield microscope and can intuitively select information from the sampled volume for further analysis. The first advantage is that the tilted raw data volume can be set directly during acquisition, allowing the user to use conventional 3D visualization of the sample, rather than only obtaining the tilted raw data volume in post-processing. In a particular embodiment, the creation of the volume representation can be performed as follows:

[0021] 1) Automatically record sample volume;

[0022] 2) Erect the volume and display it to the user in the HSI, thereby:

[0023] 2.1) Displaying 3D images;

[0024] 2.2) Display x / z projection;

[0025] 2.3) Display y / z projection (or x / y projection);

[0026] 3) Select a rectangle in the 2D projection of the relevant area;

[0027] 4) Transform the selected ROI by calculating the ROI back to the recording coordinate system;

[0028] 5) Determine the ROI in camera space;

[0029] 6) Determine the range of sampling devices (such as galvanometer mirrors);

[0030] 7) Based on a given exposure time, determine the minimum speed and maximum speed of the sampling device;

[0031] 7.1) If necessary, adjust the exposure time based on the minimum / optimum speed of the sampling device;

[0032] 8) If applicable, consider different modes of sampling the device, such as continuous mode versus triggered mode, and compare their results.

[0033] As a result, users can interact with conventional 3D volumes in a Cartesian system, rather than having to consider the complex mathematical calculations and optimal parameter relationships required for light-sheet microscopy. This allows users to obtain the best possible volume with the ideal sampling setup. Previously projected regions can be updated to visualize parameter changes compared to the existing setup, and their effects can be intuitively evaluated based on previously recorded volumes without having to re-expose the sample, thus reducing light stress on the sample.

[0034] Embodiments of a first aspect of the present disclosure relate to a method for operating a light sheet microscope,

[0035] The camera space is based on the predefined angles in the light sheet relative to the optical axis of the main lens. The primary lens captures light from the volume by one or more light sheets.

[0036] By using a sample pattern of an oblique light sheet, a light sheet microscope can operate using a single primary lens (the lens located directly in front of the sample) to both emit the light sheet and capture the reflected light from the bulk.

[0037] Embodiments of a first aspect of the present disclosure relate to a method for operating a light sheet microscope,

[0038] Wherein, the one or more light sheets are emitted in a pattern based on regular positions.

[0039] Regular position-based patterns can be formed using multiple light sheets that emit substantially parallel light and / or where adjacent light sheets are always spaced the same distance apart. Using a single light sheet moving across a volume, a regular position-based pattern can be formed using a constant velocity. Using a regular position-based sampling pattern provides a simple and efficient sampling method.

[0040] Embodiments of a first aspect of the present disclosure relate to a method for operating a light sheet microscope,

[0041] wherein the one or more light sheets are emitted in a pattern based on irregular positions.

[0042] Irregularly positioned patterns can be formed using multiple light sheets that emit light at varying distances between adjacent light sheets. When a single light sheet is moved across the volume, the speed of the light sheet can be varied to create an irregularly positioned pattern. Using irregularly positioned sampling patterns allows for sampling patterns tailored to the complexity of the sample volume; for example, more light sheets can be used to sample a more complex section than a less complex section.

[0043] Embodiments of a first aspect of the present disclosure relate to a method for operating a light sheet microscope, wherein a user space is based on one or more of the following parameters:

[0044] - the angle between the light sheet and the optical axis of the main lens;

[0045] - the distance between two adjacent light sheets;

[0046] - the speed at which the volume is sampled based on one or more light sheets.

[0047] The goal of user space is to display the volume / sample as if the user were looking at it through a wide-field microscope. Therefore, an adequate transformation of the information captured in camera space is required to map the different dimensionalities introduced by light-sheet-based sampling into a consistent single-view 2D or 3D user space.

[0048] The corresponding transformation can be based on the angle between the emitted light sheet and the optical axis of the main lens. For example, based on the angle, a sampling point that provides information for a specific point in the user space (i.e., an image of the sample generated based on different light sheets) can be selected.

[0049] Additionally or alternatively, the transformation from camera space to user space can also be based on the distance between two adjacent light sheets. For example, when the distance between two adjacent light sheets is relatively small (e.g., <20 μm), a point in user space located between the two light sheets can be generated by combining information about the position of two light sheet images generated based on the two light sheets. Alternatively, a point in user space can be based on a corresponding point in an image (in camera space) based on the closest light sheet (e.g., the light sheet with the smallest distance to the point).

[0050] Additionally or alternatively, the conversion from camera space to user space can also be based on the sample speed. For example, the light sheet can be moved through the volume at a predefined speed. This speed can be substantially constant, for example, 10 ns for the entire sampling line (the line sampled by the volume / sample) in the case of a rolling shutter.

[0051] Controlled speed illumination of the sample volume can be performed to achieve a certain exposure time. For longer exposure times, and when using a single moving light sheet, the light sheet needs to be moved through the volume at a slower speed than for shorter exposure times. Additionally or alternatively, the required or expected exposure time can vary depending on the position of the light sheet at the sample. The speed of the light sheet then needs to be adjusted according to the desired exposure time. The duration that the light sheet illuminates the volume can vary depending on the position of the light sheet. Based on this functionality, the speed of the device that moves one or more light sheets through the volume / sample can be determined. Such methods can be based on galvanometer mirrors. Additionally or alternatively, such a device can be a moving sample holder.

[0052] An embodiment of a first aspect of the present disclosure relates to a method for operating a light sheet microscope, comprising the following steps:

[0053] - Transform information from camera space to user space so that the focal planes of the emitted light sheet and user space differ by an angle less than approximately 90°.

[0054] The system "sees" the volume through different images based on multiple tilted light sheets. Therefore, the system sees the volume through predefined perspectives. To transform this information into user space, i.e., monoscopic perspective, the camera space information needs to be transformed to compensate for the angle of the light sheet relative to the user's viewing axis. This can also be referred to as "erecting" the information in camera space to achieve perspective on the illuminated volume in user space.

[0055] For example, such "erecting" is performed by a transformation matrix that associates each pixel captured in camera space with a pixel in user space. Thus, one or more pixels selected from camera space can be implemented. Additionally or alternatively, multiple pixels in camera space can be interpolated to determine the pixel in user space. Thus, the pixels of the sample volume in camera space are moved to the new position, thereby completing the "erecting". The transformation can include one or more filters to remove artifacts from the image in user space.

[0056] The angle calibration between camera space and user space can be different, for example, for a single device and / or across different devices. Calibration can therefore be performed to control a predefined angle between camera space and user space. This is particularly necessary for the combination of continuous galvanometer motion and the use of a rolling shutter in the camera to increase acquisition speed. Calibration can be software-controlled. Its purpose is to detect and correct errors caused by, for example, lens aberrations. For performance reasons, calibration can be done in a single step.

[0057] An embodiment of a first aspect of the present disclosure relates to a method for operating a light sheet microscope, comprising the following steps:

[0058] - Mapping user space in a 3D Cartesian coordinate system;

[0059] - display of the xy plane of the coordinate system, in particular as a maximum intensity projection;

[0060] - Receive user selection in the xy plane.

[0061] User space can be mapped in a Cartesian coordinate system with x, y, z dimensions.

[0062] Maximum intensity projection is a technique used in imaging, particularly medical imaging, to visualize high-intensity structures within a sample volume. The method can project voxels with the highest intensity values ​​along a specific viewing direction or in a specific plane. By means of maximum intensity projection, a display of a first plane of a coordinate system can be performed. In one example, an xy plane can be defined on one side of the sample being sampled. The voxels with the highest intensity of the sample being sampled can then be depicted in this xy plane. This can enhance the user's orientation. Additionally or alternatively, the xy plane can be configured to be movable by the user via an artificial system interface. In this way, the user can position the xy plane as desired, and within the sample being sampled, and highlight the intersection of the sampled volume.

[0063] The coordinate system can be aligned with the microscope's camera system. In particular, the coordinate system's axes can be parallel to the sampling direction, i.e., the direction in which one or more light sheets sample the volume. This facilitates user-selectable transformations from user space to camera space. For example, a user-selected distance can specify the range of movement of a galvanometer-based mirror to sample the selected volume.

[0064] An embodiment of a first aspect of the present disclosure relates to a method for operating a light sheet microscope, comprising the following steps:

[0065] - Mapping user space in a 3D Cartesian coordinate system;

[0066] - display of the xz plane of the coordinate system, in particular as a maximum intensity projection;

[0067] - Receive user selection information within the xz plane.

[0068] The xz projection defines a plane perpendicular to the xy projection in the Cartesian coordinate system. The xz projection can be generated and displayed to the user in the same manner as described for the xy projection in the previous embodiment. This embodiment can be combined with the previous embodiment in particular to select a volume. Selection information can be automatically generated based on the intersection of the selected xy plane and the selected xz plane. The selection can be directly converted back to camera space. Alternatively, further processing can be performed before the conversion is performed, for example, it can be checked whether the camera system of the microscope used can capture the selected volume.

[0069] By selecting a certain block in the sample block, a region of interest (ROI) in the camera space can be determined. By sampling only the defined region of interest, the sampling process can be accelerated.

[0070] Embodiments of a first aspect of the present disclosure relate to a method for operating a light sheet microscope,

[0071] Therein, one or more parameters of the microscope, in particular the light sheet sampling range, are depicted in user space.

[0072] If the technical parameters in the galvanometer mirror change, this can be reflected in user space. For example, this can be reflected in user space by updating one or more user-selected regions of interest (ROIs) in the projection to represent the changes made to the previously calculated parameter set. A change in the size of the camera ROI will result in a change in the ROI of the corresponding projection (in user space). Changing the step size of the galvanometer mirror can update the number of sample steps and / or the ROI in the corresponding projection. In the event that the start and / or end point of the scan range changes in camera space, the potential blocks that can be sampled can change their size in user space.

[0073] Embodiments of the first aspect of the present disclosure relate to a method for operating a light-sheet microscope, wherein based on selection information and / or further information, in particular further user input, one or more of the following microscope parameters are determined:

[0074] - starting point of the sampling motion of one or more light sheets;

[0075] - the end point of the sampling movement of one or more light sheets;

[0076] - the distance between two adjacent light sheets;

[0077] - the speed at which one or more light sheets sample the volume;

[0078] -The intensity with which the light sheet excites the volume.

[0079] If the coordinate system in the user space is aligned with the sampling direction (eg the direction of a galvanometer mirror providing one or more light sheets), it is particularly straightforward to determine the start and / or end point of the sample range.

[0080] In particular, the distance between two adjacent light sheets varies depending on the extent of the volume sampled. For example, if a portion of the volume contains greater complexity, the distance can be smaller. In areas with less complexity, the distance between two adjacent light sheets can be greater. Complexity can thus be related to the arrangement and organization of components within the sample. For example, in biological samples, this could refer to cellular structure, organelle arrangement, and / or the presence of specialized structures within the cell. In materials science, it could refer to the arrangement of molecules, crystals, or fibers.

[0081] In particular, the rate at which a sample is sampled can be correlated to the required or desired exposure of a particular portion of the sample or the entire sample.

[0082] An embodiment of a first aspect of the present disclosure relates to a method for operating a light sheet microscope, comprising the following steps:

[0083] - obtaining a sampling configuration of the microscope for sampling the volume;

[0084] - Displaying one or more effects of the sampling configuration at an artificial system interface in user space.

[0085] The sampling configuration can be related to one or more of the following parameters: starting position, ending position, sampling steps, speed and / or exposure. Thus, the user can advantageously see the impact of a specific configuration, parameter changes and / or various sampling configurations.

[0086] Embodiments of a first aspect of the present disclosure relate to a method for operating a light-sheet microscope, wherein a sampling configuration is obtained from a user, in particular via a manual system interface.

[0087] In particular, the sampling configuration may be provided together with a selection of a volume of sample to be sampled. For example, this information may be provided by a user via a manual system interface.

[0088] A second aspect of the present disclosure relates to a device, in particular a light sheet microscope, configured to:

[0089] - obtaining an image of the volume in camera space, wherein the image comprises a plurality of light sheet images based on light sheets that have excited the volume and are tilted relative to the optical axis of a main lens of the microscope and are emitted at different locations in the volume;

[0090] - transforming the obtained image from camera space to user space by mapping information from one or more light sheet images to one or more single-view focal planes;

[0091] - displaying the volume in the user space at a user interface;

[0092] - receiving selection information about the portion of the volume in the user space from the user interface;

[0093] - Convert selection information from user space to camera space and then to microscope;

[0094] - providing selection information for reconversion in order to capture an image of that portion of the volume.

[0095] In particular, the light sheet microscope according to the second aspect of the present disclosure can implement the method according to the first aspect of the present disclosure. The light sheet microscope can include a computer for performing conversions from camera space to user space, conversions from user space to camera space, providing images to a user interface, and / or determining a selected image. The computer does not necessarily need to be in close proximity to the microscope's optical setup. It can be an off-premises computer.

[0096] The device according to the second aspect of the present disclosure can also be a computing device that obtains data from a microscope and provides the data to the microscope. In a specific embodiment, the network device can control multiple microscopes by receiving information about the sample from one microscope and then converting the selection information to the microscope hardware (camera space) of different microscopes. These microscopes can be different, so the conversion from user space to camera space can be different for each controlled microscope. Thus, multiple identical / similar samples can be analyzed based on selections on a single artificial user interface.

[0097] The second aspect of the present disclosure relates to a device.

[0098] wherein the device is a light sheet microscope, and

[0099] Therein, the light sheet is emitted and the image of the volume is obtained through the same primary lens.

[0100] In standard light-sheet microscopy, the light sheet is perpendicular to the optical axis of the detection lens and is provided by an additional lens. In light-sheet microscopy with a single lens, the light sheet can be emitted at an angle less than 90° relative to the optical axis of the main lens and pass through it without the need for an additional lens.

[0101] Embodiments of a second aspect of the present disclosure relate to an apparatus, wherein the apparatus is a light sheet microscope including a galvanometer system, and the galvanometer system is used to guide the light sheet to sample an image of a volume.

[0102] In an oblique light sheet microscope, a single light sheet can be used that moves through the sample volume, or multiple light sheets can be emitted at different positions along the sampling range. This can be achieved using a movable sample. Additionally or alternatively, the light sheet can be movable. This can be accomplished using a galvanometer mirror that moves a single light sheet along or through the sample volume, or focuses different light sheets at different positions along the sample volume. By using a galvanometer mirror, the light sheet can be moved rapidly. The galvanometer can be controlled to achieve a desired position or speed of the light sheet and / or a desired exposure of a portion of the sample volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Further advantages and features are obtained from the following examples, some of which are illustrated with reference to the accompanying drawings. These drawings do not always show the embodiments to scale. In particular, the dimensions of individual components may be exaggerated or reduced for clarity of description. For this reason, the drawings are at least partially schematic.

[0104] Figure 1 Oblique light sheet microscopy is shown.

[0105] Figure 2 The operation of a light sheet microscope according to an embodiment of the present disclosure is shown.

[0106] Figure 3 The erection process according to an embodiment of the present disclosure is shown.

[0107] Figure 4 The erection process according to an embodiment of the present disclosure is shown.

[0108] Figure 5 The erection process according to an embodiment of the present disclosure is shown.

[0109] Figure 6 The selection process in the user space according to an embodiment of the present disclosure is shown.

[0110] Figure 7 The selection process in the user space according to an embodiment of the present disclosure is shown.

[0111] Figure 8 The selection process in the user space according to an embodiment of the present disclosure is shown.

[0112] Figure 9 A microscope system for use with an embodiment of the present disclosure is shown.

[0113] In the following description, reference is made to the accompanying drawings which form a part of this disclosure and which illustrate specific aspects that can be understood of the present disclosure. Like reference numerals designate components that are identical or at least similar in function or structure.

[0114] In general, the disclosure of the described method also applies to the corresponding device (or apparatus) for performing the method or the corresponding system including one or more devices, and vice versa. For example, if a specific method step is described, the corresponding device may include a component for performing the described method step, even if the component is not explicitly described or represented in the figure. On the other hand, for example, if a specific device is described based on a functional unit, the corresponding method may include one or more steps for performing the described function, even if such steps are not explicitly described or represented in the figure. Similarly, the system may be provided with corresponding device components or components for performing specific method steps. Unless otherwise expressly stated, the features of the various exemplary aspects and embodiments described above or below may be combined. DETAILED DESCRIPTION

[0115] Figure 1 The operation of an oblique light sheet microscope 100 is shown. The microscope observes a sample volume 102 placed on a sample holder 104, which may include, for example, a coverslip. A light sheet 112 is directed onto the sample 102 via a primary lens 110. The primary lens 110 is also configured to observe light 106 reflected from the sample in order to sample the sample 102. The reflected light passes through a series of lenses 120, 122, 126, 128, 138 and, via a secondary objective 134 and a tertiary lens 136, is ultimately captured by a camera 140. Alternatively, the light provided by the tertiary lens 136 can also be observed by two cameras at different locations. In this case, two different reconversions may be required.

[0116] The light 118 for the light sheet 112 originates from the illumination module 132 and is deflected into the optical system of the microscope by the dichroic mirror 130. The light then passes through the lenses 128, 126, 122, 120 to the main lens 110. With the main lens, the light sheet is oriented at an angle relative to the optical axis 116. Focused on sample volume 102. The light sheet is positioned via a galvanometer mirror 124 so as to sample the entire sample 102 perpendicularly to the optical axis 116 of the main lens. The galvanometer mirror can be operated in continuous motion, particularly in combination with a rolling shutter. Alternatively, the mirror can focus the light sheet at discrete positions. The latter alternative is slower than the first because the position needs to be adjusted, which takes time. On the other hand, a rolling shutter requires synchronization of the mirror speed with the effective pixel speed of the camera sensor.

[0117] Figure 2 The operation of a light sheet microscope according to an embodiment of the present disclosure is shown. The figure includes four sub-figures a) to d) showing the "erecting" process of a sample volume illuminated by multiple light sheets.

[0118] Sub-figure a) shows the object space as it appears during sampling near the main lens 202 of a microscope. An oblique light sheet 204 is projected through the main lens 202 onto a portion of a sample volume 212 disposed on a sample holder 206. Based on the illumination of this portion of the sample volume 212 by the light sheet 204, a light sheet image 210a can be captured by the microscope. This operation is repeated with multiple light sheets projected along the scan axis 230. The different images 110a obtained are at angles 210 to the plane of the sample holder. (234) Tilt. This tilt causes a lateral shift (offset) 232 between two consecutive images.

[0119] Sub-figure b) shows the camera space. The camera space can be the perspective of the microscope, which "sees" the sample volume 212 through multiple images (sampling "points"), and each image uses light from an inclined light sheet. The camera space can also be seen as a representation of the sample being sampled on the sensor or in the memory. The light sheet images 210a are arranged next to each other into image 210b, and an overall representation 240 of the sample volume is obtained. This can be a representation of the sampling information stored in the memory. It can also be a representation of the sample captured by the camera sensor. In the camera space, the image 210b is no longer offset. However, the information of the sample 212 is not consistent from the user's perspective. The parts of the sample that belong together are not arranged together in the camera space. Therefore, the camera space is not suitable for the user to analyze the sample and adjust the microscope for further images.

[0120] Sub-figure c) shows a first user space 250 in which the sampling information of the sample 212 is transformed to achieve a user-viewpoint perspective, i.e., the sample is depicted in a perspective that is physically consistent, although not necessarily identical to that in object space. For the user space, the information of image 210b is transformed into image 210c so that the content of image 210c is consistent with a single-viewpoint perspective. This may result in a rotation 236 of the depicted sample in user space relative to the sample in object space, such that one side of the sample is laterally displaced by a distance 238. In addition, when depicting the transformed volume in user space, the size of the transformed volume can be increased. This will be explained in detail later.

[0121] Sub-image d) shows a further version of user space 260, in which the image of sub-image c) in user space is rotated about angle 236. This has the advantage that the orientation of the sample in user space is equal to the orientation of the sample in object space (sub-image a)). In other words, the user perceives the sample in the same orientation as if it were located in front of the microscope.

[0122] Figure 3 A three-dimensional erection process 300 according to an embodiment of the present disclosure is shown. The sample is arranged in object space in a Cartesian reference frame 302, which includes three orthogonal axes x, y, and z. Based on multiple light sheets projected at different positions, multiple oblique sampled volumes 304 can be captured by a light sheet microscope camera.

[0123] The user space representation is depicted in a Cartesian reference frame 312. Converting the captured volume 304 into user space involves tilt correction / straightening the data 304 recorded at an angle. As a result, a new vertical ("upright") volume 314 can be formed. Thus, the information (e.g., voxel) at each location in the vertical volume 314 is based on information from one or more obliquely sampled volumes 304. For example, a voxel in the vertical volume 314 can be based on an interpolation of data from several obliquely sampled volumes that carry information for a corresponding location in the vertical volume 314.

[0124] Based on the erected information 314, a focal plane 316 may be selected that is orthogonal to the optical axis of the main lens.

[0125] Figure 4 A two-dimensional erection process 400 according to an embodiment of the present disclosure is shown. In the upper sub-figure, a sample 402 is sampled based on multiple tilted light sheets 404. A virtual focal plane as would be seen through a wide-field microscope is depicted from the side as line 406.

[0126] In the lower subfigure, the erection process is illustrated two-dimensionally in object space. Sample 402 and virtual focal plane 406 are framed within Cartesian axes x and y. Nine images captured based on nine light sheets 404 are shown along their centerlines 410. Voxels 412, 414, and other voxels along the same line define a single-view focal plane (in this case, this plane is simply a line, as only two dimensions, x and y, are depicted in the figure). Left voxel 412 can be inferred based on first image (sample point) 410a. The leftmost of the three voxels 412 is located far from image 410a. Therefore, the image may not carry much information about this voxel, and the voxel will remain quite dark. The right voxel of the three voxels 412 is located immediately adjacent to the first light sheet image 410a and can therefore be interpolated with good quality based solely on the first light sheet 410a. This voxel can be interpolated based on the shortest distance to light sheet image 410a. Voxel 414 is located between images 410b and 410c. Therefore, voxel 414 can be determined based on interpolation based on images 410b and 410c.

[0127] Figure 5 Operations 500 of a light sheet microscope according to an embodiment of the present disclosure are shown, wherein details of user space transformation are explained.

[0128] The volume sampled in user space is shown as a rectangle at 502. The plane of the first light sheet 510 emitted by the microscope during a sample operation is depicted within volume 502. This light sheet is tilted at an angle 516 and has a projected width 512 (e.g., 150 μm) in the Cartesian coordinate system in the x-direction and a projected height 514 (e.g., 132 μm) in the y-direction. The plane of the last light sheet 520 for the sample operation has a projected width 522 in the x-direction and a projected height 524 in the y-direction. The values ​​of the last light sheet 520 can be equal to or at least similar to the values ​​of the dimensions of the first light sheet 510.

[0129] The sample operation is performed using a galvanometer mirror with an operating width 530 (e.g., 354 μm). The sum of the operating and projected widths 512 (or 522) using 530 provides the total length of the sampled volume in user space. Thus, due to the tilted sampling, a sampled volume using 354 μm in object space (e.g., due to the maximum scan range) has a length in user space of 354 μm + 150 μm = 504 μm.

[0130] It is possible to constrain the volume in user space to match the width of the volume in camera space. However, this would mean discarding information that has already been sampled and could potentially harm the sampled volume due to phototoxicity. On the other hand, using all sampled information could result in blank spaces, especially in the corners of the sampled volume.

[0131] Figure 6 Figure 1 illustrates a selection process in user space according to an embodiment of the present disclosure. Subfigure a) shows a sampled sample volume 602. In particular, this sampled volume can be the maximum sampling volume, i.e., the volume shown is based on the maximum scanning range of the light-sheet microscope. Subfigures b) and c) each depict a two-dimensional projection of volume 602.

[0132] Sub-figure b) depicts a projection onto the xy plane 612. The xy plane projection 612 corresponds to the viewpoint 604 of the 3D volume 602. Sub-figure c) depicts a projection onto the xz plane 622. The xz plane projection 622 corresponds to the viewpoint 606 of the 3D volume 602. Both projections 612 and 622 are shown by a user-facing human system interface (HSI). Furthermore, the HSI can show the entire volume 602. The HSI also depicts the ROIs 614 and 624 in the projection planes and the resulting ROI 608 in the volume.

[0133] The user can now indicate a specific area, particularly a rectangular area, in the projections 612, 622. Thus, the user can define an ROI 608 within the sampled volume 602. Since the user displays this information in user space, the user does not need to consider the tilt of the light sheet, the angle with respect to the detection plane, and / or the sampling range used for sample sampling, such as the range of a galvanometer mirror.

[0134] After the user has selected the desired information, the selected information is automatically reconverted from the user space to the microscope's camera space for sampling. The microscope need not be co-located with the computer running the software for converting and / or mapping the sampled volume 502 and / or reconverting the selected information. Based on the reconverted selected information, the selected volume can be sampled. Advantageously, by sampling only a portion of a particular sample volume, sampling time can be shortened.

[0135] Figure 7 Shown in more detail in user space Figure 6 selection process. Sub-figure a) shows the largest sample block 602 that was sampled, and sub-figure b) and sub-figure c) depict two-dimensional projections 612, 622 of the block 602. The regions of interest (ROIs) 614, 624 identified in the xy plane and the xz plane are the areas indicated in the projection planes. Based on these ROIs, two auxiliary blocks are calculated. Each auxiliary block has a rectangular shape because the indicated ROIs 614, 624 are also rectangular. The first auxiliary block based on the selected ROI of sub-figure b) is defined by its corners 704a, 704b, 704c, 704d. The second auxiliary block based on the selected ROI of sub-figure c) is defined by its corners 706a, 706b, 706c, 706d. The selected block 608 is determined based on the intersection of the two auxiliary blocks.

[0136] Based on the identified volume, the light sheet angle can be used to calculate the correct ROI and scanning distance of the tilted light sheet in camera space, as well as the start and end points of the galvanometer mirror movement. The generated light sheet volume can include, in particular, the minimum number of light sheets required to capture the selected volume 602 in object space.

[0137] Figure 8 FIG. 4 shows a selection process in user space according to an embodiment of the present disclosure. Figure 6 and Figure 7Similarly, sub-figure a) shows the largest sample volume 802 that was sampled, and sub-figures b) and c) depict two-dimensional projections 812 and 822 of volume 802. In this case, the ROI is selected based on the projected planes xy and yz, as in the embodiment of the previous figure. However, ROIs 814 and 824 are larger than in the embodiment of the previous figure, resulting in a larger volume of 3D ROI 804. In addition, in the xy projection 812 of the ROI selection, planes 816 of the light sheet-based image (i.e., sampling points) are indicated. In particular, these light sheet-based planes may vary depending on the configuration of sampling parameters (such as the distance between two adjacent sampling points, light sheet speed, exposure time, etc.).

[0138] Essentially, a pixel can be the step size or distance the galvanometer mirror moves during the exposure time. If a larger step size or a faster speed is chosen for the galvanometer mirror, the spatial scan of the sample may need to be visualized again using the new settings so that the user can determine if this is appropriate for the sample.

[0139] Some embodiments relate to a microscope comprising a combination of Figures 1 to 8 Alternatively, the microscope may be combined with one or more of the systems described herein. Figures 1 to 8 is part of or connected to one or more of the described systems.

[0140] Figure 9 A schematic diagram of a system 900 for performing the methods described herein is shown. System 900 includes a microscope 910 and a computer system 920. Microscope 910 is configured to capture images and is connected to computer system 920. Computer system 920 is configured to perform at least a portion of the methods described herein. Computer system 920 can be configured to execute a machine learning algorithm. Computer system 920 and microscope 910 can be separate entities, but can also be integrated into a common housing. Computer system 920 can be part of a central processing system of microscope 910, and / or computer system 920 can be part of a subassembly of microscope 910, such as a sensor, actuator, camera, or lighting unit of microscope 910.

[0141] The computer system 920 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed across different locations, e.g., at a local client and / or one or more remote server farms and / or data centers). The computer system 920 may include any circuit or combination of circuits. In one embodiment, the computer system 920 may include one or more processors, which may be of any type. As used herein, a processor may represent any type of computing circuit, such as a microscope or microscope component (e.g., a camera), or any other type of processor or processing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA). Other types of circuits that may be included in the computer system 920 may be custom circuits, application specific integrated circuits (ASICs), and the like, such as one or more circuits (e.g., communication circuits) for wireless devices (e.g., mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems). The computer system 920 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for handling removable media, such as compact disks (CDs), flash memory cards, digital video disks (DVDs), etc. The computer system 920 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touch screen, voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system 920.

[0142] Part or all of the method steps can be performed by (or using) hardware devices, such as processors, microprocessors, programmable computers or electronic circuits. In some embodiments, one or more of some of the main method steps can be performed by such devices.

[0143] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. They may be implemented using a non-transitory storage medium (e.g., a digital storage medium such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory), which stores electronically readable control signals that cooperate with (or can cooperate with) a programmable computer system to execute the corresponding method. Therefore, the digital storage medium may be computer-readable.

[0144] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0145] Generally, the embodiments of the present invention can be implemented as a computer program product with a program code, when the computer program product runs on a computer, the program code is used to perform one of the methods.For example, the program code can be stored on a machine-readable carrier.

[0146] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0147] In other words, one embodiment of the present invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0148] Therefore, another embodiment of the present invention is a storage medium (or data carrier or computer-readable medium) having stored thereon a computer program for performing one of the methods described herein when executed by a processor. The data carrier, digital storage medium or recorded medium is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein, comprising a processor and a storage medium.

[0149] Therefore, another embodiment of the present invention is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.For example, the data stream or the sequence of signals can be transmitted via a data communication connection (for example via the Internet).

[0150] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0151] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0152] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0153] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can collaborate with a microprocessor to perform one of the methods described herein. Typically, these methods are preferably performed by any hardware device.

[0154] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0155] Although some aspects are described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of the corresponding apparatus.

[0156] Reference Signs List

[0157] 100 Oblique Light Sheet Microscope

[0158] 102 sample blocks

[0159] 104 sample racks

[0160] 106 Reflected Light

[0161] 110 main lens

[0162] 112 light sheets

[0163] 116 Main lens optical axis

[0164] 118 Light for light sheets

[0165] 120 First tube lens

[0166] 122 First scanning lens

[0167] 124 Galvanometer Mirror

[0168] 126 Second scanning lens

[0169] 128 Second tube lens

[0170] 130 Dichroic Mirror

[0171] 132 Irradiation Module

[0172] 134 Secondary objective lens

[0173] 136 third-level objective lens

[0174] 138 Third tube lens

[0175] 140 cameras

[0176] 202 main lens

[0177] 204 tilted light sheet

[0178] 206 sample rack

[0179] 210a Light sheet image

[0180] 210b Image in camera space

[0181] 212 sample blocks

[0182] 230 Scan axis

[0183] 232 Image Offset

[0184] 234 Light Sheet Angle

[0185] 236 Rotation in user space

[0186] 238 Displacement Distance

[0187] 240 Overall sample representation

[0188] 250 First User Space

[0189] 260 Second User Space

[0190] 300 Erection Process

[0191] 302 Object Space Cartesian Reference System

[0192] 304 sampled blocks

[0193] 312 User Space Cartesian Reference System

[0194] 314 vertical blocks

[0195] 316 Selected focal plane

[0196] 400 Erection Process

[0197] 402 Sample

[0198] 404 tilted light sheet

[0199] 406 Virtual Focal Plane

[0200] 410 light sheet images

[0201] 410a First sampling point

[0202] 410b intermediate image

[0203] 410c middle image

[0204] 412 left voxels

[0205] 414 voxels

[0206] 500 User Space Transition

[0207] 502 Userspace Block

[0208] 510 First Light

[0209] 512 Projection width

[0210] 514 Projection height

[0211] 516 corners

[0212] 520 The last light piece

[0213] 522 Projection width

[0214] 524 Projection height

[0215] 530 sampling range

[0216] 602 largest sample block

[0217] 604 xy perspective

[0218] 606 xz viewpoint perspective

[0219] 608ROI

[0220] 612xy projection

[0221] 614ROI

[0222] 622xz projection

[0223] 624ROI

[0224] 704a Left front corner

[0225] 704b right front corner

[0226] 704c Left rear corner

[0227] 704d right rear corner

[0228] 706a upper rear corner

[0229] 706b front upper corner

[0230] 706c rear lower corner

[0231] 706d front lower corner

[0232] 802 Maximum sample block

[0233] 804 Selected

[0234] 812 Block Projection

[0235] 814ROI

[0236] 816 Light Sheet Image Plane

[0237] 822 Block Projection

[0238] 824ROI

[0239] 900 Microscope System

[0240] 910 Microscope

[0241] 920 Computer

Claims

1. A method for operating a light sheet microscope, The following steps are involved: - capturing an image (240) of the volume (212, 608) in the camera space (240) by exciting the volume (212, 608) with one or more light sheets (204), which are tilted relative to the optical axis of the main lens of the microscope and are emitted at different positions (210a) in the volume (212), and by capturing a light sheet image (210b) of each of the one or more excitation light sheets; - transforming the captured image (240) from the camera space (240) to user space (250, 260) by mapping information from one or more light sheet images to one or more single-view focal planes (412); - displaying the volume in the user space (250, 206, 602) at a user interface; - receiving, from the user interface, selection information (614, 624) about a portion (608) of the volume (212, 602) in the user space; - converting the selection information (614, 624) from the user space (250, 260) back to the camera space (240) to control the microscope to capture an image based on the selection information (614, 624); - capturing a selected image of said portion (608) of said volume (212, 602).

2. The method according to claim 1, in, The camera space (240) is based on one or more light sheets (204) tilted at a predefined angle φ to the optical axis (116) of a main lens (110) that captures light from the volume (102, 212).

3. The method according to any one of the preceding claims, in, The one or more light sheets (204) are emitted in a pattern based on regular positions.

4. The method according to any one of the preceding claims, in, The one or more light sheets (204) are emitted in a pattern based on irregular positions.

5. The method according to any one of the preceding claims, in, The user space (250, 260) is based on one or more of the following parameters: - the angle between the light sheet (204) and the optical axis (116) of the main lens (110); - the distance between two adjacent light sheets (204); - a speed at which the volume (212) is sampled based on one or more light sheets.

6. The method according to any one of the preceding claims, The following steps are involved: - transforming information from camera space (240) to user space (250, 260) such that the focal planes of the emitted light sheet (204) and the user space (406) differ by an angle of less than approximately 90°.

7. The method according to any one of the preceding claims, The following steps are involved: - mapping the user space in a three-dimensional Cartesian coordinate system (204); - displaying the xy plane (612) of the coordinate system, in particular as a maximum intensity projection; - Receiving a user selection within the xy plane (614).

8. The method according to the preceding claim, The following steps are involved: - mapping the user space in a three-dimensional Cartesian coordinate system (204); - displaying the xz plane (622) of the coordinate system, in particular as a maximum intensity projection; - receiving user selection information in the xz plane (624).

9. The method according to the preceding claim, in, One or more parameters of the microscope are depicted in the user space (250, 260), in particular the light sheet sampling range (816).

10. The method according to any one of the preceding claims, in, Based on the selection information (612, 624) and / or further information, in particular further user input, one or more of the following microscope parameters are determined: - a starting point for the sampling movement of the one or more light sheets (204); - the end point of the sampling movement of the one or more light sheets; - the distance between two adjacent light sheets; - the speed at which the one or more light sheets sample the volume; - the intensity with which the light sheet excites the volume.

11. The method according to any one of the preceding claims, The following steps are involved: - obtaining a sampling configuration of the microscope for sampling the volume; - displaying one or more effects of the sampling configuration at a manual system interface in the user space.

12. The method according to the preceding claim, in, The sampling configuration is obtained from a user, in particular via the manual system interface.

13. A device, in particular a light sheet microscope, Configured to: - obtaining an image of the volume (212) in camera space (240), wherein The image comprises a plurality of light sheet images based on light sheets that have excited the volume and are tilted relative to the optical axis of the main lens of the microscope and are emitted at different locations (210a) in the volume (212); - transforming the acquired image (240) from the camera space (240) to user space (250, 260) by mapping information from one or more light sheet images to one or more single-view focal planes (412); - displaying the volume (212) in the user space (240, 602) at a user interface; - receiving, from the user interface, selection information (614, 624) about a portion (608) of the volume in the user space (240); - converting the selection information from the user space to the camera space and then to the microscope; - providing selection information for said reconversion in order to capture an image of said portion (608) of said volume (602).

14. The device according to the preceding claim, in, The apparatus is a light sheet microscope (100), and Therein, the light sheet is emitted and the image of the volume is obtained through the same main lens (110).

15. The device according to any one of the preceding claims, in, The apparatus is a light sheet microscope (100) comprising a galvanometer system, with which the light sheet is directed to sample the image of the volume.

Images