Optical systems for light sheet microscopes

Abstract

This disclosure provides an optical system for light sheet microscopy. [Solution] The optical system includes a processor configured to perform a method for determining a volume of interest (VOI) in a sample, the method being to process scanning image data obtained by scanning through a sample in at least one scanning direction using a light sheet, the processing of the scanning image data including determining the boundaries of an object in the scanning image data along at least two axes, and, based on the determined boundaries, defining a proposed volume of interest (VOI) such that the boundaries along each of the at least two axes are included in the proposed volume of interest (VOI).

Description

Technical Field

[0001] The present application is directed to an optical system for a light sheet microscope, a method for determining a volume of interest in a sample, a computer program, and a computer-readable medium.

Background Art

[0002] By determining a volume of interest containing an object to be analyzed in a sample, for example when performing an analysis in a microscopy application, especially when the analysis does not need to cover the entire sample, the analysis time can be shortened.

[0003] Generally, a light sheet microscopy method is applied to a transparent sample. In this case, it is difficult to visually identify the object, and for an object extending in the depth direction, the identification of the range of the object becomes more complicated, so it can be particularly difficult to identify the volume of interest.

[0004] Therefore, generally, it is difficult to determine the volume of interest in a sample, for example, to perform sample analysis in the volume of interest. Known methods for selecting the volume of interest are often based on the user manually selecting the volume of interest based on visual evaluation and their expertise. This risks introducing significant inaccuracies, which may require a large margin of error in the analysis or the need for trial and error.

[0005] However, especially for the reasons described above, not only is user-based selection hindered, but furthermore, the development of automated techniques for identifying the volume of interest with the required accuracy becomes difficult, so there is no automated method that can replace such manual selection of the volume of interest with appropriate accuracy.

Summary of the Invention

Problems to be Solved by the Invention

[0006] Therefore, an object of the present disclosure is to provide a method for determining a volume of interest in a sample with higher accuracy and speed. [Means for solving the problem]

[0007] This disclosure provides optical systems, methods, computer programs, and computer-readable media as described in each independent claim. Preferred embodiments are described in each dependent claim.

[0008] This disclosure provides an optical system for a light-sheet microscope, wherein the optical system includes a processor configured to perform a method for determining a volume of interest (VOI) in a sample. The method comprises processing scanning image data obtained by scanning a sample through a light sheet in at least one scanning direction. Processing the scanning image data comprises determining the boundaries of an object in the scanning image data along at least two axes, and, based on the determined boundaries, defining a proposed volume of interest (VOI) such that the boundaries along each of the at least two axes are included in the proposed volume of interest (VOI).

[0009] The advantage of the approach of this disclosure lies in its ability to quickly and accurately identify the volume of interest. The inventors have surprisingly discovered that the use of scanning data from the optical system itself enables accurate and rapid determination of the proposed volume of interest. Thus, this disclosure addresses the challenges outlined above and provides a method for determining the volume of interest with greater accuracy and speed.

[0010] The proposed volume of interest (VOI) can also be optionally defined such that multiple objects in the scan image data are included in the proposed volume of interest, as will be explained in more detail below.

[0011] The sample may be a liquid and / or gel-like and / or solid volume material. Scanning image data can be acquired using a sample placed, for example, in a well of a well plate, on the upper surface of a microscope slide, or on other support structures suitable for light-sheet microscopy applications. The sample may, for example, contain an object, such as an organoid. The sample may contain an object, such as an organoid, suspended in a gel or liquid.

[0012] The volume of interest, according to this disclosure, may be a portion of the sample volume. Criteria for defining the proposed volume of interest, such as which objects to include in the volume of interest and / or how many objects to include in the volume of interest, can be predetermined. Such criteria will depend on the field application, for example, the objects to be analyzed.

[0013] The objects to be analyzed may be, for example, organoids.

[0014] To detect the boundaries of an object, known image processing methods for object detection can be used. These methods may rely on various parameters associated with neighboring pixels. Objects can be determined using focus-based and / or contrast-based image processing techniques. Machine learning can also be used for object detection.

[0015] According to this disclosure, the boundary included in a VOI may include a VOI that extends at least to that boundary, or in other words, the boundary of a VOI may coincide with or extend beyond that boundary.

[0016] At least a subset of the object boundaries can form the outer boundary of the proposed VOI. Alternatively or additionally, the outer boundary of the proposed VOI may be shifted by a predetermined amount from the object boundaries, particularly outward, along each axis. If multiple objects are included in the VOI, the outer boundary of the VOI may include a subset of the boundaries of each object and / or may be shifted by a predetermined amount from the boundaries of each subset of the object boundaries along each axis.

[0017] The at least two axes may be strictly two axes, for example, the X and Z axes, or the Y and Z axes, or the X and Y axes, or three axes, for example, the X, Y, and Z axes. As an example, if the entire sample is inside the line length of the microscope, the VOI is not necessarily limited to a dimension for practical purposes, because (in that dimension) the entire sample is automatically imaged regardless of any boundary setting of the VOI.

[0018] It can be understood that an axis has two directions that define the dimensions along that axis.

[0019] Based on boundaries along two or more axes, proposed VOIs can be defined by defining VOIs of predetermined shapes, such as rectangular, cuboidal, circular, or spherical, based on predetermined rules. The processor can be configured to determine proposed VOIs by fitting predetermined shapes to all or a subset of boundaries. A subset of the object's boundaries can be selected, for example, when multiple objects are included in the proposed volume of interest and / or when the proposed volume of interest conforms to boundary conditions, such as conforming to restricted dimensions along one axis.

[0020] A VOI (Volume of Interest) does not necessarily have to be defined by a complete 3D shape. For example, a VOI can be defined by a subset of its boundary coordinates. This may be sufficient when the VOI is used for a specific purpose, such as selecting settings for subsequent analysis, especially when it is not necessary to render and display the VOI to the user.

[0021] A light sheet microscope may be an oblique-plane microscope (OPM).

[0022] Light sheet microscopy, such as fluorescence light sheet microscopy, allows scanning of a sample through a light sheet, such as a magnified laser beam. The light sheet may generally be non-parallel to the sample surface and the scanning direction. Fluorescence light sheet microscopy allows detection of fluorescence emitted from the sample based on the light sheet.

[0023] Scanning image data may be image data acquired by orienting a light sheet obliquely to the sample surface.

[0024] The scanning image data may be scanning image data received directly from the scanning process, and / or scanning image data retrieved from a data storage system which may be a local and / or remote system, and / or a distributed system as an optional means.

[0025] As will be explained in detail below, the scanning image data may include, for example, scanning image data acquired by moving the sample, for example, scanning image data acquired by moving the stage on which the sample is mounted, and / or scanning image data acquired by moving an optical component, in particular a scanner mirror, also referred to below as a galvanometer mirror, which will be detailed later.

[0026] According to the present disclosure, the proposed VOI can be automatically selected as the VOI or presented to the user for user selection. Automatically selecting the proposed VOI as the VOI can be performed without user interaction. This enables rapid progress and provides results that are still sufficiently accurate according to the on-site case. Presenting the proposed VOI to the user for user selection can be performed in a manner described in more detail below. A user interface can be used for the steps here, and the proposed VOI can be rendered and displayed via the user interface.

[0027] According to the present disclosure, processing the scanned image data can include at least two of determining a first boundary of an object in the sample in the scanned image data along a first axis parallel to the sample surface, and deriving the start and end portions of the object along the first axis; determining a second boundary of the object in the scanned image data along a second axis parallel to the sample surface and different from the first axis, and deriving the start and end portions of the object along the second axis; and determining a third boundary of the object in the scanned image data along a third axis perpendicular to the sample surface, also referred to as the Z axis, and deriving the start and end portions of the object along the third axis. Defining the proposed volume of interest VOI can include defining a volume that extends at least from the start to the end of the object along each axis, for example, each of the first axis, the second axis, and the third axis, as the proposed volume of interest VOI.

[0028] In particular, the first axis may be perpendicular to the second axis and / or the third axis. Optionally, the first axis, the second axis, or the third axis may be parallel to the scanning direction. Hereinafter, the first axis and the second axis may also be referred to as the X axis and the Y axis.

[0029] [ According to the present disclosure, defining the proposed volume of interest (VOI) can include defining a volume that extends at least from the beginning to the end of the object along each of a first axis, a second axis, and a third axis. In many cases, the volume here can provide a sufficient coverage rate along one axis with a line length (e.g., the width of the detector and / or the light sheet), such that for example, it can cover substantially the entire sample along that axis, and thus it may be sufficient to use only two axes. Thus, as an example, each dimension of the proposed VOI may be sufficient to be determined only along the other two dimensions.

[0030] The beginning and the end of the object along one axis can become a first boundary and a second boundary, respectively, when a direction parallel to the axis is given. When the scanning direction is the same as the above direction, the beginning is scanned earlier than the end.

[0031] According to the present disclosure, at least one scanning direction S can be parallel to at least one of the first axis, the second axis, and the third axis. That is, in the same direction as the scanning direction, boundaries along the one or more parallel axes, particularly the beginning and the end of the object, can be determined. For practical purposes, the dimensions of the object and the corresponding VOI in the scanning direction are particularly important for subsequent analysis of the object. Therefore, by determining these appropriately, an improved overall result can be obtained.

[0032] According to the present disclosure, the scanned image data includes a plurality of first images obtained by moving a sample stage in a direction parallel to the sample surface of the sample, also referred to as the scanning direction S, and thereby scanning through the sample using a light sheet, and alternatively or additionally, a plurality of second images obtained while moving the light sheet by operating one or more optical components to move the light sheet through the sample. The optical components may include a scanner mirror, also referred to as a galvanometer mirror for example.

[0033] Sample scanning is typically performed by moving a light sheet using optical components, such as a galvanometer mirror. This allows for rapid and accurate scanning. To extend the scannable range, this range can be expanded by moving the sample stage during or between scans.

[0034] As an example where both optical components, and therefore both the light sheet and the sample stage, are moved, the optical components can be driven continuously and the stage can be driven in steps. To operate the optical components and acquire scanning image data, for example, multiple second images, scanning can also be performed during the dead time of each stage. As an optional means, scanning can also be performed while moving the stage, for example, to acquire multiple first images.

[0035] Processing scan image data may include stitching the image data and determining the boundaries of the object in the stitched volume. Determining the boundaries and proposed VOIs in the stitched volume may be particularly advantageous when the stage is moved as part of the image acquisition.

[0036] According to this disclosure, processing scan image data may involve individually determining the boundaries in at least one of a plurality of first images and / or individually determining the boundaries in at least one of a plurality of second images. In other words, the boundaries can be individually determined in at least a subset of the plurality of first images and / or a subset of the plurality of second images, where each subset may be a single image from the plurality of first images and / or the plurality of second images. In this scenario, image stitching can be omitted, resulting in lower computational costs and lower requirements regarding the available images. For example, stitching may require sufficient overlap of image data.

[0037] In this example, stitching is not required to determine the boundary. In other words, the stitching step can be omitted. The boundary can be determined using only a single 2D image. This is because the light sheet provides sufficient volume information, allowing for boundary identification in multiple dimensions. This is not the case, for example, with conventional imaging.

[0038] The optical system according to this disclosure may include a sample stage and can be configured to move the sample stage to acquire scanning image data. Using the movement of the sample stage allows for a wider range of motion than when using optical components alone. The sample stage can be moved in steps or continuously.

[0039] The optical system according to this disclosure may include optical components, such as a galvanometer mirror, which can be configured to move a light sheet and acquire scanning image data. Such scanning is faster and more precise compared to moving a sample stage.

[0040] According to this disclosure, processing scan image data can include the detection of multiple objects, and the proposed volume of interest (VOI) can be determined to include two or more of the multiple objects. For this purpose, if the multiple objects are arranged consecutively along a single axis, the proposed VOI can extend along that axis from the outermost boundary of at least the outermost object among the multiple objects. For example, if the objects are arranged in a direction along the axis, the proposed VOI can extend at least from the beginning of the first object in that direction to the end of the last object in that direction.

[0041] According to this disclosure, the processor can be configured to automatically set the proposed volume of interest (VOI) as the VOI to be used in subsequent procedures, and / or to render the proposed volume of interest (VOI) or a derived volume derived from the proposed volume of interest (VOI) on a user interface, which allows the user to set the VOI for subsequent procedures by user input.

[0042] The user interface may include a display device configured to render the proposed VOI, and an input device configured to receive user input, particularly user input for setting the VOI. The user input device and the display device may be configured as an integrated unit, for example, a touch panel display, or separately, for example, a display and a keyboard and / or mouse.

[0043] The volume derived from the proposed volume of interest can be derived by performing processing steps such as deskewing (also called deskewing) and / or projection of the proposed volume of interest. This allows for rendering that shows a view representing the VOI in the manner expected by the user. Rendering the image data before deskewing and / or projection does not provide a continuous representation of the actual volume, making it difficult for the user to accurately select the VOI. Known methods for deskewing and / or projection can be used.

[0044] The Disclosure also provides a method for determining a volume of interest (VOI) in a sample, the method comprising processing scanning image data obtained by scanning the sample with a light sheet of an optical system for a light sheet microscope, wherein processing the scanning image data includes determining the boundaries of an object in the scanning image data along at least two axes, and defining, based on the determined boundaries, a proposed volume of interest (VOI) such that the boundaries along each of the axes are included in the proposed volume of interest (VOI).

[0045] In particular, the optical systems used in the methods of this disclosure may be, for example, the optical systems outlined above or claimed by this disclosure.

[0046] The method of this disclosure may include at least two of the following: processing scan image data, determining a first boundary of an object in the sample in the scan image data along a first axis X parallel to the sample surface, and deriving the start and end of the object along the first axis; determining a second boundary of the object in the scan image data along a second axis Y parallel to the sample surface and different from the first axis X, and deriving the start and end of the object along the second axis; and determining a third boundary of the object in the scan image data along a third axis Z perpendicular to the sample surface, and deriving the start and end of the object along the third axis. In this example, defining the proposed volume of interest (VOI) may include defining the proposed volume of interest (VOI) as the volume extending at least from the start to the end of the object along each of the axes X, Y, and Z.

[0047] In particular, the method according to this disclosure may include defining the proposed volume of interest (VOI) by defining a volume that extends at least from the beginning to the end of the object along each of the first axis X, the second axis Y, and the third axis Z.

[0048] In the method according to the present disclosure, at least one scanning direction S may be parallel to at least one of the first axis X, the second axis Y, and the third axis Z.

[0049] In the method according to this disclosure, the image data may include a plurality of first images acquired by moving a sample stage in a direction S parallel to the sample surface of the sample, thereby scanning through the sample using a light sheet, and / or a plurality of second images acquired while moving the light sheet by operating one or more optical components to move the light sheet through the sample.

[0050] The method according to this disclosure may include processing scan image data by stitching the images of the scan image data and determining the boundaries of an object in the stitched volume.

[0051] Alternatively or additionally, the methods of the present disclosure may include processing scan image data by individually determining the boundaries in at least one of a plurality of first images and / or individually determining the boundaries in at least one of a plurality of second images.

[0052] The method disclosed herein may include acquiring scanned image data.

[0053] In particular, the method of the present disclosure may include moving a sample stage to acquire scanning image data, and / or moving a light sheet by operating an optical component to acquire scanning image data.

[0054] The method according to this disclosure may involve processing scan image data to include the detection of multiple objects, and the proposed volume of interest (VOI) can be determined to include two or more of the multiple objects.

[0055] The method according to this disclosure may include automatically setting the proposed volume of interest (VOI) as a VOI to be used in a subsequent procedure, and / or rendering the proposed volume of interest (VOI) or a derived volume derived from the proposed volume of interest (VOI) on a user interface on which the user can input a VOI for a subsequent procedure.

[0056] In the systems and methods of this disclosure, machine learning techniques or AI techniques may be employed for at least some of the steps, for example, for object detection and / or object boundary identification.

[0057] This disclosure also provides a computer program, including program code for performing the method described herein, particularly as outlined above.

[0058] The disclosure also provides a computer-readable medium that stores instructions causing a processor to perform the method outlined above, in particular, when executed by the processor.

[0059] It should be understood that the characteristics and advantages described above in the context of optical systems also apply equally to corresponding methods, computer programs, and computer-readable media.

[0060] The present disclosure will be described below with reference to accompanying drawings that provide background information and illustrate specific embodiments of the present disclosure. However, the scope of the present invention is not limited to the specific features disclosed in the context of the drawings. [Brief explanation of the drawing]

[0061] [Figure 1a] This is a schematic diagram showing the optical system, sample, and volume of interest observed from two viewpoints according to the present disclosure. [Figure 1b] This is a schematic diagram showing the optical system, sample, and volume of interest observed from two viewpoints according to the present disclosure. [Figure 1c]This is a schematic diagram showing the optical system, sample, and volume of interest observed from two viewpoints according to the present disclosure. [Figure 1d] This figure illustrates an example case in which two objects arranged along the x-axis are contained within a volume of interest. [Figure 2] This is a schematic diagram showing a light sheet microscope including the optical system 100 according to this disclosure. [Figure 3] This is a schematic diagram illustrating the method described herein. [Figure 4] This diagram shows the skew removal process. [Figure 5] This figure shows the system as disclosed herein. [Modes for carrying out the invention]

[0062] Figure 1a shows an optical system 100 for a light-sheet microscope 10 (shown in Figure 2), which includes a processor 102 configured to perform a method for determining a volume of interest VOI 200 in a sample 300. For example, the processor can be configured to perform any of the methods of this disclosure, as described, for example, in the context of Figure 3.

[0063] The sample and VOI are shown for illustrative purposes only, but are not part of the optical system. The light sheet microscope may be, for example, an oblique-plane microscope (OPM), or a microscope using other light sheet imaging techniques. The light sheet 400 formed by the light sheet microscope is illustrated for illustrative purposes only.

[0064] The optical system may optionally include a user interface 500 that allows user input to configure a VOI for a subsequent procedure. The user interface may include a display device configured to render the proposed VOI, and an input device configured to receive user input, in particular user input for configuring the VOI. The user input device and the display device may be configured as an integrated unit, for example, a touch panel display, or separately, for example, a display and a keyboard and / or mouse.

[0065] The optical system may include a sample stage 600 as an optional means and can be configured to move the sample stage 600 to acquire scanning image data.

[0066] The optical system may optionally include an optical component 700, such as a galvanometer mirror, and may also be configured to operate the optical component to move the light sheet 400 in order to acquire scanning image data.

[0067] The sample stage 600 of the optical system, the optical components 700, and further components as optional means, such as the component 104 used for acquiring scanning image data, can be collectively referred to as a scanning system.

[0068] The processor 102 is configured to process scan image data acquired by scanning the sample 300 in at least one scanning direction S using a light sheet 400 oriented obliquely to the sample surface 304, as shown in Figure 1a, for example. The at least one scanning direction S may, for example, be parallel to at least one of a first axis X, a second axis Y, and a third axis Z. In one example, each axis may be perpendicular to the others. At least one of these axes may be parallel to the scanning direction.

[0069] Processing scan image data involves determining the boundaries 302x, 302y, 302z of the object in the scan image data along at least two axes. That is, the start and end points of the object can be determined for each of the at least two axes. This means that the determined boundary for each axis can contain two values ​​representing the start and end coordinates of the object along that axis.

[0070] The processor is configured to define a proposed volume of interest (VOI) based on the determined boundaries 302x, 302y, 302z such that the boundaries 302x, 302y, 302z along at least two axes X, Y, and Z are included in the proposed volume of interest (VOI).

[0071] In some examples, the processor is configured to determine the start and end of an object along the X and Z axes, respectively, or along the Y and Z axes, respectively, or along the X and Y axes, respectively, as part of processing the scan image data. In another example, the start and end of an object are determined along the X, Y, and Z axes, respectively.

[0072] Specifically, as part of processing the scanning image data, the processor 102 determines the first boundary 302x of the object 302 within the sample 300 in the scanning image data along a first axis X parallel to the sample surface 304, and derives the start 302x-1 and end 302x-2 of the object 302 along the first axis X, and the object in the scanning image data along a second axis Y that is parallel to the sample surface 304 and different from the first axis X. The system can be configured to perform at least two of the following: determining the second boundary 302y of object 302 and deriving the start 302y-1 and end 302y-2 of object 302 along the second axis Y; and determining the third boundary 302z of object 302 in the scanning image data along the third axis Z perpendicular to the sample surface 304 and deriving the start 302z-1 and end 302z-2 of object 302 along the third axis Z.

[0073] In this example, the processor 102 is configurable to define a proposed volume of interest (VOI), where defining the proposed volume of interest (VOI) includes defining the volume as the proposed volume of interest (VOI) that extends along each axis X, Y, Z at least from the starting point 302x-1, 302y-1, 302z-1 to the ending point 302x-2, 302y-2, 302z-2 of the object 302. That is.

[0074] For example, the processor can be configured to define a volume extending at least from the beginning 302x-1, 302y-1, 302z-1 to the end 302x-2, 302y-2, 302z-2 of the object 302, along each of the first axis X, second axis Y, and third axis Z, as part of defining the proposed volume of interest (VOI). In other words, the proposed volume of interest can be defined in three dimensions unfolded along axes X, Y, and Z. Alternatively, a definition in only two dimensions can be made along the Z axis and one of the X and Y axes.

[0075] The scanning image data may include a plurality of first images obtained by moving the sample stage 600 in a direction S parallel to the sample surface 304 of the sample 300, thereby scanning through the sample 300 using the light sheet 400, and / or a plurality of second images obtained while moving the light sheet 400 by operating one or more optical components 700, such as a galvanometer mirror, to move the light sheet 400 through the sample 300.

[0076] In this example, the processor 102 can be configured, as part of processing the scan image data, to stitch the images of the scan image data and determine the boundaries of the object 302 in the stitched volume. Alternatively or additionally, in this example, the processor 102 can be configured, as part of processing the scan image data, to individually determine the boundaries in each of at least one of a plurality of first images and / or to individually determine the boundaries in each of at least one of a plurality of second images.

[0077] The processor 102 can be configured, as part of processing the scan image data, to detect a plurality of objects 302 and to determine the proposed volume of interest VOI such that the proposed volume of interest includes two or more of the plurality of objects 302, as shown in Figure 1d, for example.

[0078] The processor can be configured to automatically set the proposed volume of interest (VOI) as the VOI to be used in subsequent procedures, and / or render the proposed volume of interest (VOI) or the derived volume derived from the proposed volume of interest (VOI) on the user interface 500 described above, and / or receive user input to set the VOI for subsequent procedures. The processor can be configured to perform processing on the proposed VOI, e.g., projection and / or deskew, before rendering, thereby obtaining the derived volume derived from the volume of interest to be rendered.

[0079] Figure 1b shows the potential volume of interest 302 and its beginning and end in the X and Z directions (side view or vertical section) to illustrate the features described above. Figure 1c shows the potential volume of interest 302 and its beginning and end in the X and Y directions (top view or horizontal section). Figure 1d shows an exemplary case in which two objects arranged along the X axis are contained within the volume of interest. In this example, the proposed volume of interest extends along the X axis at least from the beginning 302x-1 of the first object (shown on the left) to the end 302x-2 of the second object (shown on the right).

[0080] This disclosure also provides a light-sheet microscope 10 including an optical system 100 shown in Figure 1a. Such a microscope is shown in Figure 2.

[0081] This disclosure also provides a method for determining the volume of interest (VOI) 200 in sample 300. An exemplary method is described below and shown in Figure 3.

[0082] As an example, the optical system used in the method of this disclosure may be, for example, the optical system of this disclosure outlined in Figures 1a, 1b, and 2 or in the context of the claims.

[0083] The method includes, in step S12, processing scanning image data acquired by scanning through the sample 300 using the light sheet 400 of the optical system 100 for the light sheet microscope 10, wherein processing the scanning image data includes determining the boundaries 302x, 302y, 302z of the object 302 in the scanning image data along at least two axes X, Y, Z. The boundaries of a single object or the boundaries of multiple objects can be determined.

[0084] Known image processing methods for object detection can be used to determine the boundaries of objects. These methods may rely on various parameters associated with neighboring pixels. Objects can be determined using focus-based and / or contrast-based image processing techniques. Machine learning is also available for object detection.

[0085] The method here further includes, in step S13, defining the proposed volume of interest (VOI) based on the determined boundary such that the boundary along each axis is included in the proposed volume of interest (VOI).

[0086] Step S12, which processes the scanned image data, may include determining the start and end points of the object in at least two directions, i.e., along at least two axes. In non-limiting examples, one of the axes may be parallel to the scanning direction S.

[0087] In this case, the volume of interest (VOI) can be defined by defining a volume that extends at least from each beginning to each end of the object. The boundary of the volume of interest in further directions does not necessarily have to be defined, or the boundary of the volume of interest in at least some further directions may be defined by using a predetermined shape of the volume of interest, such as a cuboid or a sphere, and dimensionally designing the volume of interest so that its beginning and end are included at its boundary.

[0088] For example, in order to accomplish this, step S12, which processes the scanned image data, may include at least two of the following steps S12a to S12c.

[0089] Step S12a includes determining a first boundary 302x of the object 302 within the sample 300 in the scanning image data along a first axis X parallel to the sample surface 304, and deriving the start 302x-1 and end 302x-2 of the object 302 along the first axis X.

[0090] Step S12b includes determining a second boundary 302y of the object 302 in the scanning image data along a second axis Y that is parallel to the sample surface 304 and different from the first axis X, and deriving the start 302y-1 and end 302y-2 of the object 302 along the second axis Y.

[0091] Step S12c includes determining the third boundary 302z of the object 302 in the scanning image data along a third axis Z perpendicular to the sample surface 304, and deriving the start 302z-1 and end 302z-2 of the object 302 along the third axis Z.

[0092] For example, decisions here can be made for the X and Z axes, or for the Y and Z axes, or for the X and Y axes, or for all three X, Y, and Z axes.

[0093] After at least two of steps S12a, S12b, and S12c have been applied, the volume of interest becomes definable in step S13, in which case the volume of interest VOI may include defining in step S13a the volume extending along each axis X, Y, Z, in particular along each axis from at least the starting part 302x-1, 302y-1, 302z-1 to the ending part 302x-2, 302y-2, 302z-2 of the object 302 as the proposed volume of interest VOI.

[0094] Step S12, which processes the scan image data, may include detecting a plurality of objects 302, and step S13, which determines the proposed volume of interest (VOI), may be determined to include two or more of the plurality of objects 302. In this case, for example, in steps S12a, S12b, and / or S12c, the start of the first object and the end of the last object of two or more objects arranged along each direction that are included in the proposed VOI can be determined, in directions extending along the first axis, the second axis, and / or the third axis, respectively. Furthermore, in this case, for example, in step S13a, the proposed volume of interest (VOI) may be determined to extend at least from the start of the first object to the end of the last object along each direction.

[0095] In step S10, which is an optional means, the scanned image data can be received directly from the scanning process and / or retrieved from a data storage system which may be a local and / or remote system, or even a distributed system, which is an optional means.

[0096] Before step S11, scanning image data can be acquired in step S10 as an optional means.

[0097] Specifically, in step S10a, which is an optional means, the sample stage 600 can be moved to acquire scanning image data. The scanning direction S may be parallel to the sample surface of the sample. For example, a sample stage 600 as shown in the context of Figure 1a or Figure 2 can be used. Alternatively, or in addition to step S10a, in step S10b, which is an optional means, the light sheet 400 itself can be moved to acquire scanning image data. This can be done, for example, by operating an optical component 700 for moving the light sheet, such as a galvanometer mirror. For this purpose, an optical component as described in the context of Figure 1a or Figure 2 can be used.

[0098] In this way, the scanning image data may include a plurality of first images obtained by moving the sample stage 600 in a direction S parallel to the sample surface 304 of the sample 300 in step S10a as an optional means, thereby scanning through the sample 300 using the light sheet 400, and / or a plurality of second images obtained while moving the light sheet 400 by operating one or more optical components 700 to move the light sheet 400 through the sample 300 in step S10b as an optional means.

[0099] In one example, optical components, such as a galvanometer mirror, can be driven continuously, while the stage can be moved in steps. Scanning can be performed by activating the optical components and moving the light sheet during each dead time of the stage.

[0100] Therefore, scanning can yield multiple consecutive images along the scanning direction. In this case, processing the scanned image data may include stitching the images of the scanned image data in step S12c, which is an optional means. In this case, determining the boundaries of the object 302 can be done within the stitched volume. Stitching techniques known in the art can also be used.

[0101] Alternatively or additionally, the boundary is determinable in each of a subset of several first images, each having a individually determined boundary, and in particular in at least one of the several first images. Similarly, the boundary is determinable in each of a subset of several second images, each having a individually determined boundary, and in particular in at least one of the several second images.

[0102] After the proposed volume of interest (VOI) has been determined, the method may, as an optional means, include in step S14, i.e., in step S14a, automatically setting the proposed volume of interest (VOI) as the VOI to be used in a subsequent procedure, e.g., for sample analysis, by microscopy, for example, using the light sheet microscope described above. Alternatively or additionally, in step S14, i.e., in step S14b, the method may include rendering the proposed volume of interest (VOI) or a derived volume derived from the proposed volume of interest (VOI) on a user interface 500 that allows the user to set the VOI for a subsequent procedure by user input. This can be done using a user interface such as the one described in the context of Figure 1a.

[0103] Before rendering the proposed VOI, the scanned image data can be processed, which may include image projection and / or deskewing of the image data. This allows the user to interact with the VOI in a view familiar to them. This is illustrated in Figure 4, which is just one example. Deskewing of image data may include processing the images of obliquely acquired image data, for example, by shifting their position and / or rotating them, so that these image data form a continuous and undistorted representation of the volume that will be depicted when stacked. Image projection may include projecting the volumetric image data onto a 2D projection, for example, to enable rendering to a display. After the VOI is selected by user input via user input, reconstruction into volumetric image data becomes possible. Methods for deskewing, projection, and reconstruction known in the art can be used; therefore, a detailed explanation is omitted here.

[0104] In the optical systems and methods described herein, machine learning techniques or AI techniques may be used for at least some of the steps, for example, for object detection and / or object boundary identification.

[0105] This disclosure also provides a computer program, including program code for performing the method described herein, particularly as outlined above.

[0106] The disclosure also provides a computer-readable medium storing instructions for causing a processor to perform the method outlined above, in particular, when executed by the processor.

[0107] This disclosure provides a system 900 comprising, for example, a microscope 10 and a computing device 800, as shown in Figure 2. Such a system is shown in Figure 5.

[0108] In this disclosure, several embodiments relate to a microscope 10 including an optical system 100 described in relation to Figures 1a and 2. The microscope may be part of a system 900 or connected to a system 900, as shown in Figure 5. Figures 1a and 2 show schematic diagrams of an optical system 100 and a microscope 10 including the system 100, respectively, configured to perform the methods described herein, particularly using a processor 102. The system 900, in particular the optical system, includes a processor 102. The processor may be part of a computing device 800. The microscope 10 and / or system 900 are configured to perform imaging by scanning, in particular to acquire scanning image data, and are connectable to a computing device 800. The computing device 800, in particular the processor 102, is configured to perform at least a portion of the methods described herein. The computing device 800, in particular the processor, can be configured to perform machine learning algorithms. The computing device 800 and the microscope 10 can be configured in a common housing, as a whole entity, or as partially separate entities, but integrated together. The computing device 800 may be part of the central processing system of the microscope 10, and / or part of a subcomponent of the microscope 10, such as a sensor, actor, camera, or lighting unit of the microscope 10.

[0109] The computing device 800 may be a local computer device (e.g., a personal computer, laptop computer, tablet computer, or mobile phone) comprising one or more processors and one or more storage devices, or it may be a distributed computing device (e.g., a cloud computing system comprising one or more processors and one or more storage devices distributed across various locations, e.g., local clients and / or one or more remote server farms and / or data centers). The computing device 800 may include any circuit or combination of circuits. In one embodiment, the computing device 800 may include one or more processors of any kind. As used herein, a processor may mean, but is not limited to, any kind of computing circuit such as a microprocessor in a microscope or microscope component (e.g., a camera), a microcontroller, a composite instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multicore processor, a field-programmable gate array (FPGA), or any other kind of processor or processing circuit.Other types of circuits that may be included in the computing device 800 may be custom circuits, application-specific integrated circuits (ASICs), etc., such as one or more circuits (communication circuits, etc.) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computing device 800 may also include one or more storage devices that may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives and / or one or more drives that handle removable media such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc. The computing device 800 may also include a display device, one or more speakers, a keyboard and / or a controller that may include a mouse, a trackball, a touchscreen, a voice recognition device, or any other devices that enable a user of the system to input information into and receive information from the computing device 800.

[0110] Some or all of the method steps can be performed by hardware devices (or their use), such as a processor, microprocessor, programmable computer, or electronic circuit, and in some embodiments, one or more of the most important method steps can be performed by these devices or computing devices.

[0111] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation is feasible using a non-transient recording medium, which is a digital recording medium, etc., that stores electronically readable control signals and cooperates (or can cooperate) with a programmable computer system to carry out each method. Examples include floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs and EPROMs, EEPROMs, or FLASH memory. Thus, the digital recording medium may be computer-readable.

[0112] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system or computing device so as to perform any of the methods described herein.

[0113] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code, which operates to perform one of the methods when the computer program product is executed on a computer. This program code may be stored, for example, on a machine-readable carrier.

[0114] Another embodiment includes a computer program stored in a machine-readable carrier for carrying out any of the methods described herein.

[0115] Therefore, in other words, embodiments of the present invention are computer programs having program code for carrying out any of the methods described herein when the computer program is executed on a computer.

[0116] Accordingly, another embodiment of the present invention is a recording medium (or data carrier or computer-readable medium) containing a stored computer program for carrying out any of the methods described herein when executed by a processor. The data carrier, digital recording medium, or recording medium is typically tangible and / or non-transient. Another embodiment of the present invention is the apparatus, computing device, or microscope system described herein, comprising a processor and a storage medium.

[0117] Therefore, another embodiment of the present invention is a data stream or signal sequence representing a computer program for carrying out any of the methods described herein. The data stream or signal sequence may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0118] Another embodiment includes a processing means, for example, a computer, computing device, computer system, or programmable logic device configured or adapted to perform any of the methods described herein.

[0119] Another embodiment includes a computer or computing device on which a computer program for performing any of the methods described herein is installed.

[0120] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for carrying out any of the methods described herein to a receiver. The receiver may be, for example, a computer, computing device, mobile device, storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0121] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to carry out any of the methods described herein. Generally, the methods are advantageously carried out by any hardware device.

[0122] Embodiments may be based on the use of machine learning models or machine learning algorithms. Instead of relying on models and inference, machine learning may refer to algorithms and statistical models that a computer system can use to perform a particular task without using explicit instructions. For example, machine learning may use data transformations that are inferred from the analysis of historical data and / or training data, instead of rule-based data transformations. For example, image content may be analyzed using a machine learning model or a machine learning algorithm. For a machine learning model to analyze image content, the machine learning model may be trained with training images as input and training content information as output. By training a machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model “learns” to recognize image content so that image content not included in the training data becomes recognizable using the machine learning model. The same principle may be used in the same way for other types of sensor data: by training a machine learning model with training sensor data and a desired output, the machine learning model “learns” the transformation between sensor data and output, which can then be used to provide output based on non-trained sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) may be preprocessed to obtain feature vectors that can be used as input to a machine learning model.

[0123] A machine learning model may be trained using training input data. The example above uses a training method called "supervised learning." In supervised learning, a machine learning model is trained using multiple training samples, each of which may contain multiple input data values ​​and multiple desired output values; that is, each training sample is associated with a desired output value. By specifying both the training samples and the desired output values, the machine learning model "learns" during training which output values ​​to provide based on input samples similar to the provided samples. In addition to supervised learning, semi-supervised learning may be used. In semi-supervised learning, some of the training samples lack corresponding desired output values. Supervised learning may be based on a supervised learning algorithm (e.g., a classification algorithm, a regression algorithm, or a similarity learning algorithm). A classification algorithm may be used if the output is limited to a limited set of values ​​(categorical variables), i.e., the input is classified into one of a limited set of values. A regression algorithm may be used if the output may have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms, but are based on learning from examples using a similarity function that measures how similar or related two objects are. In addition to supervised or semi-supervised learning, unsupervised learning may be used to train machine learning models. In unsupervised learning, input data (only) may be provided, and unsupervised learning algorithms may be used to find structure in the input data (for example, by grouping or clustering the input data, or by finding commonalities in the data). Clustering is the process of assigning input data containing multiple input values ​​into multiple subsets (clusters), so that input values ​​within the same cluster are similar according to one or more (predefined) similarity criteria, but are not similar to input values ​​in another cluster.

[0124] Reinforcement learning is a third group of machine learning algorithms. In other words, reinforcement learning may be used to train machine learning models. In reinforcement learning, one or more software actors (referred to as “software agents”) are trained to take actions in their surroundings. A reward is calculated based on the actions taken. Reinforcement learning is based on training one or more software agents to choose actions that result in software agents that perform better on a given task, with cumulative rewards increasing (as revealed by the increase in rewards).

[0125] Furthermore, several techniques may be applied as part of a machine learning algorithm. For example, feature representation learning may be used. In other words, a machine learning model may be trained at least partially using feature representation learning, and / or a machine learning algorithm may include feature representation learning components. A feature representation learning algorithm, which may be called a representation learning algorithm, may not only store information in its own input but may also transform the information to make it useful, often as a preprocessing step before performing classification or prediction. Feature representation learning may be based, for example, on principal component analysis or cluster analysis.

[0126] In some examples, anomaly detection (i.e., outlier detection) may be used, which aims to provide the identification of input values ​​that raise suspicion by being significantly different from the majority of the input or training data. In other words, a machine learning model may be trained with anomaly detection, at least in part, and / or a machine learning algorithm may include anomaly detection components.

[0127] In some examples, a machine learning algorithm may use a decision tree as its predictive model. In other words, a machine learning model may be based on a decision tree. In a decision tree, observations about an item (e.g., a set of input values) may be represented by branches of the decision tree, and the output values ​​corresponding to these items may be represented by leaves of the decision tree. A decision tree may support both discrete and continuous values ​​as output values. When discrete values ​​are used, the decision tree may be represented as a classification tree, and when continuous values ​​are used, the decision tree may be represented as a regression tree.

[0128] Correlation rules are another technique that can be used in machine learning algorithms. In other words, a machine learning model may be based on one or more correlation rules. Correlation rules are created by identifying relationships between variables in a large amount of data. A machine learning algorithm may identify and / or utilize one or more correlational rules that represent knowledge derived from the data. These rules may be used, for example, to store, manipulate, or apply knowledge.

[0129] Machine learning algorithms are typically based on machine learning models. In other words, the term “machine learning algorithm” may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term “machine learning model” may refer to a set of data structures and / or rules that represent learned knowledge (for example, based on training performed by a machine learning algorithm). In embodiments, usage of a machine learning algorithm may mean usage of one underlying machine learning model (or multiple underlying machine learning models). Usage of a machine learning model may mean that a machine learning model and / or a set of data structures / rules that are a machine learning model are trained by a machine learning algorithm.

[0130] For example, a machine learning model may be an artificial neural network (ANN). An ANN is a system influenced by biological neural networks, such as those found in the retina or brain. An ANN consists of multiple interconnected nodes and multiple junctions, so-called edges, between the nodes. Typically, there are three types of nodes: input nodes that receive input values, hidden nodes that are (simply) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transmit information from one node to another. The output of a node may be defined as a (nonlinear) function of its input (e.g., the sum of its inputs). The input of a node may be used in a function based on the "weights" of the edges or nodes that provide the input. The weights of nodes and / or edges may be adjusted during the learning process. In other words, training an artificial neural network may involve adjusting the weights of the nodes and / or edges of the artificial neural network to obtain a desired output for a given input.

[0131] Alternatively, a machine learning model may be a support vector machine, a random forest model, or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with a relevant learning algorithm that can be used to analyze data (e.g., in classification or regression analysis). A support vector machine may be trained by providing inputs with multiple training input values ​​belonging to one of two categories. A support vector machine may be trained to assign new input values ​​to one of two categories. Alternatively, a machine learning model may be a Bayesian network, which is a stochastic directed acyclic graphical model. A Bayesian network may use a directed acyclic graph to represent a set of random variables and their conditional dependencies. Alternatively, a machine learning model may be based on a search algorithm and a genetic algorithm, which is a heuristic method that mimics the process of natural selection.

[0132] As used herein, the term "and / or" includes all possible combinations of one or more of the related items and may be abbreviated as " / ".

[0133] While several embodiments have been described in the context of the apparatus, it is clear that these embodiments also represent descriptions of the corresponding methods, where blocks or apparatus correspond to steps or features of steps. Similarly, embodiments described in the context of steps also represent descriptions of the corresponding blocks, items, or features of the corresponding apparatus. [Explanation of Symbols]

[0134] 10 Light Sheet Microscopes 100 Optical Systems 102 processors 104 Further parts 200 Interesting Warfare 300 samples 304 Sample surface 302x,302y,302z border 302x-1, 302y-1, 302z-1 Starting point of the object 302x-2, 302y-2, 302z-2 End of object 400 Light Seat 500 User Interfaces 600 sample stages 700 Optical Components 800 Computing Devices 900 System S scanning direction X,Y,Z axis

Claims

1. An optical system (100) for a light sheet microscope (10), The optical system includes a processor (102) configured to perform a method for determining a volume of interest (VOI) (200) in a sample (300), wherein the method is Processing scan image data obtained by scanning the sample (300) through a light sheet (400) in at least one scanning direction (S), wherein processing the scan image data includes determining the boundaries (302x, 302y, 302z) of the object in the scan image data along at least two axes. Based on the determined boundary (302x, 302y, 302z), the proposed volume of interest VOI is defined such that the boundary (302x, 302y, 302z) along each of the at least two axes (X, Y, Z) is included in the proposed volume of interest VOI, An optical system (100) including the optical system.

2. Processing the aforementioned scanning image data is The first boundary (302x) of the object (302) within the sample (300) in the scanning image data is determined along a first axis (X) parallel to the sample surface (304), and the start (302x-1) and end (302x-2) of the object (302) along the first axis (X) are derived. The second boundary (302y) of the object (302) in the scanning image data is determined along a second axis (Y) that is parallel to the sample surface (304) and different from the first axis (X), and the start (302y-1) and end (302y-2) of the object (302) along the second axis (Y) are derived. The third boundary (302z) of the object (302) in the scanning image data is determined along a third axis (Z) perpendicular to the sample surface (304), and the start (302z-1) and end (302z-2) of the object (302) along the third axis (Z) are derived. It includes at least two of the following: Defining the proposed volume of interest VOI involves defining the proposed volume of interest VOI as the volume extending along each axis (X, Y, Z) at least from the starting point (302x-1, 302y-1, 302z-1) to the ending point (302x-2, 302y-2, 302z-2) of the object (302). The optical system (100) according to claim 1.

3. Defining the proposed volume of interest VOI includes defining a volume that extends at least along each of the first axis (X), the second axis (Y), and the third axis (Z) of the object (302) from its starting point (302x-1, 302y-1, 302z-1) to its ending point (302x-2, 302y-2, 302z-2). The optical system (100) according to claim 2.

4. The at least one scanning direction (S) is parallel to at least one of the first axis (X), the second axis (Y), and the third axis (Z). The optical system (100) according to claim 2 or 3.

5. The data of the scanning image is The sample stage (600) is moved in a direction (S) parallel to the sample surface (304) of the sample (300), thereby obtaining a plurality of first images by scanning through the sample (300) using a light sheet (400), By operating one or more optical components (700) to move the light sheet (400) through the sample (300), a plurality of second images acquired while moving the light sheet (400) are obtained, Including at least one of the following: The optical system (100) according to any one of claims 1 to 4.

6. Processing the aforementioned scanning image data is The process involves stitching together the images from the aforementioned scanning data. Determining the boundary of the object (302) in the stitched volume, including, The optical system (100) according to claim 5.

7. Processing the aforementioned scanning image data is To individually determine the boundary in at least one of the plurality of first images, To individually determine the boundary in at least one of the plurality of second images, Including at least one of the following: The optical system (100) according to claim 5 or 6.

8. The optical system (100) includes a sample stage (600) and is configured to move the sample stage (600) in order to acquire scanning image data. The optical system (100) according to any one of claims 1 to 7.

9. The optical system (100) includes an optical component (700) and is configured to move a light sheet (400) by operating the optical component (700) in order to acquire scanning image data. The optical system (100) according to any one of claims 1 to 8.

10. Processing the scan image data includes detecting a plurality of objects (302), and determining that the proposed volume of interest VOI includes two or more of the plurality of objects (302). The optical system (100) according to any one of claims 1 to 9.

11. The aforementioned processor, The proposed volume of interest (VOI) is automatically set as the VOI to be used in subsequent procedures. Rendering the proposed volume of interest VOI or a derived volume derived from the proposed volume of interest VOI onto a user interface (500) that allows the user to input the setting of the VOI for subsequent procedures. It is configured to do at least one of the following: The optical system (100) according to any one of claims 1 to 10.

12. A method for determining the volume of interest VOI (200) in a sample (300), wherein the method is: Processing scanning image data (S12) obtained by scanning a sample (300) using a light sheet (400) of an optical system (100) for a light sheet microscope (10), wherein processing the scanning image data includes determining the boundaries (302x, 302y, 302z) of the object (302) in the scanning image data along at least two axes (X, Y, Z), Based on the determined boundary, the proposed volume of interest VOI is defined such that the boundary along each of the axes is included in the proposed volume of interest VOI (S13), A method that includes this.

13. The optical system (100) is the optical system (100) according to any one of claims 1 to 11. The method according to claim 12.

14. A computer program comprising program code for performing the method described in claim 12 or 13 when the computer program is executed on a processor.

15. A computer-readable medium storing instructions for causing a processor to perform the method described in claim 12 or 13 when executed by the processor.

Images