Method and system for constructing a digital image depicting an artificially stained specimen - Patents.com

Abstract

The inventive concept relates to a method (30) and apparatus (10) for training a machine learning model to construct a digital image (602) depicting an artificially stained sample, the method (30) comprising: receiving (S300) a training set of digital images of an unstained sample, the training set of digital images being obtained by illuminating the unstained sample from a plurality of directions and capturing a digital image for each of the plurality of directions; receiving (S302) a ground truth comprising digital images of a stained sample, the stained sample being formed by applying a staining agent to the unstained sample; and training (S304) a machine learning model to construct a digital image (602) depicting the artificially stained sample using the received training set of digital images of the unstained sample and the received ground truth. The inventive concept further relates to a microscope system (20) and a method (40) for constructing a digital image (602) depicting an artificially stained sample.

Description

[Technical field]

[0001] The inventive concepts relate to methods and apparatus for training machine learning models for constructing digital images that represent artificially stained samples. The inventive concepts further relate to methods and microscope systems for constructing digital images that represent artificially stained samples. [Background technology]

[0002] 2. Background of the Invention In the field of digital microscopy, a common task is to find and identify objects within a sample, for example in the fields of hematology and cytology, where a specific cell type needs to be found and identified in order to establish a diagnosis for the patient from whom the sample was taken.

[0003] Samples are commonly stained to enhance contrast within the sample and allow objects of interest within the sample to be detected. For example, white blood cells are semi-transparent and therefore difficult to identify under a microscope without staining the sample. However, the chemicals used for staining can be toxic, which requires laboratory personnel to follow safety procedures (e.g., use of protective equipment and / or fume hoods) to avoid exposure to such chemicals. Additionally, the staining process is a complex process and can increase the time required for analysis. Due to its complexity, it is often difficult to ensure consistent results from the staining process. For example, depending on the quality of the chemicals used, the degree of staining as well as the color of the resulting sample may vary. Additionally, staining of samples prior to analysis can increase the economic costs associated with the analysis due to the chemicals required, safety procedures, and additional time required.

[0004] Also, it is usually beneficial to quickly screen a sample for a particular type of cell. Such screening requires a compromise between speed and accuracy. To perform high-precision screening, the sample generally needs to be greatly magnified and the cells imaged and analyzed. Thus, only a portion of the sample is imaged at a time, and to screen the entire sample, many individual positions of the sample must be imaged, leading to a time-consuming screening process. Therefore, in order to reduce the time required for screening, it is possible to reduce the number of imaging positions. However, to screen the entire sample, the magnification needs to be reduced, which in turn reduces the accuracy of the screening, leading to a screening process where cell types may not be properly discovered and identified.

[0005] Therefore, there is a need in the art for improvements. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention aims to at least partially mitigate, alleviate or eliminate one or more of the above identified deficiencies and disadvantages in the art, singly or in any combination, and to at least solve the above problems. [Means for solving the problem]

[0007] According to a first aspect, a method for training a machine learning model to construct a digital image depicting an artificially stained sample is provided. The method of the first aspect includes receiving a training set of digital images of an unstained sample, the training set of digital images being obtained by illuminating the unstained sample from a plurality of directions and capturing a digital image for each of the plurality of directions; receiving ground truth including digital images of stained samples, the stained samples being formed by applying a staining agent to the unstained sample; and using the received training set of digital images of the unstained sample and the received ground truth to train a machine learning model to construct a digital image depicting the artificially stained sample.

[0008] In the context of the present disclosure, the expression "digital image depicting an artificially stained sample" should be interpreted as a computer-generated digital image of an unstained sample. This digital image may be similar or even identical to the digital image of a stained sample. Thus, the depicted artificially stained sample may have a similar contrast to the depicted stained sample (actually stained by a staining agent). In other words, the depicted artificially stained sample may reproduce the sample actually stained by a staining agent.

[0009] In the context of the present disclosure, the expression "unstained sample" should be interpreted as a sample to which no staining agent is applied.Unstained samples may have relatively low contrast compared to stained samples.The contrast of unstained samples may be so low that it is difficult or even impossible to distinguish features of unstained samples in digital images of unstained samples using conventional microscopy.

[0010] In the context of the present disclosure, the expression "ground truth" should be interpreted as information that is known to be authentic and / or true. Thus, in this context, the ground truth may represent a stained sample, since a machine learning model is trained to construct a digital image that depicts an artificially stained sample. Thus, in this context, the ground truth may comprise a digital image of the stained sample. The ground truth may comprise a digital color image of the stained sample. Stated differently, the ground truth digital image may depict the stained sample in color.

[0011] In the context of this disclosure, the term "stain" should be interpreted as one or more chemical agents that can be used to enhance the contrast (e.g., change color) of a sample. The stain can be selected depending on the type of sample and / or what objects in the sample are of interest. For example, in hematology, samples (e.g., blood smears) are typically stained using May-Grünwald-Giemsa (MGG), Wright-Giemsa (WG), or Wright. As a further example, in pathology, samples are typically stained using hematoxylin and eosin (H&E). Different stains may enhance contrast for different objects and / or substances in a sample. For example, some stains enhance contrast, such as proteins. Thus, which stain to use can be selected depending on the sample and / or the type of object of interest in the sample.

[0012] Thus, the machine learning model is trained to correlate a training set of digital images of unstained samples with ground truth (e.g., digital images of stained samples). The machine learning model may be iteratively and / or recursively trained until the difference between the output of the machine learning model (i.e., digital images depicting artificially stained samples) and the ground truth (e.g., digital images of stained samples) is less than a predetermined threshold. The smaller the difference between the output of the machine learning model and the ground truth, the higher the accuracy of the constructed digital image depicting the artificially stained sample provided by the machine learning model. In other words, the smaller the difference between the output of the machine learning model and the ground truth, the higher the likelihood that the constructed digital image depicting the artificially stained sample will reproduce the digital image of the stained sample to a higher degree. Thus, it is preferable to minimize the difference between the output of the machine learning model and the ground truth. The machine learning model may be trained to construct digital images of artificially stained samples of a plurality of different sample types. In such cases, a machine learning model can be trained for each sample type using a training set of digital images of samples of that sample type and the corresponding ground truth associated with each sample type.

[0013] By illuminating the sample from multiple different directions and capturing digital images for each of the multiple directions, information about the unstained sample regarding finer details than would normally be resolvable by a conventional microscope (i.e., using conventional microscope illumination) used to image the unstained sample may be captured. This may be understood as different portions of Fourier space (i.e., spatial frequency domain) associated with the unstained sample being imaged for different illumination directions. This technique is known in the art as Fourier ptychography. Furthermore, by illuminating the unstained sample from multiple different directions and capturing digital images for each of the multiple directions, information about the refractive index associated with the unstained sample may be captured. This may be understood as the effect of light refraction being dependent on the angle of incidence of the light illuminating the unstained sample and the refractive index of the unstained sample. Information about the refractive index of the unstained sample may in turn enable phase information (commonly referred to in the art as quantitative phase) associated with the unstained sample to be determined. Because the multiple digital images contain information related to one or more of fine details of the unstained sample, refractive index associated with the unstained sample, and phase information associated with the unstained sample, this information may be used in training a machine learning model, which in turn may enable a machine learning model to be trained to more accurately construct a digital image depicting an artificially stained sample than would be allowed if the multiple digital images were captured from only one direction or by using conventional microscopy. Using conventional microscopy (e.g., by illuminating the unstained sample from most of multiple directions up to the numerical aperture of the microscope objective lens used to image the unstained sample), it may be difficult or even impossible to capture information related to refractive index associated with the unstained sample and / or phase information associated with the unstained sample.Illuminating the unstained sample from multiple different directions may further allow for capturing information relating to finer details of the unstained sample than would normally be permitted by the microscope objective lens used to image the unstained sample. Thus, a microscope objective lens with a relatively low magnification may be used to capture information relating to finer details of the unstained sample. By using a microscope objective lens with a relatively low magnification, a larger portion of the unstained sample may be imaged at each imaging position. Thus, the entire unstained sample may be scanned by imaging at relatively fewer positions, which may allow for faster scanning of the unstained sample.

[0014] Thus, in accordance with the concepts of the present invention, a plurality of digital images of an unstained sample can be used to train a machine learning model for constructing a digital image that reproduces a stained sample. The trained machine learning model can then be used to construct a digital image that depicts a stained sample (i.e., an artificially stained sample) without applying a staining agent to the unstained sample.

[0015] The method of the first aspect may further include obtaining a training set of digital images of the unstained sample by illuminating the unstained sample from a plurality of directions and capturing a digital image of the unstained sample for each of the plurality of directions.

[0016] A training set of digital images may be obtained by illuminating an unstained sample with white light from multiple directions and capturing a digital image for each of the multiple directions.

[0017] In the context of the present disclosure, the expression "white light" should be interpreted as light having a relatively broad wavelength spectrum. White light may, for example, be similar to sunlight. The spectrum of white light may include most of the visible light spectrum, as opposed to light having a narrow spectrum (e.g., monochromatic light).

[0018] A related advantage is that white light (i.e., broad spectrum light) may allow more information about the sample to be captured in the training set of digital images that may be missing in parts of the spectrum not covered when using a narrowband light source. That information is used by the trained machine learning model in constructing digital images that depict the artificially stained sample. Imaging applications that utilize Fourier ptychography generally require monochromatic light, and thus, in prior art systems, multiple differently colored monochromatic lights may be required to capture multispectral information about the sample. Thus, the use of white light may reduce the total number of digital images required to capture multispectral information about the sample compared to prior art systems that utilize Fourier ptychography. Additionally, the use of differently colored monochromatic lights may only allow information about the sample in those specific wavelength ranges, and because white light may have a broader spectrum, the use of white light may allow information about the sample in a broader wavelength range compared to using monochromatic light.

[0019] The training set of digital images can be acquired using a microscope objective and an image sensor, and at least one of the multiple directions can correspond to an angle greater than the numerical aperture of the microscope objective.

[0020] The numerical aperture of a microscope objective may be a dimensionless number related to the range of angles over which the microscope objective accepts light. Thus, a direction greater than the numerical aperture may be understood to be a direction corresponding to an angle greater than the range of angles over which the microscope objective accepts light.

[0021] By illuminating the sample from an angle larger than the numerical aperture of the microscope objective, the digital image captured for that illumination angle may contain information about higher spatial frequencies and therefore finer details of the unstained sample than the microscope objective would normally allow. This allows the microscope objective to capture phase information related to the unstained sample and information related to details that the microscope objective would not normally be able to resolve, which can be used to train machine learning models. In other words, illuminating the unstained sample from an angle larger than the numerical aperture of the microscope objective can improve the training of machine learning models to construct digital images that represent artificially stained samples.

[0022] The method of the first aspect may further include applying a staining agent to an unstained sample to form a stained sample; and acquiring a digital image of the stained sample to form a ground truth comprising the digital image of the stained sample. In other words, the stained sample and the unstained sample may be the same sample, differing only with respect to the applied staining agent.

[0023] A related advantage is the ability to pair features (e.g., objects) in a stained sample with corresponding features in an unstained sample, allowing for improved training of machine learning models to construct digital images that depict the artificially stained sample.

[0024] The act of acquiring a digital image of the stained specimen may include illuminating the stained specimen from a subset of the multiple directions simultaneously and capturing a digital image of the stained specimen while the stained specimen is illuminated from the subset of the multiple directions simultaneously.

[0025] In the context of the present disclosure, the expression "subset of the multiple directions" should be interpreted as one or more directions of the multiple directions. The subset directions and the number of directions can be selected such that the illumination of the stained sample is similar to conventional microscope illumination (e.g., bright field illumination). Thus, the digital image of the stained sample can resemble a digital image captured by a conventional microscope system. The subset of directions can be, for example, most or all of the multiple directions.

[0026] A related advantage is that the same microscope system, and in particular the same illumination system, can be used to capture the training set of digital images of unstained and stained samples. This allows for more efficient handling of unstained samples during collection of the digital images used for training. For example, the unstained sample can be stained while still stationary within the microscope system after the training set of digital images has been captured. Thus, improved pairing between features of the stained sample and corresponding features of the unstained sample can be achieved. Using the same illumination system can further reduce the economic costs associated with collecting the digital images required to train the machine learning model.

[0027] The method of the first aspect may further include receiving a reconstruction set of digital images of the stained sample, the reconstruction set may be obtained by illuminating the stained sample from a plurality of directions and capturing a digital image for each of the plurality of directions; and reconstructing the digital image of the stained sample using a computational imaging technique and the received reconstruction set of digital images. In the context of the present disclosure, the term "computational imaging technique" should be interpreted as a computational process that can form a digital image by combining information of a plurality of digital images taken under various conditions (e.g., different lighting conditions). Examples of computational imaging techniques that may be suitable in this context are Fourier ptychography and machine learning. The computational imaging technique may be a reconstruction machine learning model trained to reconstruct the digital image of the stained sample. The reconstructed digital image of the stained sample may have a resolution similar to the resolution of the digital images of the reconstruction set. The reconstructed digital image of the stained sample may have a resolution relatively higher than the resolution of one or more digital images of the reconstruction set. Because the reconstruction set is obtained by illuminating the stained sample from multiple directions and capturing digital images for each of the multiple directions, it may be permissible to reconstruct digital images having a resolution relatively higher than the resolution of the digital images in the reconstruction set.

[0028] The method of the first aspect may further include the steps of: forming a stained sample by applying a staining agent to an unstained sample; and obtaining a set of digital image reconstructions of the stained sample by illuminating the stained sample from a plurality of directions and capturing a digital image of the stained sample for each of the plurality of directions.

[0029] The reconstructed set of digital images may be acquired using a microscope objective and an image sensor, where at least one of the multiple directions corresponds to an angle greater than the numerical aperture of the microscope objective.

[0030] By illuminating the stained sample from an angle larger than the numerical aperture of the microscope objective, the digital image captured for that illumination angle may contain information about higher spatial frequencies and therefore finer details of the stained sample than the microscope objective would normally allow. This allows the microscope objective to capture phase information related to the stained sample and information related to details that the microscope objective would not normally be able to resolve, which can be used by computational imaging techniques in reconstructing the digital image of the stained sample. In other words, by illuminating the stained sample from an angle larger than the numerical aperture of the microscope objective, the reconstruction of the digital image of the stained sample can be improved. In particular, a digital image of the stained sample having a relatively higher resolution than the digital images of the reconstruction set can be reconstructed.

[0031] The ground truth may consist of digital images having a relatively higher resolution than the digital images of the training set of digital images.

[0032] A related advantage is that the machine learning model can be trained to construct a digital image of an artificially stained sample that has a relatively higher resolution than one or more digital images of the training set. In other words, the trained machine learning model can be used to construct a digital image depicting an artificially stained sample that has a relatively higher resolution than one or more of the digital images input to the trained machine learning model. Thus, the digital image (e.g., the digital image of an unstained sample) used as the input of the trained machine learning model can be captured using a microscope objective lens with a low numerical aperture, while allowing the constructed digital image depicting the artificially stained sample to have a similar resolution to the digital image captured using a microscope objective lens with a relatively high numerical aperture.

[0033] According to a second aspect, there is provided a method of constructing a digital image depicting an artificially stained sample. The method of the second aspect includes constructing a digital image depicting the artificially stained sample by receiving an input set of digital images of an unstained sample, the input set of digital images being obtained by illuminating the unstained sample from a plurality of directions and capturing a digital image for each of the plurality of directions, inputting the input set of digital images into a machine learning model trained according to the method of the first aspect, and receiving an output from the machine learning model including the digital image depicting the artificially stained sample.

[0034] By inputting an input set of digital images into the machine learning model trained by the method of the first aspect, the process of imaging the unstained sample may become more efficient, since the digital image of the unstained sample is used to output a digital image of the artificially stained sample from the trained machine learning model. In other words, the trained machine learning model can reproduce the digital image of the stained sample to output a digital image of the artificially stained sample. Thus, there is no need to apply a staining agent to the unstained sample before imaging. This, in turn, may allow for faster and / or more cost-effective imaging of the sample. Furthermore, the need for manual handling of potentially toxic chemicals (e.g., staining agents or other related chemicals) required during staining may be eliminated, which may allow for a safer imaging process for people (e.g., laboratory personnel).

[0035] An input set of digital images of an unstained sample may be acquired using a microscope objective and an image sensor, and at least one of the multiple directions may correspond to an angle greater than the numerical aperture of the microscope objective.

[0036] The input set of digital images may be obtained by illuminating an unstained sample with white light from multiple directions and capturing a digital image for each of the multiple directions.

[0037] The above-mentioned features of the first aspect also apply to this second aspect, where applicable: to avoid undue repetition, please see above.

[0038] According to a third aspect, there is provided an apparatus for training a machine learning model for constructing digital images depicting artificially stained samples, the apparatus comprising: a circuit configured to perform a first receiving function configured to receive a training set of digital images, the training set of digital images being obtained by illuminating an unstained sample from a plurality of directions and capturing a digital image of the unstained sample for each of the plurality of directions; a second receiving function configured to receive a ground truth consisting of digital images of stained samples, the stained samples being formed by applying a staining agent to the unstained sample; and a training function configured to use the received training set of digital images of unstained samples and the received ground truth to train a machine learning model for constructing digital images depicting the artificially stained sample according to the method of the first aspect.

[0039] A training set can be obtained by illuminating an unstained sample with white light from multiple directions and capturing digital images for each of the multiple directions.

[0040] The above mentioned features of the first and / or second aspect also apply to this third aspect, where applicable: in order to avoid excessive repetition, please see above.

[0041] According to a fourth aspect, there is provided a microscope system comprising: an illumination system configured to illuminate an unstained sample from a plurality of directions; an image sensor; at least one microscope objective lens arranged to image the unstained sample onto the image sensor; and circuitry configured to perform an acquisition function configured to control the illumination system to illuminate the unstained sample from each of the plurality of directions and to control the image sensor to capture a digital image for each of the plurality of directions, thereby acquiring an input set of digital images, and an image construction function configured to input the input set of digital images to a machine learning model trained according to the method of the first aspect, and to receive an output from the machine learning model including a digital image depicting the artificially stained sample.

[0042] The illumination system may include a plurality of light sources, each of which may be configured to emit white light, and each of which may be configured to illuminate the unstained sample from one of a plurality of directions.

[0043] A related advantage is the ability to capture information about a sample that pertains to a broader range of the electromagnetic spectrum, as compared to illumination systems with narrowband light sources (eg, monochromatic light sources).

[0044] A further related advantage is that fewer light sources can be used to capture multispectral information about a sample, as compared to illumination systems with narrowband (e.g., monochromatic) light sources. This is because light sources configured to emit white light can emit light over a broader range of wavelengths, as compared to narrowband (e.g., monochromatic) light sources typically used in Fourier ptychography applications. Thus, fewer light sources configured to emit white light may be required to cover a particular range of the electromagnetic spectrum, as compared to narrowband light sources.

[0045] The illumination system can include a plurality of light sources arranged on a curved surface, the curved surface being concave along at least one direction along the surface, and each of the plurality of light sources can be configured to illuminate the unstained sample from one of a plurality of directions.

[0046] Arranging multiple light sources on a curved surface may be advantageous in that the distance from each light source to the current imaging position of the microscope system (i.e., the position or portion of the unstained sample currently being imaged) is similar. Because the distances are similar, the intensity of light emanating from each light source may be similar at the current imaging position. This may be understood as the effect of the inverse square law. Thus, the unstained sample may be illuminated by light having similar intensity for each direction in multiple directions, which may allow for more homogeneous illumination of the unstained sample independent of the illumination direction. It may be advantageous to configure the illumination system such that the distance from each light source to the current imaging position is large enough that each light source can be treated as a point source. This may allow the light to be quasi-coherent at the current imaging position. Thus, the distance from each light source to the current imaging position may be selected such that the intensity of light from each light source at the current imaging position is high enough to generate an input set of digital images.

[0047] The curved surface may be formed with facets, or stated differently, the curved surface may be constructed from a number of flat surfaces.

[0048] A related advantage is that the lighting system is easier to manufacture, reducing the associated economic costs.

[0049] A further related advantage is that the lighting system may be modular, which may facilitate replacing one or more light sources (e.g., if a light source is broken and / or defective).

[0050] The numerical aperture of the at least one microscope objective may be 0.4 or less. In other words, the magnification of the at least one microscope objective may be 20x or less.

[0051] A related advantage is that a large portion of the unstained sample can be imaged at one time, compared to a microscope objective having a high numerical aperture. This, in turn, can allow for a reduction in the number of individual imaging positions required to image a large portion of the unstained sample. Thus, the time required to image a large portion of the unstained sample can be reduced. This can be advantageous, particularly when the machine learning model is trained to construct a digital image having a relatively greater resolution than the digital images input to the machine learning model (i.e., the input set of digital images). Thus, the unstained sample can be imaged more quickly, while the digital image depicting the artificially stained sample can have a relatively higher resolution than the resolution normally permitted by at least one microscope objective.

[0052] The above-mentioned features of the first, second and / or third aspects also apply to this fourth aspect, where applicable: in order to avoid excessive repetition, please see above.

[0053] According to a fifth aspect there is provided a non-transitory computer readable storage medium comprising program code portions which, when executed on a device having processing capability, perform a method according to the first aspect or a method according to the second aspect.

[0054] The above-mentioned features of the first, second, third and / or fourth aspects also apply to this fifth aspect, where applicable: in order to avoid undue repetition, please see above.

[0055] Further scope of applicability of the present disclosure will become apparent from the detailed description set forth below. However, it should be understood that the detailed description and specific examples, while indicating preferred variations of the inventive concept, are given by way of illustration only, since various changes and modifications within the scope of the inventive concept will become apparent to those skilled in the art from this detailed description.

[0056] Thus, it is to be understood that the inventive concept is not limited to the particular steps of the described method or components of the described system, as such methods and systems may vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting. It should be noted that, as used in this specification and the appended claims, the articles "a," "an," "the," and "said" are intended to mean that there are one or more elements, unless the context clearly dictates otherwise. Thus, for example, reference to "a unit" or "the unit" may include a plurality of devices, etc. Furthermore, the use of "comprising," "including," "containing," and similar terms does not exclude other elements or steps.

[0057] BRIEF DESCRIPTION OF THE DRAWINGS The above and other aspects of the inventive concept will now be described in more detail with reference to the accompanying drawings, which show variations of the inventive concept. The figures should not be considered as limiting the inventive concept to a particular variation, but are used to explain and understand the inventive concept. As shown in the figures, the sizes of layers and regions have been exaggerated for illustrative purposes, and are therefore provided to illustrate the general structure of the variations of the inventive concept. Like reference numerals refer to like elements throughout. [Brief description of the drawings]

[0058] [Figure 1] FIG. 1 illustrates an apparatus for training a machine learning model. [Diagram 2]FIG. 2 is a diagram showing a microscope system. [Diagram 3] FIG. 3 shows a block scheme of a method for training a machine learning model to construct digital images depicting artificially stained samples. [Figure 4] FIG. 4 shows a block scheme of how a trained machine learning model is used to construct a digital image depicting an artificially stained sample. [Diagram 5] FIG. 5 shows an illumination system having a curved surface formed by facets. [Figure 6A] FIG. 6A shows a digital image of an unstained sample. [Figure 6B] FIG. 6B shows a digital image of an artificially stained sample. [Figure 6C] FIG. 6C shows a digital image of the stained sample. [Figure 7] FIG. 7 illustrates a non-transitory computer-readable storage medium. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0059] Detailed Description The inventive concepts will now be more fully described with reference to the accompanying drawings, in which presently preferred versions of the inventive concepts are shown, however, the inventive concepts may be embodied in many different forms and should not be construed as limited to the versions set forth herein, but rather these versions are provided for thoroughness and completeness and to fully convey the scope of the inventive concepts to those skilled in the art.

[0060] First, an apparatus 10 and method 30 for training a machine learning model for constructing a digital image depicting an artificially stained specimen will be described with reference to FIGS. 1 and 3. FIG.

[0061] FIG. 1 shows an apparatus 10 for training a machine learning model for constructing a digital image depicting an artificially stained sample. Here, a "digital image depicting an artificially stained sample" should be interpreted as a digital image of an unstained sample colored by a computer. This digital image may be similar or even identical to a digital image of a stained sample (i.e., an unstained sample to which a staining agent has been applied). Thus, the depicted artificially stained sample may have a similar contrast to the depicted stained sample (actually stained by a staining agent). The apparatus 10 may be a computing device. Examples of suitable computing devices include computers, servers, smartphones, tablets, etc. The apparatus 10 may further be implemented as part of a cloud server and / or a distributed computing facility. It is further understood that the apparatus 10 may comprise further components, such as an input device (mouse, keyboard, touch screen, etc.) and / or a display. The apparatus 10 may further comprise a power source, such as a power connection, a battery, etc. The apparatus 10 comprises a circuit 100. As shown in the example of FIG. 1, the circuit 100 may include one or more of a memory 110, a processing unit 120, a transceiver 130, and a data bus 140. The memory 110, the processing unit 120, and the transceiver 130 may communicate via the data bus 140. The processing unit 120 may be a central processing unit (CPU). The transceiver 130 may be configured to communicate with an external device. For example, the transceiver 130 may be configured to communicate with a server, a computer external peripheral device (e.g., an external storage device), and the like. The external device may be a local device or a remote device (e.g., a cloud server). The transceiver 130 may be configured to communicate with the external device via an external network (e.g., a local area network, the Internet, etc.). The transceiver 130 may be configured for wireless and / or wired communication. Suitable technologies for wireless communication are known to those skilled in the art. Some non-limiting examples include Wi-Fi and Near Field Communication (NFC). Suitable techniques for wired communications are known to those skilled in the art.Some non-limiting examples include USB, Ethernet, and Firewire.

[0062] The memory 110 may be a non-transitory computer-readable storage medium. The memory 110 may be a random access memory. The memory 110 may be a non-volatile memory. As shown in the example of FIG. 1, the memory 110 may store program code portions corresponding to one or more functions. The program code portions may be executable by a processing unit to thereby perform the functions. Thus, when it is mentioned that the circuit 100 is configured to perform a particular function, the processing unit 120 may execute the program code portions corresponding to the particular function that may be stored on the memory 110. However, it should be understood that one or more functions of the circuit 100 may be implemented in hardware and / or in a particular integrated circuit. For example, one or more functions may be implemented using a field programmable gate array (FPGA). Stated differently, one or more functions of the circuit 100 may be implemented in hardware or software, or as a combination of the two.

[0063] The circuit 100 is configured to perform a first receiving function 1100, a second receiving function 1102, and a training function 1104. As shown in FIG. 1, the circuit 100 may be further configured to perform one or more of a reconstruction function 1106 and a third receiving function 1108.

[0064] The first receiving function 1100 is configured to receive a training set of digital images. The training set of digital images is obtained by illuminating an unstained sample from a plurality of directions and capturing a digital image of the unstained sample for each of the plurality of directions. In other words, the training set of digital images of the unstained sample may be obtained by illuminating an unstained sample from a plurality of directions and capturing a digital image of the unstained sample for each of the plurality of directions. The training set of digital images may be obtained by illuminating an unstained sample with white light from a plurality of directions and capturing a digital image for each of the plurality of directions. In other words, the training set of digital images of the unstained sample may be obtained by illuminating an unstained sample with white light from a plurality of directions and capturing a digital image of the unstained sample for each of the plurality of directions. The unstained sample may be a sample to which no staining agent has been applied. The unstained sample may have a relatively low contrast compared to a stained sample. The contrast of an unstained sample may be so low that it is difficult or even impossible to image features of the unstained sample using conventional microscopy. For example, white blood cells are usually almost transparent, making it difficult to image the sample without staining it. The first receiving function 1100 may be configured to receive a training set of digital images via the transceiver 130. For example, the training set of digital images may be captured using an external microscope system and then transmitted to the transceiver 130 of the device 10. As a further example, the device 10 may form part of a microscope system and the training set of digital images may be received from an image sensor of the microscope system. The memory 110 may be configured to store the training set of digital images and the first receiving function 1100 may be configured to receive the training set of digital images from the memory 110. The training set of digital images may be acquired using a microscope objective and an image sensor. The unstained sample may be illuminated sequentially from at least one of a plurality of directions.Each digital image of the training set of digital images may be captured when the unstained sample is illuminated from at least one of a plurality of directions. Thus, the unstained sample may be illuminated from a single direction of the plurality of directions or may be illuminated from multiple directions of the plurality of directions simultaneously. At least one of the plurality of directions may correspond to an angle greater than the numerical aperture of the microscope objective. The numerical aperture of the microscope objective may be a dimensionless number related to the range of angles at which the microscope objective accepts light. Thus, a direction greater than the numerical aperture may be understood as a direction corresponding to an angle greater than the range of angles at which the microscope objective is configured to accept light when used for bright field microscopy. Because the unstained sample may be illuminated from a plurality of different directions and digital images may be captured for each of the plurality of directions, information of the unstained sample relating to finer details than can normally be resolved by the microscope objective used to image the unstained sample may be captured. This may be understood as different portions of Fourier space (i.e., spatial frequency domain) associated with the unstained sample being imaged for different illumination directions. This technique is known in the art as Fourier ptychography. In general, in Fourier ptychography, high spatial frequencies in Fourier space associated with a sample (e.g., an unstained sample) are sampled when the sample is illuminated from a direction corresponding to a large angle of incidence. Thus, even higher spatial frequencies in Fourier space may be sampled when the unstained sample is illuminated from a direction corresponding to an angle larger than the numerical aperture of the microscope objective. This is possible because light is scattered by the unstained sample and some of the light scattered by the unstained sample may be collected by the microscope objective. This illumination technique may further enable information about the refractive index associated with the unstained sample to be captured by at least one microscope objective. This can be understood as an effect, where the refraction of light depends on the angle of incidence of the light illuminating the unstained sample and the refractive index of the unstained sample.Information about the refractive index of the unstained sample may then allow for determining phase information (commonly referred to in the art as quantitative phase) associated with the unstained sample. Because the digital images include information related to one or more of the fine details of the unstained sample, the refractive index associated with the unstained sample, and the phase information associated with the unstained sample, this information may be used in training a machine learning model, which may then allow the machine learning model to be trained to more accurately construct a digital image depicting the artificially stained sample than would be permitted if the digital images were captured from only one direction. In other words, the machine learning model may be trained to more accurately construct a digital image depicting the artificially stained sample than would be permitted if the digital images were captured from only one direction or using a conventional microscope (e.g., using bright field illumination). It should be understood that information related to one or more of fine details of an unstained sample, refractive index associated with an unstained sample, and phase information associated with an unstained sample may be captured by illuminating an unstained sample from one or more of a plurality of different directions at a time, for example, from a subset of the plurality of directions. The subset of the plurality of directions may include directions corresponding to different portions of the Fourier space of the unstained sample. The different portions of the Fourier space of the unstained sample may be partially overlapping or non-overlapping. Thus, information related to fine details of an unstained sample may be captured while using a microscope objective having a relatively low magnification. For example, a microscope objective having a numerical aperture of 0.4 may capture information related to fine details by illuminating an unstained sample from multiple directions as well as a microscope objective having a numerical aperture of 1.25 used in conventional microscopy (e.g., bright field illumination). Stated another way, by illuminating an unstained sample from multiple directions, a microscope objective with 20x magnification can be used with traditional microscopy (e.g., using brightfield illumination) to capture as much information about fine detail as a microscope objective with 100x magnification.It should be understood that the magnifications and numerical apertures mentioned above are exemplary and the invention may be practiced with other magnifications and / or numerical apertures. A suitable microscope system comprising a microscope objective and an image sensor is described in relation to FIG.

[0065] The second receiving function 1102 is configured to receive ground truth comprising a digital image of a stained sample. The ground truth may be information known to be authentic and / or true. In this context, the ground truth may represent a stained sample since the machine learning model is trained to construct a digital image that depicts an artificially stained sample. For example, the ground truth may comprise a digital image of a stained sample. The ground truth may be received via the transceiver 130. For example, the ground truth may be formed on a different device and / or stored on a different device and transmitted to the device 10 via the transceiver 130. The stained sample is formed by applying a staining agent to an unstained sample. The stained sample may be formed such that the relative positions of features (e.g., objects) in the stained sample remain substantially unperturbed compared to the unstained sample. In other words, the features of the unstained sample and the corresponding features of the stained sample can be in the same relative positions (i.e., relative positions to the unstained sample or the stained sample). This allows features of the stained sample to be correlated (or paired) with corresponding features of the unstained sample. Alternatively or additionally, the machine learning model can be trained using CycleGAN. As known in the art, CycleGAN is a technique that can train an image-to-image transformation model without paired examples (i.e., without having paired examples of features in stained samples and features in unstained samples). This can be advantageous because the machine learning model can be trained without the features of the stained sample being correlated (or paired) with the corresponding features of the unstained sample, allowing the unstained sample to be stained in a simpler way.

[0066] A staining agent may be one or more chemical agents configured to enhance the contrast of an unstained sample when applied to the unstained sample (e.g., by changing the color of the unstained sample). A staining agent may be selected depending on the type of sample and / or what objects in the sample are of interest. For example, in hematology, samples (e.g., blood smears) are typically stained using May-Grünwald-Giemsa (MGG), Wright-Giemsa (WG), or Wright. As a further example, in pathology, samples are typically stained using Hematoxylin and Eosin (H&E). Different staining agents may enhance contrast for different objects and / or substances in a sample. For example, some staining agents enhance contrast, such as proteins. Thus, which staining agent to use may be selected depending on the sample and / or depending on the type of object of interest in the sample. A ground truth that constitutes a digital image of a stained sample may be formed by acquiring a digital image of a stained sample. A digital image of a stained sample may be acquired by simultaneously illuminating the stained sample from a subset of multiple orientations. The subset of the multiple directions may be one or more of the multiple directions. The subset directions and the number of directions may be selected such that the illumination of the stained sample is similar to conventional microscope illumination (e.g., bright field illumination). The subset of directions may be, for example, most or all of the multiple directions. The digital image of the stained sample may be captured while the stained sample is simultaneously illuminated from the subset of the multiple directions. Thus, the digital image of the stained sample may be similar to a digital image captured by a conventional microscope system using, for example, bright field illumination. A training set of digital images may be acquired prior to the ground truth digital image. This is advantageous because the training set may comprise digital images of an unstained sample and the ground truth may comprise digital images of the same sample with a stain applied.

[0067] The digital image of the stained sample may be obtained in a manner other than the above. For example, the digital image of the stained sample may be reconstructed using a computational imaging technique. Here, the term "computational imaging technique" should be understood as a computational process that can form a digital image by combining information from multiple digital images captured under various conditions (e.g., different lighting conditions). Examples of computational imaging techniques that may be suitable in this context are Fourier ptychography and machine learning. The computational imaging technique may be a reconstruction machine learning model trained to reconstruct the digital image of the stained sample. The reconstructed digital image of the stained sample may have a resolution similar to the resolution of the digital images of the reconstruction set. The reconstructed digital image of the stained sample may have a resolution relatively higher than the resolution of one or more digital images of the reconstruction set. Since the reconstruction set is obtained by illuminating the stained sample from multiple directions and capturing digital images for each of the multiple directions, reconstruction of a digital image having a resolution relatively higher than the resolution of the digital images of the reconstruction set may be permitted. Thus, the third receiving function 1108 may be configured to receive the reconstruction set of the digital images of the stained sample. The reconstruction set may be received via the transceiver 130. For example, the reconstruction set may be formed and / or stored on a different device and transmitted to the device 10 via the transceiver 130. The reconstruction set may be obtained by illuminating the stained sample from a plurality of directions and capturing a digital image for each of the plurality of directions. Stated another way, the reconstruction set of digital images of the stained sample may be obtained by illuminating the stained sample from a plurality of directions and capturing a digital image of the stained sample for each of the plurality of directions. The reconstruction set of digital images may be obtained using a microscope objective and an image sensor, where at least one of the plurality of directions may correspond to an angle greater than the numerical aperture of the microscope objective.Thus, the reconstruction set of digital images of the stained sample may be obtained in a similar manner to the training set of digital images of the unstained sample, and therefore the above description relating to the training set applies mutatis mutandis to the reconstruction set of digital images of the stained sample. The reconstruction functionality 1106 may be configured to reconstruct the digital image of the stained sample using the computational imaging technique and the received reconstructed digital image set. By illuminating the stained sample from an angle larger than the numerical aperture of the microscope objective, the digital image captured for that illumination angle may contain information about higher spatial frequencies and therefore finer details of the stained sample than the microscope objective would normally allow. The illumination technique may further enable the microscope objective to capture phase information related to the stained sample and information related to details that are not normally resolvable by the microscope objective, which information may be used by the computational imaging technique in reconstructing the digital image of the stained sample. In other words, by illuminating the stained sample from an angle larger than the numerical aperture of the microscope objective, the reconstruction of the digital image of the stained sample may be improved. In particular, a digital image of the stained sample having a relatively higher resolution than the digital images of the reconstruction set may be reconstructed. Thus, a digital image of the stained sample may be reconstructed from the reconstructed set of digital images of the stained sample using computational imaging techniques and receiving the reconstructed set of digital images of the stained sample.

[0068] The ground truth may comprise a digital image having a relatively higher resolution than the digital images of the training set of digital images. The digital image of the stained sample may be a digital image having a relatively higher resolution than each digital image of the training set of digital images. As described above, the digital image of the stained sample may be reconstructed using computational imaging techniques, and the reconstructed digital image may have a relatively higher resolution than the resolution of the digital images of the reconstructed set of digital images. Thus, the computational imaging techniques may be configured to reconstruct the digital image of the stained sample such that the resolution of the reconstructed digital image is higher than the one or more digital images of the training set of digital images of the unstained sample. Alternatively, or additionally, the digital image of the stained sample may be captured using a microscope objective having a relatively higher magnification than the microscope objective used to capture the one or more digital images of the training set of digital images of the unstained sample. In other words, the microscope objective lens used to capture the digital image of the stained sample may have a relatively large magnification, so that the digital image may depict a smaller area of ​​the stained sample compared to an image captured using the microscope objective lens used to capture the training set of digital images. For this reason, the digital image of the stained sample may be formed from multiple individual digital images of the stained sample, for example, by combining (e.g., stitching) the multiple individual digital images. This allows the digital image of the stained sample to depict an area of ​​the stained sample that corresponds to an area of ​​the unstained sample imaged by the microscope objective lens used to image the training set of digital images. Alternatively, each digital image of the training set may be cropped such that the area of ​​the unstained sample depicted in each cropped digital image of the training set is similar and / or equivalent to the area of ​​the stained sample in the digital image of the stained sample.

[0069] The training function 1104 is configured to train a machine learning model to construct a digital image depicting an artificially stained sample using the received training set of digital images of the unstained sample and the received ground truth. The machine learning model may be a convolutional neural network. The machine learning model may be a convolutional neural network suitable for digital image construction. The training function 1104 is configured to receive ground truth comprising a digital image of a stained sample. The training function 1104 may receive the ground truth from the second receiving function 1102. The training function 1104 is further configured to obtain a training set of digital images of the unstained sample. As described above, the training set of digital images of the unstained sample is obtained by illuminating the unstained sample from multiple directions (possibly with white light) and capturing a digital image for each of the multiple directions. The training function 1104 is further configured to train the machine learning model to construct a digital image depicting an artificially stained sample using the training set of digital images of the unstained sample and the received ground truth. Thus, in the concept of the present invention, a machine learning model can be trained to construct a digital image that reproduces a stained sample using a plurality of digital images of unstained samples. Thus, the trained machine learning model can be used to construct a digital image that depicts a stained sample (i.e., an artificially stained sample) without applying a staining agent to the unstained sample. When the ground truth comprises a digital image of a stained sample that has a relatively higher resolution than one or more digital images of a training set of digital images of unstained samples, the machine learning model can be trained to construct a digital image of an artificially stained sample that has a relatively higher resolution than one or more digital images of the digital images (e.g., training set) that are input to the machine learning model.The training function 1104 may be configured to iteratively and / or recursively train the machine learning model until the difference between the output of the machine learning model (i.e., a digital image depicting an artificially stained sample) and the ground truth (e.g., a digital image depicting a stained sample) is smaller than a predetermined threshold. Thus, the training function 1104 may train the machine learning model to correlate a training set of digital images of unstained samples to the ground truth (e.g., a digital image depicting a stained sample). A smaller difference between the output of the machine learning model and the ground truth may indicate a higher accuracy of the digital image depicting the artificially stained sample provided by the machine learning model. Thus, preferably, the difference between the output of the machine learning model and the ground truth may be minimized. Those skilled in the art will appreciate that the function to be minimized (e.g., a loss function) may be associated with a tolerance range. For example, the loss function may be considered to be minimized even if the minimized loss function has a value that is not a local and / or global minimum.

[0070] The machine learning model may be trained using multiple training sets and multiple corresponding ground truths. In other words, the machine learning model may be trained using multiple different samples. This can improve the training of the machine learning model to construct digital images depicting artificially stained samples. The machine learning model may be trained to construct digital images of artificially stained samples of multiple different sample types. In that case, the machine learning model may be trained using, for each sample type, a training set of digital images of unstained samples of that sample type and corresponding ground truths associated with stained samples of each sample type (e.g., digital images of stained samples of that sample type). This allows the trained machine learning model to construct digital images depicting artificially stained samples of different sample types.

[0071] A microscope system 20 and a method 40 for constructing a digital image depicting an artificially stained sample will now be described with reference to Figs. 2 and 4. Fig. 2 shows a microscope system 20. The microscope system 20 of Fig. 2 is suitable for acquiring a set of digital images of unstained samples to be used by a machine learning model trained according to the above to construct a digital image depicting an artificially stained sample. Furthermore, the microscope system 20 of Fig. 2 may be suitable for acquiring a training set of digital images of unstained samples and / or digital images of stained samples to be used to train a machine learning model according to the above. The microscope system 20 comprises an illumination system 260, an image sensor 270, at least one microscope objective lens 280, and a circuit 200. The microscope system 20 may further comprise a sample holder 290 as shown in the example of Fig. 2. The circuit 200 may comprise one or more of a memory 210, a processing unit 220, a transceiver 230, and a data bus 240. The processing unit 220 may include a central processing unit (CPU) and / or a graphics processing unit (GPU). The transceiver 230 may be configured to communicate with an external device. For example, the transceiver 230 may be configured to communicate with a server, a computer external peripheral (e.g., an external storage device), and the like. The external device may be a local device or a remote device (e.g., a cloud server). The transceiver 230 may be configured to communicate with the external device over an external network (e.g., a local area network, the Internet, etc.). The transceiver 230 may be configured for wireless and / or wired communication. Suitable technologies for wireless communication are known to those skilled in the art. Some non-limiting examples include Wi-Fi and Near Field Communication (NFC). Suitable technologies for wired communication are known to those skilled in the art. Non-limiting examples include USB, Ethernet, Firewire. The memory 210, the processing unit 220, and the transceiver 230 may communicate via a data bus 240.The lighting system 260 and / or the image sensor 270 may be configured to communicate with the circuit 200 via the transceiver 230 as shown in FIG. 2. Additionally or alternatively, the lighting system 260 and / or the image sensor 270 may be configured to communicate directly (e.g., via a wired connection) with the data bus 240. The memory 210 may be a non-transitory computer-readable storage medium. The memory 210 may be a random access memory. The memory 210 may be a non-volatile memory. As shown in the example of FIG. 2, the memory 210 may store program code portions corresponding to one or more functions. The program code portions may be executable by the processing unit 220 thereby to perform the functions. Thus, when the circuit 200 is referred to as being configured to perform a particular function, the processing unit 220 may execute the program code portions corresponding to the particular function that may be stored on the memory 210. However, it should be understood that one or more functions of the circuit 200 may be implemented in hardware and / or in a particular integrated circuit. For example, one or more functions may be implemented using a field programmable gate array (FPGA). Stated another way, one or more functions of circuit 200 may be implemented in hardware or software, or as a combination of the two.

[0072] While image sensor 270 is shown alone in FIG. 2, it should be understood that image sensor 270 may be built into a camera. In the example of FIG. 2, sample holder 290 is a microscope slide having unstained sample 292 applied thereto. It should be understood that sample 292 may be covered with a cover slip (not shown in FIG. 2). Sample holder 290 may be configured to hold a sample 292 to be analyzed. Sample holder 290 may be movable (e.g., by being coupled to a manual and / or motorized stage) such that sample 292 may be moved such that different portions of sample 292 may be imaged by at least one microscope objective 280.

[0073] The illumination system 260 is configured to illuminate the unstained sample 292 from multiple directions. As shown in FIG. 2, the illumination system may be composed of multiple light sources 261. The light sources may be light emitting diodes (LEDs). The light sources may be lasers. The light sources may emit incoherent light, quasi-coherent light, or coherent light. Each light source of the illumination system 260 may be arranged to illuminate the sample 292 from one of multiple directions 262. The illumination system 260 may be configured to simultaneously illuminate the sample 292 with one or more of the multiple light sources 261. In other words, the illumination system 260 may be configured to simultaneously illuminate the sample 292 from one or more of the multiple directions 262. Each light source of the multiple light sources 261 may be configured to emit white light. The light sources may be LEDs configured to emit white light. The LED configured to emit white light may be composed of an LED configured to emit blue light and a phosphor layer configured to convert the emitted blue light to white light. For example, the LED configured to emit white light may be formed by, for example, a blue LED covered with a phosphor layer that emits white light when illuminated with blue light. Alternatively or additionally, the light source may be a laser configured to emit white light. For example, light emitted from a laser may be converted to light having a broader spectral bandwidth. One example of such a conversion process is known in the art as supercontinuum generation. As further shown in FIG. 2, the plurality of light sources 261 may be disposed on a curved surface 264 that is concave along at least one direction along the surface. Each light source of the plurality of light sources 261 may be configured to illuminate the unstained sample 292 from one of a plurality of directions. As shown in the example of FIG. 2, the curved surface 264 may be concave along at least one direction along the surface 264. For example, the curved surface 264 may be a cylindrical surface. The curved surface 264 may be concave along two perpendicular directions along the surface.For example, the curved surface 264 may have a shape similar to a segment of a sphere. The segment of a sphere may be a spherical cap or a spherical dome. Arranging the multiple light sources 261 on the curved surface 264 is advantageous in that the distance R from each light source to the current imaging position P of the microscope system 20 is similar. Because the distances are similar, the intensity of light emanating from each light source may be similar at the current imaging position P. This may be understood as the effect of the inverse square law. Thus, the sample 292 may be illuminated by light having a similar intensity for each of the multiple directions 262, which may allow for a more homogeneous illumination of the sample 292 independent of the illumination direction. The distance R from each light source to the current imaging position P may range from 4 cm to 15 cm. It may be advantageous to configure the illumination system 260 such that the distance R from each light source to the current imaging position P is large enough to treat each light source as a point source. This may allow the light to be quasi-coherent at the current imaging position P. Thus, the distance R from each light source to the current imaging position P may be greater than 15 cm, provided that the intensity of light from each light source at the current imaging position is high enough to generate a series of digital images. In particular, if one or more of the multiple light sources are lasers, the distance R between each light source may be greater than 15 cm. However, it should be understood that the multiple light sources 261 may be arranged on a flat surface or on a surface having an irregular shape. Furthermore, it should be understood that FIG. 2 shows a cross section of the microscope system 20, in particular the illumination system 260. Thus, the curved surface 264 of the illumination system 260 illustrated in FIG. 2 may be a cylindrical surface or a part of a sphere (or a quasi-sphere). The curved surface 264 of the illumination device 260 may be bowl-shaped. The curved surface 264 may be formed of facets 265, which is shown in the example of FIG. 5. In other words, the curved surface 264 may be formed of multiple flat surfaces. The curved surface 264 may be a part of a quasi-sphere consisting of multiple facets or segments. Thus, the curved surface 264 may be part of a surface of a polyhedron. An example of such a polyhedron may be a truncated icosahedron. The light sources 261 may be arranged on the facets 265.Each light source may be arranged such that it is configured to emit light in a direction substantially parallel to the normal of the associated facet. As with the embodiment illustrated in FIG. 2, it should be understood that FIG. 5 illustrates a cross section of the illumination device 260. Thus, the curved surface 264 of the illumination system 260 illustrated in FIG. 5 may be a portion of a quasi-cylindrical surface or a quasi-spherical surface. The curved surface 264 of the illumination system 260 in FIG. 5 may have a shape similar to a bowl. Thus, while the facets 265 in FIG. 5 are illustrated with lines, it should be understood that each facet 265 may be a flat surface having at least three sides. For example, the curved surface 264 may be formed with facets having five sides and facets having six sides (e.g., similar to the inner surface of a football or soccer ball). Although the curved surface 264 in FIG. 5 is illustrated as a continuous surface, it should be understood that each facet may be separate. Thus, the curved surface may be formed by multiple parts, and each facet may be formed by one or more parts. It should further be understood that each portion may include one or more facets. Moreover, such portions may be disposed in contact with adjacent portions or at a distance from adjacent portions. A single portion may include all of the facets. It should further be understood that the number of facets 265 of the illumination system 260 of FIG. 5 is exemplary and that other numbers of facets 265 may be used to form the curved surface 264 of the illumination system 260. It should further be understood that the number of light sources on each facet 265 is exemplary only and that the number may vary.

[0074] The at least one microscope objective 280 is arranged to image the unstained sample 292 onto the image sensor 270. The at least one microscope objective 280 may comprise a first microscope objective. The numerical aperture of the first microscope objective may be 0.4 or less. In other words, the magnification of the first microscope objective may be 20 times or less. Thus, a larger portion of the unstained sample 292 may be imaged at one time, as compared to a microscope objective having a relatively higher numerical aperture. This, in turn, may allow for a reduction in the number of individual imaging positions required to image a larger portion of the unstained sample 292. Thus, the time required to image a larger portion of the unstained sample 292 may be thereby reduced. This may be advantageous, particularly when the machine learning model is trained to construct a digital image having a relatively greater resolution than the digital images input to the machine learning model (i.e., the input set of digital images). Thus, the unstained sample 292 may be imaged more quickly, while the digital images depicting the artificially stained sample may have a relatively higher resolution than the resolution that at least one microscope objective lens would normally allow. The first microscope objective lens may be used in capturing the digital images of the unstained sample 292. The digital images captured using the first microscope objective lens may be used when training the machine learning model (i.e., a training set of digital images of the unstained sample) and / or when constructing the digital images depicting the artificially stained sample using the trained machine learning model. It should further be understood that the first microscope objective lens may be used to form a ground truth. In other words, the first microscope objective lens may be used to capture digital images of the stained sample that are used as ground truth when the machine learning model is trained.

[0075] At least one microscope 280 may include a second microscope objective lens having a magnification of 100x and / or a numerical aperture of 1.25. The second microscope objective lens may be used to acquire digital images having a relatively higher resolution than the digital images acquired using the first microscope objective lens. Thus, the second microscope objective lens may be used to acquire digital images (e.g., digital images of stained samples) for use in training the machine learning model. In other words, the digital images acquired using the second microscope objective lens are used to form the ground truth used in training the machine learning model.

[0076] The numerical aperture and magnification of the first microscope objective lens and / or the second microscope objective lens are exemplary and may be selected, for example, depending on the type of the unstained sample 292. For example, the numerical aperture of the first microscope objective lens may have a magnification of 10 times and / or a numerical aperture of 0.25. It should be understood that the microscope system 20 may include additional optical lenses that may be used with the at least one microscope objective lens 280 to image the unstained sample 292 onto the image sensor 270. For example, the microscope system may include at least one relay lens 285 arranged such that the unstained sample 292 may be imaged onto the image sensor 270 by the at least one microscope objective lens 280 and the at least one relay lens 285, as shown in the example of FIG. 2. It should be further understood that the at least one relay lens 285 may be selected (e.g., focal length, material, size, etc.) depending on the magnification and / or numerical aperture of the at least one microscope objective lens 280. Thus, each microscope objective lens of the at least one microscope objective lens 280 may have a corresponding relay lens of the at least one relay lens. The at least one microscope objective lens 280 may be movable in a longitudinal direction Z by being coupled to a manual stage and / or a motorized stage. The longitudinal direction Z may be parallel to the optical axis of the microscope system 20. In other words, the at least one microscope objective lens 280 may be movable in a light collection direction of the microscope system 20. Alternatively or additionally, the sample holder 290 may be movable along the longitudinal direction Z. The at least one microscope objective lens 280 and / or the sample holder 290 may be movable in a direction such that the unstained sample 292 may be moved such that a focused image may be captured by the image sensor 270 (assuming the illumination system 260 is configured for bright field illumination). The longitudinal direction Z of the at least one microscope objective lens 280 may be controlled by the circuit 200. For example, the circuit 200 may be configured to perform a focus function (not shown in FIG. 2) configured to adjust the position of at least one microscope objective lens 280 along the longitudinal direction Z.The focus function may be configured to automatically adjust the position of the at least one microscope objective lens 280 along the longitudinal direction Z. In other words, the focus function may be an autofocus function.

[0077] The circuit 200 is configured to perform an acquisition function 2100 and an image construction function 2102 .

[0078] The acquisition function 2100 is configured to acquire an input set of digital images by being configured to control the illumination system 260 to illuminate the unstained sample 292 from each of a plurality of directions, such that the input set of digital images of the unstained sample 292 may be acquired using at least one microscope objective 280 and image sensor 270. At least one of the plurality of directions 262 may correspond to an angle greater than the numerical aperture 282 of the microscope objective 280. The acquisition function 2100 may be configured to acquire an input set of digital images by being configured to control the illumination system 260 to illuminate the unstained sample 292 with white light from each of a plurality of directions.

[0079] The acquisition function 2100 is further configured to control the image sensor 270 to capture a digital image for each of the multiple directions. In other words, the acquisition function 2100 may be configured to receive an input set of digital images of an unstained sample 292. The input set of digital images may be acquired by illuminating the unstained sample 292 from the multiple directions 262 and capturing a digital image for each of the multiple directions 262. The input set of digital images may be acquired by illuminating the unstained sample 292 with white light from the multiple directions 262 and capturing a digital image for each of the multiple directions 262. As previously mentioned, at least one of the multiple directions may correspond to an angle greater than the numerical aperture 282 of the microscope objective lens 280. For example, the direction 2620 of FIG. 2 may correspond to an angle greater than the numerical aperture 282 of at least one microscope objective lens 280. Light that is incident on at least one microscope objective lens 280 from a direction 2620 without being scattered may not be able to propagate through the microscope objective lens 280 (i.e., the angle of incidence of light from this direction may be outside the numerical aperture 282 of the microscope objective lens 280) to the image sensor 270. Thus, light from this direction may need to be scattered by the sample 292 in order to propagate through the microscope objective lens 280 to the image sensor 270.

[0080] The image construction function 2102 is configured to input an input set of digital images to a machine learning model trained as described in connection with FIG. 1 and FIG. 3. The image construction function 2102 is further configured to receive an output from the machine learning model comprising a digital image depicting the artificially stained sample. In other words, the image construction function 2102 may be configured to construct a digital image depicting the artificially stained sample by inputting an input set of digital images to a trained machine learning model (e.g., trained in the manner described in connection with FIG. 1 and FIG. 3) and receiving an output from the trained machine learning model comprising a digital image depicting the artificially stained sample. Therefore, the process of imaging the unstained sample 292 may be more efficient because a digital image depicting the artificially stained sample is output from the trained machine learning model using a digital image of the unstained sample 292. Thus, there is no need to apply a staining agent to the unstained sample 292 before imaging. This, in turn, may enable faster and / or more cost-effective imaging of the unstained sample 292. Additionally, it may eliminate the need for manual handling of potentially toxic chemicals required during staining (e.g., stains or other associated chemicals), thereby allowing for a safer imaging process for people (e.g., laboratory personnel).

[0081] FIG. 6A shows a digital image 600 of an unstained sample captured using a 20x microscope objective. In this case, the unstained sample comprises red blood cells 610 and white blood cells 620. In this example, the type of white blood cells 620 is a segmented neutrophil. As seen in FIG. 6A, some features of the unstained sample (e.g., red blood cells 610) are visible. However, capturing an input set of digital images of the unstained sample shown in FIG. 6A and inputting the input set into a trained machine learning model results in the digital image 602 shown in FIG. 6B. In FIG. 6B, it is clear that more features are visible in the artificially stained sample depicted in the digital image 602. In particular, features near the center (in this case, white blood cells 620) are clearly visible in the constructed digital image 602 depicting the artificially stained sample, but are essentially invisible in the digital image 600 of the unstained sample depicted in FIG. 6A. As is further evident from FIG. 6B, some features visible in the digital image 600 of the unstained sample (e.g., red blood cells 610) have increased contrast in the constructed digital image 602 depicting the artificially stained sample. If stains (in this case comprising May-Grünwald eosin methylene blue and Giemsa-azur eosin methylene blue) are applied to the unstained sample depicted in FIG. 6A and a digital image 604 is taken using conventional microscopy (in this case bright field illumination), the result is shown in FIG. 6C. Comparing FIG. 6B and FIG. 6C, it is evident that the image 602 of the artificially stained sample (i.e., FIG. 6B) is very similar and nearly equivalent to the image 604 of the actual stained sample (i.e., FIG. 6C).

[0082] 3 is a block diagram of a method 30 for training a machine learning model to construct a digital image 602 depicting an artificially stained sample. The method 30 includes receiving a training set of digital images of an unstained sample S300, the training set of digital images being obtained by illuminating the unstained sample from a plurality of directions and capturing a digital image for each of the plurality of directions; receiving a ground truth comprising digital images of a stained sample S302, the stained sample being formed by applying a staining agent to the unstained sample; and training a machine learning model S304, using the received training set of digital images of the unstained sample and the received ground truth to construct a digital image 602 depicting the artificially stained sample. The method 30 may further include an acquiring step S306, where a training set of digital images of the unstained sample is acquired by illuminating the unstained sample from a plurality of directions S308 and capturing a digital image of the unstained sample for each of the plurality of directions S310. The training set of digital images may be acquired by illuminating the unstained sample with white light from a plurality of directions and capturing a digital image for each of the plurality of directions. The training set of digital images may be acquired using a microscope objective and an image sensor, where at least one of the plurality of directions may correspond to an angle greater than the numerical aperture of the microscope objective. The method 30 may further include applying a staining agent to the unstained sample S312 to form a stained sample S314; and acquiring digital images of the stained sample S316 to form a ground truth comprising digital images of the stained sample. Acquiring a digital image of the stained sample S316 may include illuminating the stained sample from a subset of the multiple directions simultaneously S318 and capturing a digital image of the stained sample while the stained sample is illuminated from the subset of the multiple directions simultaneously S319.The method 30 may further include receiving a reconstruction set of digital images of the stained sample S320, where the reconstruction set may be obtained by illuminating the stained sample from a plurality of directions and capturing a digital image for each of the plurality of directions; and reconstructing a digital image of the stained sample using computational imaging techniques S322. The method 30 may further include applying a staining agent to an unstained sample S312 to form a stained sample; and acquiring a reconstruction set of digital images of the stained sample S324, where the reconstruction set of digital images of the stained sample is obtained by illuminating the stained sample from a plurality of directions S326 and capturing a digital image of the stained sample for each of the plurality of directions S328. The reconstruction set of digital images may be acquired using a microscope objective and an image sensor, where at least one of the plurality of directions may correspond to an angle greater than the numerical aperture of the microscope objective. The ground truth may comprise digital images having a relatively higher resolution than the digital images of the training set of digital images.

[0083] 4 is a block scheme of a method 40 for constructing a digital image 602 depicting an artificially stained sample. The method 40 includes receiving an input set of digital images of an unstained sample S400, the input set of digital images being obtained by illuminating the unstained sample from a plurality of directions and capturing a digital image for each of the plurality of directions; constructing a digital image depicting the artificially stained sample S402 by inputting the input set of digital images into a machine learning model trained according to the method 30 shown in FIG. 3 S404 and receiving an output from the machine learning model comprising a digital image 602 depicting the artificially stained sample S406. The input set of digital images of the unstained sample may be acquired using a microscope objective and an image sensor, and at least one of the plurality of directions may correspond to an angle greater than the numerical aperture of the microscope objective. The input set of digital images may be obtained by illuminating an unstained sample with white light from multiple directions and capturing a digital image for each of the multiple directions.

[0084] 7 illustrates a non-transitory computer readable storage medium 50. The non-transitory computer readable storage medium 50 comprises program code portions which, when executed on a device having processing capabilities, perform the method 30 illustrated in FIG. 3 or the method 40 illustrated in FIG.

[0085] Those skilled in the art will be aware of machine learning, and in particular, how a machine learning model may be trained and / or how a trained machine learning model may be used. However, briefly, the machine learning model may be a type of supervised machine learning model, e.g., a network such as U-net or Pix2pix. The machine learning model may be a transformer-based network such as SwinIR. The machine learning model may be a convolutional neural network. The machine learning model is trained to predict a desired output using exemplary input training data and ground truth, i.e., "correct" or "true" output. In other words, the ground truth is used as a label for the input training data. The input training data may comprise data related to different outcomes, and each input training data may thereby be associated with a ground truth related to that particular input training data. Thus, each input training data may be labeled with an associated ground truth (i.e., "correct" or "true" output). The machine learning model may be composed of multiple layers of neurons, and each neuron may represent a mathematical operation applied to the input training data. Typically, a machine learning model is composed of an input layer, one or more hidden layers, and an output layer. The first layer is called the input layer. The output of each layer of the machine learning model (except the output layer) is fed to the subsequent layer, which generates a new output. The new output may be further fed to the subsequent layer. The output of the machine learning model may be the output of the output layer. This process may be repeated for all layers of the machine learning model. In general, each layer further includes an activation function. The activation function may further define the output of the neurons of the layer. For example, the activation function may ensure that the output from the layer is not too large or too small (e.g., tending toward positive or negative infinity). Additionally, the activation function may introduce nonlinearity into the machine learning model.During the training process, weights and / or biases associated with the neurons of a layer may be adjusted until the machine learning model generates predictions for the input training data that reflect the ground truth. Each neuron may be configured to multiply the input to the neuron by a weight associated with that neuron. Each neuron may be further configured to add a bias associated with that neuron to the input. In other words, the output from a neuron may be the sum of the bias associated with the neuron and the product of the weight associated with the neuron and the input. The weights and biases may be adjusted in a recursive or iterative process. This is known in the art as backpropagation. A convolutional neural network may be a type of neural network consisting of one or more layers that represent convolution operations. In this context, the input training data comprises a digital image. The digital image may be represented as a matrix (or as an array), where each element of the matrix (or array) may represent a corresponding pixel of the digital image. Thus, the value of the element may represent a pixel value of the corresponding pixel of the digital image. Thus, the inputs and outputs to the machine learning model may be numerical values ​​(e.g., matrices or arrays) that represent the digital image. In this context, the input is a collection of digital images (i.e., a training set or input set). Thus, the input to the machine learning model may be a plurality of matrices, or a three-dimensional matrix. It should be understood that the machine learning model may receive further inputs during training. An example of such an input includes the type of staining agent used in forming the stained sample. Such inputs may be used in constructing a digital image depicting the artificially stained sample using the trained machine learning model. Furthermore, in this context, the output is a digital image. Thus, the output of the machine learning model may be a matrix representing the constructed digital image depicting the artificially stained sample.

[0086] Those skilled in the art will understand that the inventive concept is in no way limited to the preferred variants described above, but rather many modifications and variations are possible within the scope of the appended claims.

[0087] For example, multiple machine learning models can be trained for different staining agents. Thus, the type of artificial staining of the constructed digital image can be selected by inputting an input set of digital images to a machine learning model trained with that type of staining agent. Furthermore, it should be understood that instead of training multiple machine learning models for different staining agents, a single machine learning model can be trained for multiple different specimen and staining agent types. For example, the type of staining agent can be used as an additional input to the machine learning model during training. Thus, the learned machine learning model can take as input an input set of digital images and the type of staining agent used for artificial staining. The trained machine learning model can be used to output a digital image depicting an artificially stained specimen that reproduces the specimen stained with the type of staining agent used as input to the trained machine learning model.

[0088] Although the construction of a digital image depicting an artificially stained sample is described in relation to the microscope system of FIG. 2, it should be understood that the construction of the digital image may be performed on a computing device. Thus, the computing device may be configured to receive an input set of digital images and input the received input set of digital images into a machine learning model trained according to the above. The input set of digital images may be captured, for example, using the microscope system of FIG. 2 and transmitted to the computing device (i.e., the computing device may be configured to receive the input set of digital images from the microscope system).

[0089] Additionally, variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

Claims

1. 1. A method (30) of training a machine learning model for constructing a digital image (602) depicting an artificially stained specimen, comprising: receiving (S300) a training set of digital images of an unstained sample, the training set of digital images being obtained by illuminating the unstained sample from a plurality of directions and capturing a digital image for each of the plurality of directions; receiving (S302) ground truth comprising a digital image of a stained sample, the stained sample being formed by applying a stain to the unstained sample; training (S304) the machine learning model to construct digital images (602) depicting artificially stained samples using the received training set of digital images of the unstained samples and the received ground truth; A method (30).

2. acquiring the training set of digital images of the unstained samples (S306), a step (S308) of illuminating the unstained sample from multiple directions; capturing a digital image of the unstained sample for each of the plurality of directions (S310); and acquiring the information (S306). The method (30) of claim 1, further comprising:

3. 10. The method of claim 1, wherein the training set of digital images is obtained by illuminating the unstained sample with white light from the multiple directions and capturing a digital image for each of the multiple directions.

4. 2. The method (30) of claim 1, wherein the training set of digital images is acquired using a microscope objective and an image sensor, and at least one direction of the plurality of directions corresponds to an angle greater than the numerical aperture of the microscope objective.

5. applying a staining agent to the unstained sample (S312) to form a stained sample; forming the ground truth (S314) comprising a digital image of the stained sample, acquiring a digital image of the stained sample (S316); and forming a step (S314) The method (30) of claim 1, further comprising:

6. The step of acquiring a digital image of the stained sample (S316) includes: simultaneously illuminating the stained specimen from a subset of the plurality of directions (S318); capturing (S319) a digital image of the stained specimen while the stained specimen is simultaneously illuminated from the subset of the plurality of directions; The method (30) of claim 5, comprising:

7. receiving (S320) a reconstructed set of digital images of the stained specimen; a receiving step (S320) in which the reconstruction set is obtained by illuminating the stained sample from a plurality of directions and capturing a digital image for each of the plurality of directions; reconstructing (S322) a digital image of the stained specimen using computational imaging techniques and the received reconstruction set of digital images; The method (30) of claim 1, further comprising:

8. applying the staining agent to the unstained sample (S312) to form the stained sample; Obtaining the reconstruction set of digital images of the stained specimen (S324), a step (S326) of illuminating the stained sample from multiple directions; capturing a digital image of the stained specimen for each of the plurality of directions (S328); and an acquiring step (S324) The method (30) of claim 7, further comprising:

9. 8. The method (30) of claim 7, wherein the reconstructed set of digital images is acquired using a microscope objective and an image sensor, and at least one direction of the plurality of directions corresponds to an angle greater than the numerical aperture of the microscope objective.

10. The method (30) of claim 1 , wherein the ground truth comprises digital images having a relatively higher resolution than digital images in the training set of digital images.

11. 1. A method (40) for constructing a digital image (602) depicting an artificially stained specimen, said method (40) comprising: receiving (S400) an input set of digital images of an unstained sample, the input set of digital images being obtained by illuminating the unstained sample from a plurality of directions and capturing a digital image for each of the plurality of directions; Constructing a digital image (S402) depicting the artificially stained sample, inputting the input set of digital images into a machine learning model trained according to the method of claim 1 (S404); receiving (S406) an output from the machine learning model comprising a digital image (602) depicting the artificially stained sample; The construction step (S402) A method (40) comprising:

12. 12. The method (40) of claim 11, wherein the input set of digital images of the unstained sample is acquired using a microscope objective and an image sensor, and at least one direction of the plurality of directions corresponds to an angle greater than the numerical aperture of the microscope objective.

13. 12. The method (40) of claim 11, wherein the input set of digital images is obtained by illuminating the unstained sample with white light from the plurality of directions and capturing a digital image for each of the plurality of directions.

14. 1. An apparatus (10) for training a machine learning model for constructing a digital image depicting an artificially stained specimen, the apparatus (10) comprising a circuit (100) comprising: a first receiving function (1100) configured to receive a training set of digital images, the training set of digital images being obtained by illuminating an unstained sample from a plurality of directions and capturing a digital image of the unstained sample for each of the plurality of directions; a second receiving function (1102) configured to receive ground truth comprising a digital image of a stained sample, the stained sample being formed by applying a staining agent to the unstained sample; a training function (1104) configured to use the received training set of digital images of the unstained sample and the received ground truth to train a machine learning model for constructing a digital image depicting the artificially stained sample, according to the method of any one of claims 1 to 10; A circuit (100) configured to perform An apparatus (10) comprising:

15. 15. The apparatus (10) of claim 14, wherein the training set is obtained by illuminating the unstained sample with white light from the plurality of directions and capturing a digital image for each of the plurality of directions.

16. A microscope system (20), comprising: an illumination system (260) configured to illuminate an unstained sample (292) from multiple directions (262); an image sensor (270); at least one microscope objective (280) positioned to image the unstained sample (292) onto the image sensor (270); A circuit (200) comprising: An acquisition function (2100), controlling the illumination system (260) to illuminate the unstained sample (292) from each of the plurality of directions (262); configured to control the image sensor (270) to capture a digital image in each of the plurality of directions (262), thereby obtaining an input set of digital images. An acquisition function (2100) configured as follows: An image construction function (2102), inputting said input set of digital images into a machine learning model trained by the method (30) of any one of claims 1 to 9; receiving an output from the machine learning model comprising a digital image (602) depicting the artificially stained sample; An image construction function (2102) configured as follows: a circuit (200) configured to perform A microscope system (20) comprising:

17. 17. The microscope system (20) of claim 16, wherein the illumination system (260) comprises a plurality of light sources, each light source of the plurality of light sources configured to emit white light.

18. 17. The microscope system (20) of claim 16, wherein the illumination system (260) comprises a plurality of light sources arranged on a curved surface (264), the curved surface (264) being concave along at least one direction along the curved surface (264), and each of the plurality of light sources (261) being configured to illuminate the unstained sample (292) from one of the plurality of directions.

19. 20. The microscope system (20) of claim 18, wherein the curved surface (264) is formed with facets (265).

20. 17. The microscope system of claim 16, wherein the at least one microscope objective lens (280) has a numerical aperture of 0.4 or less.

21. A non-transitory computer-readable storage medium (50) comprising program code portions which, when executed on a device having processing capabilities, perform the method (30) of any one of claims 1 to 10 or the method (40) of any one of claims 11 to 13.

Images