Method, device and system for analyzing a sample

Abstract

An apparatus (10) and a method (30) for training a machine learning model for analyzing a sample are provided. The method (30) includes receiving (S300) a ground truth comprising a classification of at least a portion of the sample; obtaining (S302) a training set of digital images of the sample by illuminating the sample from a plurality of directions and capturing a digital image for each of the plurality of directions; and training (S304) a machine learning model for analyzing the sample using the training set and the received ground truth. Additionally, a microscope system (20) and a method (40) for analyzing a sample are provided.

Description

[Technical field]

[0001] The present invention relates to a method, an apparatus and a system for analysing a sample. [Background technology]

[0002] 2. Background of the Invention In the field of digital microscopy, a common task is to find and identify objects within a sample, for example in the fields of hematology and cytology, where a specific cell type needs to be found and identified in order to establish a diagnosis for the patient from whom the sample was taken.

[0003] It is usually beneficial to quickly screen a sample for a particular type of cell. Such screening requires a compromise between speed and accuracy. To perform high-precision screening, the sample generally needs to be greatly magnified, which allows the cell type to be determined. However, this has the opposite effect: only a small portion of the sample is imaged at one time. To screen the entire sample at high magnification, many individual positions of the sample must be imaged, leading to a time-consuming screening process. Therefore, to reduce the time required for screening, it is conceivable to reduce the number of imaging positions. However, when screening the entire sample, the magnification needs to be reduced. On the other hand, this reduces the accuracy of the screening, which may lead to a screening process where the cell types are not properly found and identified.

[0004] Therefore, there is a need in the art for improvements. Summary of the Invention [Problem to be solved by the invention]

[0005] In view of the above, it is an object of the inventive concept to overcome or at least partially alleviate the above mentioned problems. In particular, it is an object of the inventive concept to provide an improved and more efficient method of analyzing samples with a digital microscope and classifying the samples with high speed, high performance and high accuracy. It is an object of the inventive concept to provide a method for classifying samples with less computational power. [Means for solving the problem]

[0006] According to a first aspect, there is provided a method of training a machine learning model for analyzing a sample, the method comprising: receiving a ground truth comprising a classification of at least a portion of the sample; obtaining a training set of digital images of the sample by illuminating the sample from a plurality of directions and capturing a digital image for each of the plurality of directions; and training a machine learning model for analyzing the sample using the training set of digital images and the received ground truth.

[0007] In the context of the present disclosure, the expression "ground truth" should be interpreted as information that is known to be authentic and / or true. Thus, in this context, ground truth may represent the true classification of the samples as the machine learning model is trained to classify the samples. The true classification may be performed, for example, by a human manually classifying the samples and / or by a device configured to classify the samples using conventional techniques (e.g., object recognition of digital images, etc.).

[0008] In the context of the present disclosure, the expression "a portion of a sample" should be interpreted as a region of the sample. A portion of a sample may be an area corresponding to the size of a single object (e.g., a cell) within the sample, an area corresponding to the size of multiple objects within the sample, the entire sample, or an area of ​​any size in between. Thus, the classification of a portion of a sample can be understood as the classification of objects and / or regions within the sample.

[0009] Thus, the machine learning model can be trained to correlate a training set of digital images to the ground truth. The machine learning model can be iteratively and / or recursively trained until the difference between the classification output of the machine learning model and the ground truth is less than a predetermined threshold. A smaller difference between the classification output and the ground truth can indicate a higher accuracy of classification by the machine learning model. The machine learning model is trained using a plurality of samples. In such a case, the machine learning model can be trained using a training set of digital images of each sample of the plurality of samples and a corresponding ground truth associated with each sample of the plurality of samples. The use of multiple samples in training the machine learning model can improve the classification provided using the trained machine learning model.

[0010] By illuminating a sample from multiple different directions and capturing digital images for each of the multiple directions, information about the sample may be captured that relates to finer details than would normally be resolvable by a conventional microscope (i.e., using conventional microscope illumination) used to image the sample. This may be understood as different portions of the Fourier space (i.e., spatial frequency domain) associated with the sample being imaged for different illumination directions. This technique is known in the art as Fourier Ptychography. Furthermore, by illuminating a sample from multiple different directions and capturing digital images for each of the multiple directions, information about the refractive index associated with the sample may be captured. This may be understood as the effect that the refraction of light is dependent on the angle of incidence of the light illuminating the sample and the refractive index of the sample. Information about the refractive index of the sample may in turn enable phase information associated with the sample (commonly referred to in the art as quantitative phase) to be determined. Since the multiple digital images contain information related to one or more of fine details of the sample, refractive index associated with the sample, and phase information associated with the sample, this information may be used in training a machine learning model, which in turn may enable the machine learning model to be trained to classify the sample more accurately than would be permitted if multiple digital images were captured from only one direction. In other words, by illuminating the sample from multiple different directions, it may be possible to capture information related to details of the sample that are finer than would normally be permitted by the microscope objective lens used to image the sample. Thus, information related to fine details can be captured even using a microscope objective lens with a relatively low magnification. By using a microscope objective lens with a relatively low magnification, a large portion of the sample can be imaged at each imaging position. Thus, by imaging at relatively few positions, the entire sample can be scanned, which in turn allows the sample to be scanned faster.

[0011] Thus, the inventive concept allows for rapid classification of samples directly from the digital images without requiring highly detailed images of the sample (i.e., images having relatively higher magnification than the digital images.) A further advantage is that the samples can be classified by trained machine learning models instead of humans, which may allow for faster and more reliable classification.

[0012] The sample may be an unstained sample. This may be avoided because information related to the refractive index of the sample and / or phase information related to the sample may be captured by illuminating the sample from multiple directions. This information (or variation within the sample) may be used when training a machine learning model, which may then be used to classify unstained samples.

[0013] A related advantage is that the trained machine learning model can be used to classify unstained samples, thereby avoiding the associated staining steps, thereby eliminating the use of toxic or hazardous chemicals, and further reducing the financial costs associated with sample classification and / or the time required to classify the samples.

[0014] The training set can be obtained by illuminating a sample with white light from multiple directions and capturing digital images for each of the multiple directions.

[0015] In the context of the present disclosure, the expression "white light" should be interpreted as light having a relatively broad wavelength spectrum. White light may, for example, be similar to sunlight. The spectrum of white light may include most of the visible light spectrum, as opposed to light having a narrow spectrum (e.g., monochromatic light).

[0016] A related advantage is that white light (i.e., broad spectrum light) may allow more information about the sample to be captured in a training set of digital images that may be missing in parts of the spectrum not covered when using a narrowband light source. That information is used by the trained machine learning model in analyzing the sample. Imaging applications that utilize Fourier ptychography generally require monochromatic light, and thus, in prior art systems, multiple different colored monochromatic lights may be required to capture multispectral information about the sample. Thus, the use of white light may reduce the total number of digital images required to capture multispectral information about the sample compared to prior art systems that utilize Fourier ptychography. Additionally, the use of different colored monochromatic lights may only allow information about the sample in those specific wavelength ranges, and because white light may have a broader spectrum, the use of white light may allow information about the sample to be captured in a broader wavelength range compared to using monochromatic light.

[0017] The machine learning model may be a convolutional neural network.

[0018] A related advantage is that the convolutional neural network can detect significant features associated with a sample in the multiple digital images without human supervision because the multiple digital images may contain information, such as height and / or position, associated with objects present in the sample.

[0019] As a further related advantage, convolutional neural networks can provide increased accuracy in image recognition compared to other neural networks, which can lead to improved classification of samples.

[0020] The ground truth may further comprise a location of at least a portion within the sample, and the step of training the machine learning model may further include training the machine learning model to determine a location of at least a portion within the sample by training using the training set of digital images and the received ground truth until a difference between a location output of the machine learning model is less than a further predetermined threshold.

[0021] At least a portion within the sample may include an object of interest (e.g., a cell), and thus the location of the at least a portion may be the location of the object of interest. The location of the at least a portion may be a relative location of the sample.

[0022] A related advantage is that when a trained machine learning model is used to analyze a sample, the location of portions of the sample that are of interest (e.g., portions that are assigned to a particular class) can be determined.

[0023] The ground truth may further comprise a dimension of at least a portion within the sample, and the step of training the machine learning model may further include training the machine learning model to determine a dimension of at least a portion of the sample using the training set of digital images and the received ground truth, by training the machine learning model until a difference between a dimension output of the machine learning model is less than a further predetermined threshold.

[0024] A related advantage is that when a trained machine learning model is used to analyze a sample, dimensions (e.g., shape and / or size) of portions of the sample that are of interest (e.g., portions that are assigned to a particular class) may be determined.

[0025] The ground truth may comprise a classification for each of a number of portions of the specimen.

[0026] A related advantage is that machine learning models can be trained to classify different parts of a sample into different classes, further improving the analysis of the sample using the trained machine learning model.

[0027] The ground truth may further comprise respective positions of a plurality of features within the sample.

[0028] A related advantage is that the machine learning model can be trained to output the locations of portions of a sample that are of interest (e.g., portions that are classified into one or more particular classes), thereby further improving analysis of the sample using the trained machine learning model.

[0029] The training set of digital images can be acquired using a microscope objective and an image sensor, where at least one of the multiple directions can correspond to an angle greater than the numerical aperture of the microscope objective.

[0030] The numerical aperture of a microscope objective may be a dimensionless number related to the range of angles that the microscope objective accepts light into. Thus, directions greater than the numerical aperture can be understood as directions that correspond to angles greater than the range of angles that the microscope objective is configured for brightfield microscopy.

[0031] By illuminating a sample from an angle larger than the numerical aperture of the microscope objective, images captured at that illumination angle may contain information about higher spatial frequencies and therefore finer details of the sample than the microscope objective would normally allow. This allows the microscope objective to capture phase information related to the sample and information related to details that the microscope objective would not normally be able to resolve, which can be used to train machine learning models. In other words, illuminating a sample from an angle larger than the numerical aperture of the microscope objective can improve the training of machine learning models.

[0032] In a second aspect, there is provided a method of analysing a sample, the method comprising: receiving an input set of digital images of the sample, the input set of digital images being obtained by illuminating the sample from a plurality of directions and capturing a digital image for each of the plurality of directions; analysing the sample by inputting the input set of digital images into a machine learning model trained according to the method of the first aspect, and receiving output from the machine learning model including a classification of at least a portion of the sample.

[0033] By inputting an input set of received digital images of the sample into a machine learning model trained according to the method of the first aspect, the process of classifying a portion of the sample may be made more efficient, as a classification of the sample is output directly from the trained machine learning model. Compared to classification techniques performed on images taken at high magnification, a machine learning model trained according to the method of the first aspect may provide a highly accurate classification of the portion of the sample with greater efficiency (e.g., due to the need for less computational resources and / or memory) while allowing the use of digital images taken at relatively low magnification.

[0034] The sample may be an unstained sample. When the sample is an unstained sample, the input set of digital images is comprised of digital images of the unstained sample. In such a case, the machine learning model may be trained using a training set of digital images comprised of digital images of the unstained sample.

[0035] The input set of digital images may be obtained by illuminating the sample with white light from multiple directions and capturing a digital image for each of the multiple directions.

[0036] The output may further include a location of at least the portion within the sample.

[0037] The output may further include a dimension of at least a portion within the sample.

[0038] An input set of digital images of the sample can be acquired using a microscope objective and an image sensor, where at least one of the multiple directions can correspond to an angle greater than the numerical aperture of the microscope objective.

[0039] The above-mentioned features of the first aspect also apply to this second aspect, where applicable: to avoid undue repetition, please see above.

[0040] According to a third aspect, there is provided an apparatus for training a machine learning model, the apparatus comprising a circuit configured to perform: a first receiving function configured to receive a training set of digital images, the training set of digital images being acquired by illuminating a sample from a plurality of directions and capturing digital images of at least a portion of the sample for each of the plurality of directions; a second receiving function configured to receive ground truth; and a training function configured to train the machine learning model according to the method of the first aspect using the received ground truth and the acquired training set of digital images.

[0041] The training set can be obtained by illuminating a sample with white light from multiple directions and capturing digital images for each of the multiple directions.

[0042] The circuitry of the apparatus of the third aspect may be further configured to perform: a third receiving function configured to receive a high resolution digital image of the sample; and a determining function configured to classify at least a portion of the sample by classifying a corresponding portion of the high resolution digital image of the sample and form a ground truth comprising a classification of the at least a portion of the sample, and the second receiving function may be configured to receive the formed ground truth from the determining function.

[0043] In this context, a "high resolution digital image of a sample" should be interpreted as a digital image of the sample having a relatively higher resolution than each digital image of the plurality of digital images. Furthermore, a high resolution digital image of a sample may be a digital image of the sample captured with a microscope objective having a relatively higher magnification than the microscope objective used to capture each digital image of the training set of digital images. Thus, details visible in the high resolution image may not be visible in the digital image captured with the microscope objective used to capture the training digital images.

[0044] A related advantage is that the apparatus may determine the ground truth by classifying the portions of the sample using a digital image of the sample having a relatively higher resolution than each digital image of the plurality of digital images, such that an accurate ground truth (i.e., classification of the portions of the sample) may be acceptable.

[0045] A further related advantage is that the device itself can determine the ground truth, thereby eliminating or at least reducing the need for human input (e.g., classification of sample portions performed by a human).

[0046] The training function may be configured to train the machine learning model using a first subset of the training set of digital images. The circuitry may be further configured to perform a reconstruction function configured to reconstruct a high resolution digital image of the sample from a second subset of the training set of digital images, the high resolution image having a higher resolution than a resolution of the digital images of the training set. The third receiving function may be configured to receive the high resolution digital image of the sample from the reconstruction function.

[0047] A high-resolution digital image of the sample can be reconstructed from the second subset using Fourier ptychography. A high-resolution digital image of the sample can be reconstructed by inputting the second subset to a further machine learning model trained to reconstruct a high-resolution image of the sample from a plurality of digital images acquired by illuminating the sample from a plurality of directions and capturing a digital image for each of the plurality of directions.

[0048] A related advantage is that the same microscope system can be used to capture the digital images used as input for training (i.e., the first subset of the training set of digital images) and the digital images used to determine the ground truth (i.e., the high-resolution digital images reconstructed from the second subset of the training set of digital images), thereby enabling improved training of the machine learning model. In other words, because the same system is used to capture the digital images of the sample, the sample can remain stationary, such that each digital image in the training set of digital images can be a digital image of the same portion of the sample. This can improve the training of the machine learning model.

[0049] A further related advantage is that the time required to train a machine learning model is reduced, since all the digital images required for training can be acquired during common imaging processes.

[0050] The determining functionality may be further configured to determine a position of at least a portion within the sample by determining a position of a corresponding portion of a high resolution digital image of the sample, and the ground truth may further include the determined position of the at least a portion of the sample.

[0051] The above mentioned features of the first and / or second aspect also apply to this third aspect, where applicable: in order to avoid excessive repetition, please see above.

[0052] According to a fourth aspect, there is provided a microscope system comprising: an illumination system configured to illuminate a sample from a plurality of directions; an image sensor; at least one microscope objective lens arranged to image the sample onto the image sensor; and a circuit configured to perform an illumination function configured to control the illumination system to sequentially illuminate the sample from a plurality of directions, a capture function configured to control the image sensor to obtain an input set of digital images, the input set of digital images being obtained by illuminating the sample from a plurality of directions and capturing a digital image for each of the plurality of directions, and an analysis function configured to analyze the sample by inputting the input set of digital images to a machine learning model trained according to the method of the first aspect and receiving from the machine learning model an output including a classification of at least a portion of the sample into at least one class.

[0053] The sample may be an unstained sample.

[0054] The illumination system may comprise a plurality of light sources, each of which may be configured to emit white light.

[0055] A related advantage is the ability to capture information about a sample that pertains to a broader range of the electromagnetic spectrum, as compared to illumination systems with narrowband light sources (eg, monochromatic light sources).

[0056] A further related advantage is that fewer light sources can be used to capture multispectral information about a sample, as compared to illumination systems with narrowband (e.g., monochromatic) light sources. This is because light sources configured to emit white light can emit light over a broader range of wavelengths, as compared to narrowband (e.g., monochromatic) light sources typically used in Fourier ptychography applications. Thus, fewer light sources configured to emit white light may be required to cover a particular range of the electromagnetic spectrum, as compared to narrowband light sources.

[0057] The illumination system may include a plurality of light sources arranged on a curved surface that is concave along at least one direction along the surface, and each light source of the plurality of light sources may be configured to illuminate the sample from one of a plurality of directions.

[0058] Arranging multiple light sources on a curved surface may be advantageous in that the distance from each light source to the current imaging position of the microscope system (i.e., the position or portion of the unstained sample currently being imaged) is similar. Because this distance is similar, the intensity of light emitted from each light source may be similar at the current imaging position. This may be understood as the effect of the inverse square law. Thus, the unstained sample may be illuminated by light having similar intensity for each direction in multiple directions, which may allow for more homogeneous illumination of the unstained sample independent of the illumination direction. It may be advantageous to configure the illumination system such that the distance from each light source to the current imaging position is large enough that each light source can be treated as a point source. This may make the light quasi-coherent at the current imaging position. Thus, the distance from each light source to the current imaging position may be selected such that the intensity of light from each light source at the current imaging position is high enough to generate an input set of digital images.

[0059] The curved surface may be formed with facets, or stated differently, the curved surface may be constructed from a number of flat surfaces.

[0060] A related advantage is that the lighting system is easier to manufacture, reducing the associated economic costs.

[0061] A further related advantage is that the lighting system may be modular, which may facilitate replacing one or more light sources (e.g., if a light source is broken and / or defective). For example, one or more light sources present on a facet may be removed and replaced (e.g., by removing the facet) without replacing light sources present on other facets.

[0062] The above-mentioned features of the first, second and / or third aspects also apply to this fourth aspect, where applicable: in order to avoid excessive repetition, please see above.

[0063] According to a fifth aspect, there is provided a non-transitory computer readable storage medium comprising program code portions which, when executed on a device having processing capability, perform a method according to the first aspect.

[0064] The above-mentioned features of the first, second, third and / or fourth aspects also apply to this fifth aspect, where applicable: in order to avoid undue repetition, please see above.

[0065] The above and other aspects of the inventive concept will now be described in more detail with reference to the accompanying drawings. The figures should not be considered as limiting the inventive concept to a particular variant, but are used to explain and understand the inventive concept. As shown in the figures, the relative distances between different elements, sizes of layers and regions are exaggerated for illustrative purposes, and are therefore provided to explain the general structure of the variant of the inventive concept. Like reference numerals refer to like elements throughout. [Brief description of the drawings]

[0066] [Figure 1] FIG. 1 illustrates an apparatus for training a machine learning model. [Diagram 2] FIG. 2 is a diagram showing the microscope stem. [Diagram 3] FIG. 3 is a box scheme diagram of a method for training a machine learning model to analyze a sample. [Figure 4] FIG. 4 is a box scheme diagram of a method for analyzing a sample using a trained machine learning model. [Diagram 5] FIG. 5 shows an illumination system having a faceted curved surface. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0067] Detailed Description The inventive concepts will now be more fully described with reference to the accompanying drawings, in which presently preferred versions of the inventive concepts are shown, however, the inventive concepts may be embodied in many different forms and should not be construed as limited to the versions set forth herein, but rather these versions are provided for thoroughness and completeness and to fully convey the scope of the inventive concepts to those skilled in the art.

[0068] First, a method 30 and an apparatus 10 for training a machine learning model for analyzing a sample will be described with reference to Figs. 1 and 3. Fig. 1 shows an apparatus 10 for training a machine learning model for analyzing a sample. The apparatus 10 may be a computing device. Examples of suitable computing devices include computers, servers, smartphones, tablets, etc. The apparatus 10 may further be implemented as part of a cloud server and / or a distributed computing facility. It should be further understood that the apparatus 10 may comprise further components, such as an input device (mouse, keyboard, touch screen, etc.) and / or a display. The apparatus 10 may further comprise a power source, such as a power connection, a battery, etc. The apparatus 10 comprises a circuit 100. As shown in the example of Fig. 1, the circuit 100 may comprise one or more of a memory 110, a processing unit 120, a transceiver 130, and a data bus 140. The memory 110, the processing unit 120, and the transceiver 130 may communicate via the data bus 140. The processing unit 120 may be a central processing unit (CPU). The transceiver 130 may be configured to communicate with an external device. For example, the transceiver 130 may be configured to communicate with a server, a computer external peripheral device (e.g., an external storage device), etc. The external device may be a local device or a remote device (e.g., a cloud server). The transceiver 130 may be configured to communicate with the external device over an external network (e.g., a local area network, the Internet, etc.). The transceiver 130 may be configured for wireless communication and / or wired communication. Suitable technologies for wireless communication are known to those skilled in the art. Some non-limiting examples include Wi-Fi and Near Field Communication (NFC). Suitable technologies for wired communication are known to those skilled in the art. Some non-limiting examples include USB, Ethernet, Firewire.

[0069] The memory 110 may be a non-transitory computer-readable storage medium. As shown in the example of FIG. 1, the memory 110 may store program code portions corresponding to one or more functions 1100, 1102, 1104, 1106, 1108, 1110. The program code portions may be executable by a processing unit to thereby perform the function. Thus, when it is mentioned that the circuit 100 is configured to perform a particular function, the processing unit 120 may execute the program code portions corresponding to the particular function that may be stored on the memory 110. However, it should be understood that one or more functions of the circuit 100 may be implemented in hardware and / or in a particular integrated circuit. For example, one or more functions may be implemented using a field programmable gate array (FPGA). In other words, one or more functions of the circuit 100 may be implemented in hardware or software, or as a combination of the two.

[0070] The circuit 100 is configured to perform a first receiving function 1100, a second receiving function 1102, and a training function 1104. The circuit 100 may be further configured to perform one or more of a third receiving function 1106, a determining function 1108, and a reconstructing function 1110.

[0071] The first receiving function 1100 is configured to receive a training set of digital images. The training set of digital images is acquired (S302) by illuminating a sample from a plurality of directions and capturing a digital image of at least a portion of the sample for each of the plurality of directions. The training set of digital images may be acquired by illuminating a sample with white light from a plurality of directions and capturing a digital image for each of the plurality of directions. Stated differently, the training set of digital images of a sample may be acquired by illuminating a sample with white light from a plurality of directions and capturing a digital image of the sample for each of the plurality of directions. The sample may be an unstained sample. Thus, the training set of digital images may include a digital image of an unstained sample. The first receiving function 1100 may be configured to receive the training set of digital images via the transceiver 130. For example, the training set of digital images is captured using an external microscope system and then transmitted to the transceiver 130 of the device 10. As a further example, the device 10 may form part of a microscope system and the training set of digital images may be received from an image sensor of the microscope system. The memory 110 may be configured to store a training set of digital images, and the first receiving function 110 may be configured to receive the training set of digital images from the memory 110. The training set of digital images may be acquired using a microscope objective and an image sensor. The sample may be illuminated sequentially from at least one of a plurality of directions. Each digital image of the training set of digital images may be captured when the sample is illuminated from at least one of a plurality of directions. Thus, the sample may be illuminated from a single one of a plurality of directions, or may be illuminated from multiple one of a plurality of directions simultaneously.

[0072] At least one of the multiple directions may correspond to an angle greater than the numerical aperture of the microscope objective. Because the sample is imaged from multiple different directions and digital images are captured for each of the multiple directions, information about the sample may be captured that relates to finer details than can normally be resolved by the microscope objective used to image the sample. This may be understood as different portions of Fourier space (i.e., spatial frequency domain) associated with the sample being imaged for different illumination directions. This technique is known in the art as Fourier ptychography. Generally, in Fourier ptychography, high spatial frequencies of Fourier space associated with a sample are sampled when the sample is illuminated from a direction corresponding to a large angle of incidence. Thus, even higher spatial frequencies of Fourier space may be sampled when the sample is illuminated from a direction corresponding to an angle greater than the numerical aperture of the microscope objective. This is possible because light is scattered by the sample and some of the light scattered by the sample may be collected by the microscope objective. Since the multiple digital images contain information related to fine details of the sample, this information may be used in training a machine learning model, which in turn may enable a machine learning model to be trained to analyze the sample more accurately than would be allowed if multiple digital images were captured from only one direction or when using a conventional microscope. In other words, illuminating the sample from one of multiple different directions at a time may capture information about finer details of the sample than would normally be allowed by a microscope objective. It should be appreciated that more detailed information of the sample and phase information related to the sample may be obtained by illuminating the sample from more than one of the multiple different directions at the same time, for example, from a subset of the multiple directions. The subset of the multiple directions may include directions corresponding to different portions of the Fourier space of the sample. The different portions of the Fourier space of the sample may be partially overlapping or non-overlapping. Thus, information related to fine details of the sample may be captured while using a microscope objective having a relatively low magnification.For example, a microscope objective lens having a numerical aperture of 0.4 can illuminate a sample from multiple directions and capture information related to fine details as well as a microscope objective lens having a numerical aperture of 1.25 used in conventional microscopy. In other words, a microscope objective lens having a magnification of 20x can illuminate a sample from multiple directions and capture information related to fine details as well as a microscope objective lens having a magnification of 100x used in conventional microscopy. It should be understood that the magnifications and numerical apertures mentioned above are exemplary and the invention can be practiced with other magnifications and / or numerical apertures. A suitable microscope system including a microscope objective lens and an image sensor is described in relation to FIG. 2.

[0073] The second receiving function 1102 is configured to receive (S300) ground truth, where ground truth is information that is known to be authentic and / or true. In this context, the ground truth may represent a true classification of at least a portion of the sample, since a machine learning model is trained to classify the sample. The true classification may be performed, for example, by a human manually classifying the sample and / or by an apparatus configured to classify the sample using conventional techniques (e.g., object recognition of digital images, etc.). Thus, the ground truth (i.e., the true classification of the sample) may be received by manual input. The manual input may be performed via an input device (not shown in FIG. 1 ) connected to the device 10, such as a keyboard, mouse, etc. The ground truth may be received via the transceiver 130. For example, the classification may be performed and / or stored on another device and transmitted to the device 10 via the transceiver 130. If the sample is an unstained sample, the ground truth may include a classification of the unstained sample. Alternatively, the ground truth may include a classification of the stained sample. Stated another way, the training set of digital images may include a digital image of an unstained sample. After the training set of digital images of the unstained sample is acquired, the unstained sample may be stained. The ground truth may thereby include a classification of the stained sample. Thus, the machine learning model may be trained using the training set of digital images of the unstained sample and the ground truth is a classification of the stained sample. The ground truth may include a location of at least a portion of the sample. The ground truth may include a dimension of at least a portion of the sample. The dimension of at least a portion of the sample may be one or more of a height, a width, a shape of at least a portion of the sample. The ground truth may include a classification of each of a plurality of portions of the sample. Thus, the machine learning model may be trained to classify different portions of a single sample into different classes. The ground truth may further include a location of each of a plurality of portions within the sample.The ground truth may further include dimensions of each of a plurality of features within the sample. The machine learning model may be additionally trained to output locations and / or dimensions of features of interest of the sample (e.g., features that are classified into one or more particular classes).

[0074] The third receiving function 1106 may be configured to receive a high-resolution digital image of the sample. The high-resolution digital image of the sample may be captured using a microscope objective lens having a relatively greater magnification than the microscope objective lens used to capture the training set of digital images. In other words, since the microscope objective lens used to capture the high-resolution image may have a relatively greater magnification, the high-resolution image may be a digital image of a smaller area of ​​the sample compared to an image captured using the microscope objective lens used to capture the training set of digital images. Thus, the high-resolution digital image of the sample may be formed from multiple high-resolution digital images of the sample. For example, by combining (e.g., stitching) the multiple high-resolution digital images. The combined image may thereby be an image of an area of ​​the sample comparable to the digital images of the training set of digital images. The high-resolution digital image may be a digital image of the sample having a relatively higher resolution than each digital image of the training set of digital images. Furthermore, the high-resolution digital image of the sample (or each of the multiple high-resolution images) may be a digital image of the sample taken using a microscope objective lens having a relatively higher magnification than the microscope objective lens used in taking each of the digital images of the training set of digital images. Thus, details visible in the high resolution image may not be visible in a digital image taken with the microscope objective used to take the training set of digital images. If the sample is an unstained sample, the high resolution digital image of the sample may be a high resolution digital image of the unstained sample. Alternatively, the high resolution digital image of the sample may be a high resolution digital image of a stained sample. Thus, a training set of digital images of an unstained sample may be acquired, and the unstained sample may be stained after the training set is acquired. After the sample is stained, a high resolution digital image of the stained sample may be acquired. The reconstruction functionality 1110 may be configured to reconstruct a high resolution digital image of the sample from a subset of the training set of digital images.In this case, the reconstructed high-resolution digital image may be a digital image of an unstained sample (if the sample is unstained). The reconstruction function 1110 may be configured to reconstruct a high-resolution image of the sample from a set of digital images of stained samples. In this case, the reconstructed high-resolution digital image may be a digital image of a stained sample. The high-resolution image may have a higher resolution than the resolution of the digital images of the training set. The high-resolution digital image of the sample may have been reconstructed from the second subset using Fourier ptychography. The high-resolution digital image of the sample may be reconstructed by inputting the second subset into a further machine learning model that is trained to reconstruct a high-resolution image of the sample from a plurality of digital images obtained by illuminating the sample from a plurality of directions and capturing a digital image for each of the plurality of directions. The third receiving function may be configured to receive the high-resolution digital image of the sample from the reconstruction function.

[0075] The determiner 1108 may be configured to classify at least a portion of the sample by classifying a corresponding portion of the high resolution digital image of the sample. The determiner 1108 may be configured to classify multiple portions of the sample.

[0076] The decision function 1108 may be configured to classify the corresponding portion of the high resolution digital image using an automatic classification algorithm. The automatic classification algorithm may use image recognition to identify one or more objects in the sample. The decision function 1108 may be configured to compare the identified one or more objects to a database of entries that associate a classification of the sample with an object in the sample. For example, the decision function 1108 may be configured to identify cancer cells in a blood sample, where the database associates the cancer cells with an appropriate classification (e.g., "cancer", "infection", etc.). As an example, classifying the sample may include determining whether the objects in the sample are associated with a class, thereby allowing the number of objects in the sample associated with the class to be counted. For example, when screening for leukemia, the objective may be to count the number of white blood cells in the sample. As another example, classifying the sample may include determining whether at least one object belonging to a class is present in the sample. For example, when screening for malaria, the sample may be classified as infected or not infected depending on whether at least one red blood cell infected with the malaria parasite is present in the sample. Similarly, when screening for cervical precancerous lesions that may progress to cervical cancer, the sample may be classified as healthy or unhealthy depending on whether the sample has at least one abnormal cell. The decision function 1108 may be configured to receive manual input from a human. The manual input may be, for example, a classification of the sample and / or the detected object. In such a case, the device 10 may be configured to display the results of the object detection and may prompt the human to classify the object using an input device (e.g., a keyboard and mouse, and / or a touch screen) in communication with the device 10.

[0077] The determining function 1108 may be further configured to determine a position of at least one portion of the sample by determining a position of a corresponding portion in a high-resolution digital image of the sample. The determining function 1108 may be further configured to determine a position of each of the multiple portions in the sample if configured to classify the multiple portions of the sample. The position may be determined, for example, by object recognition. For example, the sample may be disposed on a surface comprising markings. The markings on the surface may provide information related to a relative position on the surface. Thus, the markings may be identified using object recognition and the relative position on the surface may be decoded from the markings. Such markings may be further used to determine a relative position within the high-resolution digital image if the high-resolution digital image is a combination of multiple high-resolution digital images. The determining function 1108 may be configured to receive a current relative position of the sample from a microscope system used to capture the high-resolution digital image. The relative position of the sample imaged in the high-resolution digital image may be stored as metadata of the high-resolution digital image. If the high-resolution digital image is a combination of multiple high-resolution digital images, such information may be further used to determine a relative position within the high-resolution digital image.

[0078] The determining function 1108 may be further configured to form a ground truth comprising a classification of the at least one portion of the sample. The ground truth may further include a determined location of the at least one portion within the sample. The second receiving function 1102 may be configured to receive the formed ground truth from the determining function 1108.

[0079] The training function 1104 is configured to train (S304) a machine learning model to analyze the portion of the sample using the received ground truth and the training set of acquired digital images. The machine learning model may be a convolutional neural network. The machine learning model may be a convolutional neural network suitable for object detection and / or classification. The training function 1104 is configured to receive a ground truth comprising a classification of at least a portion of the sample. The training function 1104 may receive the ground truth from the second receiving function 1102. The training function 1104 is further configured to obtain a training set of digital images of the sample. As described above, the training set of digital images is obtained by illuminating the sample from multiple directions (possibly with white light) and taking a digital image for each of the multiple directions. The training function 1104 is further configured to train the machine learning model to analyze the sample using the training set of digital images and the received ground truth.

[0080] The training function 1104 may be configured to iteratively and / or recursively train the machine learning model until the difference between the classification output of the machine learning model and the ground truth is less than a predetermined threshold. Thus, the training function 1104 may train the machine learning model to correlate a training set of digital images with the ground truth. A smaller difference between the classification output from the machine learning model and the ground truth may indicate a higher accuracy of the classification provided by the trained machine learning model. The training function 1104 may be configured to train the machine learning model until a loss function is less than a predetermined threshold. The loss function may be determined from the difference between the classification output of the machine learning model and the ground truth. The machine learning model may be trained until the loss function is minimized. Those skilled in the art will appreciate that minimizing a function, e.g., a loss function, may be associated with a tolerance value. For example, a loss function may be considered minimized even if the minimized loss function has a value that is not a local and / or global minimum.

[0081] As mentioned above, the ground truth may further include a location of at least a portion within the sample. In such a case, the training functionality 1104 may be further configured to train (S306) the machine learning model using the training set of digital images and the received ground truth until a difference between the location output of the machine learning model is less than an additional predefined threshold. This allows the machine learning model to be trained to determine a location of at least a portion within the sample. This allows the trained machine learning model to locate an object within the sample.

[0082] As mentioned above, the ground truth may further include dimensions of at least a portion within the sample. In such a case, the training functionality 1104 may be configured to train (S307) the machine learning model using the training set of digital images and the received ground truth until a difference between the dimensional output of the machine learning model is less than a predetermined dimensional threshold. This may train the machine learning model to determine dimensions of at least a portion within the sample. This may enable the trained machine learning model to determine the size and / or shape of an object within the sample.

[0083] If a subset of the training set of digital images is used to reconstruct the high-resolution image, the training functionality 1104 may be configured to train the machine learning model using a further subset of the training set of digital images. In other words, the training functionality 1104 may be configured to train the machine learning model using a first subset of the training set of digital images, and the reconstruction functionality 1110 may be configured to reconstruct the high-resolution digital image using a second subset of the training set of digital images. The first and second subsets may be partially overlapping subsets of the training set of digital images. The first and second subsets may be non-overlapping subsets of the training set of digital images.

[0084] A method 40 for analysing a sample 292 and a microscope system 20 suitable for analysing a sample 292 will now be described with reference to Figures 2 and 4. Figure 2 illustrates a microscope system 20. The microscope system 20 of Figure 2 is suitable for acquiring a set of digital images of a sample to be analysed by a machine learning model trained in accordance with the above. Furthermore, the microscope system of Figure 2 may be suitable for acquiring a training set of digital images used to train a machine learning model in accordance with the above.

[0085] As shown in FIG. 2, the microscope system 20 includes an illumination system 260, an image sensor 270, at least one microscope objective lens 280, and a circuit 200. The system may further include a sample holder 290, as shown in the example of FIG. 2. The circuit 200 may include one or more of a memory 210, a processing unit 220, a transceiver 230, and a data bus 240. The processing unit 220 may include a central processing unit (CPU) and / or a graphics processing unit (GPU). The processing unit 220 may be comprised of a CPU and a GPU. The transceiver 230 may be configured to communicate with an external device. For example, the transceiver 230 may be configured to communicate with a server, a computer external peripheral (e.g., an external storage device), and the like. The external device may be a local device or a remote device (e.g., a cloud server). The transceiver 230 may be configured to communicate with the external device via an external network (e.g., a local area network, the Internet, etc.). The transceiver 230 may be configured for wireless and / or wired communication. Suitable technologies for wireless communication are known to those skilled in the art. Some non-limiting examples include Wi-Fi and Near Field Communication (NFC). Suitable technologies for wired communication are known to those skilled in the art. Non-limiting examples include USB, Ethernet, and Firewire. The memory 210, the processing unit 220, and the transceiver 230 may communicate via a data bus 240. The lighting system 260 and / or the image sensor 270 may be configured to communicate with the circuit 200 via the transceiver 230 as shown in FIG. 2. Additionally or alternatively, the lighting system 260 and / or the image sensor 270 may be configured to communicate directly (e.g., via a wired connection) with the data bus 240. The memory 210 may be a non-transitory computer-readable storage medium. 2, memory 210 may store program code portions corresponding to one or more functions 2100, 2102, 2104. The program code portions may be executable by a processing unit 220 to thereby perform the functions.Thus, when circuit 200 is referred to as being configured to perform a particular function, processing unit 220 may execute portions of program code corresponding to the particular function that may be stored on memory 210. However, it should be understood that one or more functions of circuit 200 may be implemented in hardware and / or in a particular integrated circuit. For example, one or more functions may be implemented using a field programmable gate array (FPGA). Stated differently, one or more functions of circuit 200 may be implemented in hardware or software, or as a combination of the two.

[0086] While the image sensor 270 is shown alone in FIG. 2, it should be understood that the image sensor 270 may be built into a camera. In the example of FIG. 2, the sample holder 290 is a microscope slide onto which a sample 292 has been applied. In the example of FIG. 2, the sample is an unstained sample. However, as previously described, the sample may be a stained sample. It should be understood that the sample 292 may be covered with a cover slip (not shown in FIG. 2). The sample holder 290 may be configured to hold a sample 292 to be analyzed. The sample holder 290 may be movable (e.g., by being coupled to a manual and / or motorized stage) such that the sample 292 may be moved such that different portions of the sample 292 may be imaged by the at least one microscope objective 280.

[0087] The illumination system 260 is configured to illuminate the sample 292 from multiple directions 262. As shown in the example of FIG. 2, the illumination system 260 may be composed of multiple light sources 261. The light sources may be light-emitting diodes (LEDs). The light sources may be lasers. The light sources may emit incoherent light, quasi-coherent light, or coherent light. Each light source of the illumination system 260 may be arranged to illuminate the sample 292 from one of the multiple directions 262. The illumination system 260 may be configured to simultaneously illuminate the sample 292 with one or more of the multiple light sources 261. In other words, the illumination system 260 may be configured to simultaneously illuminate the sample 292 from one or more of the multiple directions 262. Each light source of the multiple light sources 261 may be configured to emit white light. The light sources may be LEDs configured to emit white light. The LED configured to emit white light may be composed of an LED configured to emit blue light and a phosphor layer configured to convert the emitted blue light into white light. For example, the LED configured to emit white light may be formed by a blue LED covered with a phosphor layer that emits white light when illuminated with blue light. Alternatively or additionally, the light source may be a laser configured to emit white light. For example, light emitted from a laser may be converted into light with a broader spectral bandwidth. One example of such a conversion process is known in the art as supercontinuum generation. As shown in the example of FIG. 2, the multiple light sources 261 of the illumination system 260 may be arranged on a curved surface 264. As shown in the example of FIG. 2, the curved surface 264 may be concave along at least one direction along the surface 264. For example, the curved surface 264 may be a cylindrical surface. The curved surface 264 may be concave along two perpendicular directions along the surface. For example, the curved surface 264 may have a shape similar to a segment of a sphere. The sphere segment may be a spherical cap or a spherical dome.Arranging multiple light sources on a curved surface is advantageous in that the distance R from each light source to the current imaging position P of the microscope system 20 is similar. Because the distances are similar, the intensity of light emitted from each light source may be similar at the current imaging position P. This can be understood as an inverse square law effect. Thus, the sample 292 may be illuminated by light having a similar intensity for each of the multiple directions 262, which may in turn allow for a more homogenous illumination of the sample 292 independent of the illumination direction. The distance R from each light source to the current imaging position P may range from 4 cm to 15 cm. It may be advantageous to configure the illumination system such that the distance R from each light source to the current imaging position P is large enough to allow each light source to be treated as a point source. This may allow the light to be quasi-coherent at the current imaging position P. Thus, the distance R from each light source to the current imaging position P may be greater than 15 cm if the intensity of light from each light source at the current imaging position is high enough to generate multiple images. In particular, if one or more of the multiple light sources are lasers, the distance R between each light source may be greater than 15 cm. However, it should be understood that the multiple light sources may be arranged on a flat surface or on a surface having an irregular shape. Furthermore, it should be understood that FIG. 2 shows a cross section of the microscope system 20, in particular the illumination system 260. Thus, the curved surface of the illumination system 260 illustrated in FIG. 2 may be a cylindrical surface or a part of a sphere (or a quasi-sphere). The curved surface 264 of the illumination device 260 may be bowl-shaped. The curved surface 264 may be formed of facets 265, which is shown in the example of FIG. 5. In other words, the curved surface 264 may be formed of multiple flat surfaces. The curved surface 264 may be a part of a quasi-sphere consisting of multiple facets or segments. Thus, the curved surface 264 may be a part of a surface of a polyhedron. An example of such a polyhedron may be a truncated icosahedron. The multiple light sources 261 may be arranged on the facets 265. Each light source may be arranged such that the light source is configured to emit light in a direction substantially parallel to the normal of the associated facet.It should be understood that, similar to the embodiment illustrated in FIG. 2, FIG. 5 illustrates a cross section of the illumination device 260. Thus, the curved surface 264 of the illumination system 260 illustrated in FIG. 5 may be a portion of a quasi-cylindrical or quasi-spherical surface. Thus, although the facets 265 in FIG. 5 are illustrated with lines, it should be understood that each facet 265 may be a flat surface having at least three sides. For example, the curved surface 264 may be formed with facets having five sides and facets having six sides (e.g., similar to the inner surface of a soccer ball or football). Although the curved surface 264 in FIG. 5 is illustrated as a continuous surface, it should be understood that each facet may be separate. Thus, the curved surface may be formed by multiple portions, and each facet may be formed by one or more portions. Furthermore, it should be understood that each portion may include one or more facets. Furthermore, such portions may be disposed in contact with adjacent portions or at a distance from adjacent portions. A single portion may include all the facets. 5 is an example, and other numbers of facets 265 may be used to form the curved surface 264 of the illumination system 260. Furthermore, it should be understood that the number of light sources on each facet 265 is an example, and the number may vary.

[0088] At least one microscope objective 280 is positioned to image the sample 292 onto the image sensor 270. The at least one microscope objective 280 may comprise a first microscope objective having a magnification of 20x and / or a numerical aperture of 0.4. The first microscope objective may be used to acquire a training set of digital images. The at least one microscope 280 may comprise a second microscope objective having a magnification of 100x and / or a numerical aperture of 1.25. The second microscope objective may be used to acquire a high-resolution image of the sample, and a portion of the high-resolution image of the sample may be classified and used to form the ground truth. The numerical aperture and magnification of the first microscope objective and / or the second microscope objective are exemplary and may be selected depending on, for example, the type of sample 292. For example, the numerical aperture of the first microscope objective may have a magnification of 10x and / or a numerical aperture of 0.25. It should be understood that the microscope system 20 may include additional optical lenses that may be used together with the at least one microscope objective lens 280 to image the sample 292 onto the image sensor 270. For example, the microscope system may include at least one relay lens 285 arranged such that the sample 292 may be imaged onto the image sensor 270 by the at least one microscope objective lens 280 and the at least one relay lens 285, as shown in the example of FIG. 2. Furthermore, it should be understood that the at least one relay lens 285 may be selected (e.g., focal length, material, size, etc.) depending on the magnification and / or numerical aperture of the at least one microscope objective lens 280. The at least one microscope objective lens 280 may be movable in a longitudinal direction Z by being coupled to a manual stage and / or a motorized stage. The longitudinal direction Z may be parallel to the optical axis of the microscope system 20. In other words, the at least one microscope objective lens 280 may be movable in a light collection direction of the microscope system 20. Alternatively, or in addition, the sample holder 290 may be movable along the longitudinal direction Z.The at least one microscope objective 280 and / or the sample holder 290 may be movable in a direction such that the sample 292 may be moved so that a focused image may be captured by the image sensor 270. The longitudinal direction Z of the at least one microscope objective 280 may be controlled by the circuit 200. For example, the circuit 200 may be configured to perform a focus function (not shown in FIG. 2 ) configured to adjust the position of the at least one microscope objective 280 along the longitudinal direction Z. The focus function may be configured to automatically adjust the position of the at least one microscope objective 280 along the longitudinal direction Z. In other words, the focus function may be an autofocus function.

[0089] The circuit 200 is configured to perform an illumination function 2100, a capture function 2102, and an analysis function 2104.

[0090] The illumination function 2100 is configured to control the illumination system 260 to sequentially illuminate the sample from a plurality of directions 262. The illumination function 2100 may be configured to control the illumination system 260 to illuminate the sample 292 with white light from each of the plurality of directions 262.

[0091] The capture function 2102 is configured to control the image sensor 270 to capture an input set of digital images. In other words, the capture function 2102 may be configured to receive (S400) an input set of digital images. The input set of digital images is captured by illuminating the sample 292 from a plurality of directions 262 and capturing a digital image for each of the plurality of directions 262. The input set of digital images may be captured by illuminating the sample 292 with white light from a plurality of directions 262 and capturing a digital image for each of the plurality of directions 262. As previously mentioned, at least one of the plurality of directions may correspond to an angle larger than the numerical aperture 282 of the microscope objective lens 280. For example, the direction 2620 of FIG. 2 may correspond to an angle larger than the numerical aperture 282 of the microscope objective lens 280. Light that enters at least one microscope objective 280 from a direction 2620 without being scattered may not be able to propagate through the microscope objective 280 (i.e., the angle of incidence of light from this direction may be outside the numerical aperture 282 of the microscope objective 280). Thus, light from this direction may need to be scattered by the sample 292 in order to propagate through the microscope objective 280.

[0092] The analysis function 2104 is configured to analyze the sample 292 (S402) by being configured to input (S404) an input set of digital images to a machine learning model trained in the manner described above. Since the sample in the example of FIG. 2 is an unstained sample, the machine learning model is trained using a training set of digital images including digital images of unstained samples. The analysis function 2104 is further configured to receive (S406) an output from the machine learning model including a classification of at least a portion of the sample 292 into at least one class. The output may further include a position of at least a portion within the sample 292. The output may further include a dimension of at least a portion within the sample 292. The at least a portion of the sample 292 may correspond to a portion of the sample 292 at the current imaging position P. The sample 292 may be moved such that some portions of the sample 292 are imaged in a manner similar to that described above. At least one portion within the sample 292 may correspond to an object within the sample 292. In other words, at least a portion within the sample 292 may be smaller or larger than the area of ​​the sample 292 at the current imaging position P.

[0093] Thus, the sample 292 can be classified directly from an input set of digital images without the need for a highly detailed digital image of the sample 292. Such a highly detailed digital image may be, for example, a digital image captured using a microscope objective having a relatively higher magnification than the microscope objective 280 used to capture the input set of digital images, or a digital image reconstructed using Fourier ptychography. Thus, the sample 292 can be classified by a trained machine learning model instead of a human, which may allow for a faster and / or more reliable classification of at least a portion of the sample 292. This may be further advantageous because reconstructing a high-resolution image using Fourier ptychography microscopy requires combining a large number of digital images (typically hundreds) into a reconstructed image by repeatedly transforming the digital images to and from the spatial frequency domain, e.g., by a Fourier transform, in order to properly combine them into a digital image having a higher resolution than the original digital images. The reconstruction process is therefore very computationally intensive and requires a large amount of memory since the original images and the reconstructed high-resolution images must be stored simultaneously. Thus, by avoiding such an iterative reconstruction process, the sample 292 can be analyzed in less time and using fewer computational resources. Furthermore, because the digital image reconstructed using computational image processing (e.g., Fourier ptychography) is related to the vertical level of the sample 292 (i.e., the level along the optical axis of the microscope objective lens), the inventive concepts may be more suitable for samples consisting of objects at different vertical levels (i.e., the level along the optical axis of the microscope system 20). This may be important for blood samples consisting of blood cells where it is important to distinguish between objects within the blood cells and objects above (or below) the blood cells. Since a machine learning model may be trained to directly classify such features of the sample 292, the trained machine learning model may directly classify the sample as including such features, whereas methods using the reconstructed images would require reconstructing digital images at multiple different vertical levels within the sample, thereby requiring more processing resources and memory.

[0094] Those skilled in the art will be aware of machine learning, and in particular, how a machine learning model may be trained and / or how a trained machine learning model may be used. However, briefly, the machine learning model may be a type of supervised machine learning model, such as a classification network, such as Xception, VGG, ResNet, EfficientNet, or Inception. The machine learning model may be a transformer-based network and / or an object detection network, such as Single Shot Detector, Yolo, Faster R-CNN, RetinaNet, or Spatial Pyramid Pooling. The machine learning model may be a convolutional neural network. The machine learning model is trained to predict a desired output using example input training data and a ground truth, i.e., a "correct" or "true" output. In other words, the ground truth is used as a label for the input training data. The input training data may comprise data related to different outcomes, and each input training data may thereby be associated with a ground truth related to that particular input training data. Thus, each input training data may be labeled with an associated ground truth (i.e., the "correct" or "true" output). A machine learning model may be composed of multiple layers of neurons, where each neuron may represent a mathematical operation applied to the input training data. Typically, a machine learning model is composed of an input layer, one or more hidden layers, and an output layer. The first layer is called the input layer. The output of each layer of the machine learning model (except the output layer) is fed to a subsequent layer, which generates a new output. The new output may be further fed to a subsequent layer. The output of the machine learning model may be the output of the output layer. This process may be repeated for all layers of the machine learning model. Typically, each layer further includes an activation function. The activation function may further define the output of the neurons of the layer.For example, the activation function may ensure that the output from the layer is not too large or too small (e.g., tending towards positive or negative infinity). Additionally, the activation function may introduce nonlinearity into the machine learning model. During the training process, weights and / or biases associated with the neurons of the layer may be adjusted until the machine learning model produces predictions for the input training data that reflect the ground truth. Each neuron may be configured to multiply the input to the neuron by a weight associated with that neuron. Each neuron may further be configured to add a bias associated with that neuron to the input. In other words, the output from a neuron may be the sum of the bias associated with the neuron and the product of the weight associated with the neuron and the input. The weights and biases may be adjusted in a recursive or iterative process. This is known in the art as backpropagation. A convolutional neural network may be a type of neural network consisting of one or more layers that represent convolution operations. In this context, the input training data comprises a digital image. The digital image may be represented as a matrix (or as an array), and each element of the matrix (or array) may represent a corresponding pixel of the digital image. Thereby, the value of the element may represent the pixel value of the corresponding pixel of the digital image. Thus, the input to the machine learning model may be a numerical input (e.g., a matrix or an array). The output of the machine learning model may be a numerical output. However, the numerical output of the machine learning model may be associated with a classification. For example, the machine learning model outputs numerical values ​​(e.g., integers), each numerical value being associated with a class. The output of the machine learning model may be a vector (or array) of floating point numbers. Each floating point number in the vector / array may be associated with a probability that the input belongs to the class associated with the position of the floating point number in the vector / array. Thus, the size (or length) of the output vector / array corresponds to the number of classes into which the machine learning model classifies (or is trained to classify) the input.If the output of the machine learning model is a vector / array, the input may be determined to belong to (i.e., classified into) the class associated with the location of the vector / array with the highest probability. Those skilled in the art will appreciate that the above description may also be applied to other characteristics. For example, the floating point numbers of the output vector / array may alternatively or additionally be associated with the location and / or dimensions (e.g., height, width, and shape) of a box that constitutes a portion of the sample, for example, related to an object of interest within the sample.

[0095] Those skilled in the art will appreciate that the inventive concept is in no way limited to the preferred variants described above, but rather many modifications and variations are possible within the scope of the appended claims.

[0096] For example, although the machine learning model has been described as being trained on only one sample, it should be understood that the machine learning model may be trained to analyze multiple samples. In such a case, the machine learning model may be trained using a training set of digital images of each sample of the multiple samples and a corresponding ground truth associated with each sample of the multiple samples.

[0097] It should further be understood that a digital image of a sample to be analyzed may be captured using a microscope system and then analyzed on another device (e.g., a computer, a server, a cloud server, etc.). Thus, it is also possible to capture a digital image using the microscope system 20 of FIG. 2 without using the analysis function 2104. The captured digital image may then be analyzed by a separate device comprising circuitry configured to perform the analysis function 2104. In such a case, the circuitry of the separate device may be configured to perform a receiving function configured to receive (S400) an input set of digital images. The receiving function may be configured to receive the input set of digital images from another storage device (a server, a cloud server, an external storage device, etc.) and / or from the microscope system used to capture the digital images.

[0098] Additionally, variations on the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

Claims

1. A method (30) for training a machine learning model for analyzing a sample, comprising: Receiving ground truth comprising a classification of at least a portion of the samples (S300); obtaining a training set of digital images of the sample by illuminating the sample from a plurality of directions and capturing a digital image for each of the plurality of directions (S302); training the machine learning model for analyzing the sample using the training set of digital images and the received ground truth (S304); A method (30), comprising:

2. The method of claim 1 , wherein the sample is an unstained sample.

3. 3. The method of claim 1 or 2, wherein the training set is obtained by illuminating the sample with white light from the multiple directions and capturing a digital image for each of the multiple directions.

4. The method (30) of any one of claims 1 to 3, wherein the machine learning model is a convolutional neural network.

5. The ground truth further comprises a location of the at least a portion of the specimen, and the step of training the machine learning model (S304) comprises: training (S306) the machine learning model to determine a location of the at least a portion of the specimen by training using the training set of digital images and the received ground truth until a difference between a location output of the machine learning model is less than an additional predetermined threshold; The method (30) of any one of claims 1 to 4, further comprising:

6. The ground truth comprises dimensions of the at least a portion of the specimen, and the step of training the machine learning model (S304) comprises: training (S307) the machine learning model to determine dimensions of the at least a portion of the specimen by training using the training set of digital images and the received ground truth until a difference between a dimension output of the machine learning model is less than an additional predetermined threshold; The method (30) of any one of claims 1 to 5, further comprising:

7. The method (30) of any one of claims 1 to 6, wherein the ground truth comprises a classification of each of a plurality of parts of the specimen.

8. The method of claim 7 , wherein the ground truth further comprises a location of each of the plurality of portions of the specimen.

9. 9. The method (30) of any one of claims 1 to 8, wherein the training set of digital images is acquired using a microscope objective and an image sensor, and at least one direction of the plurality of directions corresponds to an angle greater than the numerical aperture of the microscope objective.

10. A method (40) for analyzing a sample, comprising: receiving (S400) an input set of digital images of a specimen, the input set of digital images being obtained by illuminating the specimen from a plurality of directions and capturing a digital image for each of the plurality of directions; Analyzing the sample (S402), Inputting (S404) the input set of digital images into a machine learning model trained by the method (30) of any one of claims 1 to 9; receiving an output from the machine learning model, the output including a classification of at least a portion of the sample (S406); Analyzing the data (S402) A method (40).

11. The method (40) of claim 10, wherein the sample is an unstained sample.

12. 12. The method (40) of claim 10 or 11, wherein the input set of digital images is obtained by illuminating the sample with white light from the multiple directions and capturing a digital image for each of the multiple directions.

13. The method (40) of any one of claims 10 to 12, wherein the output comprises a position of the at least a portion of the sample.

14. 14. The method (40) of any one of claims 10 to 13, wherein the input set of digital images of the sample are acquired using a microscope objective and an image sensor, and at least one direction of the plurality of directions corresponds to an angle greater than a numerical aperture of the microscope objective.

15. An apparatus (10) for training a machine learning model, comprising a circuit (100), the circuit (100) comprising: a first receiving function (1100) configured to receive a training set of digital images, the training set of digital images being obtained by illuminating a sample from a plurality of directions and capturing digital images of at least a portion of the sample for each of the plurality of directions; a second receiving function (1102) configured to receive ground truth; Using the received ground truth and the acquired training set of digital images, a method (30) according to any one of claims 1 to 9 is performed, A training function (1104) configured to train a machine learning model; A circuit (100) configured to perform Apparatus (10).

16. 16. The apparatus (10) of claim 15, wherein the training set is obtained by illuminating the sample with white light from the multiple directions and capturing a digital image for each of the multiple directions.

17. The circuit (100) comprises: a third receiving function (1106) configured to receive a high resolution digital image of the sample; A decision function (1108), classifying at least a portion of the sample by classifying a corresponding portion of the high resolution digital image of the sample; forming a ground truth comprising the classification of the at least a portion of the samples; A decision function (1108) configured as follows: and further configured to execute 17. The apparatus (10) of claim 15 or 16, wherein the second receiving function (1102) is configured to receive the formed ground truth from the determining function (1108).

18. The training function (1104) is configured to train the machine learning model using a first subset of the training set of digital images, and the circuit (100) further comprises: a reconstruction function (1110) configured to reconstruct a high resolution digital image of the sample from a second subset of the training set of digital images, the high resolution image having a higher resolution than a resolution of the digital images of the training set; and further configured to execute 20. The apparatus (10) of claim 17, wherein the third receiving function (1106) is configured to receive the high resolution digital image of the sample from the reconstruction function (1110).

19. The decision function (1108) determining a location of a corresponding portion of the high resolution digital image of the sample; further configured to determine a position of the at least a portion of the sample; The ground truth may be determined based on the determined position of the at least a portion of the specimen.

19. The apparatus (10) of claim 17 or 18, further comprising:

20. A microscope system (20), comprising: an illumination system (260) configured to illuminate the sample (292) from multiple directions (262); An image sensor (270); at least one microscope objective (280) positioned to image the sample (292) onto the image sensor (270); A circuit (200), comprising: an illumination function (2100) configured to control the illumination system (260) to sequentially illuminate the sample (292) from the multiple directions (262); a capture function (2102) configured to control the image sensor (270) to acquire an input set of digital images, the input set of digital images being acquired by illuminating the sample (292) from a plurality of directions (262) and capturing a digital image for each of the plurality of directions (262); An analysis function (2104), Inputting said input set of digital images into a machine learning model trained by the method (30) of any one of claims 1 to 9; an analysis function (2104) configured to analyze the sample (292) by being configured to receive output from the machine learning model, the output including a classification of at least a portion of the sample (292) into at least one class; A circuit (100) configured to perform A microscope system (20) comprising:

21. 21. The microscope system (20) of claim 20, wherein the illumination system (260) comprises a plurality of light sources (261), each of the plurality of light sources (261) being configured to emit white light.

22. A microscope system (20) as described in claim 20 or 21, wherein the illumination system (260) comprises a plurality of light sources (261) arranged on a curved surface (264), the curved surface (264) being concave along at least one direction along the surface (264), and each light source of the plurality of light sources (261) is configured to illuminate the sample (292) from one of the plurality of directions.

23. 23. The microscope system (20) of claim 22, wherein the curved surface (264) is formed with facets (265).

24. A non-transitory computer readable storage medium comprising program code portions which, when executed on a device having processing capabilities, perform the method (30) of any one of claims 1 to 9 or the method (40) of any one of claims 10 to 14.

Images