Method and system for constructing a digital image depicting a focused sample

Abstract

The present invention relates to a method (30) and apparatus (10) for training a machine learning model to construct a digital image depicting a focused sample. The method includes obtaining a training set of digital images of a training sample (S300), which is done by positioning the training sample at a focal position outside the range between the near and far limits of the depth of field of the microscope objective lens (S302), illuminating the training sample with a plurality of illumination patterns using an illumination system (S304), and capturing a digital image of the training sample for each of the plurality of illumination patterns (S306), thereby obtaining a training set of digital images of a training sample (S300), receiving ground truth including a digital image depicting a focused training sample (S308), and training a machine learning model to construct a digital image depicting a focused sample using the training set of digital images and the ground truth (S310). The present invention further relates to a microscope system (20) and a method (40) for constructing a digital image of a focused sample.

Description

Technical Field

[0001] The present invention concept relates to a method and apparatus for training a machine learning model to construct a digital image depicting a focused sample. The present invention concept further relates to a method and a microscope system for constructing a digital image depicting a focused sample.

Background Art

[0002] In the field of digital microscopy, a typical task is to find and identify objects within a sample. For example, in hematology, cytology, and pathology, specific cell types may be found and identified in order to establish a diagnosis for the patient from whom the sample was taken.

[0003] There are various techniques for imaging a sample. One of the simplest forms of a digital microscope is a bright-field microscope, where the sample is illuminated from below with white light and imaged from above.

[0004] However, a drawback of such imaging techniques today is that the sample needs to be carefully positioned with respect to the microscope objective lens in order to obtain a focused image. This places high demands on the accuracy of the mechanism of the microscope system and also results in a time-consuming process.

[0005] Therefore, there is a need for improvement in the art.

Summary of the Invention

Problems to be Solved by the Invention

[0006] An object of the present invention is, at least in part, to mitigate, alleviate, or eliminate one or more of the deficiencies and drawbacks in the art identified above, alone or in any combination, and to solve at least the above-mentioned problems.

Means for Solving the Problems

[0007] The inventors of this invention have developed a machine learning method to realize a simple, accurate, and effective method and microscopy system for rendering a focused image of a sample without the need to precisely position the sample relative to the microscope objective lens, using Fourier ptychographic microscopy (FPM). Therefore, a more time-efficient system can be realized that does not require a highly precise sample positioning process. Furthermore, the less precise process can be made less expensive, thereby enabling a more affordable system.

[0008] According to a first aspect, a method is provided for training a machine learning model to construct a digital image depicting a focused sample. The method according to the first aspect includes obtaining a training set of digital images of a training sample by positioning the training sample at a focal position outside the range of positions between the near limit and far limit of the depth of field of a microscope objective lens, illuminating the training sample with a plurality of illumination patterns using an illumination system, and capturing a digital image of the training sample for each of the illumination patterns, receiving ground truth including a digital image depicting a focused training sample, and training a machine learning model to construct a digital image depicting a focused sample using the training set of digital images and the ground truth.

[0009] It should be noted that a training set of digital images may include digital images of multiple training samples. Therefore, a training set of digital images may include one or more sets of digital images for one or more training samples. Each set of digital images may have a corresponding ground truth.

[0010] In this specification, the term "training," as in "training sample," is used to refer to a sample used during training, as opposed to a general sample to which a machine learning model is trained to construct its digital image. The digital image subsequently constructed can depict a general sample in focus. The machine learning model can, of course, construct a digital image depicting the training sample. The machine learning model can also construct digital images of other samples that may not be part of the training set of digital images. In other words, after training, the machine learning model can construct digital images of samples that were not used to train the machine learning model. In this specification, depending on the context, the term "sample" may be used simply to refer to either a sample used during training or a sample input to a trained machine learning model.

[0011] In the context of this disclosure, the phrase “digital image of a focused sample” should be interpreted as a computer-generated digital image of a sample that is acceptablely in focus. This digital image may be similar to or identical to a digital image of a sample captured by the use of conventional techniques, such as by the use of a bright-field microscope, while the sample is in focus with respect to the microscope (e.g., at the best focus). Thus, the constructed digital image of a sample can replicate a drawing of a focused sample, but in an improved manner as described in the following disclosure. In this context, the phrase “focused sample” means that the sample is in focus.

[0012] In the context of this disclosure, the term “depth of field” of a microscope objective lens should be interpreted as the distance between the furthest and closest points from the microscope objective lens at which an object is acceptablely in focus. In this specification, “near limit” refers to the closest point of acceptable focus. In this specification, “far limit” refers to the furthest point of acceptable focus. Therefore, for example, when a training sample is “in focus,” this specification means that, at the time of imaging, the training sample was positioned at the focal point within the depth of field of the microscope objective lens used. The term “focal point” refers to a position along the optical axis of the microscope objective lens.

[0013] In the context of this disclosure, the term “ground truth” should be interpreted as information that is real and / or known to be true. Therefore, in this context, a machine learning model capable of constructing a digital image of a focused sample is trained using a training set of ground truth and digital images, so that the ground truth can represent an actual focused image of the training sample. Such an actual focused digital image of the training sample may be captured using conventional techniques, for example, by bright-field microscopy.

[0014] In the context of this disclosure, the term “illumination pattern” may be interpreted as different ways of illuminating a sample with one or more of a plurality of light sources. Different illumination patterns can be formed, for example, by illuminating the sample from one or more of a plurality of directions, and / or by changing the number of light sources among a plurality of light-emitting light sources.

[0015] When it is stated that "the training sample is positioned at the focal point relative to the microscope objective lens," it should be understood that the microscope objective lens may be positioned instead of, or in addition to, the training sample. Therefore, the method according to the first embodiment may include positioning the microscope objective lens relative to the training sample such that the training sample is at a focal point outside the range of positions between the near and far limits of the depth of field of the microscope objective lens.

[0016] A machine learning model can be trained to correlate a training set of digital images of training samples with ground truth (e.g., a digital image of a focused sample). The machine learning model can be trained iteratively and / or recursively until the difference between the machine learning model's output (i.e., a constructed digital image of a training sample) and ground truth (i.e., a digital image of a focused training sample) falls below a predetermined threshold. A smaller difference between the machine learning model's output and ground truth indicates greater accuracy in the constructed digital image of the sample provided by the machine learning model. In other words, a smaller difference between the machine learning model's output and ground truth indicates a greater degree of replication of the digital image of the focused sample. Therefore, preferably, the difference between the machine learning model's output and ground truth can be minimized. The machine learning model may be trained to construct digital images of samples for multiple different sample types. In that case, the machine learning model may be trained for each sample type using a training set of digital images of training samples of that sample type, and the corresponding ground truth associated with each sample type.

[0017] By illuminating a training sample with multiple illumination patterns and capturing a digital image for each of these patterns, information about finer details of the training sample can be captured compared to what is typically resolvable by conventional microscopes used to image the training sample (i.e., using conventional microscopy illumination such as bright-field illumination patterns). This can be understood as different parts of the Fourier space (i.e., the spatial frequency domain) associated with the training sample being imaged for different illumination directions. This technique is known in the art as Fourier ptychography microscopy (FPM). Furthermore, by illuminating the training sample with multiple illumination patterns and capturing a digital image for each of these patterns, information about the refractive index (or spatial distribution of the refractive index) associated with the training sample may be captured. This can be understood as the effect that the refraction of light depends on the angle of incidence of the light illuminating the sample and the refractive index of the training sample. Information about the refractive index of the training sample can be used to determine the phase information (commonly referred to in the art as quantitative phase) associated with the sample. The phase information can be used to replicate the training sample at different focal positions. In other words, even if a training set of digital images is acquired when the training sample is positioned outside the range between the near and far limits of the depth of field of the microscope objective lens (i.e., the training sample is out of focus), the training set of digital images still contains information that can be used to construct a digital image depicting a focused training sample. This is because the training sample was illuminated by multiple lighting patterns when the training set was acquired.Therefore, since multiple digital images contain information associated with one or more of the following: fine details of the sample, the refractive index associated with the sample, and phase information associated with the sample, this information can be used in training a machine learning model, thereby allowing the machine learning model to be trained to construct a digital image that more accurately depicts a focused training sample than would be possible if multiple digital images were captured using a single illumination pattern (e.g., conventional bright-field illumination, conventional dark-field illumination, or from a single direction) or by conventional microscopy. Using conventional microscopy (e.g., illuminating the sample from most multiple illumination patterns up to the numerical aperture of the microscope objective lens used to image the training sample), it may be difficult, or even impossible, to capture information associated with the refractive index and / or phase information associated with the training sample. In other words, because digital images captured using conventional microscopy may simultaneously contain information about the refraction of light striking from directions corresponding to overlapping portions of the Fourier space associated with the training sample, it may be impossible to determine the phase information and / or information associated with the refractive index of the training sample using such techniques. In other words, it may not be possible to determine such information related to a training sample using conventional microscopy. By illuminating the training sample with multiple illumination patterns, it may be possible to capture information related to finer details of the sample than would normally be possible with the microscope objective lens used to capture a digital image of the training sample. Therefore, a relatively low magnification microscope objective lens can be used while still being able to capture information related to these finer parts of the training sample. By using a relatively low magnification microscope objective lens, it is possible to capture a digital image of a larger portion of the training sample at each imaging position.Therefore, by capturing digital images at relatively fewer locations, the entire training sample can be scanned, thereby enabling faster scanning of the training sample.

[0018] Furthermore, additional information captured from the training sample allows the machine learning model to construct an image depicting a focused sample, even if the input image to the machine learning model is out of focus, or at least not in the best possible focus. This eliminates the need for the meticulous and time-consuming process of attempting to position the training sample to focus on the microscope objective lens. It also reduces the mechanical requirements for the microscope system used to position the training sample, as precise focusing is not required. Moreover, the reduced positioning requirements can decrease the time associated with positioning and / or imaging the training sample. Finally, the need for refocusing when changing training samples may also be eliminated for the same reasons mentioned above.

[0019] The resolution of the digital image depicting the focused training sample can be relatively higher than the resolution of at least one digital image in the training set of digital images.

[0020] In addition to the above, machine learning models can be trained to construct digital images that depict the training sample at a higher resolution than what is possible with the microscope objective lens used to capture the training set of digital images when using conventional microscope illumination (e.g., bright-field microscope).

[0021] Therefore, the concept of the present invention makes it possible to train a machine learning model that constructs a digital image that replicates the sample at a relatively high resolution using multiple digital images of the sample at a relatively low resolution. Thus, by using the trained machine learning model, a digital image depicting the sample can be constructed without the need to use a microscope objective lens with a relatively high magnification.

[0022] The aforementioned focal position may be one of a plurality of focal positions, and obtaining a training set of digital images of the training sample may further include positioning the training sample at each of the plurality of focal positions with respect to the microscope objective lens, and the acts of illuminating the training sample and capturing digital images of the training sample may be performed for each of the plurality of focal positions.

[0023] The performance of a machine learning model can be improved by training it using digital images in which the training sample is drawn at multiple focal points. For example, greater flexibility can be achieved in determining which focal point the sample can be positioned at in order to construct a digital image of the sample in focus using the trained machine learning model.

[0024] At least one of the multiple focal positions can be within the range of positions between the near limit and far limit of the depth of field of the microscope objective lens.

[0025] A related advantage is that the machine learning model can learn to construct a digital image of the sample even if the sample is already in focus, at least to some extent. Therefore, the proposed method can be used to construct a digital image of the sample in focus, using any focal position.

[0026] The act of receiving ground truth can include positioning a training sample at a focal position within a range of positions between the near limit and the far limit of the depth of field of the microscope objective lens with respect to the microscope objective lens, illuminating the training sample with a bright-field illumination pattern, and using the microscope objective lens to capture a digital image depicting the in-focus training sample while the training sample is illuminated with the bright-field illumination pattern.

[0027] Of the training set of digital images, the digital images may be captured using a first microscope objective lens. The ground truth may be captured using a second microscope objective lens. The numerical aperture of the second microscope objective lens may be greater than the numerical aperture of the first microscope objective lens. The magnification of the second microscope objective lens may be higher than the magnification of the first microscope objective lens.

[0028] As used herein, the phrase "bright-field illumination pattern" refers to an illumination pattern similar to the illumination used in a bright-field microscope and refers to an illumination pattern in which the sample is illuminated from the opposite side when the sample is imaged from one side. Thus, ground truth can be acquired with a bright-field microscope. The bright-field illumination pattern can be formed, for example, using the illumination source of a conventional microscopy method.

[0029] The digital image depicting the in-focus training sample (i.e., the ground truth) is captured by a microscope objective lens having a numerical aperture higher than that of the microscope objective lens when the training set was captured, so that the machine learning model can be trained to replicate the sample with a relatively high-resolution image from a set of relatively low-resolution images.

[0030] The illumination system can include a plurality of light sources, each of the plurality of light sources can be configured to illuminate the training sample from one of a plurality of directions, and each of the plurality of illumination patterns can be formed by one or more of the plurality of light sources.

[0031] The illumination patterns of multiple illumination patterns can be formed by illuminating (i.e., emitting light from) one or more of multiple light sources. Thus, each illumination pattern can be formed by simultaneously illuminating the training sample from one or more directions out of a group of directions. Each illumination pattern of multiple illumination patterns can be formed by simultaneously illuminating the training sample from one or more directions corresponding to non-overlapping portions of the Fourier space associated with the training sample. Of two different illumination patterns, at least two illumination directions can correspond to overlapping portions of the Fourier space associated with the training sample. A bright-field illumination pattern may be formed by simultaneously illuminating the training sample with most of the light-emitting diodes among a group of light-emitting diodes. For example, a bright-field illumination pattern may be formed by simultaneously illuminating the training sample with all (or almost all) of the light-emitting diodes of a group of white light-emitting diodes. A bright-field illumination pattern may be formed by simultaneously illuminating the training sample from directions corresponding to and less than the numerical aperture of the microscope objective lens used to image the training sample.

[0032] A possible and associated advantage is that different illumination patterns can be formed without physically moving any part of the device used to capture images of the training sample, and / or without moving the training sample itself.

[0033] At least one of the multiple directions can correspond to an angle greater than the numerical aperture of the microscope objective lens used to acquire the training set of digital images.

[0034] The numerical aperture of a microscope objective lens can be a dimensionless number related to the range of angles in which the objective lens accepts light. Therefore, directions greater than the numerical aperture of the microscope objective lens can be understood as corresponding to angles greater than the range of angles in which the objective lens accepts light.

[0035] By illuminating a training sample from a direction corresponding to an angle greater than the numerical aperture of the microscope objective lens, the digital image captured at that illumination angle can contain information about higher spatial frequencies of the training sample, and consequently, finer details of the training sample, than is normally possible with the microscope objective lens (i.e., using conventional microscope illumination). This allows the microscope objective lens to capture phase information associated with the training sample, as well as information about details of the training sample that are not normally resolvable by the microscope objective lens, and this information can be used to train a machine learning model. In other words, illuminating a training sample from a direction corresponding to an angle greater than the numerical aperture of the microscope objective lens enables improved training of a machine learning model for constructing a digital image depicting a focused sample.

[0036] The advantages and / or beneficial effects have been examined above with reference to the training samples. However, it should be understood that these advantages and / or beneficial effects may also be applicable when a digital image of a sample is used to construct a digital image depicting the focused sample using a machine learning model trained according to the first embodiment.

[0037] A second aspect provides a method for constructing a digital image of a focused sample. The method according to the second aspect includes receiving an input set of digital images of a sample, the input set of digital images being obtained by positioning the sample at a focal position relative to a microscope objective lens, illuminating the sample with a plurality of illumination patterns by an illumination system, and capturing a digital image of the sample for each of the illumination patterns; and constructing a digital image of a focused sample, which includes inputting the input set of digital images into a machine learning model trained according to the method of the first aspect, and receiving an output from the machine learning model that includes a constructed digital image of a focused sample.

[0038] When it is stated that "the sample is positioned at the focal point relative to the microscope objective lens," it should be understood that it may be the microscope objective lens that is being positioned instead of the sample. Therefore, the method according to the second embodiment may include positioning the microscope objective lens relative to the sample such that the sample is at the focal point.

[0039] Multiple lighting patterns used to acquire an input set of digital images may be formed in the same way as multiple lighting patterns used to acquire a training set of digital images used to train a machine learning model. For example, the lighting patterns may have the same direction, the same LED position, the same intensity, color, etc.

[0040] By inputting a set of digital images into a machine learning model trained according to the method of the first embodiment, the process of imaging a sample in focus can be made more efficient, as a digital image depicting a focused sample can be output from the trained machine learning model using digital images of the sample that do not need to be captured in focus. In other words, the proposed method makes it possible to position the sample at a focal position outside or inside the depth of field of the microscope objective lens used. It should be further understood that the proposed method makes it possible to position the sample such that a first part of the sample is inside the depth of field and a second part of the sample is outside the depth of field. For example, the sample may be tilted relative to the microscope objective lens, meaning that part of the sample may be in focus while other parts are not. Thus, the time-consuming process of adjusting the position of the sample relative to the microscope objective lens in order to focus on the sample can be avoided.

[0041] The act of receiving a set of digital image inputs of a sample may include positioning the sample at a focal point outside the range between the near and far limits of the depth of field of the microscope objective lens, illuminating the sample with multiple illumination patterns using an illumination system, and acquiring a set of digital image inputs of the sample by capturing a digital image of the sample for each of the multiple illumination patterns.

[0042] The illumination system may include multiple light sources, each of which may be configured to illuminate the sample from one of several directions, and each of the multiple illumination patterns may be formed by one or more of the multiple light sources.

[0043] At least one of the multiple directions can correspond to an angle greater than the numerical aperture of the microscope objective lens used to acquire the input set of digital images.

[0044] The features of the first embodiment described above also apply to this second embodiment, where applicable. References to the above will be made to avoid excessive repetition.

[0045] According to a third embodiment, an apparatus is provided for training a machine learning model to construct a digital image of a focused sample. The apparatus includes a circuit configured to perform a first receiving function, which is configured to receive a training set of digital images of a training sample, the first receiving function being acquired by positioning the training sample at a focal position outside the range of positions between the near and far limits of the depth of field of a microscope objective lens, illuminating the training sample with a plurality of illumination patterns by an illumination system, and capturing a digital image of the training sample for each of the illumination patterns, the circuit further configured to perform a second receiving function, which is configured to receive ground truth including a digital image of a focused training sample, and a training function, which is configured to train a machine learning model to construct a digital image of a focused sample using the training set of images and the ground truth. The apparatus for training a machine learning model may be configured to perform the method according to the first embodiment.

[0046] The features described above in the first and / or second aspects also apply to this third aspect, where applicable. References above are made to avoid excessive repetition.

[0047] According to a fourth aspect, a microscope system is provided. The microscope system comprises an illumination system configured to illuminate a sample with a plurality of illumination patterns; an image sensor configured to capture a digital image of the sample; a microscope objective lens configured to image the sample onto the image sensor; and a circuit configured to perform an acquisition function, which is configured to acquire a set of digital image inputs by performing an acquisition function, which is configured to perform an image construction function, which is configured to input the set of digital image inputs to a machine learning model trained according to the method of the first aspect, and to receive an output from the machine learning model, which includes a constructed digital image depicting a focused sample.

[0048] The illumination system may include multiple light sources, each of which may be configured to illuminate the sample from one of several directions, and each of the multiple illumination patterns may be formed by one or more of the multiple light sources.

[0049] At least one of the multiple directions can correspond to an angle greater than the numerical aperture of the microscope objective lens used to image the sample.

[0050] Multiple light sources can be arranged on a curved surface that is concave along at least one direction.

[0051] Placing multiple light sources on a curved surface can be advantageous in that the distance from each light source to the current imaging position of the microscope system (i.e., the position or portion of the sample currently being imaged) may be similar. Because these distances are similar, the intensity of light emitted from each light source at the current imaging position may be similar. This can be understood as the effect of the inverse square law. Therefore, the sample can be illuminated by light of similar intensity from each light source, thereby enabling more homogeneous illumination of the sample, independent of the direction of illumination. It may be advantageous to configure the illumination system such that the distance from each light source to the current imaging position is large enough so that each light source can be treated as a point source. The distance from each light source to the current imaging position may be chosen such that the intensity of light from each light source at the current imaging position is high enough to produce an input set of digital images.

[0052] A curved surface may be formed by facets. In other words, a curved surface can be constructed from multiple flat surfaces. Therefore, each section of a curved surface may be a flat surface.

[0053] One of the associated advantages is that lighting systems can be manufactured more easily, thereby reducing associated economic costs.

[0054] Another associated advantage is that the lighting system may be modular. This can make it easier to replace one or more light sources (for example, if a light source breaks and / or is defective).

[0055] The numerical aperture of the microscope objective lens may be 0.4 or less. In other words, the microscope objective lens can have a magnification of 20x or less.

[0056] One associated advantage is that, compared to microscope objectives with larger numerical apertures, a larger portion of the sample can be imaged at once. This can reduce the number of individual imaging positions required to image most of the sample. As a result, the time required to image most of the sample can be shortened. This can be particularly advantageous because machine learning models can be trained to construct digital images with relatively higher resolution than the digital images input to the machine learning model (i.e., the digital images of the input set). Thus, while a larger portion (or the entirety) of the sample can be imaged more quickly, the resulting digital image of the sample can have a relatively higher resolution than what a microscope objective lens typically allows (using conventional microscope illumination).

[0057] The features described above in the first, second, and / or third aspects also apply to this fourth aspect, where applicable. References to the above are made to avoid excessive repetition.

[0058] According to a fifth aspect, a non-temporary computer-readable storage medium is provided. The non-temporary computer-readable storage medium includes a program code portion that, when executed on a device having processing capabilities, performs the method according to the second aspect.

[0059] The features described above in the first, second, third, and / or fourth aspects also apply to this fifth aspect, where applicable. References to the above are made to avoid excessive repetition.

[0060] Further scope of the applicability of this disclosure will become apparent from the detailed description given below. However, various changes and modifications within the scope of the concept of the invention will become apparent to those skilled in the art from this detailed description, so it should be understood that the detailed description and specific examples, while illustrating preferred variations of the concept of the invention, are given for illustrative purposes only.

[0061] Therefore, since the methods or systems described may change, it should be understood that the concepts of the present invention are not limited to specific steps of such methods or component parts of such systems. Also, it should be understood that the terms used herein are merely for describing specific embodiments and are not intended to be limiting. It should be noted that, as used herein and in the appended claims, the articles “a,” “an,” “the,” and “said” are intended to mean that one or more elements exist unless the context specifically states otherwise. For example, a reference to “a unit” or “the unit” may include multiple devices. Furthermore, the words and similar expressions “comprising,” “including,” and “containing” do not preclude other elements or steps.

[0062] Next, the above and other embodiments of the concept of the present invention will be described in more detail with reference to the accompanying drawings illustrating variations of the concept of the present invention. The drawings should not be considered as limiting the concept of the invention to specific variations, but rather are used to illustrate and understand the concept of the invention. As shown in the drawings, the sizes of layers and regions are exaggerated for illustrative purposes and are therefore provided to illustrate the general structure of variations of the concept of the present invention. Similar reference numerals throughout refer to similar elements. [Brief explanation of the drawing]

[0063] [Figure 1] This is a diagram of a device for training machine learning models. [Figure 2] This is a diagram showing a microscope system. [Figure 3] This is a block scheme diagram of a method for training a machine learning model to construct a digital image depicting a focused sample. [Figure 4] This is a block scheme diagram of a method for constructing a digital image depicting a focused sample using a trained machine learning model. [Figure 5] This is a diagram of a lighting system having a curved surface formed by facets. [Figure 6] This is a diagram of a non-temporary computer-readable storage medium. [Figure 7A] This is a diagram of a digital image of the sample. [Figure 7B] This is a diagram of a constructed digital image of a focused sample. [Figure 8] This is a schematic diagram of the depth of field of a microscope objective lens. [Modes for carrying out the invention]

[0064] Next, the concept of the present invention is described more fully below with reference to the accompanying drawings showing currently preferred modifications of the concept. However, the concept of the present invention can be implemented in many different forms and should not be construed as being limited to the modifications described herein, but rather these modifications are provided for thoroughness and completeness and fully convey the scope of the concept of the present invention to those skilled in the art.

[0065] Next, apparatus 10 and method 30 for training a machine learning model to construct a digital image depicting a focused sample are described with reference to Figures 1 and 3.

[0066] Figure 1 shows, as an example, apparatus 10 for training a machine learning model to construct a digital image depicting a focused sample. Apparatus 10 is shown as a schematic diagram. Therefore, please note that the size, shape, and position of different elements in the drawing are not limiting in any way and are merely for illustrative purposes.

[0067] Device 10 may be a computing device. Suitable examples of computing devices include computers, servers, smartphones, tablets, etc. Device 10 may be further implemented as part of a cloud server and / or distributed computing deployment. It should be further understood that Device 10 may include further components, such as input devices (mouse, keyboard, touchscreen, etc.) and / or displays. Device 10 may further include a power supply, such as a power connection, batteries, etc. Device 10 comprises a circuit 100. As shown in the example in Figure 1, the circuit 100 may comprise one or more of the following: memory 110, processing unit 120, transceiver 130, and data bus 140. The memory 110, processing unit 120, and transceiver 130 may be configured to communicate via the data bus 140. An accompanying control line and / or address bus may also be present between two or more of the memory 110, processing unit 120, and transceiver 130.

[0068] The processing unit 120 may include, for example, one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, and a microprocessor. The processing unit 120 may be configured to execute program code stored in the memory 110 in order to perform the functions and operations of the device 10.

[0069] The transceiver 130 can be configured to communicate with an external device. The transceiver 130 can transmit data to device 10 and receive data from device 10. For example, the transceiver 130 may be configured to communicate with a server, a computer peripheral (e.g., external storage), etc. The external device may be a local device or a remote device (e.g., a cloud server). The transceiver 130 may be configured to communicate with the external device via an external network (e.g., a local area network, the internet, etc.). The transceiver 130 may be configured for wireless and / or wired communication. Appropriate techniques for wireless communication are known to those skilled in the art. Some non-limiting examples include Wi-Fi, Bluetooth, and Near Field Communication (NFC). Appropriate techniques for wired communication are known to those skilled in the art. Some non-limiting examples include USB, Ethernet, and Firewire.

[0070] Memory 110 may be a non-temporary computer-readable storage medium. Memory 110 may be random-access memory. Memory 110 may be non-volatile memory. Memory 110 may include one or more of the following: buffers, flash memory, hard drives, removable media, volatile memory, non-volatile memory, random-access memory (RAM), and other suitable devices. In a typical configuration, memory may include non-volatile memory for long-term data storage and volatile memory that functions as system memory for device 10. Memory 110 can exchange data within circuit 100 via the data bus 140.

[0071] As shown in the example in Figure 1, memory 110 can store program code portions 1100, 1102, 1104, and 1106 corresponding to one or more functions. These program code portions 1100, 1102, 1104, and 1106 may be executable by a processing unit 120 that thereby performs the function. Therefore, when it is mentioned that circuit 100 is configured to perform a particular function, the processing unit 120 can execute program code portions corresponding to that particular function that can be stored in memory 110. However, it should be understood that one or more functions of circuit 100 may be implemented in hardware and / or in a particular integrated circuit. For example, one or more functions may be implemented using a field-programmable gate array (FPGA). In other words, one or more functions of circuit 100 can be implemented in hardware, software, or a combination of both.

[0072] Circuit 100 is configured to perform a first receiving function 1100, a second receiving function 1102, and a training function 1104. Circuit 100 may be configured to perform additional functions.

[0073] The first receiving function 1100 is configured to receive a training set of digital images of a training sample. The training set of digital images is formed (has been formed) by positioning the training sample at a focal position outside the range between the near and far limits of the depth of field of the microscope objective lens, illuminating the training sample with multiple illumination patterns using an illumination system, and capturing a digital image of the training sample for each of the multiple illumination patterns.

[0074] The first receiving function 1100 may be configured to receive training sets of digital images via the transceiver 130. For example, the training sets of digital images may be captured using an external microscope system and then transmitted to the transceiver 130 of the device 10. As a further example, the device 10 may form part of a microscope system, and the training sets of digital images may be received from the image sensors of the microscope system. As yet another example, the training sets of digital images may be stored in a storage device (e.g., an external or internal storage device) and received from that storage device. The circuit 100 may be configured to execute an instruction function 1106 configured to send instructions on how to capture the training sets of images to the microscope system via the transceiver 130. The memory 110 may be configured to store training sets of digital images, and the first receiving function 1100 may be configured to receive training sets of digital images from the memory 110.

[0075] A training set of digital images can be acquired (or captured) using a first microscope objective lens and an image sensor. For example, the training set of digital images may be acquired using a microscope system 20, as further described below in relation to Figure 2. The training sample is positioned at the focal position relative to the first microscope objective lens and can be illuminated sequentially by an illumination system with each of a plurality of illumination patterns. Each digital image in the training set of digital images may be captured when the training sample is illuminated with one of the illumination patterns. The focal position may be outside the range of positions between the near and far limits of the depth of field of the first microscope objective lens. The receiving function 1100 may be configured to acquire the training set of digital images in accordance with the above.

[0076] The focal position may be one of a plurality of focal positions. Obtaining a training set of digital images of a training sample may further include positioning the training sample at each of the plurality of focal positions with respect to the microscope objective lens (or first microscope objective lens). Illuminating the training sample and capturing a digital image of the training sample can be performed for each of the plurality of focal positions. At least one of the plurality of focal positions may be within the range of positions between the near limit and far limit of the depth of field of the microscope objective lens.

[0077] The illumination system may consist of multiple light sources. Each of the multiple light sources may be configured to illuminate the training sample from one of several directions. In other words, each light source can illuminate the training sample from a different (or unique) direction. Different illumination patterns among the multiple illumination patterns can be formed by illuminating the training sample with at least one of the multiple light sources. In other words, each illumination pattern among the multiple illumination patterns may be formed by illuminating the sample from at least one of several directions. Thus, the training sample can be illuminated by a single light source among the multiple light sources (or from a single direction among several directions). Alternatively, the training sample may be illuminated simultaneously by several light sources among the multiple light sources (or from several directions among several directions). In such cases, the directions used to form a single illumination pattern can correspond to non-overlapping portions of the Fourier space associated with the training sample. The light sources may be white light-emitting diodes (LEDs). By using white LEDs, the training set of digital images can include information about the training sample associated with a relatively wide wavelength range, especially compared to using narrowband light (e.g., lasers, monochromatic LEDs). This allows machine learning models to be trained to construct color images of the training sample (or sample).

[0078] At least one of several directions can correspond to an angle greater than the numerical aperture of the first microscope objective lens used to capture digital images of the training set of digital images. Since the training sample is illuminated from several different directions and a digital image can be captured for each of the several directions, information about finer parts of the training sample can be captured than what is normally resolvable by the microscope objective lens used to image the training sample. This can be understood as information from different parts of the Fourier space (i.e., the spatial frequency domain) associated with the training sample being captured for different illumination directions. This technique is known in the art as Fourier ptychography. In general, in Fourier ptychography, when a sample is illuminated from a direction corresponding to a large angle of incidence, high spatial frequencies are sampled in the Fourier space associated with that sample. Therefore, when the sample is illuminated from a direction corresponding to an angle greater than the numerical aperture of the first microscope objective lens, even higher spatial frequencies in Fourier space may be sampled. This is possible because light is scattered by the sample, and some of the light scattered by the sample can be collected by a first microscope objective lens. This illumination technique can further be used to capture information about the refractive index (or spatial distribution of refractive index) associated with the sample by a microscope objective lens (e.g., a first microscope objective lens) used to image the sample. This can be understood as the effect of light refraction, which depends on the angle of incidence of the light illuminating the sample and the refractive index of the sample. Information about the refractive index of the sample can be used to determine phase information (commonly referred to in the art as quantitative phase) associated with the sample.Since multiple digital images contain information associated with one or more of the following: fine details of the sample, the refractive index associated with the sample, and phase information associated with the sample, this information can be used in training a machine learning model, thereby allowing the model to be trained to construct a more accurate digital image of the focused sample than would be possible if multiple digital images were captured from only one direction. In other words, a machine learning model can be trained to construct a more accurate digital image of the focused sample than would be possible if multiple digital images were captured from only one direction, or if they were captured using a conventional microscope (e.g., with bright-field illumination). It should be understood that information associated with one or more of the finer details of the sample, the refractive index associated with the sample, and phase information associated with the sample can be captured by illuminating the sample from two or more different directions at once, for example, from a subset of multiple directions. A subset of multiple directions may include directions corresponding to non-overlapping portions of the Fourier space of the sample. Therefore, while microscope objective lenses with relatively low magnification can be used, it is still possible to capture information about fine details of the sample (i.e., details finer than what a microscope objective lens can normally enable using conventional microscope illumination). For example, by illuminating the sample from multiple directions, a microscope objective lens with a numerical aperture of 0.4 can capture information about fine details to the same extent as a microscope objective lens with a numerical aperture of 1.25 used in conventional microscopy (e.g., using bright-field illumination). In other words, by illuminating the sample from multiple directions, a microscope objective lens with a magnification of 20x can capture information about fine details to the same extent as a microscope objective lens with a magnification of 100x used in conventional microscopy (e.g., using bright-field illumination). It should be understood that the above magnifications and numerical apertures are merely examples, and the present invention can be similarly implemented with other magnifications and / or numerical apertures.A suitable microscope system, including a microscope objective lens and an image sensor, is described in relation to Figure 2.

[0079] The second receiving function 1102 is configured to receive ground truth, which includes a digital image of a focused training sample. The resolution of the digital image of the focused training sample (i.e., ground truth) can be relatively higher than the resolution of at least one digital image in the training set of digital images. For example, as a non-limiting example, the digital image of the focused training sample can be captured using a second microscope objective lens having a numerical aperture of 1.25 (or 100x magnification). The resolution of the digital image of the focused training sample (i.e., ground truth) can be relatively higher than the resolution of each digital image in the training set of digital images. Ground truth can be information that is known to be real and / or true. In this context, since the machine learning model is trained to construct a digital image of a focused sample, ground truth can represent a "correct" representation of the training sample. For example, ground truth can include digital images of the training sample captured using different imaging techniques. Ground truth can be received via transceiver 130. For example, ground truth can be formed on different devices and / or stored on different devices and transmitted to device 10 via transceiver 130.

[0080] The command function 1106 may be further configured to transmit commands to the microscope system via the transceiver 130 regarding how to acquire a digital image of ground truth. Ground truth can be acquired by positioning a training sample at a focal position relative to the microscope objective lens, within a range of positions between the near and far limits of the depth of field of the microscope objective lens; illuminating the training sample with a bright-field illumination pattern; and using the microscope objective lens to capture a digital image of the focused training sample while the training sample is illuminated. Thus, a digital image of ground truth can be captured when the training sample is positioned at a focal position within the depth of field of the microscope objective lens. The numerical aperture of the second microscope objective lens (i.e., the microscope objective lens used to acquire ground truth) may be greater than the numerical aperture of the first microscope objective lens (i.e., the microscope objective lens used to acquire the training set of digital images). Thus, the resolution of ground truth can be relatively higher than the resolution of at least one of the digital images in the training set of digital images. A second microscope objective lens used to capture a digital image of a focused training sample may have a relatively larger numerical aperture (and therefore higher magnification) than the first microscope objective lens. Therefore, the digital image of the focused training sample can depict a smaller area of ​​the training sample compared to an image captured using the first microscope objective lens used to capture a training set of digital images. For this reason, the digital image of the focused sample may be formed from multiple individual digital images of the sample, for example, by combining multiple individual digital images (e.g., by stitching). This allows the digital image of the focused training sample to depict an area of ​​the training sample comparable to the area of ​​the training sample imaged by the first microscope objective lens used to capture a training set of digital images.Alternatively, each digital image in the training set may be cropped so that the area of ​​the training sample drawn on each cropped digital image in the training set is similar to and / or equivalent to the area of ​​the training sample drawn on the digital image of the focused training sample.

[0081] A bright-field illumination pattern may be formed by simultaneously illuminating the training sample from a subset of multiple illumination patterns used to illuminate the training sample during the acquisition of a training set of digital images. In other words, a bright-field illumination pattern can be formed by illuminating the training sample from a subset of multiple directions realized by multiple light sources in the illumination system, i.e., from one or more of the multiple directions. The directions of the subset, and the number of directions, may be selected so that the illumination of the training sample is similar to conventional microscope illumination (e.g., bright-field illumination). The subset of directions may be, for example, most of the multiple directions, or all of the multiple directions. The subset of directions may correspond to angles less than or equal to the numerical aperture of the microscope objective lens used to capture the digital image of ground truth. A digital image of a focused training sample may be captured while the training sample is simultaneously illuminated from the subset of multiple directions. Thus, a digital image of a focused training sample may be similar to a focused digital image captured by a conventional microscope system using, for example, a bright-field microscope. The training set of digital images may be acquired prior to the digital images of ground truth. Alternatively, or in addition, the bright-field illumination pattern can be formed by illuminating the training sample with a light source suitable for conventional microscopy (i.e., bright-field microscopy).

[0082] The training function 1104 is configured to train a machine learning model to construct a digital image depicting a focused sample using a training set of digital images and its ground truth. Thus, the training function 1104 (or training device 10) can be configured to perform method 30 as described in relation to Figure 3. The machine learning model may be a convolutional neural network. The machine learning model may be a convolutional neural network suitable for constructing digital images.

[0083] The training function 1104 can be configured to train the machine learning model iteratively and / or recursively until the difference between the output of the machine learning model (i.e., a constructed digital image depicting a focused sample) and the ground truth (e.g., a digital image depicting a focused training sample) is smaller than a predetermined threshold. Thus, the training function 1104 can train the machine learning model to correlate a training set of digital images of training samples with the ground truth (e.g., a digital image depicting a focused training sample). A smaller difference between the output of the machine learning model and the ground truth indicates a higher accuracy of the constructed digital image depicting a focused sample provided by the machine learning model. Therefore, preferably, the difference between the output of the machine learning model and the ground truth can be minimized. Those skilled in the art will recognize that a minimization function (e.g., a loss function) may be associated with tolerance. For example, a loss function can be considered minimized even if the minimized loss function has a value that is not a local minimum and / or global minimum.

[0084] A machine learning model can be trained using multiple training sets and multiple corresponding ground truths. In other words, a machine learning model may be trained using multiple different samples. This can improve the training of the machine learning model for constructing digital images of focused samples. A machine learning model may be trained to construct digital images of focused samples of multiple different sample types. In such cases, for each sample type, the machine learning model may be trained using a training set of digital images of training samples of that sample type, and the corresponding ground truth associated with samples of each sample type (e.g., a focused digital image of a sample of that sample type). This allows the trained machine learning model to construct digital images of focused samples of different sample types. The trained machine learning model can construct digital images of focused samples that were not used during training.

[0085] Next, a microscope system 20 and method 40 for constructing a digital image of a focused sample will be described with reference to Figures 2 and 4.

[0086] Figure 2 shows a microscope system 20. The microscope system 20 in Figure 2 may be suitable for acquiring a training set of digital images of a training sample used when training a machine learning model to construct a digital image of a focused sample, as described above in relation to the apparatus 10 in Figure 1, or in relation to the method described later in relation to Figure 3. Furthermore, the microscope system 20 in Figure 2 may be suitable for acquiring a digital image of a focused training sample (i.e., ground truth) used to train a machine learning model. Furthermore, the microscope system 20 in Figure 2 may be suitable for acquiring an input set of digital images of a sample used by a machine learning model to construct a digital image of a focused sample (as described further later in relation to Figures 2 and 4).

[0087] The microscope system 20 includes an illumination system 260, an image sensor 270, a microscope objective lens 280, and a circuit 200. The microscope system 20 may further include a sample holder 290, as shown in the example in Figure 2. The microscope system 20 may also include further components, such as an input device (mouse, keyboard, touchscreen, etc.) and / or a display.

[0088] The circuit 200 may include one or more of the following: memory 210, processing unit 220, transceiver 230, and data bus 240. The processing unit 220 may include a central processing unit (CPU) and / or graphics processing unit (GPU). The transceiver 230 may be configured to communicate with an external device. For example, the transceiver 230 may be configured to communicate with a server, a computer peripheral (e.g., external storage), etc. The external device may be a local device or a remote device (e.g., a cloud server). The transceiver 230 may be configured to communicate with the external device via an external network (e.g., a local area network, the Internet, etc.). The transceiver 230 may be configured for wireless and / or wired communication. Appropriate techniques for wireless communication are known to those skilled in the art. Some non-limiting examples include Wi-Fi, Bluetooth, and Near Field Communication (NFC). Appropriate techniques for wired communication are known to those skilled in the art. Some non-limiting examples include USB, Ethernet, and Firewire. The memory 210, processing unit 220, and transceiver 230 can communicate via the data bus 240. The illumination system 260 and / or image sensor 270 may be configured to communicate with the circuit 200 via the transceiver 230, as shown in Figure 2. In addition, the illumination system 260 and / or image sensor 270 may be configured to communicate directly with the data bus 240 (e.g., via a wired connection). The memory 210 may be a non-temporary computer-readable storage medium. The memory 210 may be random-access memory. The memory 210 may be non-volatile memory. The memory 210 may include one or more of the following: a buffer, flash memory, hard drive, removable medium, volatile memory, non-volatile memory, random-access memory (RAM), or other suitable devices. In a typical configuration, the memory may include non-volatile memory for long-term data storage and volatile memory serving as system memory for the microscope system 20.The memory 210 can exchange data within the circuit 200 via the data bus 240. As shown in the example in Figure 2, the memory 210 can store program code portions 2100, 2102 corresponding to one or more functions. The program code portions 2100, 2102 may be executable by a processing unit 220 that thereby performs the function. Thus, when it is mentioned that the circuit 200 is configured to perform a particular function, the processing unit 220 can execute a program code portion corresponding to that particular function that can be stored in the memory 210. However, it should be understood that one or more functions of the circuit 200 may be implemented in hardware and / or in a particular integrated circuit. For example, one or more functions may be implemented using a field-programmable gate array (FPGA). In other words, one or more functions of the circuit 200 can be implemented in hardware, software, or a combination of both.

[0089] Although Figure 2 shows the image sensor 270 as a standalone unit, it should be understood that the image sensor 270 may be included in a camera (not shown in Figure 2). In the example in Figure 2, the sample holder 290 is a microscope slide coated with the sample 292. It should be understood that the sample 292 may be covered with a coverslip (not shown in Figure 2). The sample holder 290 may be configured to hold the sample 292 to be analyzed. The sample holder 290 may be movable (for example, by being coupled to a manual and / or motorized stage) so that the sample 292 can be moved so that different parts of the sample 292 can be imaged by the microscope objective lens 280.

[0090] The illumination system 260 is configured to illuminate the sample 292 with multiple illumination patterns. As shown in Figure 2, the illumination system 260 may include multiple light sources 261. The light sources may be configured to illuminate the sample with broad-spectrum light (i.e., white light). Alternatively, the illumination system may illuminate the sample with narrow-spectrum light (e.g., monochromatic light). The light sources 261 may be light-emitting diodes (LEDs). The LEDs may be white LEDs or color LEDs. A white LED can be, for example, a blue LED covered with a layer of fluorescent material that emits white light when illuminated. The LEDs may be any type of LED, such as a regular LED bulb (i.e., conventional LEDs and inorganic LEDs), a graphene LED, or an LED commonly found in displays (e.g., quantum dot LEDs (QLEDs) or organic LEDs (OLEDs)). However, other types of LEDs may also be used. The illumination system 260 may include, for example, multiple lasers, and the light emitted from each of the multiple lasers may be converted into light having a wider spectral bandwidth. One example of such a conversion process is sometimes called supercontinuum generation. The light source can emit incoherent light, quasi-coherent light, or coherent light.

[0091] Multiple illumination patterns can be formed by illuminating the sample 292 with one or more of the multiple light sources 261. In other words, multiple illumination patterns can also be formed by illuminating the sample 292 with just one of the multiple light sources 261. Alternatively, or as a combination, multiple illumination patterns can be formed by simultaneously illuminating the sample 292 with a subset of the multiple light sources 261.

[0092] Each of the multiple light sources 261 may be positioned to illuminate the sample 292 from one of the multiple directions 262. The illumination system 260 can be configured to illuminate the sample 292 simultaneously with one or more of the multiple light sources 261. In other words, the illumination system 260 can be configured to illuminate the sample 292 simultaneously from one or more of the multiple directions 262. At least one of the multiple directions can correspond to an angle greater than the numerical aperture of the microscope objective lens 280.

[0093] As further shown in Figure 2, multiple light sources 261 may be arranged on a curved surface 264. As shown in the example in Figure 2, the curved surface 264 may be concave along at least one direction along the surface 264. For example, the curved surface 264 may be a cylindrical surface. The curved surface 264 may be concave along two perpendicular directions along the surface. For example, the curved surface 264 may have a shape similar to a segment of a sphere. The segment of the sphere may be a spherical cap or a spherical dome. Arranging multiple light sources 261 on a curved surface 264 may be advantageous in that the distance R from each light source to the current imaging position P of the microscope system 20 may be similar. Because this distance is similar, the intensity of the light emitted from each light source at the current imaging position P may be similar. This may be understood as the effect of the inverse square law. Thus, the sample 292 can be illuminated by light having similar intensity in each direction in multiple directions 262, thereby enabling more homogeneous illumination of the sample 292 that is independent of the illumination direction. The distance R from each light source to the current imaging position P can range from 4 cm to 15 cm. It may be advantageous to configure the illumination system 260 such that the distance R from each light source to the current imaging position P is large enough so that each light source can be treated as a point source. Therefore, the distance R from each light source to the current imaging position P may be greater than 15 cm if the light intensity from each light source at the current imaging position is high enough to produce a set of digital images. However, it should be understood that the multiple light sources 261 may be arranged on a flat surface or on a surface having an irregular shape. Furthermore, it should be understood that Figure 2 shows a cross-section of the microscope system 20, in particular the illumination system 260. Therefore, the curved surface 264 of the illumination system 260 shown in Figure 2 may be part of a cylindrical surface or a spherical surface (or a quasi-spherical surface). The curved surface 264 of the illumination system 260 may be bowl-shaped. The curved surface 264 may be formed by facets 265, as shown in the example in Figure 5. In other words, the curved surface 264 may be formed from multiple flat surfaces. Therefore, each section of the curved surface 264 may be flat.The curved surface 264 may be part of a quasi-spherical surface containing multiple facets or segments. Thus, the curved surface 264 may be part of a polyhedron surface. An example of such a polyhedron may be a truncated icosahedron. Multiple light sources 261 may be arranged on the facet 265. Each light source can be arranged to emit light in a direction substantially parallel to the normal of the facet to which the light source is associated. It should be understood that, as with the example shown in Figure 2, Figure 5 shows a cross-section of the illumination system 260. Thus, the curved surface 264 of the illumination system 260 shown in Figure 5 may be part of a quasi-cylindrical surface or a quasi-spherical surface. The curved surface 264 of the illumination system 260 in Figure 5 may have a bowl-like shape. Thus, it should be understood that although the facet 265 in Figure 5 is illustrated by lines, each facet 265 may be a flat surface having at least three facets. For example, the curved surface 264 can be formed from facets having five facets and facets having six facets (similar to the inner surface of a football or soccer ball, for example). Although the curved surface 264 in Figure 5 is illustrated as a continuous surface, it should be understood that each facet may be distinct. Thus, the curved surface may be formed from multiple parts, and each facet may be formed from one or more parts. It should be further understood that each part may contain one or more facets. Furthermore, such parts may be positioned in contact with adjacent parts or at a certain distance from adjacent parts. A single part may contain all the facets. It should be further understood that the number of facets 265 of the lighting system 260 in Figure 5 is merely an example, and other numbers of facets 265 may be used to form the curved surface 264 of the lighting system 260. Furthermore, it should be understood that the number of light sources on each facet 265 is merely an example, and that number may vary.

[0094] As shown in Figure 2, the microscope objective lens 280 is positioned to image the sample 292 on the image sensor 270. The microscope system 20 can include multiple microscope objective lenses. Therefore, the microscope system 20 can include at least one microscope objective lens 280. This allows the microscope system 20 to image the sample with different microscope objective lenses. For example, the microscope system 20 can include a first microscope objective lens. The numerical aperture of the first microscope objective lens may be 0.4 or less. In other words, the first microscope objective lens can have a magnification of 20x or less. Therefore, compared to microscope objective lenses with relatively large numerical apertures, a larger portion of the sample 292 can be imaged at once. This can reduce the number of individual imaging positions required to image most of the sample 292. Thus, the time required to image most of the sample 292 can be reduced. This can be particularly advantageous when a machine learning model is trained to construct digital images with a relatively higher resolution than the digital images input to the machine learning model (i.e., the digital images of the input set). Therefore, sample 292 can be imaged more quickly, while the constructed digital image depicting the focused sample can have a relatively higher resolution than the resolution typically achievable with the first microscope objective lens. The first microscope objective lens can be used to capture a digital image of sample 292. The digital image captured using the first microscope objective lens can be used when training a machine learning model (i.e., a training set of digital images of the sample) and / or when obtaining an input set of digital images of the sample to be fed into a machine learning model trained to construct a digital image depicting the focused sample.

[0095] The microscope system 20 can also be configured to change the relative position between the sample 292 and the microscope objective lens 280 in order to position the sample at different focal positions. This can be achieved by moving the sample holder 290 toward the microscope objective lens 290 and / or away from the microscope objective lens 290. Alternatively, or in addition, this can be achieved by moving the microscope objective lens toward the sample 292 and / or away from the microscope objective lens.

[0096] The microscope system 20 may further include a second microscope objective lens having, for example, a magnification of 100x and / or an numerical aperture of 1.25. The second microscope objective lens can be used to acquire a digital image having a relatively higher resolution than the digital image acquired using the first microscope objective lens. Thus, the second microscope objective lens can be used to acquire a digital image (i.e., ground truth) depicting a focused sample 292 for use when training a machine learning model. In other words, the digital image captured using the second microscope objective lens can be used to form ground truth for use when training a machine learning model. As described above in relation to Figure 1, ground truth may be acquired by capturing an image of the sample when it is illuminated with a bright-field illumination pattern and positioned at a focal position within a range of positions between the near and far limits of the depth of field of the microscope objective lens. The bright-field illumination pattern may be formed by simultaneously illuminating the sample with a subset of multiple light sources 261. A bright-field illumination pattern may be formed by simultaneously illuminating the sample with most (or all) of the multiple light sources 261. Furthermore, the microscope system 20 may include light sources (not shown in Figure 2) used in conventional microscopy (i.e., bright-field microscopy). Such light sources can be used to create a bright-field illumination pattern.

[0097] It should be understood that the numerical aperture and magnification of the first and / or second microscope objective lenses are merely examples and may be selected depending on the type of sample 292, for example. For example, the numerical aperture of the first microscope objective lens may have a magnification of 10x and / or a numerical aperture of 0.25. Preferably, the numerical aperture of the second microscope objective lens may be 2 to 3 times greater than that of the first microscope objective lens. This is just an example, and it should be understood that the numerical aperture of the second microscope objective lens may be greater than 2 to 3 times that of the first microscope objective lens. This may be advantageous in that a machine learning model can construct a focused digital image of the sample that corresponds to an image of the sample taken with a numerical aperture 2 to 3 times (or more) the numerical aperture from which the input image was taken. It should be understood that the microscope system 20 may include further optics that can be used together with at least one microscope objective lens 280 to image the sample 292 onto the image sensor 270. For example, the microscope system may include at least one relay lens 285, arranged so that a sample 292 is imaged by an image sensor 270, as shown in the example in Figure 2. It should be further understood that the at least one relay lens 285 may be selected depending on the magnification and / or numerical aperture of the at least one microscope objective lens 280 (e.g., focal length, material, size, etc.). Thus, each microscope objective lens 280 may have a corresponding relay lens among the at least one relay lens. The at least one microscope objective lens 280 may be movable in the longitudinal direction Z by being coupled to a manual and / or motorized stage. The longitudinal direction Z may be parallel to the optical axis of the microscope system 20. In other words, the at least one microscope objective lens 280 may be movable in the focusing direction of the microscope system 20. Therefore, different focal positions of the sample 292 (or training sample) relative to the microscope objective lens 280 may be set by moving the microscope objective lens 280 along the longitudinal direction Z.Alternatively, or in addition, the sample holder 290 may be movable along the longitudinal direction Z. The position of at least one microscope objective lens 280 along the longitudinal direction Z may be controlled by the circuit 200. For example, the circuit 200 may be configured to perform a focusing function (not shown in Figure 2) configured to adjust the position of at least one microscope objective lens 280 along the longitudinal direction Z. The focusing function may be configured to automatically adjust the position of at least one microscope objective lens 280 along the longitudinal direction Z.

[0098] The circuit 200 is configured to perform an acquisition function 2100 and an image reconstruction function 2102.

[0099] The acquisition function 2100 is configured to acquire a set of digital image inputs. The acquisition function 2100 is configured to position the sample 292 at the focal point relative to the microscope objective lens 280.

[0100] The acquisition function 2100 is further configured to control the illumination system 260 so that it illuminates the sample 292 with each of the multiple illumination patterns 262.

[0101] The acquisition function 2100 is further configured to control the image sensor 270 to capture a digital image of the sample 292 for each of the multiple illumination patterns 262.

[0102] As mentioned above, each light source may be configured to illuminate the sample from one of several directions. At least one of the multiple directions can correspond to an angle greater than the numerical aperture 282 of the microscope objective lens 280. For example, direction 2620 in Figure 2 can correspond to an angle greater than the numerical aperture 282 of at least one microscope objective lens 280. Light entering at least one microscope objective lens 280 from direction 2620 without scattering can be prevented from propagating through the microscope objective lens 280 to the image sensor 270 (i.e., the angle of incidence of light from this direction can be outside the numerical aperture 282 of the microscope objective lens 280). Therefore, light from this direction may need to be scattered by the sample 292 in order to propagate through the microscope objective lens 280 to the image sensor 270.

[0103] The image construction function 2102 is configured to input a set of digital images into a machine learning model that has been trained as described in relation to Figures 1 and 3.

[0104] The image reconstruction function 2102 is further configured to receive an output from a machine learning model that includes a constructed digital image depicting the focused sample. In other words, the image reconstruction function 2102 can be configured to construct a digital image depicting the focused sample by inputting a set of digital image inputs into a trained machine learning model (e.g., trained in the manner described in relation to Figures 1 and 3) and receiving an output from the trained machine learning model that includes a digital image depicting the focused sample. Thus, the process of imaging sample 292 may be more efficient, as a machine learning model trained with relatively low-resolution digital images of sample 292 outputs a relatively high-resolution digital image depicting the focused sample.

[0105] The construction of a digital image depicting a focused sample is described in relation to the microscope system in Figure 2, but it should be understood that the construction of the digital image may also be implemented on a computing device. Therefore, the computing device can be configured to receive a set of digital image inputs and to input the received set of digital image inputs into a machine learning model trained according to the above. The machine learning model may be stored and run on the computing device. The set of digital image inputs may be captured, for example, using the microscope system in Figure 2 and sent to the computing device (i.e., the computing device may be configured to receive the set of digital image inputs from the microscope system).

[0106] Figure 3 is a block scheme diagram of Method 30 for training a machine learning model to construct a digital image 702 depicting a focused sample. In other words, the constructed digital image 702 depicts a focused sample. Method 30 may also be a computer implementation method. Various steps are described in more detail below. Although illustrated in a specific order, one or more steps of Method 30 may be performed in parallel, partially in parallel, or multiple times in any suitable order.

[0107] A training set of digital images of the training sample is acquired in S300. The training set of digital images of the training sample is acquired in S302 by positioning the training sample at a focal position outside the range between the near and far limits of the depth of field of the microscope objective lens, illuminating the training sample with multiple illumination patterns using an illumination system in S304, and capturing a digital image of the training sample for each of the multiple illumination patterns in S306. Positioning the training sample in S302 can be rephrased as being at a focal position far from the optimal focal position, rather than at the depth of field (or half the depth of field) of the microscope objective lens.

[0108] S308 receives ground truth, which includes a digital image of a focused training sample. In other words, ground truth may be obtained when the training sample is positioned within a range of positions between the near and far limits of the depth of field of the microscope objective lens used. The resolution of the digital image of the focused training sample can be relatively higher than the resolution of at least one digital image in the training set of digital images.

[0109] The machine learning model is trained to construct a digital image depicting a focused sample using a training set of digital images and ground truth (S310).

[0110] The focal position may be one of a plurality of focal positions. Acquiring a training set of digital images of the training sample S300 may further include positioning the training sample at each of the plurality of focal positions relative to the microscope objective lens S302. The act of illuminating the training sample S304 and the act of capturing a digital image of the training sample S306 can be performed for each of the plurality of focal positions. Each of the plurality of focal positions may be outside the range of positions between the near and far limits of the depth of field of the microscope objective lens (i.e., outside the depth of field). Alternatively, at least one of the plurality of focal positions may be within the range of positions between the near and far limits of the depth of field of the microscope objective lens. In other words, at least one digital image of the training set of digital images may capture the training sample at a position closer to the optimal focal position than the depth of field. Most of the plurality of focal positions may be outside the depth of field of the microscope objective lens.

[0111] The S308 action of receiving ground truth may include positioning the training sample at a focal position relative to the microscope objective lens within the range of positions between the near and far limits of the depth of field of the microscope objective lens (S312), illuminating the training sample with a bright-field illumination pattern (S314), and using the microscope objective lens to capture a digital image of the focused training sample while it is illuminated (S316). In other words, ground truth can be acquired using a conventional bright-field microscope. The bright-field illumination pattern can be realized by a conventional light source. Alternatively, the bright-field illumination pattern may be realized by most of the multiple light sources of the illumination system (described later) emitting light simultaneously.

[0112] The illumination system may consist of multiple light sources. Each of the multiple light sources may be configured to illuminate the training sample from one of several directions. Each illumination pattern may be formed by one or more of the multiple light sources. The illumination system may be one of the illumination systems described above in relation to Figures 2 and 5.

[0113] At least one of the multiple directions can correspond to an angle greater than the numerical aperture of the microscope objective lens used to acquire the training set of digital images.

[0114] Figure 4 is a block scheme diagram of method 40 for constructing a digital image 702 depicting a focused sample. In other words, the constructed digital image depicts the focused sample. Method 40 may also be a computer implementation method. Various steps are described in more detail below. Although illustrated in a specific order, one or more steps of method 40 may be performed in parallel, partially in parallel, or multiple times in any suitable order.

[0115] S400 receives an input set of digital images of the sample. The input set of digital images can be obtained by positioning the sample at the focal point relative to the microscope objective lens, illuminating the sample with multiple illumination patterns using an illumination system, and capturing a digital image of the sample for each of the multiple illumination patterns. The multiple illumination patterns used to obtain the input set of digital images may be the same as the multiple illumination patterns used to obtain the training set of digital images when training a machine learning model (i.e., the method described above in relation to Figure 3).

[0116] S402 constructs a digital image of the focused sample. The digital image of the focused sample is constructed by inputting a set of digital image inputs into a machine learning model trained according to method 30 described above in relation to Figure 3, S404, and receiving an output from the machine learning model, S406, which includes the constructed digital image of the focused sample. The act of receiving the output S406 may include obtaining the output by processing the input set of input images with the machine learning model.

[0117] The resolution of the constructed digital image can be made relatively higher than the resolution of at least one digital image in the input set of digital images. As mentioned above, this can be achieved by using digital images of the sample illuminated by multiple lighting patterns.

[0118] The act of receiving an input set of digital images of a sample S400 may include acquiring an input set of digital images of a sample by positioning the sample at a focal position outside the depth of field of the microscope objective lens S408-1, illuminating the sample with multiple illumination patterns using an illumination system S408-2, and capturing a digital image of the sample for each of the multiple illumination patterns S408-3.

[0119] The illumination system may include multiple light sources. Each of the multiple light sources may be configured to illuminate the sample from one of several directions. Each of the multiple illumination patterns may be formed by one or more of the multiple light sources.

[0120] At least one of the multiple directions can correspond to an angle greater than the numerical aperture of the microscope objective lens used to acquire the input set of digital images.

[0121] Figure 6 shows a non-temporary computer-readable storage medium 60. The non-temporary computer-readable storage medium 60 includes a program code portion that, when executed on a device having processing capabilities, performs method 40 as described in relation to Figure 4.

[0122] Figures 7A and 7B are diagrams of digital images of two different samples. In this example, the samples are pathological specimens.

[0123] Figure 7A shows a digital image 700 of a sample captured using a microscope objective lens with 10x magnification. As seen in Figure 7A, the digital image 700 is captured out of focus. In particular, the digital image 700 is captured at a distance of 15 microns from the optimal focal position. As shown herein, the out-of-focus digital image 700 may represent a digital image of a training sample captured under the same conditions (excluding the illumination pattern) as the training set of digital images used to train the machine learning model described above. The out-of-focus digital image 700 of a sample, as shown herein, may also represent a digital image of a sample captured under the same conditions (excluding the illumination pattern) as the input set of digital images fed into a machine learning model trained to construct a digital image depicting a focused sample.

[0124] Figure 7B shows a digital image 702 of the same sample as in Figure 7A, but constructed by a machine learning model from a digital image captured under similar conditions to the digital image 700 illustrated in Figure 7A. In other words, digital image 702 is an example of output from a machine learning model trained to construct a digital image depicting a focused sample. Comparing the constructed digital image 702 with the digital image 700 in Figure 7A, it is clear that the constructed digital image 702 is better in focus, facilitating improved analysis of the depicted sample. Thus, a machine learning model can construct the focused digital image 702 in Figure 7B using a digital image similar to the digital image 700 in Figure 7A.

[0125] Figure 8 illustrates, as an example, how the focal position relates to the depth of field of the microscope objective lens 800, as well as to the near and far limits of the depth of field. Illustrated here is a sample 802 placed on a sample holder 804. The sample 802 is positioned at the focal point relative to the microscope objective lens 800, indicated by line L-4. The position of the sample 802 can be defined as the central plane of the sample holder 804, as shown here. Alternatively, the position of the sample 802 may be defined as a plane of the sample, such as the top, bottom, or central plane of the sample. The position of the sample can be defined along the optical axis AO of the microscope objective lens 800. Therefore, the focal position of the sample relative to the microscope objective lens 800 can be adjusted by moving the sample holder 804 and / or the microscope objective lens 800 along the optical axis so that the distance D3 between the microscope objective lens (indicated by L-1) and the sample 802 increases or decreases. The position L-1 of the microscope objective lens can be defined as the surface of the microscope objective lens 800 facing the sample 802 (i.e., the surface closest to the sample 802). Those skilled in the art will notice other suitable ways in which the position of the microscope objective lens 800 can be defined.

[0126] The depth of field (indicated by the interval I-1) of the microscope objective lens 800 is defined as the distance between the furthest and closest points that constitute an acceptable focal point. In other words, this can be identified as the zone or interval of acceptable sharpness in the image produced by the microscope objective lens 800. Thus, the depth of field I-1 can be defined by the range of positions between the near limit (indicated by line L-2) and the far limit (indicated by line L-5). The near limit L-2 may be considered as the minimum distance D4 between the microscope objective lens 800 and the position where the object being imaged is considered to be acceptablely sharp. Correspondingly, the far limit L-5 can be considered as the maximum distance D2 between the microscope objective lens 800 and the position where the object being imaged is considered to be acceptablely sharp.

[0127] Between these two positions (i.e., between the near limit L-2 and the far limit L-5), an optimal focal position (indicated by line L-3) can be defined. An object imaged at a distance D3 from the microscope objective lens (i.e., at the optimal focal position L-3) can be in the best possible focus.

[0128] In the illustrated example, the focal position L-4 where sample 802 is positioned is within the depth of field I-1 of the microscope objective lens 800. The captured digital image of sample 802 at this position can represent ground truth. However, as described above, the training set of digital images or the input set of digital images of sample 802 can be captured at one or more focal positions outside and / or inside the depth of field I-1 of the microscope objective lens 800, as illustrated by the dashed sample and sample holder.

[0129] Those skilled in the art will be aware of machine learning, particularly how machine learning models can be trained and / or how trained machine learning models can be used. However, to put it simply, a machine learning model may be a type of supervised machine learning model, such as a network like U-net or Pix2pix. A machine learning model may be a transformer-based network like SwinIR. A machine learning model may be a convolutional neural network. A machine learning model can be trained to predict a desired output using exemplary input training data and ground truth, i.e., the "correct" or "true" output. In other words, ground truth can be used as a label for the input training data. The input training data may contain data related to different outcomes, and each input training data may be associated with the ground truth associated with that particular input training data. Thus, each input training data can be labeled with the associated ground truth (i.e., the "correct" or "true" output). Different types of loss functions may be used to evaluate how well the predicted output compares to the ground truth. A machine learning model can be trained to decrease (or, depending on the loss function, increase) the value of a loss function. In other words, training a machine learning model can be thought of as minimizing (or maximizing) a loss function. The loss function may be a pixel-level loss function, i.e., a loss function that compares the output to ground truth pixel by pixel. Alternatively, the loss function may be a non-pixel-level loss function, i.e., a loss function that views the image as a whole.

[0130] A machine learning model can contain multiple layers of neurons, each neuron representing a mathematical operation applied to the input training data. Typically, a machine learning model includes an input layer, one or more hidden layers, and an output layer. The first layer is sometimes referred to as the input layer. In a machine learning model, the output of each layer (except the output layer) can be fed to subsequent layers, which produce new outputs. These new outputs may be fed to even more subsequent layers. The output of a machine learning model may also be the output of the output layer. This process can be repeated for all layers of a machine learning model. Typically, each layer further includes an activation function. The activation function can further define the output of the neurons in the layer. For example, the activation function can ensure that the output from the layer does not become too large or too small (e.g., tend toward positive or negative infinity). Furthermore, the activation function can introduce nonlinearity into the machine learning model. During the training process, the weights and / or biases associated with the neurons in the layer can be adjusted until the machine learning model produces predictions for the input training data that reflect ground truth. Each neuron can be configured to multiply the input to the neuron by a weight associated with that neuron. Each neuron can further be configured to add a bias associated with that neuron to the input. In other words, the output from a neuron is the sum of the bias associated with the neuron and the product of the weights associated with the neuron and the input. The weights and biases can be adjusted in a recursive and / or iterative process, which is known in the art as backpropagation. A convolutional neural network is a type of neural network that includes one or more layers that represent convolutional operations. In this context, the input training data includes digital images. Digital images can be represented as matrices (or arrays), where each element of the matrix (or array) can represent a corresponding pixel in the digital image. Thus, the value of an element can represent the pixel value of the corresponding pixel in the digital image.Therefore, the inputs and outputs to a machine learning model may be numerical representations (e.g., matrices or arrays) of digital images. In this context, the input is a set of digital images (i.e., a training set or input set). Thus, the input to a machine learning model may be multiple matrices or a three-dimensional matrix. That is, it should be understood that a machine learning model can receive additional inputs during training. One example of such input may be the type of sample being imaged. Such inputs can then be used with the trained machine learning model to construct a digital image depicting the focused sample. Furthermore, in this context, the output is a digital image. Thus, the output of a machine learning model may be a matrix representing the constructed digital image depicting the focused sample.

[0131] Those skilled in the art will understand that the concept of the present invention is by no means limited to the preferred modifications described above. Rather, many modifications and variations are possible within the scope of the appended claims.

[0132] In addition, variations in the disclosed modifications can be understood and implemented by those skilled in the art when practicing the claimed invention, based on an examination of the drawings, disclosures, and the attached claims.

Claims

1. A method (30) for training a machine learning model to construct a digital image depicting a focused sample, This involves obtaining a training set of digital images of training samples (S300), The training sample is positioned at a focal point outside the range between the near limit and far limit of the depth of field of the microscope objective lens (S302). The lighting system illuminates the training sample with multiple lighting patterns (S304), and For each of the multiple lighting patterns, a digital image of the training sample is captured (S306). This involves obtaining a training set of digital images of the training sample (S300), Receiving ground truth including a digital image of the focused training sample (S308), Using the training set of digital images and the ground truth, the machine learning model is trained to construct the digital image depicting the focused sample (S310). Method (30), including the method (30).

2. The method according to claim 1 (30), wherein the resolution of the digital image depicting the focused training sample is relatively higher than the resolution of at least one digital image in the training set of digital images.

3. The aforementioned focal position is one of a plurality of focal positions, and obtaining a training set of the digital images of the training sample (S300) is Position the training sample at each of the multiple focal positions relative to the microscope objective lens (S302) It further includes, The method according to claim 1 or 2 (30), wherein the act of illuminating the training sample (S304) and the act of capturing a digital image of the training sample (S306) are performed for each of the multiple focal positions.

4. The method according to claim 3 (30), wherein at least one of the plurality of focal positions is within the range of the position between the near limit and the far limit of the depth of field of the microscope objective lens.

5. The act of receiving the aforementioned ground truth (S308) Position the training sample with respect to the microscope objective lens at a focal position within the range of the position between the near limit and the far limit of the depth of field of the microscope objective lens (S312). Illuminating the training sample with a bright-field illumination pattern (S314), and Using the microscope objective lens, capture the digital image of the focused training sample while the training sample is illuminated with the bright-field illumination pattern (S316). The method according to any one of claims 1 to 4 (30).

6. The method according to any one of claims 1 to 5 (30), wherein the illumination system comprises a plurality of light sources, each of the plurality of light sources is configured to illuminate the training sample from one of a plurality of directions, and each of the plurality of illumination patterns is formed by one or more of the plurality of light sources.

7. The method according to claim 6 (30), wherein at least one of the plurality of directions corresponds to an angle greater than the numerical aperture of the microscope objective lens used to acquire the training set of digital images.

8. A method (40) for constructing a digital image depicting a focused sample, The process involves receiving an input set of digital images of the sample (S400), which is obtained by positioning the sample at the focal position relative to the microscope objective lens, illuminating the sample with multiple illumination patterns using an illumination system, and capturing a digital image of the sample for each of the multiple illumination patterns. This involves constructing a digital image depicting the focused sample (S402). Inputting the aforementioned set of digital images into a machine learning model trained according to the method described in any one of claims 1 to 7 (30) (S404), and The machine learning model receives an output including the constructed digital image depicting the focused sample (S406). By doing so, the digital image depicting the focused sample is constructed (S402). Method (40), including the following.

9. The method according to claim 8 (40), wherein the illumination system comprises a plurality of light sources, each of the plurality of light sources is configured to illuminate the sample from one of a plurality of directions, and each of the plurality of illumination patterns is formed by one or more of the plurality of light sources.

10. The method according to claim 9 (40), wherein at least one of the plurality of directions corresponds to an angle greater than the numerical aperture of the microscope objective lens used to acquire the input set of digital images.

11. A device (10) for training a machine learning model to construct a digital image depicting a focused sample, A first receiving function (1100) configured to receive a training set of digital images of a training sample, wherein the training set of digital images of the training sample is acquired by positioning the training sample at a focal position outside the range between the near and far limits of the depth of field of the microscope objective lens, illuminating the training sample with a plurality of illumination patterns using an illumination system, and capturing a digital image of the training sample for each of the plurality of illumination patterns. The circuit (100) is configured to perform the following: The aforementioned circuit is A second receiving function (1102) is configured to receive ground truth including a digital image of the focused training sample, A training function (1104) is configured to train the machine learning model to construct the digital image depicting a focused sample using the training set of digital images and the ground truth. A device (10) further configured to perform the following actions.

12. A microscope system (20), A lighting system (260) configured to illuminate a sample (292) with multiple lighting patterns (262), An image sensor (270) configured to capture a digital image of the sample (292), A microscope objective lens (280) configured to image the sample (292) onto the image sensor (270), Circuit (200), The acquisition function (2100) Position the sample (292) at the focal point relative to the microscope objective lens (280). Controlling the illumination system (260) to illuminate the sample (292) with each of the plurality of illumination patterns (262), and The image sensor (270) is controlled to capture a digital image of the sample (292) for each of the plurality of illumination patterns (262). An acquisition function (2100) is configured to acquire a set of digital image inputs by performing the following: A circuit (200) configured to perform the following: Equipped with, The circuit (200) is Image reconstruction function (2102), Inputting the aforementioned set of digital images into a machine learning model trained according to the method (30) described in any one of claims 1 to 7, and The machine learning model receives an output that includes the constructed digital image depicting the focused sample. Image construction function (2102) configured to perform A microscope system (20) is further configured to perform the following actions.

13. The microscope system (20) according to claim 12, wherein the illumination system (260) comprises a plurality of light sources (261), each of the plurality of light sources (261) is configured to illuminate the sample (292) from one of a plurality of directions, and each of the plurality of illumination patterns (262) is formed by one or more of the plurality of light sources (261).

14. The microscope system (20) according to claim 13, wherein at least one of the plurality of directions corresponds to an angle greater than the numerical aperture of the microscope objective lens (280) used to image the sample (292).

15. The microscope system (20) according to claim 13 or 14, wherein the plurality of light sources (261) are arranged on a curved surface (264) that is concave along at least one direction along the surface.

16. A non-temporary computer-readable storage medium comprising a program code portion that, when executed on a device having processing capabilities, performs the method (40) according to any one of claims 8 to 10.

Images