Construction of digital image depicting sample

Abstract

The inventive concept relates to a method and apparatus for training a machine learning model to construct a digital image depicting a sample. The method comprises: receiving (S300) a digital image training set of a training sample, where the digital image training set is acquired (S310) using a first imaging system (100) by illuminating (S312) the training sample with a plurality of illumination modes and capturing (S314) a digital image for each of the plurality of illumination modes; receiving (S320) real data comprising a digital image depicting the training sample, where the digital image of the real data is acquired (S330) using a second imaging system (200) exhibiting a lower imaging error level than the first imaging system (100); and training (S340) the machine learning model using the received training set of digital images and the received real data to construct a digital image depicting the sample. The inventive concept further relates to a method and a microscope device for constructing a digital image depicting a sample.

Description

Technical Field

[0001] The inventive concept relates to the construction of a digital image depicting a sample. Background Art

[0002] In the field of digital microscopy, a typical task is to find and identify objects within a sample. For example, in hematology, cytology, and pathology, specific cell types can be found and identified in order to establish a diagnosis for the patient from whom the sample was taken.

[0003] There are different techniques for imaging samples. One of the simplest forms of digital microscopy is brightfield microscopy, in which the sample is illuminated from below with white light and imaged from above. Typically, the images of the sample are of very high quality, meaning they closely replicate the sample. However, to produce such high-quality sample images, the microscope system used must typically be complex. For example, optical and digital errors in imaging must be reduced in order to capture high-quality images. This is typically achieved by using very complex optical components (i.e., complex microscope objectives) and high-performance image sensors in the microscope system. However, a disadvantage of such microscope systems is that the increased complexity and performance typically results in higher economic costs. Another disadvantage is that complex microscope systems are typically more sensitive to misalignment of the microscope system. For example, if there is relative misalignment between the microscope objective and other optical components (e.g., relay lenses), the image sensor, and / or the sample being imaged, the quality of the captured sample image may be lower. This, in turn, increases the risk of patient misdiagnosis. Summary of the Invention

[0004] In view of the foregoing, an object of the present inventive concept is to provide a method and apparatus for training a machine learning model to construct a digital image depicting a sample.

[0005] Another object is to provide a method and a microscope apparatus for constructing a digital image depicting a sample.

[0006] Another object is to at least partially mitigate, alleviate or eliminate one or more of the above mentioned deficiencies in the art.

[0007] According to a first aspect, a method for training a machine learning model to construct a digital image depicting a sample is provided. The digital image depicting the sample may be a digital microscopic image. The method comprises: receiving a training set of digital images of the training sample, wherein the training set of digital images was acquired using a first imaging system by illuminating the training sample with a plurality of illumination modes and capturing a digital image for each of the plurality of illumination modes; receiving real data comprising a digital image depicting the training sample, wherein the digital image of the real data was acquired using a second imaging system that exhibits a lower level of imaging error than the first imaging system; and training a machine learning model using the received training set of digital images and the received real data to construct a digital image depicting the sample.

[0008] In the context of the present disclosure, the wording "training" in "training samples" should be interpreted as samples used during the training of the machine learning model, rather than general samples of which the trained machine learning model is able to construct digital images. The constructed digital images may then depict the general samples. The machine learning model may certainly be able to construct digital images that depict the training samples. However, the machine learning model may also be able to construct digital images of other samples, which may not be part of the training set of digital images. In other words, the machine learning model may be able to construct digital images of samples after training that were not used to train the machine learning model. Depending on the context, the use of "samples" alone may be used herein to refer to samples used during training or samples that are input into the trained machine learning model.

[0009] In the context of the present disclosure, the term "illumination mode" can be interpreted as different ways of illuminating a sample using an illumination system. Each illumination mode can be formed, for example, by illuminating the sample simultaneously from one or more of a plurality of directions and / or by varying the number of light sources emitted in the illumination system.

[0010] In the context of this disclosure, the term "real data" should be interpreted as information that is known to be true and / or factual. Thus, in this context, since a machine learning model is trained to construct digital images using real data and a training set of digital images, the real data may refer to digital images depicting training samples. Such digital images depicting training samples may be acquired or captured using conventional techniques.

[0011] The machine learning model can be trained to associate a digital image training set with real data (e.g., a digital image depicting a training sample). The machine learning model can be trained iteratively and / or recursively until the difference between the output of the machine learning model (i.e., the constructed digital image) and the real data (i.e., the digital image depicting the training sample) is less than a predetermined threshold. A smaller difference between the output of the machine learning model and the real data can indicate that the accuracy of the constructed digital image provided by the machine learning model is higher. In other words, a smaller difference between the output of the machine learning model and the real data can indicate that the constructed digital image can reproduce the digital image depicting the training sample to a higher degree. Therefore, preferably, the difference between the output of the machine learning model and the real data can be minimized. The machine learning model can be trained to construct digital images of samples for a variety of different sample types. In this case, for each sample type, the machine learning model can be trained using a digital image training set of training samples of that sample type and the corresponding real data associated with the corresponding sample type.

[0012] By illuminating a training sample with multiple illumination modes and capturing digital images for each of the multiple illumination modes, phase information associated with the training sample (often referred to in the art as quantitative phase) can be determined. This can be understood as capturing information about different portions of Fourier space (i.e., the spatial frequency domain) associated with the training sample for different illumination directions. This technique may be referred to in the art as Fourier phantom microscopy (FPM). This phase information can then be used to reproduce the training sample at different focal positions (i.e., the position of the training sample relative to the image-forming components of the first imaging system used to capture the digital image of the training sample). For example, even if a training set of digital images is acquired using a first imaging system that exhibits a relatively large degree of imaging error (compared to a second imaging system), the training set of digital images still includes phase information that can be used to construct a digital image depicting the training sample that is similar to the digital image captured using the second imaging system. This is because the training sample was illuminated with multiple illumination modes when acquiring the training set. Furthermore, information about the refractive index (or spatial distribution of the refractive index) associated with the training sample can be captured. This can be understood as the refractive effect of light depending on the angle of incidence of the light illuminating the training sample and the refractive index of the training sample. Furthermore, by illuminating the training sample with multiple illumination modes and capturing a digital image for each of the multiple illumination modes, information about finer details of the training sample can be captured than can typically be resolved using, for example, a conventional microscope used to image the training sample.

[0013] Thus, because the digital image training set includes information associated with one or more of: fine details of the training sample, a refractive index associated with the training sample, and phase information associated with the training sample, this information can be used to train a machine learning model, which in turn can allow the machine learning model to be trained to more accurately construct digital images depicting the training sample than would be permitted if the digital image training set were captured using a first imaging system and using a single illumination mode (e.g., conventional brightfield illumination, conventional darkfield illumination, or from a single direction) or by using conventional microscopy. It may be difficult or even impossible to capture information associated with: a refractive index associated with the training sample and / or phase information associated with the training sample using conventional microscopy. In other words, because digital images captured using conventional microscopy may simultaneously contain information about the refraction of light incident from directions corresponding to overlapping portions of Fourier space associated with the training sample, it may not be possible to determine phase information associated with the training sample and / or information related to the refractive index of the training sample using such techniques. In other words, it may not be possible to determine information related to the training sample using conventional microscopy. Illuminating the training sample with multiple illumination modes can further allow for the capture of information related to details of the training sample that are finer than would typically be permitted by the first imaging system used to capture the training set of digital images of the training sample (using conventional microscopy illumination). Thus, an imaging system with a relatively low magnification can be used while still being able to capture information related to such fine details of the training sample. Using an imaging system with a relatively low magnification can, in turn, allow for the capture of digital images of a larger portion of the training sample at each imaging position. Thus, the entire training sample can be scanned by capturing digital images at relatively few positions, which can, in turn, allow for faster and / or more efficient scanning of the training sample.

[0014] Another related advantage is that because the level of optical aberrations in the second imaging system (i.e., in the digital images of the real data) is lower than in the first imaging system (i.e., in the digital images of the digital image training set), the machine learning model can be trained to reduce (or preferably eliminate) such optical aberrations in the output (i.e., the constructed digital images).

[0015] Conventional techniques used to reconstruct images using FPM typically require that the refractive index variation near the imaged sample (e.g., the training sample) is not too high, otherwise the reconstruction of a digital image of such a sample will fail to converge. For example, this can be a problem for samples with structured (e.g., rough) surfaces. The refractive index of such a sample can vary significantly along its surface. For example, due to surface roughness, there may be many interfaces between the sample and the air in a plane close to the sample surface, resulting in large variations in the refractive index along that plane. To address these issues, immersion oil and / or a coverslip are typically applied to the sample, which can reduce the degree of structuring of the sample surface. Furthermore, the refractive index of the immersion oil can typically be more similar to the refractive index of the sample than to the refractive index of air. Advantageously, it has been found that by using a machine learning model trained according to the first aspect, the requirements regarding the refractive index variation can be less stringent, thereby reducing (or even eliminating) the need for immersion oil and / or a coverslip.

[0016] The image forming components of the second imaging system can exhibit a lower level of optical aberration than the image forming components of the first imaging system, whereby the image of the object in best focus formed by the image forming components of the second imaging system can be more similar to the imaged object than the image of the object in best focus formed by the image forming components of the first imaging system.

[0017] A related advantage is that the image-forming assembly of the first imaging system can be simpler (eg, have fewer parts) than the image-forming assembly of the second imaging system. This, in turn, can reduce the economic costs associated with the first imaging system.

[0018] Another related advantage is that the image forming assembly of the first imaging system can include lower quality components than the image forming assembly of the second imaging system. This, in turn, can reduce the economic costs associated with the first imaging system.

[0019] Thus, a trained machine learning model can allow the use of a simpler and / or less expensive imaging system while still being able to construct a digital image depicting the sample that is similar to a digital image depicting the sample captured using a more complex imaging system and / or a higher quality imaging system (e.g., a second imaging system).

[0020] The number of imaging elements of the image-forming assembly of the first imaging system can be lower than the number of imaging elements of the image-forming assembly of the second imaging system, and thus the image-forming assembly of the second imaging system can have a lower level of optical aberration than the image-forming assembly of the first imaging system. In other words, the image-forming assembly of the second imaging system can exhibit a lower level of optical aberration than the image-forming assembly of the first imaging system.

[0021] A related advantage is that the image-forming components of the first imaging system can be simpler (e.g., have fewer parts) than the image-forming components of the second imaging system. This, in turn, can reduce the economic costs associated with the first imaging system. Thus, the trained machine learning model can allow the use of a simpler and / or less expensive imaging system while still being able to construct a digital image depicting the sample that is similar to a digital image depicting the sample captured using a more complex and / or more expensive imaging system (e.g., the second imaging system).

[0022] The image forming component of the second imaging system may include a microscope objective.

[0023] A related advantage is that the quality and / or resolution of digital images of real data can be improved. Typically, microscope objectives are configured to both produce magnified images of an object and to reduce (preferably minimize) imaging errors. Therefore, digital images captured using microscope objectives can allow trained machine learning models to construct digital images with higher quality and / or resolution.

[0024] The image forming assembly of the first imaging system may include only one imaging element.

[0025] A related advantage is that the image-forming components of the first imaging system can be simpler, i.e., include only one imaging element. This, in turn, can reduce the economic costs associated with the first imaging system. Thus, the trained machine learning model can allow the use of a simpler and / or less expensive imaging system while still being able to construct a digital image depicting the sample that is similar to a digital image depicting the sample captured using a more complex and / or more expensive imaging system (e.g., the second imaging system).

[0026] Optical aberrations may include one or more of: spherical aberration; astigmatism; coma; image distortion; chromatic aberration; and Petzval field curvature.

[0027] The image sensor of the second imaging system may exhibit one or more of: a greater number of pixels; a higher pixel resolution; a larger pixel size; a higher dynamic range; and a lower noise level compared to the image sensor of the first imaging system.

[0028] A related advantage is that the image sensor of the first imaging system can be less expensive than the image sensor of the second imaging system. Thus, the trained machine learning model can allow the use of the less expensive imaging system while still being able to construct a digital image depicting the sample that is similar to a digital image depicting the sample captured using a more expensive imaging system (e.g., the second imaging system).

[0029] The first imaging system may include a sample positioning assembly configured to move the sample along a first plane, and the second imaging system may include a sample positioning assembly configured to move the sample along a second plane, the second plane may exhibit a higher degree of flatness than the first plane. The second plane may be substantially parallel to the first plane. In other words, when the sample moves, the accuracy and / or repeatability of the sample focus position of the second imaging system (i.e., the position of the sample relative to the image forming assembly of the imaging system) may be higher than that of the first imaging system. Therefore, the second imaging system can position the sample to be in focus with higher accuracy and / or better repeatability than the first imaging system. This is advantageous because the sample positioning assembly of the first imaging system can be simpler and / or cheaper than the sample positioning assembly of the second imaging system. Therefore, the trained machine learning model can allow the use of a simpler and / or cheaper imaging system while still being able to construct a digital image depicting the sample that is similar to a digital image depicting the sample captured using a more expensive imaging system (e.g., the second imaging system).

[0030] The first imaging system may include an illumination subsystem. The illumination subsystem may include a plurality of light sources, each of the plurality of light sources may be configured to illuminate the training sample from one of a plurality of directions; and each of the plurality of illumination patterns may be formed by one or more of the plurality of light sources.

[0031] A related advantage is that the illumination subsystem of the first imaging system can switch between different illumination modes more quickly and / or more reliably than an illumination subsystem including a single movable light source capable of illuminating a location from different directions.

[0032] Another related advantage is that digital images of the training sample can be captured under different lighting conditions, so that more information of the training sample (eg, information associated with the refractive index, etc.) can be captured in the captured digital images.

[0033] At least one of the plurality of directions may correspond to an angle greater than a numerical aperture of an image forming component of the first imaging system.

[0034] In the context of the present disclosure, the term "numerical aperture of an image-forming assembly" should be interpreted as a dimensionless number associated with the angular range over which the image-forming assembly accepts light. Thus, a direction corresponding to an angle greater than the numerical aperture of the image-forming assembly may be a direction corresponding to an angle outside (e.g., greater than) the angular range over which the image-forming assembly accepts light.

[0035] By illuminating the training sample from a direction corresponding to an angle greater than the numerical aperture of the image-forming component of the first imaging system, a digital image captured for that illumination angle can include information about higher spatial frequencies of the training sample, and thereby include finer details of the training sample than would normally be permitted by the image-forming component of the first imaging system (e.g., using conventional illumination). This, in turn, can allow the image-forming component of the first imaging system to capture phase information associated with the training sample and / or information relating to details of the training sample that would normally be indistinguishable from the image-forming component of the first imaging system, which can be used to train a machine learning model. In other words, illuminating the training sample from a direction corresponding to an angle greater than the numerical aperture of the image-forming component of the first imaging system can allow for an improved machine learning model capable of constructing a digital image depicting the sample.

[0036] Two or more of the plurality of directions may correspond to overlapping portions of Fourier space associated with the training samples, and wherein each of the plurality of illumination patterns may be formed by illuminating the training samples from one or more directions corresponding to non-overlapping portions of Fourier space associated with the training samples.

[0037] A related advantage is that the training sample can be illuminated from several directions simultaneously, thereby increasing the amount of information associated with the training sample captured in each captured digital image. This, in turn, can reduce the number of digital images required while still allowing sufficient information associated with the training sample (or samples) to be captured (i.e., information sufficient for the training process and / or the process of constructing a digital image using the trained machine learning model). Reducing the number of digital images that need to be captured can reduce the time required for image capture, which can make the system for constructing digital images depicting the sample more time-efficient. Further, reducing the number of captured digital images can also reduce the memory requirements associated with the training process (e.g., the memory required to store the captured digital images). For the same reason, the memory requirements associated with the process of constructing a digital image depicting the sample using the trained machine learning model can also be reduced. Furthermore, by simultaneously illuminating the training samples from directions corresponding to non-overlapping portions of Fourier space associated with the training samples, different portions of Fourier space can be sampled simultaneously. This, in turn, can reduce (or even avoid) the entanglement (mixing) of information (i.e., information associated with the training samples) from different directions.

[0038] According to a second aspect, a method for constructing a digital image depicting a sample is provided. The method comprises: receiving an input set of digital images of the sample, wherein the input set of digital images is acquired using an imaging system by illuminating the sample with a plurality of illumination modes and capturing a digital image for each of the plurality of illumination modes; inputting the input set of digital images into a machine learning model trained according to the method of the first aspect; and receiving an output from the machine learning model comprising a digital image depicting the sample.

[0039] By using a machine learning model trained according to the method of the first aspect, a digital image depicting the sample can be constructed that is of a higher quality than would typically be achievable by the imaging system used to acquire the input set of digital images (e.g., using conventional lighting). This, in turn, can allow the imaging system used to acquire the input set to have lower performance. For example, an imaging system may exhibit a relatively high level of imaging error (e.g., due to optical aberrations in the image forming components of the imaging system), and the digital image depicting the sample that is constructed may be more similar to a digital image captured using an imaging system that exhibits a lower level of imaging error. The multiple illumination modes used when acquiring the digital image input set may be substantially the same (or even identical) to the multiple illumination modes used when acquiring the digital image training set used to train the machine learning model according to the method of the first aspect. The multiple illumination modes used when acquiring the digital image input set may be a subset of the multiple illumination modes used when acquiring the digital image training set used to train the machine learning model according to the method of the first aspect.

[0040] The above features of the first aspect also apply to this second aspect where applicable. To avoid excessive repetitions, reference is made to the above.

[0041] According to a third aspect, a device for training a machine learning model to construct a digital image depicting a sample is provided. The device includes circuitry configured to perform: a first receiving function configured to receive a training set of digital images, wherein the training set of digital images is acquired using a first imaging system by illuminating the training sample with a plurality of illumination modes and capturing a digital image of the training sample for each of the plurality of illumination modes; a second receiving function configured to receive real data including digital images depicting the training sample, wherein the digital images of the real data are acquired using a second imaging system that exhibits a lower level of imaging error than the first imaging system; and a training function configured to train a machine learning model to construct a digital image depicting the sample using the received training set of digital images and the received real data.

[0042] The device can be configured to train a machine learning model according to the method of the first aspect.

[0043] The above features of the first and / or second aspects also apply to this third aspect, where applicable. To avoid excessive repetition, reference is made to the above.

[0044] According to a fourth aspect, a microscope apparatus is provided. The microscope apparatus comprises: an imaging system comprising: an image sensor, an image forming assembly configured to image a sample onto the image sensor, and an illumination subsystem configured to illuminate the sample from multiple directions; and circuitry configured to perform: an acquisition function configured to acquire a set of digital image inputs by controlling the illumination subsystem to illuminate the sample with multiple illumination modes and controlling the image sensor to capture a digital image for each of the multiple illumination modes; and an image construction function configured to input the set of digital image inputs into a machine learning model trained according to the method of the first aspect, and receive output from the machine learning model comprising a digital image depicting the sample.

[0045] The above features of the first, second and / or third aspects also apply to this fourth aspect, where applicable. To avoid excessive repetition, reference is made to the above.

[0046] According to a fifth aspect, a non-transitory computer-readable storage medium is provided, comprising program code portions which, when executed on a device having processing capabilities, perform the method according to the first aspect or the method according to the second aspect.

[0047] The above features of the first, second, third and / or fourth aspects also apply to the fifth aspect, where applicable. To avoid excessive repetition, reference is made to the above.

[0048] Further features and advantages of the present invention will become clear upon studying the appended claims and the following description. Those skilled in the art will recognize that different features of the present invention may be combined to produce variations other than those described below without departing from the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Various aspects of the present inventive concept, including its particular features and advantages, will be readily understood from the following detailed description and accompanying drawings, in which:

[0050] Figure 1A A schematic diagram of a microscope apparatus suitable for acquiring a training set of digital images for training a machine learning model and / or an input set of digital images for constructing a digital image depicting a sample using the trained machine learning model is illustrated.

[0051] Figure 1B A schematic diagram of a second imaging system suitable for acquiring digital images of real data is illustrated.

[0052] Figure 2 A schematic diagram of an apparatus for training a machine learning model to construct a digital image depicting a sample is illustrated.

[0053] Figure 3A is a block diagram of a method for training a machine learning model to construct a digital image depicting a sample.

[0054] Figure 3B is a block diagram of method steps for obtaining a training set of digital images for training a machine learning model.

[0055] Figure 3C is a block diagram of method steps for obtaining digital images of real data for training a machine learning model.

[0056] Figure 4A is a block diagram of a method for constructing a digital image depicting a sample.

[0057] Figure 4B is a block diagram of method steps for obtaining a set of digital image inputs for constructing a digital image depicting a sample.

[0058] Figure 5 is a schematic diagram of a non-transitory computer-readable storage medium. DETAILED DESCRIPTION

[0059] The present invention will now be described more fully below with reference to the accompanying drawings, in which presently preferred variations of the present invention are shown and discussed. However, the present invention can be implemented in many different forms and should not be construed as limited to the variations set forth herein; rather, these variations are provided for thoroughness and completeness and to fully convey the scope of the present invention to those skilled in the art. As illustrated in the accompanying drawings, features may be exaggerated for illustrative purposes and, therefore, may be provided to illustrate the overall structure of variations of the present invention. Throughout this specification, like reference numerals refer to like elements.

[0060] Now refer to Figure 1A and Figure 1B A microscope system 10 is described that is suitable for acquiring a set of digital images that can be used as a digital image training set when training a machine learning model to construct a digital image depicting a sample, and / or can be used as a digital image input set when using the trained machine learning model to construct a digital image depicting a sample.

[0061] Figure 1A Schematic diagram of the microscope device 10. Figure 1AAs shown, microscopy apparatus 10 includes an imaging system 100 and a circuit system 110. Imaging system 100 includes an image sensor 102, an image forming assembly 104, and an illumination subsystem 106.

[0062] Imaging system 100 may further include a sample positioning assembly 108. Sample positioning assembly 108 may be configured to hold sample 1080. Sample 1080 may be a biological sample. Sample 1080 may be a cytological sample. Examples of biological samples include, but are not limited to, blood, plasma, bone marrow fluid, etc. The sample may also be a non-biological sample. Non-biological samples may be man-made or naturally occurring. Examples of non-biological samples include, but are not limited to, integrated circuits, optical components, microstructures, minerals, metals, etc. Sample positioning assembly 108 may be configured to move sample 1080. Sample positioning assembly 108 may be configured to move sample 1080 along a plane. The normal of the plane may be substantially parallel to optical axis 101 of imaging system 100. It should be further understood that sample positioning assembly 108 may be configured to move the sample in a direction substantially parallel to optical axis 101 of imaging system 100. In other words, sample positioning assembly 108 may be configured to move sample 1080 in the focus direction of imaging system 100. Circuitry 110 may be configured to control sample positioning assembly 108. For example, circuitry 110 can be configured to perform sample positioning functionality 1124. Sample positioning functionality 1124 can be configured to control the position of sample 1080 using sample positioning assembly 108.

[0063] Although the circuit system 110 Figure 1A 100 , it will be appreciated that the circuit system 110 may form part of an electronic device. The electronic device may be, for example, a computer, a server, a smartphone, etc. The electronic device may be a local electronic device (i.e., arranged near the imaging system 100) or a remote electronic device. Non-limiting examples of remote electronic devices may be servers, cloud servers, remote computers, remote smartphones, etc. It will be further appreciated that the functionality of the circuit system 110 may be distributed across more than one electronic device. The electronic device may include additional components, such as an input device (mouse, keyboard, touch screen, etc.) and / or a display. Figure 1AAs shown, the circuit system 110 may include one or more of a memory 112, a processing unit 114, a communication interface 116, and a data bus 118. The memory 112, the processing unit 114, and the communication interface 116 may communicate (e.g., exchange data) via the data bus 118. The processing unit 114 may include a central processing unit (CPU) and / or a graphics processing unit (GPU). The communication interface 116 may be configured to communicate with an external device. For example, the communication interface 116 may be configured to communicate with a server, a computer, an external peripheral device (e.g., an external storage device), etc. The external device may be a local device or a remote device (e.g., a cloud server). The communication interface 116 may be configured to communicate with the external device via an external network (e.g., a local area network, the Internet, etc.). The communication interface 116 may include a transceiver. The communication interface 116 may be configured for wireless and / or wired communication. Technologies suitable for wireless communication are known to those skilled in the art. Some non-limiting examples include Wi-Fi and near-field communication (NFC). Technologies suitable for wired communication are known to those skilled in the art. Some non-limiting examples include USB, Ethernet, and FireWire.

[0064] The memory 112 may be a non-transitory computer-readable storage medium. The memory 112 may be a random access memory. The memory 112 may be a non-volatile memory. Figure 1A As shown in the example of , memory 112 can store program code portions 1120, 1122, 1124, 1126, 1128 corresponding to one or more functions. Program code portions 1120, 1122, 1124, 1126, 1128 can be executed by processing unit 114, which thereby performs the function. Therefore, when it is mentioned that circuit system 110 is configured to perform a specific function, processing unit 114 can execute program code portions 1120, 1122, 1124, 1126, 1128 corresponding to the specific function, which can be stored on memory 112. However, it should be understood that one or more functions of circuit system 110 can be implemented in hardware and / or in a specific integrated circuit. For example, a field programmable gate array (FPGA) can be used to implement one or more functions. Therefore, one or more functions of circuit system 110 can be implemented in hardware or software or a combination of the two.

[0065] Although the image sensor 102 Figure 1A102 is shown as a separate entity, but it should be understood that the image sensor 102 may form part of the camera. The image sensor 102 may include a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. The image sensor 102 may capture digital color images. To this end, the image sensor 102 may include a color filter (e.g., a Bayer filter) to allow the image sensor 102 to capture color information of light incident on the image sensor 102. Figure 1A As shown in the example of FIG, image sensor 102 can communicate with circuitry 110 via communication interface 116. However, it should be understood that image sensor 102 can communicate with circuitry 110 via data bus 118.

[0066] Image forming assembly 104 is configured to image sample 1080 onto image sensor 102. For example, image forming assembly 104 can be positioned such that the object plane of image forming assembly 104 coincides with sample 1080 and the image plane of image forming assembly 104 coincides with image sensor 102. It will be appreciated that whether an image depicting sample 1080 is formed on image sensor 102 may depend on the illumination of sample 1080. For example, if sample 1080 is illuminated using a manner similar to conventional microscope illumination (e.g., brightfield illumination), an image similar to sample 1080 may be formed on image sensor 102. However, if sample 1080 is illuminated with light from only a few directions, or even a single direction, the image formed on image sensor 102 may not resemble sample 1080. For example, sample 1080 may be illuminated with light from one or more directions corresponding to angles greater than the numerical aperture 1040 of image forming assembly 104. In such cases, the image formed on image sensor 102 may not resemble sample 1080. However, even though the images may not resemble (or depict) sample 1080, the images may include information associated with sample 1080, and this information may be used in training the machine learning model and / or in constructing digital images depicting the sample using the trained machine learning model.

[0067] The image forming assembly 104 can be moved along a direction Z that is substantially parallel to the optical axis 101 of the imaging system 100. In other words, the image forming assembly 104 can be moved in a focusing direction of the imaging system 100. The image forming assembly 104 can be moved along the direction Z by being coupled to a manual and / or motorized stage (not shown). The image forming assembly 104 and / or the sample positioning assembly 108 can be movable so that the image sensor 102 can capture a focused image of the sample 1080. The position of the image forming assembly 104 along the direction Z can be controlled by the circuit system 110. For example, the circuit system 110 can be configured to perform a focusing function 1126 that is configured to adjust the position of the image forming assembly 104 along the direction Z. The focusing function 1126 can be configured to automatically adjust the position of the image forming assembly 104 along the direction Z. In other words, the focusing function 1126 can be an automatic focusing function. As Figure 1A As shown in the example of FIG, the focus function 1126 can control the position of the image forming assembly 104 along the direction Z by communicating via the communication interface 116. However, it should be understood that the focus function 1126 can control the position by communicating via the data bus 118.

[0068] The image forming assembly 104 may include a microscope objective lens ( Figure 1A (not shown in the figure). The image forming component 104 may include one or more imaging elements. In this context, the one or more imaging elements may be one or more elements configured to image the sample 1080 onto the image sensor 102. For example, the one or more imaging elements may include one or more of a pinhole and an optical lens. The one or more imaging elements may be one or more refractive optical elements. The one or more imaging elements may include one or more optical lenses, for example, in the form of a microscope objective or lens assembly. Figure 1A As shown in the example of , the image forming assembly 104 of the imaging system 100 can include only one imaging element. For example, the image forming assembly 104 can include only one optical lens. The image forming assembly 104 can therefore have a lower complexity. This, in turn, can reduce the economic costs associated with the imaging system 100. However, it should be understood that the imaging system 100 can include additional components, such as one or more of an aperture, a window, a color filter, etc. Such additional components are not considered imaging elements in the context of the present application. For example, the imaging system 100 including a window and an image forming assembly 104 consisting of only one optical lens should be interpreted as including only one imaging element.

[0069] The illumination subsystem 106 is configured to illuminate the sample 1080 from a plurality of directions 1060. The illumination subsystem 106 may include a plurality of light sources 1062. Figure 1AAs shown in the example of , multiple light sources 1062 can be arranged on the curved surface 1064. Figure 1A As illustrated in the example, curved surface 1064 can be concave along at least one direction along surface 1064. For example, curved surface 1064 can be a cylindrical surface. Curved surface 1064 can be concave along two perpendicular directions along the surface. For example, curved surface 1064 can have a shape similar to a spherical segment. A spherical segment can be a spherical cap or a spherical dome. Arranging multiple light sources 1062 on curved surface 1064 can be advantageous because the distance R from each light source to the current imaging position P of imaging system 100 can be similar. Because these distances R are similar, the intensity of light emitted from each of the multiple light sources 1062 can be similar at the current imaging position P. This can be understood as an effect of the inverse square law. As a result, sample 1080 can be illuminated by light having similar intensity in each of the multiple directions 1060, which in turn can allow sample 1080 to be more similarly illuminated regardless of the illumination direction. The distance R from each light source to the current imaging position P can be in the range of 4 cm to 15 cm. It may be advantageous to configure the illumination subsystem 106 so that the distance R from each light source to the current imaging position P is large enough so that each light source can be considered a point light source. Thus, given that the intensity of light from each light source at the current imaging position is high enough to generate a digital image set, the distance R from each light source to the current imaging position P may be greater than 15 cm. However, it should be understood that the plurality of light sources 1062 may be arranged on a flat surface or a surface having an irregular shape. It should be further understood that Figure 1A The diagram shows a cross section of the microscope apparatus 10, in particular a cross section of the illumination subsystem 106. Thus, Figure 1AThe curved surface 1064 of the illustrated illumination subsystem 106 can be a portion of a cylindrical surface or a spherical surface (or a quasi-spherical surface). The curved surface 1064 of the illumination subsystem 106 can be bowl-shaped. The curved surface 1064 can be formed by facets (not shown). In other words, the curved surface 1064 can be formed by multiple flat surfaces. Thus, the curved surface 1064 can be segmentally flat. The curved surface 1064 can be a portion of a quasi-spherical surface comprising multiple facets or segments. Thus, the curved surface 1064 can be a portion of a polyhedron surface. An example of such a polyhedron is a truncated icosahedron. Multiple light sources 1062 can be arranged on the facets. Each light source can be arranged so that the light source is configured to emit light in a direction substantially parallel to the normal of the associated facet. Each facet can be a flat surface having at least three sides. For example, the curved surface 1064 can be formed by a facet having five sides and a facet having six sides (e.g., similar to the inside surface of a football or soccer ball). It should be understood that each facet can be a separate entity. Thus, the curved surface 1064 can be formed from multiple sections, and each facet can be formed from one or more sections. It should be further understood that each section can include one or more facets. Furthermore, such sections can be arranged to contact adjacent sections, or can be arranged to be spaced a certain distance from adjacent sections. A single section can include all facets.

[0070] Each of the plurality of light sources 1062 can be configured to illuminate the sample 1080 from one of the plurality of directions 1060. The plurality of light sources 1062 can include one or more adjustable light sources. One or more of the position, direction of emitted light, color of emitted light, and intensity of emitted light of the one or more adjustable light sources can be adjustable. One or more of the plurality of light sources 1062 can be configured to emit broad-spectrum light (i.e., white light or pseudo-white light). Alternatively or in addition, one or more of the plurality of light sources 1062 can be configured to emit narrow-spectrum light (e.g., monochromatic light or quasi-monochromatic light). The light source can be a light-emitting diode (LED). The LED can be a white LED or a colored LED. The white LED can be formed, for example, by a blue LED covered with a layer of fluorescent material that emits white light when illuminated. By using white LEDs, the digital image set can include information about the sample 1080 associated with a relatively wide range of wavelengths, especially compared to when using narrow-band light (e.g., laser, monochromatic LED, etc.). This in turn can allow the machine learning model to use more information about the sample 1080 during training or when constructing a digital image depicting the sample. The LED can be any type of LED, such as a common LED bulb (i.e., a conventional and inorganic LED), a graphene LED, or an LED typically found in displays (e.g., a quantum dot LED (QLED) or an organic LED (OLED)). However, other types of LEDs can also be used. The illumination subsystem 106 can, for example, include a plurality of lasers, and the light emitted from each of the plurality of lasers can be converted into light having a wider spectral bandwidth. An example of such a conversion process can be referred to as supercontinuum generation. The light source can emit incoherent light, quasi-coherent light, or coherent light.

[0071] At least one direction 1061 of the plurality of directions 1060 may correspond to an angle greater than the numerical aperture 1040 of the image forming component 104 of the imaging system 100. The numerical aperture 1040 of the image forming component 104 may be a dimensionless number associated with the angular range over which the image forming component 104 accepts light. Thus, a direction 1061 (e.g., Figure 1AThe directions 1064 in the image forming assembly 104 may be directions 1061 corresponding to angles outside (e.g., greater than) the range of angles at which the image forming assembly 104 receives light. Because the sample 1080 is illuminated from multiple different directions, information about finer details of the training sample can be captured, details that are finer than those that the image forming assembly 104 used to image the sample 1080 can typically resolve. This can be understood as capturing information about different portions of Fourier space (i.e., the spatial frequency domain) associated with the sample 1080 for different illumination directions. This technique may be referred to in the art as Fourier ptychography. Generally, in Fourier ptychography, when the sample is illuminated from a direction corresponding to a large angle of incidence, high spatial frequencies in the Fourier space associated with the sample can be sampled. Therefore, when the sample 1080 is illuminated from a direction 1061 corresponding to an angle greater than the numerical aperture 1040 of the image forming assembly 104, even higher spatial frequencies in the Fourier space associated with the sample 1080 can be sampled. This is possible because light is scattered by the sample 1080, and a portion of the light scattered by the sample 1080 can be collected by the image forming assembly 104. Illuminating the sample 1080 from multiple directions 1060 can further allow the image forming assembly 104 to capture information about the refractive index (or spatial distribution of the refractive index) associated with the sample 1080. This can be understood as the refractive effect of light depending on the angle of incidence of the light illuminating the sample 1080 and the refractive index of the sample 1080. Information about the refractive index of the sample 1080 can, in turn, allow phase information associated with the sample 1080 (often referred to in the art as quantitative phase) to be determined. It should be understood that by illuminating the sample 1080 from more than one of the multiple directions 1060 at a time (e.g., from a subset of the multiple directions 1060), information related to one or more of the following can be captured: finer details of the sample 1080, the refractive index associated with the sample 1080, and phase information associated with the sample 1080. The subset of the multiple directions can include directions corresponding to non-overlapping portions of the Fourier space of the sample 1080. However, two or more directions in plurality of directions 1060 may correspond to overlapping portions of Fourier space associated with sample 1080. For example, two adjacent light sources in plurality of light sources 1062 may be configured to emit light in directions corresponding to overlapping portions of Fourier space associated with sample 1080.

[0072] The function that circuit system 110 is configured to perform depends on whether the digital image set is used to train a machine learning model (i.e., as a digital image training set) or to construct digital images depicting a sample using the trained machine learning model (i.e., as a digital image input set). However, for both purposes, circuit system 110 is configured to perform acquisition function 1120. Acquisition function 1120 is configured to acquire a set of digital images. To this end, acquisition function 1120 is configured to control illumination subsystem 106 to illuminate sample 1080 using a plurality of illumination modes, and to control image sensor 102 to capture a digital image for each of the plurality of illumination modes. Each digital image in the set of digital images can be associated with one of the plurality of illumination modes. Each of the plurality of illumination modes can be formed, for example, by simultaneously illuminating sample 1080 from one or more of plurality of directions 1060 and / or by varying the number of light sources emitted by plurality of light sources 1062 of illumination subsystem 106. Each of the plurality of illumination modes can be formed by illuminating sample 1080 from one or more directions corresponding to non-overlapping portions of Fourier space associated with sample 1080. For example, each of the plurality of illumination modes can be formed by illuminating sample 1080 from only one of plurality of directions 1060. Each of the plurality of illumination modes can be formed by one or more light sources from plurality of light sources 1062. In other words, each of the plurality of illumination modes can be formed by simultaneously emitting light from one or more light sources from plurality of light sources 1062. However, two different illumination modes from the plurality of illumination modes can be formed by illuminating sample 1080 from one or more directions corresponding to at least partially overlapping portions of Fourier space associated with sample 1080. By including multiple light sources 1062, illumination subsystem 106 of imaging system 100 can switch between different illumination modes more quickly and / or more reliably than an illumination subsystem including a movable light source capable of illuminating a location from different directions. Sample 1080 can be illuminated under different lighting conditions (i.e., by using different illumination modes), and more information about sample 1080 (e.g., information associated with refractive index, phase, and / or finer details) can be collected by capturing digital images of the sample 1080 when the sample 1080 is illuminated under different lighting conditions.

[0073] Because the acquired digital image set may include information associated with one or more of: fine details of the sample 1080, a refractive index associated with the sample 1080, and phase information associated with the sample 1080, this information may be used by the machine learning model (either during training or when constructing a digital image depicting the sample using the trained machine learning model). This, in turn, may allow the trained machine learning model to more accurately construct a digital image depicting the sample than would be permitted if the digital image set were captured from only one direction. In other words, this may allow the trained machine learning model to more accurately construct a digital image depicting the sample than would be permitted if the digital image set were captured from only one direction or using a conventional microscope (e.g., using brightfield illumination).

[0074] When training a machine learning model to construct a digital image depicting a sample, the acquired digital image set can be used as a digital image training set. In this context, the sample 1080 can be referred to as a training sample. In this case, the circuit system 110 is configured to perform a receiving function configured to receive real data including a digital image depicting the sample 1080. However, the digital image of the real data is acquired using the second imaging system 200. In the following, Figure 1A The imaging system 100 may be referred to as a first imaging system 100 . Figure 1B is a schematic diagram of a second imaging system 200. The second imaging system 200 may be different from the first imaging system 100. Figure 1B As shown in the example of FIG, the second imaging system 200 may include one or more of an image sensor 202, an image forming component 204, an illumination subsystem 206, and a sample positioning component 208. Figure 1B Although shown as a separate entity, it will be appreciated that the image sensor 202 of the second imaging system 200 may form part of an electronic device such as a camera. Figure 1B As shown in the example of FIG, the sample positioning assembly 208 of the second imaging system 200 can be configured to hold a sample 2080. Since the second imaging system 200 is used to capture digital images of real data to be used during training, Figure 1B The sample 2080 may be a training sample. The illumination subsystem 206 of the second imaging system 200 may include one or more light sources. Figure 1BAs shown in the example of , the illumination subsystem 206 may include a conventional light source suitable for microscopy (e.g., brightfield microscopy). The illumination subsystem 206 may be configured to illuminate the training sample 2080 with a brightfield illumination pattern. Here, the brightfield illumination pattern may be an illumination pattern similar to conventional microscope illumination. Such a brightfield illumination pattern may be formed by a conventional light source suitable for microscopy. In this case, the brightfield illumination pattern may be very similar to (or even identical to) conventional microscope illumination. However, it should be understood that Figure 1A The illumination subsystem 106 in the embodiment can be used to form a bright field illumination mode. Figure 1A Most (or all) of the light sources of the illumination subsystem 106 are used to form a bright field illumination mode similar to conventional microscope illumination. For example, Figure 1A The lighting subsystem 106 is less than (or may be equal to) Figure 2 A light source emitting light in a direction corresponding to the angle of the numerical aperture 2040 of the image forming assembly 204 of the second imaging system 200 shown in B can be used to form an illumination pattern similar to conventional microscope illumination. Using an illumination pattern similar to conventional microscope illumination can allow a digital image similar to the sample 2080 to be captured using the second imaging system 200.

[0075] Compare Figure 1A and Figure 1B As can be seen, the first imaging system 100 and the second imaging system 200 can be similar. However, a key difference is that the second imaging system 200 exhibits a lower level of imaging errors than the first imaging system 100. The different levels of imaging errors may be due to structural and / or technological differences between the first imaging system 100 and the second imaging system 200. For example, the image forming assembly 104 of the first imaging system 100 can be simpler (e.g., have fewer components) than the image forming assembly 204 of the second imaging system 200. In general, image forming assemblies can be designed to reduce imaging errors, but they thereby become more complex (i.e., include more components). Alternatively or additionally, the quality of the components of the image forming assembly 104 of the first imaging system 100 can be lower than the quality of the components of the image forming assembly 204 of the second imaging system 200. These differences, in turn, can reduce the economic costs associated with the first imaging system 100 compared to the second imaging system 200.

[0076] Image forming assembly 204 of second imaging system 200 may exhibit lower levels of optical aberrations than image forming assembly 104 of first imaging system 100. Optical aberrations may include one or more of the following: spherical aberration, astigmatism, coma, image distortion, chromatic aberration, and Petzval field curvature. The imaging error levels of first imaging system 100 may be correlated with the optical aberration levels of image forming assembly 104 of first imaging system 100. The imaging error levels of second imaging system 200 may be correlated with the optical aberration levels of image forming assembly 204 of second imaging system 200. Thus, an image of an object in best focus formed by image forming assembly 204 of second imaging system 200 may be more similar to the imaged object than an image of an object in best focus formed by image forming assembly 104 of first imaging system 100. The image formed by image forming assembly 204 of second imaging system 200 may be, for example, sharper than a corresponding image formed by image forming assembly 104 of first imaging system 100. The image of a subject formed by the image forming assembly 204 of the second imaging system 200 can be more similar to an ideal image of the subject than the image of the subject formed by the image forming assembly 104 of the first imaging system 100. In the art, an ideal image is an image formed by an imaging system without aberrations. In the context of the present disclosure, the similarity between the image of the subject formed by the image forming assembly and the ideal image of the subject can be referred to as the quality of the image formed by the image forming assembly. Therefore, a higher-quality image of the subject can be more similar to the ideal image of the subject than a lower-quality image of the subject.

[0077] The number of imaging elements of the image forming component 104 of the first imaging system 100 can be lower than the number of imaging elements of the image forming component 204 of the second imaging system 200, whereby the image forming component 204 of the second imaging system 200 can have a lower level of optical aberration than the image forming component 104 of the first imaging system 100. Typically, a complex image forming component (e.g., a microscope objective) includes several imaging elements (e.g., optical lenses) that are arranged to reduce the level of optical aberration. Therefore, reducing the number of imaging elements (e.g., lenses, etc.) of an image forming component may increase the level of optical aberration of the image forming component. For example, the image forming component 204 of the second imaging system 200 may include a microscope objective. The microscope objective may include a complex arrangement of multiple optical lenses of different types. Typically, a microscope objective is designed to produce a magnified image of an object and reduce (preferably minimize) imaging errors to produce a high-quality image of the object. However, such a microscope objective is typically associated with a higher economic cost. By using a microscope objective in the image forming component 204 of the second imaging system 200, the quality and / or resolution of the digital image of the real data can thereby be improved. In combination with Figure 1AThe image forming assembly 204 of the second imaging system 200 may be more complex (i.e., include a greater number of imaging elements) than the image forming assembly 104 of the first imaging system 100 described above. For example, the image forming assembly 104 of the first imaging system 100 may include a lower quality microscope objective than the microscope objective of the image forming assembly 204 of the second imaging system 200. Figure 1A As shown, the image forming component 104 of the first imaging system 100 may include only one imaging element (e.g., an optical lens). The quality of the image produced by only one imaging element may be lower than the quality of the image produced by the image forming component 204 of the second imaging system 200 (e.g., a microscope objective).

[0078] The lower imaging error levels of the second imaging system 200 compared to the first imaging system 100 have been described above as optical aberrations. However, it should be understood that the present inventive concept is not limited to optical aberrations. Alternatively or additionally, the image sensor 202 of the second imaging system 200 may exhibit one or more of a greater number of pixels, a higher pixel resolution, a larger pixel size, a higher dynamic range, and a lower noise level compared to the image sensor 102 of the first imaging system 100. Consequently, the digital image quality of the digital images produced by the image sensor 202 of the second imaging system 200 may be higher than the digital image quality of the digital images produced by the image sensor 102 of the first imaging system 100. This means that the image sensor 102 of the first imaging system 100 may be less expensive than the image sensor 202 of the second imaging system 200. As a further example, the sample positioning assembly 108 of the first imaging system 100 may be configured to move the sample along a first plane, and the sample positioning assembly 208 of the second imaging system 200 may be configured to move the sample along a second plane, which may exhibit a higher degree of flatness than the first plane. In other words, when the sample is moved, the second imaging system 200 can have a more accurate and / or repeatable sample focus position (i.e., the position of the sample relative to the image-forming components of the imaging system) than the first imaging system 100. Consequently, the second imaging system 200 can position the sample 2080 in focus with greater accuracy and / or better repeatability than the first imaging system 100. Consequently, the sample positioning assembly 108 of the first imaging system 100 can be simpler and / or more economically affordable than the sample positioning assembly 208 of the second imaging system 200.

[0079] It will be appreciated that the second imaging system 200 may form part of a separate microscope arrangement. In this case, the second imaging system 200 may be used. Figure 1AThe microscope apparatus 100 can acquire a training set of digital images, and the training sample 1080 can then be moved to a separate microscope apparatus so that the second imaging system 200 can acquire digital images of real data. The sample 1080 can be moved automatically (e.g., by using a motorized device) or manually between the first imaging system 100 and the second imaging system 200. However, it should be understood that Figure 1A The microscope apparatus 10 may include a first imaging system 100 and a second imaging system 200. In this case, the first imaging system 100 and the second imaging system 200 may be interchangeable. For example, the first imaging system 100 may be replaced with the second imaging system 200. Furthermore, the first imaging system 100 and the second imaging system 200 may share one or more components. As a specific example, the first imaging system 100 and the second imaging system 200 may share an image sensor, an illumination subsystem, and a sample positioning assembly. In this specific example, the image forming assembly 104 of the first imaging system 100 is different from the image forming assembly 204 of the second imaging system 200. If the first imaging system 100 and the second imaging system 200 share a sample positioning assembly, the shared sample positioning assembly may be used to move training samples between the first imaging system 100 and the second imaging system 200. For example, the shared sample positioning system may allow training samples to be moved between the first imaging system 100 and the second imaging system 200, allowing the first image forming assembly 104 to be used to acquire a training set of digital images, while allowing the second image forming assembly 204 to be used to acquire digital images of real data. Alternatively or additionally, the image forming assemblies 104, 204 of the first imaging system 100 and the second imaging system 200 may be interchangeable. For example, the image forming assemblies 104, 204 of the first imaging system 100 and the second imaging system 200 may be mounted on a turntable. The turntable may be rotatable. Thus, depending on the current setting of the turntable, the first image forming assembly 104 or the second image forming assembly 204 may be positioned to image a training sample onto a (possibly shared) image sensor.

[0080] The acquired digital image set can be used as a digital image training set for training a machine learning model to construct a digital image depicting a sample. To this end, the circuit system 110 can be further configured to execute a training function 1128. The training function 1128 can be configured to use the acquired digital image training set and the acquired real data to train the machine learning model to construct a digital image depicting the sample. Therefore, the training function 1128 can be configured to train the machine learning model to associate the digital image training set with the real data (e.g., the digital image depicting the training sample). The training function 1128 can be configured to iteratively and / or recursively train the machine learning model until the difference between the output of the machine learning model (i.e., the constructed digital image) and the real data (i.e., the digital image depicting the training sample) is less than a predetermined threshold. A smaller difference between the output of the machine learning model and the real data can indicate that the constructed digital image provided by the machine learning model has a higher degree of accuracy. In other words, a smaller difference between the output of the machine learning model and the real data can indicate that the constructed digital image can more closely replicate the digital image acquired using the second imaging system 200 and depicting the training sample. Therefore, preferably, the difference between the output of the machine learning model and the real data can be minimized. The training functionality 1128 can be configured to train the machine learning model to construct digital images of samples for a plurality of different sample types. In this case, for each sample type, the machine learning model can be trained using a training set of digital images of training samples of that sample type and corresponding real data associated with the respective sample type.

[0081] The trained machine learning model can use the digital image input set acquired by the first imaging system 100 to construct digital images depicting the sample and having a quality comparable to (or the same as) that of digital images acquired by the second imaging system 200 (e.g., the digital images of the real data used during training). In other words, the trained machine learning model can use the digital image input set to construct digital images depicting the sample and exhibiting a lower level of imaging error than the imaging system used to capture the digital image input set is typically capable of. For example, because the level of imaging error (e.g., optical aberration) of the second imaging system 200 (i.e., in the digital images of the real data) is lower than that of the first imaging system 100 (i.e., in the digital image training set), the machine learning model can be trained to eliminate such imaging errors in its output (i.e., the constructed digital images).

[0082] If the acquired digital image set is to be used as a digital image input set to use a trained machine learning model (e.g., similar to the above and / or in combination with Figure 2 、 Figure 3A or Figure 3BIn this case, the circuit system 110 may be further configured to execute an image construction function 1122. The image construction function 1122 is configured to input the digital image input set into the trained machine learning model and receive an output from the machine learning model including a digital image depicting the sample 1080. Figure 1A Because the image construction function 1122 is configured to use the trained machine learning model, a simpler and / or less expensive imaging system (e.g., the first imaging system 100) can be used to acquire the digital image input set (i.e., using Figure 1A 10 acquired using the microscope device 10), the image construction function 1122 is capable of constructing a digital image depicting the sample that is similar to a digital image depicting the sample acquired using a more complex imaging system and / or a higher quality imaging system (e.g., the second imaging system 200). For example, acquiring digital images of real data using a microscope objective (i.e., where the image forming component 204 of the second imaging system 200 includes a microscope objective) can allow the trained machine learning model to construct digital images of higher quality and / or resolution from a relatively low quality digital image set (i.e., the digital images in the digital image input set). As described above, a simpler imaging system can be used to acquire the digital image input set. For example, the imaging system can include only one imaging element (e.g., an optical lens), and the trained machine learning model can use the digital image input set acquired by the imaging system to construct a digital image that can be similar to (or identical to) a digital image acquired using the microscope objective. Thus, the trained machine learning model may allow a simpler and / or less expensive microscope apparatus 10 to be used to acquire a digital image input set, while still being able to construct a digital image depicting the sample that is similar to a digital image acquired using a more expensive microscope apparatus (e.g., a microscope apparatus that includes the second imaging system 200). In other words, after the machine learning model has been trained, the second imaging system 200 may no longer be required to construct a high-quality digital image of the sample because the first imaging system 100 is sufficient to acquire the digital image input set.

[0083] Although the training of the machine learning model has been described in conjunction with the microscope apparatus 10 , it should be understood that the training may be performed in a separate device 60 . Figure 2 is a schematic diagram of an apparatus 60 for training a machine learning model to construct a digital image depicting a sample. Figure 2 Device 60 may be a computing device (e.g., a computer, server, cloud server, smartphone, etc.). Device 60 may include additional components, such as an input device (mouse, keyboard, touch screen, etc.) and / or a display. Device 60 includes circuit system 610. Figure 2The circuit system 610 shown in FIG. 6 can be similar to the circuit system 610 in conjunction with FIG. Figure 1A and Figure 1B The circuit system 110 is described. Therefore, Figure 2 The circuit system 610 shown in FIG. 6 may include one or more of a memory 612 , a processing unit 614 , a communication interface 616 , and a data bus 618 . Figure 1A and Figure 1B The description of the corresponding features of the circuit system 110 applies where applicable Figure 2 To avoid excessive repetition, reference is made to the above. The circuit system 610 is configured to perform a first receiving function 6120, a second receiving function 6122, and a training function 6124. The first receiving function 6120 is configured to receive a training set of digital images. As described above, the circuit system 610 is configured to perform a first receiving function 6120, a second receiving function 6122, and a training function 6124. Figure 1A The set of digital images acquired by the microscope apparatus 10 shown can be used as a digital image training set. Figure 1A The circuit system 110 illustrated in FIG receives the digital image training set. However, it should be understood that the first receiving function 6120 can also be configured to receive the digital image training set from other sources. For example, the digital image training set can be stored on a remote device (e.g., a server, etc.) or a local device (e.g., a computer, a memory 612, etc.), and the first receiving function 6120 can be configured to receive the digital image training set therefrom. The first receiving function 6120 can receive the digital image training set via the communication interface 616. As described above, the digital image training set is acquired using the first imaging system 100 by illuminating the training sample with a plurality of illumination modes and capturing a digital image of the training sample for each of the plurality of illumination modes. The second receiving function 6122 is configured to receive real data including a digital image depicting the training sample. The digital image of the real data is acquired using the second imaging system 200. As described above, the second imaging system 200 exhibits a lower level of imaging error than the first imaging system 100. It can be combined with Figure 1A and Figure 1B The second receiving function 6122 can be configured to receive a digital image of real data from a microscope device for capturing a digital image of real data. For example, when using Figure 1A Microscope device 10 (with Figure 1B In the case of a second imaging system 200 as shown in FIG. 1 , the second receiving function 6122 may be configured to receive a digital image of the real data. Figure 1AThe circuit system 110 shown receives a digital image of real data. However, it should be understood that the second receiving function 6122 can also be configured to receive a digital image of real data from other sources. For example, the digital image of real data can be stored on a remote device (e.g., a server, etc.) or a local device (e.g., a computer, memory 612, etc.), and the second receiving function 6122 can be configured to receive the digital image of real data therefrom. The second receiving function 6122 can receive the digital image of real data via the communication interface 616. The training function 6124 is configured to use the received digital image training set and the received real data to train the machine learning model to construct a digital image depicting the sample. Figure 2 The training function 6124 can be configured to Figure 1A The machine learning model is trained in the same manner as the training function 1128. To avoid unnecessary repetition, refer to the above.

[0084] Figure 3A is a block diagram of a method 30 for training a machine learning model to construct a digital image depicting a sample. Figure 3A The method 30 may be a computer-implemented method. The method 30 includes receiving S300 a training set of digital images of training samples. Figure 3B As shown, the digital image training set is acquired S310 using the first imaging system 100 by illuminating S312 the training sample with a plurality of illumination modes and capturing S314 a digital image for each of the plurality of illumination modes. It should be understood that the method 30 may include acquiring S310 the digital image training set. Thus, as Figure 3B As shown, the method 30 may further include acquiring S310 a training set of digital images using the first imaging system 100 by illuminating S312 training samples with a plurality of illumination modes, and capturing S314 a digital image for each of the plurality of illumination modes.

[0085] The method 30 further comprises receiving S320 real data comprising digital images depicting the training samples. Figure 3C The figure shows a digital image of the real data acquired S330 using a second imaging system 200 that exhibits a lower level of imaging error than the first imaging system 100. The digital image of the real data can be acquired S330 using the second imaging system 200 by illuminating S332 a training sample with a bright field illumination mode, and capturing S334 a digital image while the training sample is illuminated with the bright field illumination mode. The bright field illumination mode can be an illumination mode similar to (or the same as) the illumination provided by a conventional light source. The bright field illumination mode can be formed by a conventional light source (e.g., a conventional light source suitable for microscopy). It should be understood that the method 30 may include acquiring a digital image of the real data S330. Therefore, as Figure 3CAs shown, the method 30 may further include acquiring S330 a digital image of the real data using the second imaging system 200 by illuminating S332 the training sample with a brightfield illumination mode, and capturing S334 a digital image of the real data while illuminating the training sample with the brightfield illumination mode.

[0086] The method 30 further comprises training S340 a machine learning model using the received digital image training set and the received real data to construct a digital image depicting the sample.

[0087] Image forming assembly 204 of second imaging system 200 may exhibit lower levels of optical aberrations than image forming assembly 104 of first imaging system 100, thereby forming an image of an object in best focus formed by image forming assembly 204 of second imaging system 200 that is more similar to the imaged object than the image of an object in best focus formed by image forming assembly 104 of first imaging system 100. The number of imaging elements of image forming assembly 104 of first imaging system 100 may be lower than the number of imaging elements of image forming assembly 204 of second imaging system 200, thereby forming an image forming assembly 204 of second imaging system 200 that is lower than the image forming assembly 104 of first imaging system 100. Image forming assembly 204 of second imaging system 200 may include a microscope objective. Image forming assembly 104 of first imaging system 100 may include only one imaging element. Optical aberrations may include one or more of the following: spherical aberration; astigmatism; coma; image distortion; chromatic aberration; and Petzval field curvature. Image sensor 202 of second imaging system 200 can exhibit one or more of the following compared to image sensor 102 of first imaging system 100: a greater number of pixels; a higher pixel resolution; a larger pixel size; a higher dynamic range; and a lower noise level. First imaging system 100 can include sample positioning assembly 108 configured to move the sample along a first plane, and second imaging system 200 can include sample positioning assembly 208 configured to move the sample along a second plane. The second plane can exhibit a higher degree of planarity than the first plane. First imaging system 100 can include illumination subsystem 106. Illumination subsystem 106 can include multiple light sources 1062. Each of the multiple light sources 1062 can be configured to illuminate training sample 1080 from one of multiple directions 1060. Each illumination pattern in a plurality of illumination patterns can be formed by one or more of the multiple light sources 1062. At least one direction 1061 in the plurality of directions 1060 can correspond to an angle greater than the numerical aperture 1040 of image forming assembly 104 of first imaging system 100. Two or more directions in plurality of directions 1060 may correspond to overlapping portions of Fourier space associated with training sample 1080, and wherein each of plurality of illumination patterns may be formed by illuminating training sample 1080 from one or more directions corresponding to non-overlapping portions of Fourier space associated with training sample 1080. Figure 3A 、 Figure 3B and / or Figure 3C Method 30 can be used Figure 1A and Figure 1B The microscope device 10 shown is implemented.

[0088] Figure 4Ais a block diagram of a method 40 for constructing a digital image depicting a sample. Figure 4A The method 40 may be a computer-implemented method. The method 40 comprises receiving S400 a digital image input set of samples. Figure 4B As shown, a digital image input set is acquired S410 using an imaging system by illuminating S412 a sample with a plurality of illumination modes and capturing S414 a digital image for each of the plurality of illumination modes. It should be understood that method 40 may include acquiring S410 a digital image input set. Thus, as Figure 4B As shown, the method 40 may further include acquiring S410 a set of digital image inputs using the imaging system by illuminating S412 the sample with a plurality of illumination modes and capturing S414 a digital image for each of the plurality of illumination modes. The method 40 further includes inputting S420 the set of digital image inputs into a machine learning model trained according to the previously described method 30. The method 40 further includes receiving S430 an output from the machine learning model comprising a digital image depicting the sample. Figure 4A and / or Figure 4B Method 40 can be used Figure 1A and Figure 1B The microscope device 10 shown or using Figure 2 The device 60 shown is used to perform.

[0089] Figure 5 is a schematic diagram of a non-transitory computer readable storage medium 50. The non-transitory computer readable storage medium 50 includes program code portions that, when executed on a device having processing capabilities, perform operations according to Figure 3A The illustrated method 30 or Figure 4A The method 40 is shown.

[0090] Those skilled in the art will understand machine learning, and in particular how machine learning models can be trained and / or how trained machine learning models can be used. However, in short, the machine learning model can be a supervised machine learning model, for example, a network such as U-net or Pix2pix. The machine learning model can be a transformer-based network, such as SwinIR. The machine learning model can be a convolutional neural network. The machine learning model can be trained to predict the desired output using example input training data and real data (i.e., "correct" or "factual" outputs). In other words, the real data can be used as labels for the input training data. The input training data can include data related to different outcomes, and each input training data can thereby be associated with the real data associated with that particular input training data. Therefore, each input training data can be labeled with the associated real data (i.e., "correct" or "factual" output). The machine learning model can include multiple layers of neurons, and each neuron can represent a mathematical operation applied to the input training data. Typically, the machine learning model includes an input layer, one or more hidden layers, and an output layer. The first layer can be called the input layer. The output of each layer in the machine learning model (except the output layer) can be fed to a subsequent layer, which in turn produces a new output. The new output can be fed to further subsequent layers. The output of the machine learning model can be the output of the output layer. This process can be repeated for all layers in the machine learning model. Typically, each layer further includes an activation function. The activation function can further define the output of the neurons in that layer. For example, the activation function can ensure that the output of the layer is not too large or too small (for example, tending towards positive infinity or negative infinity). Further, the activation function can introduce nonlinearity into the machine learning model. During the training process, the weights and / or biases associated with the neurons of the layer can be adjusted until the machine learning model produces predictions that reflect real data for the input training data. Each neuron can be configured to multiply the input of the neuron by the weight associated with the neuron. Each neuron can be further configured to add the bias associated with the neuron to the input. In other words, the output from the neuron can be the sum of the product of the bias associated with the neuron and the weight associated with the neuron and the input. The weights and biases can be adjusted in a recursive and / or iterative process. This can be referred to as backpropagation in the art. A convolutional neural network can be a neural network that includes one or more layers representing a convolution operation. In this context, the input training data includes a digital image. A digital image can be represented as a matrix (or array), and each element in the matrix (or array) can represent a corresponding pixel of the digital image. The value of the element can thus represent the pixel value and / or color value of the corresponding pixel in the digital image. Therefore, the input and output of the machine learning model can be numbers (e.g., matrices or arrays) representing the digital image.In this context, the input is a set of digital images (i.e., a training set or input set). Thus, the input to the machine learning model can be a plurality of matrices or a three-dimensional matrix. However, it should be understood that the machine learning model can accept further inputs during training. In this particular case, the machine learning model is trained using a digital image training set and real data. The first digital image training set is acquired in a manner similar to the first digital image input set. In other words, the lighting patterns used when acquiring the digital image training set and the digital image input set can be similar or identical. The machine learning model can be trained using the digital image training set as input and the real data as the desired output. In other words, the machine learning model can be trained until the difference between the output of the machine learning model and the real data is less than a threshold. This difference can be described in the art as a loss function. It may be preferable to train the machine learning model until the loss function is minimized. In other words, the machine learning model can be trained until the difference between the output of the machine learning model and the real data is minimized. The training process can be repeated for multiple different training samples (e.g., different training samples of the same and / or different types), which can allow the machine learning model to construct digital images of a wider range of sample types and / or to construct digital images with higher accuracy.

[0091] A person skilled in the art realizes that the inventive concept is by no means limited to the preferred variants described above. On the contrary, many modifications and variations are possible within the scope of the appended claims.

[0092] For example, despite Figure 1A The diagram shows the Figure 1A The training sample (or samples) is illuminated with a plurality of illumination patterns from one side (ie, the side facing the illumination subsystem 106) and from the opposite side (ie, Figure 1A The image is taken from the side facing the image forming assembly 104), but it should be understood that the training sample (or sample) can also be illuminated with multiple illumination modes from one side and imaged from the same side. Figure 1A It will be further understood that the illumination pattern may be formed by illuminating the training sample (or samples) from only one side of the sample simultaneously or by illuminating the training sample (or samples) from two opposite sides of the sample simultaneously.

[0093] Additionally, variations to the disclosed variations can be understood and effected by the skilled artisan in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

Claims

1. A method (30) for training a machine learning model to construct a digital image depicting a sample, the method comprising: A digital image training set of training samples is received (S300), wherein the digital image training set is acquired (S310) using a first imaging system (100) by the following steps: Illuminating (S312) the training sample with a plurality of illumination modes, and capturing (S314) a digital image for each of the plurality of illumination patterns; receiving (S320) real data including a digital image depicting the training sample, wherein the digital image of the real data is acquired (S330) using a second imaging system (200) exhibiting a lower imaging error level than the first imaging system (100); and The machine learning model is trained (S340) using the received digital image training set and the received real data to construct a digital image depicting the sample.

2. The method (30) according to claim 1, wherein: The image forming component (204) of the second imaging system (200) exhibits a lower level of optical aberration than the image forming component (104) of the first imaging system (100), whereby an image of an object in best focus formed by the image forming component (204) of the second imaging system (200) is more similar to the imaged object than the image of the object in best focus formed by the image forming component (104) of the first imaging system (100).

3. The method (30) according to claim 2, wherein: The number of imaging elements of the image forming assembly (104) of the first imaging system (100) is lower than the number of imaging elements of the image forming assembly (204) of the second imaging system (200), whereby the image forming assembly (204) of the second imaging system (200) has a lower level of optical aberration than the image forming assembly (104) of the first imaging system (100).

4. The method (30) according to claim 2 or 3, wherein: The image forming component (204) of the second imaging system (200) includes a microscope objective.

5. The method (30) according to any one of claims 2 to 4, wherein: The image forming assembly (104) of the first imaging system (100) includes only one imaging element.

6. The method (30) according to any one of claims 2 to 5, wherein: The optical aberrations include one or more of the following: Spherical aberration; astigmatism; coma; Image distortion; Chromatic aberration; and Petzval Field Song.

7. The method (30) according to any one of claims 1 to 6, wherein: The image sensor (202) of the second imaging system (200) exhibits one or more of the following as compared to the image sensor (102) of the first imaging system (100): Larger pixel count; Higher pixel resolution; Larger pixel size; Higher dynamic range; and Lower noise levels.

8. The method (30) according to any one of claims 1 to 7, wherein: The first imaging system (100) includes a sample positioning assembly (108) configured to move a sample (1080) along a first plane, and the second imaging system (200) includes a sample positioning assembly (208) configured to move a sample (2080) along a second plane, the second plane exhibiting a higher degree of flatness than the first plane.

9. The method (30) according to any one of claims 1 to 8, wherein: The first imaging system (100) includes an illumination subsystem (106), the illumination subsystem (106) including a plurality of light sources (1062), each of the plurality of light sources (1062) being configured to illuminate the training sample (1080) from one of a plurality of directions (1060); and Each of the plurality of lighting modes is formed by one or more light sources among the plurality of light sources (1062).

10. The method (30) according to claim 9, wherein: At least one direction (1061) of the plurality of directions (1060) corresponds to an angle that is larger than a numerical aperture (1040) of an image forming assembly (104) of the first imaging system (100).

11. The method (30) according to any one of claims 9 or 10, wherein: Two or more of the plurality of directions (1060) correspond to overlapping portions of Fourier space associated with the training sample (1080), and wherein each of the plurality of illumination patterns is formed by illuminating the training sample (1080) from one or more directions corresponding to non-overlapping portions of Fourier space associated with the training sample (1080).

12. A method (40) for constructing a digital image depicting a sample, the method comprising: A digital image input set of a sample (1080) is received (S400), wherein the digital image input set is acquired (S410) using an imaging system (100) by: illuminating (S412) the sample (1080) with a plurality of illumination modes, and capturing (S414) a digital image for each of the plurality of illumination patterns; inputting (S420) the digital image input set into a machine learning model trained according to the method (30) of any one of claims 1 to 11; and An output is received (S430) from the machine learning model including a digital image depicting the sample (1080).

13. An apparatus (20) for training a machine learning model to construct a digital image depicting a sample, the apparatus (20) comprising circuitry (210) configured to perform: A first receiving function (2100) is configured to receive a training set of digital images, wherein: The digital image training set is acquired using a first imaging system (100) by: illuminating a training sample (1080) with a plurality of illumination patterns and capturing a digital image of the training sample for each of the plurality of illumination patterns; a second receiving function (2102) configured to receive real data comprising a digital image depicting the training sample, wherein the digital image of the real data is acquired using a second imaging system (200) exhibiting a lower level of imaging error than the first imaging system (100); as well as A training function (2104) is configured to train the machine learning model using the received digital image training set and the received real data to construct a digital image depicting the sample.

14. A microscope device (10), comprising: An imaging system (100), comprising: an image sensor (102), an image forming assembly (104) configured to image a sample (1080) onto the image sensor (102), and an illumination subsystem (106) configured to illuminate the sample (1080) from multiple directions; and Circuitry (110) configured to perform: An acquisition function (1120) configured to acquire a digital image input set by being configured to: controlling the illumination subsystem (106) to illuminate the sample (1080) with a plurality of illumination modes, and controlling the image sensor (102) to capture a digital image for each of the plurality of illumination modes; and An image building function (1122) is configured to: inputting the digital image input set into a machine learning model trained according to the method (30) of any one of claims 1 to 11, and An output is received from the machine learning model including a digital image depicting the sample (1080).

15. A non-transitory computer-readable storage medium (50), comprising program code portions for performing the method (30) according to any one of claims 1 to 11 or the method (40) according to claim 12 when the program code portions are executed on a device having processing capabilities.

Images