Construction of digital image depicting sample

Abstract

The inventive concept relates to a method and apparatus for training a machine learning model for constructing a digital image depicting a sample. The method (30) comprises: receiving (S300) a digital image training set of a training sample (1080), where the digital image training set is acquired (S310) using a first microscope objective (104) by illuminating (S312) the training sample (1080) with a plurality of illumination modes and capturing (S314) a digital image for each of the plurality of illumination modes; receiving (S320) real data comprising a digital image depicting the training sample (1080), where the digital image of the real data is acquired (S330) using a second microscope objective (204) and by irradiating (S332) the training sample with a conventional microscopy illumination mode, and capturing (S334) the digital image while illuminating the training sample (1080) with the conventional microscopy illumination mode; and training (S340) the machine learning model using the received training set of digital images and the received real data for constructing a digital image depicting the training sample (1080); and wherein the second microscope objective (204) has a higher numerical aperture than the first microscope objective (104), whereby the resolution of the digital image of the real data is higher than the resolution of at least one digital image in the training set of digital images. The inventive concept further relates to a method and a microscope device for constructing a digital image depicting a sample.

Description

TECHNICAL FIELD

[0001] The present inventive concept relates to the construction of a digital image depicting a sample. BACKGROUND

[0002] In the field of digital microscopy, a typical task is to find and identify objects within a sample. For example, in hematology, cytology and pathology, specific cell types can be found and identified in order to establish a diagnosis for a patient from which the sample was taken. Another example is the imaging of manufactured components. For example, microscopes are often used to image electrical circuits and microelectronic devices for quality control.

[0003] There are different techniques for imaging a sample. One of the simplest forms of microscopy is brightfield microscopy, in which a sample is illuminated from below and imaged from above. Typically, sample images captured with a microscope have a very high magnification (e.g. 100x), i.e. they are able to reveal fine details of the sample. However, a drawback of high magnification microscope objectives is that the field of view of such microscope objectives is typically small. Therefore, each image captured with a high magnification microscope reveals only a small portion of the entire sample. This means that, in order to scan an entire sample, a large number of images need to be captured, each at a different location of the sample. This in turn can become very time consuming and therefore can increase the associated economic costs.

[0004] Further, high magnification microscope objectives typically comprise complex optical arrangements in order to produce high quality images (e.g. to achieve low levels of aberration at high magnification). This in turn increases the economic costs associated with a microscope system comprising such microscope objectives. SUMMARY

[0005] In view of the above, it is an object of the present inventive concept to provide a method and apparatus for training a machine learning model for constructing a digital image depicting a sample.

[0006] It is a further object to provide a method and microscope system for constructing a digital image depicting a sample.

[0007] It is a further object to provide a method and microscope system for constructing a digital image depicting a sample at a relatively high resolution using a digital image constructed at a relatively low resolution.

[0008] It is a further object to at least partially mitigate, alleviate, or eliminate one or more of the above-identified deficiencies in the art.

[0009] According to a first aspect, a method is provided for training a machine learning model to construct digital images depicting a sample. The method includes: receiving a training set of digital images of a training sample, wherein the training set is acquired using a first microscope objective by illuminating the training sample with a plurality of illumination modes and capturing digital images for each of the plurality of illumination modes; receiving real data comprising digital images depicting the training sample, wherein the digital images of the real data are acquired using a second microscope objective by illuminating the training sample with a conventional microscopy illumination mode and capturing digital images while illuminating the training sample with the conventional microscopy illumination mode; and training the machine learning model using the received training set of digital images and the received real data to construct digital images depicting the training sample; and wherein the second microscope objective has a higher numerical aperture than the first microscope objective, whereby the resolution of the digital images of the real data is higher than the resolution of at least one digital image in the training set of digital images.

[0010] The method according to the first aspect may further include: using a first microscope objective and acquiring a digital image training set of the training sample by illuminating the training sample with multiple illumination modes and capturing a digital image of the sample for each of the multiple illumination modes. The method according to the first aspect may further include: using a second microscope objective and acquiring a digital image of the training sample by illuminating the training sample with a conventional microscopy illumination mode; and capturing the digital image while illuminating the training sample with the conventional microscopy illumination mode.

[0011] In the context of this disclosure, the word "training" in "training samples" should be interpreted as the samples used during the training of a machine learning model, not as general samples that a trained machine learning model can use to construct its digital images. The constructed digital images can then depict general samples. A machine learning model can certainly be able to construct digital images depicting training samples. However, a machine learning model can also be able to construct digital images of other samples that are not part of the digital image training set. In other words, a machine learning model, after training, can be able to construct digital images of samples that were not used to train the machine learning model. Depending on the context, the term "sample" may be used herein to refer only to samples used during training or samples input into a trained machine learning model.

[0012] In the context of this disclosure, the term "illumination mode" can be interpreted as different ways of illuminating a sample using an illumination system. Each illumination mode can be formed, for example, by illuminating the sample simultaneously from one or more of a plurality of directions and / or by varying the number of light sources emitting light in the illumination system.

[0013] In the context of this disclosure, the term "real data" should be interpreted as information that is known to be true and / or consistent with the facts. Therefore, in this context, since machine learning models are trained to construct digital images using real data and a training set of digital images, real data can represent digital images depicting the training samples.

[0014] In the context of this disclosure, the term "conventional microscopy illumination mode" should be interpreted as an illumination mode similar to (or identical to) conventional microscope illumination. In other words, a conventional microscopy illumination mode can be an illumination mode similar to (or identical to) illumination provided by a conventional light source suitable for microscopy. A conventional microscopy illumination mode can be formed by a conventional light source (e.g., a conventional light source suitable for microscopy). A conventional microscopy illumination mode can be an illumination mode suitable for capturing conventional microscopic images of a training sample. For example, a conventional microscopy illumination mode can include one or more of brightfield illumination, darkfield illumination, and phase-contrast illumination modes. It should be further understood that a conventional microscopy illumination mode can illuminate the front (i.e., the side of the training sample facing the second microscope objective) and / or the back (i.e., the side of the training sample opposite to the front).

[0015] In the context of this disclosure, the term "numerical aperture of a microscope objective" should be interpreted as a dimensionless number associated with the angular range of light received by the microscope objective. Therefore, a direction corresponding to an angle greater than the numerical aperture of the microscope objective can be a direction corresponding to an angle outside (e.g., greater than) the angular range of light received by the microscope objective. Generally, microscope objectives with higher numerical apertures can have higher magnification than microscope objectives with lower numerical apertures.

[0016] A machine learning model can be trained to associate a training set of digital images with real data (e.g., digital images captured and depicting training samples using a second microscope objective). The machine learning model can be trained iteratively and / or recursively until the difference between the machine learning model's output (i.e., the constructed digital image) and the real data (i.e., the digital image captured and depicting training samples using a second microscope objective) is less than a predetermined threshold. A smaller difference between the machine learning model's output and the real data indicates higher accuracy of the constructed digital image provided by the machine learning model. In other words, a smaller difference between the machine learning model's output and the real data indicates that the constructed digital image can reproduce the digital image captured and depicting the training sample using a second microscope objective to a greater extent. Therefore, preferably, the difference between the machine learning model's output and the real data can be minimized. The machine learning model can be trained to construct digital images of samples for multiple different sample types. In this case, for each sample type, the machine learning model can be trained using a training set of digital images of training samples for that sample type and the corresponding real data associated with the respective sample type.

[0017] By illuminating a training sample with multiple illumination modes and capturing digital images for each of these modes, phase information (often referred to in the art as quantitative phase) associated with the training sample can be determined. This can be understood as capturing information about different parts of the Fourier space (i.e., the spatial frequency domain) associated with the training sample for different illumination directions. This technique can be referred to in the art as Fourier stacked microscopy (FPM). The phase information can then be used to reproduce the training sample at different focal positions (i.e., the position of the training sample relative to the first microscope objective used to capture the digital image of the training sample). Furthermore, information about the refractive index (or spatial distribution of the refractive index) associated with the training sample can be captured. This can be understood as the refractive effect of light depending on the angle of incidence of the light illuminating the training sample and the refractive index of the training sample. Moreover, by illuminating the training sample with multiple illumination modes and capturing digital images for each of these modes, information about finer details of the training sample can be captured, details that are typically resolvable by microscopes using, for example, conventional light sources. Therefore, even when acquiring a digital image training set using a first microscope objective with a numerical aperture smaller than that of the second microscope objective, the digital image training set still includes information that can be used by a trained machine learning model to construct digital images depicting the training samples, similar to digital images captured using the second microscope objective (i.e., with higher resolution). This is because the training samples are illuminated with multiple illumination modes when acquiring the training set.

[0018] Therefore, since the digital image training set includes information associated with one or more of the following: fine details of the training sample, the refractive index associated with the training sample, and phase information associated with the training sample, this information can be used to train a machine learning model. This allows the machine learning model to be trained to construct a digital image of the training sample at a relatively higher resolution than is permitted when using a first microscope objective and a single illumination mode (e.g., conventional microscopy illumination or from a single direction) or when capturing the digital image training set using conventional microscopy. Using conventional microscopy, it may be difficult or even impossible to capture the information associated with the following: the refractive index associated with the training sample and / or the phase information associated with the training sample. In other words, since a digital image captured using conventional microscopy may simultaneously contain information about the refraction of light incident from a direction corresponding to the overlapping portion of the Fourier space associated with the training sample, it may be impossible to determine the phase information associated with the training sample and / or the information related to the refractive index of the training sample using this technique. In other words, it may be impossible to determine the information associated with the training sample using conventional microscopy. Illuminating the training sample with multiple illumination modes further allows for the capture of information related to the details of the training sample that are finer than what the first microscope objective used to capture the digital image training set of the training sample typically allows (using conventional microscopic illumination). Therefore, microscope objectives with relatively low magnification (i.e., low numerical aperture) can be used while still being able to capture information related to this fine detail of the training sample. Using microscope objectives with relatively low magnification further allows for the capture of a larger portion of the digital image of the training sample at each imaging location. Thus, by capturing digital images at relatively fewer imaging locations, the entire training sample can be scanned, which in turn allows for faster and / or more efficient scanning of the training sample.

[0019] Each of the multiple illumination patterns can be formed by illuminating the training sample using one or more of the multiple light sources.

[0020] The related advantage is that it allows for more reliable and / or faster switching between different lighting modes compared to using a single movable light source that can illuminate a location from different directions.

[0021] One or more of the plurality of light sources can be configured to emit multicolor light; and each of the plurality of illumination patterns can be formed by illuminating the training sample with multicolor light.

[0022] In the context of this disclosure, the term "polychromatic light" should be interpreted as light having a spectrum that includes a relatively wide range of wavelengths. The spectrum of polychromatic light can be continuous. The spectrum of polychromatic light can be at least partially continuous. The spectrum of polychromatic light can include a relatively broad spectrum. The spectrum of polychromatic light can be discontinuous. The spectrum of polychromatic light can include a combination of a relatively broad spectrum and a relatively narrow spectrum. Examples of polychromatic light can be white light, quasi-white light, and / or pseudo-white light. White light, quasi-white light, and / or pseudo-white light can, for example, resemble sunlight. In contrast to light with a narrower spectrum (e.g., monochromatic light or quasi-monochromatic light), the spectrum of white light can include most of the visible light spectrum. Polychromatic light can have a spectral width of 100 nm or greater. Polychromatic light can have a spectral width of 400 nm or less. Narrow-spectrum light can have a spectral width of 50 nm or less. For example, narrow-spectrum light can have a spectral width of less than 20 nm. Here, "spectral width" can be the wavelength range of light emitted by one or more light sources. For example, spectral width can be the range of most (or all) wavelengths of light emitted by one or more light sources. Spectral width can be full width at half maximum (FWHM) or any other suitable metric used in the art.

[0023] A related advantage is that multicolor light (e.g., white light, quasi-white light, or pseudo-white light) allows for the capture of more information about the training samples in the digital image training set, information that would otherwise be missed in uncovered spectral portions when using narrowband light sources. This information can then be used by a trained machine learning model when constructing a digital image depicting the sample (which can be even more similar to the sample). For example, by using multicolor light (e.g., white light, quasi-white light, or pseudo-white light) when acquiring the digital image training set, the trained machine learning model may be able to construct a color digital image depicting the sample. In imaging applications utilizing Fourier overlay imaging, light with a spectral width of approximately 50 nm or less (e.g., 20 nm or less) is often required. Therefore, in existing systems, it may be necessary to illuminate the sample sequentially and / or individually with several different colors of narrowband light (e.g., monochromatic or quasi-monochromatic light) to capture multispectral information about the sample. Thus, using multicolor light reduces the total number of digital images required to capture multispectral information about the sample compared to existing systems utilizing Fourier overlay imaging. Furthermore, using monochromatic or quasi-monochromatic light of different colors may only allow capturing information about samples within those specific wavelength ranges, while using multicolor light, due to its potentially broader spectrum, allows capturing information about samples across a wider wavelength range compared to using monochromatic or quasi-monochromatic light.

[0024] Each of the multiple light sources can be configured to illuminate the training sample from one of the multiple directions.

[0025] A related advantage is that digital images of training samples can be captured under different lighting conditions (i.e., with light from different directions), thereby allowing more information about the training samples (e.g., information related to refractive index, etc.) to be captured in the captured digital images.

[0026] At least one of the multiple directions may correspond to an angle greater than the numerical aperture of the first microscope objective.

[0027] By illuminating the training sample from an angle corresponding to an angle larger than the numerical aperture of the image-forming components of the first imaging system, the digital image captured for this illumination angle can include information about higher spatial frequencies of the training sample, and thus include finer details of the training sample than typically allowed by the first microscope objective (e.g., using conventional microscopic illumination). This, in turn, allows the capture of phase information associated with the training sample and / or information related to details of the training sample that are typically indistinguishable by the first microscope objective, which can be used to train a machine learning model. In other words, illuminating the training sample from an angle corresponding to an angle larger than the numerical aperture of the first microscope objective allows for the construction of improved machine learning models capable of depicting digital images of the sample.

[0028] Two or more of the multiple directions may correspond to the overlapping portion of the Fourier space associated with the training sample, and each illumination pattern may be formed by illuminating the training sample from one or more directions corresponding to the non-overlapping portion of the Fourier space associated with the training sample.

[0029] A related advantage is that training samples can be illuminated simultaneously from several directions. In other words, the amount of information associated with the training samples captured in each captured digital image can be increased. This, in turn, reduces the number of digital images required while still allowing enough information associated with the training samples to be captured (i.e., enough for the training process and / or the process of constructing digital images using a trained machine learning model). Reducing the number of digital images that need to be captured reduces the time required for image capture, which can make the system more time-efficient in constructing digital images depicting the samples. Furthermore, reducing the number of captured digital images also reduces the memory requirements associated with the training process (e.g., the memory required to store the captured digital images). For the same reason, the memory requirements associated with the process of constructing digital images depicting the samples using a trained machine learning model can also be reduced.

[0030] Furthermore, by simultaneously illuminating the training sample from directions corresponding to the non-overlapping portions of the Fourier space associated with the training sample, different portions of the Fourier space can be sampled simultaneously. This, in turn, can reduce (or even avoid) the entanglement (mixing) of information from different directions (i.e., information associated with the training sample).

[0031] Since two or more of the multiple directions can correspond to the overlapping portions of the Fourier space associated with the training samples, the directions forming different illumination patterns can also correspond to the overlapping portions of the Fourier space associated with the training samples. This, in turn, allows for the capture of larger and potentially continuous information related to the Fourier space associated with the training samples in the digital images of the digital image training set.

[0032] According to a second aspect, a method for constructing a digital image depicting a sample is provided. The method includes: receiving a digital image input set of the sample, wherein the digital image input set is obtained by illuminating the sample with a plurality of illumination modes and capturing digital images for each of the plurality of illumination modes; constructing a digital image depicting the sample by: inputting the digital image input set into a machine learning model trained according to the method of the first aspect, and receiving an output including the digital image depicting the sample from the machine learning model.

[0033] A microscope objective having a numerical aperture similar to or the same as that of the first microscope objective used to acquire the digital image training set can be used to acquire the digital image input set of the sample. The multiple illumination modes used when acquiring the digital image input set of the sample can be similar to or the same as the multiple illumination modes used to acquire the digital image training set.

[0034] By using a machine learning model trained according to the method of the first aspect, a digital image depicting the sample can be constructed with a higher resolution than that typically achievable with microscope objectives (e.g., using conventional illumination) used to acquire the digital image input set. This, in turn, allows the use of microscope objectives with lower numerical apertures for acquiring the input set. The multiple illumination modes used when acquiring the digital image input set can be substantially the same (or even identical) to the multiple illumination modes used when acquiring the digital image training set for training the machine learning model according to the method of the first aspect. The multiple illumination modes used when acquiring the digital image input set can be a subset of the multiple illumination modes used when acquiring the digital image training set for training the machine learning model according to the method of the first aspect.

[0035] Another related advantage is that trained machine learning models may be able to construct digital images depicting a sample from a set of digital images captured without the use of microscope oil immersion and / or coverslips. In other words, trained machine learning models may be able to construct digital images depicting a sample without reducing the refractive index variation within the sample. Typically, conventional FPM (i.e., reconstructing digital images depicting a sample by iteratively transforming the digital image input set into Fourier space) may perform poorly without the use of microscope oil immersion and / or coverslips. This is because conventional FPM assumes the sample is very thin (i.e., thin compared to the wavelength of the light illuminating the sample), and surface roughness can affect the thin sample assumption. Furthermore, without the use of microscope oil immersion and / or coverslips, the sample may be in direct contact with air. This can be problematic because the refractive index difference between air and the sample can be relatively large, and this difference can negatively impact the performance of conventional FPM. Generally, conventional FPM may perform better when the phase shift between the sample and its surrounding environment is relatively small. Without the use of microscope immersion oil and / or coverslips, the refractive index variation within a sample can become too large (i.e., corresponding to a phase shift greater than 2π), potentially causing the iterative process for reconstructing digital images to fail. It has been found that by using a machine learning model trained according to the first aspect, this requirement for refractive index variation is less stringent, thus allowing the capture of digital image input sets without the use of microscope immersion oil and / or coverslips.

[0036] The aforementioned features of the first aspect also apply to the second aspect where applicable. To avoid excessive repetition, refer to the above.

[0037] According to a third aspect, an apparatus is provided for training a machine learning model to construct digital images depicting a sample. The apparatus includes circuitry configured to perform: a first receiving function configured to receive a training set of digital images of a training sample, wherein the training set is acquired using a first microscope objective by illuminating the training sample with a plurality of illumination modes and capturing a digital image for each of the plurality of illumination modes; a second receiving function configured to receive real data comprising digital images depicting the training sample, wherein the digital images of the real data are acquired using a second microscope objective by illuminating the training sample with a conventional microscopy illumination mode and capturing the digital image while illuminating the training sample with the conventional microscopy illumination mode; and a training function configured to train a machine learning model using the received digital image training set and the received real data to construct digital images depicting the training sample; and wherein the second microscope objective has a higher numerical aperture than the first microscope objective, whereby the resolution of the digital images of the real data is higher than the resolution of at least one digital image in the digital image training set.

[0038] The device can be configured to train a machine learning model according to the method of the first aspect.

[0039] The features described above in the first and / or second aspects also apply to this third aspect where applicable. To avoid excessive repetition, refer to the preceding text.

[0040] According to a fourth aspect, a microscope system is provided. The microscope system includes: an illumination system configured to illuminate a sample with multiple illumination modes; an image sensor; a microscope objective arranged to image the sample onto the image sensor; and circuitry configured to perform: an acquisition function configured to acquire a set of digital image inputs, which is achieved by being configured to: control the illumination system to illuminate the sample with each of the multiple illumination modes, and control the image sensor to capture digital images for each of the multiple illumination modes; and the circuitry is further configured to perform: an image construction function configured to: input the set of digital image inputs into a machine learning model trained according to the method of the first aspect, and receive from the machine learning model an output including a constructed digital image depicting the sample.

[0041] A lighting system may include multiple light sources.

[0042] One or more of the plurality of light sources can be configured to emit polychromatic light; and each of the plurality of illumination modes can be formed by illuminating the sample with polychromatic light.

[0043] Each of the multiple light sources can be configured to illuminate the sample from one of the multiple directions.

[0044] At least one of the multiple directions can correspond to an angle larger than the numerical aperture of the microscope objective.

[0045] Two or more of the multiple directions may correspond to the overlapping portion of the Fourier space associated with the sample; and wherein the acquisition function may be configured to control the illumination system such that each illumination pattern is formed by illuminating the sample from one or more directions corresponding to the non-overlapping portion of the Fourier space associated with the sample.

[0046] The features described above in the first, second, and / or third aspects also apply to this fourth aspect where applicable. To avoid excessive repetition, refer to the preceding text.

[0047] According to a fifth aspect, a non-transitory computer-readable storage medium is provided. This non-transitory computer-readable storage medium includes program code portions that, when executed on a device with processing capabilities, perform the method according to the first aspect or the method according to the second aspect.

[0048] The features described above in the first, second, third, and / or fourth aspects also apply to this fifth aspect where applicable. To avoid excessive repetition, refer to the preceding text.

[0049] Further features and advantages of the inventive concept will become clear upon reading the appended claims and the following description. Those skilled in the art will recognize that different features of the inventive concept can be combined to produce variations beyond those described below without departing from its scope. Attached Figure Description

[0050] Various aspects of the inventive concept, including its particular features and advantages, will be readily understood from the following detailed description and accompanying drawings, in which:

[0051] Figure 1A A schematic diagram of a microscope system is shown, which is suitable for acquiring a digital image training set for training a machine learning model and / or a digital image input set for constructing digital images depicting a sample using a trained machine learning model.

[0052] Figure 1B A schematic diagram of a second microscope objective and an additional illumination system suitable for acquiring digital images of real data is shown.

[0053] Figure 2A schematic diagram of a device used to train a machine learning model to construct a digital image depicting a sample is shown.

[0054] Figure 3A It is a block diagram of a method for training machine learning models to construct digital images that depict samples.

[0055] Figure 3B This is a block diagram of the steps involved in obtaining a training set of digital images for training machine learning models.

[0056] Figure 3C It is a block diagram of the method steps for obtaining digital images of real data used to train machine learning models.

[0057] Figure 4A This is a block diagram of a method for constructing digital images depicting a sample.

[0058] Figure 4B This is a block diagram of the method steps for obtaining a digital image input set used to construct a digital image depicting the sample.

[0059] Figure 4C This is a block diagram of the method steps for constructing a digital image depicting a sample.

[0060] Figure 5 This is a schematic diagram of a non-transitory computer-readable storage medium. Detailed Implementation

[0061] The inventive concept will now be described more fully below with reference to the accompanying drawings, in which presently preferred variations of the inventive concept are shown and discussed. However, the inventive concept can be embodied in many different forms and should not be construed as being limited to the variations set forth herein; rather, these variations are provided to achieve thoroughness and completeness and to fully communicate the scope of the inventive concept to those skilled in the art. As shown in the drawings, features can be enlarged for illustrative purposes, and thus the overall structure for illustrating variations of the inventive concept can be provided. Throughout the specification, the same reference numerals refer to the same elements.

[0062] Now refer to Figure 1A and Figure 1B A microscope system 10 is described that is suitable for acquiring a set of digital images, which can be used as a training set of digital images when training a machine learning model to construct digital images depicting a sample, and / or as an input set of digital images when constructing digital images depicting a sample using a trained machine learning model.

[0063] Figure 1A This is a schematic diagram of microscope system 10. (As shown) Figure 1AAs shown, the microscope system 10 includes an imaging system 100 and a circuit 110. The imaging system 100 includes an image sensor 102, a microscope objective lens 104, and an illumination system 106.

[0064] The microscope system 10 may further include a sample positioning component 108. The sample positioning component 108 may be configured to hold a sample 1080. The sample 1080 may be a biological sample. The sample 1080 may be a cytological sample. Examples of biological samples include, but are not limited to, blood, plasma, bone marrow fluid, etc. The sample may also be a non-biological sample. Non-biological samples may be artificial or naturally occurring. Examples of non-biological samples include, but are not limited to, integrated circuits, optical components, microstructures, minerals, metals, etc. The sample positioning component 108 may be configured to move the sample 1080. The sample positioning component 108 may be configured to move the sample 1080 along a plane. The normal to this plane may be substantially parallel to the optical axis 101 of the microscope objective 104. It should be further understood that the sample positioning component 108 may be configured to move the sample in a direction substantially parallel to the optical axis 101 of the microscope objective 104. In other words, the sample positioning component 108 may be configured to move the sample 1080 in the focusing direction of the microscope objective 104. The circuitry 110 may be configured to control the sample positioning component 108. For example, circuit 110 can be configured to perform sample positioning function 1124. Sample positioning function 1124 can be configured to use sample positioning component 108 to control the position of sample 1080.

[0065] Although circuit 110 is in Figure 1A While shown as a separate entity, it should be understood that circuitry 110 can form part of an electronic device. The electronic device may be, for example, a computer, server, smartphone, etc. The electronic device can be a local electronic device (i.e., located near the imaging system 100) or a remote electronic device. Non-limiting examples of remote electronic devices may include servers, cloud servers, remote computers, remote smartphones, etc. It should be further understood that the functionality of circuitry 110 can be distributed across more than one electronic device. The electronic device may include additional components such as input devices (mouse, keyboard, touchscreen, etc.) and / or a display. Figure 1AAs shown, circuitry 110 may include one or more of a memory 112, a processing unit 114, a communication interface 116, and a data bus 118. The memory 112, processing unit 114, and communication interface 116 may communicate (e.g., exchange data) via the data bus 118. Processing unit 114 may include a central processing unit (CPU) and / or a graphics processing unit (GPU). Communication interface 116 may be configured to communicate with external devices. For example, communication interface 116 may be configured to communicate with servers, computers, external peripheral devices (e.g., external storage devices), etc. External devices may be local devices or remote devices (e.g., cloud servers). Communication interface 116 may be configured to communicate with external devices via an external network (e.g., a local area network, the Internet, etc.). Communication interface 116 may include a transceiver. Communication interface 116 may be configured for wireless and / or wired communication. Technologies suitable for wireless communication are known to those skilled in the art. Some non-limiting examples include Wi-Fi and Near Field Communication (NFC). Technologies suitable for wired communication are known to those skilled in the art. Some non-limiting examples include USB, Ethernet, and FireWire.

[0066] Memory 112 may be a non-transitory computer-readable storage medium. Memory 112 may be random access memory. Memory 112 may be non-volatile memory. For example... Figure 1A As illustrated in the example, memory 112 may store program code portions 1120, 1122, 1124, 1126, and 1128 corresponding to one or more functions. Program code portions 1120, 1122, 1124, 1126, and 1128 may be executed by processing unit 114, which thereby performs the functions. Therefore, when reference is made that circuit 110 is configured to perform a specific function, processing unit 114 may execute program code portions 1120, 1122, 1124, 1126, and 1128 corresponding to the specific function, which may be stored in memory 112. However, it should be understood that one or more functions of circuit 110 may be implemented in hardware and / or in a specific integrated circuit. For example, one or more functions may be implemented using a field-programmable gate array (FPGA). Therefore, one or more functions of circuit 110 may be implemented in hardware or software or a combination of both.

[0067] Although image sensor 102 is Figure 1AWhile shown as a separate entity, it should be understood that image sensor 102 can be part of a camera. Image sensor 102 can include, for example, a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor. Image sensor 102 can capture digital color images. For this purpose, image sensor 102 may include a color filter (e.g., a Bayer filter) to allow image sensor 102 to capture color information of the light incident on image sensor 102. Image sensor 102 may be able to capture digital grayscale images, and in this case, image sensor 102 may not include a color filter. Figure 1A As shown in the example, image sensor 102 can communicate with circuit 110 via communication interface 116. However, it should be understood that image sensor 102 can also communicate with circuit 110 via data bus 118.

[0068] Microscope objective 104 is arranged to image sample 1080 onto image sensor 102. For example, microscope objective 104 may be positioned such that the object plane of microscope objective 104 coincides with sample 1080 and the image plane of microscope objective 104 coincides with image sensor 102. It should be understood that whether an image depicting sample 1080 is formed on image sensor 102 may depend on the illumination of sample 1080. For example, if sample 1080 is illuminated using conventional microscope illumination (e.g., bright field illumination), an image similar to sample 1080 can be formed on image sensor 102. However, if sample 1080 is illuminated with light from only a few directions or even a single direction, the image formed on image sensor 102 may be dissimilar to sample 1080. For example, sample 1080 may be illuminated with light from one or more directions corresponding to an angle greater than the numerical aperture 1040 of microscope objective 104. In this case, the image formed on image sensor 102 may be dissimilar to sample 1080. However, even if the images may not resemble (or depict) sample 1080, they may include information associated with sample 1080, and this information may be used when training a machine learning model and / or when constructing digital images depicting the sample using a trained machine learning model.

[0069] Microscope objective 104 is movable along a direction Z substantially parallel to the optical axis 101 of microscope objective 104. In other words, microscope objective 104 is movable in the focusing direction. Microscope objective 104 can be moved along direction Z by coupling to a manual and / or motorized stage (not shown). Microscope objective 104 and / or sample positioning assembly 108 can be movable, allowing image sensor 102 to capture a focused image of sample 1080. The position of microscope objective 104 along direction Z can be controlled by circuitry 110. For example, circuitry 110 can be configured to execute focusing function 1126, which is configured to adjust the position of microscope objective 104 along direction Z. Focusing function 1126 can be configured to automatically adjust the position of microscope objective 104 along direction Z. In other words, focusing function 1126 can be an autofocus function. Figure 1A As shown in the example, the focusing function 1126 can control the position of the microscope objective 104 along the Z direction via communication interface 116. However, it should be understood that the focusing function 1126 can also control this position via communication via data bus 118.

[0070] Although not in Figure 1A The example shown is for illustrative purposes only; however, it should be understood that imaging system 100 may include additional components, such as one or more of apertures, lenses, windows, color filters, etc. For example, imaging system 100 may further include a relay lens 185. The relay lens 185 may be aligned with microscope objective 104.

[0071] The illumination system 106 is configured to illuminate the sample 1080 with multiple illumination modes. The illumination system 106 may include multiple light sources 1062. The multiple light sources 1062 may include one or more adjustable light sources. For example, the color and / or intensity of the light emitted by the one or more adjustable light sources may be adjustable. For example, the circuitry may be configured to control one or more adjustable light sources. Figure 1A As shown in the example, multiple light sources 1062 can be arranged on a curved surface 1064. For example... Figure 1AAs illustrated in the example, the curved surface 1064 can be concave along at least one direction of the surface 1064. For example, the curved surface 1064 can be a cylinder. The curved surface 1064 can be concave along two perpendicular directions of the surface. For example, the curved surface 1064 can have a shape similar to a spherical cross section. The spherical cross section can be a spherical cap or a spherical dome. It may be advantageous to arrange multiple light sources 1062 on the curved surface 1064 because the distance R from each light source to the current imaging position P of the imaging system 100 is likely to be similar. Because this distance R is similar, the intensity of the light emitted from each of the multiple light sources 1062 can be similar at the current imaging position P. This can be understood as an effect of the inverse square law. Therefore, the sample 1080 can be illuminated by light with similar intensity for each of the multiple directions 1060, which in turn allows the sample 1080 to be illuminated more similarly independent of the illumination direction. The distance R from each light source to the current imaging position P can be in the range of 4 cm to 15 cm. It may be advantageous to configure the illumination system 106 such that the distance R from each light source to the current imaging position P is large enough that each light source can be considered a point light source. Therefore, given that the intensity of light from each light source at the current imaging position is high enough to produce a digital image set, the distance R from each light source to the current imaging position P can be greater than 15 cm. However, it should be understood that the multiple light sources 1062 can be arranged on a flat surface or on a surface with an irregular shape. It should be further understood that... Figure 1A A cross-section of the microscope system 10 is shown, particularly a cross-section of the illumination system 106. Therefore, Figure 1AThe curved surface 1064 of the illustrated lighting system 106 may be part of a cylinder or a sphere (or quasi-sphere). The curved surface 1064 of the lighting system 106 may be bowl-shaped. The curved surface 1064 may be formed by facets (not shown). In other words, the curved surface 1064 may be formed by multiple flat surfaces. Therefore, the curved surface 1064 may be segmented flat. The curved surface 1064 may be part of a quasi-sphere comprising multiple facets or segments. Therefore, the curved surface 1064 may be part of a polyhedral surface. An example of such a polyhedron may be a truncated icosahedron. Multiple light sources 1062 may be arranged on the facets. Each light source may be arranged such that the light source is configured to emit light in a direction substantially parallel to the normal of the associated facet. Each facet may be a flat surface having at least three edges. For example, the curved surface 1064 may be formed by facets having five edges and facets having six edges (e.g., similar to the inner surface of a soccer ball or football). It should be understood that each facet may be a separate entity. Therefore, the curved surface 1064 can be formed from multiple portions, and each facet can be formed from one or more portions. It should be further understood that each portion can include one or more facets. Furthermore, such portions can be arranged to contact adjacent portions, or can be arranged to be at a distance from adjacent portions. A single portion can include all facets.

[0072] One or more of a plurality of light sources can be configured to emit multicolor light. One or more of a plurality of light sources can be configured to emit narrowband light. Each of a plurality of illumination modes can be formed by illuminating a sample with multicolor light and / or narrowband light. One of the plurality of light sources may include one or more light emitters. Each of the plurality of light sources may include one or more light emitters. The light emitter may be a light-emitting diode (LED). For example, one of the plurality of light sources may include one or more LEDs. The light emitter may be configured to emit multicolor light and / or narrowband light (e.g., monochromatic or quasi-monochromatic light). Multicolor light can be generated by emitting light simultaneously (or at least partially simultaneously) from two or more light emitters, each light emitter being configured to emit narrowband light (e.g., light of different colors, monochromatic light, and / or quasi-monochromatic light). In other words, one or more of the light sources may include two or more light emitters, wherein a first light emitter of the two or more light emitters may be configured to emit light having a first spectrum, and a second light emitter of the two or more light emitters may be configured to emit light having a second spectrum different from the first spectrum, thereby the spectrum of the multicolor light may include the first spectrum and the second spectrum. The first and second spectra may partially overlap. The first and second spectra may not overlap. The spectra of polychromatic light may be continuous or discontinuous. It should be understood that a light source may include multiple light emitters. Each of the multiple light emitters may be configured to emit light with a corresponding spectrum. The spectra of the light emitted by the multiple light emitters may partially overlap and / or not overlap. In this document, polychromatic light should be understood as light having a spectrum including a relatively wide wavelength range, while narrow-spectrum light should be understood as light having a relatively narrow spectrum (i.e., including a relatively narrow wavelength range). For example, one or more of white light, quasi-white light, pseudo-white light, and broadband light may be examples of polychromatic light. Monochromatic light and / or quasi-monochromatic light may be examples of narrow-spectrum light. Polychromatic light may have a spectral width of 100 nm or greater. Polychromatic light may have a spectral width of 400 nm or less. In this case, polychromatic light may correspond to electromagnetic radiation having wavelengths within the visible spectrum. Narrow-spectrum light may have a spectral width of 50 nm or less. For example, narrow-spectrum light may have a spectral width of 20 nm or less. Multicolor light can be emitted from an LED configured to emit white light. An LED configured to emit white light can be referred to as a white LED. A white LED can be formed, for example, from a blue LED coated with a layer of fluorescent material that emits white light when illuminated. By using multicolor light, the digital image set can include information about the sample 1080 associated with a relatively wide wavelength range, especially compared to using narrowband light (e.g., lasers, monochromatic LEDs, etc.). This, in turn, allows machine learning models to use more information about the sample 1080 during training and / or when constructing digital images depicting the sample.LEDs can be any type of LED, such as ordinary LED bulbs (i.e., conventional and inorganic LEDs), graphene LEDs, or LEDs commonly found in displays (e.g., quantum dot LEDs (QLEDs) or organic LEDs (OLEDs)). However, other types of LEDs can also be used.

[0073] Each of the plurality of light sources 1062 can be configured to illuminate the sample 1080 from one of the plurality of directions 1060. At least one direction 1061 of the plurality of directions 1060 can correspond to an angle greater than the numerical aperture 1040 of the microscope objective 104 of the imaging system 100. The numerical aperture 1040 of the microscope objective 104 can be a dimensionless number associated with the angular range of light received by the microscope objective 104. Therefore, the direction 1061 corresponding to an angle greater than the numerical aperture 1040 of the microscope objective 104 (e.g., Figure 1AThe direction 1064 in the image can be a direction 1061 corresponding to an angle outside (e.g., greater than) the angular range of light received by the microscope objective 104. Because the sample 1080 is illuminated from multiple different directions, information about finer details of the training sample can be captured, details that are finer than those typically resolved by the microscope objective 104 used to image the sample 1080. This can be understood as capturing information about different parts of the Fourier space (i.e., the spatial frequency domain) associated with the sample 1080 for different illumination directions. This technique may be referred to in the art as Fourier stack imaging. Typically, in Fourier stack imaging, when the sample is illuminated from a direction corresponding to a large incident angle, high spatial frequencies in the Fourier space associated with that sample can be sampled. Therefore, when the sample 1080 is illuminated from a direction 1061 corresponding to an angle greater than the numerical aperture 1040 of the microscope objective 104, even higher spatial frequencies in the Fourier space associated with the sample 1080 can be sampled. This is possible because light is scattered by sample 1080, and a portion of the scattered light can be collected by microscope objective 104. Illuminating sample 1080 from multiple directions 1060 further allows microscope objective 104 to capture information about the refractive index (or spatial distribution of the refractive index) associated with sample 1080. This can be understood as the refractive effect of light depending on the angle of incidence of the light illuminating sample 1080 and the refractive index of sample 1080. Information about the refractive index of sample 1080 can then allow the determination of phase information associated with sample 1080 (commonly referred to in the art as quantitative phase). It should be understood that information related to one or more of the following can be captured by illuminating sample 1080 at a time from more than one of the multiple directions 1060 (e.g., from a subset of the multiple directions 1060): finer details of sample 1080, the refractive index associated with sample 1080, and the phase information associated with sample 1080. A subset of the multiple directions may include directions corresponding to the non-overlapping portions of the Fourier space of sample 1080. However, two or more of the multiple directions 1060 may correspond to the overlapping portion of the Fourier space associated with the sample 1080. For example, two adjacent light sources among the multiple light sources 1062 may be configured to emit light in directions corresponding to the overlapping portion of the Fourier space associated with the sample 1080.

[0074] The function that circuit 110 is configured to perform depends on whether the digital image set is used to train a machine learning model (i.e., as a digital image training set) or to construct digital images depicting the sample using the trained machine learning model (i.e., as a digital image input set for the trained machine learning model). However, for both purposes, circuit 110 is configured to perform acquisition function 1120. Acquisition function 1120 is configured to acquire the digital image set. For this purpose, acquisition function 1120 is configured to control illumination system 106 to illuminate sample 1080 with each of a plurality of illumination modes, and to control image sensor 102 to capture digital images for each of the plurality of illumination modes. Each digital image in the digital image set may be associated with one of the plurality of illumination modes. Each of the plurality of illumination modes may be formed, for example, by simultaneously illuminating sample 1080 from one or more of a plurality of directions 1060, and / or by changing the number of light sources emitting light from the plurality of light sources 1062 of illumination system 106. Acquisition function 1120 may be configured to control and / or adjust one or more of the plurality of light sources. Each of a plurality of illumination patterns can be formed by illuminating the sample 1080 from one or more directions corresponding to the non-overlapping portion of the Fourier space associated with the sample 1080. Acquisition function 1120 can be configured to control illumination system 106 such that each illumination pattern is formed by illuminating the sample 1080 from one or more directions corresponding to the non-overlapping portion of the Fourier space associated with the sample 1080. For example, each of a plurality of illumination patterns can be formed by illuminating the sample 1080 from only one of a plurality of directions 1060. Each of a plurality of illumination patterns can be formed by one or more light sources 1062. In other words, each of a plurality of illumination patterns can be formed by simultaneously emitting light from one or more of a plurality of light sources 1062. However, two different illumination patterns among a plurality of illumination patterns can be formed by illuminating the sample 1080 from one or more directions corresponding to at least a partial overlap of the Fourier space associated with the sample 1080. For example, a first illumination mode can be formed by illuminating the sample 1080 from a first set of directions corresponding to the non-overlapping portion of the Fourier space associated with the sample 1080, and a second illumination mode can be formed by illuminating the sample 1080 from a second set of directions corresponding to the non-overlapping portion of the Fourier space associated with the sample 1080. One direction in the first set and one direction in the second set can correspond to the overlapping portion of the Fourier space associated with the sample 1080. By including multiple light sources 1062, the illumination system 106 of the imaging system 100 can switch between different illumination modes faster and / or more reliably compared to an illumination system that includes movable light sources capable of illuminating a location from different directions.The sample 1080 can be illuminated under different lighting conditions (i.e., by using different lighting modes), and more information about the sample 1080 (e.g., information related to refractive index, phase, and / or finer details) can be collected by capturing digital images of the sample when it is illuminated under different lighting conditions.

[0075] Because the acquired digital image set can include information associated with one or more of the following: fine details of sample 1080, the refractive index associated with sample 1080, and phase information associated with sample 1080, this information can be used by a machine learning model (whether during training or when constructing a digital image depicting the sample using a trained machine learning model). This, in turn, allows a trained machine learning model to construct a digital image depicting the sample at a higher resolution than is allowed when capturing the digital image set from only one direction. In other words, this allows a trained machine learning model to construct a digital image depicting the sample at a higher resolution than is allowed when capturing the digital image set from only one direction or using a conventional microscope (e.g., using bright-field illumination).

[0076] When training a machine learning model to construct digital images depicting a sample, the acquired set of digital images can be used as a digital image training set. In this context, sample 1080 can be referred to as the training sample. In this case, circuit 110 is configured to perform a receiving function that is configured to receive real data including digital images depicting the training sample 1080. However, the digital images of the real data are acquired using a second microscope objective 204. Second microscope objective 204. In the following text, Figure 1B The microscope objective 104 can be referred to as the first microscope objective 104. The second microscope objective 204 has a higher numerical aperture than the first microscope objective 104. Figure 1B This is a schematic diagram of the second microscope objective lens 204. Although Figure 1A While not explicitly shown, it is understood that additional optics can be used with the second microscope objective 204. As a specific example, the second microscope objective 204 can be used in combination with a relay lens (similar to...). Figure 1B The first microscope objective 104 and the relay lens 185). Figure 1B In the example, training sample 1080 and sample positioning assembly 108 are positioned between the second microscope objective 204 and the additional illumination system 206. Figure 1AAs illustrated in the example, the additional illumination system 206 may include a conventional light source suitable for microscopy (e.g., conventional microscopy). The additional illumination system 206 may be configured to illuminate the training sample 2080 using conventional microscopy illumination (e.g., bright-field illumination, dark-field illumination, etc.). It should be understood that the additional illumination system 206 may be configured to illuminate the front side (i.e., the side facing the second microscope objective 204) and / or the back side (i.e., the side opposite to the front side) of the training sample 1080. The primary difference between the first microscope objective 104 and the second microscope objective 204 is that the second microscope objective 204 has a higher numerical aperture than the first microscope objective 104. Therefore, the resolution of the digital image of the real data is higher than the resolution of at least one digital image in the digital image training set. The resolution of the digital image of the real data may be higher than the resolution of each digital image in the digital image training set. The numerical aperture 1040 of the first microscope objective 104 may be 0.5 or smaller. For example, the numerical aperture 1040 of the first microscope objective 104 can be 0.4 or 0.25. The numerical aperture 2040 of the second microscope objective 204 can be 0.75 or larger. For example, the numerical aperture 2040 of the second microscope objective 204 can be 1.25. It should be understood that microscope immersion oil can be used when capturing digital images of real data using the second microscope objective 204. As a non-limiting example, a numerical aperture of 0.25 (corresponding to 10x magnification) can be used to capture the training set, and a numerical aperture of 0.75 (corresponding to 40x magnification) can be used to capture digital images of real data. As another non-limiting example, a numerical aperture of 0.5 (corresponding to 20x magnification) can be used to capture the training set, and a numerical aperture of 1.25 (corresponding to 100x magnification) can be used to capture digital images of real data. It should be understood that other combinations of numerical apertures can be used when capturing digital images of training sets and real data.

[0077] It should be understood that the second microscope objective 204 and the additional illumination system 206 can form part of a separate microscope system (not shown). In this case, it is possible to use... Figure 1A The microscope system 10 acquires a digital image training set, and the training sample 1080 can then be moved to a separate microscope system so that the second microscope objective 204 can acquire digital images of the real data. The sample 1080 can be moved automatically (e.g., using motorized equipment) or manually between the microscope system 10 and the separate microscope system. However, it should be understood that... Figure 1AThe microscope system 10 may include a first microscope objective 104, an illumination system 106, a second microscope objective 204, and an additional illumination system 206. In this configuration, the sample positioning assembly 108 may be configured to move a sample 1080 to a first position where the training sample 1080 can be imaged by the first microscope objective 104 (using the illumination system 106), and to a second position where the training sample 1080 can be imaged by the second microscope objective 204 (using the additional illumination system 206). In this configuration, the microscope system 10 may include an additional image sensor (not shown) configured to capture digital images of real data. However, Figure 1A The image sensor 102 can be used to capture digital images of real data. For example, additional optics (e.g., one or more mirrors, lenses, and beam splitters) and / or additional components (e.g., a translation stage configured to move the image sensor 102) can be used for this purpose.

[0078] The first microscope objective 104 and the second microscope objective 204 are interchangeable. For example, the first microscope objective 104 and the second microscope objective 204 can be mounted on a turntable. The turntable can be rotatable. Thus, depending on the current setting of the turntable, the first microscope objective 104 or the second microscope objective 204 can be positioned to image the training sample 1080 onto the image sensor 102. In this case, the illumination system 106 and the additional illumination system 206 are interchangeable (e.g., they can be moved manually or using a motorized translation stage). The illumination system 106 and the additional illumination system 206 can form part of a single illumination system (not shown). Figure 1A The illumination system 106 and the additional illumination system 206 can be the same illumination system. In this case, the conventional microscopy illumination mode can be... Figure 1A A subset of light sources is formed in the lighting system 106. For example, Figure 1B The light source of the lighting system 106 is simultaneously along with (and may be equal to) a light source less than (and possibly ... Figure 2 The light emitted from the second microscope objective 204, whose numerical aperture 2040 corresponds to the angle of the emitted light, can form an illumination pattern similar to conventional microscopy illumination (e.g., conventional microscopy illumination mode). Additionally, a light diffuser (not shown) can be placed between the sample 1080 and the illumination system to diffuse the simultaneously emitted light. By using a light diffuser, the conventional microscopy illumination mode may be more similar to illumination provided by a conventional microscopy illumination source.

[0079] The acquired digital image set can be used as a training set for training a machine learning model to construct digital images depicting the sample. To this end, circuit 110 can be further configured to execute training function 1128. Training function 1128 can be configured to train a machine learning model using the acquired digital image training set and acquired real data to construct digital images depicting the sample. Therefore, training function 1128 can be configured to train a machine learning model to correlate the digital image training set with real data (e.g., digital images depicting the training sample). Training function 1128 can be configured to iteratively and / or recursively train the machine learning model until the difference between the output of the machine learning model (i.e., the constructed digital image) and the real data (i.e., the digital image depicting the training sample) is less than a predetermined threshold. A smaller difference between the output of the machine learning model and the real data can indicate a higher accuracy of the constructed digital image provided by the machine learning model. In other words, a smaller difference between the output of the machine learning model and the real data can indicate that the constructed digital image can reproduce to a greater extent the digital image acquired using the second microscope objective 204 and depicting the training sample 1080. Therefore, preferably, the difference between the output of the machine learning model and the real data can be minimized. The training function 1128 can be configured to train the machine learning model to construct digital images of samples for multiple different sample types. In this case, for each sample type, the machine learning model can be trained using a training set of digital images of training samples for that sample type and the corresponding real data associated with that sample type.

[0080] A trained machine learning model can use a set of digital images acquired by the first imaging system 100 to construct digital images depicting the sample with a resolution comparable to (or the same as) that of digital images acquired using the second microscope objective 204 (e.g., digital images of real data used during training). In other words, a trained machine learning model can use a set of digital images to construct digital images depicting the sample with a higher resolution than is typically achievable using the microscope objective used to capture the digital image input set (e.g., with conventional microscopic illumination).

[0081] If the acquired set of digital images is to be used as a digital image input set to utilize a trained machine learning model (e.g., in a manner similar to and / or combined with the above), Figure 3A , Figure 3B or Figure 1AThe described method (trained) constructs a digital image depicting the sample. In this case, circuit 110 can be further configured to perform image construction function 1122. Image construction function 1122 is configured to input a set of digital images into a trained machine learning model and receive from the trained machine learning model an output including a digital image depicting sample 1080. Return to Figure 1A The microscope system 10. Since the image construction function 1122 is configured to use a trained machine learning model, a digital image input set can be acquired using the first microscope objective 104 (i.e., using...). Figure 2 The first microscope objective 1122 can construct a digital image depicting the sample, similar to the digital image depicting the sample acquired using the second microscope objective 204, from a set of relatively low-resolution digital images (i.e., digital images in the digital image input set). In other words, a trained machine learning model may be able to construct a higher-resolution digital image from a relatively low-resolution set of digital images (i.e., digital images in the digital image input set). Therefore, a trained machine learning model can allow the acquisition of a digital image input set using a cheaper microscope system 10, while still being able to construct a digital image depicting the sample, similar to the digital image acquired using a more expensive microscope system. In other words, after the machine learning model has been trained, a second microscope objective may no longer be needed to construct a high-quality digital image of the sample, as the first microscope objective is sufficient to acquire the digital image input set.

[0082] Although the training of the machine learning model has been described in conjunction with the microscope system 10, it should be understood that training can be performed in a separate device 60. Figure 2 This is a schematic diagram of a device 60 used to train machine learning models to construct digital images depicting samples. Figure 2 Device 60 may be a computing device (e.g., a computer, server, cloud server, smartphone, etc.). Device 60 may include additional components, such as input devices (mouse, keyboard, touchscreen, etc.) and / or a display. Device 60 includes circuitry 610. Figure 1A The circuit 610 shown can be similarly combined Figure 1B and Figure 2 The circuit 110 is described. Therefore, Figure 1A The circuit 610 shown may include one or more of a memory 612, a processing unit 614, a communication interface 616, and a data bus 618. Figure 2 The description of the corresponding features of circuit 110 is applicable where applicable. Figure 1AThese features of circuit 610. To avoid excessive repetition, refer to the above. Circuit 610 is configured to perform a first receiving function 6120, a second receiving function 6122, and a training function 6124. The first receiving function 6120 is configured to receive a digital image training set. As described above, using Figure 1A The set of digital images acquired by the microscope system 10 shown can be used as a digital image training set. Therefore, the first receiving function 6120 can be configured to receive images from... Figure 1A The circuit 110 shown receives a digital image training set. However, it should be understood that the first receiving function 6120 can also be configured to receive the digital image training set from other sources. For example, the digital image training set can be stored on a remote device (e.g., a server, etc.) or a local device (e.g., a computer, memory 612, etc.), and the first receiving function 6120 can be configured to receive the digital image training set therefrom. The first receiving function 6120 can receive the digital image training set via a communication interface 616. As described above, the digital image training set is acquired using the first microscope objective 104 by illuminating the training sample 1080 with multiple illumination modes and capturing digital images for each of the multiple illumination modes. The second receiving function 6122 is configured to receive real data including digital images depicting the training sample 1080. The digital images of the real data are acquired using the second microscope objective 204 by illuminating the training sample 1080 with a conventional microscopy illumination mode and capturing digital images while illuminating the training sample 1080 with a conventional microscopy illumination mode. As described above, the second microscope objective 204 has a higher numerical aperture than the first microscope objective 104, thereby providing a higher resolution digital image of the real data than at least one digital image in the digital image training set. This can be achieved by combining... Figure 1B and Figure 1A The description is used to acquire digital images of real data. The second receiving function 6122 can be configured to receive digital images of real data from a microscope system used to capture digital images of real data. For example, when using... Figure 1B Microscope system 10 (with Figure 1A In the case of acquiring digital images of real data using the second microscope objective 204 shown, and possibly an additional illumination system 206, the second receiving function 6122 can be configured to receive data from... Figure 2The circuit 110 shown receives digital images of real data. However, it should be understood that the second receiving function 6122 can also be configured to receive digital images of real data from other sources. For example, digital images of real data can be stored on a remote device (e.g., a server, etc.) or a local device (e.g., a computer, memory 612, etc.), and the second receiving function 6122 can be configured to receive digital images of real data from there. The second receiving function 6122 can receive digital images of real data via communication interface 616. The training function 6124 is configured to train a machine learning model using the received digital image training set and the received real data to construct digital images depicting samples. Figure 1A The training function 6124 can be configured to be used with Figure 3A The training function 1128 trains the machine learning model in the same way. To avoid unnecessary repetition, please refer to the above.

[0083] Figure 3A This is a block diagram of a method 30 for training machine learning models to construct digital images depicting samples. Figure 3B Method 30 can be a computer-implemented method. Method 30 includes receiving a digital image training set of S300 training samples 1080. For example... Figure 3B As shown, the S310 digital image training set is acquired using the first microscope objective 104 by illuminating the S312 training sample 1080 with multiple illumination modes and capturing the S314 digital image for each of the multiple illumination modes. It should be understood that method 30 may include acquiring the S310 digital image training set. Therefore, as... Figure 3C As shown, method 30 may further include using a first microscope objective 104 and acquiring a digital image training set S310 by illuminating the training sample 1080 with multiple illumination modes and capturing a digital image S314 for each of the multiple illumination modes.

[0084] Method 30 further includes receiving, in S320, real data comprising digital images depicting the training sample 1080. (e.g.) Figure 3C As shown, a digital image of the real data S330 is acquired using a second microscope objective 204 by illuminating the training sample 1080 with conventional microscopy illumination mode (S332) and simultaneously capturing a digital image of S334. The second microscope objective 204 has a higher numerical aperture than the first microscope objective 104, thus the resolution of the digital image of the real data is higher than the resolution of at least one digital image in the digital image training set. It should be understood that method 30 may include acquiring a digital image of the real data S330. Therefore, as... Figure 3AAs shown, method 30 may further include using a second microscope objective and acquiring a digital image of the real data of S330 by illuminating the training sample 1080 of S332 with conventional microscopy illumination mode and capturing a digital image of the real data of S334 while illuminating the training sample 1080 of S332 with conventional microscopy illumination mode.

[0085] Method 30 further includes training an S340 machine learning model using the received digital image training set and the received real data to construct a digital image depicting the sample. Each of the multiple illumination patterns can be formed by illuminating the training sample 1080 using one or more of the multiple light sources 1062. The light sources can be light-emitting diodes configured to emit multicolor light. Each of the multiple illumination patterns can be formed by illuminating the training sample 1080 with multicolor light. Each of the multiple light sources 1062 can be configured to illuminate the training sample 1080 from one of the multiple directions 1060. At least one direction 1061 of the multiple directions 1060 can correspond to an angle greater than the numerical aperture 1040 of the first microscope objective 104. Two or more directions of the multiple directions 1060 can correspond to the overlapping portion of the Fourier space associated with the training sample 1080. Each of the multiple illumination patterns can be formed by illuminating the training sample 1080 from one or more directions corresponding to the non-overlapping portion of the Fourier space associated with the training sample 1080. Figure 3B , Figure 3C and / or Figure 1A Method 30 can be used Figure 1B and Figure 4A The microscope system 10 shown is used to perform this operation.

[0086] Figure 4A This is a block diagram of a method 40 for constructing a digital image depicting a sample. Figure 4B Method 40 can be a computer-implemented method. Method 40 includes receiving a digital image input set of the S400 sample. For example... Figure 4B As shown, the digital image input set is obtained by illuminating the S412 sample with multiple illumination modes and capturing a digital image of S414 for each of these illumination modes. It should be understood that method 40 may include obtaining the digital image input set of S410. Therefore, as... Figure 4A As shown, method 40 may further include acquiring the S410 digital image input set by illuminating the S412 sample with multiple illumination modes and capturing a S414 digital image for each of the multiple illumination modes. Figure 4C and Figure 4AAs shown, method 40 further includes constructing a digital image depicting the sample in S420 by inputting the digital image input set into a machine learning model trained according to the previously described method 30. Figure 4C and Figure 4A As shown, method 40 further includes receiving, in step S424, an output from the machine learning model including a digital image depicting the sample. Figure 4B , Figure 4C and / or Figure 1A Method 40 can be used Figure 1B and Figure 2 The microscope system 10 shown or using Figure 5 The device 60 shown is used to perform this action.

[0087] Figure 3A This is a schematic diagram of a non-transitory computer-readable storage medium 50. The non-transitory computer-readable storage medium 50 includes program code portions that, when executed on a device with processing capabilities, perform actions according to… Figure 4A Method 30 or shown Figure 4C and Figure 4A Method 40 is shown.

[0088] Technicians will understand machine learning, particularly how machine learning models can be trained and / or how trained machine learning models can be used. In short, however, a machine learning model can be a supervised machine learning model, such as networks like U-net or Pix2pix. A machine learning model can be a transformer-based network, such as SwinIR. A machine learning model can be a convolutional neural network. A machine learning model can be trained to predict the desired output using example input training data and real data (i.e., “correct” or “factual” outputs). In other words, real data can be used as labels for the input training data. Input training data can include data related to different outcomes, and each input training data can thus be associated with the real data associated with that particular input training data. Therefore, each input training data can be labeled with the associated real data (i.e., “correct” or “factual” outputs). A machine learning model can include multiple layers of neurons, and each neuron can represent a mathematical operation applied to the input training data. Typically, a machine learning model includes an input layer, one or more hidden layers, and an output layer. The first layer can be called the input layer. The output of each layer in the machine learning model (except the output layer) can be fed into subsequent layers, which in turn produce new outputs. The new output can be fed into further subsequent layers. The output of the machine learning model can be the output of the output layer. This process can be repeated for all layers in the machine learning model. Typically, each layer further includes an activation function. The activation function can further define the output of the neurons in that layer. For example, the activation function can ensure that the output of the layer is not too large or too small (e.g., tending towards positive or negative infinity). Furthermore, the activation function can introduce nonlinearity into the machine learning model. During the training process, the weights and / or biases associated with the neurons of the layers can be adjusted until the machine learning model produces predictions that reflect the real data for the input training data. Each neuron can be configured to multiply its input by the weights associated with that neuron. Each neuron can be further configured to add the bias associated with that neuron to its input. In other words, the output from a neuron can be the sum of the product of the bias associated with that neuron and the weights associated with that neuron with its input. The weights and biases can be adjusted during the recursive process and / or iteration. This can be referred to as backpropagation in the art. A convolutional neural network can be a neural network that includes one or more layers representing convolution operations. In this context, the input training data includes digital images. A digital image can be represented as a matrix (or array), and each element in the matrix (or array) can represent a corresponding pixel in the digital image. The value of an element can thus represent the pixel value and / or color value of the corresponding pixel in the digital image. Therefore, the input and output of a machine learning model can be numbers (e.g., matrices or arrays) representing a digital image.In this context, the input is a set of digital images (i.e., the training set or input set). Therefore, the input to the machine learning model can be multiple matrices or a three-dimensional matrix. However, it should be understood that the machine learning model can accept further input during training. In this particular case, the machine learning model is trained using a digital image training set and real data. The acquisition of the first digital image training set is similar to the acquisition of the first digital image input set. In other words, the lighting patterns used when acquiring the digital image training set and the digital image input set may be similar or the same. The machine learning model can be trained using the digital image training set as input and real data as the desired output. In other words, the machine learning model can be trained until the difference between the machine learning model's output and the real data is less than a threshold. This difference can be described in the art as a loss function. It is probably preferable to train the machine learning model until the loss function is minimized. In other words, the machine learning model can be trained until the difference between the machine learning model's output and the real data is minimized. The training process can be repeated for multiple different training samples (e.g., different training samples of the same and / or different types), which allows the machine learning model to construct digital images of a wider range of sample types and / or construct digital images with higher accuracy.

[0089] Those skilled in the art will recognize that the inventive concept is by no means limited to the preferred variations described above. Rather, many modifications and variations are possible within the scope of the appended claims.

[0090] For example, in ​ The image shows the back side of sample 1080 being illuminated. However, it should be understood that the illumination system 106 may alternatively or additionally be configured to illuminate the front side of sample 1080 (i.e., the side of sample 1080 facing the microscope objective 104). Thus, an illumination pattern can be formed by illuminating the back side and / or the front side of sample 1080.

[0091] Furthermore, variations of the disclosed variants can be understood and implemented by a person skilled in the art when practicing the claimed invention by studying the drawings, the disclosure, and the appended claims.

Claims

1. A method (30) for training a machine learning model to construct a digital image depicting a sample, the method (30) comprising: Receive (S300) a digital image training set of training sample (1080), wherein the digital image training set is acquired using a first microscope objective (104) and by illuminating (S310) the training sample (1080) with multiple illumination modes and capturing (S314) a digital image for each of the multiple illumination modes. Receiving (S320) real data including a digital image depicting the training sample (1080), wherein the digital image of the real data is acquired using a second microscope objective (204) and by: illuminating the training sample with conventional microscopic illumination mode (S332), and capturing (S334) the digital image while illuminating the training sample (1080) with the conventional microscopic illumination mode; and The machine learning model is trained (S340) using the received digital image training set and the received real data to construct digital images depicting the training sample (1080); and The second microscope objective (204) has a higher numerical aperture than the first microscope objective (104), thereby the resolution of the digital image of the real data is higher than the resolution of at least one digital image in the digital image training set.

2. The method (30) according to claim 1, wherein, Each of the multiple illumination patterns is formed by illuminating the training sample (1080) with one or more of the multiple light sources.

3. The method (30) according to claim 2, wherein, One or more of the plurality of light sources (1062) are configured to emit polychromatic light; and each of the plurality of illumination patterns is formed by illuminating the training sample (1080) with polychromatic light.

4. The method (30) according to claim 2 or 3, wherein, Each of the plurality of light sources (1062) is configured to illuminate the training sample (1080) from one of the plurality of directions 1060.

5. The method (30) according to claim 4, wherein, At least one of the multiple directions (1060) corresponds to an angle greater than the numerical aperture (1040) of the first microscope objective (104).

6. The method (30) according to any one of claims 4 or 5, wherein, Two or more of the plurality of directions (1060) correspond to the overlapping portion of the Fourier space associated with the training sample (1080), and wherein each illumination pattern is formed by illuminating the training sample (1080) from one or more directions corresponding to the non-overlapping portion of the Fourier space associated with the training sample (1080).

7. A method (40) for constructing a digital image depicting a sample, the method (40) comprising: Receive (S400) a digital image input set of the sample, wherein the digital image input set is acquired (S410) by illuminating the sample with multiple illumination modes (S412) and capturing (S414) digital images for each of the multiple illumination modes; A digital image depicting the sample is constructed (S420) in the following manner: The digital image input set is input (S422) into the machine learning model trained by the method (30) according to any one of claims 1 to 6, and The machine learning model receives (S424) an output including a digital image depicting the sample.

8. An apparatus (60) for training a machine learning model to construct a digital image depicting a sample, the apparatus (60) including circuitry (610) configured to perform: A first receiving function (6120) is configured to receive a digital image training set of training samples (1080), wherein, The digital image training set was acquired using a first microscope objective (104) by illuminating the training sample (1080) with multiple illumination modes and capturing digital images for each of the multiple illumination modes. A second receiving function (6122) is configured to receive real data including a digital image depicting the training sample (1080), wherein the digital image of the real data is acquired using a second microscope objective (204) by illuminating the training sample (1080) with conventional microscopic illumination mode and capturing the digital image while illuminating the training sample (1080) with conventional microscopic illumination mode; and Training function (6124), configured to train the machine learning model using a received digital image training set and received real data to construct digital images depicting the training samples; and The second microscope objective (204) has a higher numerical aperture than the first microscope objective (104), thereby the resolution of the digital image of the real data is higher than the resolution of at least one digital image in the digital image training set.

9. A microscope system (10), comprising: An illumination system (106) is configured to illuminate a sample (1080) with multiple illumination modes; Image sensor (102); A microscope objective (104) is arranged to image the sample (1080) onto the image sensor (102); and Circuit (100), which is configured to perform: An acquisition function (1120) is configured to acquire a digital image input set, which is achieved by being configured to perform the following operations: The lighting system (106) is controlled to illuminate the sample (1080) with each of the plurality of lighting modes, and The image sensor (102) is controlled to capture digital images for each of the plurality of lighting modes, and The circuit (100) is further configured to perform: Image construction function (1122), which is configured as follows: The digital image input set is fed into a machine learning model trained by the method according to any one of claims 1 to 6, and The machine learning model receives output including a constructed digital image depicting the sample.

10. The microscope system (10) according to claim 9, wherein, The lighting system (106) includes multiple light sources (1062).

11. The microscope system (10) according to claim 10, wherein, One or more of the plurality of light sources (1062) are configured to emit polychromatic light; and each of the plurality of illumination modes is formed by illuminating the sample (1080) with polychromatic light.

12. The microscope system (10) according to claim 10 or 11, wherein, Each of the plurality of light sources (1062) is configured to illuminate the sample (1080) from one of the plurality of directions 1060.

13. The microscope system (10) according to claim 12, wherein, At least one of the multiple directions (1060) corresponds to an angle greater than the numerical aperture (1040) of the microscope objective (104).

14. The microscope system (10) according to claim 12 or 13, wherein, Two or more of the plurality of directions (1060) correspond to the overlapping portion of the Fourier space associated with the sample (1080); and wherein the acquisition function (1120) is configured to control the illumination system (106) such that each illumination pattern is formed by illuminating the sample (1080) from one or more directions corresponding to the non-overlapping portion of the Fourier space associated with the sample (1080).

15. A non-transitory computer-readable storage medium (50) comprising program code portions that, when executed on a device having processing capabilities, perform the method (30) according to any one of claims 1 to 6 or the method (40) according to claim 7.

Images