Lensless color high-speed microscopic imaging system based on coding illumination

Abstract

The invention discloses a coded illumination lensless color high-speed microscopic imaging system based on an FPGA (Field Programmable Gate Array), and belongs to the technical field of microscopic imaging. A main body of the lensless color high-speed microscopic imaging system based on coded illumination disclosed by the invention is composed of an LED matrix light source, a sample table, a CMOS image sensor and a deep learning computing platform, and the specific working process is as follows: in an imaging stage, an FPGA controls the LED matrix light source to be turned on and off at a high speed; light rays with multiple wavelengths are emitted in a random illumination channel mode to irradiate a sample on the sample table, and an image sensor collects a diffraction pattern and sends the diffraction pattern to a computing platform; according to the method, TIE and an iterative phase retrieval method are combined, a diffraction object field is refocused in a digital mode through retrieval of a quantitative phase and back propagation, phase distribution of an object plane is reconstructed, and a color dynamic image of a sample is finally obtained by applying a color migration model of a GAN neural network. The system is simple in structure, dyeing or marking is avoided, the imaging process is automatic and intelligent, the imaging speed is high, the quality is high, and color dynamic imaging can be achieved.

Description

Technical Field

[0001] The invention relates to the technical field of microscopic imaging, and mainly to a lens-free color high-speed microscopic imaging system based on coded illumination. Background Art

[0002] As an imaging tool widely used in many fields such as life science research, optical microscopy has greatly promoted the progress and development of life science and other fields. It is often used to observe life phenomena such as cell movement, vesicle fusion, and lipid metabolism. However, optical microscopy still faces some challenges, including relatively low system automation and high cost. In addition, when observing biological samples, it is usually necessary to stain and label the samples first, which limits its convenience to a certain extent. At the same time, when performing high-speed dynamic imaging, the structure of the microscope often interferes with the imaging process, thereby affecting the imaging quality.

[0003] In recent years, lensless microscopes based on non-interference phase recovery technology have developed rapidly, relaxing the strict requirements on beam coherence, simplifying and reducing the size of lensless or on-chip microscopy systems, and achieving compactness and high-throughput imaging. Lensless microscopes only need to use image sensors to measure the light intensity distribution of objects at different defocus planes or different wavelengths to achieve quantitative phase reconstruction, and then use the color migration model of the GAN neural network to process the reconstructed image into a color image. This technology can avoid the influence of staining or labeling on the observation time of cells, and has the advantages of small size, compact structure, low system cost, and easy portability compared to traditional microscopes. However, many lensless imaging devices still have a translation stage, which is relatively complex in structure, slow in imaging speed, unable to perform color imaging, and unable to achieve high-speed dynamic imaging.

[0004] In view of these problems, the present invention proposes a lens-free color high-speed microscopic imaging system based on coded illumination, which can avoid staining or marking of different samples, and can be intelligent, high-speed, and form color dynamic images. Summary of the invention

[0005] The purpose of the present invention is to provide a lensless color high-speed microscopic imaging system based on coded illumination in view of the problems existing in the above-mentioned existing optical microscopes and lensless microscopes, so as to solve the problems of high cost of the displacement stage, inability to perform high-speed dynamic imaging, and need to be dyed to restore the original color in the prior art. To achieve the above-mentioned purpose, the present invention relates to a lensless color high-speed microscopic imaging system based on coded illumination. It is implemented by the following technical scheme: The present invention is implemented by a lensless color high-speed microscopic imaging system based on coded illumination, which is composed of an FPGA, an LED matrix light source, a sample stage, an image sensor, an outer frame, and a deep learning computing platform, characterized in that: the outer frame is placed horizontally, the LED matrix light source is fixed on the top layer of the outer frame, and the light source emission direction faces vertically to the base, each LED can provide quasi-monochromatic illumination with a narrow spectral width and approximately spatial coherence, the sample stage is arranged on the side of the LED matrix light source close to the base, the image sensor is arranged on the side of the sample stage close to the base, and is connected to the computing platform through USB Type-C, the FPGA is arranged on the side of the LED matrix light source close to the base, and the plane of the FPGA close to the base is fixed to coincide with the plane of the outer frame base.

[0006] Optionally, the overall frame is made of plastic or alloy and is in a rectangular parallelepiped shape.

[0007] Optionally, the axial distance between the LED matrix light source and the sample stage is about 3.25-3.30 cm, and the axial distance between the image sensor and the sample stage is about 1-2 mm.

[0008] Optionally, the LED matrix light source is a 4*4 color common anode LED matrix, the wavelengths of the LED light source are red light λ=635nm, green light λ=515nm, blue light λ=465nm, the spectrum width is basically around 20nm, and the diameter of the light-emitting element is 150μm.

[0009] Optionally, the sample stage includes a metal slide clamp for fixing a slide containing the sample.

[0010] Optionally, the image sensor is CMOS, model is MTP3091, and the image sensor includes a USB Type-C interface, and the interface is used to connect the image sensor to the deep learning computing platform for data transmission.

[0011] Technical solution: An image restoration algorithm based on a coded illumination method, characterized in that: the FPGA controls the LED matrix light source to quickly turn on and off a specified number and position of LED lights to form coded illumination, and then the computing platform captures three diffraction patterns of different illumination wavelengths (red, green, and blue), converts the effective defocus distances of the red and blue channel images, and uses the green channel image as the central reference image. The phase distribution of the image plane can be retrieved by solving TIE using a fast discrete cosine transform (DCT) under homogeneous Neumann boundary conditions.

[0012] Combining the TIE (Transfer Intensity Equation) algorithm and the Gerchberg–Saxton (GS) iterative phase recovery method, it is characterized in that: the deep learning computing platform uses the TIE reconstructed phase as the initial input of the Gerchberg-Saxton type iterative phase retrieval algorithm, and uses multiple RGB iterations for phase recovery.

[0013] The color high-speed image reconstruction method is characterized in that: the deep learning computing platform uses the color migration model of the GAN neural network to process the reconstructed image, minimizes the difference in color and texture details through the zero-sum competition game of the generator network (GN) and the discriminator network (DN), realizes the color restoration of the original image, and then processes the obtained color image to form a dynamic image.

[0014] The beneficial effects of the present invention are as follows: the lensless color high-speed microscopic imaging system based on coded illumination provided by the present application comprises: an LED matrix light source, an FPGA, a sample stage, a CMOS image sensor and a deep learning computing platform, wherein the LED matrix light source, the sample stage, the CMOS image sensor, the FPGA and the overall bracket arranged in sequence constitute an imaging main body and are connected to the deep learning computing platform; the LED matrix light source is placed at the top of the entire system, irradiates the sample stage below it, and is captured by the image sensor below the sample stage, and the FPGA connected to the LED matrix light source is placed at the bottom of the entire device, and during the imaging stage, the FPGA controls the LED matrix light source to turn on and off at high speed, and emits light of multiple wavelengths in the form of coded illumination channels to illuminate the sample on the sample stage, and the image sensor collects the diffraction pattern and sends it to the computing platform; by combining TIE with an iterative phase retrieval method, the quantitative phase of the sample can be retrieved through an effective phase retrieval algorithm, and by retrieving the quantitative phase, the diffraction object field can be digitally refocused through back propagation, and then the color migration model of the GAN neural network is used to process the reconstructed image, and then the obtained color image is processed to form a dynamic image, and finally a color dynamic image of the sample is obtained. Not only does it make the system structure simple and avoid staining or labeling, but it also makes the imaging process automated and intelligent, and it also has fast imaging speed and high quality, and can achieve color dynamic imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 Schematic diagram of the process of the lens-free color high-speed microscopy imaging system based on coded illumination.

[0017] Figure 2 Schematic diagram of the imaging subject of the lensless color high-speed microscopy imaging system based on coded illumination.

[0018] Figure 3 It is a partial schematic diagram of the LED lighting strategy in the coded lighting method.

[0019] Icons: 1-LED matrix light source; 2-processed light beam; 3-sample stage; 4-image sensor; 5-FPGA; 6-deep learning computing platform; DETAILED DESCRIPTION

[0020] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0021] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0023] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0024] In order to make the implementation process of the present invention clearer, it will be described in detail below with reference to the accompanying drawings.

[0025] The present application provides a lensless color high-speed microscopic imaging system based on coded illumination, which is composed of an LED matrix light source, an FPGA, a sample stage, a CMOS image sensor, and a deep learning computing platform. It is characterized in that: the outer frame 5 is placed horizontally, the LED matrix light source 1 is fixed on the top layer of the outer frame 5, and the light source emission direction is vertically facing the base, each LED can provide quasi-monochromatic illumination with a narrow spectrum width and approximately spatial coherence, the sample stage 2 is arranged on the side of the LED matrix light source 1 close to the base, the CMOS image sensor 3 is arranged on the side of the sample stage 2 close to the base, and is connected to the computing platform 7 via USB Type-C, the FPGA5 is arranged on the side of the LED matrix light source 1 close to the base, and the plane of the FPGA5 close to the base is fixed to coincide with the plane of the outer frame base.

[0026] Optionally, the overall frame is made of plastic or alloy and is in a rectangular parallelepiped shape.

[0027] Optionally, the axial distance between the LED matrix light source and the sample stage is about 3.2-3.305 cm, and the axial distance between the image sensor and the sample stage is about 1-2 mm.

[0028] Optionally, the LED matrix light source is a 4*4 color common anode LED matrix, the wavelengths of the LED light source are red light λ=635nm, green light λ=515nm, blue light λ=465nm, the spectrum width is basically around 20nm, and the diameter of the light-emitting element is 150μm.

[0029] Optionally, the sample stage includes a metal slide clamp for fixing a slide containing the sample.

[0030] Optionally, the image sensor is CMOS, and the image sensor includes a USB Type-C interface, and the interface is used to connect the image sensor to the deep learning computing platform for data transmission.

[0031] The image restoration algorithm based on the coded illumination method is characterized in that: the FPGA controls the LED matrix light source to turn on and off the LED lights of a specified number and position at a high speed to form coded illumination, and then the computing platform captures three diffraction patterns of different illumination wavelengths (red, green, and blue), converts the effective defocus distances of the red and blue channel images, and uses the green channel image as the central reference image. The phase distribution of the image plane can be retrieved by solving TIE using a fast discrete cosine transform (DCT) under homogeneous Neumann boundary conditions.

[0032] Combining the TIE (Transfer Intensity Equation) algorithm and the Gerchberg–Saxton (GS) iterative phase retrieval method, it is characterized in that: the deep learning computing platform uses the TIE reconstructed phase as the initial input of the Gerchberg-Saxton type iterative phase retrieval algorithm, using 5 RGB iterations.

[0033] The color high-speed image reconstruction method is characterized in that: the deep learning computing platform uses the color migration model of the GAN neural network to process the reconstructed image, minimizes the difference in color and texture details through the zero-sum competition game of the generator network (GN) and the discriminator network (DN), realizes the color restoration of the original image, and then processes the obtained color image to form a dynamic image.

[0034] The present application relates to a lensless color high-speed microscopic imaging system based on coded illumination, which adopts the imaging method of coded illumination. During imaging, the LED matrix light source 1 is placed at the top of the entire device, and the emitted light 2 illuminates the sample stage 3 below it, and is captured by the CMOS image sensor 4 below the sample stage 3. The FPGA 5 connected to the LED matrix light source 1 is placed at the bottom of the entire device. The FPGA 5 controls the LED matrix light source 1 to light up and down at high speed, so as to randomly illuminate the channels (such as Figure 3 ) emits light 2 of multiple wavelengths to illuminate the sample on the sample stage 4, and the CMOS image sensor 4 collects the diffraction pattern and transmits it to the computing platform 6 through the USB Type-C connection; by combining TIE with the iterative phase retrieval method, the quantitative phase of the sample can be retrieved through an effective phase retrieval algorithm. By retrieving the quantitative phase, the diffraction field can be digitally refocused through back propagation, and then the reconstructed image can be processed using the color migration model of the GAN neural network, and then the obtained color image is processed to form a dynamic image, and finally a color dynamic image of the sample is obtained. Not only does it make the system structure simple, the imaging process is automated and intelligent, but it also has a fast imaging speed and high quality, and can achieve color imaging.

Claims

1. A lensless color high-speed microscopic imaging system based on coded illumination, characterized in that: The invention is composed of an FPGA, an LED matrix light source, a sample stage, a CMOS image sensor, an outer frame and a deep learning computing platform, and is characterized in that: the outer frame is placed horizontally, the LED matrix light source is fixed on the uppermost layer of the outer frame, and the emission direction of the light source faces vertically to the base, each LED can provide quasi-monochromatic illumination with narrow spectral width and approximately spatial coherence, the sample stage is arranged on the side of the LED matrix light source close to the base, the image sensor is arranged on the side of the sample stage close to the base, and is connected to the computing platform via a USB Type-C, the FPGA is arranged on the side of the LED matrix light source close to the base, and the plane of the FPGA close to the base is fixedly overlapped with the plane of the outer frame base, the image sensor collects a diffraction image and transmits it to the deep learning computing platform, and an image recovery algorithm based on a coded illumination method is used on the deep learning computing platform, and phase recovery is realized by combining a TIE (light intensity transmission equation) algorithm and a Gerchberg–Saxton (GS) iterative phase recovery method, and then a color high-speed image reconstruction method is used to complete color high-speed dynamic imaging.

2. The lensless color high-speed microscopic imaging system based on coded illumination according to claim 1, characterized in that: The overall frame is made of plastic or alloy and is in a rectangular parallelepiped shape.

3. The lensless color high-speed microscopic imaging system based on coded illumination according to claim 1, characterized in that: The axial distance between the LED matrix light source and the sample stage is about 3.25-3.30 cm, and the axial distance between the CMOS image sensor and the sample stage is about 1-2 mm.

4. The lensless color high-speed microscopic imaging system based on coded illumination according to claim 1, characterized in that: The LED matrix light source is a 4*4 color common anode LED matrix, the wavelengths of the LED light source are red light λ=635nm, green light λ=515nm, blue light λ=465nm, the spectrum width is basically around 20nm, and the diameter of the light-emitting element is 150μm.

5. The lensless color high-speed microscopic imaging system based on coded illumination according to claim 1, characterized in that: The sample stage comprises a metal pressing clip for fixing a glass slide containing a sample.

6. The lensless color high-speed microscopic imaging system based on coded illumination according to claim 1, characterized in that: The image sensor is a CMOS, model MTP3091, and includes a USB Type-C interface, which is used to connect the image sensor to the deep learning computing platform for data transmission.

7. The image restoration algorithm based on coded illumination method according to claim 1, characterized in that: The FPGA controls the LED matrix light source to turn on and off the specified number and position of LED lights at high speed to form coded lighting. The computing platform then captures three diffraction patterns of different illumination wavelengths (red, green, and blue), converts the effective defocus distances of the red and blue channel images, and uses the green channel image as the central reference image. The phase distribution of the image plane can be retrieved by solving TIE using a fast discrete cosine transform (DCT) under homogeneous Neumann boundary conditions.

8. The method of combining the TIE (Transfer Intensity Equation) algorithm and the Gerchberg–Saxton (GS) iterative phase recovery method according to claim 1, characterized in that: The described deep learning computing platform uses the TIE reconstructed phase as the initial input to a Gerchberg-Saxton type iterative phase retrieval algorithm, using multiple RGB iterations for phase recovery.

9. The color high-speed image reconstruction method according to claim 1, characterized in that: The deep learning computing platform uses the color transfer model of the GAN neural network to process the reconstructed image, minimizes the difference in color and texture details through the zero-sum competition game between the generator network (GN) and the discriminator network (DN), realizes the color restoration of the original image, and then processes the obtained color image to form a dynamic image.

Images