Multi-plane parallel imaging system and imaging method

Abstract

The invention provides a multi-plane parallel imaging system and an imaging method. A light beam emitted by a to-be-tested sample enters an objective lens from the objective lens; the first light splitting component is used for receiving light beams emitted by the objective lens and reflecting and transmitting the incident light beams to form first transmission light and first reflection light, and the first transmission light serves as incident light of the next light splitting component and sequentially forms Nth transmission light and Nth reflection light; the image acquisition device acquires an imaged image of the Nth transmission light and acquires imaged images of the first reflected light to the Nth reflected light. According to the multi-plane parallel imaging system and the imaging method provided by the invention, the light splitting unit is introduced into the light path, so that images of a plurality of planes can be acquired at the same time under the condition of single exposure; the exposure frequency required by large-depth-of-field imaging is reduced, and the imaging efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of optical imaging technology, and in particular to a multi-plane parallel imaging system and imaging method. Background Art

[0002] Optical microscopy is widely used in industry and medicine, such as circuit boards, semiconductors, and biological tissue detection imaging. In order to increase the depth of field of imaging, a common method is to fuse the images after taking a series of images at different axial positions (CN106134176B, CN112529951B), which significantly slows down the imaging process. In the field of microscopy, there are currently several innovative methods to solve this problem. For example, an axially elongated point spread function (PSF) can extend the depth of field of the system, making the imaging insensitive to surface changes in the sample, thereby avoiding the need for axial scanning, but the generation of special PSFs requires customized or expensive wavefront modulation elements. In addition, in addition to hardware improvements, artificial neural networks such as generative adversarial networks (GAN), recurrent neural networks (RNN), and U-Net can be used to successfully restore defocused images in wide-field imaging modes. However, such methods are effective within the inherent axial PSF range of the system, which can only extend the depth of field by tens of microns, making it difficult to capture the concave and convex surfaces of the sample.

[0003] The existing technology has three major defects. The method of extending the depth of field based on axial scanning is slow because it requires sequentially taking images at different depths of field. The imaging method based on the axially elongated point spread function usually requires special optical elements, which will significantly increase the system cost. The depth of field expansion solution based on software algorithms has a limited extended depth of field. Summary of the invention

[0004] In view of this, the present invention provides a multi-plane parallel imaging system and a measurement method to achieve a large depth of field and is suitable for common wide-field imaging.

[0005] To solve the above problems, this application adopts the following technical solutions:

[0006] One of the purposes of this application is to provide a multi-plane parallel imaging system, comprising:

[0007] A stage, on which a sample to be tested is fixed;

[0008] An objective lens, the objective lens is used to receive the light beam emitted by the sample to be tested;

[0009] A light splitting unit, the light splitting unit comprising a first light splitting component, a second light splitting component, ... and an Nth light splitting component sequentially arranged along a light beam propagation direction, wherein N is a natural number greater than or equal to 1, the first light splitting component is used to receive the light beam emitted by the objective lens, and to reflect and transmit the incident light beam to form a first transmitted light and a first reflected light, the first transmitted light serves as the incident light of the next light splitting component, and sequentially forms an Nth transmitted light and an Nth reflected light;

[0010] An image acquisition device is used to acquire the imaging image of the Nth transmitted light and the imaging images of the first reflected light to the Nth reflected light.

[0011] In some embodiments, a first tube lens is further provided between the objective lens and the light splitting unit, and the first tube lens is used to receive the light beam passing through the objective lens and then incident on the light splitting unit.

[0012] In some of the embodiments, a first tube lens is further provided between the light splitting unit and the image acquisition device, and the first tube lens is used to receive the Nth transmitted light and input the Nth transmitted light to the image acquisition device.

[0013] In some embodiments, the light splitting component is a light splitting prism or a light splitting plate.

[0014] In some of the embodiments, the image acquisition device is an area array camera or a point acquisition sensor or a linear sensor, and the point acquisition sensor is a photomultiplier tube or a photodiode.

[0015] In some embodiments, N second tube lenses are further included, and the first reflected light, the second reflected light and the Nth reflected light reflected by the first light splitting component, the second light splitting component, ... and the Nth light splitting component are respectively incident on the corresponding second tube lenses and are collected by the image acquisition device after passing through the second tube lenses.

[0016] In some embodiments, there are N image acquisition devices, and each of the image acquisition devices is responsible for acquiring an imaging image.

[0017] In some of the embodiments, the image acquisition device is also electrically connected to a translation stage controller, and the translation stage controller is used to control the position of the image acquisition device, and the sample is made conjugate with the image by moving the translation stage controller.

[0018] In some of the embodiments, an artificial neural network is used to process the images captured by the image acquisition device, and the artificial neural network includes CNN or U-Net.

[0019] The second object of the present application is to provide an imaging method of the multi-plane parallel imaging system, comprising the following steps:

[0020] Place the sample to be tested on the stage;

[0021] The light beam emitted by the sample to be tested enters the objective lens;

[0022] The first light splitting component is used to receive the incident light beam from the objective lens, and reflect and transmit the incident light beam to form a first transmitted light and a first reflected light, the first transmitted light serves as the incident light of the next light splitting component, and sequentially forms an Nth transmitted light and an Nth reflected light;

[0023] The image acquisition device acquires the imaging image of the Nth transmitted light and acquires the imaging images of the first reflected light to the Nth reflected light.

[0024] In some embodiments, the first reflected light, the second reflected light and the Nth reflected light reflected by the first light splitting component, the second light splitting component, ... and the Nth light splitting component are also incident on the corresponding second tube lens respectively, and are collected by the image acquisition device and imaged after passing through the second tube lens.

[0025] In some of the embodiments, there is also a step of using an artificial neural network to process the images captured by the image acquisition device, and the artificial neural network includes CNN or U-Net.

[0026] In some of the embodiments, the image acquisition device is also electrically connected to a translation stage controller, and the translation stage controller is used to control the position of the image acquisition device, and the sample is made conjugate with the image by moving the translation stage controller.

[0027] This application adopts the above technical solution, and its beneficial effects are as follows:

[0028] In the first aspect, the multi-plane parallel imaging system and imaging method provided by the present application, the light beam emitted by the sample to be tested is incident on the objective lens and enters the spectroscopic unit; the first spectroscopic component is used to receive the light beam incident on the first tube lens, and reflects and transmits the incident light beam to form a first transmitted light and a first reflected light, the first transmitted light serves as the incident light of the next spectroscopic component, and sequentially forms an Nth transmitted light and an Nth reflected light; the image acquisition device acquires an imaging image of the Nth transmitted light and an imaging image of the first reflected light to the Nth reflected light. The multi-plane parallel imaging system and imaging method provided by the present application introduce a spectroscopic unit in the optical path, and can simultaneously acquire images of multiple planes under the condition of a single exposure, thereby reducing the number of exposures required for large depth of field imaging and improving imaging efficiency.

[0029] Secondly, the imaging method of the multi-plane parallel imaging system provided in the present application uses an artificial neural network to process the images captured by the image acquisition device, thereby avoiding the problem of discontinuous sampling planes caused by the limited number of imaging planes. When the image captured by the multi-plane imaging system is slightly out of focus, the artificial neural network can restore the original image captured by the multi-plane imaging system to a clear and in-focus state, thereby effectively improving the imaging quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 A schematic diagram of the structure of a multi-plane parallel imaging system provided in an embodiment of the present application.

[0032] Figure 2 It is a schematic diagram of the structure of a multi-plane parallel imaging system provided in another embodiment of the present application.

[0033] Figure 3 It is a flowchart of the steps of an imaging method of a multi-plane parallel imaging system provided by an embodiment of the present invention.

[0034] Figure 4 This is the image processing process of artificial neural network algorithm training and application provided by the embodiment of the present invention.

[0035] Figure 5 The figure is a comparison of the effects of photographing stripes on a resolution test target by a multi-plane imaging system provided by an embodiment of the present invention and a common single-plane system.

[0036] Figure 6 This is a comparison of the effects of traditional single-plane imaging and multi-plane imaging on photographing uneven biological tissues provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0038] In the description of the present application, it should be understood that the terms "upper", "lower", "horizontal", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0039] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0040] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0041] See also Figure 1 , is a schematic diagram of the structure of a multi-plane parallel imaging system provided in an embodiment of the present application, including a stage (not shown), an objective lens 110, a first tube lens 120, a light splitting unit 130 and an image acquisition device 140. The technical solution implemented is described in detail below.

[0042] The sample to be tested is fixed on the stage.

[0043] In this embodiment, the stage is movable, and the sample to be tested is moved by moving the stage.

[0044] The objective lens 110 is used to receive the light beam emitted by the sample to be tested.

[0045] The spectroscopic unit 130 includes a first spectroscopic component, a second spectroscopic component, ... and an Nth spectroscopic component arranged in sequence along the propagation direction of the light beam, wherein N is a natural number greater than or equal to 1, and the first spectroscopic component is used to receive the incident light beam from the objective lens 110, and reflect and transmit the incident light beam to form a first transmitted light and a first reflected light, and the first transmitted light serves as the incident light of the next spectroscopic component, and sequentially forms the Nth transmitted light and the Nth reflected light.

[0046] In this embodiment, the light splitting component is a light splitting prism or a light splitting plate or other light splitting elements.

[0047] The image acquisition device 140 is used to acquire the imaging image of the Nth transmitted light and the imaging images of the first reflected light to the Nth reflected light.

[0048] Furthermore, a first tube lens 120 is provided between the objective lens 110 and the spectroscopic unit 130, and the first tube lens 120 is used to receive the light beam passing through the objective lens 110 and then incident on the spectroscopic unit 130, or a first tube lens 120 is provided between the spectroscopic unit 130 and the image acquisition device 140, and the first tube lens 120 is used to receive the N-th transmitted light and input the N-th transmitted light into the image acquisition device 140.

[0049] In this embodiment, the image acquisition device 140 is an area array camera or a point acquisition sensor or a linear sensor, and the point acquisition sensor is a photomultiplier tube or a photodiode.

[0050] See also Figure 2 The present embodiment also provides a multi-plane parallel imaging system including N second tube lenses 150. The first reflected light, the second reflected light and the Nth reflected light reflected by the first light splitting component, the second light splitting component, ... and the Nth light splitting component are respectively incident on the corresponding second tube lenses 150 and are collected by the image acquisition device 140 after passing through the second tube lenses 150.

[0051] In this embodiment, there are N image acquisition devices 140, and each of the image acquisition devices is responsible for acquiring an image.

[0052] Furthermore, the image acquisition device 140 is also electrically connected to a translation stage controller (not shown), and the translation stage controller is used to control the position of the image acquisition device, and the sample and the image are conjugated by moving the translation stage controller.

[0053] Please refer to Figure 2 , this embodiment takes a double plane as an example. Suppose it is necessary to capture images of two points A and B on the sample, point A is located at the focus of the objective lens, and point B is slightly away from the focus. An image sensor is placed on the imaging plane. The existing technical solution can usually only capture one imaging plane at each exposure, point A is imaged at A', and point B cannot be clearly imaged on imaging plane 1 because it is not in focus. After shooting point A, it is necessary to move the sample or the objective lens so that point B is located at the focus of the objective lens before exposure and shooting can be performed to obtain a clear image. Therefore, only one plane image can be captured each time the exposure is performed; in this embodiment, a beam splitter prism is added after the first tube lens 120, so that the light emitted by the sample is divided into two, and the additional imaging plane 2 is conjugate with the sample plane where point B is located, so that the system can simultaneously clearly image the conjugate point B' of point B.

[0054] It can be understood that in order to accurately control the position of the imaging plane, a translation stage controller is provided to control the position of the image acquisition device, and the sample is made conjugate with the image by moving the translation stage controller.

[0055] It can be understood that the present embodiment is not limited to the above structure, and a plurality of beam splitters can be arranged along the propagation direction of the light beam. The first beam splitter is used to receive the light beam incident from the first tube lens 120, and to reflect and transmit the incident light beam to form the first transmitted light and the first reflected light. The first transmitted light serves as the incident light of the next beam splitter, and sequentially forms the Nth transmitted light and the Nth reflected light. The image acquisition device then acquires the imaging image of the Nth transmitted light and the imaging images of the first reflected light to the Nth reflected light, so that images of multiple planes of the sample can be captured in parallel at the same time. No matter how many planes need to be captured, as long as there are enough image acquisition devices, images of multiple planes can be acquired simultaneously under the condition of a single exposure, thereby improving the imaging speed.

[0056] See also Figure 3 The imaging method of the multi-plane parallel imaging system provided in the above embodiment of the present application is as follows:

[0057] Step S110: placing the sample to be tested on the stage.

[0058] It can be understood that the sample to be tested is first placed on the stage of the imaging system. After the sample position is preliminarily adjusted, the imaging area is set according to the size of the sample, such as the range to be photographed in the horizontal and axial directions. After the setting is completed, the translation stage will move to the first position to be photographed and then send a capture signal to the image acquisition devices located on different planes.

[0059] Step S120: the light beam emitted by the sample to be tested enters the objective lens.

[0060] Step S130: The first spectroscopic component is used to receive the light beam emitted by the objective lens, and to reflect and transmit the incident light beam to form the first transmitted light and the first reflected light. The first transmitted light serves as the incident light of the next spectroscopic component, and sequentially forms the Nth transmitted light and the Nth reflected light.

[0061] Furthermore, the first reflected light, the second reflected light and the Nth reflected light reflected by the first light splitting component, the second light splitting component, ... and the Nth light splitting component are also incident on the corresponding second tube lens respectively, and are collected by the image acquisition device and imaged after passing through the second tube lens.

[0062] Step S140: the image acquisition device acquires the imaging image of the Nth transmitted light and acquires the imaging images of the first reflected light to the Nth reflected light.

[0063] It can be understood that in the multi-plane parallel imaging method provided by this embodiment, the image acquisition device that receives the acquisition signal will acquire images in parallel. Unlike the conventional sequential multi-plane imaging method achieved by moving the translation stage or the sample, the multi-plane acquisition proposed by the present invention is parallel, which can effectively reduce the image acquisition time. When the image acquisition of an area is completed, the program will determine whether all the set imaging areas are covered. If not, it will move to the next position to be shot and perform multi-plane imaging again. When the shooting covers all the imaging areas, the acquisition is completed.

[0064] It can be understood that in order to accurately control the position of the imaging plane, a translation stage controller is provided to control the position of the image acquisition device, and the sample is made conjugate with the image by moving the translation stage controller, thereby achieving better imaging.

[0065] The imaging method provided in the above-mentioned embodiment of the present application introduces a spectroscopic unit in the optical path, which can capture images of multiple planes of the sample in parallel. No matter how many planes need to be captured, as long as there are enough image acquisition devices, images of multiple planes can be obtained simultaneously under the condition of a single exposure, thereby improving the imaging speed.

[0066] It should be noted that: the imaging system and imaging method provided in the above-mentioned embodiments of the present application set up the parallel acquisition of multiple plane images. In a real imaging scene, the sample planes to be acquired are continuous. For example, there may be a plane to be photographed between points A and B. If only the two planes where A and B are located are photographed, the image between the two planes will be slightly blurred due to defocus. Therefore, in order to avoid multiple movements of the sample or the objective lens or placing too many image acquisition devices in the imaging system, the present application also adopts an image virtual focusing network based on an artificial neural network, which can further convert the multi-plane images captured by the optical system into clear in-focus images. The following is a detailed description of the dual-plane processing flow as an example.

[0067] See also Figure 4 , showing the image processing process of multi-plane imaging artificial neural network algorithm training and application. The input of the algorithm is the original image collected by the dual-plane image acquisition device, and the output is a clear sample in-focus image. Training the network requires input and output image data sets. The input image can be directly collected by the imaging system, and the output target image can be obtained by collecting images of multiple focal planes and fusing them. Figure 4Part a) shows a method for obtaining a target image in an embodiment, which is also one of the common technical solutions for obtaining clear images with a large depth of field. This example shows a tilted sample. It is necessary to acquire multiple images in the axial direction and then fuse the acquired images to obtain a clear image with a large depth of field. The number of axial images acquired must meet the Nyquist sampling theorem. Figure 4 Part b) shows the multi-plane image processing method of the present invention. The present invention uses the obtained clear image as the learning target of the artificial neural network, uses the image collected by the dual-plane imaging device as the network input, and allows the network to learn the correspondence between input and output, so as to achieve the effect of restoring a clear large depth of field image with only sparsely collected axial images. The number of axially collected images can be less than that of the existing technical solutions, thereby improving the imaging efficiency.

[0068] See also Figure 5 , showing the comparison between the multi-plane imaging system and the ordinary single-plane system in capturing the stripes on the resolution test target. Figure 5 As can be seen in a), when the axial position Z = 0 μm, the image is in focus, but once it leaves the focal plane, the image becomes blurred and the intensity drops rapidly, and the system depth of field is only 10 μm. The results of the dual-plane imaging system are shown in Figure 5 In b), Z1 and Z2 represent the positions of two image sensors respectively, and the two planes are 10μm apart. After multi-plane system shooting and neural network processing, clear images can be obtained in the range of Z1 from -40μm to 30μm, and the system depth of field reaches more than 70μm.

[0069] See also Figure 6 , showing the comparison between traditional single-plane imaging and multi-plane imaging in capturing uneven biological tissues. Figure 6 Figure a) is an image taken at Z = -10 μm using only single-plane imaging. It can be seen that the lower right part of the image is not in focus. After restoration using a neural network, image O1 is obtained, and the image clarity is improved to a certain extent. Figure 6 b) is the imaging effect when Z = 10μm. The upper left corner of the original image is blurry. After network restoration, image O2 is obtained, and the overall clarity is also improved. Intensive shooting from Z = -10μm to Z = 10μm can obtain a clear target image, such as Figure 6 By calculating the error between the target image and O1 and O2, it can be seen that even after the restoration, there is still a large error. The error image is shown in Figure 6 Right. Output image O obtained by using multi-plane images as network input 1，2 Shown in Figure 6 c) The middle column, and the target image is the same as O 1，2The error is shown in Figure 6 In the far right of middle c), it can be seen that the overall error is reduced compared with the single-plane imaging method.

[0070] The multi-plane parallel imaging system and imaging method provided by the present application introduce a spectroscopic unit in the optical path, which can simultaneously acquire images of multiple planes under the condition of a single exposure, thereby reducing the number of exposures required for large depth of field imaging and improving imaging efficiency. In addition, an artificial neural network is used to process the images acquired by the image acquisition device, thereby avoiding the problem of discontinuous sampling planes caused by the limited number of imaging planes. When the image acquired by the multi-plane imaging system is slightly out of focus, the original image acquired by the multi-plane imaging system can be restored to a clear and in-focus state through the artificial neural network, thereby effectively improving the imaging quality.

[0071] The above are only preferred embodiments of the present application, and only specifically describe the technical principles of the present application. These descriptions are only for explaining the principles of the present application and cannot be interpreted as limiting the scope of protection of the present application in any way. Based on the explanation here, any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application, and other specific implementation methods of the present application that can be associated with the technicians in this field without creative work, should be included in the scope of protection of the present application.

Claims

1. A multi-plane parallel imaging system, characterized in that: include: A stage, on which a sample to be tested is fixed; Objective lens; The objective lens is used to receive the light beam emitted by the sample to be tested; A light splitting unit, the light splitting unit comprising a first light splitting component, a second light splitting component, ... and an Nth light splitting component sequentially arranged along a light beam propagation direction, wherein N is a natural number greater than or equal to 1, the first light splitting component is used to receive the light beam emitted by the objective lens, and to reflect and transmit the incident light beam to form a first transmitted light and a first reflected light, the first transmitted light serves as the incident light of the next light splitting component, and sequentially forms an Nth transmitted light and an Nth reflected light; An image acquisition device is used to acquire the imaging image of the Nth transmitted light and the imaging images of the first reflected light to the Nth reflected light.

2. The multi-plane parallel imaging system according to claim 1, characterized in that: A first tube lens is further arranged between the objective lens and the light splitting unit, and the first tube lens is used to receive the light beam passing through the objective lens and then incident on the light splitting unit.

3. The multi-plane parallel imaging system according to claim 1, characterized in that: A first tube lens is further provided between the light splitting unit and the image acquisition device, and the first tube lens is used for receiving the Nth transmitted light and inputting the Nth transmitted light into the image acquisition device.

4. The multi-plane parallel imaging system according to claim 1, characterized in that: The light splitting component is a light splitting prism or a light splitting plate.

5. The multi-plane parallel imaging system according to claim 1, characterized in that: The image acquisition device is an area array camera or a point acquisition sensor or a linear sensor, and the point acquisition sensor is a photomultiplier tube or a photodiode.

6. The multi-plane parallel imaging system according to claim 1, characterized in that: It also includes N second tube lenses, and the first reflected light, the second reflected light and the Nth reflected light reflected by the first splitting component, the second splitting component, ... and the Nth splitting component are respectively incident on the corresponding second tube lenses and are collected by the image collection device after passing through the second tube lenses.

7. The multi-plane parallel imaging system according to claim 1, characterized in that: There are N image acquisition devices, and each of the image acquisition devices is respectively configured to acquire an imaging image.

8. The multi-plane parallel imaging system according to claim 1 or 6, characterized in that: The image acquisition device is also connected to a translation stage controller, and the translation stage controller is used to control the position of the image acquisition device. The sample and the image are conjugated by moving the translation stage controller.

9. The multi-plane parallel imaging system according to claim 1, characterized in that: An artificial neural network is used to process the images collected by the image collection device, and the artificial neural network includes CNN or U-Net.

10. An imaging method of a multi-plane parallel imaging system as claimed in claim 1, characterized in that: The steps include: Place the sample to be tested on the stage; The light beam emitted by the sample to be tested enters the objective lens; The first light splitting component is used to receive the light beam emitted by the objective lens, and reflect and transmit the incident light beam to form a first transmitted light and a first reflected light, the first transmitted light serves as the incident light of the next light splitting component, and sequentially forms an Nth transmitted light and an Nth reflected light; The image acquisition device acquires the imaging image of the Nth transmitted light and acquires the imaging images of the first reflected light to the Nth reflected light.

11. The imaging method according to claim 10, characterized in that: The first reflected light, the second reflected light and the Nth reflected light reflected by the first light splitting component, the second light splitting component, ... and the Nth light splitting component are also incident on the corresponding second tube lens respectively, and are collected by the image acquisition device and imaged after passing through the second tube lens.

12. The imaging method according to claim 10, characterized in that: The method also includes the step of using an artificial neural network to process the image captured by the image capture device, wherein the artificial neural network includes CNN or U-Net.

13. The imaging method according to claim 10, characterized in that: The method also includes the step of using an artificial neural network to process the image captured by the image capture device, wherein the artificial neural network includes CNN or U-Net.

14. The imaging method according to claim 11, characterized in that: The image acquisition device is also connected to a translation stage controller, and the translation stage controller is used to control the position of the image acquisition device. The sample and the image are conjugated by moving the translation stage controller.

Images