A signal extraction and image reconstruction method for wide-field illumination confocal microscopy imaging system

Abstract

The present invention is a signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system, comprising the following steps: using each photosensitive pixel of a camera as a single-pixel detector to reconstruct an image; selecting signals emitted by object points conjugated to the photosensitive pixel of the camera from the reconstructed image, and reconstructing to obtain a diffraction image of the object point; defining a circle and an annulus from each reconstructed image with the image pixel point corresponding to each photosensitive pixel as the center, accumulating the grayscale values ​​corresponding to the pixel points on the circle and the annulus, and assigning the grayscale values ​​to the image pixel points corresponding to the photosensitive pixel conjugated to the object point, and reconstructing to obtain a confocal image I. 圆 and I 环 According to the conjugate mapping relationship between the modulation unit of the spatial light modulator and the photosensitive pixel of the camera, the confocal image I 圆 and I 环 Perform coordinate transformation to obtain the confocal image I′ formed by the object point conjugate with all modulation units of the spatial light modulator 圆 and I′ 环 .

Description

Technical Field

[0001] The present invention relates to the technical field of optical microscopic imaging, and more particularly to a signal extraction and image reconstruction method for a wide-field illumination confocal microscopic imaging system. Background Art

[0002] Laser confocal microscopy technology is currently used in many fields. Most confocal microscopy techniques use single-point scanning imaging (M. Minsky, US Patent #3013467, Microscopy Apparatus (1957)). The disadvantage of this method is that it can only scan and measure one pixel on the sample at a time. Therefore, an image of M×N pixels needs to be measured M×N times. The imaging speed is extremely low, and high-intensity laser illumination is required to obtain a signal with a sufficiently high signal-to-noise ratio. However, high-intensity laser illumination may affect the sample. Other methods include, for example, a spinning disk confocal microscope (D. Toomre and JB Pawley, Disk-scanning confocal microscopy. In Handbook of biological confocal microscopy (pp. 221-238). Springer, Boston, MA (2006)), a microlens array (Tiziani H J, Uhde H M. “Three-dimensional analysis by a microlens-array confocal arrangement,” Appl. Opt., 33 (4), 567-572 (1994).), etc. Compared with single-point scanning imaging, these confocal microscope imaging methods can obtain information of multiple points of the sample in one measurement, but are more likely to collect stray light emitted from the off-focus plane, reduce spatial resolution, and still have the problem of affecting biological samples.

[0003] In recent years, patent application CN112505044A has proposed a wide-field illumination confocal microscopy system and method, combining laser confocal microscopy technology with single-pixel imaging to effectively address the aforementioned issues. The imaging method proposed in this patent utilizes a single-pixel imaging method to generate a series of non-uniform structured light illuminations for a sample. A camera simultaneously captures images of the sample, and each photosensitive pixel in the camera receives a series of distinct light intensity values. Each light intensity value encompasses light emitted from multiple object points within the sample. Based on the series of light intensity values ​​recorded by each photosensitive pixel, a single-pixel imaging algorithm is used to reconstruct a diffuse speckle image of the sample. This image represents the light emitted from different object points within the sample as recorded by the camera pixel, based on the distribution of the object point's position coordinates. This distribution allows the extraction of the light intensity values ​​corresponding to the object point that forms a conjugate image with the camera pixel, thereby reducing the noise generated by light emitted from non-conjugate object points on the camera pixel. Specific implementations of this patent demonstrate that the imaging method proposed in this patent can reconstruct confocal fluorescence microscopy images with fewer measurements than the number of pixels in the sample image. However, this method cannot guarantee that the generated structured light modulates only the sample information at the focal plane of the objective lens, and not the information at out-of-focus planes. As a result, the contrast and optical sectioning capabilities of the resulting confocal microscopy images still need to be improved. In addition, the method requires at least 254 measurements, which is still a relatively large number.

[0004] Patent publication number CN114486827A is an improvement on Patent publication number CN112505044A, proposing a method and system for improving the efficiency and performance of structured illumination confocal microscopy. This method utilizes a spatial light modulator to modulate and generate a series of spatially unevenly distributed structured light beams, using only a high-frequency fringe pattern. This high-frequency fringe pattern refers to a fringe pattern corresponding to high-frequency Fourier coefficients within a Fourier exchange basis pattern, and has a high fringe density. By using only fringe patterns corresponding to high-frequency Fourier coefficients to generate a series of structured light beams, this method acquires only the high-frequency Fourier coefficients of the sample image in the Fourier transform domain. This method requires only high-frequency Fourier coefficients because the high-frequency fringe pattern has a narrow light modulation range. This ensures that the structured light beam modulates only the information of the sample in the focal plane of the objective lens, and not information in out-of-focus planes. This allows for the subsequent reconstruction of high-contrast, high-section confocal microscopy images. At the same time, since spatial light modulation of structured light is performed only based on high-frequency stripe patterns, that is, the number of Fourier basis patterns used in the overall method is reduced, the number of measurements is also greatly reduced, and compared with the method and system proposed in the invention patent with publication number CN112505044A, it can reconstruct confocal microscopy images with higher image contrast.

[0005] The prior art technologies listed above all use brightfield imaging to reconstruct confocal brightfield images in microscopic imaging. However, for unstained biological samples, phase objects, or highly transparent objects, light changes very little after passing through the sample, so the contrast of the reconstructed confocal brightfield image is often still low. To address this problem, the prior art also proposes a solution to reconstruct confocal darkfield images using darkfield imaging. Darkfield imaging refers to illuminating the sample at an angle that exceeds the numerical aperture of the objective lens, suppressing the reflected or transmitted light of the sample, collecting scattered light caused by local changes in the refractive index of the sample, filtering out low-frequency components, and allowing high-frequency components to enter the imaging system. As a result, the field of view becomes a dark background, and the edge of the sample appears bright. The advantage of darkfield imaging is that the reconstructed image has a higher contrast than brightfield imaging. Since high-frequency components are collected, structures that cannot be detected by brightfield microscopy can be detected.

[0006] There are three existing methods for confocal dark-field microscopy. The first method is to achieve dark-field imaging by offsetting the detector. In 1982, IJ Cox proposed placing a pinhole on the first dark ring in the Airy disk in the detector plane in a confocal scanning microscope. This is equivalent to eliminating the zero-frequency component at the focal plane, thus satisfying the dark-field condition. However, since the detector is offset from the conjugate point position, the signal collected is emitted from a non-conjugate point, which reduces the system's slicing capability. The second method is to achieve dark-field imaging by collecting scattered light from the sample through direct oblique illumination. In 2022, Sun H et al. used laser oblique illumination of optical elements to form dark-field imaging, collect scattered light, and detect surface defects and sub-surface defects of polished optical components. The third method is to achieve dark-field imaging by generating an annular illumination beam suitable for dark-field microscopy. In 2020, Jian Liu et al. used a pair of conical lenses to generate an annular illumination beam when detecting sub-surface defects of optical components using a confocal dark-field microscope. The annular illumination beam was then illuminated at an angle to the sample, and the reflected light was blocked by an aperture, allowing only scattered light to be received. In 2021, Jian Liu et al. proposed using dual-mode optical fiber to generate an annular beam, mitigating the intensity increase caused by diffraction in the central region of the annular beam. Replacing the annular beam generation module with optical fiber contributes to the compactness of the microscope. However, existing dark-field confocal microscopes use complex annular beam generation devices and require precise lateral point scanning devices.

[0007] Therefore, how to enable the wide-field illumination confocal microscopy system in the existing technology to be applied to confocal dark-field imaging without adding additional devices, or to further improve the imaging resolution of the wide-field illumination confocal microscopy system, or to further obtain the confocal edge image of the sample is an urgent problem to be solved. Summary of the Invention

[0008] The present invention aims to overcome at least one of the above-mentioned defects of the prior art and provide a signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system, which is used to solve the problem of how to apply the wide-field illumination confocal microscopy imaging system in the prior art to confocal dark-field imaging without changing the device, or to further improve the imaging resolution of the wide-field illumination confocal microscopy imaging system.

[0009] The technical solutions adopted in the present invention include:

[0010] In the first aspect, the present invention provides a signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system, comprising: S1: a light beam emitted by a light source is modulated by a spatial light modulator to generate K spatially unevenly distributed structured lights P1, P2, ..., P k ,…,P K ; where k = 1, 2, ..., K represents the number of structured light patterns;

[0011] The spatially unevenly distributed structured light is projected onto the sample to be measured by the tube lens and the objective lens, so that the light beam emitted after the interaction between the sample to be measured and the structured light is imaged onto the photosensitive surface of the camera;

[0012] The camera captures the image of the sample to be tested after the sample is interacted with the spatially non-uniform structured light.

[0013] S2: The photosensitive pixel D of the camera (1,1) As a single-pixel detector, each image captured from the camera Select photosensitive pixel D (1,1) The grayscale values ​​of the corresponding image pixels form a one-dimensional grayscale value sequence According to the one-dimensional gray value sequence, the image I is reconstructed using the single-pixel imaging image reconstruction method. D (1,1) , where m = 1, 2, ... M and n = 1, 2, ... N are integers, representing the serial numbers of the camera's photosensitive pixels respectively;

[0014] The other photosensitive pixels D of the camera are sequentially (1,2) ,…,D (m,n) ,…,D (M,N) As a single pixel detector, repeat step S2 to reconstruct the image sequence

[0015] S3: Image reconstructed from single-pixel imaging Select the object point W of the sample to be measured (1,1) The grayscale value of the conjugate image pixel is recorded as These grayscale values ​​are then assigned to the photosensitive pixels D (1,1) ,D (1,2),…,D (m,n) ,…,D (M,N) The corresponding image pixels are reassembled to obtain the object point W (1,1) Diffraction image produced on the photosensitive surface of the camera The sample point W to be tested (1,1) The camera's photosensitive pixel D (1,1) object-image conjugation;

[0016] Image sequences reconstructed from single-pixel imaging Select the object point W of the sample to be tested in turn (1,2) ,…,W (m,n) ,…,W (M,N) The grayscale value of the conjugate camera photosensitive surface pixel point is repeated in step S3 to obtain the object point W (1,2) ,…,W (m,n) ,…,W (M,N) Diffraction image produced on the photosensitive surface of the camera

[0017] and object point W (1,1) ,W (1,2) ,…,W (m,n) ,…,W (M,N) Image reconstructed from conjugate camera photosensitive surface pixels in single pixel imaging The coordinates in are determined by calibrating the object-image conjugate mapping relationship between the camera's photosensitive pixels and the spatial light modulator's modulation unit. This conjugate mapping can be represented by an affine transformation matrix. This affine matrix can be used to generate a calibration pattern using the spatial light modulator and project it onto the calibration sample. The camera then captures an image of this calibration pattern, extracting the coordinates of multiple feature points on the calibration pattern at the spatial light modulator and camera, and establishing feature point coordinate pairs. The affine matrix coefficients are then solved to obtain the conjugate mapping relationship.

[0018] S4: Reconstructing the diffraction image Define a photosensitive pixel D (1,1) The circle with the corresponding image pixel as the center and diameter d The image grayscale values ​​corresponding to all pixels on the circle are accumulated and assigned to the photosensitive pixel D (1,1) The corresponding image pixel point; repeat the above steps to reconstruct the diffraction image In each of them, a photosensitive pixel D is defined. (m,n) The circle with the corresponding image pixel as the center and diameter d The image grayscale values ​​corresponding to all pixels on the circle are accumulated and assigned to the photosensitive pixel D (m,n) The corresponding image pixel point, until the photosensitive pixel D (M,N) The corresponding pixel assignment is completed, and the confocal image I is obtained. 圆 . Where 0≤d<2d爱里斑 , d 爱里斑 The diameter of the first-order dark fringe of the Airy spot formed by a point of the sample to be measured on the photosensitive surface of the camera;

[0019] In the reconstructed diffraction image Define a photosensitive pixel D (1,1) The corresponding image pixel is the center of the circle, and the inner diameter is d 内 , outer diameter is d 外 Ring The image grayscale values ​​corresponding to all pixels on the ring are accumulated and assigned to the photosensitive pixel D (1,1) The corresponding image pixel point; repeat the above steps to reconstruct the diffraction image Define a photosensitive pixel D (m,n) The corresponding image pixel is the center of the circle, and the inner diameter is d 内 , outer diameter is d 外 Ring The image grayscale values ​​corresponding to all pixels on the ring are accumulated and assigned to the photosensitive pixel D (m,n) The corresponding image pixel point, until the photosensitive pixel D (M,N) The corresponding pixel assignment is completed, and the confocal image I is obtained. 环 Where 0≤d 内 <d 外 <2d 爱里斑 , d 爱里斑 The diameter of the first-order dark fringe of the Airy spot formed by a point of the sample to be measured on the photosensitive surface of the camera;

[0020] S5: Using the conjugate mapping relationship calibrated in step S3 and the confocal image I reconstructed in step S4 圆 and I 环 , obtain the object point W′ that is conjugate with the object image of all modulation units of the spatial light modulator (1,1) ,W′ (1,2) ,…,W′ (m,n) ,…,W′ (M,N) The confocal image I′ is formed 圆 and I′ 环 ;

[0021] S6: Move the sample at equal intervals along the optical axis and repeat steps S1-S5 to obtain confocal images of the sample at different depths.

[0022] Furthermore, the K structured light patterns used are Fourier basis patterns, and the Fourier coefficients corresponding to the K structured light patterns are located on concentric rings on the Fourier spectrum plane.

[0023] Furthermore, the Fourier basis pattern can be generated using a digital micromirror spatial light modulator, a liquid crystal spatial light modulator, or a beam interference spatial light modulator.

[0024] Furthermore, the method further comprises: before defining the circle in the diffraction image, using the photosensitive pixel D in the diffraction image (m,n) The corresponding image pixel point is taken as the starting point, and the search is carried out along the directions of 0°, 90°, 180°, and 270° respectively to determine the four grayscale minimum points, and the distance from each minimum point to the conjugate point is determined. The diameter of the diffraction Airy disk is calculated by d 爱里斑 =2×mean(r1,r2,r3,r4); where r1, r2, r3, r4 are the distances from the four minimum points to the starting point, and mean(·) represents the averaging operation.

[0025] The steps of the image registration method include: first creating L matrices of size M×N L is equal to the number of sampling points on the circle; from the reconstructed diffraction image Select the image grayscale value corresponding to the first sampling point on the circle and assign it to the matrix Pixel points (1,1), (1,2), ..., (m,n), ..., (M,N); repeat the above steps to reconstruct the diffraction image Select the image grayscale value corresponding to the second sampling point on the circle and assign it to the matrix The pixel points (1,1), (1,2), ..., (m,n), ..., (M,N) are assigned until all the sampling points on the circle are assigned, and L images are obtained. Select the confocal bright field image I0 as the reference and Align with image I0 and accumulate the aligned images into the confocal image.

[0026] Furthermore, in step S4, two concentric circles with different diameters d1 and d2 can be defined to obtain two different confocal images I 圆1 and I 圆2 , subtract the two images to get a new image: I 圆12 =I 圆2 -I 圆1 , where 0≤d1 <d2<2d 爱里斑 .

[0027] It is also possible to define two different inner and outer diameters d 1内 d 1外 and d 2内 d 2外 concentric rings, and obtain two different confocal images I 环1 , I 环2 , the two images are subtracted to get a new image: I 环12 =I 环2 -I 环1 , where 0≤d 1内 <d 1外 <2d爱里斑 , 0≤d 2内 <d 2外 <2d 爱里斑 , 0≤d 1内 <d 2内 <2d 爱里斑 , 0≤d 1外 <d 2外 <d 爱里斑 .

[0028] Furthermore, the step S1 further includes: obtaining the object-image conjugate mapping relationship between the camera photosensitive pixel and the spatial light modulator modulation unit through calibration, obtaining the camera photosensitive surface pixel coordinates that are conjugate with all the modulation units of the spatial light modulator according to the conjugate mapping relationship, and then combining the image captured by the camera Obtaining an image formed by pixel points conjugated with all modulation units of the spatial light modulator, the steps S2-S4 further include: according to the image formed by pixel points conjugated with all modulation units of the spatial light modulator obtained in step S1, the object point W′ having an object-image conjugated relationship with all modulation units of the spatial light modulator can be directly obtained. (1,1) ,W′ (1,2) ,…,W′ (m,n) ,…,W′ (M,N) The confocal image I′ is formed 圆 and I′ 环 .

[0029] Furthermore, the step S4 further comprises: in all the reconstructed diffraction images In the digital template, a digital template is defined, and the grayscale values ​​corresponding to all pixels in the digital template in each diffraction image are accumulated and then assigned to the photosensitive pixels D (1,1) ,D (1,2) ,…,D (m,n) ,…,D (M,N) A new confocal image can also be obtained by the corresponding image pixel points.

[0030] Furthermore, the method further comprises: reconstructing all diffraction images In the process, two different digital templates are defined, and step S4 is repeated to obtain two confocal images, and then the two confocal images are subtracted to obtain a new image.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The present invention provides a signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system. The method uses each photosensitive pixel of a camera as a single-pixel detector to reconstruct an image. The grayscale value corresponding to the image pixel point conjugate with the sample object point to be measured is selected from the reconstructed image. The diffraction image of the object point is obtained by recombining the grayscale value. In the reconstructed diffraction image, a pixel D is defined as a single pixel. (1,1) ,D (1,2) ,…,D (m,n) ,…,D (M,N) The corresponding image pixel point is a circle with the center as the circle; the image grayscale values ​​corresponding to all the pixels on the circle are accumulated and then assigned to the photosensitive pixel D respectively. (1,1) ,D (1,2) ,…,D (m,n) ,…,D (M,N) The corresponding image pixels are recombined to obtain a confocal image. By adjusting the diameter of the circle or ring, different confocal microscopic images can be obtained, including ordinary confocal bright field images, confocal images with enhanced resolution, and confocal dark field images. By defining two concentric circles or rings, reconstructing two images, and then subtracting the two images, the edge image of the sample can be obtained. Different images of the sample to be tested can also be obtained by defining different digital templates. The overall device does not require changes to the optical path, optical components, or microscope, and multiple modes of confocal microscopic imaging can be achieved in the same wide-field illumination confocal microscopic imaging system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 : (a) is the optical path diagram of the experimental system adopted in Example 1 of the present invention; (b1~b2) are the results of reconstruction of the circular ring defined in Example 1 of the present invention with an inner diameter of 0 pixels and an outer diameter of 2 pixels; (c1~c5) are schematic diagrams of the principle of searching for the position of the first-order dark ring of the Airy disk in the diffraction pattern.

[0034] Figure 2 This is a flow chart of steps S110 to S160 of the signal extraction and image reconstruction method of the wide-field illumination confocal microscopy imaging system provided in Example 1 of the present invention.

[0035] Figure 3 Different images are obtained by changing the diameter of the circle in Example 1 of the present invention: (a1-a2) represents the results of reconstruction with the inner diameter of the ring being 0 pixels and the outer diameter being 2 pixels; (b1-b2) represents the confocal image reconstructed with the inner diameter of the fixed ring being 4 pixels and the outer diameter being 6 pixels.

[0036] Figure 4This is an image reconstructed by defining two different circular rings in Example 1 of the present invention: the circular ring defined by (a1-a2) has an inner diameter of 6 pixels and an outer diameter of 8 pixels, and the circular ring defined by (b1-b2) has an inner diameter of 8 pixels and an outer diameter of 10 pixels, and (c) is the result of subtracting (b2) from (a2).

[0037] Figure 5 These are the confocal imaging results of the sample to be tested at different depths in Example 1 of the present invention: (a1-a5) represent the reconstruction results when the inner diameter of the defined circular ring is 0 pixels and the outer diameter is 2 pixels; (b1-b5) represent the reconstruction results when the inner diameter of the defined circular ring is 8 pixels and the outer diameter is 10 pixels; (c1-c5) represent the results of subtracting the confocal images reconstructed by defining two different circular rings, where one circular ring has an inner diameter of 6 pixels and an outer diameter of 8 pixels, and the other circular ring has an inner diameter of 8 pixels and an outer diameter of 10 pixels.

[0038] Figure 6 This is the optical path diagram of the experimental system used in Example 2 of the present invention.

[0039] Figure 7 1 is a flow chart of steps S210 to S260 of the signal extraction and image reconstruction method of the wide-field illumination confocal microscopy imaging system in Example 2 of the present invention.

[0040] Figure 8 In Example 2 of the present invention, different confocal imaging results of the sample to be tested are obtained by changing the diameter of the ring. DETAILED DESCRIPTION

[0041] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting the present invention. To better illustrate the following embodiments, some components in the accompanying drawings may be omitted, enlarged, or reduced in size, and do not represent actual product dimensions. Those skilled in the art will appreciate that some well-known structures and their descriptions may be omitted from the accompanying drawings.

[0042] Example 1

[0043] This embodiment provides a signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system. The wide-field illumination confocal microscopy imaging system is composed of the following components: Figure 1 As shown in (a), the system includes a monochromatic LED light source, a reflector, a spatial light modulator, a tube lens 1, a beam splitter, an objective lens, a sample, a tube lens 2, and a camera.

[0044] The central wavelength of the monochromatic LED light source is 623 nm. The spatial light modulator has a resolution of 1280 × 800 pixels and a pixel size of 7.6 μm. The focal length of tube lens 1 is 200 mm. The transmission and reflection ratio of the beamsplitter is 1:1. The numerical aperture of the objective lens is 0.4, and the magnification is 20. The focal length of tube lens 2 is 200 mm. The pixel size of the area array detector (camera) is 6.5 μm, and the resolution is 2560 × 2160 pixels.

[0045] The sample to be tested in this embodiment is an integrated circuit chip.

[0046] like Figure 2 As shown, the process of performing confocal microscopic imaging of the sample to be tested using the method provided in this embodiment includes the following steps:

[0047] S110: The light beam emitted by the light source is modulated by the spatial light modulator to generate 140 frames of spatially unevenly distributed structured light. The spatially unevenly distributed structured light is projected onto the sample to be tested by tube lens 1 and objective lens, so that the light beam emitted after the sample to be tested interacts with the structured light is imaged onto the photosensitive surface of the camera through objective lens and tube lens 2. The camera captures an image of the sample to be tested interacting with the spatially uneven structured light.

[0048] S120: Each photosensitive pixel D of the camera (1,1) ,D (1,2) ,…,D (800,800) As a single-pixel detector, the light intensity value recorded by the photosensitive pixel with the same pixel coordinates is extracted from each image taken by the camera to form a one-dimensional light intensity sequence; based on each one-dimensional light intensity sequence, the image is reconstructed using the single-pixel imaging reconstruction method.

[0049] S130: Each image reconstructed from single pixel imaging Select the object point W of the sample to be measured (1,1) ,W (1,2) ,…,W (800,800) The grayscale value of the conjugate image pixel is recombined to obtain the diffraction image generated by the light signal emitted by the object point on the photosensitive surface of the camera

[0050] and object point W (1,1) ,W (1,2) ,…,W (800,800) Image reconstructed from conjugate camera photosensitive surface pixels in single pixel imaging The coordinates in are determined by calibrating the object-image conjugate mapping relationship between the camera's photosensitive pixels and the spatial light modulator's modulation unit. This conjugate mapping relationship can be represented by an affine transformation matrix, as shown in equation (1):

[0051]

[0052] Where (x s ,y s ) represents the coordinates of the point in the spatial light modulator coordinate system, (x c ,y c ) represents the coordinates of the point in the camera coordinate system, T a Represents an affine transformation matrix.

[0053] In this embodiment, the affine matrix T a A black and white checkerboard pattern is generated by a spatial light modulator and projected onto the pattern-free area of ​​a resolution plate negative. The image of the black and white checkerboard pattern is then captured by a camera. The coordinates of the corner points on the black and white checkerboard pattern projected by the spatial light modulator and the image of the black and white checkerboard pattern captured by the camera are extracted respectively to establish a pair of corner point coordinates. Substituting the pair of corner point coordinates into equation (1), a system of linear equations can be constructed to solve the affine matrix coefficients and thereby determine the conjugate mapping relationship.

[0054] S140: Reconstructing the diffraction image In each of them, a camera photosensitive pixel D is defined. (1,1) ,D (1,2) ,…,D (800,800) The corresponding image pixel point is the circle with a diameter of d; then the grayscale values ​​corresponding to all pixels on the circle are accumulated and assigned to the camera's photosensitive pixel D (1,1) ,D (1,2) ,…,D (800,800) The corresponding image pixels are reconstructed to obtain a confocal image.

[0055] In this embodiment, the coordinates of the pixel points on the circle are decimal coordinates.

[0056] like Figure 1 As shown in (c1), in the diffraction image Before defining the circle, the camera photosensitive pixel D (1,1) ,D (1,2) ,…,D (800,800) The corresponding image pixel point is taken as the starting point, and searches are performed along the directions of 0°, 90°, 180°, and 270° to determine the four minimum points. The minimum point refers to the point with the smallest image grayscale value.

[0057] like Figure 1 (c2)- Figure 1 As shown in (c5), the distance from each minimum point to the starting point is determined, which are recorded as r1, r2, r3, and r4 respectively. The diameter of the diffraction Airy spot is calculated by the formula d 爱里斑 =2×mean(r1, r2, r3, r4), where mean(·) represents the averaging operation.

[0058] Preferably, in order to eliminate the influence of noise on the calculation results during the imaging process and improve the robustness, the same method is used for other pixel points of the camera to obtain the diffraction image The diameter of the diffraction spot in the image is calculated. The diameter of the diffraction spot in all diffraction spot images is counted, and the diameter with the largest number of counts is selected as the diameter of the diffraction spot. In this embodiment, the diameter of the diffraction spot determined by the above steps is 8 pixels.

[0059] S150: According to the conjugate mapping relationship calibrated in step S130, all modulation units (x S ,y S ) is mapped to the camera coordinate system to obtain the fractional pixel coordinates (x S2C ,y S2C ), then the integer pixel points (x C ,y C ) grayscale value, interpolation calculation of fractional pixel coordinates (x S2C ,y S2C ) corresponding to the grayscale value, thereby obtaining a confocal image formed by the object point conjugated with all the modulation units of the spatial light modulator.

[0060] S160: Move the sample along the optical axis, 1 μm at a time, and repeat steps S110-S160 at each position to reconstruct confocal images of the sample at different depths.

[0061] Figure 3 The present invention provides a method for obtaining different images of the sample to be tested by defining circles of different diameters. Figure 3 (a1-a2) represents the definition of a circle with a diameter equal to 2, and the confocal image is reconstructed; Figure 3 (b1-b2) represents the confocal image reconstructed using a ring with an inner diameter of 4 pixels and an outer diameter of 6 pixels. Comparing the two results, it is not difficult to see that the result and contrast resolution of the reconstruction using the ring are higher.

[0062] Figure 4 Given two different concentric rings, d 1内 =6 pixels, d 1外 = 8 pixels and d 2内 =8 pixels, d 2外 =10 pixels, two different confocal images are reconstructed, and then the two images are subtracted to obtain a new image, which reflects the edge information of the sample to be tested.

[0063] Figure 5Confocal images of the sample at different depths obtained using the method proposed in the present invention are given, where (a1-a5) represent the reconstruction results of a defined ring with an inner diameter of 0 pixels and an outer diameter of 2 pixels; (b1-b5) represent the reconstruction results of a defined ring with an inner diameter of 8 pixels and an outer diameter of 10 pixels; and (c1-c5) represent the results of subtracting confocal images reconstructed by two defined rings, where one ring has an inner diameter of 6 pixels and an outer diameter of 8 pixels, and the other ring has an inner diameter of 8 pixels and an outer diameter of 10 pixels.

[0064] Example 2

[0065] Based on the same concept as Example 1, this embodiment provides a signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system for imaging fluorescent samples. Figure 6 As shown, the system includes a monochromatic LED light source, a reflector, a spatial light modulator, a tube lens 1, a dichroic beam splitter, an objective lens, a sample, a tube lens 2, and a camera.

[0066] The central wavelength of the monochromatic LED light source is 470 nm. The spatial light modulator has a resolution of 1920 × 1080 pixels and a pixel size of 7.6 μm. The focal length of tube lens 1 is 400 mm. The transmission wavelength of the dichroic beamsplitter is 570 nm to 700 nm. The numerical aperture of the objective lens is 0.75, and the magnification is 20. The focal length of tube lens 2 is 200 mm. The pixel size of the camera is 6.5 μm, and the resolution is 2560 × 2160 pixels.

[0067] The sample to be tested in this embodiment is a self-fluorescent lily of the valley rhizome slice.

[0068] like Figure 7 As shown, the process of performing confocal microscopic imaging of the sample to be tested using the method provided in this embodiment includes the following steps:

[0069] S210: The light beam emitted by the light source is modulated by the spatial light modulator to generate 72 frames of spatially unevenly distributed structured light. The spatially unevenly distributed structured light is projected onto the sample to be tested by tube lens 1 and the objective lens. The camera captures an image of the sample to be tested and the spatially uneven structured light. The image size is 512×512 pixels. The light beam emitted by the sample to be tested after the interaction with the structured light is imaged onto the photosensitive surface of the camera through the objective lens and tube lens 2.

[0070] S220: Each photosensitive pixel D of the camera (1,1) ,D (1,2) ,…,D (512,512)As a single-pixel detector, the light intensity value recorded by the photosensitive pixel with the same pixel coordinates is extracted from each image taken by the camera to form a one-dimensional light intensity sequence; based on each one-dimensional light intensity sequence, the image is reconstructed using the single-pixel imaging reconstruction method.

[0071] S230: Each image reconstructed from single pixel imaging Select the object point W of the sample to be measured (1,1) ,W (1,2) ,…,W (512,512) The grayscale value of the conjugate pixel is recombined to obtain the diffraction image produced by the light signal emitted by the object point on the photosensitive surface of the camera

[0072] and object point W (1,1) ,W (1,2) ,…,W (512,512) Image reconstructed from conjugate camera photosensitive surface pixels in single pixel imaging The coordinates in are determined by calibrating the object-image conjugate mapping relationship between the camera's photosensitive pixels and the spatial light modulator's modulation unit. The method for determining this conjugate mapping relationship is similar to the method steps in Example 1, except that in this example, the calibration sample is a fluorescent microsphere sample with a thickness of approximately 2 microns.

[0073] S240: Reconstructing the diffraction image In each of them, a camera photosensitive pixel D is defined. (1,1) ,D (1,2) ,…,D (512 , 512) The corresponding image pixel is the center of the circle, and the inner diameter is d 内 , outer diameter is d 外 The grayscale values ​​of all pixels on each ring are accumulated and then assigned to the photosensitive pixels D of the camera. (1,1) ,D (1,2) ,…,D (512,512) The corresponding image pixels are recombined to obtain a confocal image.

[0074] Before defining the ring in the diffraction image, the camera photosensitive pixel D (1,1) ,D (1,2) ,…,D (512,512) The corresponding image pixel point is taken as the starting point, and the search is carried out along the directions of 0°, 90°, 180°, and 270° to determine the four minimum points. The minimum point refers to the point with the smallest intensity. The distance from each minimum point to the starting point is determined and recorded as r1, r2, r3, and r4 respectively. The diameter of the diffraction Airy disk is calculated by the formula d 爱里斑 =2×mean(r1, r2, r3, r4), where mean(·) represents the averaging operation.

[0075] Preferably, to eliminate the impact of noise on the calculation results during the imaging process and improve robustness, the same method is applied to other camera pixels to determine the diffraction spot diameter in the reconstructed diffraction image. The diffraction spot diameters of all diffraction images are counted, and the diameter with the largest number of counts is selected as the diffraction spot diameter. In this embodiment, the diffraction spot diameter d0 determined using the above steps is 10 pixels.

[0076] S250: According to the conjugate mapping relationship calibrated in step S230, all modulation units (x S ,y S ) is mapped to the camera coordinate system to obtain the fractional pixel coordinates (x S2C ,y S2C ), then the integer pixel points (x C ,y C ) grayscale value, interpolation calculation of fractional pixel coordinates (x S2C ,y S2C ) corresponding to the grayscale value, thereby obtaining a confocal image formed by the object point conjugated with all the modulation units of the spatial light modulator.

[0077] S260: Move the sample along the optical axis, 1 μm at a time, and repeat steps S210-S260 at each position to reconstruct confocal images of the sample at different depths.

[0078] like Figure 8 Shown is the result of reconstruction, where Figure 8 (a) shows the results of wide-field fluorescence imaging; Figure 8 (b1-e1) represent different circular rings defined on the intermediate diffraction pattern, with inner diameters of 0 pixels, 4 pixels, 8 pixels, and 12 pixels, and outer diameters of 2 pixels, 6 pixels, 10 pixels, and 14 pixels, respectively; Figure 8 (b2-e2) represents the result of reconstructing by directly adding the grayscale values ​​corresponding to the pixels on the ring; Figure 8 (b3-e3) represent the results of reconstruction using the registration method; compared with the locally enlarged view, it can be seen that when the inner diameter of the ring is less than or equal to 8 pixels, the reconstructed result obtained by direct addition is equivalent to the result reconstructed using the registration method; in addition, as the inner diameter of the ring increases, the resolution of the reconstructed confocal image gradually improves; when the inner diameter of the ring is 10 pixels, which is equal to the diameter of the diffraction Airy disk, the reconstructed result is a confocal dark-field image; when the inner diameter of the ring is 12 pixels, which exceeds the diameter of the diffraction Airy disk, the signal-to-noise ratio of the reconstructed result is low due to the relatively weak fluorescence signal.

[0079] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system, characterized in that: The following steps are involved: S1: The light beam emitted by the light source is modulated by the spatial light modulator to generate K spatially unevenly distributed structured lights P1, P2, ..., P k ,…,P K ; where k = 1, 2, ..., K represents the number of structured light patterns; The spatially unevenly distributed structured light is projected onto the sample to be measured by the tube lens and the objective lens, so that the light beam emitted after the interaction between the sample to be measured and the structured light is imaged onto the photosensitive surface of the camera; The camera captures the image of the sample to be tested after the sample is interacted with the spatially non-uniform structured light. S2: The photosensitive pixel D of the camera (1,1) As a single-pixel detector, each image captured from the camera Select photosensitive pixel D (1,1) The grayscale values ​​of the corresponding image pixels form a one-dimensional grayscale value sequence According to the one-dimensional gray value sequence, the image is reconstructed using the single-pixel imaging image reconstruction method. Where m = 1, 2, ... M and n = 1, 2, ... N are integers, representing the serial numbers of the camera's photosensitive pixels respectively; The other photosensitive pixels D of the camera are sequentially (1,2) ,…,D (m,n) ,…,D (M,N) As a single pixel detector, repeat step S2 to reconstruct the image sequence S3: Image reconstructed from single-pixel imaging Select the object point W of the sample to be measured (1,1) The grayscale value of the conjugate image pixel is recorded as These grayscale values ​​are then assigned to the photosensitive pixels D (1,1) ,D (1,2) ,…,D (m,n) ,…,D (M,N) The corresponding image pixels are reassembled to obtain the object point W (1,1) Diffraction image produced on the photosensitive surface of the camera The sample point W to be tested (1,1) The camera's photosensitive pixel D (1,1) object-image conjugation; Similarly, the image sequence reconstructed from single pixel imaging Select the object point W of the sample to be tested in turn (1,2) ,…,W (m,n) ,…,W (M,N) The grayscale value of the conjugate camera photosensitive surface pixel point is repeated in step S3 to obtain the object point W (1,2) ,…,W (m,n) ,…,W (M,N) Diffraction image produced on the photosensitive surface of the camera and object point W (1,1) ,W (1,2) ,…,W (m,n) ,…,W (M,N) Image reconstructed from conjugate camera photosensitive surface pixels in single pixel imaging The coordinates in are determined by calibrating the object-image conjugate mapping relationship between the camera's photosensitive pixels and the spatial light modulator's modulation unit; S4: Reconstructing the diffraction image Define a photosensitive pixel D (1,1) The circle with the corresponding image pixel as the center and diameter d The image grayscale values ​​corresponding to all pixels on the circle are accumulated and assigned to the photosensitive pixel D (1,1) The corresponding image pixel; Repeat the above steps for the remaining reconstructed diffraction images In each of them, a photosensitive pixel D is defined. (m,n) The circle with the corresponding image pixel as the center and diameter d The image grayscale values ​​corresponding to all pixels on the circle are accumulated and assigned to the photosensitive pixel D (m,n) The corresponding image pixel point, until the photosensitive pixel D (M,N) The corresponding pixel assignment is completed, and the confocal image I is obtained. 圆 ; where 0≤d<2d 爱里斑 , d 爱里斑 The diameter of the first-order dark fringe of the Airy spot formed by a point of the sample to be measured on the photosensitive surface of the camera; In the reconstructed diffraction image Define a photosensitive pixel D (1,1) The corresponding image pixel is the center of the circle, and the inner diameter is d 内 , outer diameter is d 外 Ring The image grayscale values ​​corresponding to all pixels on the ring are accumulated and assigned to the photosensitive pixel D (1,1) The corresponding image pixel; Repeat the above steps to reconstruct the diffraction image Define a photosensitive pixel D (m,n) The corresponding image pixel is the center of the circle, and the inner diameter is d 内 , outer diameter is d 外 Ring The image grayscale values ​​corresponding to all pixels on the ring are accumulated and assigned to the photosensitive pixel D (m,n) The corresponding image pixel point, until the photosensitive pixel D (M,N) The corresponding pixel assignment is completed, and the confocal image I is obtained. 环 ; where 0≤d 内 <d 外 <2d 爱里斑 , d 爱里斑 The diameter of the first-order dark fringe of the Airy spot formed by a point of the sample to be measured on the photosensitive surface of the camera; S5: Through the above conjugate mapping relationship and confocal image I 圆 and I 环 , obtain the object point W′ that is conjugate with the object image of all modulation units of the spatial light modulator (1,1) ,W′ (1,2) ,…,W′ (m,n) ,…,W′ (M,N) The confocal image I′ is formed 圆 and I′ 环 When the diameter of the circle or ring is zero, the image formed by steps S4 and S5 is a confocal bright field image I0; S6: Move the sample at equal intervals along the optical axis and repeat steps S1-S5 to obtain confocal images of the sample at different depths.

2. The signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system according to claim 1, characterized in that: In step S1 , the K structured light patterns used are Fourier basis patterns, and the Fourier coefficients corresponding to the K structured light patterns are located on concentric rings on the Fourier spectrum plane.

3. The signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system according to claim 1, characterized in that: In step S1 , a digital micromirror spatial light modulator, a liquid crystal spatial light modulator, or a beam interference spatial light modulator is used to generate a structured light pattern, where the structured light pattern is a Fourier pattern.

4. The signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system according to claim 1, characterized in that: In step S4, the grayscale values ​​corresponding to the sampling points on the circle are processed using an image registration method to obtain a confocal image I 圆 and I 环 ; The steps of the image registration method include: Create L matrices of size M×N L is equal to the number of pixels on the circle or ring; Respectively, from the reconstructed diffraction images In the example, select the image grayscale value corresponding to the first pixel on the circle or ring and assign it to the matrix The pixel points are (1,1), (1,2),…, (m,n),…, (M,N); Repeat the above steps to reconstruct the diffraction image Select the image grayscale value corresponding to the second pixel on the circle or ring and assign it to the matrix The pixel points (1,1), (1,2), ..., (m,n), ..., (M,N) are assigned until all the pixel points on the circle or ring are assigned values, and L images are obtained. Select the confocal bright field image I0 as the reference and Register with image I0, accumulate the registered images and obtain the confocal image I 圆 or I 环 ; The first pixel point on the circle or ring may be any point on the circle or ring, and the next pixel point is selected according to the set path until all the pixel points on the circle or ring are selected.

5. The signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system according to claim 1, characterized in that: In step S4, the diameter d of the diffraction Airy disk 爱里斑 The steps to determine are as follows: In the reconstructed diffraction image In the figure, the photosensitive pixels D (1,1) ,D (1,2) ,…,D (i,j) ,…,D (m,n) The corresponding image pixel point is taken as the starting point, and the search is carried out along the directions of 0°, 90°, 180°, and 270° respectively to determine the four minimum points. The distance from each minimum point to the starting point is calculated, and the diameter of the diffraction Airy spot d is obtained. 爱里斑 According to formula d 爱里斑 =2×mean(r1,r2,r3,r4)determine; Among them, r1, r2, r3, and r4 are the distances from the four minimum points to the starting point, and mean() represents the average operation.

6. The signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system according to claim 1, characterized in that: In step S4, two circles with different diameters d1 and d2 are defined to obtain two different confocal images I 圆1 and I 圆2 , subtract the two images to get a new image: I 圆12 =I 圆2 -I 圆1 , where 0≤d1 <d2<2d 爱里斑； In step S4, two different inner and outer diameters d are defined. 1内 d 1外 and d 2内 d 2外 Ring, get two different confocal images I 环1 , I 环2 , the two images are subtracted to get a new image: I 环12 =I 环2 -I 环1 , where 0≤d 1内 <d 1外 <2d 爱里斑 , 0≤d 2内 <d 2外 <2d 爱里斑 , 0≤d 1内 <d 2内 <2d 爱里斑 , 0≤d 1外 <d 2外 <d 爱里斑 .

7. The signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system according to claim 1, characterized in that: In step S4, two circles with different diameters d1 and d2 are defined to obtain two different confocal images I 圆1 and I 圆2 Then, in step S5, the object point W′ that has an object-image conjugate relationship with all modulation units of the spatial light modulator is obtained. (1,1) ,W′ (1,2) ,…,W′ (m,n) ,…,W′ (M,N) The confocal image I′ is formed 圆1 , I′ 圆2 , the two images are subtracted to get a new image: I′ 圆12 =I′ 圆2 -I′ 圆1 , where 0≤d1 <d2<d 爱里斑 ; In step S4, two different inner and outer diameters d are defined. 1内 d 1外 and d 2内 d 2外 Ring, get two different confocal images I 环1 , I 环2 Then, in step S5, the object point W′ that has an object-image conjugate relationship with all modulation units of the spatial light modulator is obtained. (1,1) ,W′ (1,2) ,…,W′ (m,n) ,…,W′ (M,N) The confocal image I′ is formed 环1 , I′ 环2 , the two images are subtracted to get a new image: I′ 环12 =I′ 环2 -I′ 环1 , where 0≤d 1内 <d 1外 <2d 爱里斑 , 0≤d 2内 <d 2外 <2d 爱里斑 , 0≤d 1内 <d 2内 <2d 爱里斑 , 0≤d 1外 <d 2外 <2d 爱里斑 .

8. The signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system according to claim 1, characterized in that: The step S1 further includes: The conjugate mapping relationship between the camera's photosensitive pixels and the spatial light modulator's modulation unit is obtained by calibration. The coordinates of the camera's photosensitive surface pixels that are conjugate with all the modulation units of the spatial light modulator are obtained by using the conjugate mapping relationship, and then combined with the image captured by the camera. Obtaining an image formed by pixel points conjugated with all modulation units of the spatial light modulator; The steps S2-S4 further include: According to the image formed by the pixel points conjugated with all the modulation units of the spatial light modulator obtained in step S1, an object point W′ having an object-image conjugated relationship with all the modulation units of the spatial light modulator is obtained. (1,1) ,W′ (1,2) ,…,W′ (m,n) ,…,W′ (M,N) The confocal image I′ is formed 圆 and I′ 环 .

9. The signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system according to claim 1, characterized in that: The step S4 further comprises: reconstructing all diffraction images In the digital template, a digital template is defined, and the grayscale values ​​corresponding to all pixels in the digital template in each diffraction image are accumulated and then assigned to the photosensitive pixels D (1,1) ,D (1,2) ,…,D (m,n) ,…,D (M,N) The corresponding image pixels are used to obtain the confocal image.

10. The signal extraction and image reconstruction method for a wide-field illumination confocal microscopy imaging system according to claim 9, characterized in that: Also includes: All diffraction images reconstructed In the process, two different digital templates are defined, and step S4 is repeated to obtain two confocal images, and then the two confocal images are subtracted to obtain a new image.

Images