Line illumination modulation multicolor imaging system

Abstract

The application discloses a line illumination modulation multicolor imaging system, belonging to the technical field of multicolor imaging, comprising: a line illumination modulation module, comprising m monochromatic light sources and a modulation light path, the wavelength of the light beam emitted by each monochromatic light source is different; each light beam passes through the modulation light path to form a line light spot focused on the focal plane of an objective lens, the illumination intensity of the line light spot on the focal plane of the objective lens is in Gaussian distribution in a first direction, the position parameters of each line light spot are different, and the line light spots of the m channels are superimposed to form m-color line illumination light; an imaging module is used for continuously scanning imaging along the first direction by using a multi-element detector with n rows of pixels, and at least m mixed images under the irradiation of the line illumination light are obtained, each mixed image corresponds to one row of pixels, and each mixed image contains signals of the m channels; and an image demodulation module is used for demodulating monochromatic images from the mixed images, and each monochromatic image corresponds to the signals of one channel. The application realizes synchronous multicolor imaging on a single detector.

Description

A line illumination modulation multicolor imaging system Technical Field

[0001] This invention relates to the field of multicolor imaging technology, and in particular to a line illumination modulation multicolor imaging system. Background Technology

[0002] By selectively labeling different structures in biological tissues with fluorophores of different colors and simultaneously acquiring multicolor fluorescence signals using multicolor microscopy, we can better analyze the spatial relationships and interactions between cells, organelles and molecules in biological tissues.

[0003] Current multicolor imaging methods typically use dichroic mirrors to spatially separate the signals from each channel and transmit them to different monochrome cameras for detection. However, this method has several problems: for fluorophores with overlapping emission spectra, dichroic mirrors cannot completely separate the fluorescence signals from different channels, resulting in crosstalk between channels; when using multiple monochrome cameras for detection, the images of each channel are offset laterally and have different detection focal planes along the axis, thus requiring complex registration processing of the original images; each channel requires a separate monochrome camera, and the number of channels in multicolor imaging is limited by the size and complexity of the system; and the manufacturing cost of the system increases with the number of monochrome cameras. Summary of the Invention

[0004] To overcome the problems of existing multicolor imaging methods, this invention provides a line illumination modulation multicolor imaging system. The technical solution is as follows:

[0005] Line illumination modulation multicolor imaging systems include:

[0006] A line illumination modulation module includes m monochromatic light sources and a modulation optical path. Each monochromatic light source emits a beam with a different wavelength. Each beam passes through the modulation optical path to form a line spot focused on the focal plane of the objective lens. The illumination intensity of the line spot on the focal plane of the objective lens is Gaussian distributed in a first direction, which is perpendicular to the extension direction of the line spot. Each line spot has different position parameters. The line spots of the m channels are superimposed to form m-color line illumination light, where m ≥ 2 and is a positive integer.

[0007] An imaging module is used to continuously scan and image along a first direction using a multi-element detector with n rows of pixels to obtain at least m mixed images under the illumination of the m-color line light. Each mixed image corresponds to one row of pixels, and each mixed image contains signals from m channels, where n ≥ m and is a positive integer.

[0008] The image demodulation module is used to demodulate a monochrome image from the mixed image using a frequency domain demodulation algorithm or a spatial domain demodulation algorithm, wherein the monochrome image corresponds to a signal of one channel.

[0009] Furthermore, the modulation optical path includes a shaping optical path for shaping the beam into a line beam, a position adjustment optical path for adjusting the position parameters of each line spot, and a projection optical path for superimposing the line spots to form an m-color line illumination light.

[0010] Furthermore, the shaping optical path includes a first dichroic mirror group for combining m light sources, and a first beam expander, a second beam expander, and a cylindrical lens arranged sequentially along the beam transmission direction after beam combining. The first dichroic mirror group includes m-1 dichroic mirrors.

[0011] Furthermore, the position adjustment optical path is located between the second beam expander and the cylindrical lens, and includes a second dichroic mirror group, a third dichroic mirror group, a first reflecting mirror, and a second reflecting mirror. The second dichroic mirror group contains m-1 dichroic mirrors, and the third dichroic mirror group contains m-1 dichroic mirrors. Each time the light beam passes through one dichroic mirror in the second dichroic mirror group, a light path is separated. The separated light paths are then combined by the corresponding dichroic mirror in the third dichroic mirror group. The remaining light beam after passing through the second dichroic mirror group enters the third dichroic mirror group for further beam combining via the first reflecting mirror and the second reflecting mirror.

[0012] Furthermore, the position adjustment optical path also includes a third reflecting mirror and a fourth reflecting mirror disposed in the optical path between the third dichroic mirror group and the cylindrical lens.

[0013] Furthermore, the spacing between the position parameters of each of the linear light spots is greater than or equal to the width of a single pixel of the multi-element detector in the object space.

[0014] Furthermore, the imaging module includes:

[0015] A scanning unit is used to continuously scan and image along a first direction using a multi-element detector with n rows of pixels, where n ≥ m;

[0016] The image patch acquisition unit acquires the strip image patch of the i-th pixel row in each frame of a sample obtained in chronological order;

[0017] The stitching unit stitches together the strip image blocks of the i-th pixel row in each frame of a sample to obtain a mixed image of the i-th pixel row, i∈n; this is repeated at least m times to obtain m mixed images.

[0018] Furthermore, the image demodulation module is used to demodulate m monochrome images from the m mixed images using a frequency domain demodulation algorithm, where each monochrome image corresponds to a signal of one channel, specifically including:

[0019] The Fourier transform unit is used to perform Fourier transform on the original spatial domain images of the m mixed images to obtain m mixed frequency domain images;

[0020] The first demodulation unit is used to combine the effective optical transfer function of the corresponding pixel row as its demodulation coefficient, perform linear elimination operation on the m mixed frequency domain images, and demodulate m monochrome frequency domain images, each of the monochrome frequency domain images containing a channel signal.

[0021] The inverse Fourier transform unit is used to perform inverse Fourier transform on m monochrome frequency domain images to obtain m monochrome images.

[0022] Furthermore, the image demodulation module is used to demodulate a monochrome image from m mixed images by sequentially eliminating the signals of m-1 non-target channels through translation alignment and linear elimination operations using a spatial domain demodulation algorithm. The monochrome image corresponds to the signal of one target channel.

[0023] Furthermore, the image demodulation module includes:

[0024] The first loop control unit is used to set the value of channel counter k to 1 and repeatedly execute the second demodulation unit. Each time it is executed, the value of the channel counter is incremented by 1 until the value of channel counter k is equal to m-1.

[0025] The second demodulation unit is used to set the value of the image counter p to 1, and repeats this process through the following sub-modules until the value of the image counter equals mk.

[0026] Translation subunits are used to blend images I k Translate it to match the blended image I k+p The signals in the k-channels are basically overlapping;

[0027] Elimination subunit, used for processing mixed image I k With mixed image I k+p Perform linear elimination to eliminate the mixed image I k+p The signal of channel k in the middle, and still denoted as I. k+p ;

[0028] The counting subunit is used to increment the value of the image counter p by 1.

[0029] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0030] The line illumination modulation multicolor imaging system provided in this invention uses a single multi-element detector for scanning imaging. Compared to imaging methods that require multiple cameras, this reduces system cost and complexity. Furthermore, by employing multiple pixel rows for detection and imaging, the images between each channel have a natural registration relationship, eliminating the need for additional registration processing and achieving multi-channel multicolor imaging. In fluorescence imaging, this line illumination modulation multicolor imaging system utilizes the differences in illumination light of different wavelengths to excite different colors of fluorescence signals, avoiding the problem of emission spectrum crosstalk. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 is a functional block diagram of the line illumination modulation multicolor imaging system of the present invention;

[0033] Figure 2 is a schematic diagram of the optical path structure for line illumination modulation multicolor imaging of the present invention;

[0034] Figure 3 is a schematic diagram of the optical path structure for another line illumination modulation multicolor imaging according to the present invention;

[0035] Figure 4 is a schematic diagram of the linear illumination light distribution of the present invention;

[0036] Figure 5 is a schematic diagram of the sample imaging acquisition process of the present invention;

[0037] Figure 6 is a functional block diagram of an image demodulation module according to the present invention;

[0038] Figure 7 is a flowchart of the frequency domain demodulation algorithm for multi-channel signals of the present invention;

[0039] Figure 8 is a functional block diagram of another image demodulation module of the present invention;

[0040] Figure 9 is a flowchart of the spatial demodulation algorithm for multi-channel signals of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0042] As shown in Figure 1, this embodiment of the invention provides a line illumination modulation multicolor imaging system, including a line illumination modulation module 10, an imaging module 20, and an image demodulation module 30.

[0043] The line illumination modulation module 10 includes m (m≥2, m is a positive integer) monochromatic light sources and modulation optical paths. Each monochromatic light source emits a beam with a different wavelength. Each beam passes through the modulation optical path to form a focused line spot on the focal plane of the objective lens. The illumination intensity of this line spot on the focal plane of the objective lens exhibits a Gaussian distribution in a first direction, which is perpendicular to the extension direction of the line spot. The position parameters of the line spots formed after modulation by different laser beams are different. The line spots from the m channels are superimposed to form m-color line illumination light.

[0044] The imaging module 20 is used to continuously scan and image along a first direction using a multi-element detector with n rows of pixels to obtain at least m mixed images under the illumination of the m-color line light. Each mixed image corresponds to one row of pixels, and each mixed image contains signals from m channels, where n ≥ m and is a positive integer.

[0045] The image demodulation module 30 is used to demodulate monochrome images from mixed images using frequency domain demodulation algorithms or spatial domain demodulation algorithms, with each monochrome image corresponding to a channel signal.

[0046] This invention provides a line illumination modulation multicolor imaging system. It employs a single multi-element detector for scanning imaging, reducing system cost and complexity compared to imaging methods requiring multiple cameras. Furthermore, by using multiple pixel rows for detection imaging, the images between each channel have a natural registration relationship, eliminating the need for additional registration processing and achieving multi-channel multicolor imaging. In fluorescence imaging, this line illumination modulation multicolor imaging system utilizes the differences in illumination light of different wavelengths to excite different colors of fluorescence signals, avoiding the problem of emission spectrum crosstalk.

[0047] It should be noted that the monochromatic light source in this line illumination modulated multicolor imaging system refers to a light source with a specific wavelength. This can be a monochromatic laser light source, a monochromatic LED (Light-emitting diode) light source, or a monochromatic light source obtained by combining a broadband light source with a narrow-band filter. Furthermore, this line illumination modulated multicolor imaging system is applicable to various scenarios, including fluorescence microscopy and non-fluorescence microscopy (i.e., the illumination wavelength is the same as the detection wavelength), and this application does not impose any limitations on these applications.

[0048] Furthermore, the modulation optical path includes a shaping optical path for shaping the laser beam into a line beam, a position adjustment optical path for adjusting the position parameters of each line spot, and a projection optical path for superimposing the line spots to form m-color line illumination light.

[0049] Referring to Figure 2, the optical path structure of the line illumination modulation multicolor imaging in fluorescence microscopy according to this application is shown. The shaping optical path includes a first dichroic mirror group 3.1, 3.2, ..., 3.(m-1) for combining m laser light sources, and a first beam expander 4, a second beam expander 5, and a cylindrical lens 12 arranged sequentially along the beam propagation direction after beam combining. The position adjustment optical path is located between the second beam expander 5 and the cylindrical lens 12, and includes a second dichroic mirror group 6.1, 6.2, ..., 6.(m-1), a third dichroic mirror group 7.1, 7.2, ..., 7.(m-1), a first reflecting mirror 8, and a second reflecting mirror 9. Each time the beam passes through a dichroic mirror in the second dichroic mirror group, a beam path is separated. The separated beam paths are combined by the corresponding dichroic mirrors in the third dichroic mirror group. The remaining beam after passing through the second dichroic mirror group enters the third dichroic mirror group for beam combining via the first reflecting mirror 8 and the second reflecting mirror 9. The projection optical path includes an illumination tube lens 13, an objective lens 14, and a fourth dichroic mirror 17 (in non-fluorescent imaging, this needs to be replaced with a combination of a semi-reflective beam splitter or a polarizing beam splitter and a quarter-glass slide, or other devices that can both reflect illumination light and transmit probe light of the same wavelength as the illumination light), used to superimpose the line light spots to form multicolor line illumination light. The line illumination modulated multicolor imaging system also includes an imaging optical path, specifically including an objective lens 14, a fourth dichroic mirror 17, an emission filter 18 (not needed in non-fluorescent imaging), a probe tube lens 19, and a multi-element detector 20.

[0050] Specifically, the first dichroic mirror group 3.1, 3.2, ..., 3.(m-1) is arranged along the direction of laser beam propagation. The laser beams emitted by the second laser source 2.1, the third laser source 2.2, ..., the m-th laser source 2.(m-1) are merged into the laser beam emitted by the first laser source 1 through one of the dichroic mirrors in the first dichroic mirror group to form a combined beam. The beam is then expanded by the first beam expander 4 and the second beam expander 5. Next, the second dichroic mirror group 6.1, 6.2, ..., 6.(m-1) separates the illumination beams from different channels and sends them into different optical paths. The optical path separated by the second dichroic mirror 6.1 is then combined by the third dichroic mirror 7.1. The remaining beam after passing through the second dichroic mirror 6.1 is separated again by the second dichroic mirror 6.2. The separated light paths are then combined by the third dichroic mirrors 7.2 and 7.1. The remaining beam after passing through the second dichroic mirror 6.2 is separated again by the second dichroic mirror 6.3. The separated light paths are then combined by the third dichroic mirrors 7.3, 7.2, and 7.1. This process continues, with the light paths separated by the second dichroic mirror 6.(m-1) being combined by the third dichroic mirrors 7.(m-1), ..., 7.2, and 7.1. The remaining beam passes through the first reflecting mirror 8 and the second reflecting mirror 9 and then enters the third dichroic mirrors 7.(m-1), ..., 7.2, and 7.1 for further beam combination. The combined beam is shaped into a line beam by the cylindrical lens 12, and then projected onto the focal plane of the objective lens via the illumination tube lens 13, the fourth dichroic mirror 17 and the objective lens 14. This line beam is used to excite the fluorescence signal of the sample 15 on the motorized translation stage 16. The fluorescence signal passes sequentially through the objective lens 14, the fourth dichroic mirror 17, the emission filter 18 and the detector tube lens 19, and is finally received by the multi-element detector 20 for imaging.

[0051] In some embodiments, the position adjustment optical path further includes a third reflecting mirror 10 and a fourth reflecting mirror 11 disposed between the third dichroic mirror group 7 and the cylindrical lens 12.

[0052] Specifically, the position of the line light spot reflected from the device being adjusted can be adjusted in the x-direction by adjusting the angle of any one or more of the second dichroic mirror group 6, the third dichroic mirror group 7, the first reflecting mirror 8, and the second reflecting mirror 9 in the optical path. Alternatively, the angle of the third reflecting mirror 10 or the fourth reflecting mirror 11 can be adjusted simultaneously to adjust the position of the line light spot in the x-direction of all channels. Preferably, the spacing between the position parameters of each line light spot is greater than or equal to the width of a single pixel of the multi-element detector in the object space.

[0053] In other embodiments, the modulation optical path can be achieved by directly generating line beams of different wavelengths from linear LEDs with different emission wavelengths, which are then projected onto the focal plane of the objective lens through a projection optical path. The position parameters of the line spot on the focal plane of the objective lens can be changed by adjusting the positions of the different linear LEDs.

[0054] Each laser source emits a laser beam with a different wavelength, and each laser beam, after passing through the modulation optical path, focuses a line spot on the focal plane of the objective lens with different position parameters. By precisely adjusting the position adjustment optical path, the line spots on the m channels can be misaligned in the x-direction, thereby achieving the purpose of separating the line spots of different channels in the x-direction.

[0055] To better understand this application, the following example of two laser light sources is used to introduce the optical path structure of a line illumination modulation multicolor imaging method in fluorescence microscopy. As shown in Figure 3, this optical path structure has only two laser light sources. Therefore, the first dichroic mirror group used to combine the laser beams in the shaping optical path has only one dichroic mirror, and the second and third dichroic mirror groups in the position adjustment optical path also only need one dichroic mirror.

[0056] Specifically, the laser beams emitted from the first laser source 1 and the second laser source 2 are first combined by the first dichroic mirror 3, and then expanded by the first beam expander 4 and the second beam expander 5. Next, the illumination beams from different channels are separated by the second dichroic mirror 6 and enter two optical paths respectively. One optical path passes through the second dichroic mirror 6 and the third dichroic mirror 7, while the other optical path passes through the second dichroic mirror 6, the first reflecting mirror 8, the second reflecting mirror 9, and the third dichroic mirror 7 in sequence. Then, the illumination beams from the two optical paths are combined again by the third dichroic mirror 7. Finally, the combined illumination beam is shaped into a line beam by the cylindrical lens 12, and then projected onto the focal plane of the objective lens through the illumination tube lens 13, the fourth dichroic mirror 17, and the objective lens 14 to excite the fluorescence signal of the sample 15 on the motorized translation stage 16. The fluorescence signal passes through the objective lens 14, the fourth dichroic mirror 17, the emission filter 18, and the detector tube lens 19 in sequence, and is finally received by the multi-element detector 20 for imaging.

[0057] Figure 4 is a schematic diagram of the line illumination beam distribution of the present invention. In some embodiments, as shown in Figure 4, the extension direction of the line spot is the y-direction, and the first direction is the x-direction perpendicular to the extension direction of the line spot. The imaging area of ​​the multi-element detector consists of n pixel rows arranged along the x-direction, the illumination intensity of the line illumination light of both channels is Gaussian distributed in the x-direction, and the spacing between the position parameters of the line spots of the two channels is 1 pixel.

[0058] In this embodiment, the multi-element detector can be an area-array CCD (charge-coupled device) or an area-array CMOS (Complementary Metal Oxide Semiconductor) camera with sub-array or ROI (Region of Interest) functionality, or it can be a linear CCD or linear CMOS camera with area array mode functionality.

[0059] Further, referring to Figure 1, the imaging module 20 includes:

[0060] The scanning unit 201 is used to continuously scan and image along a first direction using a multi-element detector with n rows of pixels, where n is greater than or equal to m;

[0061] Image patch acquisition unit 202 acquires the strip image patch of the i-th row of pixels in each frame of a sample obtained in chronological order;

[0062] The stitching unit 203 stitches together the strip image blocks of the i-th row of pixels in each frame of a sample to obtain a mixed image of the i-th row of pixels, where i∈n.

[0063] Figure 5 is a schematic diagram of the sample imaging acquisition process of the present invention. The function of the imaging module 20 is described below with reference to Figure 5. In order to facilitate subsequent image demodulation, the imaging area of ​​the camera in this embodiment is n pixel rows arranged along the x-direction. The movement direction of the sample is also along the x-direction, which facilitates the scanning and imaging of the sample under different pixels, realizes m-channel multicolor imaging, and expands the number of channels for multicolor imaging.

[0064] Referring to Figure 5(a), during the imaging acquisition process, the scanning unit 201 uses a multi-element detector with n rows of pixels to continuously scan and image along the x-direction. Specifically, the motorized translation stage 16 drives the sample 15 to move at a constant speed along the x-direction, and the single-frame exposure time of the camera is equal to the time it takes for the sample to move one pixel across the corresponding width in the object space. If any pixel row in a single frame is defined as a strip image block, then the multiple strip image blocks corresponding to that pixel row in multiple frames represent the sequential and continuous imaging of various parts of the sample.

[0065] Referring to Figure 5(b), these strip image blocks obtained from sequential imaging over time are stitched together to obtain a mixed image corresponding to each pixel row. Each mixed image contains fluorescence signals from m channels. Specifically, the i-th pixel row is selected from the n pixel rows, and the image block acquisition unit 202 acquires the strip image block of the i-th pixel row in each frame image of a sample obtained in time sequence. The stitching unit 203 stitches the strip image blocks of the i-th pixel row in each frame image of a sample sequentially to obtain a mixed image of the i-th pixel row. Different pixel rows are selected and this process is repeated at least m times to obtain m mixed images of different pixel rows.

[0066] The following mainly explains the demodulation principle and process in fluorescence imaging. The demodulation principle and process in non-fluorescence imaging are basically the same as those in fluorescence imaging, with the main difference being the different detection wavelengths, resulting in a slight difference in the effective PSF of the system. This difference can be ignored, and the method is still applicable.

[0067] Similar to the imaging principle of line scanning confocal microscopy, in this method, the mixed image corresponding to each pixel row is the convolution of the system's effective PSF (Point spread function) and the fluorescence signal distribution on the sample. The effective PSF of the system is the product of the illumination system PSF and the detection system PSF.

[0068] Taking the fluorescence signal of a single channel as an example, the effective PSF of the j-th channel in the i-th pixel row is:

[0069]

[0070] in, Indicates the lighting system PSF, Let PSF represent the detection system, i represent the row number of each pixel in the imaging region, where i∈n, j represent the type number of the fluorescent protein, which is also the channel number, and x and y represent the horizontal coordinates of the object space, with directions shown in Figures 2 and 3. The x-direction is the direction of sample movement, and the y-direction is the direction perpendicular to the sample movement. i,j This represents the distance between the i-th pixel row and the j-th channel line spot position parameter.

[0071] Therefore, the signal of the j-th channel contained in the mixed image corresponding to the i-th pixel row can be expressed as:

[0072]

[0073] Among them, f j(x,y) represents the relative concentration distribution of the j-th fluorescent protein on the sample, and "*" represents the convolution operation. As can be seen from formula (2), the fluorescence signal of the sample is modulated by both the illumination system PSF and the detection system PSF. Since the detection system PSF of each pixel of the camera is the same, the difference in modulation by the line illumination modulation module is mainly reflected in the illumination system PSF.

[0074] Under multicolor illumination, the mixed image contains m channels of fluorescence signals. Due to the misalignment of the line spots in different channels, the sample fluorescence signals of different channels are modulated differently by the line illumination for the same pixel row. The mixed image of the i-th pixel row can be represented as:

[0075] I i (x,y)=h i,1 (x,y)*f1(x,y)+h i,2 (x,y)*f2(x,y) (3)

[0076] Similarly, the blended image of the (i+1)th pixel row can be represented as:

[0077] I i+1 (x,y)=h i+1,1 (x,y)*f1(x,y)+h i+1,2 (x,y)*f2(x,y) (4)

[0078] Based on the above principles, by processing the mixed image using frequency domain demodulation and spatial domain demodulation algorithms, m monochrome images can be demodulated from the m mixed images. Each monochrome image corresponds to a channel fluorescence signal, and the channels corresponding to each monochrome image are different.

[0079] In some embodiments, a frequency domain demodulation algorithm is employed, using the optical transfer function as its modulation coefficient, to perform linear elimination on the mixed image in the frequency domain, thereby demodulating the monochrome image. Referring to Figure 6, the image demodulation module 30 includes:

[0080] Fourier transform unit 301 is used to perform Fourier transform on the original spatial domain images of m mixed images to obtain m mixed frequency domain images;

[0081] The first demodulation unit 302 is used to combine the effective optical transfer function of the corresponding pixel row as its demodulation coefficient, perform linear elimination operation on m of the mixed frequency domain images, and demodulate m monochrome frequency domain images, each of the monochrome frequency domain images containing a channel of fluorescence signal;

[0082] The inverse Fourier transform unit 303 is used to perform inverse Fourier transform on m monochrome frequency domain images to obtain m monochrome images.

[0083] Figure 7 shows the frequency domain demodulation algorithm flow for multi-channel fluorescence signals, which specifically includes:

[0084] First, the Fourier transform unit 301 performs a Fourier transform on the original spatial domain images of the m mixed images to obtain m mixed frequency domain images. Then, the first demodulation unit 302 uses the effective optical transfer function of the corresponding pixel row as its demodulation coefficient to perform linear elimination on the m mixed frequency domain images to demodulate m monochrome frequency domain images. Each monochrome frequency domain image contains a fluorescence signal of one channel. Finally, the inverse Fourier transform unit 303 performs an inverse Fourier transform on the m monochrome frequency domain images to obtain m monochrome images, each of which contains a fluorescence signal of one channel.

[0085] In other embodiments, s (n≥s>m) mixed images can be selected for demodulating m-channel monochrome images. Specifically, the first to m mixed images (denoted as the first group of mixed images) are selected from the s images. Following the frequency domain demodulation algorithm described above, the first group of m-channel monochrome images is demodulated. Then, the (m+1)th image from the s mixed images replaces any one image in the first group, and this is denoted as the second group of mixed images. Next, following the frequency domain demodulation algorithm described above, the second group of m-channel monochrome images is demodulated. Then, the (m+2)th image from the s mixed images replaces any one image in the second group, and so on, until the (s-m+1)th group of m-channel monochrome images is demodulated. Finally, the images with the same channel number in the (s-m+1)th demodulated groups are summed to obtain a monochrome image with a higher signal-to-noise ratio for the m channels.

[0086] The following example illustrates the case with two channels. Performing a Fourier transform on the spatial domain images of formulas (3) and (4), we obtain the frequency domain images of the mixed image of the i-th pixel row and the (i+1)-th pixel row, respectively:

[0087] O i (u,v)=H i,1 (u,v)×F1(u,v)+H i,2 (u,v)×F2(u,v) (5)

[0088] O i+1 (u,v)=H i+1,1 (u,v)×F1(u,v)+H i+1,2 (u,v)×F2(u,v) (6)

[0089] Where H i,1 (u,v), H i+1,1 (u,v), H i,2 (u,v), H i+1,2 (u,v), F1(u,v), and F2(u,v) represent h respectively.i,1 (x,y),h i+1,1 (x,y),h i,2 (x,y),h i+1,2 The Fourier transform results of f(x,y), f1(x,y), and f2(x,y), H i,1 (u,v), H i+1,1 (u,v), H i,2 (u,v) and H i+1,2 (u,v) can be obtained through theoretical simulation or experimental measurement.

[0090] Alternatively, a theoretical simulation method can be used, specifically: h is calculated by combining formula (1) with the theoretical expression of PSF and the relevant parameters of the actual system. i,1 (x,y),h i+1,1 (x,y),h i,2 (x,y) and h i+1,2 (x,y), and then perform Fourier transforms on them respectively.

[0091] Alternatively, an experimental method can be used, specifically by measuring the effective PSFh of different channels in different pixel rows. i,1 (x,y),h i+1,1 (x,y),h i,2 (x,y) and h i+1,2 (x,y), and then perform Fourier transforms on them respectively.

[0092] Therefore, in formulas (5) and (6), there are only two unknowns, F1(u,v) and F2(u,v). By performing linear elimination on the above mixed frequency domain image, the monochrome frequency domain images of channel 1 and channel 2 can be demodulated. Specifically, this includes:

[0093]

[0094]

[0095] Furthermore, by performing inverse Fourier transforms on the monochrome frequency domain images F1(u,v) and F2(u,v), the monochrome images of channel 1 and channel 2 can be demodulated.

[0096] Similarly, when the number of channels m>2, following formulas (5) and (6), m linear equations can be listed, each containing m unknowns. By solving these m linear equations, the monochrome image of m channels can be demodulated.

[0097] In other embodiments, a spatial domain demodulation algorithm can also be used. The spatial domain demodulation algorithm is an approximate demodulation algorithm that uses the illumination intensity as its modulation coefficient and performs linear elimination on the mixed image in the spatial domain to finally achieve multicolor demodulation. In formula (1), the effective PSF obtained by multiplying the illumination PSF and the detector PSF, which are Gaussian distributed, can still be approximated as Gaussian distributed, and its intensity decreases rapidly with the increase of the distance from the center. Therefore, we can only consider the contribution of the central region of the effective PSF to the imaging and ignore the intensity value of the region around the effective PSF. Then the effective PSF can be approximated as an impulse function. In addition, since there is a certain displacement between the illumination PSF and the detector PSF, the center of the effective PSF is no longer fixed at the origin and will have a certain displacement. The direction and magnitude of the displacement change with the direction and magnitude of the displacement between the illumination PSF and the detector PSF.

[0098] Taking the effective PSF of channel 1 in the i-th pixel row as an example, it can be expressed as:

[0099]

[0100] in, δ(x,y) represents the x-coordinate corresponding to the maximum effective PSF intensity, and δ(x,y) represents the unit impulse function.

[0101] Similarly, we can conclude that:

[0102]

[0103]

[0104]

[0105] make

[0106]

[0107]

[0108]

[0109]

[0110] Substituting formulas (9)-(16) into formulas (3) and (4), we get:

[0111]

[0112]

[0113] Where L i,1 L i,2 L i+1,1 L i+1,2 and It can be measured through simulation or experiment.

[0114] Alternatively, simulation can be used, specifically by calculating each effective PSFh using formula (1) combined with the theoretical expression of PSF and the relevant parameters of the system. i,1 (x,y),h i+1,1 (x,y),h i,2 (x,y) and h i+1,2 (x,y), the intensity value of the strongest point at the center of each effective PSF is L. i,1 L i,2 L i+1,1 L i+1,2 The displacement of the strongest point at the center of each effective PSF relative to the origin is...

[0115] Alternatively, it can be measured experimentally, specifically by: turning on the light source of channel 1 alone, scanning the sample along the x-direction, acquiring one sample image from the i-th pixel row and one from the (i+1)-th pixel row, and calculating the distance between these two images along the x-axis, which is the distance between the two images. And calculate the ratio of the average gray values ​​between the two images, which is L. i,1 / L i+1,1 Following the measurement process described above, it is also possible to measure... and L i,2 / L i+1,2 ,because and and The same variables can be transformed by translation, making them equivalent. Therefore, Equations (17) and (18) are equivalent to having only two unknown variables. By performing translation transformation and linear elimination on the mixed image, the signal of one of the channels can be eliminated, and a monochrome image containing only a single channel signal can be demodulated.

[0116] Based on the above principles, in implementation, a spatial domain demodulation algorithm is used to sequentially eliminate the fluorescence signals of m-1 non-target channels from m mixed images through translation alignment and linear elimination operations, thereby demodulating a single monochrome image corresponding to only one target channel.

[0117] Specifically, referring to Figure 8, the image demodulation module 30 includes:

[0118] The first loop control unit 304 is used to set the value of the channel counter k to 1 and repeatedly execute the second demodulation unit. Each time it is executed, the value of the channel counter is incremented by 1 until the value of the channel counter k is equal to m-1.

[0119] The second demodulation unit 305 is used to set the value of the image counter p to 1, and repeats the following sub-modules until the value of the image counter is equal to mk.

[0120] Translation subunit 3051 is used to translate the mixed image I k Translate it to match the blended image I k+p The fluorescence signals of the k-channels in the samples largely overlap.

[0121] Elimination subunit 3052 is used for processing the mixed image I k With mixed image I k+p Perform linear elimination to eliminate the mixed image I k+p The fluorescence signal of the k-channel in the image;

[0122] The counting subunit 3053 is used to increment the value of the image counter p by 1.

[0123] Figure 9 shows the spatial domain demodulation algorithm flow for the multi-channel fluorescence signal of this invention. The specific process is as follows:

[0124] Set the channel counter k and image counter p to 1. Shift the blended image corresponding to the first pixel row so that the fluorescence signal of channel 1 in the blended image corresponding to the second pixel row substantially overlaps with the fluorescence signal of channel 1 in the blended image corresponding to the second pixel row. Perform linear elimination on the two blended images to eliminate the fluorescence signal of channel 1 in the blended image corresponding to the second pixel row, and still record this as the blended image corresponding to the second pixel row. Then, increment the image counter p by 1. Shift the blended image corresponding to the first pixel row again so that the fluorescence signal of channel 1 in the blended image corresponding to the first pixel row substantially overlaps with the fluorescence signal of channel 1 in the blended image corresponding to the third pixel row. Perform linear elimination again to eliminate the fluorescence signal of channel 1 in the blended image corresponding to the third pixel row, and still record this as the blended image corresponding to the third pixel row. Then, increment the image counter p by 1 again. Repeat the above steps until the image counter p equals m-1, completing the linear elimination operation between the blended image corresponding to the first pixel row and the m-th blended image, resulting in m-1 blended images with the fluorescence signal of channel 1 eliminated.

[0125] Further, increment the channel counter k by 1 and reset the image counter p to 1. Repeat the above steps to perform translation transformation and linear elimination operations on the mixed image containing the fluorescence signal of the second channel until the image counter p equals m-2, resulting in m-2 mixed images with the fluorescence signals of the first and second channels eliminated.

[0126] Repeat the steps of incrementing the channel counter k by 1 and resetting the image counter p to 1, and repeat the above process of hybrid image translation transformation and linear elimination operation until the image counter p equals mk.

[0127] Repeat the above steps until the value of the channel counter k is equal to m-1, and finally obtain a monochrome image containing only the fluorescence signal of the m-th channel.

[0128] Here, the m channel numbers are artificially defined, meaning that any one of the m fluorescent proteins can be used as the m-th channel. Therefore, the channel number of each fluorescent protein can be adjusted according to the different target fluorescent proteins to be demodulated, and the target fluorescent protein to be demodulated can be used as the m-th channel, and then demodulation can be performed according to the above demodulation process.

[0129] Based on the above methods, by selecting different mixed images, different non-target channel fluorescence signals can be eliminated, thereby obtaining a monochrome image containing only the fluorescence signal of any target channel.

[0130] In other embodiments, s (n≥s>m) mixed images can be selected for demodulating a monochrome image with m channels. Specifically, the first to m mixed images (denoted as the first group of mixed images) are selected from the s images. Following the spatial domain demodulation algorithm described above, the monochrome image of the first group of m channels is demodulated. Then, the (m+1)th image from the s mixed images replaces any one image in the first group of images, and this is denoted as the second group of mixed images. Next, following the spatial domain demodulation algorithm described above, the monochrome image of the second group of m channels is demodulated. Then, the (m+2)th image from the s mixed images replaces any one image in the second group of images, and so on, until the monochrome image of the (s-m+1)th group of m channels is demodulated. Finally, the images with the same channel number in the demodulated (s-m+1)th groups of images are summed to obtain a monochrome image with m channels and a higher signal-to-noise ratio.

[0131] Similarly, taking the case of two channels as an example, the images g1(x,y) and g2(x,y) containing only the signals of a single channel, channel 1 and channel 2, can be represented as:

[0132]

[0133]

[0134] In formula (19) and They can be converted to each other by translation along the x-direction, with the translation distance being:

[0135]

[0136] because and Since the signs are the same and the values ​​are very small and close, the value of D is close to 0, and therefore can be ignored. and The misalignment between them can be considered as:

[0137]

[0138] Similarly,

[0139]

[0140] Furthermore, substituting formulas (22) and (23) into formulas (19) and (20), we obtain:

[0141]

[0142]

[0143] Therefore, the demodulation result g j (x,y) and the original fluorescence signal f j Aside from a certain displacement, the values ​​(x, y) differ only in a constant coefficient factor. Since this constant coefficient factor only changes the overall grayscale value of the image, the demodulation result g can be considered... j (x,y) represents the actual distribution of signals in each channel of the sample.

[0144] Synchronous multicolor imaging is achieved on a single camera. The images between each channel have a natural registration relationship, eliminating the need for additional registration processing. By simply processing the mixed image through a demodulation algorithm, a monochrome image containing the fluorescence signal of a single channel can be obtained, greatly reducing the complexity of the system.

[0145] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A line illumination modulation multicolor imaging system, characterized in that, The line illumination modulation multicolor imaging system includes: a line illumination modulation module comprising m monochromatic light sources and a modulation optical path, wherein each monochromatic light source emits a beam with a different wavelength; each beam passes through the modulation optical path to form a line spot focused on the focal plane of an objective lens, the illumination intensity of the line spot on the focal plane of the objective lens is Gaussian distributed in a first direction, the first direction being perpendicular to the extension direction of the line spot, each line spot having different position parameters, and the line spots of the m channels superimposed to form m-color line illumination light, wherein m ≥ 2 and is a positive integer; an imaging module for continuously scanning and imaging along the first direction using a multi-element detector with n rows of pixels to obtain at least m mixed images under the illumination of the m-color line illumination light, each mixed image corresponding to one row of pixels, each mixed image containing signals of m channels, wherein n ≥ m and is a positive integer; and an image demodulation module for demodulating a monochromatic image from the mixed image using a frequency domain demodulation algorithm or a spatial domain demodulation algorithm, wherein the monochromatic image corresponds to a signal of one channel; wherein the modulation... The optical path includes a shaping optical path for shaping the light beam into a line beam, a position adjustment optical path for adjusting the position parameters of each line spot, and a projection optical path for superimposing the line spots to form m-color line illumination light; the shaping optical path includes a first dichroic mirror group for combining m light sources, and a first beam expander, a second beam expander, and a cylindrical lens arranged sequentially along the beam propagation direction after beam combining, the first dichroic mirror group including m-1 dichroic mirrors; the position adjustment optical path is located between the second beam expander and the cylindrical lens, including a first... The system comprises a second dichroic mirror group, a third dichroic mirror group, a first reflecting mirror, and a second reflecting mirror. The second dichroic mirror group contains m-1 dichroic mirrors, and the third dichroic mirror group contains m-1 dichroic mirrors. Each time a beam of light passes through a dichroic mirror in the second dichroic mirror group, a light path is separated. The separated light paths are then combined by the corresponding dichroic mirrors in the third dichroic mirror group. The remaining beam of light after passing through the second dichroic mirror group enters the third dichroic mirror group for further beam combining via the first reflecting mirror and the second reflecting mirror.

2. The line illumination modulation multicolor imaging system according to claim 1, characterized in that, The position adjustment optical path also includes a third reflecting mirror and a fourth reflecting mirror disposed in the optical path between the third dichroic mirror group and the cylindrical lens.

3. The line illumination modulation multicolor imaging system according to claim 1, characterized in that, The spacing between the position parameters of each line spot is greater than or equal to the width of a single pixel of the multi-element detector in the object space.

4. The line illumination modulation multicolor imaging system according to claim 1, characterized in that, The imaging module includes: a scanning unit for continuously scanning and imaging along a first direction using a multi-element detector with n rows of pixels, where n ≥ m; an image block acquisition unit for acquiring a strip image block of the i-th pixel row in each frame image of a sample obtained in chronological order; and a stitching unit for sequentially stitching the strip image blocks of the i-th pixel row in each frame image of a sample to obtain a mixed image of the i-th pixel row, where i ∈ n.

5. The line illumination modulation multicolor imaging system according to any one of claims 1 to 4, characterized in that, The image demodulation module is used to demodulate m monochrome images from m mixed images using a frequency domain demodulation algorithm. Each monochrome image corresponds to a signal of one channel. Specifically, it includes: a Fourier transform unit, used to perform Fourier transform on the original spatial domain images of the m mixed images to obtain m mixed frequency domain images; a first demodulation unit, used to combine the effective optical transfer function of the corresponding pixel row as its demodulation coefficients to perform linear elimination operation on the m mixed frequency domain images to demodulate m monochrome frequency domain images, each monochrome frequency domain image containing a signal of one channel; and an inverse Fourier transform unit, used to perform inverse Fourier transform on the m monochrome frequency domain images to obtain m monochrome images.

6. The line illumination modulation multicolor imaging system according to any one of claims 1 to 4, characterized in that, The image demodulation module is used to demodulate a monochrome image from m mixed images by sequentially eliminating the signals of m-1 non-target channels through translation alignment and linear elimination operations using a spatial domain demodulation algorithm. The monochrome image corresponds to the signal of the target channel.

7. The line illumination modulation multicolor imaging system according to claim 6, characterized in that, The image demodulation module includes: a first loop control unit, used to set the value of channel counter k to 1 and repeatedly execute the second demodulation unit, incrementing the channel counter value by 1 each time it is executed, until the value of channel counter k equals m-1; the second demodulation unit, used to set the value of image counter p to 1 and repeatedly execute the following sub-modules until the value of image counter p equals mk; and a translation sub-unit, used to translate the mixed image I... k Translate it to match the blended image I k+p The signals in the k-channels are largely overlapping; the elimination subunit is used to process the mixed image I. k With mixed image I k+p Perform linear elimination to eliminate the mixed image I k+p The signal of channel k in the middle, and still denoted as I. k+p ; The counting subunit is used to increment the value of the image counter p by 1.

Images