Multilayer structured light microscopic imaging splicing method, system and device

Abstract

The embodiment of the invention provides a multilayer structured light microscopic imaging splicing method, system and device, and the method is executed based on a processor, and comprises the steps: obtaining a plurality of first image data sets corresponding to each of a plurality of directions of structured light of a to-be-detected sample at a first axial position; acquiring a plurality of second image data sets corresponding to each of a plurality of directions of the structured light of the to-be-detected sample at the second axial position; image splicing is carried out based on the multiple first image data sets and the multiple second image data sets, a three-dimensional spliced image is obtained, and the axial range of the three-dimensional spliced image comprises a set of the axial range, corresponding to the first axial position, of the to-be-detected sample and the axial range, corresponding to the second axial position, of the to-be-detected sample. The system comprises a first acquisition module, a second acquisition module and a splicing module.

Description

Technical Field

[0001] This specification relates to the field of microscopic imaging, and particularly to a method, system and device for stitching multi-layer structured illumination microscopy images. Background Art

[0002] Currently, among many super-resolution fluorescence microscopy techniques that break through the diffraction limit, Structured Illumination Microscopy (SIM) is widely used in live cell imaging because of its advantages of simple labeling, low light dose, low phototoxicity, and high time resolution. With the development of technology, compared with two-dimensional imaging, the current SIM technology has problems in three-dimensional structured illumination microscopy (3D-SIM) of live cells, such as slow imaging speed (usually 0.5 - 1 Hz), serious cell motion artifacts, resulting in the inability to clearly capture the rapid three-dimensional movement of live cells. Using Three-Dimensional Multiplane Structured Illumination Microscopy (3D-MP-SIM) for image reconstruction can significantly improve the imaging speed, with a maximum volumetric imaging rate of up to 11 Hz, which is 8 times more efficient than traditional techniques. However, the physical model of this technology is different from that of traditional 3D-SIM, and only the signal of a certain number of layers of samples (for example, 8 layers, with a thickness of about 1.55 μm) is collected simultaneously, which limits its imaging ability for thicker cells in practical applications.

[0003] Therefore, a method, system and device for stitching multi-layer structured illumination microscopy images are provided. By using multi-layer structured illumination microscopy technology (for example, 3D-MP-SIM technology), before image reconstruction, multi-layer image data is obtained from different axial positions to achieve stitching of any thickness, thereby breaking through the limitations of the prior art, more comprehensively capturing the morphology and structure of the sample, and achieving high-speed three-dimensional super-resolution imaging of thicker live cells. Summary of the Invention

[0004] One or more embodiments of this specification provide a method for stitching multi-layer structured light microscopic imaging, including: obtaining a plurality of first image datasets corresponding to each of a plurality of directions of structured light at a first axial position of a sample to be measured; obtaining a plurality of second image datasets corresponding to each of the plurality of directions of the structured light at a second axial position of the sample to be measured; performing image stitching based on the plurality of first image datasets and the plurality of second image datasets to obtain a three-dimensional stitched image, and the axial range of the three-dimensional stitched image includes the union of the axial range corresponding to the sample to be measured at the first axial position and the axial range corresponding to the sample to be measured at the second axial position.

[0005] One embodiment of this specification provides a multi-layer structured light microscopic imaging stitching system, including a first acquisition module, a second acquisition module, and a stitching module; the first acquisition module is configured to obtain a plurality of first image datasets corresponding to each of a plurality of directions of structured light at a first axial position of a sample to be measured; the second acquisition module is configured to obtain a plurality of second image datasets corresponding to each of the plurality of directions of the structured light at a second axial position of the sample to be measured; the stitching module is configured to perform image stitching based on the plurality of first image datasets and the plurality of second image datasets to obtain a three-dimensional stitched image, and the axial range of the three-dimensional stitched image includes the union of the axial range corresponding to the sample to be measured at the first axial position and the axial range corresponding to the sample to be measured at the second axial position. Description of the Drawings

[0006] This specification will be further described by way of exemplary embodiments, which will be described in detail through the drawings. These embodiments are not restrictive. In these embodiments, the same numbers represent the same structures, where:

[0007] Figure 1 is a block diagram of a multi-layer structured light microscopic imaging stitching system shown according to some embodiments of this specification;

[0008] Figure 2 is an exemplary flowchart of a multi-layer structured light microscopic imaging stitching method shown according to some embodiments of this specification;

[0009] Figure 3 is an imaging schematic diagram of the axial movement of a sample to be measured shown according to some embodiments of this specification;

[0010] Figure 4 is an exemplary flowchart of determining a three-dimensional stitched image shown according to some embodiments of this specification;

[0011] Figure 5It is a schematic diagram of the registration of multi-layer image data shown in some embodiments of this specification;

[0012] Figure 6 It is a schematic diagram of the optical path for generating structured light shown in some embodiments of this specification;

[0013] Figure 7 It is a schematic diagram of the principle of image reconstruction shown in some embodiments of this specification. Detailed implementation manners

[0014] To more clearly illustrate the technical solutions of the embodiments of this specification, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some examples or embodiments of this specification. For those of ordinary skill in the art, without creative efforts, this specification can also be applied to other similar scenarios according to these drawings. Unless obvious from the language context or otherwise stated, the same reference numerals in the drawings represent the same structure or operation.

[0015] It should be understood that the "system", "device", "unit" and / or "module" used herein is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the said words can be replaced by other expressions.

[0016] As shown in this specification, unless the context clearly indicates an exceptional situation, words such as "a", "an", "one" and / or "the" are not specifically singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0017] Flowcharts are used in this specification to illustrate the operations performed by the systems according to the embodiments of this specification. It should be understood that the previous or subsequent operations are not necessarily executed precisely in sequence. On the contrary, the steps can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or several steps can be removed from these processes.

[0018] Figure 1 It is a module diagram of a multi-layer structured light microscopic imaging stitching system shown in some embodiments of this specification.

[0019] In some embodiments, as Figure 1 shown, the multi-layer structured light microscopic imaging stitching system 100 may include a first acquisition module 110, a second acquisition module 120, and a stitching module 130.

[0020] The first acquisition module 110 is configured to acquire a plurality of first image datasets corresponding to each of a plurality of directions of structured light at a first axial position of a sample to be measured.

[0021] The second acquisition module 120 is configured to acquire a plurality of second image datasets corresponding to each of a plurality of directions of structured light at a second axial position of the sample to be measured.

[0022] The stitching module 130 is configured to perform image stitching based on the plurality of first image datasets and the plurality of second image datasets to obtain a three-dimensional stitched image, and the axial range of the three-dimensional stitched image includes the union of the axial range corresponding to the first axial position of the sample to be measured and the axial range corresponding to the second axial position of the sample to be measured.

[0023] For more descriptions of the first acquisition module 110, the second acquisition module 120, and the stitching module 130, see Figures 2 - 6 and related content.

[0024] In some embodiments, the first acquisition module 110, the second acquisition module 120, the stitching module 130, etc. of the multi-layer structured light microscopic imaging stitching system 100 may be integrated onto a processor. The processor is configured to process information and / or data related to the multi-layer structured light microscopic imaging stitching system 100. The processor may also process data obtained from other devices, such as a user terminal, which may include a personal computer, a mobile device, etc. In some embodiments, the processor may be configured as one or a combination of an Embedded Processor, a Graphics Processing Unit (GPU), a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), etc.

[0025] It should be noted that the above descriptions of the multi-layer structured light microscopic imaging stitching system and its modules are only for convenience of description and do not limit this specification to the scope of the examples given. It can be understood that for those skilled in the art, after understanding the principle of the system, they may, without departing from this principle, make any combination of the various modules, or form a subsystem and connect it with other modules. In some embodiments, Figure 1 the first acquisition module, the second acquisition module, and the stitching module disclosed in may be different modules in a system, or one module may implement the functions of two or more of the above modules. For example, the various modules may share a storage module, or each module may have its own storage module respectively. Such variations are all within the protection scope of this specification.

[0026] Figure 2 It is an exemplary flowchart of a multi-layer structured light microscopic imaging stitching method shown in some embodiments of this specification.

[0027] Process 200 is an exemplary process of a multi-layer structured light microscopic imaging stitching method. As Figure 2 shown, process 200 includes the following steps 210 - step 230. In some embodiments, process 200 can be executed by a processor.

[0028] Step 210, obtain a plurality of first image datasets corresponding to each polarization direction among a plurality of structured light directions at a first axial position of a sample to be measured.

[0029] The sample to be measured refers to a sample that needs to be observed. For example, a live cell sample that needs to be observed.

[0030] The first axial position refers to the observation position of the sample to be measured in the microscope imaging system, that is, the position where the stage on which the sample to be measured is placed is located axially. The axial direction is the direction along the microscope optical axis, for example, the vertical direction.

[0031] The direction of the structured light refers to the irradiation direction of the structured light used in the imaging process. Structured light refers to a light source with a specific pattern (such as stripes, etc.). The direction of the structured light can be adjusted by a spatial light modulator (SLM). In some embodiments, the structured light includes 3 directions. In some embodiments, the number of directions of the structured light can also be other numbers (for example, an integer greater than 3).

[0032] Figure 6 It is a schematic optical path diagram for generating structured light shown in some embodiments of this specification.

[0033] The structured light can be generated by an illumination component. In some embodiments, as Figure 6 shown, the illumination component includes at least one of a light source, acousto-optic tunable filters (such as AOTF1 and AOTF2), polarization beam splitters (such as PBS1 and PBS2), half-wave plates (such as HWP1 and HWP2), spatial light modulators (such as SLM1 and SLM2), polarization rotators (such as Mask1 and Mask2), dichroic mirrors (such as DM), objective lenses (such as OBJ), and one or more lenses (such as L1 - L8), etc. The light source can be a laser (such as Figure 6 Laser1 and Laser2 in

[0034] In some embodiments, the structured light in each direction has a preset polarization direction. The preset polarization direction can be s polarization (i.e., linear polarization perpendicular to the incident plane) to enhance the modulation contrast of the structured light illumination. The preset polarization direction can be adjusted by a polarization rotator (such as a high-speed liquid crystal variable retarder (HS-LCVR), a quarter-wave plate (QWP), etc.).

[0035] The first image dataset refers to the collection of image data of the structured light in a certain direction at the first axial position of the sample to be measured. There are multiple first image datasets obtained in each direction. The number of the first image datasets can be preset according to experience. For example, 10 first image datasets are obtained in each direction. The first image dataset can be obtained by a collection device. The collection device can be configured as a camera. Exemplarily, as Figure 6 shown, the collection device is a camera (including Camera1 and Camera2).

[0036] In some embodiments, the processor can obtain multiple first image datasets corresponding to each direction among multiple directions of the sample to be measured at the first axial position based on a preset microscopy imaging method.

[0037] The preset microscopy imaging method can include Three-Dimensional Multiplane Structured Illumination Microscopy (3D-MP-SIM), Three-Dimensional Structured Illumination Microscopy (3D-SIM), and / or Multiplane Structured Illumination Microscopy (MP-SIM), etc. This specification takes 3D-MP-SIM as an example for illustration and does not limit its protection scope.

[0038] Step 220: Obtain multiple second image datasets corresponding to each direction among multiple directions of the structured light at the second axial position of the sample to be measured.

[0039] The second axial position refers to another observation position after the sample to be measured moves axially in the microscope imaging system, that is, the position where the stage on which the sample to be measured is placed is located after moving a certain distance axially relative to the first axial position. The processor can axially move the sample to be measured by a preset moving method.

[0040] The second image dataset refers to the collection of image data of the sample to be measured in a certain direction at the second axial position.

[0041] In some embodiments, the processor may obtain, based on a preset microscopic imaging method, a plurality of second image data sets corresponding to each of a plurality of directions of structured light of a sample to be measured at a second axial position.

[0042] Figure 3 It is a schematic diagram of imaging during the axial movement of a sample to be measured as shown in some embodiments of this specification.

[0043] Exemplarily, as Figure 3 shown, when the sample to be measured moves in the axial direction from the position shown in (a) in Figure 3 to the position shown in (b) in Figure 3 (for example, the axial movement distance is z0): the first axial position is the axial position where the sample to be measured is located as shown in (a) in Figure 3 ; the second axial position is the axial position where the sample to be measured is located as shown in (b) in Figure 3 ; the first image data set is the set D1 of image data collected by the camera when the sample to be measured is at the position shown in (a) in Figure 3 ; and the second image data set is the set D2 of image data collected when the sample to be measured is at the position shown in (b) in Figure 3 . When the sample to be measured moves in the axial direction from the position shown in (b) in Figure 3 to the position shown in (c) in Figure 3 (for example, the axial movement distance is z0): the first axial position is the axial position of the sample to be measured as shown in (b) in Figure 3 ; the first image data set is the set D2 of image data collected by the camera when the sample to be measured is at the position shown in (b) in Figure 3 ; and the second image data set is the set D3 of image data collected by the camera when the sample to be measured is at the position shown in (c) in Figure 3 .

[0044] In some embodiments, each of the plurality of first image data sets and the plurality of second image data sets includes multi-layer image data. Each layer of image data corresponds to information of the sample to be measured on a certain layer or plane (perpendicular to the axial direction). The multi-layer image data corresponds to information of the sample to be measured on a plurality of layers or planes (perpendicular to the axial direction), and the positions of these multiple planes are different in the axial direction. In some embodiments, each image data set may include 8 layers of image data. In some embodiments, each image data set may include other numbers of layers of image data, such as 6 layers, 10 layers, etc. Exemplarily, as Figure 6 shown, each time of acquisition, each camera includes 4 different regions, corresponding to 4 different sample planes (two cameras correspond to 8 planes), each plane corresponds to one layer of image data, and a set of 8 layers of image data obtained each time of acquisition is used as an image data set.

[0045] In some embodiments, the processor may also acquire image data sets of the sample to be measured at three or more axial positions; and perform image stitching based on the image data sets at the three or more axial positions to obtain a three-dimensional stitched image. For example, the processor may separately acquire the image data sets D1, D2, and D3 of the sample to be measured at the axial positions shown in (a), (b), and (c) in Figure 3 ; and perform image stitching based on the image data sets D1, D2, and D3 to obtain a three-dimensional stitched image.

[0046] In some embodiments of the present specification, by collecting multi-layer image data, the axial image data can be expanded, so that the information in the stitched three-dimensional image data is more complete.

[0047] In some embodiments, there is a certain distance between the first axial position and the second axial position in the axial direction, and the axial distance satisfies a preset condition. Herein, the axial distance refers to the distance between the first axial position and the second axial position in the axial direction. Exemplarily, as Figure 3 shown, the axial distance is z0.

[0048] The preset condition is related to the axial period. The axial period refers to the illumination period of the structured light illumination pattern in the axial direction. When the three-dimensional structured light microscope is imaging, three beams of light interfere with each other, generating not only cosine-shaped structured light with bright and dark intervals in the transverse direction, but also periodic illumination in the axial direction. For example, if the illumination intensity of the structured light in the axial direction is a cosine or sine function, the illumination period is the period of the cosine or sine function. Exemplarily, the axial period may be 678 nm.

[0049] In some embodiments, the axial period can be calculated by formula (1):

[0050] where P is the axial period; λ is the excitation wavelength, and λ may be 488 nm; n is the refractive index of water, and n may be 1.333; NAexc is the excitation energy of the water-containing sample in 3D-MP-SIM. For example, NAexc may be 1.184.

[0051] In some embodiments, the processor may determine the axial distance based on the axial period. The axial distance is positively correlated with the axial period. For example, the axial distance may be an integer multiple of the axial period. By setting the axial distance as an integer multiple of the axial period, it can be ensured that the structured light parameters corresponding to the images at different axial positions are consistent, thereby ensuring the accuracy of stitching and avoiding stitching errors caused by inconsistent structured light parameters.

[0052] In some embodiments of the present specification, by precisely controlling the axial distance through the preset condition, the multi-layer image data acquired when the sample to be measured is at different axial positions can be successfully stitched.

[0053] In some embodiments, the preset condition is further related to the layer spacing between two adjacent layers in the multi-layer image data.

[0054] The layer spacing refers to the axial distance between two adjacent layers among multiple layers of the sample to be measured corresponding to the multi-layer image data. The layer spacing in the same microscopic imaging system can be preset and fixed, and the layer spacing can be preset according to experience. For example, the layer spacing can be 193.8 nm.

[0055] In some embodiments, the processor can determine the axial distance based on the layer spacing. The axial distance can be positively correlated with the layer spacing. For example, the processor can calculate the axial distance based on the layer spacing through formula (2), and formula (2) is as follows: z0 = k × L (2)

[0056] Wherein, z0 represents the axial distance; L represents the layer spacing; k is an integer greater than 0 and k is less than or equal to the number of layers of image data included in an image data set. For example, if an image data set includes 8 layers of image data, then k takes an integer from 1 to 8.

[0057] Determining the axial distance through the layer spacing can ensure that the number of overlapping layers corresponding to the multi-layer image data obtained at the first axial position and the second axial position is an integer, which is beneficial to the successful combination of two sets of multi-layer image data during splicing. At the same time, it ensures that a signal continuous super-resolution image of a thicker sample can be successfully reconstructed.

[0058] In some embodiments, the preset condition is that the axial distance is a common multiple of the axial period and the layer spacing. Exemplarily, the axial period is 678 nm and the layer spacing is 193.8 nm, and the axial distance can be 1.356 μm (i.e., 2 times the axial period and 7 times the layer spacing).

[0059] Determining the axial distance jointly by the axial period and the layer spacing can ensure the consistency of the illumination period before and after the movement of the sample to be measured.

[0060] Only by way of example, as Figure 3 shown, Figure 3 in (a), D1 represents a first image data set, and this first image data set can be represented by formula (3):

[0061] Wherein, D1(r) represents the first image data set; z is the axial coordinate (i.e., Z-axis coordinate) of the layer or plane corresponding to the multi-layer image data; r is the transverse coordinate, and r′ refers to the coordinate system relative to the objective lens (i.e., the XY coordinate system, the transverse coordinate system); I represents the illumination mode; H represents the point spread function of the microscopic imaging system; S is the reconstructed super-resolution image.

[0062] As shown Figure 3 in, after axially moving the sample to be measured by z0 as shown in (a) in Figure 3 , the result shown in (b) in Figure 3 is obtained. In (b) in Figure 3 , a second image data set is represented within D2, and this second image data set can be expressed by formula (4):

[0063] where D2(r) represents the second image data set; z0 is the axial distance by which the sample to be measured is moved.

[0064] Since the transverse coordinate system r′ is based on the objective lens and the illumination mode, the integration interval remains within (0, z). If the axial distance z0 is an integer multiple of the axial period of the illumination mode I, I(r′) in formula (4) satisfies formula (5): I(r′) = I(r′ + z0) (5)

[0065] Substituting formula (5) into formula (4) gives formula (6):

[0066] Replacing r′ + z0 with r ″ , formula (7) can be obtained:

[0067] Step 230, perform image stitching based on multiple first image data sets and multiple second image data sets to obtain a three-dimensional stitched image.

[0068] In some embodiments, the axial range of the three-dimensional stitched image includes the union of the axial range corresponding to the first axial position of the sample to be measured and the axial range corresponding to the second axial position of the sample to be measured. The axial range corresponding to a certain axial position of the sample to be measured refers to the axial range of the area of the sample to be measured detected by the microscopic imaging system when the stage on which the sample to be measured is placed is at this axial position. The axial range of the three-dimensional stitched image refers to the axial range of the sample to be measured shown in this three-dimensional stitched image.

[0069] Image stitching refers to combining three-dimensional images at multiple axial positions (one three-dimensional image can correspond to a multi-layer image data) to obtain a new three-dimensional image. The three-dimensional stitched image refers to the three-dimensional image of the sample to be measured after stitching. The three-dimensional stitched image can reflect the three-dimensional information of the sample to be measured in a larger axial range. For example, the axial range corresponding to the sample to be measured at the first axial position is z1 - z2, and the axial range corresponding to the sample to be measured at the second axial position is z3 - z4, where z1 < z3 ≤ z2 < z4. Then, the three-dimensional stitched image can reflect the three-dimensional information of the sample to be measured in the range of z1 - z4.

[0070] In some embodiments, the processor can perform image stitching based on multiple first image datasets and multiple second image datasets through various methods. For example, the processor can stitch the multiple first image datasets and multiple second image datasets axially to obtain a three-dimensional stitched image. If there is an overlapping part between the first image dataset and the second image dataset axially, only the data in the overlapping part of the first image dataset or the second image dataset is taken during stitching. In some embodiments, the processor can also perform spectral separation, frequency shift, and Wiener filtering on the multiple stitched three-dimensional image data to obtain a super-resolution image. For more details, see Figure 4 and related content.

[0071] Some embodiments of this specification include, but are not limited to, the following beneficial effects: (1) Images are acquired and stitched at different axial positions, expanding the imaging depth of the obtained three-dimensional stitched image, which is suitable for imaging thick samples; (2) The spectra in the transverse and axial directions are expanded; (3) By stitching the image data first and then performing reconstruction, the computational complexity of subsequent reconstruction algorithms can be reduced, and the overall efficiency of image processing can be improved.

[0072] In some embodiments, performing image stitching based on multiple first image datasets and multiple second image datasets to obtain a three-dimensional stitched image includes: registering and stitching one first image dataset from the multiple first image datasets corresponding to each direction and one corresponding second image dataset from the multiple second image datasets to obtain a set of three-dimensional image data after registration and stitching; performing spectral separation on the multiple sets of three-dimensional image data corresponding to each direction to obtain multiple spectral components; determining multiple frequency-shifted images corresponding to each direction based on the multiple spectral components and estimated parameters corresponding to each direction; and performing Wiener filtering on the multiple frequency-shifted images corresponding to each direction to obtain a super-resolution reconstructed image of the sample to be measured.

[0073] Figure 4 is an exemplary flowchart for determining a three-dimensional stitched image according to some embodiments of this specification. In some embodiments, such as Figure 4As shown, process 400 is an exemplary process for determining a three-dimensional stitched image. Process 400 can be executed based on a processor. Process 400 includes step 410 - step 440.

[0074] Step 410, register and stitch one first image dataset from each of the multiple first image datasets corresponding to each direction and a corresponding second image dataset from each of the multiple second image datasets to obtain a set of three-dimensional image data after registration and stitching.

[0075] Registration refers to the horizontal alignment operation of image datasets at different axial positions. One first image dataset includes multiple layers of image data (e.g., 8 layers of image data). One second image dataset also includes multiple layers of image data (e.g., 8 layers of image data). The set of three-dimensional image data obtained after registration also includes multiple layers of image data. Since each direction corresponds to multiple first image datasets (e.g., 10 first image datasets) and multiple second image datasets (e.g., 10 second image datasets), each direction also corresponds to multiple sets of three-dimensional image data (e.g., 10 sets of three-dimensional image data).

[0076] In some embodiments, the layers of the multiple layers of image data in the first image dataset (e.g., 8 layers of image data) may not coincide with the corresponding layers of the multiple layers of image data in the second image dataset (e.g., 8 layers of image data). In this case, during the registration operation, the multiple layers of image data (e.g., 8 layers of image data) in one first image dataset and the multiple layers of image data (e.g., 8 layers of image data) in the corresponding second image dataset are combined according to the positions of the different layers and registered and stitched to obtain a set of three-dimensional image data (e.g., 16 layers of image data).

[0077] In some embodiments, there may be overlap between the layers of the multiple layers of image data in the first image dataset and the multiple layers of image data in the second image dataset, that is, one or more layers of image data in the multiple layers of image data in the first image dataset may correspond to the same layer or multiple layers of information of the sample to be measured as one or more layers of image data in the multiple layers of image data in the second image dataset. In this case, during the registration process, for the overlapping part, one or more layers of image data in the multiple layers of image data in the first image dataset located in the overlapping part are selected for the registration operation. Or, one or more layers of image data in the multiple layers of image data in the second image dataset located in the overlapping part, together with the data of other non-overlapping layers (the data of the non-overlapping layers in the first image dataset and the second image dataset), are used for the registration operation.

[0078] For example, if one layer of the multi-layer image data of the first image dataset (e.g., 8-layer image data) coincides with one layer of the multi-layer image data of the second image dataset (e.g., 8-layer image data), then select the layer image data of the first image dataset or the second image dataset located in this one coincident layer, together with the data of other non-coincident layers (i.e., the other layer image data in the first image dataset and the second image dataset except the image data of this one coincident layer), and perform a registration operation. Another example is that if two layers of the multi-layer image data of the first image dataset (e.g., 8-layer image data) coincide with two layers of the multi-layer image data of the second image dataset (e.g., 8-layer image data), then select the layer image data of the first image dataset or the second image dataset located in these two coincident layers, together with the data of other non-coincident layers (i.e., the other layer image data in the first image dataset and the second image dataset except the image data of these two coincident layers), and perform a registration operation.

[0079] Before reconstructing the 3D-MP-SIM image, it is necessary to register the images at different depths. The purpose of registration is to align the images at different depths (i.e., axial positions) for subsequent spectral separation and reconstruction.

[0080] In some embodiments, the processor can extract first feature points from a first image dataset and second feature points from a corresponding second image dataset; match the first feature points and the second feature points to obtain the matched first feature points and second feature points; calculate a transformation matrix based on the matched first feature points and second feature points, and align and register the layer image data in a first image dataset and a corresponding second image dataset to obtain a set of registered three-dimensional image data.

[0081] Figure 5 It is a schematic diagram of the registration of multi-layer image data shown in some embodiments of this specification. Exemplarily, Figure 5 (a) in it represents the multi-layer image data composed of a set of matched first feature points and second feature points (illustrated by taking 8-layer image data as an example). Figure 5 (b) in it represents the projection of the multi-layer image data in the axial direction. Select the area where one layer of image data is located as the reference area (as Figure 5 shown, take the area where the 5th layer of image data is located as the reference area), compare the other layer image data with the reference area, and calculate the registration matrix. Figure 5 (c) in it represents the original image data (including a first image dataset and a corresponding second image dataset). The size of the circle represents the camera areas of eight different axial planes corresponding to the 8-layer image data. The overlapping image on the far right reflects the lateral misalignment between different axial planes. Then, perform fine registration using the registration matrix. Figure 5where (d) represents a set of registered three-dimensional image data. Apply the Figure 5 registration matrix obtained from (b) in Figure 5 to the original image data in (c) for registration, obtaining Figure 5 the set of registered three-dimensional image data in (d).

[0082] Step 420: Perform spectral separation on multiple sets of three-dimensional image data corresponding to each direction to obtain multiple spectral components.

[0083] Spectral separation refers to separating different frequency information in multiple sets of three-dimensional image data. A spectral component refers to an image obtained after spectral separation of three-dimensional image data. In some embodiments, the 3D-MP-SIM technique can utilize the spectral characteristics of a structured illumination pattern to separate the spectrum of a sample to be measured into multiple spectral components. In some embodiments, the spectral components may include 0th, ±1st, and ±2nd order components. In some embodiments, the processor can, based on the Fourier transform, convert multiple sets of three-dimensional image data corresponding to each direction from the spatial domain to the frequency domain, and separate different spectral components in the frequency domain through a separation matrix.

[0084] In some embodiments, spectral separation includes spectral separation in a first direction and a second direction. The first direction and the second direction are different directions of spectral separation. The first direction may be transverse, i.e., the direction perpendicular to the axial direction, such as the horizontal direction. The second direction may be the axial direction. Exemplarily, as Figure 3 and Figure 5 shown, the first direction includes the X-axis and Y-axis directions (i.e., the k xy direction), and the second direction is the Z-axis direction (i.e., the k z direction).

[0085] In some embodiments, the processor can obtain 5 original images (D 1 -D 5 ) with a phase shift of 2π / 5 in three directions d (d = 1, 2, 3) of the structured light, for a total of 15 frames of original images. Each frame of the original image is a first image dataset. Each direction corresponds to 5 frames of original images (i.e., 5 first image datasets). 8 layers of image data (i.e., image data of 8 different depths) are obtained simultaneously at each phase and direction. Thus, 120 layer image data at 5 phases and 3 directions can be obtained. Using a separation method similar to 3D-SIM, for each direction, based on 5 frames of original images (D 1 -D 5 ) with a transverse phase difference of 2π / 5, 5 three-dimensional spectral components (C m , m = ±2, ±1, 0) are initially extracted in each direction, obtaining Equation (8):

[0086] Among them, Based on the axial phase shift Δ z of the spectrum of the original image; is the three-dimensional spectrum component separated by the m-th order; M 5×5 is the separation matrix. The separation matrix can be expressed as formula (9):

[0087] Among them, is 0, is 2π / 5, is 4π / 5, is 6π / 5, is 8π / 5. The phase difference between adjacent illumination patterns is 2π / 5.

[0088] In formula (8), or The upper and lower parts of the corresponding m-th order spectrum component are mixed, which needs to be further separated. For this purpose, an optical delay line is used to generate an axial phase shift during the imaging process. When the axial phase changes to π / 2 (i.e., Δ z = π / 2), 5 transverse phase shifts are performed in 3 directions respectively to obtain another set of 15 frames of original images (each frame of the original image contains 8 layers of image data). Each frame of the original image is a second image dataset. Each direction corresponds to 5 frames of original images (i.e., 5 second image datasets). An extraction method similar to formula (8) is used to obtain formula (10):

[0089] Formula (10) is similar to formula (8), the difference is that the axial phase shift corresponding to formula (10) is π / 2. According to formula (10), the spectrum component of the -1st order corresponding to the axial phase shift of π / 2 can be calculated and the spectrum component of the +1st order

[0090] In some embodiments, the processor can be based on the 2×2 separation matrix M 2×2 , and further separate the lower half and the upper half of the -1st order spectrum component in the axial direction through formula (11): Among them is the spectrum component of the lower half of the -1st order, is the spectrum component of the upper half of the -1st order; and are the -1st order components overlapping in the axial direction. For details, see formula (8) and formula (10). The spectrum component of the lower half of the -1st order and the spectral components of the upper half can be expressed by Equation (12):

[0091] where m z is the diffraction order; is the modulation depth; is the initial transverse phase; is the initial axial phase; k is the spatial frequency domain coordinate; O(k) is the optical transfer function.

[0092] In some embodiments, for the +1 order spectrum, the lower and upper half spectral components of the +1 order can be extracted by Equation (11) and expressed by Equation (12). In some embodiments, in addition to the spectral components of ±1 order, the spectral components of 0 order and ±2 order can also be expressed by Equation (12). For example, although the spectral components are extracted twice from Equations (8) and (10), corresponding to the 0 order spectral components with axial phase shifts Δ z being 0 and π / 2 (i.e., and ), these two spectral components are theoretically equivalent and can be represented by the same symbol Similarly, the ±2 order spectral components obtained from Equations (8) and (10) (i.e., and ) are also extracted twice and denoted as where m z = 0 indicates that the spectral components of 0 order and ±2 order only have transverse movement and no axial movement. Under the symbol definition, all spectral components, including the 0 order spectrum ±1 order spectra (including the upper and lower halves) and the ±2 order spectra all 7 spectral components can be accurately expressed by Equation (12).

[0093] After separating the 7 spectral components, it is necessary to estimate the transverse vector p xy of the mode vector and the axial vector p z . The calculation method of p xy is the same as that of 3D-SIM, but 3D-MP-SIM has an additional axial vector p z . For the calculation method of the axial vector p z , please refer to the relevant description in Step 430 later.

[0094] Spectral separation is performed not only transversely but also axially, which can improve the resolution of three-dimensional images. The two-step separation matrix method reduces the condition number of the separation matrix and improves the stability of reconstruction.

[0095] Step 430: Determine the multiple frequency-shifted images corresponding to each direction based on the multiple spectral components and the estimation parameters corresponding to each direction.

[0096] Frequency shift means moving the separated spectral components to the correct positions. The purpose of frequency shift is to move the different separated spectral components to their original positions according to the estimation parameters.

[0097] The estimation parameters are parameters related to the frequency shift process. In some embodiments, the estimation parameters include a mode vector, a modulation depth, and an initial phase.

[0098] The mode vector can be the spatial frequency vector of the structured illumination pattern, and the mode vector is used to reflect the distribution and direction of the illumination pattern in space. The mode vector includes a transverse vector (such as p xy ) and an axial vector (such as p z ). The transverse vector includes the frequency vector of the structured illumination pattern in the transverse direction. The axial vector includes the frequency vector of the structured illumination pattern in the axial direction. In some embodiments, the axial vector can be obtained through theoretical calculation. In some embodiments, the axial vector p z can be calculated based on the three-dimensional relationship between the zero-order spectral component and the spectral component of the +1 order upper half . The processor can use the estimated vectors (mp xy , m z p z ) to move the high-order (i.e., m = ±1, ±2) spectral components to their original positions.

[0099] The modulation depth refers to the amplitude of the intensity change of the structured illumination pattern. The modulation depth is used to quantify the contrast of the illumination pattern. A higher modulation depth has a stronger signal contrast, thus enabling clearer resolution of the structure of the sample to be measured.

[0100] The initial phase is the phase offset of the structured illumination pattern at the reference point, and the reference point can be the coordinate origin. The initial phase is used to describe the phase starting point of the illumination pattern. The initial phase includes a transverse initial phase and an axial initial phase. The initial phase is used to correct the phase offset of the illumination pattern to ensure that the illumination patterns with different phases can be correctly aligned. In some embodiments, the processor can determine the initial phase by using complex linear regression in the overlapping region of the zero-order and high-order components.

[0101] In some embodiments, the processor can perform frequency shift on the spectral components of order 0, ±1 (including the spectral components of the upper and lower halves of ±1 order) and ±2 order respectively to obtain multiple frequency-shifted images.

[0102] In some embodiments, referring to the traditional three-dimensional structured light camera imaging, when the sample to be measured is moved up and down, the formula (13) corresponding to 3D-SIM is obtained: Among them, D(r) represents the multi-layer image data collected by the camera; I m represents the axial illumination vector, and J m represents the lateral illumination vector; S represents the sample to be measured; H represents the point spread function of the microscopic system.

[0103] For 3D-MP-SIM, samples at different depths are simultaneously imaged in different regions of the camera and photographed under a single illumination, aligning the sample coordinate z with the coordinate z′ of the illumination pattern. The processor can directly perform an axial frequency shift on the spectrum of the image of the sample to be measured to obtain the frequency-shifted image, as shown in formula (14):

[0104] Step 440, perform Wiener filtering on the multiple frequency-shifted images corresponding to each direction to obtain the super-resolution reconstructed image of the sample to be measured. Wiener filtering is performed based on the frequency domain and is used to reduce the noise of the image and enhance the signal.

[0105] In some embodiments, the processor can perform Wiener filtering on the frequency-shifted images corresponding to multiple spectral components to obtain the super-resolution reconstructed image of the sample to be measured.

[0106] The frequency domain after Fourier transform can be expressed as formula (15): Among them, k is the spatial frequency domain coordinate; is the Fourier transform of D(r); is the Fourier transform of S(r′); represents the structured light illumination vector after Fourier transform (including the axial illumination vector and the lateral illumination vector); O(k) represents the optical transfer function (OTF).

[0107] The first-order and second-order spectral components of the sample to be measured are both frequency-shifted to the position of the zero-order spectral component, obtaining a total of 7 spectral components filtered by WF-OTF. The multiple frequency-shifted images of the sample to be measured are merged and then Wiener filtering is performed to obtain the super-resolution reconstructed image of the sample to be measured.

[0108] Similar to 3D-SIM, in order to estimate the high-order spectral component of the lateral initial phase axial initial phase and modulation depth Complex linear regression is employed within the overlapping region of the 0th-order spectral component and the high-order spectral components. In some embodiments, the processor may remove the initial phase and modulation depth from the high-order spectral components and obtain Equation (16) by performing Wiener filtering on the frequency-shifted image: where p xy and p z are respectively the transverse vector and the axial vector of the mode vector; O is the optical transfer function. For the transverse vector and the axial vector, reference may be made to the relevant description in Step 430.

[0109] As previously mentioned, the 0th-order and ±2nd-order spectral components are extracted twice when the axial phase shift is 0 and π / 2 and are used in the Wiener filter to improve the signal-to-noise ratio. Correspondingly, the Wiener filter also doubles the OTFs corresponding to the 0th-order and ±2nd-order spectral components.

[0110] Figure 7 is a schematic diagram of the image reconstruction principle shown in some embodiments of this specification. As Figure 7 shown, the reconstruction using 3D-MP-SIM involves several basic steps, similar to traditional 3D-SIM, including spectral separation, parameter estimation, spectral shifting, and Wiener filtering. For the reconstruction using 3D-MP-SIM, sub-pixel registration is performed on different depths of the original image before reconstruction. These original images are converted into spectra and separated using a two-step method that initially divides the aliased spectra into 0, ±1, and ±2nd-order components, and then further separates them into the upper and lower halves of the ±1st order (i.e., -1U, +1U, -1L, and +1L). By using the separated components, the mode vector, modulation depth, and initial phase are estimated by a method similar to that of 3D-SIM. Using the above estimated parameters, all seven spectra are shifted back to their original positions and combined after Wiener filtering to obtain super-resolution reconstruction.

[0111] In some embodiments of this specification, by registering the multi-layer image data of each image dataset, the image data at different axial positions can be aligned, ensuring the precise spatial alignment of the images, which is conducive to improving the quality of subsequent image reconstruction, reducing ghosting and blurring caused by image misalignment, and enhancing the clarity and contrast of the images. Spectral separation can separate signals of different frequencies, so that each spectral component can be operated on separately during the reconstruction process. The spectral shifting operation can adjust the separated spectral components to their original positions, so that each spectral component can correctly recover the signal. Wiener filtering can effectively remove the noise in the image, improve the signal-to-noise ratio of the image, and further optimize the contrast and clarity of the image.

[0112] Some embodiments of this specification include, but are not limited to, the following beneficial effects: (1) By moving a specific axial distance (the least common multiple of the structured light period and the layer spacing), multi-layer data at different axial positions can be stitched together to ensure the continuity of the structured light information for reconstructing a super-resolution image of a thicker sample. (2) Arbitrary numbers of stitches can be achieved to realize super-resolution reconstruction of samples of any thickness.

[0113] Some embodiments of this specification provide a computer-readable storage medium storing computer instructions which, when executed by a computer, implement the multi-layer structured light microscopic imaging stitching method described in any one of the embodiments of this specification. Some embodiments of this specification provide a battery swapping cabinet control device including a processor for executing the battery swapping cabinet control method described in any one of the above technical solutions.

[0114] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only an example and does not constitute a limitation to this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are proposed in this specification, so such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this specification.

[0115] Meanwhile, this specification uses specific terms to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" mean a certain feature, structure, or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "an embodiment" or "one embodiment" or "an alternative embodiment" mentioned twice or more at different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0116] In addition, the order of the processing elements and sequences, the use of numbers, letters, or other names described in this specification does not limit the order of the processes and methods of this specification. Although some currently useful inventive embodiments are discussed through various examples in the above disclosure, it should be understood that such details only serve an illustrative purpose. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only through software solutions, such as installing the described system on existing servers or mobile devices.

[0117] Similarly, it should be noted that, in order to simplify the expressions disclosed in this specification and thus assist in the understanding of one or more embodiments of the invention, in the foregoing description of the embodiments of this specification, multiple features are sometimes grouped into one embodiment, drawing, or description thereof. In fact, the features of the embodiments are fewer than all the features of the individual embodiments disclosed above.

[0118] In some embodiments, numbers are used to describe components and the quantity of attributes. It should be understood that such numbers used to describe the embodiments are modified by the modifiers "about", "approximately", or "substantially" in some examples. Unless otherwise specified, "about", "approximately", or "substantially" indicate that the said numbers allow a variation of ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification are approximate values, and such approximate values may vary according to the characteristics required by individual embodiments. In some embodiments, the numerical parameters should consider the specified significant digits and adopt the method of retaining the general number of digits. Although the numerical ranges and parameters used in some embodiments of this specification to confirm the breadth of their scope are approximate values, in specific embodiments, such numerical settings are as precise as possible within the feasible range.

[0119] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, by way of example and not limitation, alternative configurations of the embodiments of this specification may be considered to be consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A multi-layer structured light microscopy imaging splicing method, characterized in that: include: Acquire a plurality of first image data sets corresponding to each direction in a plurality of directions of structured light of the sample to be tested at a first axial position; Acquire a plurality of second image data sets corresponding to each of the plurality of directions of the structured light of the sample to be tested at a second axial position; Image stitching is performed based on the multiple first image data sets and the multiple second image data sets to obtain a three-dimensional stitched image, wherein the axial range of the three-dimensional stitched image includes the combination of the axial range corresponding to the first axial position of the sample to be tested and the axial range corresponding to the second axial position of the sample to be tested.

2. The method according to claim 1, characterized in that Each of the plurality of first image data sets and the plurality of second image data sets includes multiple layers of image data.

3. The method according to claim 2, characterized in that The axial distance meets a preset condition, wherein the axial distance refers to the distance between the first axial position and the second axial position in the axial direction, and the preset condition is related to the axial period.

4. The method according to claim 3, characterized in that The preset condition is further related to the layer spacing between two adjacent layers in the multi-layer image data.

5. The method according to claim 4, characterized in that The preset condition is that the axial distance is a common multiple of the axial period and the interlayer spacing.

6. The method according to claim 2, characterized in that The performing image stitching based on the plurality of first image data sets and the plurality of second image data sets to obtain a three-dimensional stitched image comprises: Registering and splicing one of the plurality of first image data sets corresponding to each direction and one of the plurality of second image data sets corresponding to the first image data sets to obtain a set of registered and spliced ​​three-dimensional image data; Performing spectrum separation on the multiple groups of three-dimensional image data corresponding to each direction to obtain multiple spectrum components; Determine a plurality of frequency-shifted images corresponding to each direction based on the plurality of spectral components corresponding to each direction and the estimated parameters; Perform Wiener filtering on the multiple frequency-shifted images corresponding to each direction to obtain a super-resolution image of the sample to be tested.

7. The method according to claim 6, characterized in that The spectrum separation includes spectrum separation in a first direction and a second direction.

8. The method according to claim 6, characterized in that The estimated parameters include pattern vector, modulation depth and initial phase.

9. A multi-layer structured light microscopy imaging stitching system, comprising a first acquisition module, a second acquisition module and a stitching module; The first acquisition module is configured to acquire a plurality of first image data sets corresponding to each direction in a plurality of directions of the structured light of the sample to be tested at a first axial position; The second acquisition module is configured to acquire a plurality of second image data sets corresponding to each of the plurality of directions of the structured light of the sample to be tested at a second axial position; The stitching module is configured to perform image stitching based on the multiple first image data sets and the multiple second image data sets to obtain a three-dimensional stitched image, wherein the axial range of the three-dimensional stitched image includes the combination of the axial range corresponding to the first axial position of the sample to be tested and the axial range corresponding to the second axial position of the sample to be tested.

10. A multi-layer structured light microscopy imaging splicing device, characterized in that: The device comprises a processor, and the processor is used to execute the multi-layer structured light microscopy imaging stitching method as described in any one of claims 1 to 8.

Images