A multi-focal-plane structured light three-dimensional point cloud reconstruction method and device

Abstract

The application discloses a kind of multi-focal plane oblique light sheet structured light three-dimensional point cloud reconstruction method, device, belong to biomedical imaging technical field, including: illumination light is irradiated sample with oblique light sheet, corresponding image frame sequence is collected by multiple cameras shooting acquisition;Determine the imaging light spot of same fluorescent molecule in image frame sequence, adopt two-dimensional Gaussian function fitting imaging light spot, obtain centroid coordinate and standard deviation parameter;Obtain the calibration curve of pre-set, obtain the three-dimensional coordinates of fluorescent molecule according to standard deviation parameter in combination with calibration curve;According to the angle of the focal plane of camera and horizontal direction, transform three-dimensional coordinates into orthosteric three-dimensional coordinates;According to the moving step of displacement table, transform orthosteric three-dimensional coordinates into global three-dimensional coordinates, reconstruct the three-dimensional point cloud of multiple cameras;Obtain multi-camera registration file, according to the three-dimensional point cloud of multiple cameras is registered according to multi-camera registration file, obtains three-dimensional point cloud depth image.The application realizes the fast scanning imaging and reconstruction of large volume sample.

Description

Technical Field

[0001] This invention relates to the field of biomedical imaging technology, and in particular to a method and apparatus for reconstructing three-dimensional point clouds using multi-focal plane oblique light structured light. Background Technology

[0002] Currently, various ultra-high resolution optical microscopy imaging methods that break the optical diffraction limit have been adopted. These ultra-high resolution imaging methods can be roughly divided into two categories according to their characteristics: one is imaging technology that effectively reduces the point spread function, such as stimulated emission loss fluorescence microscopy, ground state loss super-resolution imaging, and saturated structure illumination microscopy; the other is imaging technology based on single-molecule localization, such as photoactivated localization microscopy and random optical reconstruction microscopy.

[0003] In single-molecule localization-based 3D imaging techniques, cylindrical mirrors or masks are typically used to modulate the fluorescence signal, causing the captured pattern to change with depth. This pattern information allows for the recovery of the fluorescent molecule's depth, achieving 3D localization. In cylindrical mirror 3D localization imaging, the astigmatism introduced by the cylindrical mirror causes the fluorescent molecule to not be focused at the same location in the X and Y directions. When the fluorescent molecule is positioned precisely between the two focal points, the resulting image spot is a symmetrical circle. At other locations, the intensity distribution of the image spot is an ellipse with varying ellipticity. By calibrating the change in ellipticity of the image spot with focusing depth, the position of the fluorescent molecule from the focusing plane can be determined based on the ellipticity of the image spot. Whether the ellipticity exceeds 1 indicates whether the fluorescent molecule is above or below the focusing plane.

[0004] However, the aforementioned astigmatism-based 3D localization method, when used for scanning large-volume samples, achieves a reconstruction thickness of less than 1 μm in a single image. On the other hand, traditional light-sheet microscopy for cellular or subcellular imaging often requires multi-angle imaging followed by deconvolution post-processing, resulting in massive data volumes and is time-consuming and labor-intensive. Summary of the Invention

[0005] To address the problem of low single-reconstruction thickness during large-volume sample scanning, this invention provides a method and apparatus for three-dimensional point cloud reconstruction using multi-focal-plane oblique light structured light. The technical solution is as follows:

[0006] Multi-focal-plane oblique light structured light 3D point cloud reconstruction methods include:

[0007] Illumination light is shone onto the sample through an oblique beam. The sample moves with a displacement stage in a certain step size. Multiple cameras perform continuous imaging to obtain a sequence of image frames captured by the multiple cameras. The focal planes of the multiple cameras are parallel, and the movement direction of the displacement stage is parallel to the arrangement direction of the multiple cameras.

[0008] In multiple image frame sequences, the imaging spot of the same fluorescent molecule is determined, and the imaging spot is fitted with a two-dimensional Gaussian function to obtain the centroid coordinates and standard deviation parameters of the fluorescent molecule.

[0009] A preset calibration curve is obtained, and the depth information of the fluorescent molecule is calculated based on the standard deviation parameter and the calibration curve to obtain the three-dimensional coordinates of the fluorescent molecule in the camera coordinate system.

[0010] Based on the angle between the focal planes of the multiple cameras and the horizontal direction, the three-dimensional coordinates are transformed into upright three-dimensional coordinates;

[0011] The upright three-dimensional coordinates are transformed into global three-dimensional coordinates based on the movement step size of the displacement stage, and multiple three-dimensional point clouds of the multiple cameras are reconstructed.

[0012] Obtain a multi-camera registration file, and register multiple 3D point clouds from the multiple cameras according to the multi-camera registration file to obtain a 3D point cloud depth image of the sample.

[0013] Furthermore, the preset calibration curve can be determined by the following method:

[0014] Continuous imaging is performed using the multiple cameras to obtain a sequence of image frames of the sample on the multiple cameras as calibration data;

[0015] The imaging spot of the same reference fluorescent molecule in the image frame sequence is extracted, and the standard deviation parameter of the imaging spot of the reference fluorescent molecule is obtained by fitting a two-dimensional Gaussian function.

[0016] The preset calibration curve is determined by the relationship between the standard deviation parameter of the imaging spot and the defocus depth.

[0017] Furthermore, the preset calibration curve is determined by the relationship between the standard deviation parameter of the imaging spot and the defocus depth, including:

[0018] The variance parameters of the imaging spot in the x and y directions are determined by the standard deviation parameter of the imaging spot.

[0019] The curves relating the difference in variance parameters of the imaging spot in the x and y directions to the defocus depth are fitted together and used as the preset calibration curves.

[0020] Further, the step of obtaining a preset calibration curve and calculating the depth information of the fluorescent molecule based on the standard deviation parameter and the calibration curve includes:

[0021] Obtain multiple pre-defined calibration curves corresponding to different reference fluorescent molecules and multiple ellipticity curves corresponding to different reference fluorescent molecules;

[0022] Substituting the standard deviation parameter into the preset multiple calibration curves yields multiple depth information;

[0023] The depth information is substituted into the ellipticity curve to calculate the corresponding ellipticity, and the depth corresponding to the ellipticity that is closest to the fitted ellipticity is determined as the depth of the fluorescent molecule.

[0024] Furthermore, the multi-camera registration file can be determined by the following method:

[0025] The image frame with the smallest absolute value of the difference in standard deviation parameters is determined as the focusing frame of the imaging spot;

[0026] The relative positions and orientations of the multiple cameras are determined based on the centroid coordinates of the imaging spot in the focusing frame and the movement step size of the displacement stage, and saved as a multi-camera registration file.

[0027] Furthermore, the relative position and orientation of the plurality of cameras include axial spacing and lateral spacing. The axial spacing of the plurality of cameras is the product of the frame number difference between the focused frames and the movement step size. The lateral spacing of the plurality of cameras is the difference in centroid coordinates of the imaging spot in the focused frame.

[0028] Furthermore, the three-dimensional coordinates are transformed into upright three-dimensional coordinates using the following formula:

[0029] x′=x

[0030] y′=ycos(a)-zsin(a)

[0031] z′=ysin(a)+zcos(a)

[0032] Where 'a' is the angle between the camera's focal plane and the horizontal direction.

[0033] Furthermore, the global three-dimensional coordinates are the sum of the upright three-dimensional coordinates and the three-dimensional coordinates of the displacement stage at the same moment, and the three-dimensional coordinates of the displacement stage are determined by the initial position coordinates and the movement step size.

[0034] Furthermore, the multi-camera registration file includes relative position parameters of the multiple cameras in the x-direction, relative position parameters in the y-direction, and relative position parameters in the z-direction. The relative position parameters in the x-direction and y-direction are the difference in centroid coordinates of the imaging spot in the focused frame. The relative position parameter in the z-direction is the product of the frame number difference between the focused frames and the movement step size. The 3D point cloud registration of the multiple cameras includes adding the coordinate values ​​in the x-direction of the global 3D coordinate system to the relative position parameters in the x-direction, adding the coordinate values ​​in the y-direction of the global 3D coordinate system to the relative position parameters in the y-direction, and adding the coordinate values ​​in the z-direction of the global 3D coordinate system to the relative position parameters in the z-direction.

[0035] Furthermore, the multi-focal-plane oblique light structured light three-dimensional point cloud reconstruction device includes:

[0036] An image acquisition module is used to illuminate a sample with a slanted light plate. The sample moves with a displacement stage at a certain step size and is continuously imaged by multiple cameras to acquire a sequence of image frames captured by the multiple cameras. The focal planes of the multiple cameras are parallel, and the moving direction of the displacement stage is parallel to the arrangement direction of the multiple cameras.

[0037] The spot fitting module is used to determine the imaging spot of the same fluorescent molecule in multiple image frame sequences, and to fit the imaging spot with a two-dimensional Gaussian function to obtain the centroid coordinates and standard deviation parameters of the fluorescent molecule.

[0038] The depth calculation module is used to obtain a preset calibration curve, and calculate the depth information of the fluorescent molecule based on the standard deviation parameter and the calibration curve to obtain the three-dimensional coordinates of the fluorescent molecule in the camera coordinate system.

[0039] The coordinate orthogonal module is used to transform the three-dimensional coordinates into orthogonal three-dimensional coordinates based on the angle between the focal plane of the plurality of cameras and the horizontal direction;

[0040] The coordinate transformation module is used to transform the upright three-dimensional coordinates into global three-dimensional coordinates according to the movement step size of the displacement stage, and reconstruct multiple three-dimensional point clouds of the multiple cameras;

[0041] The camera registration module is used to acquire a multi-camera registration file, register multiple 3D point clouds of the multiple cameras according to the multi-camera registration file, and obtain a 3D point cloud depth image of the sample.

[0042] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0043] This invention provides a method and apparatus for reconstructing three-dimensional point clouds using multi-focal-plane oblique light structured light. By illuminating the sample with an oblique light sheet, the optical path length of light within the sample is effectively reduced, thereby minimizing errors caused by scattering and achieving higher precision three-dimensional point cloud resolution. Simultaneously, multiple cameras are used to capture images of tissue samples within multiple focal planes during a single imaging session. Depth information of fluorescent molecules is calculated using a pre-set calibration curve to determine their three-dimensional coordinates. The reconstructed three-dimensional point clouds of fluorescent molecules from each camera are then transformed to a global coordinate system using coordinate transformation. Finally, the reconstructed three-dimensional point clouds from each camera are registered using a pre-set multi-camera registration file, resulting in a three-dimensional point cloud depth image of a large-volume sample. This expands the volume of a single imaging session and enables rapid scanning imaging and reconstruction of large-volume samples. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating an embodiment of the multi-focal oblique light structured light three-dimensional point cloud reconstruction method of the present invention;

[0046] Figure 2 This is a schematic diagram of sample scanning according to the present invention;

[0047] Figure 3 This is a schematic diagram of the coordinate system transformation of the present invention;

[0048] Figure 4 This is a flowchart of another embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of the fluorescent molecule and its calibration curve of the present invention;

[0050] Figure 6 This is a three-dimensional point cloud depth image of the present invention;

[0051] Figure 7 This is a flowchart of another embodiment of the present invention;

[0052] Figure 8 This is a schematic diagram of the dual-camera simultaneous shooting of the present invention;

[0053] Figure 9 This is a flowchart of another embodiment of the present invention;

[0054] Figure 10 This is a schematic diagram of multiple calibration curves and ellipticity curves of the present invention;

[0055] Figure 11 This is a flowchart of another embodiment of the present invention;

[0056] Figure 12 This is a functional module schematic diagram of an embodiment of the multi-focal plane oblique light structured light three-dimensional point cloud reconstruction device of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0058] like Figure 1 As shown, this embodiment of the invention provides a method for reconstructing three-dimensional point clouds using structured light from a multi-focal plane oblique light sheet, including:

[0059] S110, illumination light is projected onto the sample using an oblique beam. The sample moves with a displacement stage in steps, and continuous imaging is performed by multiple cameras to acquire a sequence of image frames captured by the multiple cameras. The focal planes of the multiple cameras are parallel, and the moving direction of the displacement stage is parallel to the arrangement direction of the multiple cameras.

[0060] Reference Figure 2 The left figure is a schematic diagram of sample scanning according to the present invention. In this embodiment, multiple cameras are used to simultaneously capture images. The relative positions of the multiple cameras are fixed, and the focal planes of the multiple cameras remain parallel. As shown in the figure, the focal planes C1, C2, C3, and C4 are parallel to each other. Illumination light is irradiated onto the sample using an oblique light plate. The sample is placed on a displacement stage, and the displacement stage moves in a certain step size. The direction of movement of the displacement stage is parallel to the arrangement direction of the multiple cameras. Specifically, the displacement stage moves uniformly along the Z direction in a certain step size, referring to... Figure 2 The right figure shows the direction and trajectory of the sample's movement. The fluorescence signal emitted by the sample is captured and imaged by a camera. Each camera captures one image frame sequence, and multiple cameras simultaneously capture multiple image frame sequences.

[0061] S120, in multiple image frame sequences, the imaging spot of the same fluorescent molecule is determined, and the imaging spot is fitted with a two-dimensional Gaussian function to obtain the centroid coordinates and standard deviation parameters of the fluorescent molecule.

[0062] Due to the influence of depth of field, a single fluorescent molecule can be captured within multiple consecutive movement steps. In the image frame sequence at multiple moments, the imaging spot of the same fluorescent molecule in space can be found. By fitting the imaging spot with a two-dimensional Gaussian function, the centroid coordinates and standard deviation parameters of the fluorescent molecule can be obtained.

[0063] S130, acquire the preset calibration curve, and calculate the depth information of the fluorescent molecule based on the standard deviation parameter and the calibration curve to obtain the three-dimensional coordinates of the fluorescent molecule in the camera coordinate system.

[0064] Specifically, the preset calibration curve can be predetermined based on previously referenced fluorescent molecule data, and is used to characterize the relationship between the difference in the standard deviation parameter of the fluorescent molecule and the defocus depth.

[0065] The depth of the fluorescent molecule can be determined by combining the standard deviation parameter with the preset calibration curve. Alternatively, the actual standard deviation parameter of the fluorescent molecule can be directly substituted into the calibration curve, and the depth of the fluorescent molecule can be calculated based on the relationship between the difference in the standard deviation parameters of the fluorescent molecule and the defocus depth.

[0066] The depth of the fluorescent molecule can be determined by combining the standard deviation parameter with the preset calibration curve. Alternatively, the actual standard deviation parameter of the fluorescent molecule can be substituted into multiple calibration curves obtained from different reference fluorescent molecules to obtain multiple depth information. The depth of the fluorescent molecule can be obtained by averaging the multiple depth information to reduce the error caused by the inaccuracy of a single calibration curve.

[0067] The depth of the fluorescent molecule is determined by combining the standard deviation parameter with the preset calibration curve. Alternatively, the actual standard deviation parameter of the fluorescent molecule can be substituted into multiple calibration curves obtained based on different reference fluorescent molecules to obtain multiple depth information. Substituting these depths into the ellipticity curve, the ellipticity of the fluorescent molecule at different depths can be obtained. Finally, these ellipticities are compared with the ellipticity obtained by fitting the imaging spot, and the depth corresponding to the ellipticity closest to the fitted ellipticity is taken as the depth of the fluorescent molecule, further improving the positioning accuracy of the fluorescent molecule depth.

[0068] S140, based on the angle between the focal plane of the plurality of cameras and the horizontal direction, transform the three-dimensional coordinates into upright three-dimensional coordinates.

[0069] Reference Figure 3 This is a schematic diagram of the coordinate system transformation of the present invention. Due to the use of oblique light irradiation, the three-dimensional coordinate system of the fluorescent molecule located in step S130 is... Figure 3In the x'y'z' coordinate system, based on the angle between the camera's focal plane and the horizontal direction, the fluorescent molecule is transformed from the x'y'z' coordinate system to the x”y”z" coordinate system to obtain the upright three-dimensional coordinates of the fluorescent molecule.

[0070] S150, the upright three-dimensional coordinates are transformed into global three-dimensional coordinates according to the movement step size of the displacement stage, and multiple three-dimensional point clouds of the multiple cameras are reconstructed.

[0071] Furthermore, referring to Figure 3 Based on the step size of the displacement stage, the fluorescent molecules are transformed from the x”y”z” coordinate system to the xyz coordinate system, the upright three-dimensional coordinates are transformed into global three-dimensional coordinates, and the three-dimensional point clouds reconstructed by each camera are transformed into the global coordinate system.

[0072] S160, Obtain a multi-camera registration file, and register multiple 3D point clouds of the multiple cameras according to the multi-camera registration file to obtain a 3D point cloud depth image of the sample.

[0073] Furthermore, the multi-camera registration file can be pre-determined based on the relative positions and poses of multiple cameras. By registering the 3D point clouds reconstructed by each camera, a 3D point cloud depth image of the sample can be obtained.

[0074] This invention illuminates the sample with a slanted light plate, effectively reducing the optical path length within the sample and thus minimizing errors caused by scattering, achieving higher precision in 3D point cloud analysis. Simultaneously, multiple cameras capture images within a single imaging session, allowing for the acquisition of tissue samples across multiple focal planes. Depth information of fluorescent molecules is calculated using a pre-defined calibration curve, determining their 3D coordinates. Coordinate transformation is then applied to convert the reconstructed 3D point clouds from each camera to a global coordinate system. Finally, a pre-defined multi-camera registration file is used to register the reconstructed 3D point clouds, resulting in 3D point cloud depth images of large-volume samples. This expands the volume of a single imaging session, enabling rapid scanning imaging and reconstruction of large-volume samples.

[0075] To further improve the accuracy of fluorescent molecule depth localization, this embodiment simultaneously employs preset calibration curves and ellipticity curves for fluorescent molecule depth reconstruction. (Refer to...) Figure 4 , Figure 4 This is a flowchart illustrating another embodiment of the multi-focal-plane oblique light structured light three-dimensional point cloud reconstruction method of the present invention, including:

[0076] S210, obtain the preset calibration curve.

[0077] Specifically, the preset calibration curve is used to characterize the relationship between the difference in standard deviation parameters of the fluorescent molecule in the x and y directions and the defocus depth, or to characterize the relationship between the difference in variance parameters of the fluorescent molecule in the x and y directions and the defocus depth. The preset calibration curve can be predetermined based on data from previously referenced fluorescent molecules and does not need to be determined repeatedly. For details on the method for determining the preset calibration curve, please refer to the subsequent step S300.

[0078] S220, obtain the preset ellipticity curve.

[0079] Furthermore, the preset ellipticity curve is used to characterize the relationship between the ratio of the standard deviation parameters of the fluorescent molecules and the defocus depth of the fluorescent molecules, or to characterize the relationship between the ratio of the variance parameters of the fluorescent molecules in the x and y directions and the defocus depth. The preset ellipticity curve can be predetermined based on data from previously referenced fluorescent molecules and does not need to be determined repeatedly.

[0080] S230: Continuous imaging is performed using multiple cameras to obtain a sequence of image frames of the sample on the multiple cameras.

[0081] By continuously imaging with multiple cameras, a sequence of image frames of a sample imaged on multiple cameras is obtained.

[0082] To handle large amounts of image frame data, this embodiment developed a parallel processing mechanism. The captured image frame sequence is divided into smaller files of 100 frames each and saved accordingly. During subsequent fluorescent molecule localization, the number of Python processing processes is set according to the number of computer processes (N). If the number of files (a) is less than the number of processes (N), the number of processing processes is set to a, and each process processes one file. If the number of files (a) is greater than the number of processes (N), the number of processing processes is set to N, and the number of files localized by each process is a / N rounded down. Through this parallel processing mechanism, a localization speed of 20 frames per second can be achieved.

[0083] S240, in multiple image frame sequences, the imaging spot of the same fluorescent molecule is determined, and the imaging spot is fitted with a two-dimensional Gaussian function to obtain the centroid coordinates and standard deviation parameters of the fluorescent molecule.

[0084] Furthermore, the imaging spot of the same fluorescent molecule is extracted from the image frame sequence, and the imaging spot is fitted with a two-dimensional Gaussian function to obtain the centroid coordinates (x, y) and standard deviation parameter (σ) of the fluorescent molecule. x , σ y ).

[0085] S250, based on the standard deviation parameter and the preset calibration curve and ellipticity curve, calculate the depth information of the fluorescent molecule to obtain the three-dimensional coordinates of the fluorescent molecule in the camera coordinate system.

[0086] By substituting the standard deviation parameter into multiple calibration curves, multiple depth information can be obtained. Substituting these depths into the ellipticity curve, the ellipticity of the fluorescent molecule at different depths can be obtained. Finally, these ellipticities are compared with the ellipticity obtained by fitting the imaging spot. The depth corresponding to the ellipticity closest to the fitted ellipticity is taken as the depth of the fluorescent molecule. Thus, the three-dimensional coordinates of the fluorescent molecule in the camera coordinate system can be determined.

[0087] S260, based on the angle between the focal plane of the plurality of cameras and the horizontal direction, transform the three-dimensional coordinates into upright three-dimensional coordinates.

[0088] Transforming the three-dimensional coordinates of fluorescent molecules in the camera coordinate system into upright three-dimensional coordinates is achieved through the following formula:

[0089] x′=x

[0090] y′=ycos(a)-zsin(a)

[0091] z′=ysin(a)+zcos(a)

[0092] Where 'a' is the angle between the camera's focal plane and the horizontal direction. The angle 'a' is determined as follows:

[0093] The direction of movement of the displacement stage is parallel to the arrangement direction of multiple cameras. If the distance of movement of the displacement stage at two adjacent moments is A, then the distance of movement of the centroid of the imaging spot in two consecutive image frames of the same camera is A*cos a. Since the pixel difference of the centroid of the imaging spot in the two image frames and the size of each pixel are known, the angle α between the focal plane of the camera and the horizontal direction can be solved.

[0094] Assuming the angle between the camera's focal plane and the horizontal direction is found to be 45 degrees, then the following transformation formula applies:

[0095] x′=x

[0096]

[0097]

[0098] Reference Figure 3 By transforming the three-dimensional coordinates of the fluorescent molecule using the above formula, the fluorescent molecule is transformed from the x'y'z' coordinate system to the x”y”z” coordinate system, thus obtaining an upright three-dimensional point cloud.

[0099] S270, the upright three-dimensional coordinates are transformed into global three-dimensional coordinates according to the movement step size of the displacement stage, and multiple three-dimensional point clouds of the multiple cameras are reconstructed.

[0100] Furthermore, transforming the fluorescent molecule from the x'y'z' coordinate system to the xyz coordinate system, and transforming the upright 3D coordinates into global 3D coordinates, is obtained by adding the upright 3D coordinates at the same moment to the 3D coordinates of the displacement stage. For example, the upright 3D coordinates of the fluorescent molecule in the x'y'z' coordinate system in the first image frame, plus the 3D coordinates of the displacement stage at that moment, constitute the global 3D coordinates of the fluorescent molecule in the xyz coordinate system in the first image frame. The 3D coordinates of the displacement stage are determined by the initial position coordinates and the movement step size of the displacement stage, as shown in the table below, which displays the 3D coordinate information of the displacement stage at different moments.

[0101]

[0102]

[0103] By transforming the three-dimensional coordinates of fluorescent molecules to the global three-dimensional coordinate system, the three-dimensional point cloud reconstruction results of multiple cameras are obtained.

[0104] S280, obtain a multi-camera registration file, register multiple 3D point clouds of the multiple cameras according to the multi-camera registration file, and obtain a 3D point cloud depth image of the sample.

[0105] Furthermore, the 3D point clouds of each camera are registered using a multi-camera registration file to obtain the final reconstruction result.

[0106] Specifically, the multi-camera registration file includes the relative position parameters of multiple cameras in the x, y, and z directions. The 3D point cloud registration process of multiple cameras involves adding the coordinate values ​​in the x direction of the global 3D coordinate system to the relative position parameters of multiple cameras in the x direction, adding the coordinate values ​​in the y direction of the global 3D coordinate system to the relative position parameters of multiple cameras in the y direction, and adding the coordinate values ​​in the z direction of the global 3D coordinate system to the relative position parameters of multiple cameras in the z direction.

[0107] Reference Figure 5 (c) represents the depth that the two cameras can image after calibrating the axial distance. After registration, the dual cameras can achieve an imaging depth twice that of the single camera, enabling rapid scanning imaging of large-volume samples.

[0108] Reference Figure 6 The left image is an image of the fluorescently labeled hippocampal region taken by a confocal microscope, and the right image is a three-dimensional point cloud depth image reconstructed by the present invention. In actual three-dimensional localization and reconstruction of biological samples, the three-dimensional point cloud reconstruction of the present invention can achieve higher accuracy.

[0109] In some embodiments, the multi-focal plane oblique light structured light three-dimensional point cloud reconstruction method further includes:

[0110] S300: Determine the preset calibration curve. (Refer to...) Figure 7 The preset calibration curve can be determined by the following methods:

[0111] Step S310: Continuous imaging is performed using the multiple cameras to obtain a sequence of image frames of the sample on the multiple cameras as calibration data.

[0112] To obtain the preset calibration curve, continuous imaging is first performed using multiple cameras, acquiring a sequence of image frames of the sample captured by these cameras as calibration data. Due to the influence of depth of field, a single fluorescent molecule can be captured within multiple consecutive movement steps. In the image frame sequence at multiple time points, the imaging spot of the same reference fluorescent molecule in space can be found, serving as the reference... Figure 8 The images are taken at the same time after dual-camera calibration, and the box shows the imaging spot of the same fluorescent molecule.

[0113] Step S320: Extract the imaging spot of the same reference fluorescent molecule in the image frame sequence, and obtain the standard deviation parameter of the imaging spot of the reference fluorescent molecule by fitting a two-dimensional Gaussian function.

[0114] The imaging spot of the same reference fluorescent molecule is extracted from the image frame sequence, and the imaging spot is fitted with a two-dimensional Gaussian function to obtain the standard deviation parameter (σ) of the imaging spot. x , σ y ).

[0115] Step S330: Determine the preset calibration curve by the relationship between the standard deviation parameter of the imaging spot and the defocus depth.

[0116] Understandably, the preset calibration curve can be a single calibration curve or multiple calibration curves. Multiple preset calibration curves can be determined by repeating steps S310 to S330 with multiple different fluorescent molecules.

[0117] Because the physical size of actual fluorescent molecules varies, the characteristics of the imaging spot differ, which can lead to positioning errors. The preset calibration curve is a single calibration curve, resulting in poor positioning accuracy. By calibrating fluorescent molecules of different sizes separately and creating multiple calibration curves, the accuracy of fluorescent molecule reconstruction can be improved.

[0118] Specifically, by fitting the relationship curve between the difference of the standard deviation parameters of the imaging spot in the x and y directions and the defocus depth, i.e., fitting σ x -σ y The calibration curve of the fluorescent molecule is obtained by ∝Z, which is used for the depth recovery of the fluorescent molecule in the subsequent three-dimensional point cloud reconstruction process.

[0119] In some embodiments, since the difference between the variance parameters of the fluorescent molecules in the x and y directions is greater than the difference between the standard deviation parameters in the x and y directions, using the relationship curve between the difference in the variance parameters of the fluorescent molecules in the x and y directions and the defocus depth to determine the calibration curve can further improve the accuracy of fluorescent molecule depth reconstruction. Specifically, referring to... Figure 9 Step S330, determining the preset calibration curve through the relationship between the standard deviation parameter of the imaging spot and the defocus depth, includes:

[0120] S331, the variance parameters of the imaging spot in the x and y directions are determined by the standard deviation parameter of the imaging spot.

[0121] S332, Fit the curve of the relationship between the difference of the variance parameters of the imaging spot in the x and y directions and the defocus depth, as the preset calibration curve.

[0122] The variance parameters of the imaging spot in the x and y directions are determined by the standard deviation parameter of the imaging spot. The relationship between the difference in variance parameters of the imaging spot in the x and y directions and the defocus depth is fitted using a third-order polynomial and used as a preset calibration curve.

[0123] Specifically, refer to Figure 5 (a) is a continuously changing image of the imaging spot of the same fluorescent molecule captured from a single camera image. The imaging spot is fitted using a two-dimensional Gaussian function to obtain its variance parameter. The relationship between the difference in variance parameters of the imaging spot in the x and y directions and the defocus depth is then fitted, as shown in the figure. Figure 5 As shown in (b), the fitting The calibration curves of the fluorescent molecules were obtained.

[0124] Reference Figure 10 The left figure shows the relationship between the difference in variance parameters of fluorescent molecules and the defocus depth, while the right figure shows the relationship between defocus depth and ellipticity. Since the imaging system of the device is fixed, the calibration curves can be reused once determined. In subsequent use, the preset calibration curves can be directly obtained. By substituting the standard deviation parameters of the fluorescent molecules into the preset calibration curves, the depth information of the fluorescent molecules can be calculated, thereby determining the three-dimensional coordinates of the fluorescent molecules in the camera coordinate system.

[0125] In some embodiments, the multi-focal plane oblique light structured light three-dimensional point cloud reconstruction method further includes:

[0126] S400: Determine the multi-camera registration file, refer to... Figure 11 Multi-camera registration files can be determined using the following methods:

[0127] S410, continuously imaging with multiple cameras to obtain a sequence of image frames of the sample on the multiple cameras.

[0128] By continuously imaging with multiple cameras, a sequence of image frames of a sample imaged on multiple cameras is obtained.

[0129] S420, in multiple image frame sequences, the imaging spot of the same fluorescent molecule is determined, and the imaging spot is fitted with a two-dimensional Gaussian function to obtain the centroid coordinates and standard deviation parameters of the fluorescent molecule.

[0130] Furthermore, the imaging spot of the same fluorescent molecule is extracted from the image frame sequence, and the imaging spot is fitted with a two-dimensional Gaussian function to obtain the centroid coordinates (x, y) and standard deviation parameter (σ) of the fluorescent molecule. x , σ y ).

[0131] S430, the image frame with the smallest absolute value of the difference in standard deviation parameters is determined as the focusing frame of the imaging spot.

[0132] The standard deviation parameter (σ) of the imaging spot is obtained by fitting a two-dimensional Gaussian function to the imaging spot. x , σ y Furthermore, the standard deviation parameter σ is determined. x -σ y The image frame closest to 0 is the focus frame. The depth Z0 of the focus frame is taken as the 0 depth, and the imaging depth of the fluorescent molecule on other image frames is recalibrated.

[0133] The following example illustrates this: Assuming the displacement stage has a step size of 100 nm, in an image frame captured by a single camera, the fluorescent molecule appears as an elliptical spot in the 7th frame. The imaging spot changes from an ellipse to a perfect circle in the 20th frame, and then gradually diffuses and disappears in the 33rd frame. In the 20th frame, the standard deviation parameter of this imaging spot is closest to 0. The 20th frame is the focusing frame of this imaging spot, and the depth Z0 is recorded as 0. The depth is -100 nm in the 19th frame, 100 nm in the 21st frame, 200 nm in the 22nd frame, and so on. The depth of each image frame acquired by the camera can be recalibrated.

[0134] S440, determine the relative position and attitude of the multiple cameras based on the centroid coordinates of the imaging spot in the focusing frame and the movement step size of the displacement stage, and save it as a multi-camera registration file.

[0135] After determining the focus frame in the image frame sequence of each camera, the positional relationship between the cameras can be calibrated based on the centroid coordinates of the imaging spot in the focus frame and the movement step size of the displacement stage.

[0136] Specifically, the axial spacing between each camera is the product of the frame number difference between the focused frames and the movement step size, and the lateral spacing between each camera is the difference in centroid coordinates of the imaging spot in the focused frame.

[0137] By determining the focal frames of the imaging spot in each camera, the relative positions and attitudes of multiple cameras in the x and y directions are calibrated, achieving multi-camera position registration. (Refer to...) Figure 5 (c) represents the depth that the two cameras can image after calibrating the axial distance. After registration, the dual cameras can achieve an imaging depth twice that of the single camera, enabling rapid scanning imaging of large-volume samples.

[0138] Reference Figure 12 , Figure 12 This is a schematic diagram of the functional modules of an embodiment of the multi-focal plane oblique light structured light three-dimensional point cloud reconstruction device of the present invention.

[0139] The multi-focal-plane oblique light structured light 3D point cloud reconstruction device includes:

[0140] Image acquisition module 10 is used to illuminate the sample with a slanted light plate. The sample moves with a displacement stage at a certain step size and is continuously imaged by multiple cameras to acquire a sequence of image frames captured by the multiple cameras. The focal planes of the multiple cameras are parallel, and the moving direction of the displacement stage is parallel to the arrangement direction of the multiple cameras.

[0141] The spot fitting module 20 is used to determine the imaging spot of the same fluorescent molecule in multiple image frame sequences, and to fit the imaging spot using a two-dimensional Gaussian function to obtain the centroid coordinates and standard deviation parameters of the fluorescent molecule.

[0142] The depth calculation module 30 is used to acquire a preset calibration curve, and calculate the depth information of the fluorescent molecule based on the standard deviation parameter and the calibration curve, so as to obtain the three-dimensional coordinates of the fluorescent molecule in the camera coordinate system.

[0143] The coordinate orthogonal module 40 is used to transform the three-dimensional coordinates into orthogonal three-dimensional coordinates based on the angle between the focal plane of the plurality of cameras and the horizontal direction.

[0144] The coordinate transformation module 50 is used to transform the upright three-dimensional coordinates into global three-dimensional coordinates according to the movement step size of the displacement stage, and reconstruct multiple three-dimensional point clouds of the multiple cameras.

[0145] The camera registration module 60 is used to acquire a multi-camera registration file, register multiple 3D point clouds of the multiple cameras according to the multi-camera registration file, and obtain a 3D point cloud depth image of the sample.

[0146] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for reconstructing three-dimensional point clouds using structured light from a multi-focal plane oblique light sheet, characterized in that, The multi-focal-plane oblique light structured light three-dimensional point cloud reconstruction method includes: Illumination light is shone onto the sample through an oblique beam. The sample moves with a displacement stage in a certain step size. Multiple cameras perform continuous imaging to obtain a sequence of image frames captured by the multiple cameras. The focal planes of the multiple cameras are parallel, and the movement direction of the displacement stage is parallel to the arrangement direction of the multiple cameras. In multiple image frame sequences, the imaging spot of the same fluorescent molecule is determined, and the imaging spot is fitted with a two-dimensional Gaussian function to obtain the centroid coordinates and standard deviation parameters of the fluorescent molecule. A preset calibration curve is obtained, and the depth information of the fluorescent molecule is calculated based on the standard deviation parameter and the calibration curve to obtain the three-dimensional coordinates of the fluorescent molecule in the camera coordinate system. Based on the angle between the focal planes of the multiple cameras and the horizontal direction, the three-dimensional coordinates are transformed into upright three-dimensional coordinates; The upright three-dimensional coordinates are transformed into global three-dimensional coordinates based on the movement step size of the displacement stage, and multiple three-dimensional point clouds of the multiple cameras are reconstructed. Obtain a multi-camera registration file, and register multiple 3D point clouds from the multiple cameras according to the multi-camera registration file to obtain a 3D point cloud depth image of the sample.

2. The method for reconstructing three-dimensional point clouds using structured light from a multi-focal plane oblique light plate according to claim 1, characterized in that, The preset calibration curve is determined by the following method: Continuous imaging is performed using the multiple cameras to obtain a sequence of image frames of the sample on the multiple cameras as calibration data; The imaging spot of the same reference fluorescent molecule in the image frame sequence is extracted, and the standard deviation parameter of the imaging spot of the reference fluorescent molecule is obtained by fitting a two-dimensional Gaussian function. The preset calibration curve is determined by the relationship between the standard deviation parameter of the imaging spot and the defocus depth.

3. The method for reconstructing three-dimensional point clouds using structured light from a multi-focal plane oblique light plate according to claim 2, characterized in that, The preset calibration curve is determined by the relationship between the standard deviation parameter of the imaging spot and the defocus depth, including: The variance parameters of the imaging spot in the x and y directions are determined by the standard deviation parameter of the imaging spot. The curves relating the difference in variance parameters of the imaging spot in the x and y directions to the defocus depth are fitted together and used as the preset calibration curves.

4. The method for reconstructing three-dimensional point clouds using structured light from a multi-focal plane oblique light sheet according to claim 1, characterized in that, The step of obtaining a preset calibration curve and calculating the depth information of the fluorescent molecule based on the standard deviation parameter and the calibration curve includes: Obtain multiple pre-defined calibration curves corresponding to different reference fluorescent molecules and multiple ellipticity curves corresponding to different reference fluorescent molecules; Substituting the standard deviation parameter into the preset multiple calibration curves yields multiple depth information; The depth information is substituted into the ellipticity curve to calculate the corresponding ellipticity, and the depth corresponding to the ellipticity that is closest to the fitted ellipticity is determined as the depth of the fluorescent molecule.

5. The method for reconstructing three-dimensional point clouds using structured light from a multi-focal plane oblique light sheet according to claim 2, characterized in that, The multi-camera registration file was determined using the following method: The image frame with the smallest absolute value of the difference in standard deviation parameters is determined as the focusing frame of the imaging spot; The relative positions and orientations of the multiple cameras are determined based on the centroid coordinates of the imaging spot in the focusing frame and the movement step size of the displacement stage, and saved as a multi-camera registration file.

6. The method for reconstructing three-dimensional point clouds using structured light from a multi-focal plane oblique light sheet according to claim 5, characterized in that, The relative positions and orientations of the multiple cameras include axial spacing and lateral spacing. The axial spacing of the multiple cameras is the product of the frame number difference between the focused frames and the movement step size. The lateral spacing of the multiple cameras is the difference in centroid coordinates of the imaging spot in the focused frame.

7. The method for reconstructing three-dimensional point clouds using structured light from a multi-focal-plane oblique light plate according to claim 1, characterized in that, The three-dimensional coordinates are transformed into upright three-dimensional coordinates using the following formula: x' = x y' = ycos(a) - zsin(a) z' = ysin(a) + zcos(a) Where 'a' is the angle between the camera's focal plane and the horizontal direction.

8. The method for reconstructing three-dimensional point clouds using structured light from a multi-focal-plane oblique light plate according to claim 1, characterized in that, The global three-dimensional coordinates are the sum of the upright three-dimensional coordinates and the three-dimensional coordinates of the displacement stage at the same moment. The three-dimensional coordinates of the displacement stage are determined by the initial position coordinates and the movement step size.

9. The method for reconstructing three-dimensional point clouds using structured light from a multi-focal plane oblique light plate according to claim 6, characterized in that, The multi-camera registration file includes the relative position parameters of the multiple cameras in the x-direction, the relative position parameters in the y-direction, and the relative position parameters in the z-direction. The relative position parameters in the x-direction and the relative position parameters in the y-direction are the difference in centroid coordinates of the imaging spot in the focused frame. The relative position parameter in the z-direction is the product of the frame number difference between the focused frames and the movement step size. The 3D point cloud registration of the multiple cameras includes adding the coordinate values ​​in the x-direction of the global 3D coordinate system to the relative position parameters in the x-direction, adding the coordinate values ​​in the y-direction of the global 3D coordinate system to the relative position parameters in the y-direction, and adding the coordinate values ​​in the z-direction of the global 3D coordinate system to the relative position parameters in the z-direction.

10. A multi-focal-plane oblique light structured light three-dimensional point cloud reconstruction device, characterized in that, The multi-focal-plane oblique light structured light three-dimensional point cloud reconstruction device includes: An image acquisition module is used to illuminate a sample with a slanted light plate. The sample moves with a displacement stage at a certain step size and is continuously imaged by multiple cameras to acquire a sequence of image frames captured by the multiple cameras. The focal planes of the multiple cameras are parallel, and the moving direction of the displacement stage is parallel to the arrangement direction of the multiple cameras. The spot fitting module is used to determine the imaging spot of the same fluorescent molecule in multiple image frame sequences, and to fit the imaging spot with a two-dimensional Gaussian function to obtain the centroid coordinates and standard deviation parameters of the fluorescent molecule. The depth calculation module is used to obtain a preset calibration curve, and calculate the depth information of the fluorescent molecule based on the standard deviation parameter and the calibration curve to obtain the three-dimensional coordinates of the fluorescent molecule in the camera coordinate system. The coordinate orthogonal module is used to transform the three-dimensional coordinates into orthogonal three-dimensional coordinates based on the angle between the focal plane of the plurality of cameras and the horizontal direction; The coordinate transformation module is used to transform the upright three-dimensional coordinates into global three-dimensional coordinates according to the movement step size of the displacement stage, and reconstruct multiple three-dimensional point clouds of the multiple cameras; The camera registration module is used to acquire a multi-camera registration file, register multiple 3D point clouds of the multiple cameras according to the multi-camera registration file, and obtain a 3D point cloud depth image of the sample.

Images