A computer vision-driven optical tomography reconstruction method and system

Abstract

Embodiments of the present application relate to the field of optical tomography technology, and in particular to a computer vision-driven optical tomography reconstruction method and system. The method includes the following steps: obtaining microscopic projection images of cells in multiple directions; identifying the cells in the microscopic projection images and cropping the microscopic projection images into images with consistent sizes; separating the cells from the background area; determining the exact positions of the cells in each frame based on the high contrast of the images after cell segmentation and precisely aligning the positions of the segmented cells; using a Harris corner detector to implement dynamic feature extraction on the cell rotation video; using an optical flow method to track the feature motion in the image sequence and calculating the rotation angle of the cells; generating a sinogram based on the microscopic projection images and the rotation angles of the corresponding cells, and reconstructing the three-dimensional intensity distribution of the cells. The present application utilizes computer vision technology to improve the speed, robustness, and accuracy of optical tomography image reconstruction.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of optical tomography, and particularly to a computer vision-driven optical tomography reconstruction method and system. Background Art

[0002] Optical tomography is a new emerging microscopic three-dimensional imaging method. By reconstructing cell projections at multiple angles through algorithms, three-dimensional imaging of subcellular structures can be achieved, thus promoting related research in fields such as cell function, interaction, and metabolism. In experiments, the method of rotating the illumination beam or rotating the cells is often used to obtain cell projections at multiple angles. However, the illumination scanning range achievable by the method of rotating the illumination beam is limited, resulting in three-dimensional anisotropic spatial resolution. The use of optical manipulation, fiber-optic manipulation, holographic optical tweezer, flow cytometry, acoustic manipulation, electromagnetic manipulation, or micro-manipulator can achieve cell rotation, so as to obtain full-angle projections of the cells to achieve optical tomography with three-dimensional isotropic spatial resolution.

[0003] The imaging method of the prior art uses the characteristic that cells randomly roll in a microfluidic channel to achieve optical tomography. The rotation angle and direction of the cells are obtained through manual image processing and phase three-dimensional modeling to achieve optical diffraction tomography. However, in the prior art, both image recognition and calibration rely on manual operations, and three-dimensional modeling of the cells needs to be pre-performed to match the corresponding rotation angles, which is time-consuming and laborious; reconstructing the 3D intensity distribution of cells from 2D projections usually requires cumbersome preprocessing, and accurately and automatically detecting the cell rotation angle is still a challenge. Summary of the Invention

[0004] The embodiments of the present application provide a computer vision-driven optical tomography reconstruction method and system, which use computer vision technology to improve the speed, robustness, and accuracy of optical tomography image reconstruction.

[0005] To solve the above technical problems, in a first aspect, an embodiment of the present application provides a computer vision-driven optical tomography reconstruction method, including the following steps: First, obtain microscopic projection images of cells in multiple directions; then, use a target detection convolutional neural network to identify cells in the microscopic projection images and crop the microscopic projection images into images with a consistent size; next, use an image segmentation deep neural network to separate cells from the background area to obtain the image of the cells after segmentation; then, based on the high contrast of the image of the cells after segmentation, determine the exact position of the cells in each frame and precisely align the positions of the segmented cells; next, use a Harris corner detector to perform dynamic feature extraction on the cell rotation video; use an optical flow method to track the feature motion in the image sequence and calculate the rotation angle of the cells; finally, generate a sinogram based on the microscopic projection images and the rotation angles of the corresponding cells, and reconstruct the three-dimensional intensity distribution of the cells based on the sinogram.

[0006] In some exemplary embodiments, using a target detection convolutional neural network to identify cells in the microscopic projection images includes: using a fine-tuned YOLOv5 model to perform target detection of cells and identify cells in the microscopic projection images; wherein, the YOLOv5 model is fine-tuned using a transfer learning method.

[0007] In some exemplary embodiments, after identifying cells in the microscopic projection images and before cropping the microscopic projection images into images with a consistent size, it further includes: extracting a region of interest from the microscopic projection images.

[0008] In some exemplary embodiments, using an image segmentation deep neural network to separate cells from the background area to obtain the image of the cells after segmentation includes: using a pre-trained U-Net network to separate cells from the background area and perform cell segmentation tasks; wherein, a large number of manually labeled microscopic projection images are used to pre-train the U-Net network so that the pre-trained U-Net network adapts to cell segmentation tasks.

[0009] In some exemplary embodiments, based on the high contrast of the image of the cells after segmentation, determining the exact position of the cells in each frame and precisely aligning the positions of the segmented cells includes: based on the high contrast of the image of the cells after segmentation, determining the exact position of the cells in each frame; precisely aligning the positions of the segmented cells by determining the minimum enclosing circle of the cell contours.

[0010] In some exemplary embodiments, a Harris corner detector is used to implement dynamic feature extraction on a cell rotation video, including: using a Harris corner detector to detect changes in the gradient direction of the edges around each pixel; determining the local structure of the image using the eigenvalues of matrix M; wherein the local structure includes planes, edges, and corners; and implementing dynamic feature extraction based on the local structure of the image and the detected changes in the gradient direction of the edges around each pixel.

[0011] In some exemplary embodiments, an optical flow method is used to track the feature motion in an image sequence and calculate the rotation angle of the cell, including: using the Lucas-Kanade optical flow method to track the feature motion in the image sequence and estimate the feature motion; and calculating the rotation angle of the cell based on the estimated feature motion.

[0012] In some exemplary embodiments, based on the microscopic projection image and the corresponding rotation angle of the cell, a sinogram is generated, and the three-dimensional intensity distribution of the cell is reconstructed based on the sinogram, including: establishing a sinogram based on the microscopic projection image and the corresponding rotation angle of the cell; wherein the sinogram is a two-dimensional representation of the cross-section of the cell captured at different angles; and using a reconstruction algorithm of inverse Radon transform to reconstruct the three-dimensional intensity distribution of the cell, thereby realizing the reconstruction of the three-dimensional intensity distribution of the cell.

[0013] In a second aspect, an optical tomography reconstruction system driven by computer vision provided by an embodiment of the present application includes: a projection image acquisition module, a cell recognition module, a cell segmentation module, a cell alignment module, a rotation angle calculation module, and a three-dimensional distribution reconstruction module connected in sequence; the projection image acquisition module is used to acquire microscopic projection images of the cell in multiple directions; the cell recognition module is used to use a target detection convolutional neural network to recognize the cells in the microscopic projection images and crop the microscopic projection images into images with a consistent size; the cell segmentation module is used to use an image segmentation deep neural network to separate the cells from the background area to obtain the image of the segmented cells; the cell alignment module is used to determine the exact position of the cells in each frame according to the high contrast of the image of the segmented cells and precisely align the positions of the segmented cells; the rotation angle calculation module is used to use a Harris corner detector to implement dynamic feature extraction on a cell rotation video; use an optical flow method to track the feature motion in an image sequence and calculate the rotation angle of the cell; the three-dimensional distribution reconstruction module is used to generate a sinogram based on the microscopic projection image and the corresponding rotation angle of the cell, and reconstruct the three-dimensional intensity distribution of the cell based on the sinogram.

[0014] In some exemplary embodiments, the cell recognition module includes a model fine-tuning module and an object detection module; wherein, the model fine-tuning module is used to fine-tune the YOLOv5 model by using the transfer learning method; the object detection module is used to perform object detection of cells by using the fine-tuned YOLOv5 model to identify the cells in the microscopic projection image; the cell segmentation module includes a pre-training module and a segmentation module; wherein, the pre-training module is used to pre-train the U-Net network by using a large number of manually labeled microscopic projection images so that the pre-trained U-Net network adapts to the cell segmentation task; the segmentation module is used to separate the cells from the background area by using the pre-trained U-Net network to perform the cell segmentation task.

[0015] The technical solution provided by the embodiment of the present application has at least the following advantages:

[0016] The embodiment of the present application provides a computer vision-driven optical tomography reconstruction method and system. The method includes the following steps: First, obtain microscopic projection images of cells in multiple directions; then, use an object detection convolutional neural network to identify the cells in the microscopic projection image and crop the microscopic projection image into images with a consistent size; next, use an image segmentation deep neural network to separate the cells from the background area to obtain the image after cell segmentation; then, determine the exact position of the cells in each frame according to the high contrast of the image after cell segmentation and precisely align the positions of the segmented cells; next, use a Harris corner detector to perform dynamic feature extraction on the cell rotation video; use an optical flow method to track the feature motion in the image sequence and calculate the rotation angle of the cells; finally, generate a sinogram based on the microscopic projection image and the rotation angle of the corresponding cells, and reconstruct the three-dimensional intensity distribution of the cells based on the sinogram.

[0017] The embodiment of the present application aims to provide a novel computer vision-driven optical tomography reconstruction method, which is driven by computer vision technology for AI autonomous tomography reconstruction, enhancing the robustness and efficiency of the cell rotation optical tomography imaging system. This method uses an object detection convolutional neural network (CNN) to perform real-time preprocessing on the projection, and at the same time uses deep learning to segment cells from the background, significantly improving the quality of 3D reconstruction and achieving alignment of cell positions between frames. Feature points are extracted using Harris corner detection, and an optical flow method is used to determine the exact rotation angle of the cells. Through the precise rotation angle and projection, finally, the three-dimensional intensity distribution of the cells is reconstructed using the inverse Radon transform. The method of the present application uses computer vision technology to improve the speed, robustness, and accuracy of optical tomography image reconstruction. This method is expected to provide a wider and more efficient optical tomography imaging method for biomedical research on single-cell analysis. Description of the Drawings

[0018] One or more embodiments are exemplarily illustrated by the pictures in the corresponding drawings. These exemplary illustrations do not constitute a limitation on the embodiments. Unless otherwise stated, the figures in the drawings do not constitute a scale limitation.

[0019] Figure 1 It is a schematic flowchart of a computer vision-driven optical tomography reconstruction method provided by an embodiment of the present application;

[0020] Figure 2 It is a flowchart of computer vision-driven autonomous optical tomography reconstruction provided by an embodiment of the present application;

[0021] Figure 3 It is a schematic structural diagram of a computer vision-driven optical tomography reconstruction system provided by an embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of tomographic imaging reconstruction of a simulated cell model using the method of the present application provided by an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of the effect of optical projection tomography reconstruction of a cell model using different methods for comparison provided by an embodiment of the present application;

[0024] Figure 6 It is a schematic diagram of reconstructing a cell model made of microbeads and hydrogel in an experiment using the method of the present application provided by an embodiment of the present application;

[0025] Figure 7 It is the intensity profile diagrams of the XY plane, XZ plane, and YZ plane, and the intensity distribution diagrams along the color marked lines of the X plane and XZ plane provided by an embodiment of the present application. Detailed implementation manners

[0026] As can be seen from the background art, there are technical problems in the prior art that both image recognition and calibration rely on manual operations, and it is necessary to perform three-dimensional modeling on cells in advance to match the corresponding rotation angles, which is time-consuming and laborious.

[0027] In the prior art, the optical tomography is usually realized by utilizing the characteristic that cells randomly roll in a microfluidic channel. The rotation angle and direction of the cells are obtained through manual image processing and phase three-dimensional modeling to achieve optical diffraction tomography. However, reconstructing the 3D intensity distribution of cells from 2D projections usually requires cumbersome preprocessing, and accurately and automatically detecting the cell rotation angle remains a challenge. Another method uses the traditional Hough transform to achieve cell position detection and calibration, and performs three-dimensional reconstruction on the calibrated cell projections. This method relies on the Hough transform to detect cells. However, the Hough transform is very sensitive to noise and cell shapes and requires fine parameter adjustment. This method still requires manual operation to identify the rotation angle of the cells and cannot achieve fully automatic optical tomography image reconstruction.

[0028] To solve the above technical problems, the embodiments of the present application provide a computer vision-driven optical tomography reconstruction method and system. The method includes the following steps: First, obtain microscopic projection images of cells in multiple directions; then, use a target detection convolutional neural network to identify the cells in the microscopic projection images and crop the microscopic projection images into images with consistent sizes; next, use an image segmentation deep neural network to separate the cells from the background area to obtain the image of the cells after segmentation; then, determine the exact positions of the cells in each frame according to the high contrast of the image of the cells after segmentation, and accurately align the positions of the segmented cells; next, use a Harris corner detector to achieve dynamic feature extraction on the cell rotation video; use the optical flow method to track the feature motion in the image sequence and calculate the rotation angle of the cells; finally, generate a sinogram based on the microscopic projection images and the rotation angles of the corresponding cells, and reconstruct the 3D intensity distribution of the cells based on the sinogram. The embodiments of the present application aim to provide a novel computer vision-driven optical tomography reconstruction method and system, which uses computer vision technology to improve the speed, robustness, and accuracy of optical tomography image reconstruction. This method is expected to provide a wider and more efficient optical tomography imaging method for biomedical research on single-cell analysis.

[0029] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art can understand that in the embodiments of the present application, many technical details are proposed to help readers better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can still be implemented.

[0030] See Figure 1 , the embodiments of the present application provide a computer vision-driven optical tomography reconstruction method, including the following steps:

[0031] Step S1: Obtain microscopic projection images of cells in multiple directions.

[0032] Step S2: Use a target detection convolutional neural network to identify cells in the microscopic projection images and crop the microscopic projection images into images with consistent sizes.

[0033] Step S3: Use an image segmentation deep neural network to separate the cells from the background region to obtain the image of the segmented cells.

[0034] Step S4: Determine the exact positions of the cells in each frame based on the high contrast of the image of the segmented cells and precisely align the positions of the segmented cells.

[0035] Step S5: Use a Harris corner detector to perform dynamic feature extraction on the cell rotation video; use the optical flow method to track the feature motion in the image sequence and calculate the rotation angle of the cells.

[0036] Step S6: Generate a sinogram based on the microscopic projection images and the corresponding rotation angles of the cells, and reconstruct the three-dimensional intensity distribution of the cells based on the sinogram.

[0037] The computer vision-driven optical tomography reconstruction method provided by this application has a core of an AI autonomous tomography reconstruction method driven by computer vision technology, enhancing the robustness and efficiency of the cell rotation optical tomography imaging system. The optical tomography reconstruction method of this application involves using a target detection convolutional neural network (CNN) to perform real-time preprocessing on the projections, while using deep learning to segment the cells from the background, significantly improving the quality of the 3D reconstruction and achieving the alignment of the cell positions between frames. Feature points are extracted using Harris corner detection, and the exact rotation angle of the cells is determined using the optical flow method. Through the exact rotation angle and projections, the three-dimensional intensity distribution of the cells is finally reconstructed using the inverse Radon transform.

[0038] Figure 2Shows the workflow of computer vision-driven autonomous optical tomography reconstruction. Among them, Figure a shows the original projection diagram (Raw data) of cells in multiple directions. Figure b shows the process of preprocessing (Pre processing) of cells. Specifically, the target detection convolutional neural network (CNN) can autonomously identify cells in the microscope image and crop the detected area through a dedicated algorithm to achieve a stable schematic diagram. Figure c shows the process of frame calibration (Frames calibration) of the image. Specifically, it is a schematic diagram of using an image segmentation deep neural network (DNN) to separate cells from the background and then carefully align the image. Figure d shows the process of rotation tracking (Rotation tracking) of cells. Specifically, it is a schematic diagram where optical flow helps to track and quantify the rotation angle. Figure e shows the process of tomographic reconstruction (Tomographic reconstruction). Specifically, a sinogram is generated based on the preprocessed projections and the corresponding rotation angles, thereby realizing the reconstruction of the three-dimensional intensity distribution.

[0039] Next, in combination with Figure 2 , the computer vision-driven optical tomography reconstruction method provided by this application will be described in detail.

[0040] First, execute step S1: Obtain microscopic projection images of cells in multiple directions. After obtaining the microscopic projection images in multiple directions, execute step S2: Use a target detection convolutional neural network to identify cells in the microscopic projection images and crop the microscopic projection images into images with consistent sizes, that is, perform cell recognition.

[0041] In some embodiments, in step S2, using a target detection convolutional neural network to identify cells in the microscopic projection images includes: using a fine-tuned YOLOv5 model to perform target detection of cells and identify the cells in the microscopic projection images; among them, the YOLOv5 model is fine-tuned using the transfer learning method.

[0042] After obtaining the microscopic projection images of cells in multiple directions, cells in the microscopic projection images are detected. This application is based on YOLOv5 for the cell recognition process. Specifically, the target detection algorithm YOLOv5 is used to autonomously identify cells in the original microscopic image. Although traditional training datasets do not include microscopic images of cells, the powerful generalization ability of YOLOv5 allows it to be used for cell detection in microscopic images through transfer learning. The pre-trained YOLOv5 is fine-tuned through a set of manually labeled microscopic cell images to optimize the network specifically for this task. This application selects "YOLOv5s" because it is the smallest and fastest version in the YOLOv5 series and is selected for its high efficiency and small model size.

[0043] After selecting YOLOv5s, transfer learning technology is used to fine-tune YOLOv5s to better adapt to the cell localization task. Transfer learning is a method that utilizes a model pre-trained on other problems and fine-tunes it to adapt to a new problem. Since YOLOv5 was initially trained on a large-scale and diverse dataset, it already has the ability to recognize various common features in images. Then, manual labeling and data augmentation are performed. To train the model, 100 manually labeled cell images are used. To enhance the robustness of the model and simulate different conditions, these labeled images undergo an extensive augmentation process, including rotation and random cropping, and are extended to 2,000 images to avoid overfitting. Loss function: Generalized Intersection over Union (GIoU) loss is used in the transfer learning of YOLOv5. Different from the traditional IoU, GIoU not only considers the overlapping area between the predicted bounding box and the ground truth bounding box, but also considers the area of the smallest enclosing box that contains the predicted bounding box and the ground truth bounding box, so it can more accurately evaluate the results of object detection. The first step of this technical solution is to use the fine-tuned YOLOv5 model for cell object detection to ensure high performance even with limited labeled data.

[0044] It should be noted that other object detection algorithms can also be used in this application: In addition to YOLOv5, there are other object detection algorithms such as Faster R-CNN, SSD, etc., which can also be trained and fine-tuned in a similar way to adapt to the cell detection task.

[0045] In some embodiments, after identifying the cells in the microscopic projection image in step S2 and before cropping the microscopic projection image into an image with a consistent size, it further includes: extracting the region of interest (ROI) from the microscopic projection image.

[0046] After cell detection, the ROI (Region of Interest) is extracted from the full-field image and cropped into a smaller image with a consistent size to stabilize the video. The OpenCV video stabilization package is used to align the cell positions in the cropped video frames. Next, cell segmentation is performed.

[0047] In some embodiments, in step S3, an image segmentation deep neural network is used to separate the cells from the background area to obtain the image after cell segmentation, including: using a pre-trained U-Net network to separate the cells from the background area and perform the cell segmentation task; among them, a large number of manually labeled microscopic projection images are used to pre-train the U-Net network so that the pre-trained U-Net network can adapt to the cell segmentation task.

[0048] The process of image segmentation is very important and has a great impact on the performance of tomographic reconstruction. Effective segmentation techniques can help distinguish biological samples from the noise in the background. Through image segmentation techniques, the contrast of the final 3D reconstruction has been significantly improved. Traditional edge detection algorithms: Commonly used ones such as the Canny filter, Sobel filter, and Hough transform are used to determine the cell boundaries in microscopic images. These methods are effective in many applications because they can capture the gradient changes on the object boundaries. However, their performance depends on the precise adjustment of parameters, and they rely heavily on the high contrast between the object and the background, which can be a problem in the imaging of biological samples. Advantages of deep learning: For 3D tomographic reconstruction, uniform segmentation is required on all frames, which can be a cumbersome task when parameters need to be fine-tuned for traditional algorithms. Deep learning methods, due to their inherent feature learning ability, are found to be advantageous here. They provide consistent and uniform segmentation between different frames without the need for cumbersome parameter adjustment. U-Net architecture: Among numerous deep learning architectures, the U-Net architecture has become a powerful tool for image segmentation tasks. Its design is simple and effective and is very suitable for biomedical image segmentation. Considering its ability to learn complex patterns from limited data, a pre-trained U-Net network is used in this application for the cell segmentation task. This pre-trained network is fine-tuned using a total of 215 manually labeled images to enable it to adapt to the specific single-cell and background segmentation task of this application. The network shows excellent performance, clearly separating the cells from the background and paving the way for high-quality tomographic reconstruction. In summary, although traditional edge detection methods perform well in some scenarios, in cell microscopic images, due to insufficient contrast or other challenges, deep learning methods such as U-Net provide a more advanced and consistent way of cell segmentation for achieving high-quality tomographic reconstruction.

[0049] It should be noted that the image segmentation method U-Net adopted in this application can also be replaced by Mask R-CNN.

[0050] In some embodiments, in step S4, according to the high contrast of the image after cell segmentation, the exact position of the cells in each frame is determined, and the positions of the segmented cells are precisely aligned, including:

[0051] Step S401: According to the high contrast of the image after cell segmentation, determine the exact position of the cells in each frame.

[0052] Step S402: Precisely align the positions of the segmented cells by determining the minimum enclosing circle of the cell contours.

[0053] After cell segmentation, precise alignment of cell positions is performed: Based on the high contrast of the images after cell segmentation, the present application determines the exact positions of the cells in each frame. This is achieved by determining the minimum enclosing circle of the cell contours.

[0054] It should be noted that the cell rotation methods of the present application include, but are not limited to, non-contact cell rotation methods such as optical manipulation, fiber-optic manipulation, holographic optical tweezers, flow cytometry, acoustic manipulation, and electromagnetic manipulation. The present application is also applicable to cell rotation methods based on electric or manual control of a micro-manipulator.

[0055] Next, the rotation angle detection is performed.

[0056] In some embodiments, the Harris corner detector is used in step S5 to achieve dynamic feature extraction on the cell rotation video, including:

[0057] Step S501: Use the Harris corner detector to detect the changes in the gradient direction of the edges around each pixel.

[0058] Step S502: Use the eigenvalues of the matrix M to determine the local structure of the image; where the local structure includes planes, edges, and corner points.

[0059] Step S503: Based on the local structure of the image and the detection of the changes in the gradient direction of the edges around each pixel, achieve dynamic feature extraction.

[0060] In some embodiments, the optical flow method is used in step S5 to track the feature motion in the image sequence and calculate the rotation angle of the cell, including:

[0061] Step S5011: Use the Lucas-Kanade optical flow method to track the feature motion in the image sequence and estimate the feature motion.

[0062] Step S5012: Based on the estimated feature motion, calculate the rotation angle of the cell.

[0063] To maintain the authenticity of tomographic reconstruction, it is crucial to accurately detect the rotation angle. The Harris corner detector is used to achieve dynamic feature extraction on the cell rotation video. This algorithm works by detecting large changes in the gradient direction of the edges around each pixel. This method is used to identify points where the edge direction changes significantly in a digital image. Mathematical principle: The mathematical principle of the Harris corner detector involves the gray value of the image, the movement of the window function, the approximation of the local autocorrelation function, and the discrimination of the local image structure. Using the eigenvalues of matrix M, this application can determine the local structure of the image, such as planes, edges, and corners. Optical flow method: After identifying the features using the Harris corner detector, the next step is to track the feature movement in the image sequence to calculate the rotation angle of the cell. The Lucas-Kanade optical flow method is a commonly used method for estimating feature movement in a video.

[0064] It should be noted that in the feature point detection step, the Harris corner detector can also be replaced by other methods such as SIFT, SURF, or ORB.

[0065] In some embodiments, in step S6, based on the microscopic projection image and the corresponding rotation angle of the cell, a sinogram is generated, and based on the sinogram, the three-dimensional intensity distribution of the cell is reconstructed, including:

[0066] Step S601, based on the microscopic projection image and the corresponding rotation angle of the cell, establish a sinogram; wherein, the sinogram is a two-dimensional representation of the cross-section of the cell captured at different angles.

[0067] Step S602, use the reconstruction algorithm of inverse Radon transform to reconstruct the three-dimensional intensity distribution of the cell, thereby realizing the reconstruction of the three-dimensional intensity distribution of the cell.

[0068] After calculating the rotation angle of the cell, a sinogram is established: Once the exact rotation angle of the cell corresponding to the two-dimensional projection is determined, this data can be used to construct a sinogram, which is a two-dimensional representation of the cross-section of the cell captured at different angles.

[0069] After establishing the sinogram, optical tomography reconstruction is performed: Finally, the reconstruction algorithm of inverse Radon transform is used to reconstruct the three-dimensional intensity distribution of the cell, thereby obtaining the reconstruction of the three-dimensional intensity distribution of the cell, contributing to a deeper understanding of its internal structure and properties.

[0070] The key points of the computer vision-driven optical tomography reconstruction method provided by this application are:

[0071] (1) Application of Computer Vision in Optical Tomography: This application uses the YOLOv5 object detection algorithm to identify cells in the original microscopic images, especially fine-tuning for specific applications in optical tomography through transfer learning.

[0072] (2) Image Segmentation Method: This application utilizes the U-Net deep learning architecture for segmenting cell microscopic images, especially its advantage in low contrast during cell microscopy imaging.

[0073] (3) Detection of Cell Rotation Angle: This application combines the Harris corner detector and the Lucas-Kanade optical flow method to dynamically extract the features of cell rotation in the video and calculate the cell rotation angle.

[0074] (4) 3D Reconstruction Technology: This application finally constructs a sinogram from the two-dimensional microscopic projection data and uses the inverse Radon transform to reconstruct the three-dimensional intensity distribution of the cells.

[0075] Compared with the prior art, the computer vision-driven optical tomography reconstruction method provided by this application has the following advantages:

[0076] (1) Automated Process: This invention combines the YOLOv5 object detection algorithm with the U-Net segmentation network, enabling automatic and accurate cell identification and segmentation in the original microscopic images, avoiding the complexity and error-proneness of manual operations.

[0077] (2) Elimination of 3D Modeling: In the method of this application, there is no need to pre-build a 3D model of the cells to match the rotation angle. By combining the Harris corner detector and the Lucas-Kanade optical flow method, this invention can dynamically extract the features of cell rotation in the video and calculate its rotation angle, greatly improving the accuracy and efficiency of angle detection.

[0078] (3) Robustness to Noise and Shape: Using deep learning methods, especially the U-Net segmentation network, has good robustness to noise and irregular cell shapes in microscopic images. Compared with the Hough transform, this greatly improves the accuracy and stability, and there is no need for complex parameter adjustment.

[0079] (4) Fully Automatic Optical Tomography Image Reconstruction: Integrating methods of cell recognition, image segmentation, rotation angle detection, and 3D reconstruction, it automatically reconstructs the 3D intensity distribution of the cells from 2D projections, avoiding manual operations and cumbersome preprocessing, and achieving fully automatic optical tomography image reconstruction.

[0080] (5) Improvement of Work Efficiency and Precision: Due to the combination of various advanced image processing technologies, this method not only greatly reduces the overall processing time but also improves the precision of image analysis.

[0081] See Figure 3 In an embodiment of the present application, a computer vision-driven optical tomography reconstruction system is further provided, including: a projection image acquisition module 101, a cell recognition module 102, a cell segmentation module 103, a cell alignment module 104, a rotation angle calculation module 105, and a three-dimensional distribution reconstruction module 106, which are connected in sequence; the projection image acquisition module 101 is used to acquire microscopic projection images of cells in multiple directions; the cell recognition module 102 is used to use a target detection convolutional neural network to recognize cells in the microscopic projection images and crop the microscopic projection images into images with a consistent size; the cell segmentation module 103 is used to use an image segmentation deep neural network to separate cells from the background area to obtain an image after cell segmentation; the cell alignment module 104 is used to determine the exact position of cells in each frame according to the high contrast of the image after cell segmentation and accurately align the positions of the segmented cells; the rotation angle calculation module 105 is used to use a Harris corner detector to perform dynamic feature extraction on a cell rotation video; use an optical flow method to track the feature motion in an image sequence and calculate the rotation angle of the cells; the three-dimensional distribution reconstruction module 106 is used to generate a sinogram according to the microscopic projection images and the rotation angles of the corresponding cells, and reconstruct the three-dimensional intensity distribution of the cells based on the sinogram.

[0082] In some embodiments, the cell recognition module 102 includes a model fine-tuning module 1021 and a target detection module 1022; wherein, the model fine-tuning module 1021 is used to fine-tune the YOLOv5 model using a transfer learning method; the target detection module 1022 is used to perform target detection of cells using the fine-tuned YOLOv5 model to recognize cells in the microscopic projection images; the cell segmentation module 103 includes a pre-training module 1031 and a segmentation module 1032; wherein, the pre-training module 1031 is used to pre-train the U-Net network using a large number of manually labeled microscopic projection images to make the pre-trained U-Net network adapt to the cell segmentation task; the segmentation module 1032 is used to use the pre-trained U-Net network to separate cells from the background area and perform the cell segmentation task.

[0083] The computer vision-driven optical tomography reconstruction method and system provided by the present application are verified through experiments and simulations, and the experimental results prove to be feasible, and the results are shown as follows.

[0084] Figure 4 Shows a schematic diagram of tomographic imaging reconstruction of a simulated cell model using this method. Among them, (a) is a schematic diagram of tracking the rotation of the cell model using the optical flow method; (b) is a schematic diagram of the rotation angle of the tracked cell model; (c) is a schematic diagram of the corrected rotation angle and the true value; (d) is a schematic diagram of the three-dimensional intensity distribution reconstruction of the cell model.

[0085] Figure 5 Shows a schematic diagram of the effect of optical projection tomography reconstruction of a cell model using different methods. (a) is a three-dimensional visualization of a simulated cell model. (b) is a cross-section of the reconstructed cell model in the XZ, XY, and YZ planes. (c) is a schematic diagram of the principle of optical tomography based on the scanning illumination angle. (d) is a cross-sectional view of the cell model reconstructed using the scanning illumination angle tomography technique. (e) is a schematic diagram of the principle of optical cell rotational tomography. (f) is a cross-sectional view of the cell model using the computer vision-driven autonomous optical tomography reconstruction method. (g) and (h) respectively represent the quantitative comparison of the intensity profiles along the colored marked broken lines on the cross-section of the reconstructed (g) XZ plane and (h) YZ plane. Red line: traditional scanning illumination angle tomography reconstruction; blue line: this calculator vision-driven tomography reconstruction.

[0086] The experimental results are evaluated below.

[0087] (1) Nature of the experiment: The present invention is verified through a simulation experiment by simulating the rotation of a cell model (cellphantom) as a reliable reference for evaluating the accuracy and effectiveness of the method.

[0088] (2) Experimental results:

[0089] The present application successfully tracked the rotation angle of the simulated cell model using the proposed optical flow technique. The average measurement error of this method for cell rotation is 1.49°, showing high precision in cell rotation tracking based on the multi-feature optical flow tracking method.

[0090] The present application reconstructed the cross-section of the cell model from the exact rotation angle of each projection using the Radon inverse transform and reconstructed the three-dimensional intensity distribution of the cell model.

[0091] (3) Effect comparison:

[0092] The present application compared the proposed automated tomography reconstruction workflow with the original cell model and the traditional illumination scanning tomography reconstruction using only limited projection angles. From the results, the proposed tomography reconstruction method showed a significant improvement in the z-axis (axial) resolution.

[0093] The present application further quantitatively compared the intensity distributions of the reconstructed tomograms and found that the proposed cell rotation tomography reconstruction (represented by the blue line) is consistent with the ground truth intensity distribution of the cell model, showing improved accuracy and superior reconstruction ability of the proposed method.

[0094] (4) Statistical error index:

[0095] This application quantifies the 3D reconstruction errors of two tomographic reconstruction methods by using statistical error metrics (MSE, MAE, RMSE). The results show that using the proposed tomographic reconstruction method can significantly reduce the 3D reconstruction errors.

[0096] This application further evaluates the accuracy of 3D reconstruction by using two evaluation metrics, MM-SSIM and PSNR. The results show that the proposed cell rotation-based tomographic method has a significant improvement in the accuracy of 3D reconstruction.

[0097] In summary, experiments, simulations, and usage have all demonstrated the feasibility and advantages of the proposed method. Compared with traditional tomographic imaging methods, the AI-based automated workflow of this application shows obvious advantages in terms of reconstruction accuracy, accuracy, and robustness.

[0098] Table 1 Comparison of accuracy data of the optical tomographic reconstruction method of this application with traditional methods

[0099]

[0100] As can be seen from Table 1, compared with the optical tomographic reconstruction method of traditional methods, the accuracy of the method of this application has been significantly improved; among them, MSE: mean squared error; MAE: mean absolute error; RMSE: root mean square error; MS-SSIM: multi-scale structural similarity; PSNR: peak signal-to-noise ratio.

[0101] In addition, the method of the present invention has also been verified in experiments. Figure 6 、 Figure 7 Shown is an actual cell model reconstructed by using the method of the present invention. The model is fixed in a dual-beam fiber optic optical trap and rotated through microfluidic control to obtain two-dimensional projections. This not only verifies the accuracy of the method of this application but also proves its versatility in different cell rotation tomographic imaging systems.

[0102] Figure 6 Shows a schematic diagram of reconstructing a cell model made of microbeads and hydrogel by using the method of this application in experiments. Among them, (a) is a top view along the Z axis; (b) is an isometric view at a 45° angle; (c) is a side view aligned with the X axis. (d) is a side view aligned with the Y axis.

[0103] The artificial cell model is made of transparent hydrogel beads and two embedded silica microspheres with diameters of 2.9 microns and 2.5 microns respectively. Through the self-developed tomographic reconstruction method of this application, three-dimensional intensity reconstruction of the artificial cell model is achieved. The two microspheres are clearly visible in the three-dimensional reconstruction and can be clearly identified in the XY, YZ, and XZ planes, thus verifying the high fidelity of the tomographic reconstruction process of this application. In addition, by carefully observing the boundary of the hydrogel, subtle texture changes can be found, indicating that the boundary material has been displaced due to the presence of the embedded microspheres, and these details are particularly obvious in the figure.

[0104] To quantitatively evaluate the tomographic reconstruction in various three-dimensional directions, Figure 7 Cross-sectional images in the XY, XZ, and YZ planes are shown. Among them, Figure (a) is the XY plane; Figure (b) is the XZ plane; Figure (c) is the intensity profile of the YZ plane, and Figure (d) is a schematic diagram of the intensity distribution along the color-marked line of the XY plane (blue) and the XZ plane (yellow), with a scale of 3μm.

[0105] These images reveal the intensity profiles of the two microspheres, showing a high contrast between the silicon microspheres and their hydrogel environment. Moreover, the resolutions achieved in the XY plane and the XZ plane are very close, indicating that nearly isotropic resolution is achieved in three-dimensional space.

[0106] Generally speaking, this AI-driven tomographic reconstruction workflow shows high precision and detail reproduction ability when processing real data, proving the effectiveness and reliability of this method in the experimental environment.

[0107] It should be noted that this algorithm can be used not only for optical tomography, but also for fields such as X-ray tomographic microscopy, terahertz tomography, fluorescence tomography, and coherent optical tomography. At the same time, tomography also has different expressions with the same meaning such as tomographic imaging.

[0108] With the above technical solutions, the embodiments of the present application provide a computer vision-driven optical tomography reconstruction method and system. The method includes the following steps: First, obtain microscopic projection images of cells in multiple directions; then, use a target detection convolutional neural network to identify the cells in the microscopic projection images and crop the microscopic projection images into images with a consistent size; next, use an image segmentation deep neural network to separate the cells from the background area to obtain the image after cell segmentation; then, determine the exact positions of the cells in each frame according to the high contrast of the image after cell segmentation and precisely align the positions of the cells; next, use a Harris corner detector to perform dynamic feature extraction on the cell rotation video; use an optical flow method to track the feature motion in the image sequence and calculate the rotation angle of the cells; finally, generate a sinogram based on the microscopic projection images and the corresponding rotation angles of the cells, and reconstruct the three-dimensional intensity distribution of the cells based on the sinogram.

[0109] The embodiments of the present application aim to provide a novel computer vision-driven optical tomography reconstruction method. Driven by computer vision technology, AI performs autonomous tomography reconstruction, enhancing the robustness and efficiency of the cell rotation optical tomography imaging system. This method uses a target detection convolutional neural network (CNN) to perform real-time preprocessing on the projections, and at the same time uses deep learning to segment the cells from the background, significantly improving the quality of 3D reconstruction and achieving alignment of cell positions between frames. The Harris corner detection is used to extract feature points, and the optical flow method is used to determine the exact rotation angle of the cells. Through the precise rotation angle and projections, finally, the three-dimensional intensity distribution of the cells is reconstructed using the Radon inverse transform. The method of the present application uses computer vision technology to improve the speed, robustness, and accuracy of optical tomography image reconstruction. This method is expected to provide a more extensive and efficient optical tomography imaging method for biomedical research on single-cell analysis.

[0110] Those of ordinary skill in the art can understand that the above embodiments are specific embodiments for implementing the present application. In actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application. Any person skilled in the art can make their own changes and modifications without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application should be subject to the scope defined by the claims.

Claims

1. A computer vision-driven optical tomography reconstruction method, characterized in that, it includes the following steps: Obtain microscopic projection images of cells in multiple directions; Use a target detection convolutional neural network to identify the cells in the microscopic projection images and crop the microscopic projection images into images with a consistent size; Use an image segmentation deep neural network to separate the cells from the background area to obtain the image of the cells after segmentation; Based on the high contrast of the image of the cells after segmentation, determine the exact positions of the cells in each frame and precisely align the positions of the segmented cells; Use a Harris corner detector to implement dynamic feature extraction on the cell rotation video; use an optical flow method to track the feature motion in the image sequence and calculate the rotation angle of the cells; Generate a sinogram based on the microscopic projection images and the rotation angles of the corresponding cells, and reconstruct the three-dimensional intensity distribution of the cells based on the sinogram.

2. The computer vision-driven optical tomography reconstruction method according to claim 1, characterized in that, The step of using a target detection convolutional neural network to identify the cells in the microscopic projection images includes: Use a fine-tuned YOLOv5 model to perform target detection of cells and identify the cells in the microscopic projection images; Among them, the YOLOv5 model is fine-tuned using a transfer learning method.

3. The computer vision-driven optical tomography reconstruction method according to claim 1, characterized in that, After identifying the cells in the microscopic projection images and before cropping the microscopic projection images into images with a consistent size, it further includes: Extract the region of interest from the microscopic projection images.

4. The computer vision-driven optical tomography reconstruction method according to claim 1, characterized in that, The step of using an image segmentation deep neural network to separate the cells from the background area to obtain the image of the cells after segmentation includes: Use a pre-trained U-Net network to separate the cells from the background area and perform the cell segmentation task; among them, A large number of manually labeled microscopic projection images are used to pre-train the U-Net network so that the pre-trained U-Net network adapts to the cell segmentation task.

5. The computer vision-driven optical tomography reconstruction method according to claim 1, characterized in that, Based on the high contrast of the image of the cells after segmentation, determine the exact positions of the cells in each frame and precisely align the positions of the segmented cells, including: Based on the high contrast of the image of the cells after segmentation, precisely align the positions of the segmented cells by determining the minimum enclosing circle of the cell contours.

6. The computer vision-driven optical tomography reconstruction method according to claim 1, characterized in that, The step of using a Harris corner detector to implement dynamic feature extraction on the cell rotation video includes: Use a Harris corner detector to detect the changes in the gradient direction of the edges around each pixel; Use the eigenvalues of the matrix M to determine the local structure of the image; where the local structure includes planes, edges and corners; based on the local structure of the image and the detection of the changes in the gradient direction of the edges around each pixel, implement dynamic feature extraction.

7. The computer vision-driven optical tomography reconstruction method according to claim 1, characterized in that, the method of using the optical flow method to track the feature motion in the image sequence and calculate the rotation angle of the cell includes: using the Lucas-Kanade optical flow method to track the feature motion in the image sequence and estimate the feature motion; calculating the rotation angle of the cell based on the estimated feature motion.

8. The computer vision-driven optical tomography reconstruction method according to claim 1, characterized in that, generating a sinogram based on the microscopic projection image and the corresponding rotation angle of the cell, and reconstructing the three-dimensional intensity distribution of the cell based on the sinogram, including: establishing a sinogram based on the microscopic projection image and the corresponding rotation angle of the cell; wherein, the sinogram is a two-dimensional representation of the cross-section of the cell captured at different angles; using the reconstruction algorithm of the inverse Radon transform to reconstruct the three-dimensional intensity distribution of the cell, thereby realizing the reconstruction of the three-dimensional intensity distribution of the cell.

9. A computer vision-driven optical tomography reconstruction system, characterized in that, comprising: a projection image acquisition module, a cell recognition module, a cell segmentation module, a cell alignment module, a rotation angle calculation module, and a three-dimensional distribution reconstruction module connected in sequence; the projection image acquisition module is used to acquire microscopic projection images of the cell in multiple directions; the cell recognition module is used to use a target detection convolutional neural network to recognize the cells in the microscopic projection image and crop the microscopic projection image into images with consistent sizes; the cell segmentation module is used to use an image segmentation deep neural network to separate the cells from the background area to obtain the image after cell segmentation; the cell alignment module is used to determine the exact position of the cell in each frame according to the high contrast of the image after cell segmentation and perform precise alignment on the segmented cells; the rotation angle calculation module is used to use a Harris corner detector to realize dynamic feature extraction on the cell rotation video; use the optical flow method to track the feature motion in the image sequence and calculate the rotation angle of the cell; the three-dimensional distribution reconstruction module is used to generate a sinogram based on the microscopic projection image and the corresponding rotation angle of the cell, and reconstruct the three-dimensional intensity distribution of the cell based on the sinogram.

10. The computer vision-driven optical tomography reconstruction system according to claim 9, characterized in that, the cell recognition module includes a model fine-tuning module and a target detection module; wherein, the model fine-tuning module is used to fine-tune the YOLOv5 model using the transfer learning method; the target detection module is used to use the fine-tuned YOLOv5 model to perform target detection of cells and recognize the cells in the microscopic projection image; the cell segmentation module includes a pre-training module and a segmentation module; wherein, the pre-training module is used to pre-train the U-Net network using a large number of manually labeled microscopic projection images so that the pre-trained U-Net network adapts to the cell segmentation task; The segmentation module is used to separate cells from the background region using a pre-trained U-Net network to perform the cell segmentation task.

Images