Image registration method, device and equipment for ODT data based on coarse calibration system

Abstract

The present invention provides an image registration method, device, and apparatus for ODT data based on a coarse calibration system. The method comprises: extracting single-layer two-dimensional data matching a widefield fluorescence image from three-dimensional refractive index data; extracting feature points from the single-layer two-dimensional data and the widefield fluorescence image to obtain an ODT feature point set and a widefield fluorescence feature point set; matching the ODT feature point set and the widefield fluorescence feature point set using an adaptive iterative algorithm improved by a clustering algorithm and iteratively generating an affine transformation matrix; constructing a differential optimizer using the affine transformation matrix as an initial solution, inputting the affine transformation matrix into the differential optimizer for optimization to obtain a perspective transformation matrix; and determining the image registration result based on the perspective transformation matrix. The method specifically implements image registration for ODT three-dimensional refractive index data and multi-channel widefield fluorescence data, thereby achieving image fusion of label-free nonspecific imaging and specific imaging methods for the same sample at the same time.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image registration method, device and equipment for ODT data based on a coarse calibration system. Background Art

[0002] Optical Diffraction Tomography (ODT) is a technology used for three-dimensional microscopic imaging. It uses a laser in conjunction with a scanning galvanometer to change the incident angle, acquire data from multiple angles of the sample, and reconstruct the three-dimensional refractive index distribution of the sample based on a reconstruction algorithm. It has the advantages of being non-invasive and high-resolution. Wide-field fluorescence imaging is a technology that uses fluorescent markers and wide-angle illumination to obtain sample images. In order to achieve the complementarity of label-free nonspecific imaging and specific imaging methods to achieve comprehensive, holistic, and dynamic observation and research of samples, it is necessary to integrate the two imaging technologies into a multimodal microscopic imaging system and realize image fusion of multimodal imaging of the same sample in the same state through image registration methods.

[0003] Traditional natural image registration typically employs the following methods: feature point extraction using a scale-invariant algorithm, followed by feature matching using brute-force matching or a k-nearest neighbor algorithm, a transformation model selected based on potential transformations, and parameter estimation using a random sampling consensus algorithm. Using these conventional registration methods for ODT data presents two difficulties: First, due to the computational imaging nature of ODT data, the data reflects the three-dimensional refractive index of the sample rather than intensity or grayscale information. Furthermore, due to the noise and errors inherent in computational imaging, accurate feature point extraction is rarely possible. Second, even when the data is preprocessed as closely as possible into binary grayscale images with similar image features to widefield fluorescence for registration, registration accuracy is low, resulting in poor results and even erroneous information that can affect judgment and decision-making. Avoiding this issue requires not only fine-tuning the hardware system to a precisely calibrated or even ideal state, but also employing a time-consuming, globally optimal brute-force registration algorithm, which significantly increases both financial and time costs. Summary of the Invention

[0004] The present invention provides an image registration method, device and equipment for ODT data based on a coarse calibration system, which is used to solve the defect that the existing technology cannot process the ODT data registration requirements, while ensuring high accuracy and high processing efficiency to achieve ODT data image registration and fusion.

[0005] The present invention provides an image registration method for ODT data based on a coarse calibration system, comprising the following steps:

[0006] Acquiring three-dimensional refractive index data of an ODT image to be registered, and extracting single-layer two-dimensional data that matches the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired at the same time for the same sample, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area;

[0007] Extracting feature points from the single-layer two-dimensional data and the wide-field fluorescence image to obtain an ODT feature point set and a wide-field fluorescence feature point set;

[0008] Based on an adaptive iterative algorithm improved by a clustering algorithm, the ODT feature point set and the wide-field fluorescence feature point set are matched and an affine transformation matrix is ​​iteratively generated;

[0009] Taking the affine transformation matrix as an initial solution, constructing a differential optimizer, inputting the affine transformation matrix into the differential optimizer for optimization, and obtaining a perspective transformation matrix;

[0010] An image registration result of the three-dimensional refractive index data and the wide-field fluorescence image is determined based on the perspective transformation matrix.

[0011] According to the image registration method for ODT data based on a coarse calibration system provided by the present invention, the fitness function of the differential optimizer includes any one of mean square error and structural similarity.

[0012] According to an image registration method for ODT data based on a coarse calibration system provided by the present invention, feature point extraction is performed on the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain an ODT feature point set and a wide-field fluorescence feature point set, including:

[0013] A scale space extreme value detection algorithm is used to extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image, respectively, to obtain the ODT feature point set and the wide-field fluorescence feature point set.

[0014] According to an image registration method for ODT data based on a coarse calibration system provided by the present invention, the steps of acquiring the single-layer two-dimensional data and the wide-field fluorescence image include:

[0015] Acquire original single-layer two-dimensional data and original wide-field fluorescence images;

[0016] After sequentially performing image binarization, image inversion, and edge feature extraction on the original single-layer two-dimensional data, holes in the edge information are filled to obtain the single-layer two-dimensional data;

[0017] The original wide-field fluorescence image is subjected to image binarization and image smoothing to obtain the wide-field fluorescence image.

[0018] According to an image registration method for ODT data based on a coarse calibration system provided by the present invention, the adaptive iterative algorithm improved based on the clustering algorithm matches the ODT feature point set and the wide-field fluorescence feature point set and iteratively generates an affine transformation matrix, further comprising:

[0019] During the iteration process, the ODT feature point set and the wide-field fluorescence feature point set that meet the error requirements are reversely screened.

[0020] According to an image registration method for ODT data based on a coarse calibration system provided by the present invention, the iterative process reversely screens the ODT feature point set and the wide-field fluorescence feature point set that meet the error requirements, including:

[0021] Step s1: determining an initial affine transformation matrix based on the ODT feature point set and the widefield fluorescence feature point set, and determining a transformed ODT image based on the initial affine transformation matrix and the ODT image to be registered;

[0022] Step s2: Based on the transformed ODT image and the wide-field fluorescence image, calculating the Manhattan distance of the transformed corresponding point sets, then using the Manhattan distance as an evaluation index, and filtering out ODT feature point sets and wide-field fluorescence feature point sets whose errors are greater than a set threshold based on the evaluation index;

[0023] Repeat steps s1 and s2 to obtain the ODT feature point set, the wide-field fluorescence feature point set, and the affine transformation matrix that meet the error requirements.

[0024] The present invention also provides an image registration system for ODT data based on a coarse calibration system, comprising the following modules:

[0025] an acquisition unit, configured to acquire three-dimensional refractive index data of an ODT image to be registered, and extract a single layer of two-dimensional data that matches the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired at the same time for the same sample, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area;

[0026] a feature point extraction unit, configured to extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image, respectively, to obtain an ODT feature point set and a wide-field fluorescence feature point set;

[0027] A matching unit, configured to match the ODT feature point set and the wide-field fluorescence feature point set based on an adaptive iterative algorithm and iteratively generate an affine transformation matrix;

[0028] A perspective transformation matrix solving unit is used to use the affine transformation matrix as an initial solution, construct a differential optimizer, input the affine transformation matrix into the differential optimizer for optimization, and obtain a perspective transformation matrix;

[0029] A determining unit is configured to determine an image registration result of the three-dimensional refractive index data and the wide-field fluorescence image based on the perspective transformation matrix.

[0030] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the image registration method for ODT data based on a coarse calibration system as described above is implemented.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image registration method for ODT data based on a coarse calibration system as described above is implemented.

[0032] The present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described image registration methods for ODT data based on a coarse calibration system.

[0033] The present invention provides an image registration method, device and equipment for ODT data based on a coarse calibration system. On the one hand, it realizes image registration specifically for ODT three-dimensional refractive index data and multi-channel wide-field fluorescence data, thereby realizing image fusion of label-free nonspecific imaging and specific imaging methods of the same sample at the same time. It can not only capture the overall structure and morphological changes of the sample, but also highlight the interactions and processes of specific biological molecules or chemical components, providing new perspectives and clues for scientific research; on the other hand, it provides an image registration method with lower time cost, higher accuracy and faster time efficiency for coarse calibration or microscopic imaging systems with large errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 This is one of the flow charts of the image registration method for ODT data based on the coarse calibration system provided by the present invention.

[0036] Figure 2It is a schematic diagram of the three-dimensional refractive index volume image of the fluorescent microsphere sample ODT provided by the present invention.

[0037] Figure 3 It is a schematic diagram of a wide-field fluorescence image of a fluorescent microsphere sample provided by the present invention.

[0038] Figure 4 Schematic diagram of the pre-processed ODT image provided by the present invention.

[0039] Figure 5 This is a fusion effect diagram of ODT three-dimensional data and multi-channel wide-field fluorescence data of the cell sample provided by the present invention.

[0040] Figure 6 This is the second flow chart of the image registration method for ODT data based on the coarse calibration system provided by the present invention.

[0041] Figure 7 It is a structural diagram of an image registration system for ODT data based on a coarse calibration system provided by the present invention.

[0042] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0044] Figure 1 This is one of the flow charts of the image registration method for ODT data based on the coarse calibration system provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 , step 130 , step 140 and step 150 .

[0045] Step 110: Acquire three-dimensional refractive index data of the ODT image to be registered, and extract a single layer of two-dimensional data that matches the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired at the same time from the same sample, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area.

[0046] Specifically, the ODT image and wide-field fluorescence image to be registered are acquired by a multimodal microscopy system at the same time from the same sample. The system state of the multimodal microscopy system is coarse calibration, which includes system errors such as different illumination angles of the two modal light sources, tilt of the sample or translation stage, and insufficient resetting accuracy of the mirror frame turntable.

[0047] Here, the multimodal microscopy imaging system includes a laser, a translation stage, two cameras, a microscope stand, a scanning galvanometer, a motorized aperture, a fluorescent light box, and optical elements; the microscope stand is mounted on the bottom surface, and the remaining hardware equipment is mounted on the microscope stand; the system uses a specific optical path and hardware imaging, where the ODT data is derived from the data collected by the camera and is calculated and imaged through a solution module.

[0048] Accordingly, standard fluorescent-stained microspheres must first be prepared as registration samples. This is because standard microspheres typically have more distinct and sharp-edged image features. For the specificity of widefield fluorescence images, all microspheres specific to a specific wavelength can be excited to emit light. A multimodal microscopy system is then used to obtain ODT images and widefield fluorescence images of the microspheres at the same time, with different fields of view but overlapping areas.

[0049] Figure 2 Schematic diagram of the three-dimensional refractive index volume image of the fluorescent microsphere sample ODT provided by the present invention, Figure 3 Schematic diagram of the wide-field fluorescence image of the fluorescent microsphere sample provided by the present invention, such as Figure 2 and Figure 3 As shown. Figure 2 The three-dimensional refractive index volume data of the ODT image to be registered is shown, which reflects the refractive index volume data of the sample within a certain z-axis range. The characteristics of different layers of the image, such as grayscale, gradient, edge information, and noise, vary greatly. It is necessary to use an algorithm to calculate and screen out the specific layer that truly corresponds to the focal plane of the wide-field fluorescence imaging data, that is, the actual height of the z-axis. The specific steps are: establish ODT image evaluation indicators; calculate the evaluation function; screen out several required optimal layers; remove local optimal solutions to obtain the final required ODT image layer corresponding to the focal plane of the wide-field fluorescence data.

[0050] For the ODT data of a specific layer that is screened out, due to its computational imaging characteristics, the data reflects the refractive index information of the sample rather than the intensity or grayscale information. In addition, due to the noise and errors specific to computational imaging, it is almost impossible to produce correct results when extracting feature points. Figure 4 is a schematic diagram of the pre-processed ODT image provided by the present invention, such as Figure 4As shown, a series of preprocessing methods can be applied to specific layers of ODT data to enhance and enhance image features similar to those of widefield fluorescence microspheres. For example, an algorithm is first designed to map the refractive index of the filtered raw single-layer 2D data to grayscale space. This is followed by image binarization, image inversion, and edge feature extraction. Finally, the edge information is filled to obtain the single-layer 2D data. The raw widefield fluorescence image is then binarized and smoothed to obtain the widefield fluorescence image.

[0051] Grayscale mapping of the image, that is, mapping the original refractive index image value from the floating point range of about -2.000000 to 1.7000000 to integer values ​​​​from 0 to 255, and at the same time, appropriately truncating the upper and lower limits of 3% to 5% according to the overall distribution of the refractive index value, in order to make the mapped image show a contrast that is easier to match

[0052] Image binarization is the process of setting the grayscale values ​​of pixels on an image to 0 or 255, making the entire image appear to be only black and white. Binarization is the simplest method of image segmentation, which can convert a grayscale image into a binary image.

[0053] Image inversion is the process of reversing the color value of each pixel in an image. In a grayscale color space, this usually means subtracting the original grayscale value of each pixel from 255.

[0054] Edge feature extraction refers to extracting pixels or regions with edge features from an image. Edges usually represent the boundaries of objects or discontinuous changes, and are important for tasks such as image segmentation, recognition, and reconstruction.

[0055] The purpose of filling holes is usually to improve the visual effect of the image and increase the accuracy of subsequent image analysis or recognition.

[0056] Image smoothing is actually a low-pass filter, which makes the image grayscale smoother, reduces the sudden gradient, and thus improves the image quality.

[0057] Step 120 : extracting feature points from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain an ODT feature point set and a wide-field fluorescence feature point set.

[0058] Specifically, feature points are extracted from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain an ODT feature point set and a wide-field fluorescence feature point set.

[0059] Here, a scale space extremum detection algorithm can be used to extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image, respectively, to obtain an ODT feature point set and a wide-field fluorescence feature point set.

[0060] In order to detect feature points at different scales, we first need to construct a scale space for the image. This is done by performing a Gaussian blur operation on the image and repeating it at different scales. Each blurred image represents a different scale version of the image, allowing the algorithm to detect features of different sizes. In the constructed scale space, extreme points are detected by comparing neighboring pixels. Specifically, each pixel is compared with pixels at adjacent scales and adjacent spatial locations. If a point appears as a local maximum (or minimum) in all comparisons, it is considered a potential feature point.

[0061] Step 130 : Based on the adaptive iterative algorithm improved by the clustering algorithm, the ODT feature point set and the wide-field fluorescence feature point set are matched and an affine transformation matrix is ​​iteratively generated.

[0062] Specifically, considering that the detected potential feature points may also contain some unreliable points, further screening is required. An adaptive iterative algorithm improved based on the clustering algorithm can be used to match the ODT feature point set and the wide-field fluorescence feature point set and iteratively generate an affine transformation matrix.

[0063] Step 140: Use the last generated affine transformation matrix as the initial solution, construct a differential optimizer, input the affine transformation matrix into the differential optimizer for optimization, and obtain a perspective transformation matrix.

[0064] Specifically, the affine transformation matrix is ​​used as the initial solution, the parameters and their estimation intervals are simplified, and the differential optimization method is adopted to globally estimate the perspective transformation matrix to compensate for possible perspective transformations and ensure the accuracy and fast convergence of the registration results.

[0065] That is, the affine transformation matrix is ​​input into the differential optimizer for optimization to obtain the perspective transformation matrix.

[0066] Here, the fitness function of the differential optimizer includes any one of mean squared error (MSE) and structural similarity (SSIM).

[0067] Here, the affine transformation matrix generated by the last iteration is taken as the initial solution for estimating the perspective transformation matrix. The affine transformation matrix is:

[0068]

[0069]

[0070]

[0071] in, is the original image coordinate, , are the transformed coordinates; is the amount of rotation, , , is the translation amount.

[0072] The perspective transformation matrix is:

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] in, is the original image coordinate, , are the transformed coordinates; is the amount of rotation, is the translation amount.

[0080] To ensure registration accuracy, the coarse calibration system primarily involves linear transformations such as rotation, translation, and scaling. However, some nonlinear transformations, namely perspective transformations, caused by the optical path, camera levelness, or stage flatness, cannot be ignored. Directly designing a global optimizer to iteratively calculate the nine parameters of the perspective transformation matrix would be extremely time-consuming. Therefore, the aforementioned steps first use a local fast iteration method to estimate the six parameters of the affine transformation matrix. Feature point pairs are then selected during the iteration process to further ensure high point set matching and registration accuracy. The feature point set and affine transformation matrix generated in the last iteration are used as input to globally estimate the perspective transformation matrix. The specific method for selecting feature point pairs is to calculate the Manhattan distance between the transformed ODT point pairs and the widefield fluorescence point pairs. This distance is used as the primary evaluation metric to establish an error feedback function, filtering out point pairs with errors greater than a set threshold.

[0081] Step 150: Determine an image registration result of the three-dimensional refractive index data and the wide-field fluorescence image based on the perspective transformation matrix.

[0082] Specifically, after the perspective transformation matrix is ​​obtained, the image registration result of the three-dimensional refractive index data and the wide-field fluorescence image can be determined based on the perspective transformation matrix.

[0083] Based on the obtained perspective transformation matrix, it is applied to the actual sample data collected subsequently to complete the image registration of optical diffraction tomography and wide-field fluorescence imaging. Based on the registration results, the fusion display of the two imaging modality data is realized. Figure 5 This is a fusion effect diagram of the ODT three-dimensional data and multi-channel wide-field fluorescence data of the cell sample provided by the present invention, such as Figure 5 As shown, the ODT image presents a super-resolution panoramic image of the cell sample, and the multi-channel wide-field fluorescence data specifically presents the cell membrane and lipid droplets.

[0084] The method provided by the embodiment of the present invention, on the one hand, realizes image registration specifically for ODT three-dimensional refractive index data and multi-channel wide-field fluorescence data, and thus realizes the image fusion of label-free nonspecific imaging and specific imaging methods of the same sample at the same time. It can not only capture the overall structure and morphological changes of the sample, but also highlight the interactions and processes of specific biological molecules or chemical components, providing new perspectives and clues for scientific research; on the other hand, for microscopic imaging systems with coarse calibration or large errors, it provides an image registration method with lower time cost, higher accuracy and faster time efficiency.

[0085] Based on the above embodiment, step 130 further includes:

[0086] During the iteration process, the ODT feature point set and the wide-field fluorescence feature point set that meet the error requirements are reversely screened.

[0087] Specifically, the detected potential feature points may contain some unreliable points, necessitating further screening. This involves using an adaptive iterative algorithm modified from a clustering algorithm to rapidly estimate the affine transformation matrix. A threshold for the number of ideal feature point pairs is set based on the information density of the calibration image. The affine transformation matrix is ​​then substituted and an error evaluation metric is used to filter out eligible feature point pairs. This process is repeated until the ideal feature point pairs meet the set threshold.

[0088] Among them, the ODT feature point set and wide-field fluorescence feature point set that meet the error requirements are screened in the iterative process, as follows:

[0089] Step s1: determining an initial affine transformation matrix based on the ODT feature point set and the wide-field fluorescence feature point set, and determining a transformed ODT image based on the initial affine transformation matrix and the ODT image to be registered.

[0090] Step s2: Based on the transformed ODT image and wide-field fluorescence image, the Manhattan distance of the transformed corresponding point set is calculated, and the Manhattan distance is used as an evaluation index. Based on the evaluation index, the ODT feature point set and the wide-field fluorescence feature point set with an error greater than a set threshold are screened out;

[0091] Repeat steps s1 and s2 to obtain an ODT feature point set, a wide-field fluorescence feature point set, and an affine transformation matrix that meet the error requirements.

[0092] That is, a feature point matching algorithm is used to match the ODT feature point set and the widefield fluorescence feature point set. An initial affine transformation matrix is ​​calculated based on the matched feature point pairs. The initial affine transformation matrix is ​​applied to the ODT image to be registered to obtain a transformed ODT image. Corresponding feature point pairs are found between the transformed ODT image and the widefield fluorescence image. The Manhattan distance between these corresponding feature point pairs is calculated. An error threshold is set to filter out feature point pairs with a Manhattan distance greater than the set threshold, which means that these feature point pairs are considered to have introduced a large error in the registration process.

[0093] The next iteration uses the filtered feature point set. The affine transformation matrix is ​​recalculated using the updated feature point set and applied to the ODT image to obtain a new transformed ODT image. A new Manhattan distance is calculated, and the feature point set is filtered again. The iteration terminates when the average or maximum Manhattan distance is less than the set error. The resulting ODT feature point set, widefield fluorescence feature point set, and affine transformation matrix are the final results.

[0094] Based on any of the above embodiments, Figure 6 This is the second flow chart of the image registration method for ODT data based on the coarse calibration system provided by the present invention, such as Figure 6 As shown, the method includes:

[0095] Step S1: Select standard fluorescent dyed microspheres as registration samples.

[0096] Step S2: acquiring an ODT image and a wide-field fluorescence image through a multimodal microscopic imaging system, wherein the ODT image includes a field of view area and a focal plane layer of the wide-field fluorescence image.

[0097] Step S3 , filtering out specific layer two-dimensional data corresponding to the focal plane of the wide-field fluorescence imaging data from the ODT three-dimensional image through an algorithm.

[0098] Step S4 , performing pre-processing such as grayscale mapping, binarization, edge detection, and hole filling on the selected specific layer ODT data, so as to make it express and enhance image features similar to wide-field fluorescence.

[0099] Step S5 , extracting feature points from the pre-processed ODT data and wide-field fluorescence imaging data using a scale-space extremum detection algorithm.

[0100] Step S6: using an adaptive iterative algorithm improved by a clustering algorithm to estimate an affine transformation matrix for the feature point pairs.

[0101] Step S7: set a threshold value for the number of ideal feature point pairs according to the information density of the calibration image, substitute it into the affine transformation matrix, and reversely screen out the feature point pairs that meet the conditions by setting an error evaluation index.

[0102] Step S8: Repeat steps S6-S7 until the ideal feature point pair meets the predetermined threshold.

[0103] Step S9: setting the initial value of the perspective transformation matrix based on the affine transformation matrix finally obtained in S8.

[0104] Step S10: design a global optimizer and use a differential optimization method to globally estimate the perspective transformation matrix.

[0105] Step S11 : applying a global estimated perspective transformation matrix to transform the ODT image into a registered image.

[0106] Step S12: Select different channels to fuse the ODT image and the wide-field fluorescence image.

[0107] The image registration system for ODT data based on a coarse calibration system provided by the present invention is described below. The image registration system for ODT data based on a coarse calibration system described below and the image registration method for ODT data based on a coarse calibration system described above can refer to each other.

[0108] Based on any of the above embodiments, the present invention provides an image registration system for ODT data based on a coarse calibration system. Figure 7 Schematic diagram of the structure of the image registration system for ODT data based on the coarse calibration system provided by the present invention. Figure 7 As shown, the system includes:

[0109] An acquisition unit 710 is configured to acquire three-dimensional refractive index data of an ODT image to be registered, and extract a single layer of two-dimensional data that matches the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired at the same time for the same sample, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area;

[0110] A feature point extraction unit 720 is configured to extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image, respectively, to obtain an ODT feature point set and a wide-field fluorescence feature point set;

[0111] A matching unit 730 is configured to match the ODT feature point set and the wide-field fluorescence feature point set based on an adaptive iterative algorithm and iteratively generate an affine transformation matrix;

[0112] A perspective transformation matrix solving unit 740 is configured to use the affine transformation matrix as an initial solution, construct a differential optimizer, input the affine transformation matrix into the differential optimizer for optimization, and obtain a perspective transformation matrix;

[0113] The determining unit 750 is configured to determine an image registration result of the three-dimensional refractive index data and the wide-field fluorescence image based on the perspective transformation matrix.

[0114] The system makes a judgment based on the registration requirements. If an over-registration result is formed, the registration matrix recorded in the computer storage medium only needs to be directly applied to the data, and there is no need to perform the above-mentioned preprocessing, feature extraction, matching, screening and other calibration processes. If no over-registration result is formed, the registration method is executed.

[0115] It also includes the selection and requirements of samples. The samples need standard fluorescent dye microspheres to ensure that the imaging quality meets the requirements.

[0116] The system provided by the embodiments of the present invention, on the one hand, realizes image registration specifically for ODT three-dimensional refractive index data and multi-channel wide-field fluorescence data, and thus realizes image fusion of label-free nonspecific imaging and specific imaging methods of the same sample at the same time. It can not only capture the overall structure and morphological changes of the sample, but also highlight the interactions and processes of specific biological molecules or chemical components, providing new perspectives and clues for scientific research; on the other hand, for microscopic imaging systems with coarse calibration or large errors, it provides an image registration method with lower time cost, higher accuracy and faster time efficiency.

[0117] Based on any of the above embodiments, the fitness function of the differential optimizer includes any one of mean square error and structural similarity.

[0118] Based on any of the above embodiments, the feature point extraction unit 720 is specifically configured to:

[0119] A scale space extreme value detection algorithm is used to extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image, respectively, to obtain the ODT feature point set and the wide-field fluorescence feature point set.

[0120] Based on any of the above embodiments, the further comprising a data acquisition unit, wherein the data acquisition unit is specifically configured to:

[0121] Acquire original single-layer two-dimensional data and original wide-field fluorescence images;

[0122] After sequentially performing image binarization, image inversion, and edge feature extraction on the original single-layer two-dimensional data, holes in the edge information are filled to obtain the single-layer two-dimensional data;

[0123] The original wide-field fluorescence image is subjected to image binarization and image smoothing to obtain the wide-field fluorescence image.

[0124] Based on any of the above embodiments, the invention further includes a reverse screening unit, wherein the reverse screening unit is specifically used to:

[0125] The anti-screening unit is used to anti-screen the ODT feature point set and the wide-field fluorescence feature point set that meet the error requirements during the iterative process.

[0126] Based on any of the above embodiments, the anti-sieve unit is specifically used for:

[0127] Step s1: determining an initial affine transformation matrix based on the ODT feature point set and the widefield fluorescence feature point set, and determining a transformed ODT image based on the initial affine transformation matrix and the ODT image to be registered;

[0128] Step s2: Based on the transformed ODT image and the wide-field fluorescence image, calculating the Manhattan distance of the transformed corresponding point sets, then using the Manhattan distance as an evaluation index, and filtering out ODT feature point sets and wide-field fluorescence feature point sets whose errors are greater than a set threshold based on the evaluation index;

[0129] Repeat steps s1 and s2 to obtain the ODT feature point set, the wide-field fluorescence feature point set, and the affine transformation matrix that meet the error requirements.

[0130] Figure 8 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 8As shown, the electronic device may include: a processor 810 , a communication interface 820 , a memory 830 and a communication bus 840 , wherein the processor 810 , the communication interface 820 and the memory 830 communicate with each other via the communication bus 840 . The processor 810 can call logic instructions in the memory 830 to execute an image registration method for ODT data based on the coarse calibration system, the method including: obtaining three-dimensional refractive index data of the ODT image to be registered, and extracting a single layer of two-dimensional data that matches the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired at the same time from the same sample, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area; feature point extraction is performed on the single layer of two-dimensional data and the wide-field fluorescence image respectively to obtain an ODT feature point set and a wide-field fluorescence feature point set; based on an adaptive iterative algorithm improved by a clustering algorithm, the ODT feature point set and the wide-field fluorescence feature point set are matched and an affine transformation matrix is ​​iteratively generated; the affine transformation matrix is ​​used as an initial solution, and a differential optimizer is constructed, and the affine transformation matrix is ​​input into the differential optimizer for optimization to obtain a perspective transformation matrix; based on the perspective transformation matrix, the image registration result of the three-dimensional refractive index data and the wide-field fluorescence image is determined.

[0131] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0132] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image registration method for ODT data based on a coarse calibration system provided by the above methods, the method comprising: obtaining three-dimensional refractive index data of the ODT image to be registered, and extracting a single layer of two-dimensional data that matches the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are obtained from the same sample at the same time, and the ODT image to be registered is obtained from the same sample at the same time. The image and the wide-field fluorescence image have an overlapping area; feature points are extracted from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain an ODT feature point set and a wide-field fluorescence feature point set; an adaptive iterative algorithm improved based on a clustering algorithm is used to match the ODT feature point set and the wide-field fluorescence feature point set and iteratively generate an affine transformation matrix; the affine transformation matrix is ​​used as an initial solution, and a differential optimizer is constructed, and the affine transformation matrix is ​​input into the differential optimizer for optimization to obtain a perspective transformation matrix; based on the perspective transformation matrix, the image registration result of the three-dimensional refractive index data and the wide-field fluorescence image is determined.

[0133] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the image registration method for ODT data based on a coarse calibration system provided by the above methods, the method comprising: acquiring three-dimensional refractive index data of the ODT image to be registered, and extracting a single layer of two-dimensional data matching the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired from the same sample at the same time, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area. domain; extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain an ODT feature point set and a wide-field fluorescence feature point set; based on an adaptive iterative algorithm improved by a clustering algorithm, match the ODT feature point set and the wide-field fluorescence feature point set and iteratively generate an affine transformation matrix; use the affine transformation matrix as an initial solution, construct a differential optimizer, input the affine transformation matrix into the differential optimizer for optimization, and obtain a perspective transformation matrix; based on the perspective transformation matrix, determine the image registration result of the three-dimensional refractive index data and the wide-field fluorescence image.

[0134] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0135] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An image registration method for ODT data based on a coarse calibration system, characterized in that: include: Acquiring three-dimensional refractive index data of an ODT image to be registered, and extracting single-layer two-dimensional data that matches the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired at the same time for the same sample, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area; Extracting feature points from the single-layer two-dimensional data and the wide-field fluorescence image to obtain an ODT feature point set and a wide-field fluorescence feature point set; Based on an adaptive iterative algorithm improved by a clustering algorithm, the ODT feature point set and the wide-field fluorescence feature point set are matched and an affine transformation matrix is ​​iteratively generated; Taking the affine transformation matrix as an initial solution, constructing a differential optimizer, inputting the affine transformation matrix into the differential optimizer for optimization, and obtaining a perspective transformation matrix; An image registration result of the three-dimensional refractive index data and the wide-field fluorescence image is determined based on the perspective transformation matrix.

2. The image registration method for ODT data based on a coarse calibration system according to claim 1, characterized in that: The fitness function of the differential optimizer includes any one of mean square error and structural similarity.

3. The image registration method for ODT data based on a coarse calibration system according to claim 1, characterized in that: The extracting feature points from the single-layer two-dimensional data and the wide-field fluorescence image to obtain an ODT feature point set and a wide-field fluorescence feature point set includes: A scale space extreme value detection algorithm is used to extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image, respectively, to obtain the ODT feature point set and the wide-field fluorescence feature point set.

4. The image registration method for ODT data based on a coarse calibration system according to any one of claims 1 to 3, characterized in that: The step of acquiring the single-layer two-dimensional data and the wide-field fluorescence image comprises: Acquire original single-layer two-dimensional data and original wide-field fluorescence images; After sequentially performing image binarization, image inversion, and edge feature extraction on the original single-layer two-dimensional data, holes in the edge information are filled to obtain the single-layer two-dimensional data; The original wide-field fluorescence image is subjected to image binarization and image smoothing to obtain the wide-field fluorescence image.

5. The image registration method for ODT data based on a coarse calibration system according to any one of claims 1 to 3, characterized in that: The adaptive iterative algorithm improved based on the clustering algorithm matches the ODT feature point set and the wide-field fluorescence feature point set and iteratively generates an affine transformation matrix, and further includes: During the iteration process, the ODT feature point set and the wide-field fluorescence feature point set that meet the error requirements are reversely screened.

6. The image registration method for ODT data based on a coarse calibration system according to claim 5, characterized in that: In the iterative process, the ODT feature point set and the wide-field fluorescence feature point set that meet the error requirements are reversely screened, including: Step s1: determining an initial affine transformation matrix based on the ODT feature point set and the widefield fluorescence feature point set, and determining a transformed ODT image based on the initial affine transformation matrix and the ODT image to be registered; Step s2: Based on the transformed ODT image and the wide-field fluorescence image, calculating the Manhattan distance of the transformed corresponding point sets, then using the Manhattan distance as an evaluation index, and filtering out ODT feature point sets and wide-field fluorescence feature point sets whose errors are greater than a set threshold based on the evaluation index; Repeat steps s1 and s2 to obtain the ODT feature point set, the wide-field fluorescence feature point set, and the affine transformation matrix that meet the error requirements.

7. An image registration system for ODT data based on a coarse calibration system, characterized in that: include: an acquisition unit, configured to acquire three-dimensional refractive index data of an ODT image to be registered, and extract a single layer of two-dimensional data that matches the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired at the same time for the same sample, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area; a feature point extraction unit, configured to extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image, respectively, to obtain an ODT feature point set and a wide-field fluorescence feature point set; A matching unit, configured to match the ODT feature point set and the wide-field fluorescence feature point set based on an adaptive iterative algorithm and iteratively generate an affine transformation matrix; A perspective transformation matrix solving unit is used to use the affine transformation matrix as an initial solution, construct a differential optimizer, input the affine transformation matrix into the differential optimizer for optimization, and obtain a perspective transformation matrix; A determining unit is configured to determine an image registration result of the three-dimensional refractive index data and the wide-field fluorescence image based on the perspective transformation matrix.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image registration method for ODT data based on a coarse calibration system according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image registration method for ODT data based on a coarse calibration system according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image registration method for ODT data based on a coarse calibration system according to any one of claims 1 to 6 is implemented.

Images