Image registration method, device and equipment for ODT data based on coarse calibration system

Abstract

The invention provides an image registration method, device and equipment for ODT data based on a coarse calibration system. The method comprises the following steps: extracting single-layer two-dimensional data matched with a wide-field fluorescence image from three-dimensional refractive index data; performing feature point extraction on the single-layer two-dimensional data and the wide-field fluorescence image to obtain an ODT feature point set and a wide-field fluorescence feature point set; based on an adaptive iterative algorithm improved by a clustering algorithm, matching and iterating the ODT feature point set and the wide-field fluorescence feature point set to generate an affine transformation matrix; the affine transformation matrix serves as an initial solution, a difference optimizer is constructed, the affine transformation matrix is input into the difference optimizer to be optimized, and a perspective transformation matrix is obtained; and determining an image registration result based on the perspective transformation matrix. The method is specially used for realizing image registration for ODT three-dimensional refractive index data and multi-channel wide-field fluorescence data so as to realize image fusion of unmarked non-specific imaging and specific imaging modes of the same sample at the same moment.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image registration method, device and equipment for ODT data based on a coarse calibration system. Background Art

[0002] Optical Diffraction Tomography (ODT) is a technology used for three-dimensional microscopic imaging. It changes the incident angle by using a laser in conjunction with a scanning galvanometer to obtain data from multiple angles of the sample, and reconstructs the three-dimensional refractive index distribution of the sample based on the reconstruction algorithm. It has the advantages of being non-invasive and high-resolution. Wide-field fluorescence imaging is a technology that uses fluorescent markers and wide-angle illumination to obtain sample images. In order to achieve the complementarity of label-free nonspecific imaging and specific imaging methods, so as to achieve comprehensive, holistic, and dynamic observation and research of samples, it is necessary to integrate the two imaging technologies into a multimodal microscopic imaging system, and realize image fusion of multimodal imaging of the same sample in the same state through image registration methods.

[0003] For the needs of traditional natural image registration, the following methods are usually used for registration: extract feature points using a scale-invariant algorithm, then use brute force matching or k-nearest neighbor algorithm for feature matching, select transformation models based on possible transformations, and finally use a random sampling consistency algorithm for parameter estimation. There are two difficulties in using general registration methods to register ODT data: First, due to its computational imaging characteristics, ODT data reflects the three-dimensional refractive index information of the sample rather than the intensity or grayscale information, and due to the specific noise and errors of computational imaging, it is almost impossible to produce correct results when extracting feature points; second, even if the data is preprocessed as much as possible into a binary grayscale image with similar image features to wide-field fluorescence for registration, its registration accuracy is low, the registration effect is poor, and even leads to erroneous information that affects judgment and decision-making. To avoid this problem, it is necessary not only to adjust the hardware system to a precise calibration or even ideal state, but also to use a very time-consuming global optimal brute force registration algorithm, which increases a lot of economic and time costs. Summary of the invention

[0004] The present invention provides an image registration method, device and equipment for ODT data based on a coarse calibration system, which is used to solve the defect that the prior art cannot process the ODT data registration requirements, while ensuring high accuracy and high processing efficiency to achieve ODT data image registration and fusion.

[0005] The present invention provides an image registration method for ODT data based on a coarse calibration system, comprising the following steps: Acquire three-dimensional refractive index data of the ODT image to be registered, and extract a single layer of two-dimensional data matching the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired at the same time of the same sample, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area; Extracting feature points from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain an ODT feature point set and a wide-field fluorescence feature point set; Based on an adaptive iterative algorithm improved by a clustering algorithm, the ODT feature point set and the wide-field fluorescence feature point set are matched and an affine transformation matrix is ​​iteratively generated; The affine transformation matrix is ​​used as an initial solution, and a differential optimizer is constructed, and the affine transformation matrix is ​​input into the differential optimizer for optimization to obtain a perspective transformation matrix; Based on the perspective transformation matrix, an image registration result of the three-dimensional refractive index data and the wide-field fluorescence image is determined.

[0006] According to an image registration method for ODT data based on a coarse calibration system provided by the present invention, the fitness function of the differential optimizer includes any one of mean square error and structural similarity.

[0007] According to an image registration method for ODT data based on a coarse calibration system provided by the present invention, feature points are extracted from the single-layer two-dimensional data and the wide-field fluorescence image to obtain an ODT feature point set and a wide-field fluorescence feature point set, including: Using a scale space extremum detection algorithm, feature points are extracted from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain the ODT feature point set and the wide-field fluorescence feature point set.

[0008] According to an image registration method for ODT data based on a coarse calibration system provided by the present invention, the steps of acquiring the single-layer two-dimensional data and the wide-field fluorescence image include: Acquire original single-layer 2D data and original wide-field fluorescence images; After sequentially performing image binarization, image inversion and edge feature extraction on the original single-layer two-dimensional data, holes are filled in the edge information to obtain the single-layer two-dimensional data; The original wide-field fluorescence image is subjected to image binarization and image smoothing to obtain the wide-field fluorescence image.

[0009] According to an image registration method for ODT data based on a coarse calibration system provided by the present invention, the adaptive iterative algorithm improved based on the clustering algorithm matches the ODT feature point set and the wide-field fluorescence feature point set and iteratively generates an affine transformation matrix, and further includes: In the iterative process, the ODT feature point set and the wide-field fluorescence feature point set that meet the error requirements are reversely screened.

[0010] According to an image registration method for ODT data based on a coarse calibration system provided by the present invention, the ODT feature point set and the wide-field fluorescence feature point set that meet the error requirements are reversely screened during the iteration process, including: Step s1: determining an initial affine transformation matrix based on the ODT feature point set and the wide-field fluorescence feature point set, and determining a transformed ODT image based on the initial affine transformation matrix and the ODT image to be registered; Step s2: based on the transformed ODT image and the wide-field fluorescence image, calculating the Manhattan distance of the transformed corresponding point set, then using the Manhattan distance as an evaluation index, and filtering out the ODT feature point set and the wide-field fluorescence feature point set whose errors are greater than a set threshold based on the evaluation index; Step s1 and step s2 are iterated repeatedly to obtain the ODT feature point set, the wide-field fluorescence feature point set and the affine transformation matrix that meet the error requirements.

[0011] The present invention also provides an image registration system for ODT data based on a coarse calibration system, comprising the following modules: An acquisition unit, used for acquiring three-dimensional refractive index data of the ODT image to be registered, and extracting a single layer of two-dimensional data matching the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired at the same time of the same sample, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area; A feature point extraction unit, used to extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image, respectively, to obtain an ODT feature point set and a wide-field fluorescence feature point set; A matching unit, used for matching the ODT feature point set and the wide-field fluorescence feature point set based on an adaptive iterative algorithm and iteratively generating an affine transformation matrix; A perspective transformation matrix solving unit, used to take the affine transformation matrix as an initial solution, construct a differential optimizer, input the affine transformation matrix into the differential optimizer for optimization, and obtain a perspective transformation matrix; A determination unit is used to determine an image registration result of the three-dimensional refractive index data and the wide-field fluorescence image based on the perspective transformation matrix.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an image registration method for ODT data based on a coarse calibration system as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image registration method for ODT data based on a coarse calibration system as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image registration method for ODT data based on a coarse calibration system as described in any one of the above is implemented.

[0015] The image registration method, device and equipment for ODT data based on a coarse calibration system provided by the present invention, on the one hand, realize image registration specifically for ODT three-dimensional refractive index data and multi-channel wide-field fluorescence data, and then realize the image fusion of label-free nonspecific imaging and specific imaging methods of the same sample at the same time, which can not only capture the overall structure and morphological changes of the sample, but also highlight the interactions and processes of specific biological molecules or chemical components, providing new perspectives and clues for scientific research; on the other hand, for coarse calibration or microscopic imaging systems with large errors, an image registration method with lower time cost, higher accuracy and faster time efficiency is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is one of the flow charts of the image registration method for ODT data based on the coarse calibration system provided by the present invention.

[0018] Figure 2 It is a schematic diagram of the three-dimensional refractive index volume image of the fluorescent microsphere sample ODT provided by the present invention.

[0019] Figure 3 It is a schematic diagram of a wide-field fluorescence image of a fluorescent microsphere sample provided by the present invention.

[0020] Figure 4 Schematic diagram of the pre-processed ODT image provided by the present invention.

[0021] Figure 5 It is a fusion effect diagram of ODT three-dimensional data and multi-channel wide-field fluorescence data of the cell sample provided by the present invention.

[0022] Figure 6 This is the second flow chart of the image registration method for ODT data based on the coarse calibration system provided by the present invention.

[0023] Figure 7 It is a structural schematic diagram of an image registration system for ODT data based on a coarse calibration system provided by the present invention.

[0024] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Figure 1 FIG. 1 is one of the flow charts of the image registration method for ODT data based on the rough calibration system provided by the present invention, such as Figure 1 As shown, the method includes step 110, step 120, step 130, step 140 and step 150.

[0027] Step 110, obtaining three-dimensional refractive index data of the ODT image to be registered, and extracting a single layer of two-dimensional data matching the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired from the same sample at the same time, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area.

[0028] Specifically, the ODT image and wide-field fluorescence image to be registered are acquired by the multimodal microscopy system at the same time for the same sample, wherein the system state of the multimodal microscopy system is coarse calibration, including system errors such as different illumination angles of the two modal light sources, tilted level of the sample or the translation stage, and insufficient resetting accuracy of the mirror frame turntable.

[0029] Here, the multimodal microscopic imaging system includes a laser, a translation stage, two cameras, a microscope stand, a scanning galvanometer, an electric aperture, a fluorescent light box and optical elements; wherein, the microscope stand is installed on the bottom surface, and the remaining hardware equipment is installed on the microscope stand; the system uses a specific optical path to coordinate hardware imaging, wherein the ODT data is derived from the data collected by the camera and is obtained by calculating the imaging through the solution module.

[0030] Accordingly, standard fluorescent dyed microspheres should be prepared as registration samples first, because the imaging of standard microspheres usually has more obvious image features with clear edges, and for the specificity of wide-field fluorescence images, specific microspheres of a specific wavelength band can all be stimulated to emit light. Then, the multimodal microscopic imaging system is used to obtain ODT images and wide-field fluorescence images of microspheres with different field sizes but overlapping areas at the same time.

[0031] Figure 2 is a schematic diagram of the three-dimensional refractive index volume image of the fluorescent microsphere sample ODT provided by the present invention, Figure 3 Schematic diagram of the wide-field fluorescence image of the fluorescent microsphere sample provided by the present invention, such as Figure 2 and Figure 3 As shown. Figure 2 The three-dimensional refractive index volume data of the ODT image to be registered shown reflects the refractive index volume data of the sample within a certain z-axis range. The characteristics of different layers of the image, such as grayscale, gradient, edge information, and noise, vary greatly. It is necessary to use an algorithm to calculate and screen out the specific layer that truly corresponds to the focal plane of the wide-field fluorescence imaging data, that is, the actual height of the z-axis. The specific steps are: establish ODT image evaluation indicators; calculate evaluation functions; screen out several required optimal layers; remove local optimal solutions to obtain the final required ODT image layer corresponding to the focal plane of the wide-field fluorescence data.

[0032] For the screened-out ODT data of a specific layer, due to its computational imaging characteristics, the data reflects the sample refractive index information rather than intensity or grayscale information, and due to the specific noise and errors of computational imaging, it is almost impossible to produce correct results when extracting feature points. Figure 4 is a schematic diagram of the pre-processed ODT image provided by the present invention, such as Figure 4 As shown, a series of preprocessing methods can be performed on a specific layer of ODT data to make it show and enhance image features similar to wide-field fluorescent microspheres. For example, for the original single-layer two-dimensional data screened out, an algorithm is first designed to map its refractive index to grayscale space, and then image binarization, image inversion and edge feature extraction are performed in sequence, and then the edge information is filled with holes to obtain single-layer two-dimensional data. The original wide-field fluorescence image is subjected to image binarization and image smoothing to obtain a wide-field fluorescence image.

[0033] Grayscale mapping of the image, that is, mapping the original refractive index image value from the floating point range of about -2.000000 to 1.7000000 to integer values ​​​​from 0 to 255, and at the same time, according to the overall distribution of the refractive index value, appropriately truncating the upper and lower limits of 3% to 5% of the value, in order to make the mapped image show a contrast that is easier to match The binarization of an image is to set the grayscale value of the pixels on the image to 0 or 255, so that the entire image presents a clear visual effect of only black and white. Binarization is the simplest method of image segmentation, which can convert a grayscale image into a binary image.

[0034] Image inversion means reversing the color value of each pixel in the image. In a grayscale color space, this usually means subtracting the original value from the grayscale value of each pixel from 255.

[0035] Edge feature extraction refers to extracting pixels or regions with edge features from an image. Edges usually represent the boundaries of objects or discontinuously changing parts, and are of great significance for tasks such as image segmentation, recognition, and reconstruction.

[0036] The purpose of filling holes is usually to improve the visual effect of the image and increase the accuracy of subsequent image analysis or recognition.

[0037] Image smoothing is actually a low-pass filter, which makes the image grayscale smoother, reduces the sudden gradient, and thus improves the image quality.

[0038] Step 120 , extracting feature points from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain an ODT feature point set and a wide-field fluorescence feature point set.

[0039] Specifically, feature points are extracted from the single-layer two-dimensional data and the wide-field fluorescence image, respectively, to obtain an ODT feature point set and a wide-field fluorescence feature point set.

[0040] Here, a scale space extremum detection algorithm can be used to extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image, respectively, to obtain an ODT feature point set and a wide-field fluorescence feature point set.

[0041] In order to detect feature points at different scales, we first need to construct a scale space for the image, which is done by performing a Gaussian blur operation on the image and repeating it at different scales. Each blurred image represents a different scale version of the image, allowing the algorithm to detect features of different sizes. In the constructed scale space, extreme points are detected by comparing neighboring pixels. Specifically, each pixel is compared with its pixels at adjacent scales and adjacent spatial locations. If a point appears as a local maximum (or minimum) in all comparisons, it is considered to be a potential feature point.

[0042] Step 130 : Based on the adaptive iterative algorithm improved by the clustering algorithm, the ODT feature point set and the wide-field fluorescence feature point set are matched and an affine transformation matrix is ​​iteratively generated.

[0043] Specifically, considering that the detected potential feature points may still contain some unreliable points, further screening is required. An adaptive iterative algorithm improved based on a clustering algorithm can be used to match the ODT feature point set and the wide-field fluorescence feature point set and iteratively generate an affine transformation matrix.

[0044] Step 140, taking the last generated affine transformation matrix as the initial solution, and constructing a differential optimizer, inputting the affine transformation matrix into the differential optimizer for optimization, and obtaining a perspective transformation matrix.

[0045] Specifically, the affine transformation matrix is ​​used as the initial solution, the parameters and their estimation intervals are simplified, and the differential optimization method is used to globally estimate the perspective transformation matrix to compensate for possible perspective transformation and ensure the accuracy and fast convergence of the registration results.

[0046] That is, the affine transformation matrix is ​​input into the differential optimizer for optimization to obtain the perspective transformation matrix.

[0047] Here, the fitness function of the differential optimizer includes any one of Mean Squared Error (MSE) and Structural Similarity (SSIM).

[0048] Here, the affine transformation matrix generated by the last iteration is taken as the initial solution for estimating the perspective transformation matrix. The affine transformation matrix is: in, is the original image coordinate, , are the transformed coordinates; is the rotation amount, , , is the translation amount.

[0049] The perspective transformation matrix is: in, is the original image coordinate, , are the transformed coordinates; is the rotation amount, is the translation amount.

[0050] To ensure the accuracy of registration, since the rough calibration system mainly has linear transformations such as rotation, translation, and scaling, but some nonlinear transformations caused by the optical path, camera levelness, or stage flatness, namely perspective transformation, cannot be ignored. It would be very time-consuming to directly design a global optimizer to iteratively calculate the 9 parameters of the perspective transformation matrix. Therefore, the above steps first use the local fast iteration method to estimate the 6 parameters of the affine transformation matrix, and select feature point pairs during the iteration process to further ensure the high matching degree of the point set and the accuracy of registration. The feature point set and affine transformation matrix generated by the last iteration are used as input to globally estimate the perspective transformation matrix. The specific method for selecting feature point pairs is to calculate the Manhattan distance between the transformed ODT point pairs and the wide-field fluorescence point pairs, and use this as the main evaluation index to establish an error feedback function to select and remove point pairs whose errors are greater than the set threshold.

[0051] Step 150: Determine an image registration result of the three-dimensional refractive index data and the wide-field fluorescence image based on the perspective transformation matrix.

[0052] Specifically, after the perspective transformation matrix is ​​obtained, the image registration result of the three-dimensional refractive index data and the wide-field fluorescence image can be determined based on the perspective transformation matrix.

[0053] Based on the obtained perspective transformation matrix, it is applied to the actual sample data collected subsequently to complete the image registration of optical diffraction tomography and wide-field fluorescence imaging. Based on the registration results, the fusion display of the two imaging modality data is realized. Figure 5 It is a fusion effect diagram of the ODT three-dimensional data and multi-channel wide-field fluorescence data of the cell sample provided by the present invention, such as Figure 5 As shown, the ODT image presents a super-resolution panoramic image of the cell sample, and the multi-channel wide-field fluorescence data specifically presents the cell membrane and lipid droplets.

[0054] The method provided in the embodiment of the present invention, on the one hand, realizes image registration specifically for ODT three-dimensional refractive index data and multi-channel wide-field fluorescence data, and further realizes image fusion of label-free nonspecific imaging and specific imaging methods of the same sample at the same time, which can not only capture the overall structure and morphological changes of the sample, but also highlight the interactions and processes of specific biological molecules or chemical components, providing new perspectives and clues for scientific research; on the other hand, for microscopic imaging systems with rough calibration or large errors, it provides an image registration method with lower time cost, higher accuracy and faster time efficiency.

[0055] Based on the above embodiment, step 130 further includes: In the iterative process, the ODT feature point set and the wide-field fluorescence feature point set that meet the error requirements are reversely screened.

[0056] Specifically, the detected potential feature points may still contain some unreliable points, so further screening is required, and the adaptive iterative algorithm improved by the clustering algorithm is used to quickly estimate the affine transformation matrix. The threshold of the number of ideal feature point pairs is set according to the information density of the calibration image, and the affine transformation matrix is ​​substituted and the feature point pairs that meet the conditions are screened out by setting the error evaluation index. Repeat the above steps until the ideal feature point pairs meet the set threshold.

[0057] Among them, the ODT feature point set and wide-field fluorescence feature point set that meet the error requirements are screened in the iterative process, as follows: Step s1: determining an initial affine transformation matrix based on the ODT feature point set and the wide-field fluorescence feature point set, and determining a transformed ODT image based on the initial affine transformation matrix and the ODT image to be registered.

[0058] Step s2: based on the transformed ODT image and the wide-field fluorescence image, the Manhattan distance of the transformed corresponding point set is calculated, and the Manhattan distance is used as an evaluation index, and the ODT feature point set and the wide-field fluorescence feature point set whose errors are greater than a set threshold are screened out based on the evaluation index; Step s1 and step s2 are iterated repeatedly to obtain an ODT feature point set, a wide-field fluorescence feature point set and an affine transformation matrix that meet the error requirements.

[0059] That is, a feature point matching algorithm is used to match the ODT feature point set and the wide-field fluorescence feature point set. The initial affine transformation matrix is ​​calculated based on the matched feature point pairs. The initial affine transformation matrix is ​​applied to the ODT image to be registered to obtain the transformed ODT image. The corresponding feature point pairs are found between the transformed ODT image and the wide-field fluorescence image. The Manhattan distance between these corresponding feature point pairs is calculated. An error threshold is set to filter out feature point pairs whose Manhattan distance is greater than the set threshold, that is, it is considered that these feature point pairs introduce large errors in the registration process.

[0060] Use the filtered feature point set for the next iteration, use the updated feature point set to recalculate the affine transformation matrix, apply the new affine transformation matrix to the ODT image, and obtain a new transformed ODT image. Calculate the new Manhattan distance and filter the feature point set again. When the average or maximum value of the Manhattan distance is less than the set error requirement, the iteration terminates. The ODT feature point set, wide-field fluorescence feature point set, and affine transformation matrix obtained at this time are the final results.

[0061] Based on any of the above embodiments, Figure 6 FIG. 2 is a flow chart of the image registration method for ODT data based on the rough calibration system provided by the present invention. Figure 6 As shown, the method includes: Step S1, selecting standard fluorescent dyed microspheres as registration samples.

[0062] Step S2, acquiring an ODT image and a wide-field fluorescence image by a multimodal microscopic imaging system, wherein the ODT image includes a field of view area and a focal plane layer of the wide-field fluorescence image.

[0063] Step S3, filtering out the specific layer two-dimensional data corresponding to the focal plane of the wide-field fluorescence imaging data from the ODT three-dimensional image through an algorithm.

[0064] Step S4, pre-processing such as grayscale mapping, binarization, edge detection, and hole filling is performed on the screened specific layer ODT data to make it show and enhance image features similar to wide-field fluorescence.

[0065] Step S5: extracting feature points from the pre-processed ODT data and wide-field fluorescence imaging data using a scale space extremum detection algorithm.

[0066] Step S6: using an adaptive iterative algorithm improved by a clustering algorithm to estimate an affine transformation matrix for the feature point pairs.

[0067] Step S7, setting a threshold value for the number of ideal feature point pairs according to the information density of the calibration image, substituting into the affine transformation matrix and inversely screening out feature point pairs that meet the conditions by setting an error evaluation index.

[0068] Step S8, repeating steps S6-S7 until the ideal feature point pair meets a predetermined threshold.

[0069] Step S9, setting the initial value of the perspective transformation matrix based on the affine transformation matrix finally obtained in S8.

[0070] Step S10, designing a global optimizer, and using a differential optimization method to globally estimate the perspective transformation matrix.

[0071] Step S11 , applying a global estimated perspective transformation matrix to transform the ODT image into a registered image.

[0072] Step S12, selecting different channels to fuse the ODT image and the wide-field fluorescence image.

[0073] The image registration system for ODT data based on a coarse calibration system provided by the present invention is described below. The image registration system for ODT data based on a coarse calibration system described below and the image registration method for ODT data based on a coarse calibration system described above can refer to each other.

[0074] Based on any of the above embodiments, the present invention provides an image registration system for ODT data based on a coarse calibration system. Figure 7 is a schematic diagram of the structure of the image registration system for ODT data based on the coarse calibration system provided by the present invention, such as Figure 7 As shown, the system includes: An acquisition unit 710 is used to acquire three-dimensional refractive index data of an ODT image to be registered, and extract a single layer of two-dimensional data matching the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired at the same time of the same sample, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area; A feature point extraction unit 720 is used to extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image, respectively, to obtain an ODT feature point set and a wide-field fluorescence feature point set; A matching unit 730, configured to match the ODT feature point set and the wide-field fluorescence feature point set based on an adaptive iterative algorithm and iteratively generate an affine transformation matrix; A perspective transformation matrix solving unit 740 is used to use the affine transformation matrix as an initial solution, construct a differential optimizer, input the affine transformation matrix into the differential optimizer for optimization, and obtain a perspective transformation matrix; The determination unit 750 is configured to determine an image registration result of the three-dimensional refractive index data and the wide-field fluorescence image based on the perspective transformation matrix.

[0075] The system makes a judgment based on the registration requirements. If an over-registration result is formed, the registration matrix recorded in the computer storage medium only needs to be directly applied to the data. There is no need to perform the above-mentioned preprocessing, feature extraction, matching, screening and other calibration processes. If no over-registration result is formed, the registration method is executed.

[0076] It also includes the selection and requirements of samples. The samples need standard fluorescent dyed microspheres to ensure that the imaging quality meets the requirements.

[0077] The system provided by the embodiment of the present invention, on the one hand, realizes image registration specifically for ODT three-dimensional refractive index data and multi-channel wide-field fluorescence data, and further realizes image fusion of label-free nonspecific imaging and specific imaging methods of the same sample at the same time, which can not only capture the overall structure and morphological changes of the sample, but also highlight the interactions and processes of specific biological molecules or chemical components, providing new perspectives and clues for scientific research; on the other hand, for microscopic imaging systems with rough calibration or large errors, it provides an image registration method with lower time cost, higher accuracy and faster time efficiency.

[0078] Based on any of the above embodiments, the fitness function of the differential optimizer includes any one of mean square error and structural similarity.

[0079] Based on any of the above embodiments, the feature point extraction unit 720 is specifically used for: Using a scale space extremum detection algorithm, feature points are extracted from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain the ODT feature point set and the wide-field fluorescence feature point set.

[0080] Based on any of the above embodiments, it further includes a data acquisition unit, wherein the data acquisition unit is specifically used to: Acquire original single-layer 2D data and original wide-field fluorescence images; After sequentially performing image binarization, image inversion and edge feature extraction on the original single-layer two-dimensional data, holes are filled in the edge information to obtain the single-layer two-dimensional data; The original wide-field fluorescence image is subjected to image binarization and image smoothing to obtain the wide-field fluorescence image.

[0081] Based on any of the above embodiments, it further includes a reverse screening unit, wherein the reverse screening unit is specifically used for: The anti-screening unit is used to anti-screen the ODT feature point set and the wide-field fluorescence feature point set that meet the error requirements during the iteration process.

[0082] Based on any of the above embodiments, the anti-sieve unit is specifically used for: Step s1: determining an initial affine transformation matrix based on the ODT feature point set and the wide-field fluorescence feature point set, and determining a transformed ODT image based on the initial affine transformation matrix and the ODT image to be registered; Step s2: based on the transformed ODT image and the wide-field fluorescence image, calculating the Manhattan distance of the transformed corresponding point set, then using the Manhattan distance as an evaluation index, and filtering out the ODT feature point set and the wide-field fluorescence feature point set whose errors are greater than a set threshold based on the evaluation index; Step s1 and step s2 are iterated repeatedly to obtain the ODT feature point set, the wide-field fluorescence feature point set and the affine transformation matrix that meet the error requirements.

[0083] Figure 8 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 8 As shown, the electronic device may include: a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the image registration method for ODT data based on the rough calibration system, the method comprising: obtaining three-dimensional refractive index data of the ODT image to be registered, and extracting a single-layer two-dimensional data matching the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are obtained from the same sample at the same time, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area; feature points are extracted from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain an ODT feature point set and a wide-field fluorescence feature point set; based on an adaptive iterative algorithm improved by a clustering algorithm, the ODT feature point set and the wide-field fluorescence feature point set are matched and an affine transformation matrix is ​​iteratively generated; the affine transformation matrix is ​​used as an initial solution, and a differential optimizer is constructed, and the affine transformation matrix is ​​input into the differential optimizer for optimization to obtain a perspective transformation matrix; based on the perspective transformation matrix, the image registration result of the three-dimensional refractive index data and the wide-field fluorescence image is determined.

[0084] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0085] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image registration method for ODT data based on a coarse calibration system provided by the above methods, the method comprising: obtaining three-dimensional refractive index data of the ODT image to be registered, and extracting a single layer of two-dimensional data matching the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are obtained from the same sample at the same time, and the ODT image to be registered is The image and the wide-field fluorescence image have an overlapping area; feature points are extracted from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain an ODT feature point set and a wide-field fluorescence feature point set; based on an adaptive iterative algorithm improved by a clustering algorithm, the ODT feature point set and the wide-field fluorescence feature point set are matched and an affine transformation matrix is ​​iteratively generated; the affine transformation matrix is ​​used as an initial solution, and a differential optimizer is constructed, and the affine transformation matrix is ​​input into the differential optimizer for optimization to obtain a perspective transformation matrix; based on the perspective transformation matrix, the image registration result of the three-dimensional refractive index data and the wide-field fluorescence image is determined.

[0086] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for image registration based on a rough calibration system for ODT data provided by the above methods is implemented, the method comprising: acquiring three-dimensional refractive index data of the ODT image to be registered, and extracting a single layer of two-dimensional data matching the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired from the same sample at the same time, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area. domain; extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain an ODT feature point set and a wide-field fluorescence feature point set; based on an adaptive iterative algorithm improved by a clustering algorithm, match the ODT feature point set and the wide-field fluorescence feature point set and iteratively generate an affine transformation matrix; use the affine transformation matrix as an initial solution, construct a differential optimizer, input the affine transformation matrix into the differential optimizer for optimization, and obtain a perspective transformation matrix; based on the perspective transformation matrix, determine the image registration result of the three-dimensional refractive index data and the wide-field fluorescence image.

[0087] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image registration method for ODT data based on a coarse calibration system, characterized in that: include: Acquire three-dimensional refractive index data of the ODT image to be registered, and extract a single layer of two-dimensional data matching the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired at the same time of the same sample, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area; Extracting feature points from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain an ODT feature point set and a wide-field fluorescence feature point set; Based on an adaptive iterative algorithm improved by a clustering algorithm, the ODT feature point set and the wide-field fluorescence feature point set are matched and an affine transformation matrix is ​​iteratively generated; The affine transformation matrix is ​​used as an initial solution, and a differential optimizer is constructed, and the affine transformation matrix is ​​input into the differential optimizer for optimization to obtain a perspective transformation matrix; Based on the perspective transformation matrix, an image registration result of the three-dimensional refractive index data and the wide-field fluorescence image is determined.

2. The image registration method for ODT data based on a coarse calibration system according to claim 1, characterized in that: The fitness function of the differential optimizer includes any one of mean square error and structural similarity.

3. The image registration method for ODT data based on a coarse calibration system according to claim 1, characterized in that: The extracting feature points of the single-layer two-dimensional data and the wide-field fluorescence image to obtain an ODT feature point set and a wide-field fluorescence feature point set includes: Using a scale space extremum detection algorithm, feature points are extracted from the single-layer two-dimensional data and the wide-field fluorescence image respectively to obtain the ODT feature point set and the wide-field fluorescence feature point set.

4. The image registration method for ODT data based on a coarse calibration system according to any one of claims 1 to 3, characterized in that: The step of acquiring the single-layer two-dimensional data and the wide-field fluorescence image comprises: Acquire original single-layer 2D data and original wide-field fluorescence images; After sequentially performing image binarization, image inversion and edge feature extraction on the original single-layer two-dimensional data, holes are filled in the edge information to obtain the single-layer two-dimensional data; The original wide-field fluorescence image is subjected to image binarization and image smoothing to obtain the wide-field fluorescence image.

5. The image registration method for ODT data based on a coarse calibration system according to any one of claims 1 to 3, characterized in that: The adaptive iterative algorithm improved based on the clustering algorithm matches the ODT feature point set and the wide-field fluorescence feature point set and iteratively generates an affine transformation matrix, and also includes: In the iterative process, the ODT feature point set and the wide-field fluorescence feature point set that meet the error requirements are reversely screened.

6. The image registration method for ODT data based on a coarse calibration system according to claim 5, characterized in that: In the iterative process, the ODT feature point set and the wide-field fluorescence feature point set that meet the error requirements are reversely screened, including: Step s1: determining an initial affine transformation matrix based on the ODT feature point set and the wide-field fluorescence feature point set, and determining a transformed ODT image based on the initial affine transformation matrix and the ODT image to be registered; Step s2: based on the transformed ODT image and the wide-field fluorescence image, calculating the Manhattan distance of the transformed corresponding point set, then using the Manhattan distance as an evaluation index, and filtering out the ODT feature point set and the wide-field fluorescence feature point set whose errors are greater than a set threshold based on the evaluation index; Step s1 and step s2 are iterated repeatedly to obtain the ODT feature point set, the wide-field fluorescence feature point set and the affine transformation matrix that meet the error requirements.

7. An image registration system for ODT data based on a coarse calibration system, characterized in that: include: An acquisition unit, used for acquiring three-dimensional refractive index data of the ODT image to be registered, and extracting a single layer of two-dimensional data matching the wide-field fluorescence image from the three-dimensional refractive index data; the ODT image to be registered and the wide-field fluorescence image are acquired at the same time of the same sample, and the ODT image to be registered and the wide-field fluorescence image have an overlapping area; A feature point extraction unit, used to extract feature points from the single-layer two-dimensional data and the wide-field fluorescence image, respectively, to obtain an ODT feature point set and a wide-field fluorescence feature point set; A matching unit, used for matching the ODT feature point set and the wide-field fluorescence feature point set based on an adaptive iterative algorithm and iteratively generating an affine transformation matrix; A perspective transformation matrix solving unit, used to take the affine transformation matrix as an initial solution, construct a differential optimizer, input the affine transformation matrix into the differential optimizer for optimization, and obtain a perspective transformation matrix; A determination unit is used to determine an image registration result of the three-dimensional refractive index data and the wide-field fluorescence image based on the perspective transformation matrix.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image registration method for ODT data based on a coarse calibration system is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image registration method for ODT data based on a coarse calibration system according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image registration method for ODT data based on a coarse calibration system according to any one of claims 1 to 6 is implemented.

Images