Microscopic image three-dimensional imaging method, device, electronic equipment and storage medium

Abstract

The present application relates to the field of image processing technology, and in particular to a three-dimensional imaging method, device, electronic device and storage medium for microscopic images, which obtains a target two-dimensional image by acquiring a nuclear image and a cytoplasm image of a target cell, performs nuclear-cytoplasm alignment processing on the nuclear image and the cytoplasm image, obtains the target two-dimensional image, performs three-dimensional conversion and image reconstruction on the target two-dimensional image to obtain an initial three-dimensional image, and performs registration and splicing processing on the initial three-dimensional image to obtain the target three-dimensional image, improves the robustness of the registration, achieves accurate registration of microscopic images at different spatial positions, and reconstructs the 3D display of pathology.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for three-dimensional imaging of microscopic images. Background Art

[0002] Microscopic imaging is a widely used optical instrument that magnifies the object being observed through the action of the optical system, displays the microscopic tissue structure, and obtains the field of view image under the microscope through the image acquisition device. The dual-channel light sheet microscope acquires dual-channel signals after staining the cell nucleus and cytoplasm, which can enhance the display of cytoplasm and nuclear information and complete the three-dimensional imaging of pathological tissue without manual pathological sectioning.

[0003] In the prior art, after dual-channel microscopic imaging, the 2D slice images collected by the microscope need to be registered and reconstructed based on a 3D imaging algorithm. For example, when collecting rock slices with a microscope, images under multiple fields of view are collected, grayscale deviation correction is performed on the image to be stitched, and then features are extracted based on the edge feature threshold method, image registration is performed based on the Hausdorff distance method, and finally multiple images are fused based on the wavelet framework to complete the stitching of 2D images. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the present invention provides a method, device, electronic device and storage medium for three-dimensional imaging of microscopic images to reconstruct the 3D display of target cells to improve the accuracy of three-dimensional imaging of microscopic images.

[0005] The first aspect of the present application provides a method for three-dimensional microscopic imaging, the method comprising:

[0006] Acquire nuclear and cytoplasmic images of target cells;

[0007] Performing nucleus-cytoplasm alignment processing on the nucleus image and the cytoplasm image to obtain a target two-dimensional image;

[0008] Performing three-dimensional conversion and image reconstruction on the target two-dimensional image to obtain an initial three-dimensional image;

[0009] The initial three-dimensional image is registered and spliced ​​to obtain a target three-dimensional image.

[0010] In an optional implementation, the performing registration and splicing processing on the initial three-dimensional image to obtain the target three-dimensional image includes:

[0011] Acquire at least first three initial three-dimensional images in the first image set to obtain a first sub-image set, and acquire at least first three initial three-dimensional images in the second image set to obtain a second sub-image set;

[0012] determining a second registration parameter of the first sub-image set and the second sub-image set;

[0013] Determining a splicing position and a fusion width of the first sub-image set and the second sub-image set;

[0014] The first sub-image set and the second sub-image set are fused according to the second registration parameter, the stitching position and the fusion width to obtain the target three-dimensional image.

[0015] In an optional implementation, the determining the second registration parameter of the first sub-image set and the second sub-image set includes:

[0016] Performing a coarse registration process on a first sub-image and a second sub-image to determine an image overlap area between the first sub-image and the second sub-image, wherein the first sub-image is a first initial three-dimensional image in a set of first sub-images, and the second sub-image is a first initial three-dimensional image in a set of second sub-images;

[0017] Extracting SIFT feature points of the overlapping area of ​​the image, and performing feature point matching on the SIFT feature points to obtain initial matching point pairs;

[0018] Eliminate the wrong matching points in the initial matching point pair to obtain the target matching point pair;

[0019] The second registration parameter is estimated according to the target matching point pair.

[0020] In an optional embodiment, performing nucleus-cytoplasm alignment processing on the nucleus image and the cytoplasm image to obtain a target two-dimensional image includes:

[0021] Extracting key features from the cell nucleus image and the cytoplasm image;

[0022] Performing feature matching on the key features to obtain a first registration parameter;

[0023] Based on the first registration parameter, spatially align the cell nucleus image and the cytoplasm image to obtain an initial two-dimensional image;

[0024] The initial two-dimensional image is subjected to pseudo-color processing according to a preset color lookup table to obtain the target two-dimensional image.

[0025] In an optional implementation, performing pseudo-color processing on the initial two-dimensional image according to a preset color lookup table to obtain the target two-dimensional image includes:

[0026] Determine the R, G, and B components of the initial two-dimensional image according to the preset color lookup table;

[0027] Staining the initial two-dimensional image according to the R, G, and B components; determining a first weight value of a cell nucleus image and a second weight value of a cytoplasm image in the stained initial two-dimensional image;

[0028] The initial two-dimensional image after dyeing is weighted and summed according to the first weight value and the second weight value to obtain an implementation target two-dimensional image.

[0029] In an optional implementation, performing three-dimensional conversion and image reconstruction on the target two-dimensional image to obtain an initial three-dimensional image includes:

[0030] Performing a three-dimensional transformation on the target two-dimensional image based on a shear transformation matrix;

[0031] The target two-dimensional image after the three-dimensional conversion is reconstructed based on the VTK third-party library to obtain the initial three-dimensional image.

[0032] In an optional implementation, the shear transformation matrix is ​​expressed by the following formula:

[0033] ;

[0034] Wherein, M is the shear transformation matrix.

[0035] A second aspect of the present application provides a microscopic three-dimensional imaging device, the device comprising:

[0036] An image acquisition module, used for acquiring a cell nucleus image and a cytoplasm image of a target cell;

[0037] An alignment processing module, used for performing nucleus-cytoplasm alignment processing on the nucleus image and the cytoplasm image to obtain a target two-dimensional image;

[0038] A three-dimensional reconstruction module, used for performing three-dimensional conversion and image reconstruction on the target two-dimensional image to obtain an initial three-dimensional image;

[0039] The three-dimensional registration module is used to perform registration and splicing processing on the initial three-dimensional image to obtain a target three-dimensional image.

[0040] A third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the three-dimensional imaging method of microscopic images when executing the computer program.

[0041] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned microscopic image three-dimensional imaging method when executed by a processor.

[0042] In summary, the three-dimensional imaging method, device, electronic device and storage medium of microscopic images provided by the present application first obtain images of the cell nucleus and cytoplasm, and achieve high-precision alignment of the cell nucleus and cytoplasm images through key feature extraction and feature matching technology, thereby ensuring the integrity and accuracy of the cell structure in subsequent three-dimensional imaging, and avoiding structural distortion or information loss caused by image misalignment; then, the shear transformation matrix is ​​used for three-dimensional conversion, and the image is reconstructed with the help of the VTK third-party library to directly generate the initial three-dimensional image, thereby improving processing efficiency and imaging quality; further, through the steps of coarse alignment, feature point matching, and erroneous matching point elimination, the alignment parameters and splicing positions between different images are accurately determined, thereby achieving efficient fusion of multiple images, reducing splicing gaps and ghosting phenomena, and improving the continuity and consistency of the final three-dimensional image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a method for three-dimensional imaging of a microscopic image shown in an embodiment of the present application;

[0044] Figure 2 is a schematic diagram of a cell nucleus image and a cytoplasm image of a target cell shown in an embodiment of the present application;

[0045] Figure 3 is a flow chart of a three-dimensional image registration method shown in an embodiment of the present application;

[0046] Figure 4 It is a functional module diagram of a microscopic image three-dimensional imaging device shown in an embodiment of the present application;

[0047] Figure 5 It is a structural schematic diagram of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0049] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0050] Reference Figure 1 FIG. 1 is a flow chart of a three-dimensional microscopic image imaging method according to an embodiment of the present application. The three-dimensional microscopic image imaging method includes the following steps.

[0051] S11, acquiring a nucleus image and a cytoplasm image of a target cell.

[0052] Wherein, the target cell refers to the subject used to collect the microscopic image. In some embodiments, the target cell can be a pathological sample, such as thyroid cancer, breast cancer, gastric cancer cell, etc.

[0053] In some embodiments, when it is necessary to collect the nucleus image and cytoplasm image of the target cell, a sample containing the target cell, such as a cell culture dish, a tissue section, etc., can be selected, and the cell morphology can be fixed using an appropriate chemical fixative, and then stained with specific dyes for the nucleus and cytoplasm, respectively, to enhance the contrast between the nucleus and cytoplasm. Specifically, DAPI can be used as a nucleus dye, and fluorescent dyes such as FITC and Cy3 can be used as cytoplasm dyes.

[0054] Since the nucleus and cytoplasm of the target cells are stained, the target cells can be observed using an optical microscope (such as a fluorescence microscope, a confocal microscope, etc.) or an electron microscope (such as a transmission electron microscope, a scanning electron microscope, etc.) equipped with a fluorescence imaging function. In an embodiment of the present application, two different fluorescence channels are first set, corresponding to the dye excitation and emission wavelengths of the nucleus and cytoplasm, respectively. The target cells are scanned layer by layer by adjusting the focusing plane of the microscope. At each level, images of the two channels are collected simultaneously to obtain a nucleus image and a cytoplasm image. Specifically, under an optical microscope, the target cell is placed on a displacement stage, and the displacement stage moves at a constant speed along the Y axis. When the displacement stage moves along the Y axis, in order to capture a wider cell area in a single exposure, the camera shoots at a 45-degree angle. When it is detected that the displacement stage moves along the Y axis, continuous shooting is performed using a camera, so that image slices of the target cell at different Y-axis positions can be obtained. Due to dual-channel imaging, the camera will collect the nuclear signal and cytoplasm signal of the target cell respectively, that is, multiple nucleus images and multiple cytoplasm images, refer to Figure 2 In other embodiments, the cell nucleus image and the cytoplasm image can also be captured by using the image capture software or camera attached to the microscope.

[0055] S12, performing nucleus-cytoplasm alignment processing on the nucleus image and the cytoplasm image to obtain a target two-dimensional image.

[0056] When the cell nucleus image and the cytoplasm image of the target cell are collected, the cell nucleus image and the cytoplasm image are subjected to cell nucleus-cytoplasm alignment processing. Specifically, the cell nucleus image and the cytoplasm image are subjected to feature extraction and matching through image processing technology (such as edge detection, feature point extraction, etc.) to determine the alignment parameters (first alignment parameters), and the cell nucleus-cytoplasm alignment processing is performed based on the first alignment parameters.

[0057] In an optional embodiment, performing nucleus-cytoplasm alignment processing on the nucleus image and the cytoplasm image to obtain a target two-dimensional image includes:

[0058] Extracting key features from the cell nucleus image and the cytoplasm image;

[0059] Performing feature matching on the key features to obtain a first registration parameter;

[0060] Based on the first registration parameter, spatially align the cell nucleus image and the cytoplasm image to obtain an initial two-dimensional image;

[0061] The initial two-dimensional image is subjected to pseudo-color processing according to a preset color lookup table to obtain the target two-dimensional image.

[0062] In some embodiments, during the feature extraction process, the electronic device can pre-test the target, that is, shoot the calibration plate, because the gradient information usually includes more significant structural information in the image, so the gradient information is extracted on the calibration plate by edge detection method, and the point with the largest gradient is selected as the key feature point. During the feature matching process, the electronic device can match the feature points corresponding to the key feature points on another corresponding calibration plate using a block matching algorithm. It should be noted that the target test can be performed multiple times, for example, the objective lens can be repeatedly disassembled and assembled, the calibration plate can be moved and focused multiple times, and multiple groups of calibration plate images can be obtained. In the process of estimating the registration parameters, the electronic device can estimate the transformation matrix (such as an affine transformation matrix or a perspective transformation matrix) based on the matched feature point pairs based on the least squares algorithm, and estimate the transformation parameters required to transform the cell nucleus image into the cytoplasm image space, that is, the registration parameters, including horizontal offset, vertical offset, rotation angle, shear value, and scaling factor.

[0063] For each target test, a registration coefficient can be obtained. The statistical average of the registration parameters obtained from N target tests is calculated as the final registration parameter H (first registration parameter). The formula is as follows:

[0064] .

[0065] After determining the first registration parameter, the electronic device can use an image transformation algorithm (such as affine transformation, perspective transformation, etc.) to spatially align the cell nucleus image and the cytoplasm image based on the first registration parameter, so that the cell nucleus image and the cytoplasm image of the target cell are in the same coordinate system, and the positions of the cell nucleus and the cytoplasm are aligned, thereby obtaining multiple two-dimensional images (initial two-dimensional images).

[0066] In some embodiments, the electronic device may further fine-tune the transformed initial two-dimensional image and verify the alignment effect. The alignment effect may be verified by visual inspection or by using quantitative evaluation indicators (such as mean square error, mutual information, etc.). If the alignment effect is not good, the above steps may be returned to perform feature point extraction, feature point matching, or registration parameter estimation again.

[0067] Furthermore, the electronic device can search a preset color lookup table to obtain the corresponding R, G, and B components. The R, G, and B components are determined according to the experiment, and the relationship between each component and the grayscale value is nonlinear. Specifically, the target cells are stained with a staining reagent. When the staining is completed, a one-to-one correspondence is made according to the grayscale of the corresponding position after the target cell imaging and the color after staining, and then an exponential function is fitted to generate a color mapping curve, that is, the corresponding R, G, and B components. For the cell nucleus and cytoplasm in the initial two-dimensional image, the R, G, and B component formulas are as follows:

[0068] ;

[0069] ;

[0070] ;

[0071] ;

[0072] ;

[0073] ;

[0074] in is an exponential function, as follows:

[0075] ;

[0076] in It is an adjustable parameter.

[0077] It should be noted that the weight values ​​of the stained nucleus image and cytoplasm image are both set to 0.5.

[0078] The electronic device further performs weighted summation to obtain a cell color image (target two-dimensional image):

[0079] ;

[0080] in, is the target two-dimensional image, is the image of the cell nucleus after staining, The stained cytoplasm is shown in Figure 2 .

[0081] In some embodiments, before performing feature extraction, the electronic device may also pre-process the cytoplasm image and the cell nucleus object, such as denoising, contrast enhancement, etc., to improve the accuracy and stability of feature extraction.

[0082] Through the above optional implementation, the positions of the nucleus and cytoplasm in the image of the same cell are ensured to be consistent, and efficient alignment of the nucleus and cytoplasm images is achieved, ensuring the accuracy of the cell structure in three-dimensional imaging. Secondly, multiple target tests are used to optimize the registration parameters to improve the alignment accuracy.

[0083] S13, performing three-dimensional conversion and image reconstruction on the target two-dimensional image to obtain an initial three-dimensional image.

[0084] In some embodiments, the electronic device first performs a three-dimensional conversion on the target two-dimensional image obtained in S12, and then continues to perform image reconstruction on the three-dimensional image obtained after the three-dimensional conversion to obtain an initial three-dimensional image.

[0085] In an optional implementation, performing three-dimensional conversion and image reconstruction on the target two-dimensional image to obtain an initial three-dimensional image includes:

[0086] Performing a three-dimensional transformation on the target two-dimensional image based on a shear transformation matrix;

[0087] The target two-dimensional image after the three-dimensional conversion is reconstructed based on the VTK third-party library to obtain the initial three-dimensional image.

[0088] (1) 3D edge transformation.

[0089] In some embodiments, the electronic device may predefine a shear transformation matrix M, and perform a three-dimensional conversion on each target two-dimensional image based on the shear transformation matrix M.

[0090] The shear transformation matrix M is calculated based on the light sheet inclination angle of 45 degrees (ie, cos45°), and the transformation matrix is ​​calculated using the following formula:

[0091] ;

[0092] Where M is a 3×3 matrix. .

[0093] For each target two-dimensional image after the nucleus-cytoplasm alignment process, each target two-dimensional image (or its corresponding plane in the 3D space) is multiplied by the transformation matrix M to achieve 3D transformation of the image set collected by the translation stage along the y-axis through 3D shear transformation, and to convert the two-dimensional image into a three-dimensional image, as follows:

[0094] .

[0095] The image set after 3D trimming transformation is regarded as the basic data for constructing three-dimensional objects.

[0096] In some embodiments, since the transformed pixel coordinates may not be integers, for the grayscale values ​​at the non-integer coordinates, the electronic device can use an interpolation method (such as bilinear interpolation, bicubic interpolation, etc.) to estimate the pixel values ​​at the non-integer coordinates.

[0097] In some embodiments, since the electronic device needs to perform a large amount of data calculations during the three-dimensional transformation process, OPENGL-based acceleration can be used to meet the time requirements of 3D imaging.

[0098] (2) 3D image reconstruction.

[0099] Furthermore, the electronic device can perform 3D reconstruction on the image after the 3D conversion. Specifically, the electronic device can perform 3D reconstruction using surface rendering based on the VTK third-party library.

[0100] In some embodiments, during the 3D image reconstruction process, the electronic device may also use sampling technology to improve the resolution in the Y-axis direction.

[0101] In some embodiments, in order to improve the reconstruction speed, during the 3D image reconstruction process, the electronic device may also perform algorithm acceleration based on a general parallel computing platform and programming model (eg, CUDA).

[0102] Through the above optional implementation, accurate 3D shear transformation and efficient VTK library are used for image reconstruction, achieving accurate conversion and reconstruction from two-dimensional to three-dimensional images. Interpolation method is used to process non-integer coordinates to ensure image quality. At the same time, OPENGL and CUDA acceleration technology are used to significantly improve data processing speed and imaging efficiency, ensuring the real-time and accuracy of three-dimensional imaging.

[0103] S14, performing registration and splicing processing on the initial three-dimensional image to obtain a target three-dimensional image.

[0104] In some embodiments, during the three-dimensional imaging process, since the size of the sample may exceed the range of a single scan, multiple scans are required (each scan obtains a piece of image data), wherein the scan strips are adjacent in the X-axis direction, but due to the influence of experimental conditions (such as slight deviation of the translation stage, movement of the sample, etc.), there may be slight position deviations between adjacent strips. Therefore, in order to obtain a complete and seamless three-dimensional image, it is necessary to perform 3D registration and splicing of adjacent scan strips to find the best alignment between adjacent strips, eliminate position deviations, and splice and merge the registered image strips into a whole three-dimensional image.

[0105] In an optional implementation, the performing registration and splicing processing on the initial three-dimensional image to obtain the target three-dimensional image includes:

[0106] Acquire at least first three initial three-dimensional images in the first image set to obtain a first sub-image set, and acquire at least first three initial three-dimensional images in the second image set to obtain a second sub-image set;

[0107] determining a second registration parameter of the first sub-image set and the second sub-image set;

[0108] Determining a splicing position and a fusion width of the first sub-image set and the second sub-image set;

[0109] The first sub-image set and the second sub-image set are fused according to the second registration parameter, the stitching position and the fusion width to obtain the target three-dimensional image.

[0110] After the 3D conversion and 3D reconstruction are completed to obtain the 3D image (initial 3D image), for each initial 3D image, the electronic device needs to further perform registration and splicing processing on two adjacent image strips in the X-axis direction of the initial 3D image. Figure 3 The electronic device first obtains at least the first three initial three-dimensional images of each of the two three-dimensional image sets (the first image set and the second image set), and the at least first three initial three-dimensional images obtained from the first image set are called the first sub-image set, and the at least first three initial three-dimensional images obtained from the combination of the second image set are called the second sub-image set. For any initial three-dimensional image of the first sub-image set and the second sub-image set, the electronic device needs to determine its corresponding registration parameter, and perform registration and splicing processing on the initial three-dimensional images in the two three-dimensional image sets based on the registration parameter.

[0111] Specifically, the electronic device first extracts the first three three-dimensional images from two adjacent 3D three-dimensional image sets A and B, and marks them as and Secondly, and To perform registration, and To perform registration, and To facilitate the registration operation in this application, the first initial three-dimensional image in the first sub-image set is The first initial 3D image in the set of (first sub-image) and the second sub-image The registration process of (the second sub-image) is taken as an example, which is specifically implemented through the following registration steps.

[0112] (1) Yes and Perform a coarse alignment.

[0113] In some embodiments, the electronic device may use a phase peak algorithm, that is, respectively and Do Fourier transform and and Transform from the spatial domain to the frequency domain. As a reference image, As the image to be matched, calculate and The cross-correlation spectrum between the frequency domains of the images is extracted, and the phase spectrum is extracted from the cross-correlation spectrum. Then, the electronic device can calculate the offset angle between adjacent images according to the movement curve of the translation stage, and determine the corresponding preset angle step and preset translation step according to the offset angle, and then calculate the phase spectrum according to the preset angle step. Query different rotation angle ranges , that is, according to the preset translation step Query translation parameters , Then, the electronic device sorts the peak values ​​of each phase spectrum and calculates the parameter corresponding to the maximum peak value. As the registration parameters, exemplary, within a preset rotation angle range (e.g., ±10°) and a translation range (e.g., ±10 pixels), the optimal rotation parameter and translation parameter are searched with a preset angle step (e.g., 1° and 1 pixel) to maximize the peak value of the phase spectrum.

[0114] (2) Determination and overlapping area.

[0115] In some embodiments, the electronic device may compare and The pixel values, feature points or other attributes at the same or similar spatial positions determine the coarse registration. and which areas are overlapping.

[0116] Secondly, the electronic device can calculate the speed of the light sheet microscopy stage according to its movement speed. and The overlap is half the image width:

[0117] ;

[0118] The moving speed refers to the physical distance x that the stage moves. According to the camera pixel resolution pix, the overlapping area can be calculated as x / pix.

[0119] Through the above-mentioned embodiment mode, by calculating and The overlapping area can determine the area for subsequent feature point searches.

[0120] (3) Feature point extraction.

[0121] In some embodiments, the electronic device can use a feature point extraction algorithm (such as SIFT, SURF, ORB, etc.) in the overlapping area to detect and extract points with significant features (such as corner points, edge points, etc.). In the embodiment of the present application, the electronic device extracts SIFT feature points, wherein SIFT feature points are a local feature description operator of an image based on scale space, which is invariant to image scaling, rotation, and even affine transformation, is insensitive to image rotation and brightness transformation, and is robust.

[0122] (4) Feature point matching.

[0123] In some embodiments, the electronic device can perform feature point matching based on SAD. Specifically, first determine the length of the corresponding window after registration based on multiple experiments, and construct a small window based on the window length to cover the feature point position determined in step (3), including the reference image and the floating image; then calculate the difference in grayscale within the window and sum it. Move the window in the floating image until the search range ends, and find the window with the smallest SAD value within the search range, that is, the best matching feature point position is found, that is, the initial matching point pair. In other embodiments, the electronic device can also use other algorithms, such as the Fast Library for Approximate Nearest Neighbors (FLANN), the strong matcher (BFMatcher), etc. to perform feature point matching and find the initial matching point pair.

[0124] In some embodiments, since the matching results may include some erroneous matches, the electronic device may use an algorithm such as a random sample consensus algorithm (RANSAC) to remove erroneous matching points in the initial matching point pair to obtain a target matching point pair.

[0125] (5) Determination and The registration coefficient of .

[0126] For the target matching point pairs obtained by feature matching, in some embodiments, the electronic device can use the RANSAC algorithm to estimate the registration parameters, that is, after the iteration, the model parameters with the largest number of inliers are output as the final registration parameters. In other embodiments, the electronic device can use the least squares method, SVD decomposition or other optimization algorithms to estimate the target matching point pairs. and The precise transformation parameters between them (i.e., the second registration parameters) include translation, rotation, scaling, and affine transformation parameters.

[0127] In some embodiments, the electronic device may further refine the registration parameters through an iterative optimization algorithm (such as gradient descent, Levenberg-Marquardt method, etc.) to improve the accuracy of the registration.

[0128] (6) Repeat the above steps to and To align and and Perform registration.

[0129] (7) Perform statistical mean operation on the above registration parameters to obtain the final registration parameter T (second registration parameter).

[0130] In completion and The registration is obtained and of Registration coefficient , and complete and , we get and The registration coefficient , and complete and The registration is obtained and The registration coefficient After that, the mean operation is performed to obtain the second registration parameter, the formula is as follows:

[0131] .

[0132] (8) Based on the second registration parameter T, each image in two adjacent 3D image sets is registered.

[0133] In some embodiments, according to the same embodiment as above, the second registration parameter T is applied to the first sub-image set and the second sub-image set for transformation.

[0134] (9) Determine the splicing position and fusion width.

[0135] In some embodiments, the electronic device can determine the stitching position pos according to the orientation of the floating image relative to the reference image and the horizontal offset component in the registration parameter, and determine the fusion width w according to multiple tests. Specifically, an initial fusion width can be set according to experience or preliminary tests, and the initial fusion width can be applied to the actual image stitching task, and the effect of the fusion area can be observed, and then the initial fusion width can be adjusted according to the effect of the fusion area. After multiple iterations of adjustment and optimization, an optimal fusion width is determined, that is, the fusion width w in the example of this application. Therefore, a certain fusion width w is specified for the stitching position pos, and in The pixel values ​​are fused using linear weighting.

[0136] In some embodiments, the electronic device may further utilize a smoothing filter (such as a Gaussian filter) to reduce artifacts or discontinuities that may appear in the fusion area.

[0137] (10) Perform the same fusion process of stitching nearby areas on all adjacent images in the two 3D image sets.

[0138] For each pair of adjacent images in the two 3D image sets (including the registered images), the fusion process in step (9) is repeated to ensure that the stitching areas between all adjacent images in the entire image set are properly fused. For example, near the stitching position, linear weighting or other fusion methods (such as multi-band fusion) are used to merge and the transformed The images are fused to generate seamless images.

[0139] In some embodiments, after the fusion is completed, a global quality assessment is performed on each target 3D image in the entire fused 3D image set to check whether there are unresolved registration errors or fusion artifacts.

[0140] In some embodiments, since the final registration coefficient is obtained by calculating the mean, before calculating the second registration parameter, the electronic device needs to determine whether the image currently being processed is the last three-dimensional image in the image set.

[0141] Through the above optional implementation, the slight position deviation between multiple scans is effectively eliminated through 3D registration and stitching processing, and seamless stitching of three-dimensional images is achieved. At the same time, through fusion width setting and linear weighted fusion, the smooth transition of the stitching area is ensured, and finally a complete and high-quality target three-dimensional image is generated.

[0142] Reference Figure 4 , which is a functional module diagram of a microscopic image three-dimensional imaging device shown in an embodiment of the present application.

[0143] In some embodiments, the microscopic image 3D imaging device 40 may include a plurality of functional modules composed of computer program segments. The computer program of each program segment of the microscopic image 3D imaging device 40 may be stored in a memory of an electronic device and executed by at least one processor to perform (see Figure 1 Description) The function of three-dimensional imaging of microscopic images.

[0144] In this embodiment, the microscopic image three-dimensional imaging device 40 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: an image acquisition module 401, an alignment processing module 402, a three-dimensional reconstruction module 403 and a three-dimensional registration module 404. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0145] The image acquisition module 401 is used to acquire the nucleus image and cytoplasm image of the target cell.

[0146] The alignment processing module 402 is used to perform nucleus-cytoplasm alignment processing on the nucleus image and the cytoplasm image to obtain a target two-dimensional image.

[0147] The three-dimensional reconstruction module 403 is used to perform three-dimensional conversion and image reconstruction on the target two-dimensional image to obtain an initial three-dimensional image. The three-dimensional registration module 404 is used to perform registration and splicing processing on the initial three-dimensional image to obtain a target three-dimensional image. The three-dimensional registration module 404 is also specifically used to: obtain at least the first three initial three-dimensional images in the first image set to obtain a first sub-image set, and obtain at least the first three initial three-dimensional images in the second image set to obtain a second sub-image set; determine the second registration parameters of the first sub-image set and the second sub-image set; determine the splicing position and fusion width of the first sub-image set and the second sub-image set; perform image fusion on the first sub-image set and the second sub-image set according to the second registration parameters, the splicing position and the fusion width to obtain the target three-dimensional image.

[0148] The three-dimensional registration module 404 is also specifically used to: perform coarse registration processing on the first sub-image and the second sub-image to determine the image overlapping area between the first sub-image and the second sub-image, the first sub-image is the first initial three-dimensional image in the first sub-image set, and the second sub-image is the first initial three-dimensional image in the second sub-image set; extract SIFT feature points in the image overlapping area, and perform feature point matching on the SIFT feature points to obtain initial matching point pairs; eliminate erroneous matching points in the initial matching point pairs to obtain target matching point pairs; and estimate the second registration parameters based on the target matching point pairs.

[0149] The alignment processing module 402 is also specifically used to: extract key features of the cell nucleus image and the cytoplasm image; perform feature matching on the key features to obtain a first alignment parameter; spatially align the cell nucleus image and the cytoplasm image based on the first alignment parameter to obtain an initial two-dimensional image; and perform pseudo-color processing on the initial two-dimensional image according to a preset color lookup table to obtain the target two-dimensional image.

[0150] The alignment processing module 402 is also specifically used to: determine the R, G, and B components of the initial two-dimensional image according to the preset color lookup table; stain the initial two-dimensional image according to the R, G, and B components; determine a first weight value of the cell nucleus image and a second weight value of the cytoplasm image in the stained initial two-dimensional image; and perform weighted summation of the stained initial two-dimensional image according to the first weight value and the second weight value to obtain a target two-dimensional image.

[0151] The three-dimensional reconstruction module 403 is further specifically used for: performing three-dimensional transformation on the target two-dimensional image based on the shear transformation matrix; and reconstructing the target two-dimensional image after the three-dimensional transformation based on the VTK third-party library to obtain the initial three-dimensional image.

[0152] It should be understood that the various variations and specific embodiments of the microscopic image three-dimensional imaging method provided in the above embodiments are also applicable to the microscopic image three-dimensional imaging device of the present embodiment. Through the above detailed description of the microscopic image three-dimensional imaging method, those skilled in the art can clearly know the implementation method of the microscopic image three-dimensional imaging device in the present embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0153] See also Figure 5 FIG. 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. In a preferred embodiment of the present application, the electronic device 5 includes a memory 51 , at least one processor 52 and at least one communication bus 53 .

[0154] Those skilled in the art should understand that Figure 5 The structure of the electronic device shown does not constitute a limitation of the embodiments of the present application, and may be either a bus structure or a star structure. The electronic device 5 may also include more or less other hardware or software than shown in the figure, or a different component arrangement.

[0155] In some embodiments, the electronic device 5 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The electronic device 5 may also include user equipment, which includes but is not limited to any electronic product that can interact with a user through a keyboard, mouse, remote control, touchpad, or voice control device, such as a personal computer, tablet computer, smart phone, digital camera, etc.

[0156] It should be noted that the electronic device 3 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.

[0157] In the above embodiments provided in the present application, it should be understood that the disclosed methods, devices, computer-readable storage media, and electronic devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple components or modules can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or components or modules, which can be electrical, mechanical or other forms.

[0158] The components described as separate components may or may not be physically separated, and the components shown as components may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the components may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each component may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0160] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0161] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0162] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0163] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for three-dimensional microscopic imaging, characterized in that: The method comprises: Acquire nuclear and cytoplasmic images of target cells; Performing nucleus-cytoplasm alignment processing on the nucleus image and the cytoplasm image to obtain a target two-dimensional image; Performing three-dimensional conversion and image reconstruction on the target two-dimensional image to obtain an initial three-dimensional image; The initial three-dimensional images are registered and stitched to obtain a target three-dimensional image, including: obtaining at least the first three initial three-dimensional images in a first image set to obtain a first sub-image set, and obtaining at least the first three initial three-dimensional images in a second image set to obtain a second sub-image set; determining a second registration parameter for the first sub-image set and the second sub-image set; determining a stitching position and a fusion width for the first sub-image set and the second sub-image set; performing image fusion on the first sub-image set and the second sub-image set according to the second registration parameter, the stitching position and the fusion width to obtain the target three-dimensional image, wherein the determining The second registration parameters of the first sub-image set and the second sub-image set include: performing coarse registration processing on the first sub-image and the second sub-image to determine the image overlapping area between the first sub-image and the second sub-image, the first sub-image is the first initial three-dimensional image in the first sub-image set, and the second sub-image is the first initial three-dimensional image in the second sub-image set; extracting SIFT feature points of the image overlapping area, and performing feature point matching on the SIFT feature points to obtain initial matching point pairs; eliminating erroneous matching points in the initial matching point pairs to obtain target matching point pairs; estimating the second registration parameters based on the target matching point pairs.

2. The three-dimensional imaging method of microscopic images according to claim 1, characterized in that: The performing nucleus-cytoplasm alignment processing on the nucleus image and the cytoplasm image to obtain a target two-dimensional image comprises: Extracting key features from the cell nucleus image and the cytoplasm image; Performing feature matching on the key features to obtain a first registration parameter; Based on the first registration parameter, spatially align the cell nucleus image and the cytoplasm image to obtain an initial two-dimensional image; The initial two-dimensional image is subjected to pseudo-color processing according to a preset color lookup table to obtain the target two-dimensional image.

3. The three-dimensional imaging method of microscopic images according to claim 2, characterized in that: The performing pseudo-color processing on the initial two-dimensional image according to a preset color lookup table to obtain the target two-dimensional image comprises: Determine the R, G, and B components of the initial two-dimensional image according to the preset color lookup table; Staining the initial two-dimensional image according to the R, G, and B components; determining a first weight value of a cell nucleus image and a second weight value of a cytoplasm image in the stained initial two-dimensional image; The initial two-dimensional image after dyeing is weighted and summed according to the first weight value and the second weight value to obtain an implementation target two-dimensional image.

4. The three-dimensional imaging method of microscopic images according to claim 1, characterized in that: The performing three-dimensional conversion and image reconstruction on the target two-dimensional image to obtain an initial three-dimensional image comprises: Performing a three-dimensional transformation on the target two-dimensional image based on a shear transformation matrix; The target two-dimensional image after the three-dimensional conversion is reconstructed based on the VTK third-party library to obtain the initial three-dimensional image.

5. The three-dimensional imaging method of microscopic images according to claim 4, characterized in that: The shear transformation matrix is ​​expressed by the following formula: ； Wherein, M is the shear transformation matrix.

6. A three-dimensional microscopic imaging device, characterized in that: The device comprises: An image acquisition module, used for acquiring a cell nucleus image and a cytoplasm image of a target cell; An alignment processing module, used for performing nucleus-cytoplasm alignment processing on the nucleus image and the cytoplasm image to obtain a target two-dimensional image; A three-dimensional reconstruction module, used for performing three-dimensional conversion and image reconstruction on the target two-dimensional image to obtain an initial three-dimensional image; A three-dimensional registration module is used to perform registration and splicing processing on the initial three-dimensional images to obtain a target three-dimensional image, including: obtaining at least the first three initial three-dimensional images in the first image set to obtain a first sub-image set, and obtaining at least the first three initial three-dimensional images in the second image set to obtain a second sub-image set; determining a second registration parameter for the first sub-image set and the second sub-image set; determining a splicing position and a fusion width for the first sub-image set and the second sub-image set; performing image fusion on the first sub-image set and the second sub-image set according to the second registration parameter, the splicing position and the fusion width to obtain the target three-dimensional image, wherein the Determining the second registration parameters of the first sub-image set and the second sub-image set includes: performing coarse registration processing on the first sub-image and the second sub-image to determine the image overlapping area between the first sub-image and the second sub-image, the first sub-image is the first initial three-dimensional image in the first sub-image set, and the second sub-image is the first initial three-dimensional image in the second sub-image set; extracting SIFT feature points in the image overlapping area, and performing feature point matching on the SIFT feature points to obtain initial matching point pairs; eliminating erroneous matching points in the initial matching point pairs to obtain target matching point pairs; estimating the second registration parameters based on the target matching point pairs.

7. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the three-dimensional imaging method of microscopic images according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional imaging method of microscopic images according to any one of claims 1 to 5 are implemented.

Images