A multi-fluorescence-channel microsphere image alignment method, device and equipment

Abstract

The present invention discloses a method, device and equipment for multi-fluorescent channel microsphere image alignment. The method acquires fluorescent microsphere images at the same position through multiple fluorescent channels respectively; preprocesses each fluorescent microsphere image to obtain a fluorescence corrected image; performs microsphere region segmentation on each fluorescence corrected image respectively, and calculates the region center points corresponding to each microsphere region respectively; intercepts pixel matrices from the fluorescence corrected images according to the region center points, and extracts corresponding feature vectors; selects any one fluorescence corrected image as a reference image, and performs pairwise registration on the remaining fluorescence corrected images according to the feature vectors to respectively determine adjustment offsets; corrects each fluorescence corrected image according to the adjustment offsets and superimposes it on the reference image to generate a fluorescence channel aligned image. Thus, by combining the gradient direction to perform offset registration on the fluorescent microsphere image, high-precision image alignment is achieved without relying on deep learning.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, device, and equipment for aligning multi-fluorescent channel microsphere images. Background Art

[0002] Microspheres are tiny spherical particles with diameters typically in the micron range. In spatial transcriptomics, to obtain spatial information, bases are often bound to microspheres, so the microspheres carry fluorescent groups and blocking groups. Then, fluorescence signals are excited by lasers, and there are four fluorescence channels (CY5, FITC, ROX, TRITC) as Figure 1 shown, enabling microsphere images to be captured by a CCD camera.

[0003] Since the bases bound to the microspheres are different, microspheres bound to different bases will light up in different fluorescence channels. Therefore, at the same position, all microspheres will light up only when the four-channel images are aligned and superimposed. Thus, when we want to confirm the binding effect of microspheres and fluorescent dyes in a no-load experiment, we need to align and superimpose the four-channel data. However, since the four pieces of data are captured under different fluorescence channels respectively, there is a certain offset between any two of the four pieces of data. The present invention designs a registration algorithm for this image situation to achieve the alignment of four-channel images.

[0004] In medical imaging technology, to more accurately diagnose diseases, it is necessary to register images of different modalities, such as CT and MRI, CT and PET, etc. Currently, biomedical image registration algorithms can be divided into traditional registration algorithms and deep learning-based algorithms. However, traditional algorithms require a large number of iterations to reach the set target optimization parameters, which is time-consuming, and are greatly affected by image quality and have poor robustness. Deep learning registration algorithms often involve the production of large-scale datasets and model training. Limited by the current small dataset of microsphere images, the training of the model cannot fully converge. At the same time, the computational cost of the algorithm is relatively high and it has a high dependence on the hardware platform, resulting in the inability of the prior art to accurately register the fluorescent images of microspheres in four channels at low cost. Summary of the Invention

[0005] The present invention provides a method, device, and equipment for aligning multi-fluorescent channel microsphere images, which solves the technical problem that the prior art cannot accurately register the fluorescent images of microspheres in four channels at low cost.

[0006] A method for aligning multi-fluorescent channel microsphere images provided in the first aspect of the present invention includes:

[0007] Obtaining fluorescent microsphere images at the same position through multi-fluorescent channels;

[0008] Preprocess each of the fluorescent microsphere images to obtain fluorescence-corrected images;

[0009] Perform microsphere region segmentation on each of the fluorescence-corrected images, and calculate the region center points corresponding to each microsphere region;

[0010] Crop pixel matrices from the fluorescence-corrected images according to the region center points, and extract corresponding feature vectors;

[0011] Select any one of the fluorescence-corrected images as a reference image, and pairwise register the remaining fluorescence-corrected images according to the feature vectors to respectively determine adjustment offsets;

[0012] Correct each of the fluorescence-corrected images according to the adjustment offsets and superimpose them on the reference image to generate a fluorescence channel alignment image.

[0013] Optionally, the preprocessing of each of the fluorescent microsphere images to obtain fluorescence-corrected images includes:

[0014] Perform a power operation on the pixel value of each pixel in each of the fluorescent microsphere images according to a preset gamma parameter to respectively obtain updated pixel values;

[0015] Generate fluorescence-corrected images using the pixels with the updated pixel values.

[0016] Optionally, the performing microsphere region segmentation on each of the fluorescence-corrected images and calculating the region center points corresponding to each microsphere region includes:

[0017] Filter the pixel values of the pixels in each of the fluorescence-corrected images according to a preset quantile threshold to obtain a rough microsphere region segmentation image;

[0018] Filter the rough microsphere region segmentation image, and identify the microsphere region edges in combination with the Laplacian operator to obtain a microsphere region edge image;

[0019] Perform a subtraction operation on the rough microsphere region segmentation image and the microsphere region edge image to generate a microsphere region segmentation image;

[0020] Calculate the pixel distance from each non-zero pixel of each microsphere region in the microsphere region segmentation image to the nearest zero pixel;

[0021] Generate a distance peak map using all the pixel distances;

[0022] Calculate the local peak points of each microsphere region in the distance peak map as the corresponding region center points.

[0023] Optionally, calculating the pixel distance from each non-zero pixel in the microsphere region segmentation image to the nearest zero pixel includes:

[0024] Locate the non-zero pixels of each microsphere region from the microsphere region segmentation image, and initialize the first distance value of the non-zero pixels to a preset maximum value;

[0025] Starting from each non-zero pixel, retrieve the nearest zero pixel in the neighborhood, and calculate the second distance value between the non-zero pixel and the nearest zero pixel;

[0026] Select the minimum value between the second distance value and the first distance value to update the first distance value until all pixels in the microsphere region are retrieved;

[0027] Normalize each of the first distance values to obtain the pixel distance from each non-zero pixel in each microsphere region to the nearest zero pixel.

[0028] Optionally, intercepting a pixel matrix from the fluorescence correction image according to each of the region center points and extracting corresponding feature vectors includes:

[0029] Taking each of the region center points as the center, intercept a pixel matrix from the fluorescence correction image according to the first pixel size, and downsample it to the second pixel size to obtain a first pixel matrix;

[0030] Taking each of the region center points as the center, intercept a second pixel matrix from the fluorescence correction image according to the second pixel size;

[0031] Divide the gradient direction range into multiple gradient bins according to a preset dimension;

[0032] Calculate the gradient direction corresponding to each pixel in the first pixel matrix and the second pixel matrix respectively;

[0033] Classify each of the gradient directions into the corresponding gradient bin, and construct a gradient direction frequency table;

[0034] Generate a first feature vector corresponding to the first pixel matrix and a second feature vector corresponding to the second pixel matrix according to the gradient direction frequency table;

[0035] Concatenate the first feature vector and the second feature vector to generate a feature vector corresponding to each of the region center points.

[0036] Optionally, calculating the gradient direction corresponding to each pixel in the first pixel matrix and the second pixel matrix respectively includes:

[0037] Traverse each pixel in the first pixel matrix and the second pixel matrix, and calculate the corresponding horizontal direction change gradient and vertical direction change gradient respectively;

[0038] Calculate the gradient ratio between the horizontal direction change gradient and the vertical direction change gradient;

[0039] Calculate the arctangent value of the gradient ratio as the gradient direction.

[0040] Optionally, the adjustment offset includes a rotation offset and a displacement offset; selecting any one of the fluorescence correction images as the reference image, and pairwise registering the remaining fluorescence correction images according to the eigenvector, and respectively determining the adjustment offset, including:

[0041] Select any one of the fluorescence correction images as the reference image;

[0042] Perform pairwise registration on the eigenvector of the reference image and the eigenvectors corresponding to the remaining fluorescence correction images to determine multiple groups of matching feature point pairs;

[0043] Locate the corresponding feature point coordinate pairs from the reference image and the remaining fluorescence correction images respectively according to each group of the matching feature point pairs;

[0044] Determine the initial rotation offset and the initial displacement offset respectively according to each group of the feature point coordinate pairs;

[0045] Select the most frequent value among the initial rotation offsets as the rotation offset;

[0046] Select the most frequent value among the initial displacement offsets as the displacement offset.

[0047] Optionally, the step of correcting each fluorescence correction image according to the adjustment offset and superimposing it on the reference image to generate a fluorescence channel alignment image includes:

[0048] After rotating according to the rotation offset corresponding to each fluorescence correction image, perform translation according to the corresponding displacement offset to obtain a target fluorescence image;

[0049] Superimpose each target fluorescence image on the reference image to generate a fluorescence channel alignment image.

[0050] The second aspect of the present invention provides a multi-fluorescence channel microsphere image alignment device, including:

[0051] An image acquisition module, configured to respectively acquire fluorescence microsphere images at the same position through multiple fluorescence channels;

[0052] An image correction module, configured to perform preprocessing on each of the fluorescence microsphere images to obtain fluorescence correction images;

[0053] A region center point positioning module, configured to perform microsphere region segmentation on each of the fluorescence-corrected images respectively, and calculate the region center points corresponding to each microsphere region respectively;

[0054] A feature vector extraction module, configured to intercept pixel matrices from the fluorescence-corrected images according to each of the region center points, and extract corresponding feature vectors;

[0055] An offset determination module, configured to select any one of the fluorescence-corrected images as a reference image, and perform pairwise registration on the remaining fluorescence-corrected images according to the feature vectors, and determine adjustment offsets respectively;

[0056] An image correction module, configured to correct each of the fluorescence-corrected images according to the adjustment offsets and superimpose them on the reference image to generate a fluorescence channel alignment image.

[0057] A third aspect of the present invention provides an electronic device, including a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor is caused to execute the steps of the multi-fluorescence channel microsphere image method according to any one of the first aspects of the present invention.

[0058] It can be seen from the above technical solutions that the present invention has the following advantages:

[0059] The present invention respectively obtains fluorescence microsphere images at the same position through multiple fluorescence channels; preprocesses each fluorescence microsphere image to obtain a fluorescence-corrected image; performs microsphere region segmentation on each fluorescence-corrected image respectively, and calculates the region center points corresponding to each microsphere region respectively; intercepts pixel matrices from the fluorescence-corrected images according to each region center point, and extracts corresponding feature vectors; selects any one of the fluorescence-corrected images as a reference image, and performs pairwise registration on the remaining fluorescence-corrected images according to the feature vectors, and determines adjustment offsets respectively; corrects each fluorescence-corrected image according to the adjustment offsets and superimposes them on the reference image to generate a fluorescence channel alignment image. Thus, by combining the gradient direction to perform offset registration on the fluorescence microsphere images, high-precision image alignment is achieved without relying on deep learning. Description of the Drawings

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to these drawings without creative efforts.

[0061] Figure 1Schematic diagram of fluorescence microsphere images obtained by four fluorescence channels in the embodiments of the present invention;

[0062] Figure 2 Flowchart of steps of a method for aligning multi-fluorescence channel microsphere images provided by embodiments of the present invention;

[0063] Figure 3 Refined flowchart of steps of a method for aligning multi-fluorescence channel microsphere images provided by embodiments of the present invention;

[0064] Figure 4 Schematic diagram of image change during the determination of the central point of the region provided by embodiments of the present invention;

[0065] Figure 5 Schematic diagram of the process of aligning multi-fluorescence channel microsphere images provided by embodiments of the present invention;

[0066] Figure 6 Structural block diagram of a device for aligning multi-fluorescence channel microsphere images provided by embodiments of the present invention. Detailed implementation manners

[0067] Due to the existence of multiple reasons such as fluorescence switching, machine jitter, and fluorescence reflection, there are certain deviations between the four-channel fluorescence microsphere images. The mechanical design principle of the microscopic imaging system determines that this deviation is mainly rotation deviation and displacement deviation in the x and y directions. For this reason, embodiments of the present invention provide a method, device, and equipment for aligning multi-fluorescence channel microsphere images, which are used to solve the technical problem that the prior art cannot accurately register the fluorescence images of microspheres in four channels at low cost.

[0068] In order to make the invention objectives, features, and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described below are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0069] Please refer to Figure 2 , Figure 2 Flowchart of steps of a method for aligning multi-fluorescence channel microsphere images provided by embodiments of the present invention.

[0070] A method for aligning multi-fluorescence channel microsphere images provided by the present invention includes:

[0071] Step 201, obtaining fluorescence microsphere images at the same position through multiple fluorescence channels respectively;

[0072] The fluorescence microsphere image refers to a fluorescence image in which bases are bound to microspheres and then the fluorescence signal is excited by a laser of a specific wavelength. In this embodiment, the fluorescence microsphere image corresponds to at least four fluorescent dyes (CY5, FITC, ROX, TRITC), and fluorescence signals are generated respectively through different excitation wavelengths.

[0073] In the embodiment of the present invention, different fluorescences can be switched through a fluorescence microscope, and fluorescence microsphere images can be taken at the same position in the chip covered with microspheres, so as to respectively obtain the fluorescence microsphere images corresponding to each fluorescence channel.

[0074] It should be noted that among the fluorescence microsphere images of multiple fluorescence channels, the microspheres will only generate fluorescence signals in a single fluorescence channel. For the convenience of subsequent image processing and maintaining image clarity, the image size of the fluorescence microsphere image can be set to 1996 * 1652 pixels, and the diameter of a single microsphere is 5 pixels.

[0075] Step 202, perform preprocessing on each fluorescence microsphere image to obtain a fluorescence corrected image;

[0076] After obtaining the fluorescence microsphere images, since the fluorescence microsphere images respectively belong to different fluorescence channels, in order to reduce the influence caused by local shadows and illumination changes in the images and at the same time suppress the interference of noise, preprocessing can be performed on each fluorescence microsphere image to adjust the contrast of the image, standardize the color space of the fluorescence microsphere image, and obtain the fluorescence corrected images corresponding to each fluorescence channel.

[0077] Step 203, perform microsphere region segmentation on each fluorescence corrected image, and calculate the region center points corresponding to each microsphere region;

[0078] After completing the preprocessing of the fluorescence microsphere images to obtain the fluorescence corrected images, microsphere region segmentation can be performed on the fluorescence corrected images, and region center points are calculated for each of the segmented microsphere regions, providing necessary conditions for generating more valuable descriptors by the subsequent multi-scale HOG (Histogram of Oriented Gradients) algorithm. Compared with the common sift feature point extraction, it can largely avoid the generation of invalid feature points or similar feature points, greatly reducing the workload of subsequent HOG descriptor calculation.

[0079] It should be noted that microsphere region segmentation refers to an image processing method for determining the pixel ranges where each microsphere is located in the fluorescence corrected image, thereby determining multiple microsphere regions, and further determining the region center points of each microsphere region according to the peak value of the distance from non-zero pixels to the nearest zero pixel in the region.

[0080] Step 204: Intercept a pixel matrix from the fluorescence-corrected image according to the center points of each region, and extract the corresponding feature vectors;

[0081] In this embodiment, after determining the center points of each microsphere region, a pixel matrix with a preset pixel size can be intercepted from the fluorescence-corrected image according to the center points of each region, the histogram of oriented gradients corresponding to the pixel matrix is calculated using the multi-scale HOG algorithm, and the feature vectors corresponding to the center points of each region are extracted according to the histogram of oriented gradients.

[0082] Step 205: Select any fluorescence-corrected image as the reference image, and pairwise register the remaining fluorescence-corrected images according to the feature vectors, and respectively determine the adjustment offsets;

[0083] In this embodiment, select any fluorescence-corrected image as the reference image, sequentially select the remaining fluorescence-corrected images for registration, compare the corresponding feature vectors of the two, and calculate the adjustment offsets of each fluorescence-corrected image as the data basis for subsequent image alignment correction.

[0084] Among them, the adjustment offset includes a rotation offset and a displacement offset, and the displacement offset includes displacement offsets in the horizontal axis direction and the vertical axis direction.

[0085] Step 206: Correct each fluorescence-corrected image according to the adjustment offset and superimpose it on the reference image to generate a fluorescence channel alignment image.

[0086] After respectively calculating the adjustment offsets between each fluorescence-corrected image and the reference image, each fluorescence-corrected image is respectively translated and rotated to offset the adjustment offset, and then the corrected fluorescence-corrected image is superimposed on the reference image. Since each fluorescence-corrected image belongs to a different fluorescence channel, and a single microsphere only emits light on one fluorescence channel, a fluorescence channel alignment image is generated after image superposition to display the fluorescence excitation states of all microspheres on a single image.

[0087] In the embodiment of the present invention, fluorescence microsphere images are respectively obtained at the same position through multiple fluorescence channels; each fluorescence microsphere image is preprocessed to obtain a fluorescence-corrected image; each fluorescence-corrected image is respectively segmented into microsphere regions, and the center points corresponding to each microsphere region are calculated; a pixel matrix is intercepted from the fluorescence-corrected image according to the center points of each region, and the corresponding feature vectors are extracted; any fluorescence-corrected image is selected as the reference image, and the remaining fluorescence-corrected images are pairwise registered according to the feature vectors, and the adjustment offsets are respectively determined; each fluorescence-corrected image is corrected according to the adjustment offset and superimposed on the reference image to generate a fluorescence channel alignment image. Thus, by combining the gradient direction to perform offset registration on the fluorescence microsphere images, high-precision image alignment is achieved without relying on deep learning.

[0088] Please refer to Figure 3 , Figure 3 , which is a refined step flowchart of a multi-fluorescent channel microsphere image alignment method provided by an embodiment of the present invention, where step 203 is refined.

[0089] A multi-fluorescent channel microsphere image alignment method provided by the present invention includes:

[0090] Step 301, obtaining fluorescence microsphere images at the same position through multiple fluorescent channels;

[0091] In an embodiment of the present invention, the specific implementation process of step 301 is similar to that of step 201, and fluorescence microsphere images are obtained with the number of fluorescent channels being four.

[0092] Step 302, preprocessing each fluorescence microsphere image to obtain a fluorescence corrected image;

[0093] In an example of the present invention, step 302 may include the following sub-steps:

[0094] Performing a power operation on the pixel value of each pixel in each fluorescence microsphere image according to a preset gamma parameter to respectively obtain updated pixel values;

[0095] Generating a fluorescence corrected image using the pixels to which the updated pixel values belong.

[0096] Since not all microspheres in the four-channel fluorescence microsphere image are lit, only some microspheres in each fluorescence microsphere image have strong brightness, which is not conducive to subsequent image processing. Therefore, in this embodiment, gamma correction is used to adjust the image brightness, light up the darker microspheres, and ensure that all microspheres in the four-channel fluorescence image are lit, facilitating subsequent image processing.

[0097] Specifically, a power operation can be performed on the pixel value of each pixel in each fluorescence microsphere image according to a preset gamma parameter to obtain the updated pixel value of each pixel, and its power operation formula is as follows:

[0098]

[0099] Among them, is the updated pixel value of the fluorescence corrected image, is the normalized pixel value of the i-th pixel in the fluorescence microsphere image, is the preset gamma parameter.

[0100] It can be known from the principle of gamma correction that when When in the low gray value region, the dynamic range becomes larger, and thus the image contrast is enhanced; in the high gray value region, the dynamic range becomes smaller, and the image contrast is reduced; since in the fluorescence microsphere image of a certain channel, only a part of the microspheres will light up, and although some microspheres have a certain brightness, they will be relatively dim, so take Selecting appropriate gamma parameters within the range for image correction can obtain better registration results, and its fluorescence corrected image is as shown in Figure 4 image a in

[0101] In the prior art, neither traditional feature point extraction algorithms nor deep learning feature points effectively utilize the unique circular geometric features of the microspheres in the image. The matching results are greatly affected by the image quality and the results are unstable. The present invention designs a feature point extraction algorithm according to the characteristics of the idle microsphere image, which can lock the feature points to the center of the idle microspheres, and can be specifically implemented through the following steps 303-308.

[0102] Step 303: Filter the pixel values of the pixels in each fluorescence corrected image according to a preset quantile threshold to obtain a rough segmentation image of the microsphere region;

[0103] After obtaining multiple fluorescence corrected images by correction, the pixel values of the pixels therein can be sorted from small to large respectively, and the sorted pixel values are filtered according to a preset quantile threshold such as 0.1 to obtain a rough segmentation image of the microsphere region, as shown in Figure 4 image b in

[0104] Step 304: Filter the rough segmentation image of the microsphere region, and combine with the Laplace operator to identify the edge of the microsphere region to obtain an edge image of the microsphere region;

[0105] After obtaining each rough segmentation image of the microsphere region, since its brightness is too high and it is difficult to accurately identify the edges of each microsphere region, each rough segmentation image of the microsphere region can be filtered, and the filtering methods include but are not limited to Gaussian filtering, mean filtering, median filtering, etc.

[0106] After the filtering is completed, the Laplace operator is used to identify the edge of the microsphere region of each filtered rough segmentation image of the microsphere region to extract the microsphere edge information, and an edge image of the microsphere region is obtained as shown in Figure 4 image c in

[0107] Among them, the Laplace operator calculates the curvature of the change of the image pixel value in the x and y directions, and its formula is as follows:

[0108]

[0109] When the Laplacian operator is applied to an image, it produces high response values in the edge regions because the second derivative in these regions is large. By setting a threshold, these high response regions can be detected, thereby identifying the edges and obtaining the edge image of the microsphere region.

[0110] Step 305: Perform a subtraction operation on the roughly segmented image of the microsphere region and the edge image of the microsphere region to generate a segmented image of the microsphere region.

[0111] In this embodiment, to highlight each microsphere region, a subtraction operation can be performed on the roughly segmented image of the microsphere region and the edge image of the microsphere region to generate a segmented image of the microsphere region as Figure 4 shown in image d.

[0112] Step 306: Calculate the pixel distance from each non-zero pixel in each microsphere region within the segmented image of the microsphere region to the nearest zero pixel.

[0113] In an example of the present invention, step 306 may include the following sub-steps:

[0114] Locate the non-zero pixels of each microsphere region from the segmented image of the microsphere region, and initialize the first distance value of the non-zero pixels to a preset maximum value.

[0115] Starting from each non-zero pixel, retrieve the nearest zero pixel in the neighborhood, and calculate the second distance value between the non-zero pixel and the nearest zero pixel.

[0116] Select the minimum value between the second distance value and the first distance value to update the first distance value until all pixels within the microsphere region have been retrieved.

[0117] Normalize each first distance value to obtain the pixel distance from each non-zero pixel in each microsphere region to the nearest zero pixel.

[0118] In this embodiment, to capture the shape and size characteristics of each microsphere region in the segmented image of the microsphere region, each foreground pixel (non-zero pixel) in the image can be replaced with its distance to the nearest background pixel (zero pixel). Specifically, create an output image with the same size as the segmented image of the microsphere region, and initialize all non-zero pixels to a very large value (such as infinity), or locate the non-zero pixels of each microsphere region from the segmented image of the microsphere region, and initialize the first distance value of the non-zero pixels to a preset maximum value. For each non-zero pixel, calculate the second distance value between the non-zero pixel and the nearest zero pixel by iteratively retrieving its nearest zero pixel. The first distance value and the second distance value can be Euclidean distances.

[0119] Update the first distance value by selecting the minimum value between the second distance value and the first distance value until all pixels within the microsphere region have been retrieved, as follows:

[0120]

[0121] Among them, is the distance value of pixel p, d x is the distance in the x direction between pixel p and the zero pixels in its neighborhood, d y is the distance in the y direction between pixel p and the zero pixels in its neighborhood.

[0122] After the update of the first distance value is completed, the first distance values are normalized, which can be achieved by the cv2.normalize() function, so as to obtain the pixel distance from each non-zero pixel in each microsphere region to the nearest zero pixel.

[0123] Step 307: Generate a distance peak map using all pixel distances;

[0124] In this embodiment, after calculating the pixel distance from each non-zero pixel to the nearest zero pixel, these distance values are presented in the form of an image, and the distance peak map is obtained as shown in Figure 4 image e. Generally, the larger the distance value, the brighter the color or the higher the gray value may be in the figure (if it is a gray-scale image). For example, the distance value can be normalized to the range of 0-255 (if it is an 8-bit image), 0 represents the minimum distance, 255 represents the maximum distance, and then each pixel is assigned the corresponding color or gray value according to this value. The generated distance peak map can intuitively display the distance distribution of each pixel in the microsphere region to the non-microsphere region, which is helpful for further analyzing information such as the boundary characteristics and shape of the microspheres.

[0125] Step 308: Calculate the local peak points of each microsphere region in the distance peak map as the corresponding region center points;

[0126] After obtaining the distance peak map, taking each microsphere region as a local region, traversing each pixel, comparing the pixel distance of the central pixel with the values of the surrounding pixels in each microsphere region, and selecting the pixel point corresponding to the central pixel as the region center point. A total of n region center points can be obtained, as shown in Figure 4 image f.

[0127] Step 309: Intercept the pixel matrix from the fluorescence correction image according to each region center point and extract the corresponding feature vectors;

[0128] In an example of the present invention, step 309 may include the following sub-steps S11-S17:

[0129] S11. Center on the center point of each region, intercept a pixel matrix from the fluorescence corrected image according to the first pixel size, and downsample it to the second pixel size to obtain a first pixel matrix;

[0130] S12. Center on the center point of each region, and intercept a second pixel matrix from the fluorescence corrected image according to the second pixel size;

[0131] Since most of the fluorescence microspheres are exactly the same size, and there are only differences in the fluorescence brightness, even if the two target microspheres are not the same microsphere, there may be similar gradient information. It is necessary to refer to the relative distance of the surrounding microspheres to accurately identify whether the two microspheres correspond. And the diameter of a single microsphere is 5 pixels. To ensure the integrity of the information around the target microsphere, a 31×31 pixel matrix is taken on the fluorescence corrected image with the center point of the region as the center point, and the matrix is downsampled to 15×15 to obtain the first pixel matrix a. At the same time, the second pixel matrix b is intercepted according to the second pixel size of 15×15 pixels.

[0132] S13. Divide the gradient direction range into multiple gradient bins according to the preset dimension;

[0133] S14. Calculate the gradient direction corresponding to each pixel in the first pixel matrix and the second pixel matrix;

[0134] Further, S14 may include the following sub-steps:

[0135] Traverse each pixel in the first pixel matrix and the second pixel matrix, and calculate the horizontal axis direction change gradient and the vertical axis direction change gradient corresponding to each respectively;

[0136] Calculate the gradient ratio between the horizontal axis direction change gradient and the vertical axis direction change gradient;

[0137] Calculate the arctangent value of the gradient ratio as the gradient direction.

[0138] In this embodiment, since a microsphere will only light up on one fluorescence channel, and the images on the other three channels will be relatively dim. The intensity of the edge gradient may be different on the images of the same microsphere in different channels, but the edge direction will not change much. Therefore, in this embodiment, only the gradient direction is considered and the amplitude is not considered when generating the gradient histogram. The gradient direction and amplitude of the first pixel matrix and the second pixel matrix are calculated by the HOG algorithm:

[0139]

[0140] The gradient direction is:

[0141]

[0142] Among them, is the change in the y direction, varies in the x direction is the gradient magnitude is the gradient direction

[0143] S15. Classify each gradient direction into the corresponding gradient bin and construct a gradient direction frequency table;

[0144] S16. Generate a first feature vector corresponding to the first pixel matrix and a second feature vector corresponding to the second pixel matrix according to the gradient direction frequency table;

[0145] S17. Concatenate the first feature vector and the second feature vector to generate a feature vector corresponding to the center point of each region.

[0146] In this embodiment, the 180-degree gradient direction can be binned into 18 gradient bins with a length of 10 degrees. By accumulating the gradient directions of each pixel into the corresponding gradient bin, the accumulated value of the gradient magnitude within each gradient bin can be obtained, forming a local gradient direction histogram. Then, find the updated frequency table of the gradient directions of each pixel in the matrix, that is, the frequency of each pixel appearing in different gradient directions. Based on the frequency table, generate a histogram feature vector a and a feature vector b with a length of 18, and combine the feature vector a and the feature vector b to generate a total feature vector with a length of 36. Each region center point generates a feature vector with a length of 36.

[0147] Step 310. Select any fluorescence-corrected image as the reference image, and pairwise register the remaining fluorescence-corrected images according to the feature vectors, and respectively determine the adjustment offsets;

[0148] In an example of the present invention, the adjustment offset includes a rotation offset and a displacement offset; Step 310 may include the following sub-steps:

[0149] Select any fluorescence-corrected image as the reference image;

[0150] Perform pairwise registration of the feature vector of the reference image and the feature vectors corresponding to the remaining fluorescence-corrected images to determine multiple groups of matching feature point pairs;

[0151] Locate the corresponding feature point coordinate pairs from the reference image and the remaining fluorescence-corrected images respectively according to each group of matching feature point pairs;

[0152] Determine the initial rotation offset and the initial displacement offset respectively according to each group of feature point coordinate pairs;

[0153] Select the most frequent value among the initial rotation offsets as the rotation offset;

[0154] Select the most frequent value among the initial displacement offsets as the displacement offset.

[0155] In the embodiments of the present invention, since the above-mentioned feature vector extraction process is performed on each fluorescence-corrected image, for each fluorescence-corrected image, n feature vectors with a length of 36 can be obtained, and each feature vector corresponds to the center point of its respective region. Any fluorescence-corrected image can be selected as the reference image. In this embodiment, the Figure 5 image in the upper left corner on the left is selected as the reference image. The feature vectors of the reference image are pairwise registered with the feature vectors corresponding to the remaining fluorescence-corrected images to determine multiple groups of matching feature point pairs, and then the corresponding feature point coordinate pairs are located from the reference image and the remaining fluorescence-corrected images respectively according to each group of matching feature point pairs. When registering two fluorescence channel images, their feature vectors are respectively taken and compared using the k-nearest neighbor (kNN) matching algorithm to return two optimal matching solutions and their corresponding coordinates in the feature point sequence, and calculate the initial rotation offset, x-direction displacement offset, and y-direction displacement offset of a group of feature points that have completed the matching. Finally, 2*n groups of displacement sequences and 2*n groups of rotation sequences are obtained. First, the initial rotation offset is statistically analyzed, and the rotation amount with the highest occurrence probability is taken as the rotation offset. Then, the displacement sequences are statistically analyzed, and the displacement amount with the highest occurrence probability is taken as the displacement offset.

[0156] Since there are only offsets in the x and y directions between the four-channel microsphere images, after the feature point matching is completed, the offset that appears the most can be statistically analyzed, which is the offset between the four-channel microsphere images. This solution can greatly avoid the errors of the algorithm and accurately find the correct offset.

[0157] Among them, k-nearest neighbor (kNN) matching is a classification algorithm in machine learning. In feature point matching, it is used to find other feature points that are most similar to a feature point. The basic idea is to calculate the distance between feature points (such as Euclidean distance), and then select the k closest points.

[0158] Step 311: Correct each fluorescence-corrected image according to the adjusted offset and superimpose it on the reference image to generate a fluorescence channel alignment image.

[0159] Further, step 311 may include the following sub-steps:

[0160] After rotating according to the rotation offset corresponding to each fluorescence-corrected image, perform translation according to the corresponding displacement offset to obtain the target fluorescence image;

[0161] Superimpose each target fluorescence image on the reference image to generate a fluorescence channel alignment image.

[0162] Such as Figure 5As shown in the four fluorescence correction images on the left, using the fluorescence correction image in the upper left corner as the reference image, after obtaining the adjustment offset between each fluorescence correction image and the reference image, rotate the fluorescence correction image according to the corresponding rotation offset, and perform translation according to the displacement offset to eliminate the adjustment offset and obtain the target fluorescence image.

[0163] Overlay each target fluorescence image onto the reference image to obtain a fluorescence channel alignment image as Figure 5 shown in the right image of.

[0164] In the embodiment of the present invention, fluorescence microsphere images are respectively obtained at the same position through multiple fluorescence channels; preprocess each fluorescence microsphere image to obtain a fluorescence correction image; perform microsphere region segmentation on each fluorescence correction image respectively, and calculate the region center points corresponding to each microsphere region; intercept pixel matrices from the fluorescence correction images according to each region center point, and extract the corresponding feature vectors; select any fluorescence correction image as the reference image, and perform pairwise registration on the remaining fluorescence correction images according to the feature vectors to respectively determine the adjustment offset; correct each fluorescence correction image according to the adjustment offset and overlay it onto the reference image to generate a fluorescence channel alignment image. Thus, by combining the gradient direction to perform offset registration on the fluorescence microsphere images, high-precision image alignment is achieved without relying on deep learning.

[0165] Please refer to Figure 6 , Figure 6 which is a structural block diagram of a multi-fluorescence channel microsphere image alignment device provided by an embodiment of the present invention.

[0166] An embodiment of the present invention provides a multi-fluorescence channel microsphere image alignment device, including:

[0167] An image acquisition module 601, configured to respectively obtain fluorescence microsphere images at the same position through multiple fluorescence channels;

[0168] An image correction module 602, configured to preprocess each fluorescence microsphere image to obtain a fluorescence correction image;

[0169] A region center point positioning module 603, configured to perform microsphere region segmentation on each fluorescence correction image respectively, and calculate the region center points corresponding to each microsphere region;

[0170] A feature vector extraction module 604, configured to intercept pixel matrices from the fluorescence correction images according to each region center point, and extract the corresponding feature vectors;

[0171] An offset determination module 605, configured to select any fluorescence correction image as the reference image, and perform pairwise registration on the remaining fluorescence correction images according to the feature vectors to respectively determine the adjustment offset;

[0172] An image correction module 606, configured to correct each fluorescence-corrected image according to an adjustment offset and superimpose it on a reference image to generate a fluorescence channel alignment image.

[0173] Optionally, the image correction module 602 is specifically configured to:

[0174] Perform a power operation on the pixel value of each pixel in each fluorescence microsphere image according to a preset gamma parameter to obtain updated pixel values respectively;

[0175] Generate a fluorescence-corrected image by using the pixels to which the updated pixel values belong.

[0176] Optionally, the region center point positioning module 603 includes:

[0177] An image rough segmentation sub-module, configured to filter the pixel values of the pixels in each fluorescence-corrected image according to a preset quantile threshold to obtain a rough segmentation image of the microsphere region;

[0178] An edge recognition sub-module, configured to filter the rough segmentation image of the microsphere region and recognize the edge of the microsphere region in combination with a Laplace operator to obtain an edge image of the microsphere region;

[0179] An image segmentation sub-module, configured to perform a subtraction operation on the rough segmentation image of the microsphere region and the edge image of the microsphere region to generate a segmentation image of the microsphere region;

[0180] A pixel distance calculation sub-module, configured to calculate the pixel distance from each non-zero pixel in each microsphere region in the segmentation image of the microsphere region to the nearest zero pixel;

[0181] A distance peak map generation sub-module, configured to generate a distance peak map by using all the pixel distances;

[0182] A region center point determination sub-module, configured to calculate the local peak points of each microsphere region in the distance peak map as the corresponding region center points.

[0183] Optionally, the pixel distance calculation sub-module is specifically configured to:

[0184] Locate the non-zero pixels of each microsphere region in the segmentation image of the microsphere region and initialize the first distance value of the non-zero pixels to a preset maximum value;

[0185] Retrieve the nearest zero pixel in the neighborhood starting from each non-zero pixel and calculate the second distance value between the non-zero pixel and the nearest zero pixel;

[0186] Select the minimum value between the second distance value and the first distance value to update the first distance value until all the pixels in the microsphere region are retrieved;

[0187] Normalize each first distance value to obtain the pixel distance from each non-zero pixel to the nearest zero pixel in each microsphere region.

[0188] Optionally, the feature vector extraction module 604 includes:

[0189] The first pixel matrix intercepting sub-module is used to intercept a pixel matrix from the fluorescence-corrected image centered on the center point of each region according to the first pixel size and downsample it to the second pixel size to obtain the first pixel matrix;

[0190] The second pixel matrix intercepting sub-module is used to intercept a second pixel matrix from the fluorescence-corrected image centered on the center point of each region according to the second pixel size;

[0191] The gradient dividing sub-module is used to divide the gradient direction range into multiple gradient bins according to a preset dimension;

[0192] The gradient direction calculating sub-module is used to calculate the gradient direction corresponding to each pixel in the first pixel matrix and the second pixel matrix respectively;

[0193] The gradient direction frequency table constructing sub-module is used to classify each gradient direction into the corresponding gradient bin and construct a gradient direction frequency table;

[0194] The vector generating sub-module is used to generate a first feature vector corresponding to the first pixel matrix and a second feature vector corresponding to the second pixel matrix according to the gradient direction frequency table;

[0195] The vector splicing sub-module is used to splice the first feature vector and the second feature vector to generate a feature vector corresponding to the center point of each region.

[0196] Optionally, the gradient direction calculating sub-module is specifically used for:

[0197] Traverse each pixel in the first pixel matrix and the second pixel matrix, and calculate the horizontal-axis direction change gradient and the vertical-axis direction change gradient corresponding to each respectively;

[0198] Calculate the gradient ratio between the horizontal-axis direction change gradient and the vertical-axis direction change gradient;

[0199] Calculate the arctangent value of the gradient ratio as the gradient direction.

[0200] Optionally, the adjustment offset includes a rotation offset and a displacement offset; the offset determining module 605 is specifically used for:

[0201] Select any fluorescence-corrected image as the reference image;

[0202] Use the feature vector of the reference image and the feature vectors corresponding to the remaining fluorescence-corrected images for pairwise registration to determine multiple groups of matching feature point pairs;

[0203] Locate the corresponding feature point coordinate pairs from the reference image and the remaining fluorescence corrected images respectively according to each group of matching feature point pairs;

[0204] Determine the initial rotation offset and the initial displacement offset respectively according to each group of feature point coordinate pairs;

[0205] Select the value with the largest quantity in the initial rotation offsets as the rotation offset;

[0206] Select the value with the largest quantity in the initial displacement offsets as the displacement offset.

[0207] Optionally, the image correction module 606 is specifically configured to:

[0208] After rotating according to the rotation offset corresponding to each fluorescence corrected image, perform translation according to the corresponding displacement offset to obtain the target fluorescence image;

[0209] Overlay each of the target fluorescence images onto the reference image to generate a fluorescence channel alignment image.

[0210] An embodiment of the present invention provides an electronic device, including a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the multi-fluorescence channel microsphere image method as described in any embodiment of the present invention.

[0211] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices, modules, and sub-modules can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0212] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces. The indirect coupling or communication connection of the devices or modules can be in an electrical, mechanical, or other form.

[0213] The modules described as separate components may or may not be physically separated. The components displayed as modules may or may not be physical modules, that is, they can be located in one place, or can be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0214] In addition, each functional module in the various embodiments of the present invention may be integrated into a processing module, may exist physically alone for each module, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0215] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for aligning multi-fluorescent channel microsphere images, characterized in that, Including: Obtaining fluorescence microsphere images at the same position through multiple fluorescence channels respectively; Preprocessing each of the fluorescence microsphere images to obtain fluorescence corrected images; Performing microsphere region segmentation on each of the fluorescence corrected images respectively, and calculating region center points corresponding to each microsphere region; Intercepting pixel matrices from the fluorescence corrected images according to the region center points, and extracting corresponding feature vectors; Selecting any one of the fluorescence corrected images as a reference image, and performing pairwise registration on the remaining fluorescence corrected images according to the feature vectors, and respectively determining adjustment offsets; Correcting each of the fluorescence corrected images according to the adjustment offset and superimposing them on the reference image to generate a fluorescence channel alignment image.

2. The method according to claim 1, wherein The preprocessing each of the fluorescence microsphere images to obtain fluorescence corrected images includes: Performing power operation on the pixel value of each pixel in each of the fluorescence microsphere images according to a preset gamma parameter to respectively obtain updated pixel values; Generating fluorescence corrected images by using the pixels belonging to the updated pixel values.

3. The method according to claim 1, wherein The performing microsphere region segmentation on each of the fluorescence corrected images respectively, and calculating region center points corresponding to each microsphere region includes: Filtering the pixel values of the pixels in each of the fluorescence corrected images according to a preset quantile threshold to obtain a rough microsphere region segmentation image; Filtering the rough microsphere region segmentation image, and identifying the microsphere region edge in combination with a Laplacian operator to obtain a microsphere region edge image; Performing a subtraction operation on the rough microsphere region segmentation image and the microsphere region edge image to generate a microsphere region segmentation image; Calculating the pixel distance from each non-zero pixel in each microsphere region in the microsphere region segmentation image to the nearest zero pixel; Generating a distance peak map by using all the pixel distances; Calculating local peak points of each microsphere region in the distance peak map as corresponding region center points.

4. The method according to claim 3, wherein The calculating the pixel distance from each non-zero pixel in each microsphere region in the microsphere region segmentation image to the nearest zero pixel includes: Locating non-zero pixels of each microsphere region from the microsphere region segmentation image, and initializing the first distance value of the non-zero pixels to a preset maximum value; Retrieving the nearest zero pixel in the neighborhood starting from each non-zero pixel, and calculating the second distance value between the non-zero pixel and the nearest zero pixel; Selecting the minimum value between the second distance value and the first distance value to update the first distance value until all pixels in the microsphere region are retrieved; Normalizing each of the first distance values to obtain the pixel distance from each non-zero pixel in each microsphere region to the nearest zero pixel.

5. The method according to claim 1, wherein The intercepting pixel matrices from the fluorescence corrected images according to the region center points, and extracting corresponding feature vectors includes: Taking each region center point as the center, intercepting a pixel matrix from the fluorescence corrected image according to a first pixel size and downsampling it to a second pixel size to obtain a first pixel matrix; Taking each region center point as the center, intercepting a second pixel matrix from the fluorescence corrected image according to the second pixel size; Dividing the gradient direction range into multiple gradient bins according to a preset dimension; Calculate the gradient direction corresponding to each pixel in the first pixel matrix and the second pixel matrix respectively; Classify each of the gradient directions into corresponding gradient bins, and construct a gradient direction frequency table; Generate a first feature vector corresponding to the first pixel matrix and a second feature vector corresponding to the second pixel matrix according to the gradient direction frequency table; Concatenate the first feature vector and the second feature vector to generate a feature vector corresponding to the center point of each region.

6. The method according to claim 5, wherein The calculating the gradient direction corresponding to each pixel in the first pixel matrix and the second pixel matrix respectively includes: Traverse each pixel in the first pixel matrix and the second pixel matrix, and calculate the gradient of change in the horizontal axis direction and the gradient of change in the vertical axis direction corresponding thereto respectively; Calculate the gradient ratio between the gradient of change in the horizontal axis direction and the gradient of change in the vertical axis direction; Calculate the arctangent value of the gradient ratio as the gradient direction.

7. The method according to claim 1, wherein The adjusting the offset includes a rotation offset and a displacement offset; selecting any one of the fluorescence corrected images as a reference image, and pairwise registering the remaining fluorescence corrected images according to the feature vectors, and respectively determining the adjustment offset includes: Select any one of the fluorescence corrected images as a reference image; Perform pairwise registration on the feature vector of the reference image and the feature vectors corresponding to the remaining fluorescence corrected images to determine multiple groups of matching feature point pairs; Locate the corresponding feature point coordinate pairs from the reference image and the remaining fluorescence corrected images respectively according to each group of the matching feature point pairs; Determine an initial rotation offset and an initial displacement offset respectively according to each group of the feature point coordinate pairs; Select the value with the largest number among the initial rotation offsets as the rotation offset; Select the value with the largest number among the initial displacement offsets as the displacement offset.

8. The method according to claim 7, characterized in that The correcting each of the fluorescence corrected images according to the adjustment offset and superimposing them on the reference image to generate a fluorescence channel aligned image includes: After rotating according to the rotation offset corresponding to each of the fluorescence corrected images, perform translation according to the corresponding displacement offset to obtain a target fluorescence image; Superimpose each of the target fluorescence images on the reference image to generate a fluorescence channel aligned image.

9. A multi-fluorescence channel microsphere image alignment device, characterized in that, Includes: An image acquisition module, configured to respectively acquire fluorescence microsphere images at the same position through multiple fluorescence channels; An image correction module, configured to perform preprocessing on each of the fluorescence microsphere images to obtain fluorescence corrected images; A region center point positioning module, configured to respectively perform microsphere region segmentation on each of the fluorescence corrected images, and calculate the region center point corresponding to each microsphere region; A feature vector extraction module, configured to intercept a pixel matrix from the fluorescence corrected image according to each of the region center points, and extract the corresponding feature vector; An offset determination module, configured to select any one of the fluorescence corrected images as a reference image, and pairwise register the remaining fluorescence corrected images according to the feature vectors, and respectively determine the adjustment offset; An image correction module, configured to correct each of the fluorescence corrected images according to the adjustment offset and superimpose them on the reference image to generate a fluorescence channel aligned image.

10. An electronic device, characterized in that, It includes a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, the processor is caused to execute the steps of the multi-fluorescent channel microsphere image method according to any one of claims 1-8.

Images