Multi-fluorescence channel microsphere image alignment method, device and equipment

Abstract

The invention discloses a multi-fluorescent-channel microsphere image alignment method, device and equipment. The method comprises the following steps: respectively acquiring fluorescent microsphere images at the same position through multiple fluorescent channels; preprocessing each fluorescent microsphere image to obtain a fluorescent correction image; performing microsphere region segmentation on each fluorescence correction image, and calculating a region center point corresponding to each microsphere region; intercepting a pixel matrix from the fluorescence correction image according to the central point of each region, and extracting a corresponding feature vector; any one fluorescence correction image is selected as a reference image, the remaining fluorescence correction images are subjected to pairwise registration according to the feature vectors, and adjustment offset is determined respectively; and correcting each fluorescence correction image according to the adjustment offset, and superposing the corrected fluorescence correction images to the reference image to generate a fluorescence channel alignment image. Therefore, high-precision image alignment is realized without depending on deep learning in a mode of performing offset registration on the fluorescent microsphere image in combination with the gradient direction.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multi-fluorescence channel microsphere image alignment method, device and equipment. Background Art

[0002] Microspheres are tiny spherical particles with diameters usually in the micrometer range. In spatial transcriptomics, in order to obtain spatial information, bases are often bound to microspheres, so the microspheres are equipped with fluorescent groups and blocking groups, and then the fluorescent signal is excited by laser, with a total of four fluorescent channels (CY5, FITC, ROX, TRITC). Figure 1 As shown, the microsphere image is captured by a CCD camera.

[0003] Since the bases bound to the microspheres are different, microspheres bound to different bases will light up in different fluorescent channels. Therefore, at the same position, all microspheres will light up only when the four channel images are aligned and superimposed. Therefore, when we want to confirm the binding effect of microspheres and fluorescent dyes in the idling experiment, we need to align and superimpose the four channel data. However, since the four data are taken under different fluorescent channels, there is a certain offset between the four data. The present invention designs a registration algorithm for the image situation to realize the alignment of the four-channel image.

[0004] In medical imaging technology, in order to more accurately judge the condition, it is necessary to register images of different modalities, such as CT and MRI, CT and PET, etc. At present, biomedical image registration algorithms can be divided into traditional registration algorithms and deep learning-based algorithms, but traditional algorithms require a large number of iterations to achieve the set target optimization parameters, which is time-consuming, and is greatly affected by image quality and is not robust. Deep learning registration algorithms often involve large-scale data set production and model training. Limited by the current small data set of microsphere images, the model training cannot fully converge. At the same time, the algorithm calculation cost is relatively high, and it is highly dependent on the hardware platform, resulting in the inability of existing technologies to accurately register microsphere fluorescence images in four channels at low cost. Summary of the invention

[0005] The present invention provides a multi-fluorescence channel microsphere image alignment method, device and equipment, which solves the technical problem that the prior art cannot accurately align the fluorescence images of microspheres in four channels at low cost.

[0006] The first aspect of the present invention provides a multi-fluorescence channel microsphere image alignment method, comprising:

[0007] Acquire fluorescent microsphere images at the same position through multiple fluorescence channels;

[0008] Preprocessing each of the fluorescent microsphere images to obtain a fluorescence correction image;

[0009] Performing microsphere region segmentation on each of the fluorescence-corrected images, and calculating the region center point corresponding to each microsphere region;

[0010] intercepting a pixel matrix from the fluorescence-corrected image according to the center point of each region, and extracting a corresponding feature vector;

[0011] Selecting any one of the fluorescence-corrected images as a reference image, and performing pairwise registration on the remaining fluorescence-corrected images according to the characteristic vectors, and determining adjustment offsets respectively;

[0012] Each of the fluorescence correction images is corrected according to the adjustment offset and superimposed on the reference image to generate a fluorescence channel alignment image.

[0013] Optionally, the preprocessing of each of the fluorescent microsphere images to obtain a fluorescence-corrected image includes:

[0014] Performing a power operation on the pixel value of each pixel in each of the fluorescent microsphere images according to a preset gamma parameter to obtain updated pixel values ​​respectively;

[0015] The pixels to which the updated pixel values ​​belong are used to generate a fluorescence-corrected image.

[0016] Optionally, the segmenting of each of the fluorescence-corrected images into microsphere regions and calculating the region center points corresponding to each of the microsphere regions include:

[0017] Filtering the pixel values ​​of the pixels in each of the fluorescence-corrected images according to a preset quantile threshold to obtain a coarse segmentation image of the microsphere region;

[0018] Filtering the coarse segmentation image of the microsphere region, and identifying the edge of the microsphere region in combination with the Laplace operator to obtain an edge image of the microsphere region;

[0019] Performing a subtraction operation on the microsphere region rough segmentation image and the microsphere region edge image to generate a microsphere region segmentation image;

[0020] Calculating the pixel distance from each non-zero pixel to the nearest zero pixel in each microsphere region in the microsphere region segmentation image;

[0021] Using all the pixel distances, generating a distance peak map;

[0022] In the distance peak map, the local peak point of each microsphere region is calculated as the corresponding region center point.

[0023] Optionally, the calculating the pixel distance from each non-zero pixel of each microsphere region in the microsphere region segmentation image to the nearest zero pixel comprises:

[0024] Locating non-zero pixels of each microsphere region from the microsphere region segmentation image, and initializing the first distance value of the non-zero pixels to a preset maximum value;

[0025] Taking each of the non-zero pixels as a starting point, searching for the nearest zero pixel in the neighborhood, and calculating a second distance value between the non-zero pixel and the nearest zero pixel;

[0026] Selecting the minimum value between the second distance value and the first distance value to update the first distance value until all pixels in the microsphere area are retrieved;

[0027] Each of the first distance values ​​is normalized to obtain a pixel distance from each non-zero pixel to the nearest zero pixel in each microsphere region.

[0028] Optionally, intercepting a pixel matrix from the fluorescence-corrected image according to the center point of each region and extracting a corresponding feature vector includes:

[0029] Taking the center point of each region as the center, intercepting a pixel matrix from the fluorescence-corrected image according to a first pixel size, and downsampling to a second pixel size to obtain a first pixel matrix;

[0030] Taking the center point of each region as the center, intercepting a second pixel matrix from the fluorescence-corrected image according to the second pixel scale;

[0031] Divide the gradient direction range into multiple gradient bins according to the preset dimension;

[0032] Calculating the gradient direction corresponding to each pixel in the first pixel matrix and the second pixel matrix respectively;

[0033] Classifying each of the gradient directions into corresponding gradient bins, and constructing a gradient direction frequency table;

[0034] generating a first eigenvector corresponding to the first pixel matrix and a second eigenvector corresponding to the second pixel matrix according to the gradient direction frequency table;

[0035] The first feature vector and the second feature vector are concatenated to generate feature vectors corresponding to the center points of each region.

[0036] Optionally, the calculating the gradient direction corresponding to each pixel in the first pixel matrix and the second pixel matrix includes:

[0037] Traversing each pixel in the first pixel matrix and the second pixel matrix, and calculating the corresponding horizontal axis direction change gradient and vertical axis direction change gradient respectively;

[0038] Calculating the gradient ratio between the gradient of the horizontal axis and the gradient of the vertical axis;

[0039] The arc tangent value of the gradient ratio is calculated as the gradient direction.

[0040] Optionally, the adjustment offset includes a rotation offset and a displacement offset; the selecting any one of the fluorescence correction images as a reference image and performing pairwise registration on the remaining fluorescence correction images according to the feature vector to respectively determine the adjustment offsets includes:

[0041] Selecting any one of the fluorescence-corrected images as a reference image;

[0042] Performing pairwise registration of the feature vectors of the reference image and feature vectors corresponding to the remaining fluorescence-corrected images to determine a plurality of sets of matching feature point pairs;

[0043] According to each group of matching feature point pairs, corresponding feature point coordinate pairs are located from the reference image and the remaining fluorescence-corrected image respectively;

[0044] Determine the initial rotation offset and the initial displacement offset according to each group of feature point coordinate pairs;

[0045] Selecting the largest value among the initial rotation offsets as the rotation offset;

[0046] The value with the largest number among the initial displacement offsets is selected as the displacement offset.

[0047] Optionally, the correcting each of the fluorescence correction images according to the adjustment offset and superimposing them onto the reference image to generate a fluorescence channel alignment image includes:

[0048] After rotating according to the rotation offset corresponding to each of the fluorescence correction images, translating according to the corresponding displacement offset to obtain a target fluorescence image;

[0049] Each of the target fluorescence images is superimposed on the reference image to generate a fluorescence channel alignment image.

[0050] A second aspect of the present invention provides a multi-fluorescence channel microsphere image alignment device, comprising:

[0051] An image acquisition module, used to acquire fluorescent microsphere images at the same position through multiple fluorescence channels;

[0052] An image correction module, used for preprocessing each of the fluorescent microsphere images to obtain a fluorescence correction image;

[0053] A regional center point positioning module is used to segment each of the fluorescence correction images into microsphere regions and calculate the regional center points corresponding to each microsphere region;

[0054] A feature vector extraction module, used to intercept a pixel matrix from the fluorescence correction image according to the center point of each region, and extract a corresponding feature vector;

[0055] An offset determination module, used for selecting any of the fluorescence-corrected images as a reference image, and performing pairwise registration on the remaining fluorescence-corrected images according to the feature vectors, and respectively determining adjustment offsets;

[0056] The image correction module is used to correct each of the fluorescence correction images according to the adjustment offset and superimpose them on the reference image to generate a fluorescence channel alignment image.

[0057] The third aspect of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multi-fluorescence channel microsphere imaging method as described in any one of the first aspect of the present invention.

[0058] It can be seen from the above technical solutions that the present invention has the following advantages:

[0059] The present invention obtains fluorescent microsphere images at the same position through multiple fluorescent channels; pre-processes each fluorescent microsphere image to obtain a fluorescent correction image; performs microsphere region segmentation on each fluorescent correction image, and calculates the regional center points corresponding to each microsphere region; intercepts the pixel matrix from the fluorescent correction image according to each regional center point, and extracts the corresponding eigenvector; selects any fluorescent correction image as a reference image, and performs pairwise registration of the remaining fluorescent correction images according to the eigenvector, and determines the adjustment offset respectively; corrects each fluorescent correction image according to the adjustment offset and superimposes it on the reference image to generate a fluorescent channel alignment image. Thus, by combining the gradient direction to offset and register the fluorescent microsphere image, high-precision image alignment can be achieved without relying on deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0061] Figure 1This is a schematic diagram of the fluorescent microsphere image obtained by four fluorescent channels in an embodiment of the present invention;

[0062] Figure 2 A flowchart of a multi-fluorescence channel microsphere image alignment method provided by an embodiment of the present invention;

[0063] Figure 3 A flowchart of the detailed steps of a multi-fluorescence channel microsphere image alignment method provided by an embodiment of the present invention;

[0064] Figure 4 A schematic diagram of image changes during the process of determining the center point of a region provided by an embodiment of the present invention;

[0065] Figure 5 A schematic diagram of a multi-fluorescence channel microsphere image alignment process provided by an embodiment of the present invention;

[0066] Figure 6 A structural block diagram of a multi-fluorescence channel microsphere image alignment device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] Due to the existence of various reasons such as switching fluorescence, machine jitter, fluorescence reflection, etc., there is a certain deviation between the four-channel fluorescent microsphere images. The mechanical design principle of the microscopic imaging system determines that the deviation is mainly rotation deviation and displacement deviation in the x-direction and y-direction. To this end, the embodiment of the present invention provides a multi-fluorescence channel microsphere image alignment method, device and equipment to solve the technical problem that the prior art cannot accurately align the fluorescence images of microspheres in four channels at low cost.

[0068] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] See also Figure 2 , Figure 2 A flowchart of the steps of a multi-fluorescence channel microsphere image alignment method provided by an embodiment of the present invention.

[0070] The present invention provides a multi-fluorescence channel microsphere image alignment method, comprising:

[0071] Step 201, acquiring fluorescent microsphere images at the same position through multiple fluorescence channels;

[0072] The fluorescent microsphere image refers to a fluorescent image of a fluorescent signal excited by a laser of a specific wavelength after the base is bound to the microsphere. In this embodiment, the fluorescent microsphere image corresponds to at least four fluorescent dyes (CY5, FITC, ROX, TRITC), and generates fluorescent signals respectively through different excitation wavelengths.

[0073] In the embodiment of the present invention, the fluorescence microscope can be switched to different fluorescences to capture the fluorescent microsphere image at the same position in the chip covered with microspheres, so as to obtain the fluorescent microsphere image corresponding to each fluorescence channel respectively.

[0074] It should be noted that in the fluorescent microsphere image of multiple fluorescent channels, the microsphere will only generate fluorescent signals in a single fluorescent channel. To facilitate subsequent image processing and maintain image clarity, the image size of the fluorescent microsphere image can be set to 1996*1652 pixels, and the diameter of a single microsphere is 5 pixels.

[0075] Step 202, preprocessing each fluorescent microsphere image to obtain a fluorescence correction image;

[0076] After obtaining the fluorescent microsphere image, since each fluorescent microsphere image belongs to a different fluorescent channel, in order to reduce the impact of local shadows and illumination changes in the image and suppress noise interference, each fluorescent microsphere image can be preprocessed to adjust the image contrast and standardize the fluorescent microsphere image in color space to obtain a fluorescent correction image corresponding to each fluorescent channel.

[0077] Step 203, performing microsphere region segmentation on each fluorescence correction image, and calculating the region center point corresponding to each microsphere region;

[0078] After completing the preprocessing of the fluorescent microsphere image to obtain the fluorescent corrected image, the fluorescent corrected image can be segmented into microsphere regions, and the region center points can be calculated for each segmented microsphere region, which provides the necessary conditions for the subsequent multi-scale HOG (Histogram of Oriented Gradients) algorithm to generate a more valuable reference descriptor. Compared with the common SIFT feature point extraction, it can avoid the generation of a large number of invalid feature points or similar feature points, greatly reducing the workload of subsequent HOG descriptor calculations.

[0079] It should be noted that microsphere region segmentation refers to an image processing method that determines the pixel range of each microsphere in the fluorescence correction image, thereby determining multiple microsphere regions, and further determining the regional center point of each microsphere region according to the peak value of the distance from the non-zero pixel to the nearest zero pixel in the region.

[0080] Step 204, intercepting a pixel matrix from the fluorescence correction image according to the center point of each region, and extracting the corresponding feature vector;

[0081] In this embodiment, after determining the regional center point corresponding to each microsphere area, a pixel matrix of a preset pixel size can be intercepted from the fluorescence correction image according to each regional center point, and the gradient direction histogram corresponding to the pixel matrix can be calculated using the multi-scale HOG algorithm. According to the gradient direction histogram, the feature vector corresponding to each regional center point is extracted.

[0082] Step 205, selecting any fluorescence-corrected image as a reference image, and performing pairwise registration on the remaining fluorescence-corrected images according to the feature vectors, and determining adjustment offsets respectively;

[0083] In this embodiment, any fluorescence corrected image is selected as the reference image, and the remaining fluorescence corrected images are selected in turn for registration, and the corresponding feature vectors of the two are compared to calculate the adjustment offset of each fluorescence corrected image as the data basis for subsequent image alignment correction.

[0084] The adjustment offset includes a rotation offset and a displacement offset, and the displacement offset includes a displacement offset in a horizontal axis direction and a vertical axis direction.

[0085] Step 206 , correct each fluorescence correction image according to the adjusted offset and superimpose it on the reference image to generate a fluorescence channel alignment image.

[0086] After calculating the adjustment offset between each fluorescence correction image and the reference image, each fluorescence correction image is translated and rotated to offset the adjustment offset, and then the corrected fluorescence correction image is superimposed on the reference image. Since each fluorescence correction image belongs to a different fluorescence channel and a single microsphere only emits light in one fluorescence channel, a fluorescence channel alignment image is generated after the images are superimposed to display the fluorescence excitation state of all microspheres in a single image.

[0087] In an embodiment of the present invention, fluorescent microsphere images are obtained at the same position through multiple fluorescent channels; each fluorescent microsphere image is preprocessed to obtain a fluorescent correction image; each fluorescent correction image is segmented into microsphere regions, and the regional center points corresponding to each microsphere region are calculated; a pixel matrix is ​​intercepted from the fluorescent correction image according to each regional center point, and the corresponding eigenvector is extracted; any fluorescent correction image is selected as a reference image, and the remaining fluorescent correction images are registered in pairs according to the eigenvector, and the adjustment offset is determined respectively; each fluorescent correction image is corrected according to the adjustment offset and superimposed on the reference image to generate a fluorescent channel alignment image. Thus, by combining the gradient direction to offset and register the fluorescent microsphere image, high-precision image alignment can be achieved without relying on deep learning.

[0088] See also Figure 3 , Figure 3 This is a flowchart of the refinement steps of a multi-fluorescence channel microsphere image alignment method provided by an embodiment of the present invention, wherein the above step 203 is refined.

[0089] The present invention provides a multi-fluorescence channel microsphere image alignment method, comprising:

[0090] Step 301, acquiring fluorescent microsphere images at the same position through multiple fluorescence channels;

[0091] In the embodiment of the present invention, the specific implementation process of step 301 is similar to step 201, and the fluorescent microsphere image is acquired with four fluorescent channels.

[0092] Step 302, preprocessing each fluorescent microsphere image to obtain a fluorescence correction image;

[0093] In one example of the present invention, step 302 may include the following sub-steps:

[0094] Performing a power operation on the pixel value of each pixel in each fluorescent microsphere image according to a preset gamma parameter to obtain updated pixel values;

[0095] The fluorescence-corrected image is generated by updating the pixel values ​​of the pixels.

[0096] Since not all microspheres in the four-channel fluorescent microsphere image are lit, only some microspheres in each fluorescent microsphere image are bright, which is not conducive to subsequent image processing. Therefore, this embodiment uses gamma correction to adjust the image brightness, illuminates the darker microspheres, and ensures that all microspheres in the four-channel fluorescent image are lit, which is convenient for subsequent image processing.

[0097] Specifically, the pixel value of each pixel in each fluorescent microsphere image can be exponentially operated by a preset gamma parameter to obtain an updated pixel value of each pixel. The exponential operation formula is as follows:

[0098]

[0099] in, is the updated pixel value of the fluorescence-corrected image, is the normalized pixel value of the ith pixel in the fluorescent microsphere image, is the preset gamma parameter.

[0100] According to the principle of gamma correction, when When the gray value is low, the dynamic range becomes larger, and the image contrast is enhanced; in the high gray value area, the dynamic range becomes smaller, and the image contrast is reduced; because in the fluorescent microsphere image of a certain channel, only a part of the microspheres will light up, and some microspheres will be relatively dim although they have a certain brightness, so Selecting appropriate gamma parameters within the range to correct the image can obtain better registration results. The fluorescence correction image is as follows: Figure 4 As shown in image a.

[0101] In the prior art, neither the traditional feature point extraction algorithm nor the deep learning feature point extraction algorithm effectively utilizes the unique circular geometric features of the microspheres in the image. The matching results are greatly affected by the image quality and the results are unstable. The present invention designs a feature point extraction algorithm based on the characteristics of the idling microsphere image, which can lock the feature points to the center of the idling microsphere. This can be achieved through the following steps 303-308.

[0102] Step 303, filtering the pixel values ​​of the pixels in each fluorescence correction image according to a preset quantile threshold to obtain a coarse segmentation image of the microsphere region;

[0103] After correcting multiple fluorescence corrected images, the pixel values ​​of the pixels therein can be sorted from small to large, and the sorted pixel values ​​can be filtered according to a preset quantile threshold such as 0.1 to obtain a rough segmentation image of the microsphere region, such as Figure 4 As shown in b.

[0104] Step 304, filtering the microsphere region coarse segmentation image, and identifying the microsphere region edge in combination with the Laplace operator to obtain the microsphere region edge image;

[0105] After obtaining each coarse segmentation image of the microsphere region, it is difficult to accurately identify the edge of each microsphere region due to its high brightness. Therefore, each coarse segmentation image of the microsphere region can be filtered. The filtering method includes but is not limited to Gaussian filtering, mean filtering, median filtering, etc.

[0106] After filtering is completed, the Laplace operator is used to identify the microsphere region edge of each microsphere region rough segmentation image after filtering to extract the microsphere edge information and obtain the microsphere region edge image as shown in Figure 4 As shown in c.

[0107] The Laplace operator calculates the curvature of the image pixel values ​​in the x and y directions, and its formula is as follows:

[0108]

[0109] When the Laplacian operator is applied to an image, it produces high response values ​​in the edge regions because the second-order derivatives in these regions are large. By setting a threshold, these high response regions can be detected, thereby identifying the edges and obtaining an edge image of the microsphere region.

[0110] Step 305, performing a subtraction operation on the microsphere region rough segmentation image and the microsphere region edge image to generate a microsphere region segmentation image;

[0111] In this embodiment, in order to highlight each microsphere region, a subtraction operation can be performed between the microsphere region rough segmentation image and the microsphere region edge image to generate a microsphere region segmentation image such as Figure 4 As shown in the image in middle d.

[0112] Step 306, calculating the pixel distance from each non-zero pixel of each microsphere region in the microsphere region segmentation image to the nearest zero pixel;

[0113] In one example of the present invention, step 306 may include the following sub-steps:

[0114] Locate non-zero pixels of each microsphere region from the microsphere region segmentation image, and initialize the first distance value of the non-zero pixels to a preset maximum value;

[0115] Taking each non-zero pixel as a starting point, searching for the nearest zero pixel in the neighborhood, and calculating the second distance value between the non-zero pixel and the nearest zero pixel;

[0116] Selecting the minimum value between the second distance value and the first distance value to update the first distance value until all pixels in the microsphere area are retrieved;

[0117] Each first distance value is normalized to obtain a pixel distance from each non-zero pixel to the nearest zero pixel in each microsphere region.

[0118] In this embodiment, in order to capture the shape and size characteristics of each microsphere region in the microsphere region segmentation image, each foreground pixel (non-zero pixel) in the image can be replaced by its distance to the nearest background pixel (zero pixel). Specifically, an output image of the same size as the microsphere region segmentation image is created, and all non-zero pixels are initialized to a large value (e.g., infinity), or the non-zero pixels of each microsphere region are located from the microsphere region segmentation image, and the first distance value of the non-zero pixel is initialized to a preset maximum value. For each non-zero pixel, the nearest zero pixel closest to it is iteratively retrieved to calculate the second distance value between the non-zero pixel and the nearest zero pixel. The first distance value and the second distance value can be Euclidean distance.

[0119] The first distance value is updated by selecting the minimum value between the second distance value and the first distance value until all pixels in the microsphere area are retrieved, as shown below:

[0120]

[0121] in, is the distance value of pixel p, d x is the distance between pixel p and the zero pixel in its neighborhood in the x direction, d y is the distance between pixel p and the zero pixel in its neighborhood in the y direction.

[0122] After the first distance values ​​are updated, each first distance value is normalized, which can be achieved by using the cv2.normalize() function, so as to obtain the pixel distance from each non-zero pixel to the nearest zero pixel in each microsphere region.

[0123] Step 307, using all pixel distances to generate a distance peak map;

[0124] In this embodiment, after calculating the pixel distance from each non-zero pixel to the nearest zero pixel, these distance values ​​are presented in the form of an image, and a distance peak image is obtained. Figure 4 As shown in the image e. Usually, the larger the distance value, the brighter the color or the higher the grayscale value (if it is a grayscale image). For example, the distance value can be normalized to the range of 0-255 (if it is an 8-bit image), 0 represents the minimum distance, 255 represents the maximum distance, and then each pixel is assigned a corresponding color or grayscale according to this value. The distance peak map generated in this way can intuitively show the distance distribution of each pixel in the microsphere area to the non-microsphere area, which is helpful for further analysis of the boundary characteristics, shape and other information of the microsphere.

[0125] Step 308, calculating the local peak point of each microsphere region in the distance peak map as the corresponding region center point;

[0126] After obtaining the distance peak map, each microsphere area is taken as the local area, and each pixel is traversed. In each microsphere area, the pixel distance of the central pixel is compared with the value of the surrounding pixels, and the pixel point corresponding to the central pixel is selected as the regional center point. A total of n regional center points can be obtained, such as Figure 4 As shown in the middle f image.

[0127] Step 309, intercepting a pixel matrix from the fluorescence correction image according to the center point of each region, and extracting the corresponding feature vector;

[0128] In an example of the present invention, step 309 may include the following sub-steps S11-S17:

[0129] S11, taking the center point of each region as the center, intercepting a pixel matrix from the fluorescence correction image according to a first pixel size, and downsampling to a second pixel size to obtain a first pixel matrix;

[0130] S12, taking the center point of each region as the center, intercepting a second pixel matrix from the fluorescence correction image according to a second pixel ruler;

[0131] Since most fluorescent microspheres are exactly the same size, with only a difference in fluorescence brightness, the two target microspheres may have similar gradient information even if they are not the same microsphere. It is necessary to refer to the relative distance of the surrounding microspheres to accurately identify whether the two microspheres correspond. The diameter of a single microsphere is 5 pixels. To ensure the integrity of the information around the target microsphere, a 31*31 pixel matrix is ​​taken on the fluorescence correction image with the center point of the region as the center point, and the matrix is ​​downsampled to 15*15 to obtain the first pixel matrix a. At the same time, the second pixel matrix b is intercepted according to the second pixel size of 15*15 pixels.

[0132] S13, dividing the gradient direction range into a plurality of gradient bins according to a preset dimension;

[0133] S14, calculating the gradient direction corresponding to each pixel in the first pixel matrix and the second pixel matrix respectively;

[0134] Further, S14 may include the following sub-steps:

[0135] Traversing each pixel in the first pixel matrix and the second pixel matrix, and calculating the corresponding horizontal axis direction change gradient and vertical axis direction change gradient respectively;

[0136] Calculate the gradient ratio between the gradient of the horizontal axis and the gradient of the vertical axis;

[0137] Calculate the inverse tangent of the gradient ratio as the gradient direction.

[0138] In this embodiment, since a microsphere will only light up in one fluorescent channel, the images in the other three channels will be relatively dim. The edge gradient intensities of the same microsphere in different channel images may be different, but the edge direction will not change much. Therefore, this embodiment only considers the gradient direction and ignores the amplitude when generating the gradient histogram, and calculates the gradient direction and amplitude of the first pixel matrix and the second pixel matrix through the HOG algorithm:

[0139]

[0140] The gradient direction is:

[0141]

[0142] in, is the change in the y direction, is the change in the x direction, is the gradient amplitude, is the gradient direction.

[0143] S15, classifying each gradient direction into a corresponding gradient bin, and constructing a gradient direction frequency table;

[0144] S16, generating a first eigenvector corresponding to the first pixel matrix and a second eigenvector corresponding to the second pixel matrix according to the gradient direction frequency table;

[0145] S17, concatenate the first eigenvector and the second eigenvector to generate a eigenvector corresponding to the center point of each region.

[0146] In this embodiment, the 180-degree gradient direction can be binned into 18 gradient bins with a length of 10 degrees. The gradient direction of each pixel is accumulated to the corresponding gradient bin, and the gradient amplitude accumulation value in each gradient bin can be obtained to form a local gradient direction histogram. Then find the gradient direction update frequency table of each pixel in the matrix, that is, the frequency of each pixel appearing in different gradient directions. Based on the frequency table, a histogram feature vector a and a feature vector b with a length of 18 are generated, and the feature vector a and the feature vector b are combined to generate a total feature vector with a length of 36, and a feature vector with a length of 36 is generated for each regional center point.

[0147] Step 310, selecting any fluorescence-corrected image as a reference image, and performing pairwise registration on the remaining fluorescence-corrected images according to the feature vectors, and determining adjustment offsets respectively;

[0148] In one example of the present invention, adjusting the offset includes a rotation offset and a displacement offset; step 310 may include the following sub-steps:

[0149] Select any fluorescence-corrected image as the reference image;

[0150] The feature vectors of the reference image are used to perform pairwise registration with the feature vectors corresponding to the remaining fluorescence-corrected images to determine multiple sets of matching feature point pairs;

[0151] According to each group of matching feature point pairs, corresponding feature point coordinate pairs are located from the reference image and the remaining fluorescence-corrected image respectively;

[0152] According to each set of feature point coordinate pairs, an initial rotation offset and an initial displacement offset are determined respectively;

[0153] Select the value with the largest number among the initial rotation offsets as the rotation offset;

[0154] The value with the largest number among the initial displacement offsets is selected as the displacement offset.

[0155] In the embodiment of the present invention, since the above-mentioned feature vector extraction process is performed on each fluorescence correction image, for each fluorescence correction image, n feature vectors with a length of 36 can be obtained, and each feature vector corresponds to the center point of its own region. Any fluorescence correction image can be selected as the reference image. In this embodiment, Figure 5 The upper left corner image on the left is used as the reference image. The feature vector of the reference image is used to perform pairwise registration with the feature vectors corresponding to the remaining fluorescence correction images to determine multiple sets of matching feature point pairs. Then, the corresponding feature point coordinate pairs are located from the reference image and the remaining fluorescence correction images according to each set of matching feature point pairs. When the two fluorescence channel images are registered, their feature vectors are taken and compared using the k-nearest neighbor (kNN) matching algorithm. The two optimal matching solutions are returned and correspond to their coordinates in the feature point sequence. The initial rotation offset, x-direction displacement offset, and y-direction displacement offset of a set of feature points that have been matched are calculated. Finally, 2*n sets of displacement sequences and 2*n sets of rotation sequences are obtained. The initial rotation offset is first counted, and the rotation with the highest probability is taken as the rotation offset. Then, the displacement sequence is counted, and the displacement with the highest probability is taken as the displacement offset.

[0156] Since there are only offsets in the x and y directions between the four-channel microsphere images, after completing the feature point matching, the most common offset can be counted as the offset between the four-channel microsphere images. This solution can avoid the algorithm error to a great extent and accurately find the correct offset.

[0157] Among them, k-nearest neighbor (kNN) matching is a classification algorithm in machine learning. In feature point matching, it is used to find other feature points that are most similar to a feature point. The basic idea is to calculate the distance between feature points (such as Euclidean distance) and then select the k points closest to it.

[0158] Step 311 , correct each fluorescence correction image according to the adjustment offset and superimpose it on the reference image to generate a fluorescence channel alignment image.

[0159] Further, step 311 may include the following sub-steps:

[0160] After rotating according to the rotation offset corresponding to each fluorescence correction image, the image is translated according to the corresponding displacement offset to obtain a target fluorescence image;

[0161] Each target fluorescence image is superimposed on the reference image to generate a fluorescence channel aligned image.

[0162] like Figure 5As shown in the four fluorescence correction images on the left, the fluorescence correction image in the upper left corner is used as the reference image. After obtaining the adjustment offset between each fluorescence correction image and the reference image, the fluorescence correction image is rotated according to the corresponding rotation offset and translated according to the displacement offset to eliminate the adjustment offset and obtain the target fluorescence image.

[0163] The fluorescence images of each target are superimposed on the reference image to obtain the fluorescence channel alignment image as shown in Figure 5 as shown in the image on the right.

[0164] In an embodiment of the present invention, fluorescent microsphere images are obtained at the same position through multiple fluorescent channels; each fluorescent microsphere image is preprocessed to obtain a fluorescent correction image; each fluorescent correction image is segmented into microsphere regions, and the regional center points corresponding to each microsphere region are calculated; a pixel matrix is ​​intercepted from the fluorescent correction image according to each regional center point, and the corresponding eigenvector is extracted; any fluorescent correction image is selected as a reference image, and the remaining fluorescent correction images are registered in pairs according to the eigenvector, and the adjustment offset is determined respectively; each fluorescent correction image is corrected according to the adjustment offset and superimposed on the reference image to generate a fluorescent channel alignment image. Thus, by combining the gradient direction to offset and register the fluorescent microsphere image, high-precision image alignment can be achieved without relying on deep learning.

[0165] See also Figure 6 , Figure 6 A structural block diagram of a multi-fluorescence channel microsphere image alignment device provided by an embodiment of the present invention.

[0166] The embodiment of the present invention provides a multi-fluorescence channel microsphere image alignment device, comprising:

[0167] An image acquisition module 601 is used to acquire fluorescent microsphere images at the same position through multiple fluorescent channels;

[0168] An image correction module 602 is used to pre-process each fluorescent microsphere image to obtain a fluorescent correction image;

[0169] The regional center point positioning module 603 is used to segment each fluorescence correction image into microsphere regions and calculate the regional center points corresponding to each microsphere region;

[0170] A feature vector extraction module 604 is used to intercept a pixel matrix from the fluorescence correction image according to the center point of each region and extract the corresponding feature vector;

[0171] An offset determination module 605 is used to select any fluorescence correction image as a reference image, and perform pairwise registration on the remaining fluorescence correction images according to the feature vectors to determine the adjustment offsets respectively;

[0172] The image correction module 606 is used to correct each fluorescence correction image according to the adjustment offset and superimpose it on the reference image to generate a fluorescence channel alignment image.

[0173] Optionally, the image correction module 602 is specifically used for:

[0174] Performing a power operation on the pixel value of each pixel in each fluorescent microsphere image according to a preset gamma parameter to obtain updated pixel values;

[0175] The fluorescence-corrected image is generated by updating the pixel values ​​of the pixels.

[0176] Optionally, the regional center point positioning module 603 includes:

[0177] The image coarse segmentation submodule is used to filter the pixel values ​​of the pixels in each fluorescence correction image according to a preset quantile threshold to obtain a coarse segmentation image of the microsphere area;

[0178] The edge recognition submodule is used to filter the rough segmentation image of the microsphere region and identify the edge of the microsphere region in combination with the Laplace operator to obtain the edge image of the microsphere region;

[0179] An image segmentation submodule is used to perform a subtraction operation using a coarse segmentation image of a microsphere region and an edge image of a microsphere region to generate a segmentation image of a microsphere region;

[0180] A pixel distance calculation submodule is used to calculate the pixel distance from each non-zero pixel to the nearest zero pixel in each microsphere region in the microsphere region segmentation image;

[0181] A distance peak map generation submodule is used to generate a distance peak map using all pixel distances;

[0182] The regional center point determination submodule is used to calculate the local peak point of each microsphere region in the distance peak map as the corresponding regional center point.

[0183] Optionally, the pixel distance calculation submodule is specifically used for:

[0184] Locate non-zero pixels of each microsphere region from the microsphere region segmentation image, and initialize the first distance value of the non-zero pixels to a preset maximum value;

[0185] Taking each non-zero pixel as a starting point, searching for the nearest zero pixel in the neighborhood, and calculating the second distance value between the non-zero pixel and the nearest zero pixel;

[0186] Selecting the minimum value between the second distance value and the first distance value to update the first distance value until all pixels in the microsphere area are retrieved;

[0187] Each first distance value is normalized to obtain a pixel distance from each non-zero pixel to the nearest zero pixel in each microsphere region.

[0188] Optionally, the feature vector extraction module 604 includes:

[0189] A first pixel matrix interception submodule is used to intercept a pixel matrix from the fluorescence correction image according to a first pixel size with the center point of each region as the center, and downsample it to a second pixel size to obtain a first pixel matrix;

[0190] A second pixel matrix interception submodule, used for intercepting a second pixel matrix from the fluorescence correction image according to a second pixel scale with the center point of each area as the center;

[0191] A gradient partitioning submodule is used to divide the gradient direction range into multiple gradient bins according to a preset dimension;

[0192] A gradient direction calculation submodule, used to calculate the gradient direction corresponding to each pixel in the first pixel matrix and the second pixel matrix respectively;

[0193] A gradient direction frequency table construction submodule is used to classify each gradient direction into the corresponding gradient bin and construct a gradient direction frequency table;

[0194] A vector generation submodule, used to generate a first eigenvector corresponding to the first pixel matrix and a second eigenvector corresponding to the second pixel matrix according to the gradient direction frequency table;

[0195] The vector concatenation submodule is used to concatenate the first eigenvector and the second eigenvector to generate a eigenvector corresponding to the center point of each region.

[0196] Optionally, the gradient direction calculation submodule is specifically used for:

[0197] Traversing each pixel in the first pixel matrix and the second pixel matrix, and calculating the corresponding horizontal axis direction change gradient and vertical axis direction change gradient respectively;

[0198] Calculate the gradient ratio between the gradient of the horizontal axis and the gradient of the vertical axis;

[0199] Calculate the inverse tangent of the gradient ratio as the gradient direction.

[0200] Optionally, the adjustment offset includes a rotation offset and a displacement offset; the offset determination module 605 is specifically used to:

[0201] Select any fluorescence-corrected image as the reference image;

[0202] The feature vectors of the reference image are used to perform pairwise registration with the feature vectors corresponding to the remaining fluorescence-corrected images to determine multiple sets of matching feature point pairs;

[0203] According to each group of matching feature point pairs, corresponding feature point coordinate pairs are located from the reference image and the remaining fluorescence-corrected image respectively;

[0204] According to each set of feature point coordinate pairs, an initial rotation offset and an initial displacement offset are determined respectively;

[0205] Select the value with the largest number among the initial rotation offsets as the rotation offset;

[0206] The value with the largest number among the initial displacement offsets is selected as the displacement offset.

[0207] Optionally, the image correction module 606 is specifically used for:

[0208] After rotating according to the rotation offset corresponding to each fluorescence correction image, the image is translated according to the corresponding displacement offset to obtain a target fluorescence image;

[0209] Each of the target fluorescence images is superimposed on the reference image to generate a fluorescence channel alignment image.

[0210] An embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multi-fluorescence channel microsphere imaging method as described in any embodiment of the present invention.

[0211] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and sub-modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0212] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0213] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0214] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional modules.

[0215] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-fluorescence channel microsphere image alignment method, characterized in that: include: Acquire fluorescent microsphere images at the same position through multiple fluorescence channels; Preprocessing each of the fluorescent microsphere images to obtain a fluorescence correction image; Performing microsphere region segmentation on each of the fluorescence-corrected images, and calculating the region center point corresponding to each microsphere region; intercepting a pixel matrix from the fluorescence-corrected image according to the center point of each region, and extracting a corresponding feature vector; Selecting any one of the fluorescence-corrected images as a reference image, and performing pairwise registration on the remaining fluorescence-corrected images according to the characteristic vectors, and determining adjustment offsets respectively; Each of the fluorescence correction images is corrected according to the adjustment offset and superimposed on the reference image to generate a fluorescence channel alignment image.

2. The method according to claim 1, characterized in that The preprocessing of each of the fluorescent microsphere images to obtain a fluorescence correction image includes: Performing a power operation on the pixel value of each pixel in each of the fluorescent microsphere images according to a preset gamma parameter to obtain updated pixel values ​​respectively; The pixels to which the updated pixel values ​​belong are used to generate a fluorescence-corrected image.

3. The method according to claim 1, characterized in that The step of segmenting each of the fluorescence-corrected images into microsphere regions and calculating the region center points corresponding to each of the microsphere regions comprises: Filtering the pixel values ​​of the pixels in each of the fluorescence-corrected images according to a preset quantile threshold to obtain a coarse segmentation image of the microsphere region; Filtering the coarse segmentation image of the microsphere region, and identifying the edge of the microsphere region in combination with the Laplace operator to obtain an edge image of the microsphere region; Performing a subtraction operation on the microsphere region rough segmentation image and the microsphere region edge image to generate a microsphere region segmentation image; Calculating the pixel distance from each non-zero pixel to the nearest zero pixel in each microsphere region in the microsphere region segmentation image; Using all the pixel distances, generating a distance peak map; In the distance peak map, the local peak point of each microsphere region is calculated as the corresponding region center point.

4. The method according to claim 3, characterized in that: The calculating the pixel distance from each non-zero pixel of each microsphere region in the microsphere region segmentation image to the nearest zero pixel comprises: Locating non-zero pixels of each microsphere region from the microsphere region segmentation image, and initializing the first distance value of the non-zero pixels to a preset maximum value; Taking each of the non-zero pixels as a starting point, searching for the nearest zero pixel in the neighborhood, and calculating a second distance value between the non-zero pixel and the nearest zero pixel; Selecting the minimum value between the second distance value and the first distance value to update the first distance value until all pixels in the microsphere area are retrieved; Each of the first distance values ​​is normalized to obtain a pixel distance from each non-zero pixel to the nearest zero pixel in each microsphere region.

5. The method according to claim 1, characterized in that The method of intercepting a pixel matrix from the fluorescence correction image according to the center point of each region and extracting a corresponding feature vector includes: Taking the center point of each region as the center, intercepting a pixel matrix from the fluorescence-corrected image according to a first pixel size, and downsampling to a second pixel size to obtain a first pixel matrix; Taking the center point of each region as the center, intercepting a second pixel matrix from the fluorescence-corrected image according to the second pixel scale; Divide the gradient direction range into multiple gradient bins according to the preset dimension; Calculating the gradient direction corresponding to each pixel in the first pixel matrix and the second pixel matrix respectively; Classifying each of the gradient directions into corresponding gradient bins, and constructing a gradient direction frequency table; generating a first eigenvector corresponding to the first pixel matrix and a second eigenvector corresponding to the second pixel matrix according to the gradient direction frequency table; The first feature vector and the second feature vector are concatenated to generate feature vectors corresponding to the center points of each region.

6. The method according to claim 5, characterized in that The calculating the gradient direction corresponding to each pixel in the first pixel matrix and the second pixel matrix respectively includes: Traversing each pixel in the first pixel matrix and the second pixel matrix, and calculating the corresponding horizontal axis direction change gradient and vertical axis direction change gradient respectively; Calculating the gradient ratio between the gradient of the horizontal axis and the gradient of the vertical axis; The arc tangent value of the gradient ratio is calculated as the gradient direction.

7. The method according to claim 1, characterized in that The adjustment offset includes a rotation offset and a displacement offset; the selecting any one of the fluorescence correction images as a reference image, and performing pairwise registration on the remaining fluorescence correction images according to the feature vector to respectively determine the adjustment offsets, includes: Selecting any one of the fluorescence-corrected images as a reference image; Performing pairwise registration of the feature vectors of the reference image and feature vectors corresponding to the remaining fluorescence-corrected images to determine a plurality of sets of matching feature point pairs; According to each group of matching feature point pairs, corresponding feature point coordinate pairs are located from the reference image and the remaining fluorescence-corrected image respectively; Determine the initial rotation offset and the initial displacement offset according to each group of feature point coordinate pairs; Selecting the largest value among the initial rotation offsets as the rotation offset; The value with the largest number among the initial displacement offsets is selected as the displacement offset.

8. The method according to claim 7, characterized in that The step of correcting each of the fluorescence correction images according to the adjustment offset and superimposing the images onto the reference image to generate a fluorescence channel alignment image includes: After rotating according to the rotation offset corresponding to each of the fluorescence correction images, translating according to the corresponding displacement offset to obtain a target fluorescence image; Each of the target fluorescence images is superimposed on the reference image to generate a fluorescence channel alignment image.

9. A multi-fluorescence channel microsphere image alignment device, characterized in that: include: An image acquisition module, used to acquire fluorescent microsphere images at the same position through multiple fluorescence channels; An image correction module, used for preprocessing each of the fluorescent microsphere images to obtain a fluorescence correction image; A regional center point positioning module is used to segment each of the fluorescence correction images into microsphere regions and calculate the regional center points corresponding to each microsphere region; A feature vector extraction module, used to intercept a pixel matrix from the fluorescence correction image according to the center point of each area, and extract a corresponding feature vector; An offset determination module, used for selecting any of the fluorescence-corrected images as a reference image, and performing pairwise registration on the remaining fluorescence-corrected images according to the feature vectors, and respectively determining adjustment offsets; The image correction module is used to correct each of the fluorescence correction images according to the adjustment offset and superimpose them on the reference image to generate a fluorescence channel alignment image.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the multi-fluorescence channel microsphere imaging method according to any one of claims 1 to 8.

Images