A micro-image integral-pixel alignment method based on sub-pixel edge offset detection

Abstract

The application discloses a microscopic image integral pixel alignment method based on sub-pixel edge offset detection and relates to the field of microscopic image processing. The application solves the problems of the existing sub-pixel registration technology, such as dependence on iterative optimization or a complex deformation model, difficulty in real-time processing, need for compensation of sensor sampling error and mechanical translation error by means of a high-resolution camera or a precision displacement table, and high system cost. The application realizes sub-pixel alignment without dependence on a high-precision displacement table through sub-pixel level offset compensation, and significantly reduces hardware cost. The method disclosed by the application extracts image edge feature points by using an edge detection algorithm, fits a gray centroid to calculate a sub-pixel offset, and counts an average offset. An affine transformation matrix is generated according to the average offset, and an image is resampled to align pixel values to an integral coordinate grid. Finally, the image edge is cropped according to an interpolation algorithm type to eliminate ghosting. The application is also applicable to the fields of fluorescent microscope multi-view field splicing, super-resolution reconstruction and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of microscopic image processing, and in particular to a microscopic image integral pixel alignment method based on sub-pixel edge offset detection. BACKGROUND

[0002] In the field of microscopic imaging, multi-field image stitching and super-resolution reconstruction technologies need to rely on high-precision image alignment. Traditional methods usually realize sub-pixel level displacement compensation through hardware (such as a piezoelectric ceramic objective table), but high-precision hardware is expensive and complex to maintain. Although the software-based method can partially replace the hardware, the existing sub-pixel registration technology (such as elastic registration and optical flow method) relies on iterative optimization or complex deformation models, and it is difficult to process in real time; sensor sampling errors and mechanical translation errors need to be compensated by high-resolution cameras or precision displacement tables, which is high in system cost. SUMMARY

[0003] The present application is to solve the problems of the existing sub-pixel registration technology, which relies on iterative optimization or complex deformation models, and is difficult to process in real time; sensor sampling errors and mechanical translation errors need to be compensated by high-resolution cameras or precision displacement tables, which is high in system cost.

[0004] To solve the above technical problems, the present application is realized by the following technical scheme:

[0005] Scheme one, the present application provides a microscopic image integral pixel alignment method based on sub-pixel edge offset detection, the method comprises the following steps:

[0006] Step 1, obtaining an original microscopic image, and pre-processing the original microscopic image;

[0007] Step 2, extracting edge feature points based on the pre-processed original microscopic image in step 1, and calculating the sub-pixel offset of the feature points by fitting the gray centroid, and calculating the statistical average offset ;

[0008] Step 3, generating an affine transformation matrix according to the statistical average offset in step 2 , and resampling the original microscopic image to align the pixel value to the integer coordinate grid;

[0009] Step 4, clipping the edge of the resampled image in step 3 to eliminate the virtual shadow introduced by interpolation, which is used to eliminate the sub-pixel edge offset of the original microscopic image, and complete the microscopic image integral pixel alignment based on sub-pixel edge offset detection.

[0010] Further, a preferred embodiment is provided, which uses a feature extraction algorithm to extract edge feature points from the pre-processed original microscopic image in step 2, and the extracted feature points are denoted as each feature point contains integer coordinates , the average offset is calculated based on the distribution of the feature points ;

[0011] The feature extraction algorithm is any one of a Canny operator or other edge detection algorithm, a scale-invariant feature transform algorithm, a speeded up robust features algorithm, or a histogram of oriented gradients algorithm.

[0012] Further, a preferred embodiment is provided, in which the feature point sub-pixel offset is calculated by fitting the gray centroid in step 2, and the method for calculating the statistical average offset is :

[0013] For each feature point, a 5 pixel neighborhood centered on the feature point is extracted, a coordinate network is created, and the intensity values of the neighborhood corresponding to the feature point are read, the edge point gray centroid is calculated by fitting a bivariate quadratic Gaussian, and the feature point sub-pixel offset is calculated;

[0014] The sub-pixel offset of each edge feature point is , wherein is the initial pixel coordinate of the feature point; is the fitted gray centroid coordinate; the average offset is is the arithmetic mean of the offsets of all feature points;

[0015] The method for calculating the edge point gray centroid by fitting a bivariate quadratic Gaussian is

[0016]

[0017] wherein A is the maximum gray value of the feature point neighborhood pixel points; B is the background gray baseline; and is the standard deviation of the Gaussian distribution, indicating the edge blur degree; and is solved by the least square method.

[0018] Further, a preferred embodiment is provided, in which the affine transformation matrix in step 3 only contains a translation operation, and the translation amount is the average offset .

[0019] Further, a preferred embodiment is provided, in which the specific method for resampling the original microscopic image in step 3 is

[0020] The target pixel is mapped to a non-integer position of the original image by the affine matrix M, and the target pixel value is generated by an interpolation algorithm according to the non-integer coordinates of the mapped position, specifically

[0021] According to the average offset An affine transformation matrix is generated in the form of:

[0022] The original microscopic image is resampled to align the pixel values to the integer coordinate grid.

[0023] The matrix transformation is performed when resampling the original microscopic image ;

[0024] wherein the target image coordinates are (u, v), the original image coordinates are (x, y), M is the affine transformation matrix, and M-1 is the inverse matrix thereof.

[0025] Further, a preferred embodiment is provided, wherein the interpolation algorithm is any one of bilinear interpolation or bicubic interpolation.

[0026] Further, a preferred embodiment is provided, wherein when the interpolation algorithm is bilinear interpolation, the interpolation grayscale value is calculated by linear weighting based on the four adjacent pixel points of the target pixel in the original image, and the specific formula is:

[0027]

[0028] wherein, is the decimal part of the sub-pixel offset;

[0029] When the interpolation algorithm is bicubic interpolation, the interpolation grayscale value is calculated by cubic polynomial kernel function weighting based on the 16 adjacent pixel points to of the target pixel in the original image, and the kernel function is:

[0030]

[0031] wherein, t is the normalized offset from the target point, and the value range of the parameter a is .

[0032] Further, a preferred embodiment is provided, wherein the clipping of the edges of the resampled image in step 4 is specifically: when the bilinear interpolation is used, the clipping width is 1.5 pixels, and when the bicubic interpolation is used, the clipping width is 3 pixels.

[0033] Option 2: A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to realize the steps of the method of option 1.

[0034] Scheme three, a computer device, comprising a memory and a processor, the memory has stored a computer program, when the processor runs the computer program stored in the memory, the processor executes the method of scheme one.

[0035] The present application has the advantages of:

[0036] The micro image integral pixel alignment method based on sub-pixel edge offset detection provided by the present application can realize sub-pixel alignment without relying on a high-precision displacement table through sub-pixel offset compensation, and significantly reduces hardware cost. The method extracts image edge feature points by using an edge detection algorithm, calculates sub-pixel offset by fitting a gray centroid, and calculates statistical average offset. An affine transformation matrix is generated according to the average offset, and the image is resampled to align the pixel value to the integer coordinate grid. Finally, the image edge is cropped according to the interpolation algorithm type to eliminate ghosting.

[0037] The present application can realize sub-pixel alignment accuracy without relying on a high-precision displacement table through sub-pixel offset correction, compensate for sensor sampling errors through an algorithm, reduce the dependence on high-resolution cameras, and greatly reduce hardware cost. Moreover, the method is simple to operate and easy to implement.

[0038] The present application is also applicable to the fields of fluorescence microscope multi-view stitching, super-resolution reconstruction, etc. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The flowchart of the micro image integral pixel alignment method based on sub-pixel edge offset detection according to the first embodiment. DETAILED DESCRIPTION

[0040] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application.

[0041] Embodiment one, the present embodiment proposes a micro image integral pixel alignment method based on sub-pixel edge offset detection, which comprises the following steps:

[0042] Step 1, obtaining an original micro image and pre-processing the original micro image;

[0043] Step 2, extracting edge feature points based on the original micro image pre-processed in step 1, calculating feature point sub-pixel offset by fitting a gray centroid, and calculating statistical average offset .

[0044] Step 3, generating an affine transformation matrix according to the statistical average offset in step 2 An affine transformation matrix is generated, and the original microscopic image is resampled to align the pixel values to the integer coordinate grid.

[0045] Step 4, cropping the edges of the resampled image in step 3 to eliminate the artifacts introduced by interpolation, for eliminating the sub-pixel edge shift of the original microscopic image, and completing the integral pixel alignment of the microscopic image based on the sub-pixel edge shift detection.

[0046] The preprocessing in the embodiment includes gray scale transformation processing, denoising and edge enhancement on the original microscopic image.

[0047] Embodiment II, the embodiment is a further limitation of the integral pixel alignment method of the microscopic image based on the sub-pixel edge shift detection according to embodiment I, using a feature extraction algorithm to extract edge feature points from the original microscopic image after preprocessing in step 2, and recording the extracted feature points as Each feature point contains an integer coordinate , and the average shift is calculated based on the distribution of the feature points .

[0048] The feature extraction algorithm is any one of Canny operator or other edge detection algorithm, scale invariant feature transform algorithm, speeded up robust features algorithm, and orientation histogram of gradients algorithm.

[0049] In the embodiment, the feature point extraction algorithm is any one of Canny operator or other edge detection algorithm, scale invariant feature transform (SIFT) algorithm, speeded up robust features (SURF) algorithm, orientation histogram of gradients (ORB) algorithm, and deep learning feature extraction model based on convolutional neural network.

[0050] Embodiment III, the embodiment is a further limitation of the integral pixel alignment method of the microscopic image based on the sub-pixel edge shift detection according to embodiment I, and the feature point sub-pixel shift is calculated by fitting the gray centroid in step 2, and the average shift is calculated by Method:

[0051] For each feature point, a 5 pixel neighborhood centered on the feature point is extracted, a coordinate network is created, and the intensity values of the neighborhood corresponding to the feature point are read, the edge point gray centroid is calculated by fitting the binary quadratic Gaussian, and the feature point sub-pixel shift is calculated.

[0052] The sub-pixel shift of each edge feature point is , wherein is the initial pixel coordinate of the feature point; is the fitted gray centroid coordinate; and the average shift is The arithmetic mean of the offset values of all feature points;

[0053] The method for calculating the gray centroid of the edge point by the binary quadratic Gaussian fitting is:

[0054]

[0055] wherein A is the maximum gray value of the neighborhood pixel points of the feature point; respectively, are the sub-pixel level coordinates of the feature point; B is the background gray baseline; is the standard deviation of the Gaussian distribution, indicating the edge blur degree; and is solved by the least square method.

[0056] In the embodiment, the method for fitting the gray centroid is the binary quadratic Gaussian fitting.

[0057] In the fourth embodiment, the affine transformation matrix in step 3 only contains a translation operation, and the translation amount is the average offset amount .

[0058] In the fifth embodiment, the method for resampling the original microscopic image in step 3 is:

[0059] The target pixel is mapped to a non-integer position of the original image by the affine matrix M, and the target pixel value is generated by an interpolation algorithm according to the non-integer coordinates of the mapped position, and specifically:

[0060] The affine transformation matrix is generated according to the average offset amount , and the form is:

[0061] The original microscopic image is resampled to align the pixel values to the integer coordinate grid.

[0062] The matrix transformation is performed when the original microscopic image is resampled .

[0063] wherein the target image coordinates are (u, v), the original image coordinates are (x, y), M is the affine transformation matrix, , and is the inverse matrix thereof.

[0064] After the matrix transformation is performed in the embodiment, the image is processed by an interpolation algorithm.

[0065] Embodiment six, the embodiment is further limited to the method for micro image integral pixel alignment based on sub-pixel edge offset detection of embodiment five, the interpolation algorithm is any one of bilinear interpolation or bicubic interpolation.

[0066] Embodiment seven, the embodiment is further limited to the method for micro image integral pixel alignment based on sub-pixel edge offset detection of embodiment six, when the interpolation algorithm is bilinear interpolation, based on the four adjacent pixel points of the target pixel in the original image , the interpolation gray value is calculated by linear weighting, and the specific formula is:

[0067]

[0068] , wherein, is the decimal part of the sub-pixel offset;

[0069] When the interpolation algorithm is bicubic interpolation, based on the 16 adjacent pixel points of the target pixel in the original image to , the interpolation gray value is calculated by cubic polynomial kernel function weighting, and the kernel function is:

[0070]

[0071] , wherein t is the normalized offset distance from the target point, and the value range of parameter a is .

[0072] Embodiment eight, the embodiment is further limited to the method for micro image integral pixel alignment based on sub-pixel edge offset detection of embodiment seven, the edge of the image resampled in step 3 is specifically clipped in step 4: the clipping width is 1.5 pixels when bilinear interpolation is used, and the clipping width is 3 pixels when bicubic interpolation is used.

[0073] Embodiment nine, the embodiment provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the method in any one of embodiments one to seven.

[0074] Embodiment ten, the embodiment provides a computer device, including a memory and a processor, the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method in any one of embodiments one to seven.

[0075] Embodiment eleven, the embodiment is used to explain the above embodiments one to ten, and the embodiment is specifically:

[0076] See Figure 1 This implementation method is described with reference to Figure 1 As shown, in step 1, a raw microscopic image is acquired using microscopic imaging technology, and the raw microscopic image is preprocessed.

[0077] Specifically, an optical microscope imaging system equipped with a stepper motor stage (translation accuracy ±1μm) and a CMOS sensor can be used to capture a sequence of live cell microscopic images to obtain raw microscopic images; then, the raw microscopic images are converted to grayscale; the images are denoised using non-local means; and the images are equalized using an adaptive histogram to enhance image edges.

[0078] In step 2, edge feature points are extracted from the preprocessed original microscopic image, and sub-pixel offsets are calculated by fitting the gray-level centroids, and the average offset is statistically analyzed. ;

[0079] In this study, the Canny operator was selected to perform edge detection and extract edge feature points. The set of extracted feature points is denoted as Each feature point Including integer coordinates The coordinate precision is integer pixels; each feature point is cropped to... Within a 5x5 pixel neighborhood centered on the target, the grayscale values ​​of the neighborhood are read, and a binary quadratic Gaussian fit is performed to calculate the grayscale centroid. ;Calculate the pixel offset of each feature point Then, the average offset of all feature points is calculated. .

[0080] In step 3, based on the average offset Generate the affine transformation matrix, in the form of: The image is then resampled to align pixel values ​​to an integer coordinate grid.

[0081] An affine matrix M is used to map the target pixels to non-integer locations in the original image. Matrix transformations are performed during image resampling. The target image coordinates are (u, v), the original image coordinates are (x, y), and M is the affine transformation matrix. It is the inverse matrix, and then the target pixel value is generated using the bilinear interpolation algorithm.

[0082] In step 4, the edges of the resampled image are cropped to eliminate ghosting introduced by interpolation, thereby eliminating subpixel edge offset of the initial image.

[0083] The width of the cropped pixel value is 1.5 pixels.

[0084] The edge feature point extraction method is SURF (Speeded Up Robust Features) feature extraction, and other steps are consistent with the specific embodiment one.

[0085] The resampling process generates target pixel values using a bicubic interpolation algorithm, and the corresponding interpolation clipping width is 3 pixels to match the spatial influence range of the bicubic interpolation algorithm. Other steps and parameters are the same as in specific embodiments one or two.

[0086] The edge feature point extraction uses a pre-trained lightweight convolutional neural network (MobileNetV3) to reduce manual intervention. This embodiment is suitable for cross-device and cross-sample microscopic image alignment, and enhances the generalization ability of the algorithm.

[0087] Wherein, the mobilenet_v3_small function is used to load the pre-trained MobileNetV3 model; the Conv2d function is used to build a lightweight feature decoding layer and compress the channel to 1 dimension; the preprocess_image function is used to standardize the initial image and realize size adaptation; the preprocess_image function is called to preprocess the input image image and use unsqueeze(0) to add a dimension in the first dimension of the tensor to adapt to the input requirements of the deep learning model.

[0088] When performing the edge feature point extraction, the extract_keypoints function is used to extract the feature points of the standardized image; the non-maximum suppression maximum_filter is used, and the neighborhood window neighborhood_size=5 and k=1024 are set.

[0089] The local extreme points are filtered, and the flattened index of the top k maximum values in the response map is found, where k is the number of extracted feature points; np.unravel_index is used to convert the one-dimensional index to a two-dimensional index; np.column_stack is used to combine the two-dimensional index into an array, and each element represents a feature point coordinate.

[0090] The edge feature point extraction uses the ORB (Oriented FAST and Rotated BRIEF) algorithm. In step 2, the ORB algorithm is used to extract feature points with rotational invariance, which is suitable for scenarios where the stage moves quickly. The number of feature points is dynamically adjusted according to the complexity of the image. After extraction, a binary quadratic Gaussian fitting is performed based on a 5x5 neighborhood, and the average offset is calculated. In step 3, bilinear interpolation is used to achieve fast resampling, and in step 4, the clipping width is 1.5 pixels. This embodiment is suitable for dynamic observation of living cells, and both processing speed and alignment accuracy are considered.

[0091] The embodiment adds low-light enhancement preprocessing on the basis of the first embodiment. In step 1, gamma correction (Hessian threshold = 500) and dark channel defogging are performed on the original fluorescence image to improve weak signal visibility; Top-Hat filtering is used to enhance fluorescence spots. In step 2, the SURF algorithm (Hessian threshold = 500) is used for feature point extraction to adapt to high-noise scenes. In step 3, bicubic interpolation is used to retain details, and in step 4, the width is cropped by 3 pixels. The embodiment is optimized for low signal-to-noise ratio (SNR≥6dB) fluorescence images to ensure sub-pixel offset detection stability.

[0092] Those skilled in the art can understand that the above description is only preferred embodiments of the present application, and the features described in each embodiment and / or technical solution of the present disclosure can be combined or combined, even if such combination or combination is not explicitly described in the present disclosure. It is not intended to limit the present application, although the present application has been described in detail with reference to the foregoing embodiments, and those skilled in the art can modify the technical solutions described in the foregoing embodiments or make equivalent replacements for part of the technical features, and any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0093] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic creative concept. Therefore, the appended technical solutions are intended to include the preferred embodiments and all changes and modifications falling within the scope of the present application. Obviously, those skilled in the art can make various modifications and changes to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and changes of the present application fall within the scope of the present application and its equivalent technology, the present application also intends to include these modifications and changes.​

Claims

1. A method for micrograph image integer alignment based on sub-pixel edge shift detection, characterized in that, The method comprises the following steps: Step 1, obtaining an original microscopic image and pre-processing the original microscopic image; Step 2, edge feature point extraction is performed based on the original microscopic image after pretreatment in step 1, and the sub-pixel offset of the feature point is calculated by fitting the gray centroid, and the average offset is calculated and counted ; Step 3, average offset as calculated in step 2 An affine transformation matrix is generated and the original microscopic image is resampled to align the pixel values to the integer coordinate grid. Step 4, clipping the edges of the image resampled in step 3 to eliminate the ghosting introduced by interpolation, for eliminating the sub-pixel edge offset of the original microscopic image, and completing the integral-pixel alignment of the microscopic image based on the sub-pixel edge offset detection.

2. The sub-pixel edge shift detection based microscopic image integer alignment method according to claim 1, wherein, extracting edge feature points from the preprocessed original microscopic image in step 2 using a feature extraction algorithm, and recording the extracted feature points as each feature point contains an integer coordinate , calculating an average offset based on the distribution of the feature points ; The feature extraction algorithm is any one of a Canny operator or other edge detection algorithm, a scale-invariant feature transform algorithm, a speeded up robust features algorithm, and a histogram of oriented gradients algorithm.

3. The sub-pixel edge shift detection based microscopic image integer alignment method according to claim 1, wherein, The feature point sub-pixel offset is calculated by fitting the gray centroid in step 2, and the statistical average offset is calculated The method is: 5 5 Pixel neighborhood, create coordinate network and read intensity value of the neighborhood corresponding to the feature point, calculate the edge point gray centroid through binary quadratic Gaussian fitting, and calculate the sub-pixel offset of the feature point; The sub-pixel offset of each edge feature point is wherein is the initial pixel coordinate of the feature point; is the fitted gray centroid coordinate; the average offset is the arithmetic mean of the offset of all feature points; The method for calculating the gray centroid of the edge point by the binary quadratic Gaussian fitting is: Wherein, A is the maximum gray value of the feature point neighborhood pixel point; Respectively, the sub-pixel level coordinates of the feature point; B is the background gray baseline; The standard deviation of the Gaussian distribution represents the edge blur degree; it is solved by the least square method.

4. The sub-pixel edge shift detection based microscopic image integer alignment method according to claim 1, wherein, The affine transformation matrix in step 3 only contains translation operations, whose translation amount is the average offset .

5. The sub-pixel edge shift detection based microscopic image integer alignment method according to claim 1, wherein, The specific method for resampling the original microscopic image in step 3 is: The target pixel is mapped to a non-integer position of the original image through an affine matrix M, and the target pixel value is generated through an interpolation algorithm according to the non-integer coordinates of the mapping position, specifically: According to the average offset An affine transformation matrix is generated, in the form of: The pixel values are resampled to align to the integral coordinate grid; Performing matrix transformation when resampling original microscopic images ; Wherein, the target image coordinates are (u, v), the original image coordinates are (x, y), M is an affine transformation matrix, is the inverse matrix thereof.

6. The sub-pixel edge shift detection based microscopic image integer pixel alignment method according to claim 5, characterized in that, The interpolation algorithm is any one of bilinear interpolation or bicubic interpolation.

7. The sub-pixel edge shift detection based microscopic image integer pixel alignment method according to claim 6, characterized in that, When the interpolation algorithm is bilinear interpolation, the four pixel points adjacent to the target pixel in the original image are used as the basis to calculate the interpolation gray value by linear weighting, and the specific formula is: wherein is the fractional part of the sub-pixel offset; When the interpolation algorithm is bicubic interpolation, the interpolation gray value is calculated by weighting based on the 16 adjacent pixel points of the target pixel in the original image to , and the kernel function is a cubic polynomial kernel function: where t is the normalized offset from the target point, and the parameter a has a value in the range .

8. The sub-pixel edge shift detection based microscopic image integer pixel alignment method according to claim 7, characterized in that, In step 4, the specific clipping of the edges of the image resampled in step 3 is that the clipping width is 1.5 pixels when the bilinear interpolation is used, and the clipping width is 3 pixels when the bicubic interpolation is used.

9. A computer storage medium having stored thereon a computer program, characterized in that The computer program is executed by a processor to realize the method of any one of claims 1-7.

10. A computer device, comprising: Comprise: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to realize the method of any one of claims 1-7.

Images