A method for registering a tissue slice image with a DAPI stained image

Abstract

The present application relates to pathological image processing technical field, provide a kind of tissue section image and DAPI staining image registration method, comprising: tissue section image and DAPI staining image are carried out gray processing, and according to same work magnification is enlarged;Obtain the feature point image of tissue section and DAPI staining;Find the best scaling ratio is 最佳 Of feature point image of tissue section and DAPI staining, rotation angle ia, horizontal displacement dx and vertical displacement dy and average point (xm, ym);With average point (xm, ym) generation multiple sub-frames, so that the tissue section feature point contained in sub-frame exceeds 200;For each sub-frame, using the best scaling ratio is 最佳 Rotation angle ia, in the case of (xm, ym) as rotation center, DAPI staining feature point is handled to generate transformation matrix tf2;Transformation matrix tf2 and DAPI staining image are generated final registration image by cv2.warpAffine function in opencv.Can tissue section image under normal microscope white light irradiation and DAPI staining image under fluorescence microscope are registered.

Description

Technical Field

[0001] This invention relates to the field of pathological image processing technology, and in particular to a method for registering tissue section images with DAPI-stained images. Background Technology

[0002] DAPI is a fluorescent dye that binds strongly to DNA. It binds to the AT base pairs in the minor groove of double-stranded DNA, with one DAPI molecule occupying three base pairs. The fluorescence intensity of DAPI molecules bound to double-stranded DNA increases approximately 20-fold. It is commonly used for apoptosis detection, followed by observation under a fluorescence microscope or flow cytometry. It is also frequently used for routine nuclear staining and double-stranded DNA staining under certain specific conditions. Cells stained with DAPI after heat shock for 3 minutes show morphological changes in the cell nucleus under a fluorescence microscope. However, in current techniques, the images of tissue sections illuminated under normal white light under a microscope and the DAPI-stained images under a fluorescence microscope require manual registration, which is cumbersome and detrimental to the observation of image features. Summary of the Invention

[0003] To address the above problems, the present invention provides the following technical solution:

[0004] This invention provides a method for registering tissue section images and DAPI-stained images, the registration method comprising:

[0005] (1) Perform grayscale processing on the tissue section images and DAPI staining images, and enlarge them at the same working magnification;

[0006] (2) Obtain feature point images of tissue sections and DAPI staining;

[0007] (3) Finding the optimal scaling ratio for feature point images of tissue sections and DAPI staining is 最佳 The rotation angle ia, the horizontal displacement dx, the vertical displacement dy, and the average point (xm, ym);

[0008] (4) Generate multiple sub-frames using the average point (xm, ym) and scale them according to the optimal scaling ratio is 最佳 The tissue slice image and the tissue slice feature point image are scaled proportionally so that the number of tissue slice feature points contained in the subframe exceeds 200.

[0009] (5) For each sub-frame, use the optimal scaling ratio is 最佳 Given the rotation angle ia, and with (xm, ym) as the rotation center, the DAPI-colored feature points are processed to generate the transformation matrix tf2;

[0010] (6) The transformation matrix tf2 and the DAPI-stained image are used to generate the final registration image using the cv2.warpAffine function in OpenCV.

[0011] Furthermore, the specific process of step (2) is as follows:

[0012] The grayscale images of tissue slices and DAPI-stained images are filtered using filters to obtain score maps. Then, OpenCV is used to calculate the connected components in the score maps. Specifically, the cv2.connectedComponentsWithStats function in OpenCV is used to calculate the connected components in the score maps, and the area of ​​the connected components is checked to see if it exceeds a threshold (the best value obtained from testing). For connected components that exceed the threshold, the coordinates (x, y) of the maximum value inside them are calculated as feature points, the maximum value is taken as the score, and the feature point images are generated by sorting them according to the score.

[0013] The filter consists of two 2D Gaussian filters from OpenCV: Gaussian filters A and B, each with a window size of 15*15 and standard values ​​of 3 and 7 respectively, and outputs a and b. The overall output of the filter is (ab)*10.

[0014] Furthermore, the specific process of step (3) is as follows:

[0015] (31) Keep the ratio of the tissue slice image and the DAPI staining image unchanged, scale the long side to 200 pixels, and scale the feature point image at the same scaling ratio; scale the scaled image and the feature point again at 0.1 intervals in the range of 0.9 to 1, and then adjust the horizontal and vertical displacements under different scaling ratios, without letting the tissue slice image not exceed the boundary of the DAPI staining image, calculate the IoU between the feature point images corresponding to the tissue slice image and the DAPI staining image respectively, find the position where the maximum IoU is located, which is the best matching position, and obtain the best matching scaling ratio s;

[0016] (32) Then enlarge the scaled tissue slice image and DAPI staining image by two times. Rotate the average value of all feature points on the tissue image as the reference, with an angle range of -5 to 5 degrees. Within the range of s-0.1 to s+0.1, adjust the horizontal and vertical displacements at intervals of 0.1. Calculate the IoU between the tissue slice image and the DAPI staining feature point image. Find the location of the maximum IoU, which is the best matching position. Find the scaling ratio is, rotation angle ia, horizontal displacement dx, and vertical displacement dy corresponding to the maximum IoU.

[0017] (33) Adjust the scaling ratio to (is-0.1)*0.9~(is+0.1)*1.1, divide it into 55 equal parts, adjust the rotation angle to ia±2 interval 0.4, the horizontal displacement to dx±4 interval 1, and the vertical displacement to dy±4 interval 1. Repeat step (32) to find the optimal scaling ratio is. 最佳 ;

[0018] The optimal scaling ratio is 最佳 Compared with the working scaling factor in step (1), the scaling factor = is 最佳 *Working scale: Scales the feature point coordinates proportionally and calculates the average value of all DAPI-colored feature points, denoted as the average point (xm, ym).

[0019] Further, the specific process of step (4) is as follows: Let the width and height of the tissue slice image be w and h respectively, then the step size of the sub-frame center point movement d = (w+h) / 12, and generate the center point coordinates of the sub-frame as (xm, ym), (xm-d, ym), (xm+d, ym), (xm, ym-d), (xm, ym+d);

[0020] Calculate the number of tissue slice feature points contained in each subframe. If it does not exceed 200, increase the subframe w by 100 and h by 100, until the number of feature points exceeds 200.

[0021] Furthermore, the specific process of step (5) is as follows:

[0022] (51) In each sub-box, use the optimal scaling ratio is 最佳 Given the rotation angle ia, with (xm, ym) as the rotation center, calculate the matched DAPI staining feature points, calculate the matched new subframe, and recalculate the tissue slice feature points within the matched new subframe.

[0023] (52) If the number of tissue slice feature points in the new subframe is less than 1 / 4 of the number of DAPI staining feature points, it is considered that the region cannot be effectively matched and no further operation is performed; otherwise, proceed to the next step.

[0024] (53) In each new subframe, the transformation matrix tf between tissue section feature points and DAPI staining feature points is calculated using the CPD related point drift algorithm;

[0025] The transformation matrix tf is used to match DAPI staining feature points, and the number of feature points in tissue slices is counted. If the number exceeds 10,000, random sampling is performed to reduce the computational load, and Euclidean distance is used to calculate the matched feature point pairs.

[0026] (54) When the number of matches in all sub-boxes exceeds 20 pairs, use the cv2.estimateAffine2D function to generate a transformation matrix tf1 for the tissue slice feature points and DAPI staining feature points in the matching pairs;

[0027] (55) Transform the DAPI-colored feature points using the transformation matrix tf1, recalculate the matching pairs, and then use the cv2.estimateAffine2D function again to generate the transformation matrix tf2.

[0028] Furthermore, the specific process of step (51) is as follows:

[0029] Using the `cv2.getRotationMatrix2D` function in OpenCV, a rotation matrix M is generated. Using the rotation matrix M and the tissue slice feature points, the matched DAPI staining feature points are calculated. Then, using the rotation matrix M and the calculated matched DAPI staining feature points, the maximum (xmax, ymax) and minimum (xmin, ymin) values ​​of the feature point coordinates after scaling and rotation are calculated to obtain a new subframe. The width of the new subframe is w = xmax - xmin, and the height is h = ymax - ymin. The vertex coordinates of the new subframe are (xmax, ymax), (xmax, ymin), (xmin, ymax), (xmin, ymin).

[0030] Then, the cv2.getRotationMatrix2D function in OpenCV is used to calculate the tissue slice feature points within the new subframe after matching.

[0031] Furthermore, in step (53), the cdist function in scipy is used to calculate the Euclidean distance between the matched DAPI staining feature points and the tissue slice feature points. The distance is required to be no more than 1 / 50 of the width of the subframe. When the number of matching pairs exceeds 3, the subframe is considered to be successfully matched.

[0032] The present invention has the following beneficial effects:

[0033] This invention provides a method for registering tissue section images with DAPI-stained images, capable of registering tissue section images illuminated under normal microscope with DAPI-stained images under fluorescence microscope. It offers high registration efficiency, minimal pixel error, and is particularly suitable for registering large images under a microscope. Attached Figure Description

[0034] Figure 1 This is a flowchart of the registration method of the present invention.

[0035] Figure 2 This is a partial effect diagram of the invention without applying the CPD-related point drift algorithm to the new subframe.

[0036] Figure 3 This is a partial effect diagram of the CPD-related point drift algorithm applied to the new subframe in this invention.

[0037] Figure 4 These are partial effect images of the DAPI-stained image after registration is completed according to the present invention.

[0038] Figure 5 These are partial effect images of tissue slices after registration according to the present invention.

[0039] Figure 6 This is an overall effect diagram after the registration of the present invention. Detailed Implementation

[0040] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments are only specific illustrations of the invention and should not be regarded as limitations on the invention. The purpose of the embodiments is to enable those skilled in the art to better understand and reproduce the technical solution of the present invention. The scope of protection of the present invention should still be determined by the scope defined in the claims.

[0041] like Figure 1 As shown, the present invention provides a method for registering tissue section images and DAPI-stained images, the registration method comprising:

[0042] S1, perform grayscale processing on the tissue section image and DAPI staining image, and enlarge them at the same working magnification;

[0043] S2, acquire feature point images of tissue sections and DAPI staining;

[0044] The specific process is as follows:

[0045] The grayscale images of tissue slices and DAPI-stained images are filtered using filters to obtain score maps. Then, OpenCV is used to calculate the connected components in the score maps. Specifically, the cv2.connectedComponentsWithStats function in OpenCV is used to calculate the connected components in the score maps, and the area of ​​the connected components is checked to see if it exceeds a threshold (the best value obtained from testing). For connected components that exceed the threshold, the coordinates (x, y) of the maximum value inside them are calculated as feature points, the maximum value is taken as the score, and the feature point images are generated by sorting them according to the score.

[0046] The filter consists of two 2D Gaussian filters from OpenCV: Gaussian filters A and B, each with a window size of 15*15 and standard values ​​of 3 and 7 respectively, and outputs a and b. The overall output of the filter is (ab)*10.

[0047] S3, Finding the optimal scaling ratio between the tissue section and the DAPI-stained feature point image is最佳 The rotation angle ia, the horizontal displacement dx, the vertical displacement dy, and the average point (xm, ym);

[0048] The specific process is as follows:

[0049] (31) Keep the ratio of the tissue slice image and the DAPI staining image unchanged, scale the long side to 200 pixels, and scale the feature point image at the same scaling ratio; scale the scaled image and the feature point again in the range of 0.9 to 1 at intervals of 0.1, and then adjust the horizontal and vertical displacements at different scaling ratios, provided that the tissue slice image does not exceed the boundary of the DAPI staining image. The tissue slice image can be moved in steps of 4. Calculate the IoU between the feature point images corresponding to the tissue slice image and the DAPI staining image respectively, find the position where the maximum IoU is located, which is the best matching position, and obtain the best matching scaling ratio s;

[0050] (32) Then enlarge the scaled tissue slice image and DAPI staining image by two times. Rotate the average value of all feature points on the tissue image as the reference, with an angle range of -5 to 5 degrees. Within the range of s-0.1 to s+0.1, adjust the horizontal and vertical displacements at intervals of 0.1. Calculate the IoU between the tissue slice image and the DAPI staining feature point image. Find the location of the maximum IoU, which is the best matching position. Find the scaling ratio is, rotation angle ia, horizontal displacement dx, and vertical displacement dy corresponding to the maximum IoU.

[0051] (33) Adjust the scaling ratio to (is-0.1)*0.9~(is+0.1)*1.1, divide it into 55 equal parts, adjust the rotation angle to ia±2 interval 0.4, the horizontal displacement to dx±4 interval 1, and the vertical displacement to dy±4 interval 1. Repeat step (32) to find the optimal scaling ratio is. 最佳 ;

[0052] The optimal scaling ratio is 最佳 Compared with the working scaling factor in step (1), the scaling factor = is 最佳 *Working scale: Scales the feature point coordinates proportionally and calculates the average value of all DAPI-colored feature points, denoted as the average point (xm, ym).

[0053] S4 generates multiple sub-frames at the average point (xm, ym) and scales them according to the optimal scaling ratio. 最佳 The tissue slice image and the tissue slice feature point image are scaled proportionally so that the number of tissue slice feature points contained in the subframe exceeds 200.

[0054] The specific process is as follows:

[0055] Let the width and height of the tissue slice image be w and h, respectively. Then the step size for moving the center point of the sub-frame is d = (w + h) / 12, and the coordinates of the center point of the sub-frame are generated as (xm, ym), (xm-d, ym), (xm+d, ym), (xm, ym-d), (xm, ym+d).

[0056] Calculate the number of tissue slice feature points contained in each subframe. If it does not exceed 200, increase the subframe w by 100 and h by 100, until the number of feature points exceeds 200.

[0057] S5, for each sub-frame, uses the optimal scaling ratio. 最佳 Given the rotation angle ia, and with (xm, ym) as the rotation center, the DAPI-colored feature points are processed to generate the transformation matrix tf2;

[0058] The specific process is as follows:

[0059] (51) In each sub-box, use the optimal scaling ratio is 最佳 Given the rotation angle ia, with (xm, ym) as the rotation center, calculate the matched DAPI staining feature points, calculate the matched new subframe, and recalculate the tissue slice feature points within the matched new subframe.

[0060] Specifically: Use the `cv2.getRotationMatrix2D` function in OpenCV to generate a rotation matrix M. Use the rotation matrix M and the tissue slice feature points to calculate the matched DAPI staining feature points. Use the rotation matrix M and the calculated matched DAPI staining feature points to calculate the maximum value (xmax, ymax) and minimum value (xmin, ymin) of the feature point coordinates after scaling and rotation, and obtain a new subframe. The width of the new subframe is w = xmax - xmin and the height is h = ymax - ymin. The vertex coordinates of the new subframe are (xmax, ymax), (xmax, ymin), (xmin, ymax), (xmin, ymin).

[0061] Then, the cv2.getRotationMatrix2D function in OpenCV is used to calculate the tissue slice feature points within the new subframe after matching.

[0062] (52) If the number of tissue slice feature points in the new subframe is less than 1 / 4 of the number of DAPI staining feature points, it is considered that the region cannot be effectively matched and no further operation is performed; otherwise, proceed to the next step.

[0063] (53) In each new subframe, the CPD correlation point drift algorithm is used to calculate the transformation matrix tf between tissue section feature points and DAPI staining feature points; the transformation matrix tf is used to match DAPI staining feature points and the number of tissue section feature points is counted; if it exceeds 10,000, random sampling is performed to reduce the amount of computation, and the Euclidean distance is used to calculate the matched feature point pairs; the cdist function in scipy is used to calculate the Euclidean distance between the matched DAPI staining feature points and tissue section feature points, and the distance is required to be no more than 1 / 50 of the width of the subframe. When the number of matching pairs exceeds 3, the subframe is considered to be successfully matched.

[0064] in, Figure 2 The center of the black circle represents the feature points of the white tissue image, while the center of the gray circle represents the feature points of the DAPI image that did not pass the CPD algorithm. Figure 3 The center of the black circle represents the feature points of the white tissue image, while the center of the gray circle represents the feature points of the DAPI image after applying the CPD algorithm.

[0065] (54) When the number of matches in all sub-boxes exceeds 20 pairs, use the cv2.estimateAffine2D function to generate a transformation matrix tf1 for the tissue slice feature points and DAPI staining feature points in the matching pairs;

[0066] (55) Transform the DAPI-colored feature points using the transformation matrix tf1, recalculate the matching pairs, and then use the cv2.estimateAffine2D function again to generate the transformation matrix tf2.

[0067] S6, the transformation matrix tf2 and the DAPI-stained image are used to generate the final registered image using the cv2.warpAffine function in OpenCV, as shown. Figure 4-6 As shown.

[0068] The registration method of this invention can register tissue section images illuminated under normal microscope with DAPI-stained images under fluorescence microscope. It boasts high registration efficiency, small pixel error, and is particularly suitable for registering large-size images under a microscope.

[0069] It should be noted that any technical features not described in detail in this invention can be implemented using any existing technology.

[0070] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

Claims

1. A method for registering tissue section images with DAPI-stained images, characterized in that, The registration method includes: (1) Perform grayscale processing on the tissue section images and DAPI staining images, and enlarge them at the same working magnification; (2) Obtain feature point images of tissue sections and DAPI staining; (3) Finding the optimal scaling ratio for feature point images of tissue sections and DAPI staining is 最佳 The rotation angle ia, the horizontal displacement dx, the vertical displacement dy, and the average point (xm, ym); (4) Generate multiple sub-frames using the average point (xm, ym) and scale them according to the optimal scaling ratio is 最佳 The tissue slice image and the tissue slice feature point image are scaled proportionally so that the number of tissue slice feature points contained in the subframe exceeds 200. (5) For each sub-frame, use the optimal scaling ratio is 最佳 Given the rotation angle ia, and with (xm, ym) as the rotation center, the DAPI-colored feature points are processed to generate the transformation matrix tf2; (6) The transformation matrix tf2 and the DAPI-stained image are used to generate the final registration image using the cv2.warpAffine function in OpenCV; The specific process of step (3) is as follows: (31) Keep the ratio of the tissue slice image and the DAPI staining image unchanged, scale the long side to 200 pixels, and scale the feature point image at the same scaling ratio. The scaled image and feature points are scaled again in increments of 0.1 within the range of 0.9 to 1. Then, under different scaling ratios, while ensuring that the tissue slice image does not exceed the boundary of the DAPI stained image, the horizontal and vertical displacements are adjusted, and the IoU between the feature point images corresponding to the tissue slice image and the DAPI stained image is calculated. The location of the maximum IoU is found, which is the best matching location, and the scaling ratio s of the best matching is obtained. (32) Then enlarge the scaled tissue slice image and DAPI staining image by two times. Rotate the average value of all feature points on the tissue image as the reference. The angle range is -5 to 5 degrees. Within the range of s-0.1 to s+0.1, adjust the horizontal and vertical displacements at 0.1 intervals. Calculate the IoU between the tissue slice and the DAPI staining feature point image. Find the position where the maximum IoU is located, which is the best matching position. Find the scaling ratio is, rotation angle ia, horizontal displacement dx and vertical displacement dy corresponding to the maximum IoU. (33) Adjust the scaling ratio to (is-0.1)*0.9~(is+0.1)*1.1, divide it into 55 equal parts, adjust the rotation angle to ia±2 interval 0.4, the horizontal displacement to dx±4 interval 1, and the vertical displacement to dy±4 interval 1. Repeat step (32) to find the optimal scaling ratio is. 最佳 ; The optimal scaling ratio is 最佳 Compared with the working scaling factor in step (1), the scaling factor = is 最佳 *Working scale: Scales the feature point coordinates proportionally and calculates the average value of all DAPI-colored feature points, denoted as the average point (xm, ym).

2. The method for registering tissue section images and DAPI-stained images according to claim 1, characterized in that, The specific process of step (2) is as follows: The grayscale images of tissue slices and DAPI-stained images are filtered using filters to obtain score maps. Then, OpenCV is used to calculate the connected components in the score maps and check whether the area of ​​the connected components exceeds a threshold. For connected components that exceed the threshold, the coordinates (x, y) of the maximum value inside them are calculated as feature points, the maximum value is used as the score, and the feature point images are generated by sorting the scores.

3. The method for registering tissue section images and DAPI-stained images according to claim 1, characterized in that, The specific process of step (4) is as follows: Let the width and height of the tissue slice image be w and h respectively, then the step size of the sub-frame center point movement d = (w+h) / 12, and generate the center point coordinates of the sub-frame as (xm, ym), (xm-d, ym), (xm+d, ym), (xm, ym-d), (xm, ym+d); Calculate the number of tissue slice feature points contained in each subframe. If it does not exceed 200, increase the subframe w by 100 and h by 100, until the number of feature points exceeds 200.

4. The method for registering tissue section images and DAPI-stained images according to claim 1, characterized in that, The specific process of step (5) is as follows: (51) In each sub-box, use the optimal scaling ratio is 最佳 Given the rotation angle ia, with (xm, ym) as the rotation center, calculate the matched DAPI staining feature points, calculate the matched new subframe, and recalculate the tissue slice feature points within the matched new subframe. (52) If the number of tissue slice feature points in the new subframe is less than 1 / 4 of the number of DAPI staining feature points, it is considered that the region cannot be effectively matched and no further operation is performed; otherwise, proceed to the next step. (53) In each new subframe, the transformation matrix tf between tissue section feature points and DAPI staining feature points is calculated using the CPD related point drift algorithm; The transformation matrix tf is used to match DAPI staining feature points, and the number of feature points in tissue slices is counted. If the number exceeds 10,000, random sampling is performed to reduce the computational load, and Euclidean distance is used to calculate the matched feature point pairs. (54) When the number of matches in all sub-boxes exceeds 20 pairs, use the cv2.estimateAffine2D function to generate a transformation matrix tf1 for the tissue slice feature points and DAPI staining feature points in the matching pairs; (55) Transform the DAPI-colored feature points using the transformation matrix tf1, recalculate the matching pairs, and then use the cv2.estimateAffine2D function again to generate the transformation matrix tf2.

5. The method for registering tissue section images and DAPI-stained images according to claim 4, characterized in that, The specific process of step (51) is as follows: The cv2.getRotationMatrix2D function in OpenCV is used to generate a rotation matrix M. The rotation matrix M and the feature points of the tissue slice are used to calculate the matching DAPI staining feature points. The rotation matrix M and the calculated matching DAPI staining feature points are then used to calculate the new matching sub-box. Then, the cv2.getRotationMatrix2D function in OpenCV is used to calculate the tissue slice feature points within the new subframe after matching.

6. The method for registering tissue section images and DAPI-stained images according to claim 4, characterized in that, In step (53), the cdist function in scipy is used to calculate the Euclidean distance between the matched DAPI staining feature points and the tissue slice feature points. The distance is required to be no more than 1 / 50 of the width of the new subframe.

Images