Method for matching and displaying different stained pathological section images

Abstract

The present invention relates to a method for matching and displaying images of different stained pathological sections, comprising the following steps: a. Divide multiple pathological sections into a main section and auxiliary sections; b. Obtain the affine transformation matrix between the main section and each auxiliary section; c. Synchronously display the main section and the auxiliary sections on a main screen and an auxiliary screen respectively. The present invention enables a pathologist to accurately observe the same area of consecutive sections, so as to improve the experience of reading slides and enable better analysis and diagnosis.

Description

Technical Field

[0001] The present invention relates to a method for matching and displaying images of pathological sections with different stains. Background Art

[0002] Pathological examination is an important criterion for the diagnosis of malignant tumors. Mainly, a pathologist makes a pathological glass slide of a diseased tissue of a certain size according to the pathological histological method, and then observes it after generating a pathological digital section through a microscope or a pathological scanning device. This makes the pathological digital section gradually become the mainstream way of reading pathological slides, and various pathological digital section display technologies are becoming increasingly mature. During the pathological section diagnosis process, a pathologist usually needs to continuously section the same diseased tissue and stain it with different reagents (such as HE staining, immunohistochemical IHC staining, etc.) to make multiple pathological glass slides. Therefore, the pathologist needs to compare and observe the pathological glass slides with different stains, exclude normal tissues and necrotic tissues, etc., and finally determine the location of the tumor area and make corresponding judgments.

[0003] Existing pathological digital section display systems usually can only display a single section, or display multiple sections without association. This makes it necessary for a pathologist to switch sections when reading images of consecutive pathological sections with different stains. However, due to the large differences in the conditions of pathological sections with different stains, it is difficult to manually move them to the same area, and existing pathological digital section display technologies cannot achieve automatic matching and display of different section images. Therefore, how to synchronously display different stained sections on the corresponding screen has become an urgent problem to be solved in the field. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for matching and displaying images of pathological sections with different stains.

[0005] To achieve the above-mentioned invention purpose, the present invention provides a method for matching and displaying images of pathological sections with different stains, including the following steps:

[0006] a. Divide multiple pathological sections into a main section and sub-sections;

[0007] b. Obtain the affine transformation matrix between the main section and each sub-section;

[0008] c. Synchronously display the main section and the sub-sections on the main screen and the sub-screens respectively.

[0009] According to one aspect of the present invention, in the step (b), feature points are extracted from each section, and image registration is performed to obtain the affine transformation matrix.

[0010] According to one aspect of the present invention, the feature point extraction includes contour feature point extraction and texture feature point extraction.

[0011] According to one aspect of the present invention, the contour feature points are extracted as follows: perform Sobel edge detection on the 1.25x pathological section image to generate a binary image, and eliminate the holes, impurities, and noises inside the edge through connected component analysis;

[0012] Use morphological filtering to eliminate the edge stripes in the section background and obtain a binary image of the section contour information;

[0013] Use the ORB operator to perform corner point operations on the binary image of the section contour information to obtain 1.25x key points;

[0014] Use an outlier detection algorithm to screen the 1.25x key points to obtain the contour feature points.

[0015] According to one aspect of the present invention, the texture feature points are extracted as follows: divide 8 - 32 effective image regions of 20x within the range of the contour feature points by a random uniform distribution method to form a 20x pathological section image;

[0016] Perform gray - level processing on the 20x pathological section image and calculate the Gaussian difference pyramid, and then use the texture information of the pathological image to apply the SIFT operator for key point detection to obtain 20x key points;

[0017] Use an outlier detection algorithm to screen the 20x key points to obtain the texture feature points.

[0018] According to one aspect of the present invention, the image registration includes the following steps:

[0019] Perform ORB edge matching on the 1.25x pathological section image, output the rough affine matrix M2 reflecting the position relationship, and complete the rough matching;

[0020] Perform SIFT texture matching on the 20x pathological section image, output the fine affine matrix M3 reflecting the position relationship, and complete the fine matching;

[0021] Fuse the rough affine matrix M2 and the fine affine matrix M3 to obtain the affine transformation matrix M4 between the input image pairs.

[0022] According to one aspect of the present invention, the rough matching includes the following steps:

[0023] Perform KNN matching on the contour feature points extracted from each pathological section, and output the pairwise corresponding matching pairs r1;

[0024] Perform random sample consensus calculation on the matching pairs r1 through the RANSAC algorithm, filter out the matching pairs with low consensus scores, obtain the matching pairs r2, and perform GMS matching on them to obtain the matching pairs r3;

[0025] Calculate the affine matrix M1 according to the matching pair r3, and perform position conversion on the matching pair r3 according to the affine matrix M1 to obtain the position set C1 of the feature points in the new coordinate domain;

[0026] Use the K-means clustering algorithm to cluster the positions of the feature points in the position set C1, remove the abnormal positions therein, obtain the position set C2, and use it to recalculate and obtain the rough affine matrix M2;

[0027] The fine matching includes the following steps:

[0028] Use the rough affine matrix M2 to perform position mapping on the 20x effective region images of each pathological section, screen and perform GMS matching on the texture feature points according to the mapping relationship, and output the pairwise corresponding matching pairs r4;

[0029] Perform random sample consensus calculation on the matching pair r4 through the RANSAC algorithm, filter out the matching pairs with low consensus scores, obtain the matching pair r5, and use it to calculate and obtain the fine affine matrix M3.

[0030] According to one aspect of the present invention, the affine transformation matrix M4 is a 2×3 matrix where M 00 represents the scale scaling amount in the X direction, M 01 represents the cosine value of the rotation angle, M 02 represents the offset T in the X direction x , M 10 represents the negative of the sine value of the rotation angle, M 11 represents the scale scaling amount in the Y direction, M 12 represents the offset T in the Y direction y ;

[0031] The image registration further includes:

[0032] Calculate the rotation angle angle of the image through the following formula:

[0033]

[0034] Calculate the position relationship after registration of different stained pathological section images through the rotation angle angle:

[0035]

[0036]

[0037]

[0038]

[0039] Among them, Scale x and Shear x are respectively the scaling and shearing in the x direction; Scale y and Shear y are respectively the scaling and shearing in the Y direction.

[0040] According to one aspect of the present invention, in the step (b), it further includes mapping the positions of each slice, including:

[0041] Obtain the affine transformation matrix M4 of the main slice P m and the secondary slice P n . Then, for any point coordinate (x m , y Pm ) in the main slice P Pm , the conversion formula for the corresponding coordinate (x n , y Pn ) in the secondary slice P Pn is:

[0042] x Pm = M 00 * x Pn + M 01 * y Pn + M 02 ;

[0043] And

[0044] y Pm = M 10 * x Pn + M 11 * y Pn + M 12 ;

[0045] Obtain the affine transformation matrix M4 between the main slice P m and another secondary slice P n+1 . Get the corresponding coordinate (x m , y Pm ) in the secondary slice P Pm for any point coordinate (x n+1 , y Pn+1 ) in the main slice P Pn+1 ), and then perform coordinate conversion through the conversion formula to complete the position conversion between the secondary slice P n and the secondary slice P n+1 .

[0046] According to one aspect of the present invention, in the step (c), after a certain pathological slice is moved,

[0047] if this slice is the main slice m, directly make other secondary slices move according to the coordinates (x m, y m ) Synchronously complete the position movement;

[0048] If the slice is a secondary slice p, first convert the coordinates (x p , y p ) after its change in the corresponding secondary screen and the main slice m through the affine transformation matrix M4 to obtain the coordinates (x m , y m ) of the main slice m after change on the main screen, and then make other secondary slices synchronously complete the position movement according to the coordinates (x m , y m );

[0049] The synchronous movement method of other secondary slices is that the main screen first sends the coordinates (x m , y m ) to the event management center, and the event management center broadcasts and synchronizes to the secondary screens where other secondary slices are located, and then through the affine transformation matrix M4 between the main screen and each secondary screen, makes other secondary slices complete the position movement according to the change of the main screen.

[0050] According to the concept of the present invention, a display method for pathological consecutive slices with different stains is proposed. Aiming at the pathological diagnosis and film reading scenario, it realizes the precise registration of pathological slices under different pathological reagent staining processes, thus greatly facilitating the pathologist's observation of the same area of consecutive slices, improving the film reading experience, and helping the pathologist to better perform analysis and diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematically showing the flowchart of the method for matching and displaying different stained pathological slice images according to an embodiment of the present invention;

[0052] Figure 2 Schematically showing the flowchart of ORB edge matching according to an embodiment of the present invention;

[0053] Figure 3 Schematically showing the flowchart of SIFT texture matching according to an embodiment of the present invention;

[0054] Figure 4 Schematically showing the schematic diagram of the registration result of different stained pathological slices according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0056] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments cannot be enumerated one by one here, but the embodiments of the present invention are not limited to the following embodiments.

[0057] For the method for matching and displaying different stained pathological section images of the present invention, first, consecutive sections of the same tissue are stained with different staining reagents, and the corresponding digital pathological sections are obtained through a pathological scanner. Then, these sections are divided into a main section and several sub-sections. Next, the affine transformation matrix between the main section and each sub-section is obtained. Finally, the main section and the sub-sections are synchronously displayed on the main screen and the sub-screen respectively. Among them, the main section can be randomly selected, generally the section in the first order is sufficient. Since the digital pathological section images are large in size and there are multiple different magnifications, when the present invention obtains the affine transformation matrix between the main section and each sub-section, feature points are extracted and screened for each section image at different magnifications of 1.25x and 20x respectively, so as to provide feature points of the pathological section images at different scales. Among them, feature point extraction includes contour feature point extraction and texture feature point extraction. In addition, during the process of staining consecutive sections of the same tissue with different reagents, errors will also be introduced, resulting in various types of positional deviations such as translation, rotation, scale scaling, and mirroring between the differently stained pathological sections. Therefore, after the feature points are extracted, the present invention also performs image registration on the images of the pathological sections at different magnifications respectively, so as to obtain the affine transformation matrix between two sections.

[0058] See Figure 1 , during the process of image registration, first perform ORB edge matching on the 1.25x pathological section image, output the rough affine matrix M2 reflecting the positional relationship, and complete the rough matching. Then perform SIFT texture matching on the 20x pathological section image, output the fine affine matrix M3 reflecting the positional relationship, and complete the fine matching. Finally, fuse the rough affine matrix M2 and the fine affine matrix M3 to obtain the affine transformation matrix M4 between the input image pairs.

[0059] See Figure 2 , for contour feature point extraction, perform sobel edge detection on the 1.25x pathological section image to generate a binary image, and eliminate the holes, impurities, and noises inside the edge through connected component analysis. Among them, the impurities and noises mainly refer to the small islands outside the tumor. Subsequently, morphological filtering is used to eliminate the edge stripes in the section background and obtain the accurate binary image of the section contour information. Then use the ORB operator to perform corner point operations on the binary section contour image (i.e., the binary image of the section contour information), obtain a large number of 1.25x key points, and then use the outlier detection algorithm to screen the 1.25x key points to obtain the correct contour feature points in the 1.25x pathological section image.

[0060] Then, these low-magnification contour feature points can be used for rough matching through the corresponding edge matching algorithm. Specifically, the contour feature points extracted from each pathological section are subjected to KNN matching, and the pairwise corresponding matching pairs r1 are output. The random sample consensus calculation is performed on the matching pairs r1 through the RANSAC algorithm, the matching pairs with low consensus scores are filtered out, the matching pairs r2 are obtained, and GMS matching is performed on them to obtain the matching pairs r3. The affine matrix M1 is calculated according to the matching pairs r3, and the positions of the results in the matching pairs r3 are converted according to the affine matrix M1 to obtain the position set C1 of the feature points in the new coordinate domain. Since the offset amounts of different position points in the same coordinate domain are basically the same, the K-means clustering algorithm is used in the present invention to cluster the positions of the feature points in the position set C1, remove the abnormal positions therein, obtain the position set C2, and use it to recalculate the rough affine matrix M2.

[0061] See Figure 3 , the texture feature points are extracted as follows: 8-32 effective image regions of 20x are divided within the position range of the contour feature points of 1.25x in a randomly uniformly distributed manner to form a 20x pathological section image. The 20x pathological section image is subjected to grayscale processing and the Gaussian difference pyramid is calculated, and then the SIFT operator is used for key point detection by applying the texture information of the pathological image to obtain the 20x key points. The 20x key points are screened by using the outlier detection algorithm to obtain the correct texture feature points in the 20x pathological section image.

[0062] Then, fine matching is performed on these high-magnification texture feature points through the corresponding texture matching algorithm. Specifically, the 20x effective region images of each pathological section are subjected to position mapping by using the rough affine matrix M2, the high-magnification texture feature points are preliminarily screened according to the mapping relationship, and GMS matching is performed on the screened texture feature points to output the pairwise corresponding matching pairs r4. Then, the random sample consensus calculation is performed on the matching pairs r4 through the RANSAC algorithm, the matching pairs with low consensus scores are filtered out, the matching pairs r5 are obtained, and the fine affine matrix M3 is calculated by using them.

[0063] Subsequently, the rough affine matrix M2 obtained by ORB edge matching and the fine affine matrix M3 obtained by SIFT texture matching can be subjected to feature fusion to obtain the accurate (i.e., corrected) affine transformation matrix M4 between the input image pairs, thereby completing the conversion of the positional relationship between the main section and each sub-section. The affine transformation matrix M4 is a 2×3 matrix which can represent the translation, rotation, scaling, and shear of the image. Among them, M 00 represents the scale scaling factor in the X direction, M 01 represents the cosine value of the rotation angle, M 02 represents the offset T in the X direction x, M 10 represents the negative of the sine value of the rotation angle, M 11 represents the scaling factor in the Y direction, M 12 represents the offset T in the Y direction y . The rotation angle angle of the image can be approximately calculated by the following formula:

[0064]

[0065] By the rotation angle angle, other components can be calculated, and the positional relationship of different stained pathological section images after registration is as follows:

[0066]

[0067]

[0068]

[0069]

[0070] where Scale x and Shear x are the scaling and shearing in the x direction respectively; Scale y and Shear y are the scaling and shearing in the Y direction respectively.

[0071] After image registration, the positions of each slice can be mapped. Specifically, after obtaining the affine transformation matrix M4 m between the main slice P n and each sub-slice P by the above image matching technique, for any point coordinate (x m , y Pm ) in the main slice P Pm , the conversion formula for the corresponding coordinate (x n , y Pn ) in the sub-slice P Pn is:

[0072] x Pm = M 00 * x Pn + M 01 * y Pn + M 02 ;

[0073] and

[0074] y Pm = M 10 * x Pn + M 11 * y Pn+M 12 .

[0075] Thus, according to the affine transformation matrix M4 between the main slice P m and another secondary slice P n+1 , the coordinates (x m , y Pm ) of any point in the main slice P Pm can be obtained, and the corresponding coordinates (x n+1 , y Pn+1 ) in the secondary slice P Pn+1 ) can be obtained. Then, through the above two coordinate conversions of x and y, the position conversion between the secondary slice P n and the secondary slice P n+1 can be completed. In this way, the mapping of the positions of multiple (N) slices can be completed.

[0076] As Figure 4 shown, the main slice is displayed on the main screen display interface, and each secondary slice is displayed on the secondary screen display interface according to the user's needs. After a certain pathological slice is moved, the position of the upper left window of the slice will change, thereby causing a change in its coordinates in the corresponding screen. If the slice is the main slice m, directly make other secondary slices move synchronously according to the coordinates (x m , y m ) after the main slice m is moved; if the slice is the secondary slice p, first record its current (i.e., changed) slice coordinates (x p , y p ) in the view of the corresponding secondary screen display interface, and perform coordinate conversion on the coordinates (x p , y p ) and the main slice m through the affine transformation matrix M4, so as to obtain the changed coordinates (x m , y m ) of the main slice m in the view of the main screen, so as to complete the view synchronization between the main screen and the secondary screen where the secondary slice p is located, and then make other secondary slices move synchronously according to the coordinates (x m , y m ).

[0077] In the present invention, the way for other secondary slices to move synchronously is to first convert the view coordinates (x m , y m)It is sent to the event management center of the used software. The event management center broadcasts and synchronizes it to the secondary screens where other secondary slices are located. Then, through the affine transformation matrix M4 between the main screen and each secondary screen, the other secondary slices complete synchronous position movement in the views of their respective screen display interfaces according to the changes of the main screen, so as to realize the real-time registration and linkage display of multiple pathological slices. In this way, when the user moves the pathological slice image on the main screen or any secondary screen, all screens can be synchronously moved and displayed according to the mapping relationship.

[0078] The above is only one embodiment of the present invention and is not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for matching and displaying images of different stained pathological sections, comprising the following steps: a. Divide multiple digital pathological slices into main slices and sub-slices; b. Obtain the affine transformation matrix between the main slice and each sub-slice; c. Synchronously display the main slice and the sub-slices on the main screen and the sub-screens respectively; In step (b), feature points are extracted from each slice, and image registration is performed to obtain the affine transformation matrix; the feature point extraction includes contour feature point extraction and texture feature point extraction; The image registration includes the following steps: Perform ORB edge matching on the 1.25x pathological slice image, output the rough affine matrix M2 reflecting the position relationship, and complete the rough matching; Perform SIFT texture matching on the 20x pathological slice image, output the fine affine matrix M3 reflecting the position relationship, and complete the fine matching; Fuse the rough affine matrix M2 and the fine affine matrix M3 to obtain the affine transformation matrix M4 between the input image pairs.

2. The method according to claim 1, characterized in that, The contour feature point extraction is as follows: perform sobel edge detection on the 1.25x pathological slice image to generate a binary image, and eliminate the holes, impurities, and noises inside the edge through connected component analysis; Use morphological filtering to eliminate the edge stripes in the slice background and obtain the binary image of the slice contour information; Use the ORB operator to perform corner point operations on the binary image of the slice contour information to obtain the 1.25x key points; Use the outlier detection algorithm to screen the 1.25x key points to obtain the contour feature points.

3. The method according to claim 2, characterized in that, The texture feature point extraction is as follows: divide 8 - 32 effective image regions of 20x within the range of the contour feature points in a randomly uniformly distributed manner to form the 20x pathological slice image; Perform gray-scale processing on the 20x pathological slice image, calculate the Gaussian difference pyramid, and then use the texture information of the pathological image to apply the SIFT operator for key point detection to obtain the 20x key points; Use the outlier detection algorithm to screen the 20x key points to obtain the texture feature points.

4. The method according to claim 1, characterized in that, The rough matching includes the following steps: Perform KNN matching on the contour feature points extracted from each pathological slice, and output the pairwise corresponding matching pairs r1; Perform random sample consensus calculation on the matching pairs r1 through the RANSAC algorithm, filter out the matching pairs with low consensus scores, obtain the matching pairs r2, and perform GMS matching on them to obtain the matching pairs r3; Calculate the affine matrix M1 according to the matching pairs r3, and perform position conversion on the matching pairs r3 according to the affine matrix M1 to obtain the position set C1 of the feature points in the new coordinate domain; Use the K-means clustering algorithm to cluster the positions of the feature points in the position set C1, remove the abnormal positions among them to obtain the position set C2, and use it to recalculate and obtain the rough affine matrix M2; The fine matching includes the following steps: Use the rough affine matrix M2 to perform position mapping on the 20x effective region images of each pathological slice, screen and perform GMS matching on the texture feature points according to the mapping relationship, and output the pairwise corresponding matching pairs r4; Perform random sample consensus calculation on the matching pairs r4 through the RANSAC algorithm, filter out the matching pairs with low consensus scores, obtain the matching pairs r5, and use them to calculate and obtain the fine affine matrix M3.

5. The method according to claim 4, characterized in that, The affine transformation matrix M4 is a 2×3 matrix where M 00 represents the scaling factor in the X direction, M 01 represents the cosine value of the rotation angle, M 02 represents the offset T in the X direction x , M 10 represents the negative of the sine value of the rotation angle, M 11 represents the scaling factor in the Y direction, M 12 represents the offset T in the Y direction y ; The image registration also includes: The rotation angle angle of the image is calculated by the following formula: The positional relationship of different stained pathological section images after registration is calculated by the rotation angle angle: Among them, Scale x and Shear x are the scaling and shearing in the X direction respectively; Scale y and Shear y are the scaling and shearing in the Y direction respectively.

6. The method according to claim 5, characterized in that, In the step (b), it also includes mapping the positions of each section, including: Obtain the main slice P m and the affine transformation matrix M4 of the secondary slice P n , then for any point coordinate (x m , y Pm ) in the main slice P Pm , the corresponding coordinate (x n , y Pn ) in the secondary slice P Pn ) has the conversion formula: x Pm = M 00 * x Pn + M 01 * y Pn + M 02 ; and y Pm = M 10 * x Pn + M 11 * y Pn + M 12 ; Obtain the main slice P m and another secondary slice P n+1 to get the affine transformation matrix M4 between them, and obtain any point coordinate (x m , y Pm ) in the main slice P Pm and its corresponding coordinate (x n+1 , y Pn+1 ) in the secondary slice P Pn+1 . Then perform coordinate transformation through the conversion formula to complete the position transformation between the secondary slice P n and the secondary slice P n+1 .

7. The method according to claim 6, wherein, In the step (c), after a certain pathological section is moved, If this slice is the main slice m, directly move other secondary slices according to the coordinates (x m , y m ) after the movement of the main slice m to synchronously complete the position movement; If the slice is a secondary slice p, first convert the coordinates (x p , y p ) after the change in the corresponding secondary screen and the primary slice m through the affine transformation matrix M4 to obtain the coordinates (x m , y m ) after the change of the primary slice m on the primary screen, and then move the positions of other secondary slices synchronously according to the coordinates (x m , y m ); The way for other secondary slices to move synchronously is that the main screen first sends the coordinates (x m , y m ) to the event management center. The event management center broadcasts and synchronizes them to the secondary screens where other secondary slices are located, and then through the affine transformation matrix M4 between the main screen and each secondary screen, other secondary slices complete position movement according to the changes of the main screen.

Images