Pathology slide oriented cross-modality image registration alignment method

Abstract

The pathological section cross-modal image registration alignment method disclosed by the application obtains H&E images and IHC images and respectively performs pyramid multi-level down-sampling, extracts a tissue mask under low resolution, performs feature extraction and matching in a mask constraint region to solve a spatial transformation matrix, then obtains an original resolution transformation matrix through scale normalization, performs spatial alignment on the IHC original image and the H&E original image, finally takes the aligned H&E image as a reference block, extracts a corresponding block in the IHC aligned image according to an inverse matrix of the transformation matrix, and outputs a pair of normalized image blocks. The application solves the technical problems of high feature point mismatching rate and insufficient registration precision caused by background interference in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a cross-modal image registration and alignment method for pathological slides. Background Technology

[0002] Whole-slide images (WSI) have become a core data foundation for clinical pathology diagnosis and artificial intelligence research. When establishing a multimodal pathology AI-assisted analysis system, it is typically necessary to perform pixel-level registration and alignment of different stained modalities of the same tissue source—namely, H&E-stained sections and IHC immunohistochemical-stained sections—to obtain spatially corresponding multimodal image pairs. This provides reliable data support for subsequent diagnostic analysis, quantitative assessment, and deep learning model training. Because H&E staining and IHC staining reflect different dimensions of biological information—H&E staining showing tissue structure and morphology, and IHC staining showing specific protein expression—the two are complementary in clinical diagnosis. Establishing a precise spatial correspondence between the two modalities is a prerequisite for multimodal pathology analysis.

[0003] However, existing cross-modal pathological image registration methods still face the following prominent problems in practical applications: First, the original resolution of whole pathological slide images is extremely high, with the size of a single image typically reaching tens of thousands to hundreds of thousands of pixels. Directly performing feature extraction and registration calculations at the original resolution results in a huge memory footprint and a long computation time, making it difficult to meet the high-throughput requirements of large-scale dataset preprocessing.

[0004] Secondly, pathological images often contain a large number of blank background areas without tissue. Existing registration methods do not spatially constrain the effective tissue area. During feature extraction, noise points in the background area are misidentified as feature points, resulting in a large number of mismatches and severely reducing registration accuracy.

[0005] Third, most methods only complete the registration on the downsampled low-resolution image, but lack a scale normalization method to accurately restore the registration relationship to the original resolution. This results in spatial misalignment and loss of organizational correspondence in the block image at the original resolution, which cannot meet the analysis requirements of pixel-level accuracy.

[0006] Fourth, existing methods typically perform image registration, spatial alignment, and block slicing as independent steps, lacking an integrated process design. The operation is cumbersome, and the block sizes are inconsistent with poor spatial correspondence, making it impossible to form a standardized pairwise dataset output that can be directly used for model training.

[0007] In summary, existing cross-modal pathological image registration methods have significant shortcomings in terms of processing efficiency, registration accuracy, scale mapping accuracy, and process standardization. Summary of the Invention

[0008] The purpose of this invention is to provide a cross-modal image registration and alignment method for pathological slides, which solves the technical problems of high feature point mismatch rate and insufficient registration accuracy caused by background interference in the prior art.

[0009] The technical solution adopted in this invention is a cross-modal image registration and alignment method for pathological sections. This method acquires H&E and IHC images and performs multi-level pyramid downsampling on each. Tissue masks are extracted at low resolution, and feature extraction and matching are performed within the mask-constrained region to solve for the spatial transformation matrix. Then, the original resolution transformation matrix is ​​obtained through scale normalization. The original IHC image and the original H&E image are spatially aligned. Finally, the aligned H&E image is used as a reference for segmentation, and corresponding segments are extracted from the aligned IHC image based on the inverse of the transformation matrix. Standardized image blocks are output in pairs.

[0010] The invention is further characterized in that, Specifically, the following steps are included: Step 1: Obtain complete image data of H&E stained pathological slide images and IHC immunohistochemical pathological slide images, and perform pyramid multi-level downsampling on them to obtain low-resolution registration layer images; Step 2: Extract tissue regions from the low-resolution registration layer images of H&E stained pathological slide images and IHC immunohistochemical pathological slide images respectively, and generate corresponding tissue masks. Step 3: Under the constraint of the tissue mask, feature point extraction and feature descriptor matching are performed only on the low-resolution registration layer images of H&E stained pathological slide images and IHC immunohistochemical pathological slide images within the tissue region, and the spatial transformation matrix under low resolution is solved. Step 4: Based on the scaling ratio corresponding to the downsampling level, scale the spatial transformation matrix at low resolution to obtain the spatial transformation matrix at the original resolution. Step 5: Based on the spatial transformation matrix at the original resolution, perform a spatial transformation on the original resolution image of the IHC immunohistochemical pathological slide image to spatially align it with the original resolution image of the H&E stained pathological slide image. Step 6: Based on the original resolution image of the aligned H&E stained pathological slide, perform standardized segmentation, and extract the corresponding segments in the aligned image of the IHC immunohistochemical pathological slide according to the inverse matrix of the spatial transformation matrix under the original resolution, and output the standardized image blocks in pairs.

[0011] The specific process of step 1 is as follows: Let the H&E stained pathological section image be H ori IHC immunohistochemical pathological section images are I oriThe image is processed using bicubic interpolation with a pyramid structure and multi-level downsampling. Pyramid level 0 represents the original highest resolution, and the resolution decreases with each subsequent level. The image size at each level is half the original size. level Where level = 0, 1, 2, ..., 9; select level ∈ (3, 6) as the low-resolution registration layer, and obtain the low-resolution registration layer image H. level and I level .

[0012] Step 2 is as follows: The low-resolution registration layer image is converted to grayscale, transforming the three-channel color image into a single-channel grayscale image. Subsequently, the Otsu algorithm was used to calculate the optimal segmentation threshold. The grayscale image was then segmented by thresholding. Pixels with grayscale values ​​less than the threshold were identified as tissue regions, and pixels with grayscale values ​​greater than or equal to the threshold were identified as blank background regions, thus generating an initial tissue mask. Finally, morphological closing operations are performed on the initial tissue mask using structuring elements to fill internal pores, remove isolated noise points, and smooth tissue edges, generating the final tissue mask.

[0013] The specific process of step 3 is as follows: Under the constraint of the tissue mask, feature points and binary feature descriptors are extracted only from the H&E low-resolution image and the IHC low-resolution image within the tissue region, respectively. The BFMatcher matching algorithm, using Hamming distance as the metric, is employed to perform cross-validation matching on the two sets of extracted feature descriptors. A smaller Hamming distance indicates a higher similarity between the feature descriptors. The formula for calculating the Hamming distance is shown below: ; In the formula, These are the binary feature descriptors of tissue regions for the H&E reference image and the IHC image to be registered, respectively. , The first two descriptors are respectively the first two descriptors. i Bit binary value, The Hamming distance; High-quality matching points are selected based on a predetermined ratio according to matching distance, and the homography matrix M at low resolution is solved and optimized using the RANSAC algorithm. low .

[0014] Step 4 specifically involves obtaining the scaling ratio corresponding to the downsampling level in Step 1, as shown in the following expression: ; In the formula, D is the downsampling scaling factor; Low-resolution registration matrix based on the scaling ratio corresponding to the downsampling level Scale normalization calculations are performed to obtain a homography matrix that can be used at the original resolution. The calculation expression is shown below: .

[0015] The specific process of step 5 is as follows: Establish the original resolution pixel coordinate mapping relationship, let (x ori ,y ori (x') represents the pixel coordinates of the IHC raw resolution image. ori ,y' ori The coordinates are the pixel coordinates mapped to the original H&E resolution image after registration. The mapping relationship is expressed as follows: ; Based on the above mapping relationship, using the original resolution image of H&E as a reference, a perspective transformation is performed on the original resolution image of IHC to complete global spatial alignment, resulting in the aligned IHC image. align .

[0016] The specific process of step 6 is as follows: H&E original resolution image H ori Based on the standard, according to the fixed size patch size Traverse the blocks and determine the starting coordinates (x0, y0) and actual width w and height h of each block; The inverse matrix of the original resolution homography matrix is ​​calculated using the following expression: ; The coordinates of the four corner points of the H&E slice are extracted based on the starting coordinates of each slice and its actual width and height. The expression is as follows: ; The corner coordinates are mapped to the aligned IHC image using an inverse matrix. align Obtain the corresponding IHC cut block corner coordinates Extract the corresponding IHC slices based on the mapped coordinates and perform perspective correction. Save the H&E slices and IHC slices in pairs according to a unified naming rule. Output a standardized pairwise dataset that can be directly used for model training.

[0017] The beneficial effects of this invention are: (1) The present invention adopts a pyramid multi-level downsampling strategy to complete feature extraction and registration calculation at a low resolution level, which greatly reduces the amount of image data and memory usage, significantly reduces the computation time, and can efficiently process massive pathological whole slice images to meet the high-throughput requirements of large-scale pathological dataset preprocessing.

[0018] By generating tissue region masks to spatially constrain the feature extraction region, feature points are extracted and matched only within the effective tissue region, completely shielding noise interference from blank background regions, effectively eliminating mismatches caused by background regions, and significantly improving the accuracy of feature matching and registration stability between cross-modal images.

[0019] (2) This invention eliminates the scale deviation caused by downsampling by performing scale normalization calculation on the homography matrix solved at low resolution, accurately restores the low resolution registration relationship to the original resolution, ensures the pixel-level spatial alignment accuracy between the IHC image and the H&E image at the original resolution, and avoids the problem of tissue structure misalignment caused by scale transformation.

[0020] Simultaneously, image reading, downsampling, mask generation, feature registration, matrix normalization, original resolution alignment, and synchronous block segmentation are integrated into a single process. While completing the registration and alignment, it directly outputs standardized paired image blocks with strictly corresponding spatial positions at the original resolution. Each block has a uniform size and accurate spatial correspondence, which can be directly used as the training input for deep learning models, eliminating the need for manual post-processing steps.

[0021] (3) This invention is applicable to cross-modal registration and alignment between multiple modal pathological images such as H&E staining and IHC staining. It has good adaptability to differences in image size, staining style, and tissue morphology. It can be widely used in various scenarios such as the construction of training datasets for pathological AI models, multimodal image fusion analysis, virtual pathological staining, and digital pathological quantitative analysis. Attached Figure Description

[0022] Figure 1a This is an image of the H&E-stained pathological slide to be processed in Embodiment 2 of the present invention; Figure 1b These are IHC immunohistochemical pathological slide images to be processed in Example 2 of this invention; Figure 2a This invention is for Figure 1a Image of H&E stained pathological section after grayscale processing; Figure 2b This invention is for Figure 1b Image of H&E stained pathological section after grayscale processing; Figure 3a This invention is for Figure 2aGenerate H&E stained pathological section images after tissue masking; Figure 3b This invention is for Figure 2b Generate H&E stained pathological section images after tissue masking; Figure 4 This is a schematic diagram of feature point matching between H&E and IHC in this invention; Figure 5a This is a schematic diagram showing the overlay comparison of H&E and IHC before registration in Embodiment 3 of the present invention; Figure 5b This is a schematic diagram showing the superimposed comparison of H&E and IHC after registration in Embodiment 3 of the present invention; Figure 6a This is an image of the H&E-stained pathological slide to be processed in Example 7 of the present invention; Figure 6b These are IHC immunohistochemical pathological slide images to be processed in Example 7 of this invention; Figure 7a This is an image of the H&E-stained pathological slide to be processed in Example 8 of the present invention; Figure 7b These are IHC immunohistochemical pathological slide images to be processed in Example 8 of this invention; Figure 8a This is an image of the H&E-stained pathological slide to be processed in Example 9 of the present invention; Figure 8b These are IHC immunohistochemical pathological slide images to be processed in Example 9 of this invention; Figure 9a This is an image of the H&E-stained pathological slide to be processed in Example 10 of the present invention; Figure 9b The image shown is an IHC immunohistochemical pathological section to be processed in Example 10 of this invention. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Example 1 This invention provides a cross-modal image registration and alignment method for pathological sections. It acquires H&E and IHC images and performs pyramid multi-level downsampling on each. Tissue masks are extracted at low resolution, and feature extraction and matching are performed within the mask-constrained regions to solve for the spatial transformation matrix. Then, the original resolution transformation matrix is ​​obtained through scale normalization. The original IHC image and the original H&E image are spatially aligned. Finally, the aligned H&E image is used as a reference for segmentation, and corresponding segments are extracted from the aligned IHC image based on the inverse of the transformation matrix. Standardized image blocks are output in pairs.

[0025] The method of this invention, in the context of pathological image processing, aims to achieve precise registration and alignment of H&E and IHC pathological sections and standardized paired sections through downsampling preprocessing, tissue mask constraints, feature matching registration, original resolution mapping, and image segmentation. Specifically, it is implemented according to the following steps: Step 1: Obtain complete image data of H&E stained pathological slide images and IHC immunohistochemical pathological slide images, and perform pyramid multi-level downsampling on them to obtain low-resolution registration layer images; In this embodiment, the input image size is... The H&E stained pathological slide image is Size is The IHC immunohistochemical pathological section images are A multi-level downsampling process using bicubic interpolation with a pyramid structure is performed, where pyramid level 0 represents the original highest resolution, and the resolution decreases with each higher pyramid level. level For the selected pyramid level, ， Indicates correspondence level The relationship between the layer and the image size for the sliced ​​image is shown in the following formula (1): (1); In the formula, level In this embodiment, preferably For example, pyramid levels level=4 , The scaling factor corresponds to an image size that is half the original size. 4 That is, 1 / 16 of the original resolution; Acquire low-resolution registration layer images This layer can effectively preserve the core morphological features of the tissue while significantly reducing the amount of image data, thus reducing the computational load of subsequent feature extraction and registration, and obtaining a complete image at this layer.

[0026] Step 2: Extract tissue regions from the low-resolution registration layer images of H&E stained pathological slide images and IHC immunohistochemical pathological slide images respectively, and generate corresponding tissue masks. Step 3: Under the constraint of the tissue mask, feature point extraction and feature descriptor matching are performed only on the low-resolution registration layer images of H&E stained pathological slide images and IHC immunohistochemical pathological slide images within the tissue region, and the spatial transformation matrix under low resolution is solved. Step 4: Based on the scaling ratio corresponding to the downsampling level, scale the spatial transformation matrix at low resolution to obtain the spatial transformation matrix at the original resolution. Step 5: Based on the spatial transformation matrix at the original resolution, perform a spatial transformation on the original resolution image of the IHC immunohistochemical pathological slide image to spatially align it with the original resolution image of the H&E stained pathological slide image. Step 6: Based on the original resolution image of the aligned H&E stained pathological slide, perform standardized segmentation, and extract the corresponding segments in the aligned image of the IHC immunohistochemical pathological slide according to the inverse matrix of the spatial transformation matrix under the original resolution, and output the standardized image blocks in pairs.

[0027] Example 2 This embodiment, based on Embodiment 1 above, performs image preprocessing on low-resolution H&E images. With IHC images The specific process of extracting the tissue region mask, i.e., step 2, is as follows: Read the H&E low-resolution image saved in step 1 With IHC low-resolution images like Figure 1a and Figure 1b As shown, the tissue outlines are intact, with glands and stroma corresponding in distribution, and no obvious fragmentation or deformation. The H&E section resolution is 107520×96768, and the IHC section resolution is 110208×96768, with only minor differences in lateral scale, which places demands on the fine-scale adaptation capability of the registration method. The background area of ​​the sections is moderate, and the feature distribution is stable, providing a good foundation for registration.

[0028] The image is converted to grayscale. Grayscale processing involves fusing the three-channel information into single-channel brightness information, thereby maximizing the brightness difference between the tissue area and the blank background, which facilitates the subsequent separation of the tissue from the background. The grayscale formula is shown in equation (2) below: (2); In the formula, , , yes and The pixel values ​​of the red, green, and blue channels of the corresponding layers in the two modal images. , The standard grayscale weighting coefficients are used for grayscale images, as shown in the image. Figure 2a and Figure 2b As shown; Figure 2a , Figure 2b The grayscale results of H&E and IHC sections were respectively used. After fusing the RGB three channels using the standard grayscale formula, the brightness difference between the tissue area and the pure white background was significantly amplified, which can clearly distinguish the effective tissue from the blank slide.

[0029] Subsequently, threshold segmentation is performed on the grayscale image to extract tissue regions. Tissue regions are the parts of the image that contain color, cells, glands, and structures; these are the truly useful pathological tissues. A segmentation threshold is then set. T The expression for the segmentation threshold is shown in equation (3) below: (3); In the formula, For its coordinates The grayscale value of the point T To determine the optimal threshold for distinguishing between tissue and background, the Otsu algorithm is used. 255 indicates that the current pixel belongs to a tissue region, i.e., an effective area in the image containing pathological structures such as cells, glands, and stroma; 0 indicates that the current pixel belongs to a blank background, i.e., a pure white area in the image containing only glass slides and no tissue, with a grayscale value less than 255. T Classified as an organization, grayscale greater than or equal to T If it is determined to be the background, the segmentation rule formula is as follows (4): (4); Then use a size of Morphological closing operations are performed on the structural elements (preferably 5×5) to optimize the initial mask, fill the internal pores of the tissue, remove small noise points, and smooth the tissue edges to obtain the final tissue mask. The calculation formula is shown in equation (5): (5); The initial mask obtained after threshold segmentation The presence of tiny black holes, fissures, burr-like edges, or isolated noise points within the tissue can affect the stability of subsequent feature extraction. Morphological closing operations first dilate the mask and then erode it. This process fills in internal pores, eliminates edge burrs, and removes isolated noise without altering the overall position and contour of the tissue. As a result, the tissue region becomes a complete, continuous, and smooth-edged foreground area, providing a clean and accurate constraint range for subsequent feature extraction.

[0030] Final output final tissue mask like Figure 3a and Figure 3b As shown in the figure, the white area represents the identified valid pathological tissue, and the black area represents the completely masked blank background, achieving precise isolation of the tissue area. The function of this mask is to spatially constrain the subsequent feature extraction: it only allows feature extraction within the tissue area with a mask value of 255, and completely masks the blank background area with a mask value of 0, eliminating background interference and avoiding mismatch problems caused by background noise from the root, thereby improving the reliability of feature matching. Example 3 Based on Embodiment 2 above, step 3 specifically includes the following steps: S3.1, the tissue mask obtained in step 2 Under constraints, H&E low-resolution images were analyzed only within the tissue region. IHC low-resolution images Extract feature points and binary feature descriptors, denoted as: ; ; ; ; S3.2, using the existing matching algorithm (BFMatcher) with Hamming distance as the metric, cross-validation matching is performed on the tissue region feature descriptors of H&E and IHC. The Hamming distance calculation formula is shown in the following formula (6): (6); In the formula, The tissue region binary feature descriptors (length: ) for the H&E reference image and the IHC image to be registered, respectively. ), , The first two descriptors are respectively the first two descriptors. i A binary value (0 or 1). This represents the Hamming distance; a smaller value indicates a higher similarity between the feature descriptors of two tissue regions, and a stronger reliability that the corresponding feature points belong to the same tissue location. For example... Figure 4 For sample images and The diagram illustrates the matching of all feature points. The lines in the diagram represent the feature points matched between two sets of images. There are a total of 311 matching pairs. Thanks to the tissue mask constraint, all matching feature points fall within the tissue region. No invalid background features participate in the matching. After filtering the top 30% of high-quality matching points with low Hamming distance, the number of abnormal matching pairs can be greatly reduced, ensuring the effectiveness of the matching. S3.3, Solve for the homography matrix and perform robust optimization of the existing RANSAC algorithm. The homography matrix is ​​the core of describing the spatial mapping relationship between two images, denoted as . , set up For IHC low-resolution images pixel coordinates, To map the registered image to a low-resolution H&E image The pixel coordinates are expressed mathematically as shown in equation (7): (7); get The 3×3 low-resolution homography matrix is ​​expressed as shown in equation (8): (8); Finally, the comparison of the low-resolution sample images before and after registration is obtained, for example... Figure 5a and Figure 5b As shown, Figure 5a To obtain the superimposed image of the pre-registration H&E and IHC images, there is a significant spatial shift in the tissue glands and stromal structures; Figure 5b The image shows the superimposed effect after registration in this invention. The tissue structures of the two types of slices overlap, and the contours, glands, and stroma positions correspond one-to-one. This visually verifies that the low-resolution homography matrix solution of this invention can achieve accurate coarse alignment of cross-modal slices.

[0031] Example 4 This embodiment, based on Embodiment 3 above, uses the low-resolution homography matrix obtained in step 3. Combining the scaling rules of pyramid downsampling in step 1, the registration matrix is ​​scaled to eliminate the scale deviation between low-resolution registration and the original resolution. That is, the scaling ratio in step 4 is the scaling ratio corresponding to the downsampling level in step 1, as shown in the following formula (8): (8); In the formula, D is the downsampling scaling factor; Low-resolution registration matrix based on the scaling ratio corresponding to the downsampling level Scale normalization calculations are performed to obtain a homography matrix that can be used at the original resolution. The calculation expression is shown in equation (9): (9); This formula is used to normalize the scale of the registration matrix, transforming the low-resolution registration relationship into a spatial mapping relationship that adapts to the original resolution.

[0032] Example 5 This embodiment is based on Embodiment 4 above, using the original resolution homography matrix obtained in step 4. For IHC raw resolution images Perform alignment mapping to achieve pixel-level spatial alignment with the original H&E resolution image. The specific process of step 5 is as follows: Establish the original resolution pixel coordinate mapping relationship, and set IHC raw resolution image pixel coordinates, To map the registered image to the original H&E resolution image The pixel coordinates are expressed mathematically as shown in equation (10): (10); Based on the above mapping relationship, the IHC original resolution image Perform perspective transformation on the H&E original resolution image. Based on the original IHC image, global spatial alignment is performed to create a new, registered image, denoted as . .

[0033] Example 6 Based on Embodiment 5 above, step 6 of this invention is based on the aligned image obtained in step 5. and At the original resolution, the data is standardized into pairs based on strict spatial correspondence. The specific process is as follows: With H&E original images Using this as a baseline, traverse the image and slice it into pieces of a fixed size (Patch_size). Traverse the slices at a fixed step size to determine the starting coordinates of each slice. ; For each slice, start coordinates Calculate the actual width of the cut pieces. and height To avoid boundary crossing, the calculation formulas are as follows (11) and (12): (11); (12); Then, the inverse matrix of the original resolution homography matrix is ​​solved, and the calculation expression is shown in equation (13): (13); The coordinates of the four corner points of the H&E slice are extracted based on the starting coordinates of each slice and its actual width and height. The expression is shown in equation (14): (14); Map the corner coordinates to the aligned IHC image using the inverse matrix. align Obtain the corresponding IHC cut block corner coordinates The mapping expression is shown in equation (15): (15); Extract the corresponding IHC slices based on the mapped coordinates and perform perspective correction. Save the H&E slices and IHC slices in pairs according to a unified naming rule, and output a standardized pairwise dataset that can be directly used for model training.

[0034] This embodiment demonstrates, for example... Figure 1a and Figure 1b The sample images shown, after registration and alignment, achieved a structural similarity (SSIM) of 0.7789, which is excellent in multimodal pathological registration. This indicates that the global and local structural alignment accuracy is high and can meet the requirements of subsequent segmentation and model training.

[0035] Example 7 The sample images used in Example 7 are as follows: Figure 6a and Figure 6b As shown, these are multimodal pathological sections with intact tissue morphology and concentrated distribution. The overall tissue outline is clear, and the glandular structure corresponds to the stroma distribution, with no obvious tissue fragmentation or deformation. The original resolution of the H&E section is 94080×83328 pixels, and the original resolution of the IHC section is 96768×75264 pixels. There is a certain difference in the size of the two sets of sections, which places clear demands on the scale adaptation capability of the registration method. The background blank area accounts for a moderate proportion, providing sufficient and stable feature extraction areas for registration. After registration and alignment using the alignment method of this invention, the calculated structural similarity index (SSIM) is 0.7433, which is at a good level in the multimodal pathological image registration scenario.

[0036] Example 8 like Figure 7a and Figure 7b As shown, the original resolution size of the H&E slice selected in this embodiment 8 is 102144. 94080, the original resolution size of the IHC slide is 99456. Section 94080 exhibits irregular morphology, varied and tortuous edge contours, and a more fragmented arrangement of ducts and glands, resulting in a higher degree of tissue fragmentation. The differences in natural staining contrast and minor local deformations between sections are also more pronounced, significantly increasing the overall registration difficulty. After registration using the alignment method of this invention, the SSIM index for this group of samples was 0.7005. In professional scenarios of pathological multimodal (H&E-IHC) registration, this value falls within a stable, reliable, and effective range.

[0037] Example 9 The H&E staining and IHC staining paired pathological slide samples in Example 9 are as follows: Figure 8a and Figure 8b As shown, H&E's original resolution size is 115584. 69888, IHC original resolution size 110208 Sample 72576 exhibits a large, continuous, and dense tissue distribution, with a high proportion of interstitial tissue. Local cavities, cysts, and ductal structures are clearly visible, and the tissue margins are relatively intact. However, significant differences in grayscale, contrast, and color development exist between the H&E and IHC staining modalities. Slight global stretching and positional shifts are also observed. Registration requires balancing the alignment of large areas of homogeneous tissue with local characteristic structures. After registration, the SSIM structural similarity is 0.7151, which is considered a stable registration level in a multimodal pathological registration scenario with a high proportion of homogeneous interstitial tissue and significant differences in modal staining.

[0038] Example 10 The 10 sample images in this embodiment are as follows: Figure 9a and Figure 9b As shown, the original resolution of H&E is 115584×72576, and that of IHC is 115584×75264. The horizontal resolution of the slices is completely consistent, with only a slight difference in vertical height. The overall size deviation is small, providing excellent conditions for scale registration. The overall tissue exhibits a large-scale asymmetrical distribution, with sparse tissue and a very high proportion of blank background on the left, while the right side contains a large area of ​​dense functional tissue. The overall outline is narrow and irregular. Furthermore, there are significant differences in modal staining brightness and contrast between the two slices. The large area of ​​blank background easily introduces invalid feature noise, making registration difficult to resist interference. After sample registration, the SSIM structural similarity index reached 0.8130, the best level among all examples. The generated standardized paired slices can provide top-quality data support for training pathological AI models and high-precision multimodal quantitative analysis.

[0039] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0040] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cross-modal image registration and alignment method for pathological sections, characterized in that, H&E and IHC images are acquired and subjected to pyramid multi-level downsampling. Tissue masks are extracted at low resolution. Feature extraction and matching are performed within the mask-constrained regions to solve for the spatial transformation matrix. Then, the original resolution transformation matrix is ​​obtained through scale normalization. The original IHC image and the original H&E image are spatially aligned. Finally, the aligned H&E image is used as the reference for block division. The corresponding blocks are extracted from the IHC aligned image according to the inverse of the transformation matrix, and normalized image blocks are output in pairs.

2. The cross-modal image registration and alignment method for pathological sections according to claim 1, characterized in that, Specifically, the following steps are included: Step 1: Obtain complete image data of H&E stained pathological slide images and IHC immunohistochemical pathological slide images, and perform pyramid multi-level downsampling on them to obtain low-resolution registration layer images; Step 2: Extract tissue regions from the low-resolution registration layer images of H&E stained pathological slide images and IHC immunohistochemical pathological slide images respectively, and generate corresponding tissue masks. Step 3: Under the constraint of the tissue mask, feature point extraction and feature descriptor matching are performed only on the low-resolution registration layer images of H&E stained pathological slide images and IHC immunohistochemical pathological slide images within the tissue region, and the spatial transformation matrix under low resolution is solved. Step 4: Based on the scaling ratio corresponding to the downsampling level, scale the spatial transformation matrix at low resolution to obtain the spatial transformation matrix at the original resolution. Step 5: Based on the spatial transformation matrix at the original resolution, perform a spatial transformation on the original resolution image of the IHC immunohistochemical pathological slide image to spatially align it with the original resolution image of the H&E stained pathological slide image. Step 6: Based on the original resolution image of the aligned H&E stained pathological slide, perform standardized segmentation, and extract the corresponding segments in the aligned image of the IHC immunohistochemical pathological slide according to the inverse matrix of the spatial transformation matrix under the original resolution, and output the standardized image blocks in pairs.

3. The cross-modal image registration and alignment method for pathological sections according to claim 2, characterized in that, The specific process of step 1 is as follows: Let the H&E stained pathological section image be H ori IHC immunohistochemical pathological section images are I ori The image is processed using bicubic interpolation with a pyramid structure and multi-level downsampling. Pyramid level 0 represents the original highest resolution, and the resolution decreases with each subsequent level. The image size at each level is half the original size. level Where level = 0, 1, 2, ..., 9; select level ∈ (3, 6) as the low-resolution registration layer, and obtain the low-resolution registration layer image H. level and I level .

4. The cross-modal image registration and alignment method for pathological sections according to claim 2, characterized in that, The specific process of step 2 is as follows: The low-resolution registration layer image is converted to grayscale, transforming the three-channel color image into a single-channel grayscale image. Subsequently, the Otsu algorithm was used to calculate the optimal segmentation threshold. The grayscale image was then segmented by thresholding. Pixels with grayscale values ​​less than the threshold were identified as tissue regions, and pixels with grayscale values ​​greater than or equal to the threshold were identified as blank background regions, thus generating an initial tissue mask. Finally, morphological closing operations are performed on the initial tissue mask using structuring elements to fill internal pores, remove isolated noise points, and smooth tissue edges, generating the final tissue mask.

5. The cross-modal image registration and alignment method for pathological sections according to claim 2, characterized in that, The specific process of step 3 is as follows: Under the constraint of the tissue mask, feature points and binary feature descriptors are extracted only from the H&E low-resolution image and the IHC low-resolution image within the tissue region, respectively. The BFMatcher matching algorithm, using Hamming distance as the metric, is employed to perform cross-validation matching on the two sets of extracted feature descriptors. A smaller Hamming distance indicates a higher similarity between the feature descriptors. The formula for calculating the Hamming distance is shown below: ； In the formula, These are the binary feature descriptors of tissue regions for the H&E reference image and the IHC image to be registered, respectively. , The first two descriptors are respectively the first two descriptors. i Bit binary value, The Hamming distance; High-quality matching points are selected based on a predetermined ratio according to matching distance. The RANSAC algorithm is used to solve and optimize the homography matrix M at low resolution. low .

6. The cross-modal image registration and alignment method for pathological sections according to claim 2, characterized in that, Step 4 specifically involves obtaining the scaling ratio corresponding to the downsampling level in step 1, as shown in the following expression: ； In the formula, D is the downsampling scaling factor; Low-resolution registration matrix based on the scaling ratio corresponding to the downsampling level Scale normalization calculations are performed to obtain a homography matrix that can be used at the original resolution. The calculation expression is shown below: 。 7. The cross-modal image registration and alignment method for pathological sections according to claim 2, characterized in that, The specific process of step 5 is as follows: Establish the original resolution pixel coordinate mapping relationship, let (x ori ,y ori (x') represents the pixel coordinates of the IHC raw resolution image. ori ,y' ori The coordinates are the pixel coordinates mapped to the original H&E resolution image after registration. The mapping relationship is expressed as follows: ； Based on the above mapping relationship, using the original resolution image of H&E as a reference, a perspective transformation is performed on the original resolution image of IHC to complete global spatial alignment, resulting in the aligned IHC image. align .

8. The cross-modal image registration and alignment method for pathological sections according to claim 2, characterized in that, The specific process of step 6 is as follows: H&E original resolution image H ori Based on the standard, according to the fixed size patch size Traverse the blocks and determine the starting coordinates (x0, y0) and actual width w and height h of each block; The inverse matrix of the original resolution homography matrix is ​​calculated using the following expression: ； The coordinates of the four corner points of the H&E slice are extracted based on the starting coordinates of each slice and its actual width and height. The expression is as follows: ； The corner coordinates are mapped to the aligned IHC image using an inverse matrix. align Obtain the corresponding IHC cut block corner coordinates Extract the corresponding IHC slices based on the mapped coordinates and perform perspective correction. Save the H&E slices and IHC slices in pairs according to a unified naming rule. Output a standardized pairwise dataset that can be directly used for model training.

Images