Wax block and pathological section image-based section consistency discrimination method and system

Abstract

The invention discloses a slice consistency discrimination method and system based on a wax block and a pathological slice image, and relates to the technical field of tumor pathological diagnosis. The method comprises the following steps: receiving a to-be-compared wax block image and a to-be-compared pathological section image acquired by a color camera, and using a deep learning model YOLOv11 to realize positioning of a section image guided based on the tissue size in the wax block image and a tissue area in the wax block image; constructing an image alignment module and performing training to realize alignment of the wax block image and the slice image; and inputting the aligned slice and wax block images into the trained twin-tower twin neural network for consistency discrimination. According to the method, the wax block and the section image are automatically compared and analyzed through the deep learning model, technicians are assisted to quickly complete the quality control work of the pathological section, the mode of only depending on manual sampling inspection in the past is changed, the quality control efficiency is improved, the deep learning feature extraction method is adopted to carry out image alignment and judgment, and the detection accuracy is improved. And the accuracy of judging the consistency of the wax block and the slice is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of tumor pathological diagnosis, in particular to a slice consistency discrimination method and system based on wax block and pathological section images. BACKGROUND

[0002] Pathological diagnosis is the "gold standard" for the diagnosis of most diseases, especially tumor diseases. In the standard pathological workflow, formalin-fixed tissue samples need to be dehydrated, transparentized, and embedded in paraffin to make wax blocks, and then 4-6 μm thin sections are cut by a microtome and attached to glass slides, and after hematoxylin-eosin staining, pathological sections are formed. This process involves multiple technician operation steps, and the quality control of pathological sections is a key link and basis for clinical pathological quality control. At present, hospital pathology departments use manual sampling inspection methods to monitor the quality of pathological sections based on careful checking of each wax block and section. This method is time-consuming and labor-intensive, prone to errors, and difficult to standardize. Therefore, laboratory automation of pathological quality control is of great significance because it improves efficiency and accuracy by simplifying repetitive tasks, reducing human error, and ensuring consistency of results.

[0003] So far, the machine intelligent discrimination method of pathological section quality at home and abroad only discriminates through the stained pathological section alone. However, some quality problems of the section can only be determined by comparing the tissue in the wax block, such as the consistency of the tissue and the section, the integrity of the tissue section, and whether the section is contaminated. Ensuring the consistency of pathological wax blocks and sections is a very important quality control content, which can prevent label errors of sections and is the basis for other section quality control contents. The confusion of sections may have serious ethical, legal and diagnostic consequences.

[0004] Although the tissue images in pathological wax blocks and pathological sections are theoretically similar, differences such as rotation, inversion, deformation, and color change may occur during the sectioning process, which poses a great challenge to traditional machine vision. SUMMARY

[0005] The purpose of the present application is to provide a slice consistency discrimination method and system based on wax block and pathological section images, which can assist technicians to improve the quality control efficiency of pathological sections in an intelligent screening manner and reduce the misjudgment caused by human factors such as inattention when manually comparing wax block images and pathological section images.

[0006] According to the first aspect of the present application, in order to achieve the above-mentioned purpose, the present application provides the following technical scheme: a slice consistency discrimination method based on wax block and pathological section images, comprising the following steps:

[0007] The system receives images of paraffin blocks and pathological slides to be compared from a color camera, and uses the deep learning model YOLOv11 to locate tissue regions in the slides and paraffin blocks based on the tissue size within the paraffin block images.

[0008] An image alignment module is constructed, which uses a pre-trained convolutional model to extract high-dimensional features from wax block images and slice images. The wax block images and slice images are divided into sub-image patches at the feature layer. Through a trained feature mapping module, the matched wax block sub-image and slice sub-image regions have a high degree of cosine similarity. By comparing the cosine similarity between wax block sub-image patches and slice sub-image patches one by one, the corresponding positional relationship between sub-images with high similarity is preserved. The deformation matrix is ​​calculated to achieve the alignment of wax block images and slice images.

[0009] The image alignment module takes as input the slice image and the wax block image after the tissue region is located, and outputs as an aligned pair of slice and wax block images.

[0010] A dual-tower twin neural network is constructed and trained. Aligned slices and wax block images are input into the trained dual-tower twin neural network for consistency discrimination. The consistency discrimination result is output through the contrastive learning framework of the dual-tower twin neural network.

[0011] Furthermore, the YOLOv11 deep learning model is used to locate tissue regions in the wax block image based on the tissue size of the wax block image, as follows:

[0012] (21) Train the wax block image detection model, detect the tissue region in the wax block image through the trained wax block image detection model, and calculate the area of ​​the outer bounding box of the wax block tissue as the reference area.

[0013] (22) Train the slice image detection model. There are 1-6 repeated tissue regions in the slice image to be detected. The slice image detection model detects the tissue regions and records the confidence and corresponding area of ​​each detection box.

[0014] (23) Based on the tissue reference area obtained from the wax block image, set a size threshold range of 80-120% and screen candidate tissue regions in the slice image whose area meets this threshold range.

[0015] (24) In the candidate tissue regions that have passed the size screening, select the detection result with the highest confidence as the final detection output of the slice image;

[0016] (25) Output the final wax block detection results and slice detection results.

[0017] Furthermore, an image alignment method based on the feature similarity of image sub-images is designed. The wax block image and the slice image are divided into sub-image patches. By comparing the similarity between the wax block sub-image patches and the slice sub-image patches one by one, the corresponding positional relationship between sub-images with high similarity is preserved. The deformation matrix is ​​calculated to achieve the alignment of the wax block image and the slice image, as detailed below:

[0018] (31) Using the threshold method, the slice image and the wax block image after the tissue region is located are preprocessed to remove the background region and scale the image to the same pixel precision, and the image size is unified to [224,224].

[0019] (32) The wideresnet50 model pre-trained on ImageNet is used as the image feature extractor. Multi-scale image features of slice tissue and wax block tissue are extracted from the input image by the feature extractor. The features from the first to the fourth layer of the multi-scale are denoted as f1, f2, f3 and f4 respectively, with sizes of [1,256,56,56], [1,512,28,28], [1,1024,14,14] and [1,2048,7,7] respectively.

[0020] (33) The features of the second layer f2 and the third layer f3 are fused on the feature channel. The fusion method is to first upsample the f3 feature by 2 times to the same image height and width as f2, that is, upsample from [1,1024,14,14] to the size [1,1024,28,28]. Then, f2 and f3 are concatenated into one feature, and the concatenated feature is denoted as f. concat The dimensions after splicing are [1,1536,28,28];

[0021] (34) Crop the image sub-plots of the stitched feature map, and use a 3×3 sliding window to crop the stitched feature map f. concat Perform feature extraction, extracting 3×3 features each time. After extraction, shift one position to the right or down before extracting the next sub-image. Treat 3×3 as an image sub-image with sub-image size [1,1536,3,3].

[0022] (35) After feature extraction through a 3×3 sliding window, the new feature size is [1,1536,3,3,28,28], and the reduced feature dimension is [1,1536,3,3,784]. Let the new feature name be f. flatten , meaning the flattened feature, where [3,3] is the size of the image sub-patch, and is the high-dimensional feature extracted from the [24,24] pixel region of the original image. There are a total of 784 = 28 × 28 such sub-patches. The feature at the i-th position in 784 is denoted as f. flatten_i Then the positional correspondence of the i-th feature in the original image is: fflatten_i =f concat (x,y)

[0023] i=28(y-1)*x+x,x∈[1,28],y∈[1,28]

[0024] Where f flatten_i Indicates in f flatten The i-th feature out of 784 features, f concat (x,y) represents f concat The feature in the x-th row and y-th column of a 28×28 feature set;

[0025] (36) Using a pre-trained slice and wax block feature mapping module, the slice sub-plot features f flatten_x and wax block sub-block features f ' flatten_y After mapping to the same feature space, feature comparison is performed, and cosine similarity is used to calculate the similarity between the slice sub-image and the wax block sub-image, as follows:

[0026] (36.1) The slice sub-image features and wax block sub-image features are respectively processed by a pre-trained feature extraction convolutional module, and the slice is W. q Convolution module, wax block is W l The convolution module extracts similar features from the slice and wax block image modalities and aligns them to a common feature space.

[0027] (36.2) Calculate the similarity between the wax block sub-pattern features and the slice sub-pattern features that are already in the same feature space; use a double loop to calculate the feature W at each position of the slice. q ×f flatten_x W at each position of the wax block l ×f ' flatten_y The similarity is calculated using cosine similarity, with similarity values ​​ranging from -1 to 1, where -1 represents the least similar and 1 represents the most similar.

[0028] (36.3) The formula for calculating cosine similarity is:

[0029]

[0030] Among them (W) q ×f flatten_x )×(W l ×f ' flatten_y ) represents calculating the dot product of two vectors, ||W q ×f flatten_x ||×||W l ×f ' flatten_y|| represents taking the modulus of each of the two vectors and then multiplying them;

[0031] (37) Based on the positional relationship of highly similar sub-images, the deformation matrix is ​​calculated using the Homography algorithm to align the wax block and slice image, as follows:

[0032] (37.1) Key matching pair generation: Based on cosine similarity, the cosine similarity threshold is set to 0.8 to retain high similarity sub-image pairs. For each sub-image pair, the coordinates of its center point are taken as the matching point. Outlier filtering: The RANSAC algorithm is used to pre-filter erroneous matches to improve the robustness of Homography solution. Background region filtering: Any matching block in the background of the wax block and slice sub-image is filtered out to prevent the similarity of the background from affecting the matching effect.

[0033] (37.2) Homography algorithm deformation matrix calculation: Solve the 3×3 deformation matrix H using the findHomography() function of OpenCV, and select RANSAC as the robust estimation algorithm.

[0034] (37.3) Image deformation and interpolation: deform the slice image, call warpPerspective() to apply the H matrix to the slice image, and use bilinear interpolation to keep the image smooth, so as to obtain a slice image aligned with the wax block image.

[0035] Furthermore, the feature mapping module between the sliced ​​image and the wax block image sub-image is trained as follows:

[0036] (41) 300 pairs of wax block and slice images were selected as the dataset and the dataset was divided into training set and validation set. The 300 pairs of wax block and slice images were preprocessed before training by manual alignment. The tissue angle and tissue position were manually adjusted to achieve complete alignment of tissue in the image.

[0037] (42) The slice and wax block feature comparison model learns the image feature relationship between the matching regions of slice images and wax block images through model construction and feature training, and represents the matching relationship through cosine similarity.

[0038] (42.1) During model training, the wax block and slice image pairs in the training set are first divided into training and testing parts by the image feature matching model. During training, slice and wax block image pairs that are known to be from the same tissue are used, and the feature extraction module obtains the flattened feature set of the sub-image region, f. flatten and wax block sub-plot features f' flattenEach sub-image feature set has dimensions [1, 1536, 3, 3, 784]. The first dimension represents the number of images used simultaneously during each training session. The second dimension, 1536, represents the total number of features extracted for each sub-image region. The third and fourth dimensions, [3, 3], represent the size of each sub-image region in the high-dimensional space. The fifth dimension, 784, represents the total number of sub-image regions the image is divided into.

[0039] (42.2) During model training, the similarity label for subgraphs at the same position is set to 1, and the similarity label for subgraphs at different positions is set to -1; the model training strategy is to take the subgraphs at the same position once, calculate the similarity, and then update the convolution kernel W. q and W l Then, take subgraphs at different positions, calculate the similarity again, and then update the convolution kernel W. q and W l The model iterates by comparing the subgraph positions once for each round, and then taking the subgraph position for the next round.

[0040] (42.3) The loss function uses cosine similarity as the model's loss function. Based on the training and validation sets, the image alignment module is learned and trained. The image alignment module updates the model weights by minimizing the loss function until the loss function no longer decreases or tends to a stable value, thus obtaining the trained slice and wax block feature comparison model; the loss function is as follows:

[0041] L consim =1-Cos_Similarity(f q ,f l )

[0042] Where f q For the features of a slice patch, f l Features of the wax block patch.

[0043] Furthermore, the dual-tower Siamese neural network employs a contrastive learning framework, which uses a parameter-sharing dual-tower structure, with each tower consisting of three parts:

[0044] (51) Backbone network: The Transformer model with the Swing-Tiny architecture is used as the backbone network, and the weights are initialized using weights pre-trained on the ImageNet dataset;

[0045] (52) Feature processing module: The features extracted by the backbone network are L2 normalized, and then the features are mapped to a unified dimension through two layers of MLP projection heads;

[0046] (53) Loss function module: The model training is optimized using a contrastive loss function, specifically:

[0047]

[0048] Among them, D w =D w (F lakuai ,F qiepian )=||F lakuai -F qiepian ||2 represents the Euclidean distance between two samples, and Y is the label indicating whether the two samples match. Y=1 means that the two samples match, and Y=0 means that the two samples do not match.

[0049] Furthermore, the training process of the dual-tower twin neural network is as follows:

[0050] (61) Adjust the wax block image and slice image to a resolution of 224×224 and perform scale normalization using the mean and standard deviation of [0.5,0.5,0.5].

[0051] (62) Based on the processed wax block image and slice image, construct positive and negative sample pairs in a 1:1 ratio. The positive sample pair consists of the wax block image and its corresponding slice image. The negative sample pairs are automatically generated based on the data. Among the negative samples, 80% of the wax block and slice sample pairs have similar sizes, and 20% of the sample pairs have random sizes. The positive and negative sample pairs are divided into training set, validation set and test set in an 8:1:1 ratio.

[0052] (63) The dual-tower twin neural network was trained. During the training process, the AdamW optimizer was used, the initial learning rate was set to 1e-5, the cosine learning rate decay strategy was adopted, the batch size was set to 32, and a total of 20 epochs were trained.

[0053] (64) The model weights are updated by minimizing the loss function until the loss function no longer decreases or tends to a stable value, and the trained twin neural network is obtained.

[0054] Furthermore, the backbone network in the dual-tower twin neural network can also adopt ResNet-50, EfficientNet-B4, DenseNet-201, ConvNeXt-Tiny, or DINO-ViT architectures.

[0055] According to a second aspect of the present invention, the present invention provides a slide consistency discrimination system based on paraffin block and pathological slide images, for implementing the slide consistency discrimination method based on paraffin block and pathological slide images described in the first aspect, comprising:

[0056] The positioning unit is used to receive the paraffin block image and pathological slide image to be compared acquired by the color camera, and uses the deep learning model YOLOv11 to realize the localization of the tissue region in the slide image and the paraffin block image based on the tissue size of the paraffin block image.

[0057] The alignment unit is used to construct the image alignment module. It uses a pre-trained convolutional model to extract high-dimensional features from the wax block image and the slice image. The feature layer divides the wax block image and the slice image into sub-image patches. Through the trained feature mapping module, the matched wax block sub-image and slice sub-image regions have a high degree of cosine similarity. By comparing the cosine similarity between the wax block sub-image patch and the slice sub-image patch one by one, the corresponding positional relationship between the sub-images with high similarity is preserved. The deformation matrix is ​​calculated to achieve the alignment of the wax block image and the slice image.

[0058] The discriminant output unit is constructed and trained using a dual-tower twin neural network. The aligned slices and wax block images are input into the trained dual-tower twin neural network for consistency discrimination, and the consistency discrimination result is output through the contrastive learning framework of the dual-tower twin neural network.

[0059] This invention has at least the following beneficial effects:

[0060] 1. This invention uses a deep learning model to automatically compare and analyze paraffin block and slide images, assisting technicians in quickly completing the quality control of pathological slides. This changes the previous method of relying solely on manual sampling inspection, greatly improving quality control efficiency and enabling the processing of more slide samples in a shorter time, thus meeting the growing pathological diagnostic needs of hospitals.

[0061] 2. This invention addresses the differences between wax block and slice images, such as rotation, flipping, deformation, and color changes, as well as the different grayscale distributions and texture patterns of multimodal images. It employs a deep learning feature extraction method for image alignment and discrimination. This method can deeply mine the intrinsic features of the image and has better performance when processing multimodal images compared to traditional feature point detection and matching algorithms. It effectively improves the accuracy of wax block and slice consistency discrimination and reduces misjudgments caused by image differences.

[0062] 3. The image alignment method based on local regions proposed in this invention divides wax blocks and slices into sub-blocks, compares the similarity between sub-blocks one by one, and retains the positional relationship of sub-blocks with high similarity to construct alignment relationships. This local alignment strategy avoids the difficulties in global alignment caused by slice quality problems (such as tissue loss, folding, impurities, etc.), improves the accuracy of image alignment, provides a more reliable basis for subsequent consistency discrimination, and thus improves the discrimination accuracy of the entire system.

[0063] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0064] Figure 1 This is a flowchart illustrating the discrimination method described in this invention;

[0065] Figure 2 This is a three-dimensional schematic diagram of the structure of the present invention; where a is the acquired wax block and slice image; b is the tissue region of the wax block image detected by YOLOv11; c is the tissue region of the slice image detected by YOLOv11.

[0066] Figure 3 This is a schematic diagram of the structural framework of the image alignment module of the present invention;

[0067] Figure 4 This is a schematic diagram of the framework of the dual-tower twin neural network of the present invention;

[0068] Figure 5 This is a schematic diagram of the ROC curves of deep learning models using different backbones in the dual-tower twin neural network of this invention. Detailed Implementation

[0069] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0070] Example 1:

[0071] Please see Figures 1-5 This invention provides a technical solution: a method for determining the consistency of paraffin blocks and pathological slide images, comprising the following steps:

[0072] S1. Receive the paraffin block image and pathological slide image to be compared from a color camera, use the YOLOv11 deep learning model to detect tissue regions within the paraffin block image, and locate tissue regions in the slide image based on the tissue size of the paraffin block image, as follows:

[0073] (S11) Train the wax block image detection model, detect the tissue region in the wax block image through the trained wax block image detection model, and calculate the area of ​​the outer bounding box of the wax block tissue as the reference area.

[0074] (S12) Train the slice image detection model. There are 1-6 repeated tissue regions in the slice image to be detected. The slice image detection model detects the tissue regions and records the confidence and corresponding area of ​​each detection box.

[0075] (S13) Based on the tissue reference area obtained from the wax block image, set a size threshold range of 80-120% and screen candidate tissue regions in the slice image whose area meets this threshold range.

[0076] (S14) In the candidate tissue regions that have passed the size screening, select the detection result with the highest confidence as the final detection output of the slice image;

[0077] (S15) Output the final wax block detection results and slice detection results;

[0078] S2. Construct an image alignment module. Use a pre-trained convolutional model to extract high-dimensional features from the wax block image and the slice image. In the feature layer, the wax block image and the slice image are divided into sub-image patches. Through the trained feature mapping module, the matched wax block sub-image and slice sub-image regions have highly similar cosine similarity. By comparing the cosine similarity between the wax block sub-image patch and the slice sub-image patch one by one, the corresponding positional relationship between the sub-images with high similarity is preserved. The deformation matrix is ​​calculated to achieve the alignment of the wax block image and the slice image.

[0079] It should be noted that during the slide preparation process, technicians manually retrieve the specimen slides suspended in water using glass slides. Therefore, the orientation of the tissue on the slides is uncontrolled. In order to reduce the interference of image rotation angle on the subsequent intelligent identification system for the consistency between paraffin blocks and slides, it is necessary to use an algorithm to automatically align the pathological paraffin block image and the slide image in the preceding steps.

[0080] Limitations of existing technologies: Existing image alignment schemes fall into two categories. One category uses feature point detection and matching algorithms such as SIFT, SURF, and ORB to find similar key point pairs between images and estimates the optimal geometric transformation parameters using algorithms such as RANSAC to achieve image alignment. This is commonly used in scenarios such as multi-view panoramic image stitching and 3D reconstruction. However, this method requires the images to be taken in the same modality because feature point detection methods such as SIFT, SURF, and ORB rely on image gradients or local texture information (such as FAST corner points and DoG spots). The grayscale distribution and texture patterns of multimodal images may be completely different. For example, the fat region in a wax block is imaged as a smooth, light yellow region with the same color as the surrounding tissue region; while in a section, because the fat region is difficult to slide, it appears as an tissueless region, making the tissue region adjacent to the fat exhibit clear tissue texture information. In addition, due to problems such as folding and impurities caused by poor slide preparation, it exhibits texture information inconsistent with the wax block.

[0081] Another existing image alignment scheme uses deep learning feature extraction methods. Deep learning methods excel at extracting and aligning features of multimodal images at the feature layer. Existing schemes typically assume a global correspondence between two images, such as the registration of MRI and CT images. However, in the registration of paraffin blocks and slides, defects in the slide images, such as missing tissue portions, may occur, making it impossible to consider the alignment relationship of the tissue from a global perspective.

[0082] To address the aforementioned issues, this embodiment uses the positions of sub-images (patches) with high local similarity between the wax block and the slice to construct alignment relationships, thereby reducing alignment interference caused by slice quality issues in the algorithm. Specifically, both the wax block and the slice are divided into sub-image patches. Then, by comparing the similarity between the wax block and the slice sub-image patches one by one, the corresponding positional relationships between sub-images with high similarity are preserved. Finally, by calculating the deformation matrix, the alignment of the wax block image and the slice image is achieved.

[0083] like Figure 3 The image shown is a patch-based image alignment model framework. The specific alignment method is as follows:

[0084] (S21) Both the slice and the wax block image are scaled to 224×224 pixels, and the slice image and the wax block image after the tissue region is located are respectively divided into sub-image blocks;

[0085] (S22) First, the feature extractor is used to extract the image features of the slice image and the wax block image respectively. The feature extractor is a WideResNet50 model pre-trained on ImageNet. The features extracted by the feature extractor are denoted as f1, f2, f3, and f4, with dimensions of [1,256,56,56] and [1,512,28,28] respectively.

[0086] [1,1024,14,14], [1,2048,7,7], the first digit of the size represents the number of images trained in parallel in each round of model training, which can usually be 1, 8, 16, 32, etc. For the sake of explaining the principle, this embodiment will use 1 to describe the size change of the feature.

[0087] (S23) Since the features extracted by f1 are relatively shallow (first layer) and focus on texture information, they are easily affected by slight color differences and irrelevant slight textures on the surface of slices or wax blocks. This is not the representative feature required in this embodiment. Since f4 extracts the deepest features, which are abstract high-dimensional information, they are easily affected by the pre-trained ImageNet dataset and extract irrelevant information. This is also not the representative feature required in this embodiment. Therefore, this embodiment only uses the f2 and f3 features of the second and third layers of the feature extractor for subsequent image sub-image comparison.

[0088] The features from the second and third layers are fused along the feature channels. First, the f3 feature is upsampled by 2 times to the same image height and width as f2, i.e., upsampled from [1, 1024, 14, 14].

[0089] Given a value of [1, 1024, 28, 28], f2 and f3 are concatenated into a single feature, denoted as f. concat The dimensions are [1, 1536, 28, 28]. At this point, each feature in the range [1, 1536, 1, 1] contains information about a 16x16 area of ​​the original image. To increase the information content of each sub-image, a 3x3 sliding window is used to adjust the f... concat Feature extraction is performed, extracting 3×3 features each time. After extraction, shift one position to the right or down before extracting the next sub-image. This treats 3×3 as an image sub-image with a size of [1,1536,3,3]. The sub-image contains information about the image region of the original image with a width of 48×height of 48. This allows the sub-image to simultaneously contain global image information and sub-image detail information, making it easier to compare the similarity between slice and wax block sub-images while taking into account both global shape information and detailed feature information.

[0090] After feature extraction using a 3×3 sliding window, the new feature size is...

[0091] [1,1536,3,3,28,28], reduce the feature dimension to [1,1536,3,3,784], and denote the new feature name as f. flatten , meaning the flattened feature, where [3,3] is the size of the image sub-patch, which is the high-dimensional feature extracted from the [24,24] pixel region of the original image. There are a total of 784 = 28 × 28 such sub-patches, and the feature at the i-th position in 784 is denoted as , with length f. flatten_i Then the positional relationship of the i-th feature in the original image is as follows:

[0092] f flatten_i =f concat (x,y)i=28(y-1)*x+x,x∈[1,28],y∈[1,28]

[0093] Where f flatten_i Indicates in f flatten The i-th feature out of 784 features, f concat (x,y) represents f concat The feature in the x-th row and y-th column of (28×28) features;

[0094] (S24) A pre-trained slice and wax block feature mapping module is used to map the features f of the slice sub-images. flatten and wax block sub-block features f' flatten Feature comparison is performed, and the cosine similarity between the wax block sub-image and the slice sub-image is calculated, as follows:

[0095] (S24.1) The slice sub-image features and wax block sub-image features are respectively processed by a pre-trained feature extraction convolutional module, and the slice is W. q Convolution module, wax block is W l The convolution module extracts similar features from the slice and wax block image modalities and aligns them to a common feature space.

[0096] (S24.2) Calculate the similarity between the wax block sub-pattern features and the slice sub-pattern features that are already in the same feature space; use a double loop to calculate the feature W at each position of the slice. q ×f flatten_x W at each position of the wax block l ×f ' flatten_y The similarity is calculated using cosine similarity, with similarity values ​​ranging from -1 to 1, where -1 represents the least similar and 1 represents the most similar.

[0097] (S25) Based on the positional relationship of highly similar sub-images, the deformation matrix is ​​calculated using the Homography algorithm to align the wax block and slice images, as detailed below:

[0098] (S25.1) Key matching pair generation: Based on cosine similarity, the pair is filtered and a cosine similarity threshold of 0.8 is set to retain high-similarity sub-image pairs. For each sub-image pair, the coordinates of its center point are taken as the matching point. Outlier filtering: The RANSAC algorithm is used to pre-filter erroneous matches to improve the robustness of Homography solution. Background region filtering: Any matching block in the background of the wax block and slice sub-image is filtered out to prevent the similarity of the background from affecting the matching effect.

[0099] (S25.2) Homography algorithm deformation matrix calculation: Solve for the 3×3 deformation matrix H using the findHomography() function of OpenCV, and select RANSAC as the robust estimation algorithm.

[0100] (S25.3) Image deformation and interpolation: deform the slice image, call warpPerspective() to apply the H matrix to the slice image, and use bilinear interpolation to keep the image smooth, to obtain a slice image aligned with the wax block image;

[0101] Regarding the technical solution of this embodiment, the training process of the image alignment module is as follows:

[0102] We selected 300 pairs of fully aligned wax block and slice images as the dataset, and divided the dataset into a training set and a validation set. All 300 pairs of wax block and slice images were manually aligned.

[0103] The image alignment module is learned and trained based on the training set and validation set. The image alignment module updates the model weights by minimizing the loss function until the loss function no longer decreases or tends to a stable value, and then the trained image alignment module is obtained.

[0104] The image alignment module uses cosine similarity as its loss function, as detailed below:

[0105]

[0106] Among them (W) q ×f flatten_x )×(W l ×f ' flatten_y ) represents calculating the dot product of two vectors, ||W q ×f flatten_x ||×||W l ×f ' flatten_y || represents taking the modulus of each of the two vectors and then multiplying them;

[0107] Using cosine similarity as the loss function optimizes feature extraction, making the two feature extraction branches focus on the similarity information between the wax block image and the slice image;

[0108] During model testing, the model branches for wax block images and slice images are reasoned separately. Then, the cosine similarity between each wax block patch and slice patch is calculated. The calculation results are fed into the Homography method to calculate the deformation matrix. Then, the slice image is aligned to the wax block image based on the deformation matrix.

[0109] Regarding the technical solution of this embodiment, the training process of the information mapping model between the slice image and the wax block image sub-image is as follows:

[0110] The feature mapping module between the sliced ​​image and the wax block image sub-image is trained as follows:

[0111] (1) 300 pairs of wax block and slice images were selected as the dataset and the dataset was divided into training set and validation set. The 300 pairs of wax block and slice images were preprocessed before training by manual alignment. The tissue angle and tissue position were manually adjusted to achieve complete alignment of tissue in the image.

[0112] (2) The slice and wax block feature comparison model learns the image feature relationship between the matching regions of slice images and wax block images through model construction and feature training, and represents the matching relationship through cosine similarity.

[0113] (2.1) During model training, the wax block and slice image pairs in the training set are first divided into training and testing parts by the image feature matching model. During training, slice and wax block image pairs that are known to be from the same tissue are used, and the feature extraction module obtains the flattened feature set of the sub-image region, f. flatten and wax block sub-plot features f' flatten Each sub-image feature set has dimensions [1, 1536, 3, 3, 784]. The first dimension represents the number of images used simultaneously during each training session. The second dimension, 1536, represents the total number of features extracted for each sub-image region. The third and fourth dimensions, [3, 3], represent the size of each sub-image region in the high-dimensional space. The fifth dimension, 784, represents the total number of sub-image regions the image is divided into.

[0114] (2.2) During model training, the similarity label for subgraphs at the same position is set to 1, and the similarity label for subgraphs at different positions is set to -1; the model training strategy is to take the subgraphs at the same position once, calculate the similarity, and then update the convolution kernel W. q and W l Then, take subgraphs at different positions, calculate the similarity again, and then update the convolution kernel W. q and W l The model iterates by comparing the subgraph positions once for each round, and then taking the subgraph position for the next round.

[0115] (2.3) The loss function uses cosine similarity as the model's loss function. Based on the training and validation sets, the image alignment module is trained and learned. The image alignment module updates the model weights by minimizing the loss function until the loss function no longer decreases or tends to a stable value, thus obtaining the trained slice and wax block feature comparison model. The specific loss function is as follows:

[0116] L consim =1-Cos_Similarity(f q ,f l )

[0117] Where f q For the features of a slice patch, f l Features of wax block patches;

[0118] It should be further explained that the input of the image alignment module is the slice image and the wax block image after the tissue region is located, and the output is the aligned slice and wax block image pair;

[0119] S3. Construct and train a dual-tower Siamese neural network. Input the aligned slices and wax block images into the trained dual-tower Siamese neural network for consistency judgment. Then, output the consistency judgment result of the two through the contrastive learning framework of the dual-tower Siamese neural network, as follows:

[0120] Image enhancement and preprocessing:

[0121] The pathological paraffin block images are scaled uniformly to achieve scale normalization between the paraffin block and the slide images. In this embodiment, positive and negative sample pairs are constructed at a 1:1 ratio. The positive sample pair consists of the paraffin block image and its corresponding slide image. The negative sample pair is automatically generated based on the data. In order to be closer to the actual clinical situation, the negative sample pair consists of 80% negative slide samples with similar size to the paraffin block and 20% random negative slide samples. The positive and negative sample pairs are divided into training set, validation set and test set in an 8:1:1 ratio.

[0122] Deep model construction and training:

[0123] like Figure 2 As shown, this embodiment proposes a contrastive learning framework based on a dual-tower Siamese neural network. This framework adopts a parameter-sharing dual-tower structure, with each tower consisting of three parts:

[0124] (1) Backbone network: Four CNN architectures, including ResNet-50, EfficientNet-B4, DenseNet-201, and ConvNeXt-Tiny, as well as two Transformer architectures, Swin-Tiny and DINO-ViT, were tested. These networks have similar number of parameters and are all initialized with weights pre-trained on the ImageNet dataset.

[0125] It should be noted that the backbone network of this embodiment can also be replaced by ResNet-50, EfficientNet-B4, DenseNet-201, ConvNeXt-Tiny or DINO-ViT architectures. This embodiment does not make specific limitations here, and can be selected according to the actual situation.

[0126] (2) Feature processing module: First, the features extracted by the backbone network are L2 normalized, and then the features are mapped to a unified dimension through two layers of MLP projection heads;

[0127] (3) Loss function module: The model training is optimized by using the contrastive loss function.

[0128] During training, all models used the AdamW optimizer with initial learning rates of 1e-4 (CNN class) and 1e-5 (Transformer class), adopted a cosine learning rate decay strategy, and set the batch size to 32. The training lasted for 20 epochs.

[0129] The input images were uniformly adjusted to a resolution of 224×224, and corresponding normalization was applied for different architectures: the CNN model was normalized using ImageNet statistics, and the Transformer model was normalized using the mean and standard deviation of [0.5, 0.5, 0.5].

[0130] Table 1 Performance comparison of various combined deep learning models on the test set

[0131]

[0132] Figure 5 The image shows the ROC curves of various deep learning models on different backbones in the test set, such as... Figure 5 As shown in Table 1, we tested the performance of the model under different backbone networks on the collected dataset. The results showed that the model using the Swin-Tiny+Contrastive combination performed the best, with an accuracy of 87.5%.

[0133] In summary, this invention addresses the differences between wax block and slice images, such as rotation, flipping, deformation, and color changes, as well as the different grayscale distributions and texture patterns of multimodal images. It employs a deep learning feature extraction method for image alignment and discrimination. This method can deeply mine the intrinsic features of images and, compared to traditional feature point detection and matching algorithms, performs better when processing multimodal images, effectively improving the accuracy of wax block and slice consistency discrimination and reducing misjudgments caused by image differences. The image alignment method proposed in this invention, based on local image regions, divides the wax block and slice into sub-blocks, compares the similarity between sub-blocks one by one, and preserves the positional relationships of sub-blocks with high similarity to construct alignment relationships. This local alignment strategy avoids global alignment difficulties caused by slice quality issues (such as tissue loss, folding, impurities, etc.), improves the accuracy of image alignment, provides a more reliable foundation for subsequent consistency discrimination, and thus enhances the overall system's discrimination accuracy.

[0134] Example 2:

[0135] This embodiment provides a slide consistency judgment system based on paraffin block and pathological slide images, used to implement the slide consistency judgment method based on paraffin block and pathological slide images described in Embodiment 1, including:

[0136] The positioning unit is used to receive the paraffin block image and pathological slide image to be compared acquired by the color camera, and uses the deep learning model YOLOv11 to realize the localization of the tissue region in the slide image and the paraffin block image based on the tissue size of the paraffin block image.

[0137] The alignment unit is used to construct the image alignment module. It uses a pre-trained convolutional model to extract high-dimensional features from the wax block image and the slice image. The feature layer divides the wax block image and the slice image into sub-image patches. Through the trained feature mapping module, the matched wax block sub-image and slice sub-image regions have a high degree of cosine similarity. By comparing the cosine similarity between the wax block sub-image patch and the slice sub-image patch one by one, the corresponding positional relationship between the sub-images with high similarity is preserved. The deformation matrix is ​​calculated to achieve the alignment of the wax block image and the slice image.

[0138] The discriminant output unit is constructed and trained using a dual-tower twin neural network. The aligned slices and wax block images are input into the trained dual-tower twin neural network for consistency discrimination, and the consistency discrimination result is output through the contrastive learning framework of the dual-tower twin neural network.

[0139] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0140] For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances. When an element is referred to as being "assembled on," "mounted on," "fixed to," or "set on" another element, it may be directly on the other element or there may be an intermediate element present. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element present. The terms "vertical," "horizontal," "upper," "lower," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible embodiments.

[0141] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0142] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

Claims

1. A method for determining the consistency of paraffin blocks and pathological slide images, characterized in that, Includes the following steps: The system receives images of paraffin blocks and pathological slides to be compared from a color camera, and uses the deep learning model YOLOv11 to locate tissue regions in the slides and paraffin blocks based on the tissue size within the paraffin block images. An image alignment module is constructed, which uses a pre-trained convolutional model to extract high-dimensional features from wax block images and slice images. The wax block images and slice images are divided into sub-image patches at the feature layer. Through a trained feature mapping module, the matched wax block sub-image and slice sub-image regions have a high degree of cosine similarity. By comparing the cosine similarity between wax block sub-image patches and slice sub-image patches one by one, the corresponding positional relationship between sub-images with high similarity is preserved. The deformation matrix is ​​calculated to achieve the alignment of wax block images and slice images. The image alignment module takes as input the slice image and the wax block image after the tissue region is located, and outputs as an aligned pair of slice and wax block images. A dual-tower twin neural network is constructed and trained. Aligned slices and wax block images are input into the trained dual-tower twin neural network for consistency discrimination. The consistency discrimination result is output through the contrastive learning framework of the dual-tower twin neural network.

2. The method for determining the consistency of paraffin blocks and pathological slide images according to claim 1, characterized in that, The YOLOv11 deep learning model is used to locate tissue regions in a wax block image based on the tissue size of the wax block image, as follows: (21) Train the wax block image detection model, detect the tissue region in the wax block image through the trained wax block image detection model, and calculate the area of ​​the outer bounding box of the wax block tissue as the reference area. (22) Train the slice image detection model. There are 1-6 repeated tissue regions in the slice image to be detected. The slice image detection model detects the tissue regions and records the confidence and corresponding area of ​​each detection box. (23) Based on the tissue reference area obtained from the wax block image, set a size threshold range of 80-120% and screen candidate tissue regions in the slice image whose area meets this threshold range. (24) In the candidate tissue regions that have passed the size screening, select the detection result with the highest confidence as the final detection output of the slice image; (25) Output the final wax block detection results and slice detection results.

3. The method for determining the consistency of paraffin blocks and pathological slide images according to claim 2, characterized in that, This paper designs an image alignment method based on the feature similarity of image sub-images. The wax block image and the slice image are divided into sub-image patches. By comparing the similarity between the wax block sub-image patches and the slice sub-image patches one by one, the corresponding positional relationships between sub-images with high similarity are preserved. The deformation matrix is ​​then calculated to achieve the alignment of the wax block image and the slice image. The specific details are as follows: (31) Using the threshold method, the slice image and the wax block image after the tissue region is located are preprocessed to remove the background region and scale the image to the same pixel precision, and the image size is unified to [224,224]. (32) The wideresnet50 model pre-trained on ImageNet is used as the image feature extractor. Multi-scale image features of slice tissue and wax block tissue are extracted from the input image by the feature extractor. The features from the first to the fourth layer of the multi-scale are denoted as f1, f2, f3 and f4 respectively, with sizes of [1,256,56,56], [1,512,28,28], [1,1024,14,14] and [1,2048,7,7] respectively. (33) The features of the second layer f2 and the third layer f3 are fused on the feature channel. The fusion method is to first upsample the f3 feature by 2 times to the same image height and width as f2, that is, upsample from [1,1024,14,14] to the size [1,1024,28,28]. Then, f2 and f3 are concatenated into one feature, and the concatenated feature is denoted as f. concat The dimensions after splicing are [1,1536,28,28]; (34) Crop the image sub-plots of the stitched feature map, and use a 3×3 sliding window to crop the stitched feature map f. concat Perform feature extraction, extracting 3×3 features each time. After extraction, shift one position to the right or down before extracting the next sub-image. Treat 3×3 as an image sub-image with sub-image size [1,1536,3,3]. (35) After feature extraction through a 3×3 sliding window, the new feature size is [1,1536,3,3,28,28], and the reduced feature dimension is [1,1536,3,3,784]. Let the new feature name be f. flatten , meaning the flattened feature, where [3,3] is the size of the image sub-patch, and is the high-dimensional feature extracted from the [24,24] pixel region of the original image. There are a total of 784 = 28 × 28 such sub-patches. The feature at the i-th position in 784 is denoted as f. flatten_i Then the positional relationship of the i-th feature in the original image is as follows: f flatten_i =f concat (x,y) i=28(y-1)*x+x,x∈[1,28],y∈[1,28] Where f flatten_i Indicates in f flatten The i-th feature out of 784 features, f concat (x,y) represents f concat The feature in the x-th row and y-th column of a 28×28 feature set; (36) Using a pre-trained slice and wax block feature mapping module, the slice sub-plot features f flatten_x and wax block sub-block features f ' flatten_y After mapping to the same feature space, feature comparison is performed, and cosine similarity is used to calculate the similarity between the slice sub-image and the wax block sub-image, as follows: (36.1) The slice sub-image features and wax block sub-image features are respectively processed by a pre-trained feature extraction convolutional module, and the slice is W. q Convolution module, wax block is W l The convolution module extracts similar features from the slice and wax block image modalities and aligns them to a common feature space. (36.2) Calculate the similarity between the wax block sub-pattern features and the slice sub-pattern features that are already in the same feature space; use a double loop to calculate the feature W at each position of the slice. q ×f flatten_x W at each position of the wax block l ×f ' flatten_y The similarity is calculated using cosine similarity, with similarity values ​​ranging from -1 to 1, where -1 represents the least similar and 1 represents the most similar. (36.3) The formula for calculating cosine similarity is: Among them (W) q ×f flatten_x )×(W l ×f ' flatten_y ) represents calculating the dot product of two vectors, ||W q ×f flatten_x ||×||W l ×f ' flatten_y || represents taking the modulus of each of the two vectors and then multiplying them; (37) Based on the positional relationship of highly similar sub-images, the deformation matrix is ​​calculated using the Homography algorithm to align the wax block and slice image, as follows: (37.1) Key matching pairs are generated and filtered based on cosine similarity. The cosine similarity threshold is set to 0.8, and high similarity sub-graph pairs are retained. For each sub-graph pair, the coordinates of its center point are taken as the matching point. Outlier filtering uses the RANSAC algorithm to pre-filter erroneous matches, improving the robustness of Homography solutions; background region filtering removes any matching blocks in the background of the wax block and slice sub-image to prevent background similarity from affecting the matching results. (37.2) Homography algorithm deformation matrix calculation: Solve the 3×3 deformation matrix H using the findHomography() function of OpenCV, and select RANSAC as the robust estimation algorithm. (37.3) Image deformation and interpolation: deform the slice image, call warpPerspective() to apply the H matrix to the slice image, and use bilinear interpolation to keep the image smooth, so as to obtain a slice image aligned with the wax block image.

4. The feature comparison model between slices and wax blocks according to claim 3, characterized in that, The feature mapping module between the sliced ​​image and the wax block image sub-image is trained as follows: (41) 300 pairs of wax block and slice images were selected as the dataset and the dataset was divided into training set and validation set. The 300 pairs of wax block and slice images were preprocessed before training by manual alignment. The tissue angle and tissue position were manually adjusted to achieve complete alignment of tissue in the image. (42) The slice and wax block feature comparison model learns the image feature relationship between the matching regions of slice images and wax block images through model construction and feature training, and represents the matching relationship through cosine similarity. (42.1) During model training, the wax block and slice image pairs in the training set are first divided into training and testing parts by the image feature matching model. During training, slice and wax block image pairs that are known to be from the same tissue are used, and the feature extraction module obtains the flattened feature set of the sub-image region, f. flatten and wax block sub-block features f ' flatten Each sub-image feature set has dimensions [1, 1536, 3, 3, 784]. The first dimension represents the number of images used simultaneously during each training session. The second dimension, 1536, represents the total number of features extracted for each sub-image region. The third and fourth dimensions, [3, 3], represent the size of each sub-image region in the high-dimensional space. The fifth dimension, 784, represents the total number of sub-image regions the image is divided into. (42.2) During model training, the similarity label for subgraphs at the same position is set to 1, and the similarity label for subgraphs at different positions is set to -1; the model training strategy is to take the subgraphs at the same position once, calculate the similarity, and then update the convolution kernel W. q and W l Then, take subgraphs at different positions, calculate the similarity again, and then update the convolution kernel W. q and W l The model iterates by comparing the subgraph positions once for each round, and then taking the subgraph position for the next round. (42.3) The loss function uses cosine similarity as the model's loss function. Based on the training and validation sets, the image alignment module is learned and trained. The image alignment module updates the model weights by minimizing the loss function until the loss function no longer decreases or tends to a stable value, thus obtaining the trained slice and wax block feature comparison model; the loss function is as follows: L consim =1-Cos_Similarity(f q ,f l ) Where f q For the features of a slice patch, f l Features of the wax block patch.

5. The method for determining the consistency of paraffin blocks and pathological slide images according to claim 4, characterized in that: The dual-tower Siamese neural network employs a contrastive learning framework, which uses a parameter-sharing dual-tower structure. Each tower consists of three parts: (51) Backbone network: The Transformer model with the Swing-Tiny architecture is used as the backbone network, and the weights are initialized using weights pre-trained on the ImageNet dataset; (52) Feature processing module: The features extracted by the backbone network are L2 normalized, and then the features are mapped to a unified dimension through two layers of MLP projection heads; (53) Loss function module: The model training is optimized using a contrastive loss function, specifically: Among them, D w =D w (F lakuai ,F quepian )=||F lakuai -F qiepian ||2 represents the Euclidean distance between two samples, and Y is the label indicating whether the two samples match. Y=1 means that the two samples match, and Y=0 means that the two samples do not match.

6. The method for determining the consistency of paraffin blocks and pathological slide images according to claim 5, characterized in that: The training process of the dual-tower twin neural network is as follows: (61) Adjust the wax block image and slice image to a resolution of 224×224 and perform scale normalization using the mean and standard deviation of [0.5,0.5,0.5]. (62) Based on the processed wax block image and slice image, construct positive and negative sample pairs in a 1:1 ratio. The positive sample pair consists of the wax block image and its corresponding slice image. The negative sample pairs are automatically generated based on the data. Among the negative samples, 80% of the wax block and slice sample pairs have similar sizes, and 20% of the sample pairs have random sizes. The positive and negative sample pairs are divided into training set, validation set and test set in an 8:1:1 ratio. (63) The dual-tower twin neural network was trained. During the training process, the AdamW optimizer was used, the initial learning rate was set to 1e-5, the cosine learning rate decay strategy was adopted, the batch size was set to 32, and a total of 20 epochs were trained. (64) The model weights are updated by minimizing the loss function until the loss function no longer decreases or tends to a stable value, and the trained twin neural network is obtained.

7. The method for determining the consistency of paraffin blocks and pathological slide images according to claim 6, characterized in that: The backbone network in the dual-tower twin neural network can also adopt ResNet-50, EfficientNet-B4, DenseNet-201, ConvNeXt-Tiny, or DINO-ViT architectures.

8. A slide consistency discrimination system based on paraffin block and pathological slide images, used to implement the slide consistency discrimination method based on paraffin block and pathological slide images as described in any one of claims 1 to 7, characterized in that, include: The positioning unit is used to receive the paraffin block image and pathological slide image to be compared acquired by the color camera, and uses the deep learning model YOLOv11 to realize the localization of the tissue region in the slide image and the paraffin block image based on the tissue size of the paraffin block image. The alignment unit is used to construct the image alignment module. It uses a pre-trained convolutional model to extract high-dimensional features from the wax block image and the slice image. The feature layer divides the wax block image and the slice image into sub-image patches. Through the trained feature mapping module, the matched wax block sub-image and slice sub-image regions have a high degree of cosine similarity. By comparing the cosine similarity between the wax block sub-image patch and the slice sub-image patch one by one, the corresponding positional relationship between the sub-images with high similarity is preserved. The deformation matrix is ​​calculated to achieve the alignment of the wax block image and the slice image. The discriminant output unit is constructed and trained using a dual-tower twin neural network. The aligned slices and wax block images are input into the trained dual-tower twin neural network for consistency discrimination, and the consistency discrimination result is output through the contrastive learning framework of the dual-tower twin neural network.

Images