Instance segmentation method, device and readable storage medium for analysis of vascular invasion of early-stage intestinal cancer

Abstract

This paper proposes an instance segmentation method, device, and readable storage medium for analyzing vascular invasion in early-stage colorectal cancer. The method constructs an end-to-end automated process: first, digital pathology images are cropped at multiple magnifications and overlapped, and highly sensitive instance segmentation is performed using a Mask2former instance segmentation network. An efficient union-find algorithm is then used to merge overlapping masks, and an ROI expansion algorithm is used to recursively complete incomplete instances. Full-slice alignment of HE, CD-31, and D2-40 staining is achieved through rigid-nonrigid micro-registration, and instance-level tiles of the three modalities are cropped based on the registration results. Finally, a hybrid expert network enhanced by a large pathology model is used to fuse multimodal features and dynamically screen expert pathways for true and false positive discrimination. This invention provides an efficient technical solution for the precise diagnosis of early-stage colorectal cancer, suitable for pathology big data analysis and clinical auxiliary diagnosis.

Description

Technical Field

[0001] The present invention relates to the field of medical diagnosis technology, and in particular to an instance segmentation method, device and readable storage medium thereof for analyzing vascular invasion of early-stage intestinal cancer. Background Art

[0002] Colorectal cancer (CRC) is a malignant tumor. While the widespread use of early screening technologies has improved detection rates, vascular invasion (tumor cell invasion of blood vessels or lymphatic vessels) remains a significant challenge in clinical diagnosis. Vascular invasion is a key indicator of cancer metastasis and poor prognosis, and accurate detection is crucial for treatment selection. However, traditional pathological testing relies on subjective interpretation of hematoxylin and eosin (HE)-stained slides by pathologists, which is subject to significant interobserver variability (e.g., an international study shows a κ value of only 0.518). Furthermore, HE staining can easily confuse blood vessels and lymphatic vessels with similar structures (such as serum-filled lumens and carcinoma in situ), leading to misdiagnosis and missed diagnosis. Existing automated detection technologies are mostly based on single HE staining. Due to the blurred boundaries of blood vessels and lymphatic vessels and the small, irregular morphology of vascular invasion, low detection rates and high false-positive rates are common. Furthermore, the lack of efficient multimodal fusion (e.g., CD-31 and D2-40 immunohistochemical staining) and a fully automated analysis framework makes it difficult to meet the clinical demand for high-throughput, high-precision diagnosis.

[0003] Therefore, there is an urgent need for an instance segmentation method, device and readable storage medium thereof for analyzing the vascular invasion phenomenon of early-stage colorectal cancer to solve the problems existing in the prior art. Summary of the Invention

[0004] The embodiments of the present invention provide an instance segmentation method, device, and readable storage medium thereof for analyzing vascular invasion in early-stage colorectal cancer. These methods address the problems of current technologies, such as reliance on subjective interpretation by pathologists or single-modality detection, high misdiagnosis and missed diagnosis rates, large inter-observer variability, and insufficient detection capabilities for tiny / irregular vascular invasion structures.

[0005] The core technology of this invention is mainly based on a multimodal hybrid expert network enhanced by the Mask2Former instance segmentation network and a large pathology model. Through multi-scale image preprocessing, full-field mask merging post-processing, ROI expansion and completion, and multi-staining modality fusion classification, it can achieve end-to-end automated high-sensitivity detection and accurate identification of vascular invasion of early colorectal cancer.

[0006] In a first aspect, the present invention provides an instance segmentation method for analyzing vascular invasion of early-stage colorectal cancer, the method comprising the following steps:

[0007] S00, generating multi-scale image blocks adapted to the deep learning model input by overlapping and cropping the digital pathology image at multiple magnifications;

[0008] S10. Use the Mask2former instance segmentation network to perform instance segmentation of vascular invasion on multi-scale image blocks. The Mask2former instance segmentation network uses ResNet-101 as the backbone network, extracts four layers of multi-scale feature maps and inputs them into the segmentation decoding network. It then uses a preset number of query tokens and a multi-head attention mechanism to perform instance prediction.

[0009] S20. Merge the segmentation masks within the full field of view based on an efficient union-find algorithm, classify the masks with intersections as the same connected component and retain the maximum confidence score. Then, recursively expand the reasoning of instances with incomplete edges using the ROI expansion algorithm until there are no new incomplete instances.

[0010] S30, rigid registration, non-rigid registration, and micro-registration are performed on the HE-stained, CD-31-stained, and D2-40-stained whole-slice images through the registration algorithm to achieve instance-level alignment;

[0011] S40, using the suspected vascular invasion area on the HE staining as a reference, crop the same area in the registered CD-31 staining and D2-40 staining images to form a trimodal instance-level aligned image block;

[0012] S50. Input the trimodal instance-level aligned patches into the hybrid expert network enhanced by the pathology large model, dynamically select the expert pathway through the gated activation network, and fuse the multimodal features to perform true and false positive discrimination.

[0013] Furthermore, in step S00, the multiple magnifications include 2.5X, 5X, and 10X, the resolution of the image block is 1024×1024, and the overlapping area is 512 pixels.

[0014] Furthermore, in step S10, the Mask2former instance segmentation network uses a low confidence threshold during inference to retain more suspicious vascular invasion areas.

[0015] Furthermore, in step S20, the union-find algorithm processes overlapping masks through path compression and rank-based merging optimization strategy, and the merging threshold Tm is 0; the ROI expansion algorithm identifies incomplete edges by judging whether the ratio Re of the number of continuous overlapping points of the instance segmentation mask contour to the total number of points of the detection box line is greater than the threshold Te.

[0016] Furthermore, in step S30, the HE stained image is used as a reference image, the background is first filtered through foreground segmentation, and then rigid registration, non-rigid registration and micro-registration are performed in sequence to achieve full-slice-level alignment of different staining modalities.

[0017] Furthermore, in step S50, the pathology large model is a pre-trained VIT-Large model, the parameters of the first 23 layers of Transformer modules are frozen, and only the parameters of the last L-23 layers are updated to extract feature vectors for the three modal patches, where L is the total number of layers of the VIT model.

[0018] Furthermore, the hybrid expert network contains 4 experts, activating 2 experts for each staining modality. The activation probability of the expert pathway is calculated through a weight-sharing gated activation network. Additive aggregation is used for single-staining expert features, and splicing aggregation is used for multi-staining features.

[0019] In a second aspect, the present invention provides an instance segmentation device for analyzing vascular invasion of early-stage colorectal cancer, comprising:

[0020] The preprocessing module generates multi-scale image blocks that are suitable for deep learning model input by overlapping and cropping digital pathology images at multiple magnifications;

[0021] The instance segmentation module uses the Mask2former instance segmentation network to perform instance segmentation of vascular invasion on multi-scale image blocks. The Mask2former instance segmentation network uses ResNet-101 as the backbone network, extracts four layers of multi-scale feature maps, and inputs them into the segmentation decoding network. It then uses a preset number of query tokens and a multi-head attention mechanism to perform instance prediction.

[0022] The post-processing module merges the segmentation masks within the full field of view based on an efficient union-find algorithm, classifies the masks with intersections into the same connected component and retains the maximum confidence score. It then uses the ROI expansion algorithm to recursively expand the reasoning of instances with incomplete edges until there are no new incomplete instances.

[0023] The registration module uses a registration algorithm to perform rigid registration, non-rigid registration, and micro-registration on HE-stained, CD-31-stained, and D2-40-stained whole-slice images to achieve instance-level alignment.

[0024] The region cropping module uses the suspected vascular invasion area on the HE staining as a reference and crops the same area in the registered CD-31 staining and D2-40 staining images to form a trimodal instance-level aligned image patch;

[0025] The classification module inputs the trimodal instance-level aligned patches into the hybrid expert network enhanced by the pathology large model, dynamically selects the expert pathway through the gated activation network, and fuses multimodal features for true and false positive discrimination.

[0026] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the above-mentioned instance segmentation method for analyzing vascular invasion of early-stage intestinal cancer.

[0027] In a fourth aspect, the present invention provides a readable storage medium storing a computer program, wherein the computer program includes a program code for controlling a process to execute the process, and the process includes the above-mentioned instance segmentation method for analyzing the vascular invasion phenomenon of early intestinal cancer.

[0028] The main contributions and innovations of the present invention are as follows:

[0029] 1. Full process automation and efficiency:

[0030] An end-to-end automated analysis framework is constructed, which eliminates the need for human intervention from pathology image preprocessing and instance segmentation to multimodal classification decisions. This solves the problems of observer differences (for example, the κ value in international studies was only 0.518) and low diagnostic efficiency caused by traditional methods that rely on the subjective judgment of pathologists, and meets the needs of high-throughput pathology image analysis.

[0031] 2. High-sensitivity instance segmentation and detection rate improvement:

[0032] This approach utilizes a Mask2former instance segmentation network combined with a multi-scale overlapping cropping strategy, low-confidence threshold reasoning, and a Transformer multi-head attention mechanism to achieve high recall detection of small, irregular vascular invasion structures. Experimental data demonstrates that compared to existing methods (e.g., MaskRCNN+r101Recall = 89.1%), this method achieves a high recall of 96.3%, significantly reducing missed diagnoses.

[0033] 3. Multimodal combined diagnosis reduces false positives:

[0034] The system integrates three staining modalities—HE, CD-31, and D2-40—through full-slice registration and a hybrid expert network augmented by a large pathology model. This dynamic fusion of multimodal features enables true and false positive discrimination. Compared to HE staining alone (FPR = 0.542), this multimodal approach reduces the false positive rate to 38.8%, addressing the issue of misclassification caused by blurred blood / lymphatic vessel boundaries in HE staining.

[0035] 4. Post-processing optimization improves segmentation accuracy:

[0036] An efficient union-find algorithm is used to merge overlapping masks to reduce redundant detection. An ROI expansion algorithm is used to recursively complete instances of incomplete edges, ensuring complete detection of complex vascular invasions, avoiding contour loss caused by slice cropping, and improving the integrity and accuracy of segmentation results.

[0037] 5. Multi-scale adaptability and robustness:

[0038] In view of the large differences in the size of vascular invasion, 2.5X-10X multi-rate cropping and ResNet-101 multi-scale feature fusion are used to take into account both small-scale fine structures and large-scale tissue background, enhancing the model's robustness to different forms of vascular invasion, breaking through the traditional method's reliance on a single resolution or modality.

[0039] The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below so that other features, objects, and advantages of the invention are more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0041] Figure 1 is a flow chart of an instance segmentation method for analyzing vascular invasion of early-stage intestinal cancer according to an embodiment of the present invention;

[0042] Figure 2 is a flow chart of a method for performing ROI expansion post-processing on an instance segmentation network according to an embodiment of the present invention;

[0043] Figure 3 2 is a schematic diagram of a multi-stained slice registration result of constructing a multimodal hybrid expert classification network according to an embodiment of the present invention;

[0044] Figure 4 2. FIG. 1 is a diagram of a multimodal hybrid expert network architecture enhanced by a large pathology model according to an embodiment of the present invention;

[0045] Figure 5 4 is a result diagram of instance segmentation of suspected vascular invasion using the Mask2former instance segmentation model under HE staining according to an embodiment of the present invention;

[0046] Figure 6 1 is a schematic diagram of image blocks cropped based on corresponding regions of CD-31 and D2-40 staining based on an instance segmentation network under post-processing HE staining according to an embodiment of the present invention;

[0047] Figure 7 FIG. 4 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0049] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0050] Existing technologies for detecting vascular invasion in early-stage colorectal cancer lack fully automated processes, suffer from interobserver variability, and lead to lengthy diagnostic times for pathologists. Therefore, highly sensitive automated vascular invasion detection technologies are urgently needed.

[0051] Based on this, the present invention solves the problems existing in the prior art by using a multimodal hybrid expert network enhanced by a Mask2former network and a large pathological model.

[0052] Example 1

[0053] This paper proposes an instance segmentation method for analyzing vascular invasion of early colorectal cancer. Specifically, a workstation equipped with an RTX3090 GPU or more advanced models is used for model reasoning and analysis to ensure that the computing resources meet the requirements of large-scale instance segmentation and multimodal joint classification. Figure 1 , the method comprising:

[0054] S00, Data Preprocessing

[0055] The digital pathology images are cropped at multiple magnifications through overlapping to generate multi-scale image patches that are suitable for the deep learning model input; the details are as follows:

[0056] 1) Sample preparation:

[0057] 80X high-resolution digital pathology images of colorectal cancer were collected as input data for the analysis of vascular invasion of early-stage colorectal cancer.

[0058] 2) Image Cropping:

[0059] The entire image is segmented into 1024×1024 pixel blocks using overlapping cropping technology at multiple magnifications of 2.5X, 5X, and 10X, with an overlap size of 512 pixels, and then normalized to adapt to the input requirements of the Mask2former segmentation model.

[0060] S10, low confidence threshold vascular invasion instance segmentation:

[0061] The Mask2former instance segmentation network is used to perform instance segmentation of vascular invasion on multi-scale image blocks. The Mask2former instance segmentation network uses ResNet-101 as the backbone network, extracts four layers of multi-scale feature maps, and inputs them into the segmentation decoding network. Instance prediction is performed using a preset number of query tokens and a multi-head attention mechanism. The details are as follows:

[0062] 1) Instance segmentation execution:

[0063] The instance segmentation algorithm can obtain the number and outline of vascular invasion, providing instance-level information for subsequent multimodal classification. The pre-trained ResNet-101 is used as the front backbone network of the vascular invasion instance segmentation network, and the overall instance segmentation network uses the Mask2former architecture. The pathological image block containing vascular invasion is first encoded into a low-resolution feature vector by the feature extractor ResNet-101, and then input into the pixel decoder. The pixel decoder is a fully connected neural network that can decode the feature vector into pixel-level prediction results, which can better preserve image details compared to the traditional decoder that decodes to the original image size. The pixel-level decoder decodes the features to 4 different scales, and then sends the feature maps of the first 3 scales to the Transformer decoder for decoding. The Transformer decoder is a decoding module composed of a stack of mask attention and self-attention. The formula of the self-attention mechanism is:

[0064] (1)

[0065] X l Indicates the l The output of the layer. Q l , K l , V l Represent the query, key, and value matrices, respectively, which are usually linear transformations of the current layer input. is the transpose of the key matrix. Softmax is a normalization function used to convert the attention scores into a probability distribution. Indicates thel The input of the layer is the output of the previous layer. This formula describes the calculation process of the self-attention layer, where softmax( ) calculates the attention weights, which are then used to weight the sum value matrix V l Finally, the result of this weighted sum is combined with the input Adding , we can achieve residual connection. Residual connection helps to alleviate the gradient vanishing problem in deep networks and promote the flow of information. In practical applications, this formula is usually accompanied by some scaling factors, such as d k (in d k is the dimension of the key vector) to prevent the softmax function’s input from being too large, causing the gradient to vanish. The formula for the masked attention mechanism is:

[0066] (2)

[0067] (3)

[0068] In masked cross attention, the feature map serves as the Value vector and the Key vector, and the target query vector serves as the Query vector. Through masking operations, the cross attention can be limited to the foreground area of the predicted mask rather than the complete feature map. After cross attention fusion, the global dependency is captured through the self-attention mechanism. The Transformer decoder realizes multi-scale target detection by processing feature maps of different resolutions. In addition to multi-scale target detection at a fixed magnification, the present invention also enhances the multi-scale adaptability of the model in a wider range by inputting pictures of different magnifications into the network. The instance segmentation framework adopts the cross entropy loss function and the Dice loss function during the training process. During inference, multi-scale 1024x1024 tiles (with an overlapping area of 512) of 2.5X, 5X and 10X are input for multi-scale inference.

[0069] S20. Post-processing of vascular invasion instance segmentation results:

[0070] The segmentation masks within the full field of view are merged based on an efficient union-find algorithm. Masks with intersections are classified as the same connected component and the maximum confidence score is retained. Instances with incomplete edges are then recursively expanded using the ROI expansion algorithm until there are no new incomplete instances. The details are as follows:

[0071] 1) Post-processing and merging of instance segmentation results based on union-find:

[0072] Under the conditions of multi-scale reasoning and overlapping reasoning in step S10, many overlapping areas will be generated in the full field of view of the pathological image, which will affect the subsequent judgment of the pathologist.

[0073] This invention relates to a union-find algorithm for pathology image analysis. This algorithm aims to address the issue of the large number of overlapping regions within the full field of view of a pathology image, which can affect the pathologist's subsequent judgment under multi-scale and overlapping reasoning conditions. Due to the large size of pathology images, the inference results contain numerous masked regions. This invention proposes using an efficient union-find algorithm to merge these masked regions, object detection boxes, and confidence scores.

[0074] In the specific implementation process, a unique identifier is first assigned to each mask, target detection box and corresponding confidence score, and the set is initialized and found so that each element initially exists independently as its own root node. Then, the connected component to which each element belongs is determined by performing a search operation. If two masks are found to have an intersection ( IOU > T m ), the masked regions are classified as the same connected component through a merge operation. During the merge, the maximum confidence score between the two masks or object detection boxes is selected as the confidence score for the merged mask or object detection box. Due to the significant differences in the scale of vascular invasions, large vascular invasions receive high confidence when inferred at low magnification, while small vascular invasions receive high confidence when inferred at high magnification. Therefore, when merging overlapping vascular regions, selecting the maximum confidence score as the confidence score for the merged suspected vascular invasion instance aligns with the confidence distribution characteristics of multi-scale reasoning.

[0075] in T m is the threshold, which is an adjustable hyperparameter during post-processing. In this application case, it is set to 0. However, for some specific scenarios, such as densely distributed vascular invasion, T m The value can be set to a value greater than 0 to avoid merging different vascular invasion instances. In addition, to further improve the efficiency of the algorithm, the present invention also adopts the optimization strategy of path compression and rank merging to reduce the complexity of the search and merging operations, ensuring that the algorithm can efficiently and accurately process overlapping areas in pathology images, thereby providing pathologists with clearer and more accurate analysis results.

[0076] 2) ROI-based extended mask completion:

[0077] refer to Figure 2, the present invention provides a method based on ROI expansion mask completion, which is used to deal with the problem of incomplete edge instances that may occur when cutting blocks from the full-field pathology image for reasoning. Under the conditions of multi-scale reasoning and overlapping reasoning, many overlapping areas will be generated in the full-field pathology image, and these overlapping areas may affect the subsequent judgment of the pathologist. Therefore, the present invention first detects the incomplete edge, and then expands the area to the outside of the incomplete edge to obtain a new block for reasoning until a complete edge is obtained. The ROI expansion algorithm is based on an assumption: for the same vascular invasion instance, reasoning under the complete field of view will obtain a higher confidence than reasoning under the local field of view. Therefore, for the prediction confidence after ROI expansion C e If it is greater than or equal to the confidence of the original prediction C o , then use the new prediction result to replace the old prediction result. If C e Less than C o This process ensures the accuracy of the reasoning results and avoids the impact of incomplete edge instances on pathological diagnosis, thereby improving the reliability and effectiveness of pathological image analysis. The criterion for incomplete edges is the number of consecutive overlapping points of the straight line corresponding to the detection box in the instance segmentation mask contour. N o Total number of points in line with the detection frame N a Ratio R e ( N o / N a ) is greater than T e 。 in T e is the threshold, which is an adjustable hyperparameter in the post-processing process and is set to 0.2 in this application case.

[0078] S30, rigid registration:

[0079] The registration algorithm is used to perform rigid registration, non-rigid registration, and micro-registration on the whole-slide images stained with HE, CD-31, and D2-40 to achieve instance-level alignment. The details are as follows:

[0080] In order to align pathological sections of different staining modalities, the present invention uses a full-slice-level image registration algorithm to perform full-slice-level registration on HE, CD-31, and D2-40 sections. The registration process is independent of steps S00, S10, and S20.

[0081] Figure 3The results of the multi-stained slide alignment algorithm are shown. Using the HE slide as the reference image, the CD-31 and D2-40 slides were aligned to the HE image. First, a foreground segmentation algorithm was applied to the WSI image to filter out the background blank areas. This process generates a mask by calculating the difference between the color of each pixel and the background color. The specific steps are as follows:

[0082] First, the image is converted to the CAM16-UCS color space to obtain three channels: L (brightness), A, and B. For brightfield images, the background is assumed to be bright, so the background color is defined as the average LAB value of pixels with brightness higher than 99% of all pixels.

[0083] Next, the Euclidean distance between the LAB color of each pixel and the background LAB color is calculated to generate a new image D, where a larger value indicates that the pixel color is more different from the background color.

[0084] Then, Otsu threshold segmentation is applied to the image D, and pixels above the threshold are identified as foreground, thus generating a binary mask. The image is then normalized and features are detected in each image.

[0085] The purpose of normalization is to make the images to be registered as similar as possible. For immunofluorescence (IF) images, the DAPI channel is the best choice for registration. However, if immunohistochemistry (IHC) images are used, normalization preprocessing is required to make them look more similar. The specific steps include:

[0086] Convert the RGB image to the CAM16-UCS color space, set parameters, convert it back to RGB, and then convert it to grayscale and invert it to darken the background and lighten the tissue. This method is more accurate and robust than simple grayscale conversion or histogram equalization.

[0087] After preprocessing, all images (IHC and / or IF) were further normalized to make the pixel intensity distribution more similar. A cubic interpolation fit was performed by calculating the 5th, mean, and 95th percentile of the pixel values. Total variation (TV) denoising was used to slightly smooth the images to preserve edges and reduce noise interference.

[0088] Clustering and ranking are then used to align features in the reference image and calculate a preliminary rigid registration transformation. The rigid registration process uses the BRISK detector and VGG descriptor to extract features and removes most outliers through brute force matching and RANSAC.

[0089] To further improve accuracy, the Tukey boxplot method is used to filter out mismatched points. Subsequently, the registration algorithm constructs a similarity matrix based on the number of feature matches and infers the optimal image order through hierarchical clustering, so that each image is surrounded by its two most similar images. After determining the image order, the images are sequentially aligned to a reference image (such as the middle image) to form a continuous registration sequence. To ensure the stability of tissue structure matching, the algorithm incorporates neighborhood matching filtering, retaining only feature points shared within a local image window.

[0090] Finally, mutual information can be optionally incorporated to optimize alignment accuracy. After rigid registration, non-rigid registration is performed at a higher resolution to correct for morphological changes and distortions in tissue pathology. Finally, micro-registration is performed based on non-rigid registration. Micro-registration, also a non-rigid registration, aims to correct for minor deformations and distortions at higher resolutions. After the registration network, the aligned WSIs of different staining modalities can achieve instance-level alignment of vascular invasion, laying the foundation for the subsequent three-modal true and false vessel classification network.

[0091] S40, multiple staining and cropping of suspected vascular invasion areas:

[0092] Using the suspected vascular invasion area on the HE staining as a reference, the same area is cropped from the registered CD-31 and D2-40 staining images to form a trimodal instance-level aligned image block; the details are as follows:

[0093] Figure 5 The Mask2former instance segmentation model demonstrates the instance segmentation results of suspected vascular invasion under HE staining. Based on the full-slice level registration results in step S30, the multi-stained suspected vascular invasion region pairing is obtained by cropping the corresponding coordinates of the suspected vascular invasion.

[0094] Since the registration process aligns the spatial positions of the differently stained sections, the coordinates of the suspected vascular invasion under HE staining correspond to the cropped areas under immunohistochemical staining with the same anatomical range and spatial position. Figure 6 The figure shows a schematic diagram of the corresponding area cropped by the instance segmentation network based on post-processed HE staining on CD-31 and D2-40 staining. The figure shows that the multi-stained vascular invasion areas at the tile level are well aligned, providing a basis for the subsequent construction of the multimodal classification model.

[0095] S50, Pathology Large Model Enhanced Multimodal Hybrid Expert Classification Framework:

[0096] The trimodal instance-level aligned patches are fed into a hybrid expert network augmented by a large pathology model. The expert pathway is dynamically selected through a gated activation network, and multimodal features are integrated to perform true and false positive discrimination. The details are as follows:

[0097] Figure 4 The multimodal hybrid expert network architecture enhanced by the pathology large model is demonstrated. The general pathology large model is self-supervised pre-trained on a large-scale pathology image dataset, thereby gaining the ability to represent general pathology features. Based on the capabilities of the pre-trained pathology basic model, more accurate true and false positive identification of vascular invasion can be achieved. Due to the large differences in the scale of vascular invasion itself, the scale of the cropped suspicious vascular invasion area is not uniform. Therefore, the present invention limits the size of the suspicious vascular invasion block to the range of 224x224 to 1024x1024 through adaptive level selection of the pathology image pyramid.

[0098] For vascular invasion images with different staining, the present invention uses a pathology basic model that does not share weights to extract pathology features. In order to allow the pathology basic model to have a better feature space perception of vascular invasion data, the model freezes the previous state in the pathology basic model (VIT architecture) during the training phase. N The parameters of the Transformer module are only given to the last LN The ability to update the parameters of the Transofmer module enables it to achieve domain adaptation during the training process, where L is the number of layers of the Transformer in the VIT model. The suspected vascular invasion blocks stained with CD-31, D2-40 and HE are used to obtain the corresponding feature vectors through the pathological basic model. E CD-31 , E D2-40 and E HE , the encoding dimension of the feature vector is D , D It depends on the encoding dimension of the pathology model. The feature vectors of different staining are first passed through the weight-sharing gated activation network to obtain the activation probability of different expert pathways. The number of experts in the mixed expert model is set to N e , the number of single-color modality activation experts is N s ( N s <=N e ). First, the features of the same coloring by different experts are fused, and then the mixed expert features of different colorings are fused. The expert network uses an input dimension of D i , the hidden layer dimension is D h , the output layer dimension is D o The training process uses random dropout regularization to prevent overfitting.

[0099] 1) Model training parameter table:

[0100] Table 1

[0101]

[0102] 2) Training dataset construction table:

[0103] Table 2

[0104]

[0105] 3) Experimental verification:

[0106] As shown in Table 3, the Mask2former vascular invasion screening network based on multi-scale inference has an extremely high vascular invasion detection rate.

[0107] Table 3

[0108]

[0109] Note: ms stands for multi-scale reasoning (multi-scale)

[0110] As shown in Table 4, the false positive detection rate of vascular invasion can be reduced based on the combined prediction of multiple staining.

[0111] Table 4

[0112]

[0113] 4) Data preservation and visual analysis:

[0114] After true and false positive vessel inference in step S30, a large number of false positives (blood vessels, glands, etc.) inferred by the mask2former model in step S10 can be eliminated by setting an appropriate confidence threshold. All results for a single pathology slide are then stored in a pickle file format, containing the confidence score, target detection bounding box, and segmentation mask for each vascular invasion instance. This invention provides a format conversion tool that converts the pickle results into a format compatible with Qupath Engineering Software for visual analysis.

[0115] Through the above steps, the present invention realizes fully automated analysis of vascular invasion. The complete process from suspicious vessel extraction to multimodal classification ensures sensitivity and accuracy, and provides reliable data support and technical tools for pathological research and diagnosis.

[0116] The present invention is applicable to various pathology laboratories, hospitals and research institutions, and has broad application prospects in the following fields:

[0117] 1. Vascular invasion diagnosis: By performing highly sensitive instance segmentation of vascular invasion in early colorectal cancer and multimodal joint optimization of false positive screening, a reliable basis is provided for early diagnosis and treatment planning of colorectal cancer.

[0118] 2. Multi-organ generalization: Vascular invasion phenomena in different tissues and organs are similar, so the vascular invasion detection algorithm can be fine-tuned and migrated to multiple organs for application.

[0119] 3. Pathology big data processing: It has the ability to process massive amounts of high-resolution pathology images, providing a solid technical foundation for pathology artificial intelligence research and big data analysis.

[0120] 4. Basic medical research: Provide efficient tools for colorectal pathology and ultrastructural biology research, and promote in-depth research on the pathogenesis of basement membrane-related diseases.

[0121] 5. Education and Training: Provide automated auxiliary tools for pathology teaching and training, reduce the difficulty of learning complex microsurgery operations, and improve teaching efficiency.

[0122] This invention is based on efficient and precise automation technology and is widely applicable to various application scenarios in the medical and scientific research fields. It has important practical value and promotion prospects.

[0123] Example 2

[0124] Based on the same concept, the present invention also proposes an instance segmentation device for analyzing vascular invasion of early-stage colorectal cancer, comprising:

[0125] The preprocessing module generates multi-scale image blocks that are suitable for deep learning model input by overlapping and cropping digital pathology images at multiple magnifications;

[0126] The instance segmentation module uses the Mask2former instance segmentation network to perform instance segmentation of vascular invasion on multi-scale image blocks. The Mask2former instance segmentation network uses ResNet-101 as the backbone network, extracts four layers of multi-scale feature maps, and inputs them into the segmentation decoding network. It then uses a preset number of query tokens and a multi-head attention mechanism to perform instance prediction.

[0127] The post-processing module merges the segmentation masks within the full field of view based on an efficient union-find algorithm, classifies the masks with intersections into the same connected component and retains the maximum confidence score. It then uses the ROI expansion algorithm to recursively expand the reasoning of instances with incomplete edges until there are no new incomplete instances.

[0128] The registration module uses a registration algorithm to perform rigid registration, non-rigid registration, and micro-registration on HE-stained, CD-31-stained, and D2-40-stained whole-slice images to achieve instance-level alignment.

[0129] The region cropping module uses the suspected vascular invasion area on the HE staining as a reference and crops the same area in the registered CD-31 staining and D2-40 staining images to form a trimodal instance-level aligned image patch;

[0130] The classification module inputs the trimodal instance-level aligned patches into the hybrid expert network enhanced by the pathology large model, dynamically selects the expert pathway through the gated activation network, and fuses multimodal features for true and false positive discrimination.

[0131] Example 3

[0132] This embodiment also provides an electronic device, referring to Figure 7 , includes a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0133] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits for implementing the embodiments of the present invention.

[0134] Memory 404 may include a large-capacity memory 404 for data or instructions. By way of example, and not limitation, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to the data processing device. In certain embodiments, memory 404 is non-volatile memory. In certain embodiments, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0135] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .

[0136] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any one of the instance segmentation methods for analyzing vascular invasion of early-stage intestinal cancer in the above embodiments.

[0137] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0138] Transmission device 406 can be used to receive or transmit data via a network. Specific examples of such networks may include wired or wireless networks provided by the electronic device's communications provider. In one embodiment, the transmission device includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 406 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0139] The input / output device 408 is used to input or output information.

[0140] Example 4

[0141] This embodiment also provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process. The process includes an instance segmentation method for analyzing vascular invasion phenomenon of early intestinal cancer according to embodiment one.

[0142] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be repeated here.

[0143] In general, various embodiments may be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0144] The embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components that are configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, it should be noted at this point that, for example, Figure 1 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or memory blocks implemented within the processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0145] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The above embodiments merely illustrate several embodiments of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of the present invention. Therefore, the scope of the present invention shall be determined by the appended claims.

Claims

1. An instance segmentation method for analyzing vascular invasion of early-stage colorectal cancer, characterized in that: The following steps are involved: S00, generating multi-scale image blocks adapted to the deep learning model input by overlapping and cropping the digital pathology image at multiple magnifications; S10, performing instance segmentation of vascular invasion on the multi-scale image block using a Mask2former instance segmentation network, wherein the Mask2former instance segmentation network uses ResNet-101 as a backbone network, extracts four layers of multi-scale feature maps, inputs them into a segmentation decoding network, and performs instance prediction using a preset number of query tokens and a multi-head attention mechanism; S20. Merge the segmentation masks within the full field of view based on an efficient union-find algorithm, classify the masks with intersections as the same connected component and retain the maximum confidence score. Then, recursively expand the reasoning of instances with incomplete edges using the ROI expansion algorithm until there are no new incomplete instances. Among them, the union-find algorithm processes overlapping masks through path compression and rank-based merging optimization strategy, and the merging threshold T m The ROI expansion algorithm identifies incomplete edges by judging whether the ratio Re of the number of consecutive overlapping points of the instance segmentation mask contour to the total number of points of the detection box line is greater than the threshold Te; S30, rigid registration, non-rigid registration, and micro-registration are performed on the HE-stained, CD-31-stained, and D2-40-stained whole-slice images through the registration algorithm to achieve instance-level alignment; S40, using the suspected vascular invasion area on the HE staining as a reference, crop the same area in the registered CD-31 staining and D2-40 staining images to form a trimodal instance-level aligned image block; S50, inputting the trimodal instance-level aligned image blocks into a hybrid expert network enhanced by a large pathology model, dynamically selecting expert pathways through a gated activation network, and fusing multimodal features to perform true and false positive discrimination; Among them, the pathology large model is a pre-trained VIT-Large model, which freezes the parameters of the first 23 layers of Transformer modules and only updates the parameters of the last L-23 layers to extract feature vectors for the three modal blocks, where L is the total number of layers of the VIT model; the hybrid expert network contains 4 experts, activating 2 experts for each staining modality, and calculates the expert pathway activation probability through a weight-sharing gated activation network. Additive aggregation is used for single-staining expert features, and splicing aggregation is used for multi-staining features.

2. The instance segmentation method for analyzing vascular invasion of early-stage colorectal cancer according to claim 1, characterized in that: In step S00, the multiple magnifications include 2.5X, 5X and 10X, the resolution of the image block is 1024×1024, and the overlapping area is 512 pixels.

3. The instance segmentation method for analyzing vascular invasion of early-stage colorectal cancer according to claim 1, characterized in that: In step S10, the Mask2former instance segmentation network adopts a low confidence threshold during reasoning to retain more suspicious vascular invasion areas.

4. The instance segmentation method for analyzing vascular invasion of early-stage colorectal cancer according to claim 1, characterized in that: In step S30, the HE stained image is used as the reference image, and the background is first filtered through foreground segmentation, and then rigid registration, non-rigid registration and micro-registration are performed in sequence to achieve full-slice-level alignment of different staining modalities.

5. An instance segmentation device for analyzing vascular invasion of early-stage colorectal cancer, characterized in that: include: The preprocessing module generates multi-scale image blocks that are suitable for deep learning model input by overlapping and cropping digital pathology images at multiple magnifications; An instance segmentation module uses a Mask2former instance segmentation network to perform instance segmentation of vascular invasion on the multi-scale image blocks. The Mask2former instance segmentation network uses ResNet-101 as the backbone network, extracts four layers of multi-scale feature maps, inputs them into the segmentation decoding network, and performs instance prediction using a preset number of query tokens and a multi-head attention mechanism; The post-processing module merges the segmentation masks within the full field of view based on an efficient union-find algorithm, classifies the masks with intersections into the same connected component and retains the maximum confidence score, and then recursively expands the incomplete edge instances through the ROI expansion algorithm until there are no new incomplete instances; the union-find algorithm processes overlapping masks through path compression and rank-based merging optimization strategy, and the merging threshold T m The ROI expansion algorithm identifies incomplete edges by judging whether the ratio Re of the number of consecutive overlapping points of the instance segmentation mask contour to the total number of points of the detection box line is greater than the threshold Te; The registration module uses a registration algorithm to perform rigid registration, non-rigid registration, and micro-registration on HE-stained, CD-31-stained, and D2-40-stained whole-slice images to achieve instance-level alignment. The region cropping module uses the suspected vascular invasion area on the HE staining as a reference and crops the same area in the registered CD-31 staining and D2-40 staining images to form a trimodal instance-level aligned image patch; The classification module inputs the instance-level aligned tiles of the three modalities into the hybrid expert network enhanced by the pathology large model, dynamically selects expert pathways through the gated activation network, and fuses multimodal features to perform true and false positive discrimination; wherein, the pathology large model is a pre-trained VIT-Large model, the parameters of the first 23 layers of Transformer modules are frozen, and only the parameters of the last L-23 layers are updated to extract feature vectors for the three modal tiles respectively, where L is the total number of layers of the VIT model; the hybrid expert network contains 4 experts, 2 experts are activated for each staining modality, and the expert pathway activation probability is calculated through a weight-sharing gated activation network. Additive aggregation is used for single-staining expert features, and splicing aggregation is used for multi-staining features.

6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the instance segmentation method for analyzing vascular invasion of early-stage intestinal cancer according to any one of claims 1 to 4.

7. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, wherein the process includes the instance segmentation method for analyzing vascular invasion phenomenon of early-stage intestinal cancer according to any one of claims 1 to 4.

Images