A brain glioma whole section pathological image intelligent analysis method based on a double-path analysis framework

Abstract

The application discloses a brain glioma whole slice pathological image intelligent analysis method based on a double-path analysis framework, and specifically comprises the following steps: S1, whole slice image preprocessing, utilizing a channel threshold segmentation algorithm, dividing each whole slice image into different foreground tissue regions and background regions, cutting the foreground tissue regions into a plurality of fixed-size image blocks, extracting a feature vector of each image block, and forming a feature cluster; S2, establishing a dynamic sampling mechanism, utilizing a two-stage attention network to perform dynamic multi-instance learning on the feature cluster of each whole slice image, and finally obtaining slice-level feature representation of the whole slice image; and S3, quantifying tumor microenvironment spatial relationship information, obtaining feature representation of a spatial relationship quantization graph, and splicing the feature representation with the slice-level feature representation in the S2, to obtain a final prediction result. The application improves the prediction efficiency and robustness, and improves medical decision speed and quality.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to an intelligent analysis method for whole-section pathological images of gliomas based on a dual-path analysis framework. Background Technology

[0002] Gliomas are the most common primary intracranial tumors in adults, exhibiting varying degrees of invasiveness and prognosis. Isocitrate dehydrogenase (IDH) mutations in diffuse gliomas are the defining genetic event in adult diffuse gliomas, while telomerase reverse transcriptase (TERT) mutations are more prevalent in glioblastomas and oligodendrogliomas, suggesting a more aggressive tumor behavior. However, current molecular subtyping is primarily achieved through immunohistochemical molecular sequencing, which is not only resource-intensive but also time-consuming. Therefore, timely and accurate identification of glioma molecular stratification based on these biomarkers is crucial for providing more precise and targeted treatment.

[0003] Whole-section images, especially those stained with hematoxylin and eosin (HE), not only provide information about the tumor microenvironment but also directly reflect changes in the cell nucleus and tissue, making them the gold standard for the histological diagnosis of gliomas. With the help of artificial intelligence, this microscopic information can be rapidly extracted and used for accurate tumor diagnosis. However, some challenges remain in the analysis and modeling of whole-section images, particularly in how to effectively extract key tumor microenvironment features from ultra-high-resolution images.

[0004] Full-slice images are typically tens of thousands of pixels in size, and due to memory limitations, traditional deep learning methods cannot be directly trained end-to-end. To address this, a patch-based multi-instance learning approach has been proposed, allowing for the processing of full-slice images through weakly supervised network training, even with only slice-level labels. Among these, attention-based multi-instance learning, by further assigning weights to each instance within a packet, can better highlight the role of key instances. However, directly applying attention-based fusion to thousands of image patches reduces efficiency, and a large amount of redundant information hinders the extraction of crucial local information. Furthermore, many other instance learning methods neglect the information of the tumor microenvironment within the full-slice image and the spatial relationships between image patches. This oversight can lead to information loss, which is crucial in pathology diagnosis, as cells, tissues, or lesions often exhibit spatial local correlations within the image.

[0005] Therefore, with the increasing demand for tumor microenvironment characteristics, this invention aims to provide an intelligent analysis method for whole-section pathological images of gliomas based on a dual-pathway analysis framework. While providing high performance, it can effectively classify and identify whole-section pathological images, and the entire processing is completed automatically by computer without human intervention.

[0006] It is understood that the above statements only provide background information related to the present invention and do not necessarily constitute prior art. Summary of the Invention

[0007] The purpose of this invention is to provide an intelligent analysis method for whole-section pathological images of gliomas based on a dual-path analysis framework. While providing high performance, it can effectively classify and identify whole-section pathological images, and the entire processing is completed automatically by computer without human intervention.

[0008] To achieve the above objectives, this invention provides an intelligent analysis method for whole-slice pathological images of gliomas based on a dual-path analysis framework, specifically comprising the following steps: S1, preprocessing of the whole-slice images: using a channel threshold segmentation algorithm, each whole-slice image is divided into different foreground tissue regions and background regions, and the foreground tissue regions are cut into several image blocks of fixed size. Then, the feature vector of each image block is extracted to form a feature cluster; S2, establishing a dynamic sampling mechanism: using a two-stage attention network to perform dynamic multi-instance learning on the feature clusters of each whole-slice image, and finally obtaining the slice-level feature representation of the whole-slice image; S3, quantifying the spatial relationship information of the tumor microenvironment, obtaining the feature representation of the spatial relationship quantification map, and concatenating it with the slice-level feature representation in S2 to obtain the final prediction result.

[0009] Furthermore, in S1, dividing each full-slice image into different foreground tissue regions and background regions specifically includes the following steps: K1, reading the full-slice image, converting it to the HSV color space and performing median filtering; K2, performing binarization processing on the image under a set saturation channel threshold; K3, performing morphological closing operation on the binarized image in K2.

[0010] Furthermore, the saturation channel threshold ranges from 10 to 30.

[0011] Furthermore, the dynamic sampling mechanism includes central sampling and dynamic random sampling.

[0012] Furthermore, the two-stage attention network uses two parallel fully connected layers L and U to obtain low-level spatial features, and uses a softmax function and a mapping layer M to obtain attention scores.

[0013]

[0014] In the formula, a i Let be the attention score of the i-th image patch feature in the full slice image, · be the dot product, and tanh and sigm be the tanh function and sigmoid function, respectively. and Image patch features for different inputs

[0015] Furthermore, in the first-stage attention network, two dynamic sampling mechanisms are used to extract K image patch features of size 768 dimensions respectively. The two 768-dimensional features are then mapped to 512 dimensions through the projection layer of the first-stage attention network. Attention scores are calculated for the two 512-dimensional features respectively. The initial 768-dimensional image patch features are sorted according to the attention scores. The image patch features with higher scores extracted by the two sampling mechanisms are selected and concatenated to obtain a 768-dimensional concatenated feature representation, thus completing the first-stage attention network.

[0016] Furthermore, in the second-stage attention network, the dimension of the stitched-level feature representation is compressed again to 512 dimensions using the projection layer of the second-stage attention network, and the attention score of the corresponding i-th image patch feature is calculated, thereby obtaining the slice-level feature representation and completing the second-stage attention network calculation, as shown in the following formula:

[0017]

[0018] In the formula, h slide For slice-level feature representation, s j For attention score, This represents the splicing-level feature representation.

[0019] Furthermore, S3 specifically includes the following steps: Y1. Center sampling is performed on the M full-slice images in the training set, and K center image blocks are sampled from each slice to obtain M×K center image blocks. All center image blocks are then clustered using K-means to form G groups; Y2. Center image blocks are extracted from each group again using the center sampling method to obtain G representative image blocks; Y3. The similarity distance between all image blocks in each full-slice image and these G representative image blocks is calculated, and the representative image blocks with the closest similarity are assigned to the corresponding group; Y4. For each full-slice image in the training set, the proportion of image blocks in each group is calculated to obtain G... The proportional values ​​of all training set images are used to form an M×G matrix, and an M×1 label is used to perform a T-test to obtain G T-statistics. The T-statistics of the G image patch groups are then ranked to obtain new group values. Next, by assigning RGB color values ​​to the group values ​​of the image patches, the mapped image patches are placed back onto their corresponding positions in the full medical image dataset to obtain a spatial relationship quantization map. This spatial relationship quantization map is then encoded using a ResNet18 residual neural network to extract its latent features. Finally, the slice-level feature representation is concatenated with the feature representation of the spatial relationship quantization map and fed into a classifier for slice-level prediction to obtain the final prediction result.

[0020] Furthermore, the similarity distance is calculated using the cosine similarity distance formula:

[0021]

[0022] In the formula, A and B represent the feature vectors of the image patch.

[0023] In summary, this invention provides an intelligent analysis method for whole-section pathological images of gliomas based on a dual-path analysis framework. The dynamic multi-instance learning path, based on a two-stage attention mechanism, effectively selects and focuses on analyzing image information within key regions, minimizing interference from non-critical image blocks and improving the model's ability to extract key features of the tumor microenvironment. Secondly, the spatial relationship information quantification path utilizes spatial information quantification methods to enhance the model's ability to analyze spatial relationships between images, enabling the model to better understand and characterize the complexity of the tumor microenvironment. Finally, it automates the entire pathological image analysis process, significantly reducing the need for manual intervention and improving prediction efficiency and robustness. This makes the invention more convenient and efficient for clinical applications, contributing to faster and better medical decision-making. Furthermore, validation on public datasets demonstrates the significant advantages of this invention in improving the accuracy of glioma molecular subtyping and prognostic prediction. Attached Figure Description

[0024] Figure 1This is a flowchart illustrating the specific method of the present invention;

[0025] Figure 2 This is a schematic diagram of the Kaplan-Meier survival curve for the prognostic prediction results of this invention. Detailed Implementation

[0026] The following is in conjunction with the appendix Figures 1-2 The present invention will be further illustrated by describing a preferred embodiment in detail.

[0027] It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions. They are only used to facilitate and clarify the purpose of illustrating the embodiments of the present invention, and are not intended to limit the implementation conditions of the present invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationship, or adjustments to the size should still fall within the scope of the technical content disclosed in the present invention, provided that they do not affect the effects and objectives that the present invention can produce.

[0028] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only the expressly listed elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0029] like Figure 1 As shown, this invention provides an intelligent analysis method for whole-section pathological images of gliomas based on a dual-path analysis framework, specifically including the following steps:

[0030] S1. Preprocessing of the full slice image: Using the channel threshold segmentation algorithm, each full slice image is divided into different foreground tissue regions and background regions. The foreground tissue regions are then cut into several fixed-size image blocks. The feature vector of each image block is then extracted to form feature clustering.

[0031] S2. Establish a dynamic sampling mechanism, use a two-stage attention network to perform dynamic multi-instance learning for feature clustering of each full-slice image, and finally obtain the slice-level feature representation of the full-slice image;

[0032] S3. Quantify the spatial relationship information of the tumor microenvironment, obtain the feature representation of the spatial relationship quantification map, and splice it with the slice-level feature representation in S2 to obtain the final prediction result.

[0033] Specifically, in step S1, the full-slice image is first read, converted to the HSV color space, and then subjected to median filtering. Next, the image is binarized at a set saturation channel threshold, and morphological closing operations are performed on the binarized image to divide each full-slice image into foreground and background regions. The saturation channel threshold ranges from 10 to 30; in a preferred embodiment of the invention, the saturation channel threshold is 20.

[0034] Furthermore, the foreground tissue region is cut into non-overlapping image blocks of a fixed size of 256×256 pixels by a sliding window. Then, a pre-trained encoder is used to extract the features of the non-overlapping image blocks to obtain the corresponding feature vectors. Next, a clustering algorithm is used to divide the feature vectors into K feature clusters, where i = 1, 2, ..., K.

[0035] The features in the i-th feature cluster can be represented as: This can be named the feature cluster of the i-th feature, where d represents the number of image patch features contained in the i-th feature cluster, and 768 represents the feature dimension. Therefore, each full slice image can be represented as F = {C1, C2, ..., C}. K This can also be named the feature clustering pool F. Specifically, an image patch with an original dimension of 256×256×3 becomes a 768-dimensional feature vector after being encoded by the encoder (i.e., feature extraction of the image patch). Here, a pre-trained encoder based on the Transformer network structure is used, so the output dimension is 768.

[0036] Furthermore, in S2, the dynamic sampling mechanism has two types: center sampling and dynamic random sampling. Center sampling refers to extracting the image patch features at the cluster center of each category within the feature clustering pool F to represent the overall features of the cluster. Dynamic random sampling refers to randomly selecting an image patch feature from each feature cluster during each network iteration.

[0037] S2 comprises two stages. In the first stage, the attention network extracts K 768-dimensional image patch features for each full-slice image using two dynamic sampling mechanisms. Among them, F cs F is represented as the feature obtained through center sampling. drs This represents features obtained through dynamic random sampling; and to further enhance the ability of features to represent image patch information, a projection layer is used. The above 768-dimensional feature F cs ,F drs Mapped to 512 dimensions, as shown in the following formula:

[0038]

[0039] In the formula, layernorm is the normalization function, and ReLU is the activation function; thus, low-dimensional features are obtained. as well as in, This can be represented as the features obtained through center sampling being mapped from 768-dimensional features to 512-dimensional features; This is represented as the mapping of features obtained through dynamic random sampling from 768-dimensional features to 512-dimensional features; W p This is represented as the projection layer structure of the first-stage attention network.

[0040] Then, and The feature vector is input into a two-stage attention network, which uses two parallel fully connected layers. and To obtain the underlying spatial features, and use the softmax function and mapping layer To obtain attention score a i This score represents the impact of each image patch on the final full-slice image prediction; where the attention score of the i-th image patch feature in the full-slice image is determined by the input image patch features. The decision is as follows:

[0041]

[0042] In the formula, · represents the dot product, and tanh and sign are the tanh function and sigmoid function, respectively.

[0043] Then, based on the obtained attention score, the input image patch features F cs and F drs By sorting, we can obtain the top t image patches with the highest scores, i.e. and The concatenated feature representation obtained by concatenating these two features This completes the first stage of the attention network. We then begin the second stage of the attention network, representing the spliced-level features. As input to the attention network in the second stage.

[0044] Similarly, a projection layer is used in the second-stage attention network. The splicing level feature representation The feature dimension is compressed to 512 dimensions, where W q This is represented as the projection layer structure of the second-stage attention network, from which we can obtain... And calculate the attention score of the corresponding i-th image patch feature. Finally, the slice-level feature representation h can be obtained by weighted aggregation based on the attention scores of each image patch. slide This completes the second stage of attention network computation, as shown in the following formula:

[0045]

[0046] Furthermore, in S3, to mine tumor microenvironment information, center sampling is first performed on all whole-slice images (a total of M images) in the training set. K center image patches are sampled from each slice, resulting in M×K center image patches. All these center image patches are then clustered using K-means to divide them into G groups. Next, center image patches are extracted from each group again using the center sampling method, yielding G representative image patches. The cosine similarity distance between all image patches in each whole-slice image and these G representative image patches is calculated using the following formula, where A and B represent the feature vectors of the image patches. Each image patch is assigned to a corresponding group based on the representative image patch with the closest similarity. The formula is:

[0047]

[0048] Next, for each full-slice image in the training set, the proportion of image patches in each group is calculated, resulting in G proportion values. These proportion values ​​from all training set images are then combined into an M×G matrix, and a T-test is performed using M×1 labels to obtain G T-statistics. The larger the absolute value of the T-statistic, the more significant the difference between the image patches in that group and full-slice images of different categories, indicating a stronger discriminative ability for that image patch type. By ranking the T-statistics of the G image patch groups, new group values ​​are obtained.

[0049] Subsequently, by assigning RGB color values ​​to the group values ​​of the image patches, specifically mapping smaller group values ​​(such as group 1) to bluer image patches in the RGB image, and larger group values ​​(such as group 256) to redder image patches, the mapped image patches are placed back into their corresponding positions in the full slice image to obtain a spatial relationship quantization map. The color of the quantization map represents the correlation between the image patch and the classification result, and the spatial relationship quantization map needs to be encoded using a residual neural network ResNet18 to extract the latent features h of the spatial relationship quantization map. spatial .

[0050] Finally, the slice-level feature representation h obtained in S2 is... slide The feature representation h of the spatial relationship quantization map obtained in step S3 spatial The data is then stitched together and fed into a classifier for slice-level prediction to obtain the final prediction result.

[0051] It is understood that, in the preferred embodiment of the present invention, the total image M = 345, the sampled and extracted image blocks K = 100, and the number of groups G = 256.

[0052] Specifically, in a preferred embodiment of the present invention, the Adam optimizer is used for training, with a learning rate of 1e-4, weight decay of 5e-5, and dropout of 0.5 to reduce overfitting. Cross-entropy loss is used as the loss function for 100 iterations of training. The dataset is then divided into training, validation, and test sets in a 4:1:1 ratio. The model is trained using the training set and its performance is evaluated using the test set. Five-fold cross-validation is used on all datasets. To evaluate and compare classification performance, the present invention employs widely accepted evaluation metrics such as accuracy (ACC), F1 score, area under the receiver operating characteristic (ROC) curve (AUC), and C-index. The network is deployed using Python and PyTorch frameworks. Both training and inference processes are executed on workstations equipped with Intel Xeon Silver 4114 (20-core 2.20GHz) CPUs and NVIDIA GeForce TITAN V GPUs.

[0053] Experimental data analysis:

[0054] Table 1 shows the comparison results of the method of the present invention with four existing methods in molecular subtyping prediction and prognosis prediction of gliomas. The present invention employs two innovative technologies: dynamic multi-instance learning based on a two-stage attention mechanism and quantification of spatial relationship information of the tumor microenvironment. The present invention uses validation set data of 87 cases and independent test set data of 90 cases, and the images used are whole sections of gliomas stained with HE.

[0055] Compared to the image analysis methods used in Methods 1, 2, 3, and 4, this invention significantly improves the analytical capabilities of whole-slice images. The method of this invention performs exceptionally well in predicting key molecular markers such as IDH mutations, TERT mutations, and 1p / 19q co-deletion. It improves prediction accuracy by 2%–5% compared to existing technologies, the area under the ROC curve by 3%–6%, and the F1 score by 4%–8%, significantly outperforming any one of Methods 1, 2, 3, and 4 in all three evaluation metrics.

[0056] Table 2 shows the results of the method of the present invention in prognostic prediction. The C-index obtained by the inference is 5% to 8% higher than that obtained by WHO classification and IDH mutation status. In particular, combining the prediction results of the method of the present invention with two clinical information, WHO classification and IDH mutation status, can further improve the predictive ability of prognosis, and the C-index can be improved by about 1%.

[0057] Table 1. Prediction results of molecular typing using different methods (IDH, TERT, 1p / 19q)

[0058]

[0059] Table 2. Clinical information and C-index for prognostic prediction using the method of this invention.

[0060]

[0061] Figure 2 The figure shows the Kaplan-Meier survival curves for the prognostic prediction results of this invention. The figure illustrates the stratification quality between the low-risk and high-risk patient groups predicted by the model obtained in this invention for the dataset of this study. Patients are divided into high-risk and low-risk categories based on the risk scores generated by the model. The Log-rank test is used to evaluate the statistical difference between the two cohorts, and the p-value is less than 0.0001, indicating a highly significant statistical difference.

[0062] In summary, this invention presents an intelligent analysis method for whole-section pathological images of gliomas based on a dual-path analysis framework. First, the dynamic multi-instance learning path based on a two-stage attention mechanism effectively selects and focuses on analyzing image information within key regions, minimizing interference from non-critical image blocks and improving the model's ability to extract key features of the tumor microenvironment. Second, the spatial relationship information quantification path utilizes spatial information quantification methods to enhance the model's ability to analyze spatial relationships between images, enabling the model to better understand and characterize the complexity of the tumor microenvironment. Finally, the entire pathological image analysis process is automated, significantly reducing the need for manual intervention and improving prediction efficiency and robustness. This makes the invention more convenient and efficient for clinical applications, contributing to faster and better medical decision-making. Furthermore, validation on public datasets demonstrates the significant advantages of this invention in improving the accuracy of molecular subtyping and prognostic prediction of gliomas.

[0063] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A method for intelligent analysis of whole-section pathological images of gliomas based on a dual-path analysis framework, characterized in that, Specifically, it includes the following steps: S1. Preprocessing of the full slice image: Using the channel threshold segmentation algorithm, each full slice image is divided into different foreground tissue regions and background regions. The foreground tissue regions are then cut into several fixed-size image blocks. The feature vector of each image block is then extracted to form feature clustering. S2. Establish a dynamic sampling mechanism, use a two-stage attention network to perform dynamic multi-instance learning for feature clustering of each full-slice image, and finally obtain the slice-level feature representation of the full-slice image; S3. Quantify the spatial relationship information of the tumor microenvironment to obtain the feature representation of the spatial relationship quantification map, and splice it with the slice-level feature representation in S2 to obtain the final prediction result; In the first stage of the attention network, two dynamic sampling mechanisms are used to extract K image patch features of size 768 dimensions. The two 768-dimensional features are then mapped to 512 dimensions through the projection layer of the first stage attention network. Attention scores are calculated for the two 512-dimensional features. The initial 768-dimensional image patch features are sorted according to the attention scores. The image patch features with higher scores extracted by the two sampling mechanisms are selected and concatenated to obtain a 768-dimensional concatenated feature representation, thus completing the first stage of the attention network. In the second-stage attention network, the dimension of the stitched-level feature representation is compressed again to 512 dimensions using the projection layer of the second-stage attention network, and the attention score of the corresponding i-th image patch feature is calculated, thereby obtaining the slice-level feature representation and completing the second-stage attention network calculation, as shown in the following formula: ； In the formula, This represents slice-level features. For attention score, This represents the image block-level features.

2. The intelligent analysis method for whole-section pathological images of gliomas based on a dual-path analysis framework as described in claim 1, characterized in that, In step S1, dividing each full-slice image into different foreground tissue regions and background regions specifically includes the following steps: K1. Read the full slice image, convert it to the HSV color space and perform median filtering; K2. Binarize the image under the set saturation channel threshold; K3. Perform morphological closing operation on the binarized image in K2.

3. The intelligent analysis method for whole-section pathological images of gliomas based on a dual-path analysis framework as described in claim 2, characterized in that, The saturation channel threshold ranges from 10 to 30.

4. The intelligent analysis method for whole-section pathological images of gliomas based on a dual-path analysis framework as described in claim 1, characterized in that, The dynamic sampling mechanism includes central sampling and dynamic random sampling.

5. The intelligent analysis method for whole-section pathological images of gliomas based on a dual-path analysis framework as described in claim 1, characterized in that, The two-stage attention network uses two parallel fully connected layers. and To obtain the underlying spatial features, and use the softmax function and mapping layer To obtain attention scores, ； In the formula, Let be the attention score of the i-th image patch feature in the full slice image, · be the dot product, and tanh and sigm be the tanh function and sigmoid function, respectively. and These are image patch features from different inputs.

6. The intelligent analysis method for whole-section pathological images of gliomas based on a dual-path analysis framework as described in claim 1, characterized in that, S3 specifically includes the following steps: Y1. Perform center sampling on M full-slice images in the training set, sample K center image patches from each slice, obtain M×K center image patches, and perform K-means clustering on all center image patches again to divide them into G groups; Y2. Extract the central image block from each group again using the center sampling method to obtain G representative image blocks; Y3. Calculate the similarity distance between all image patches in each full slice image and these G representative image patches, and assign the representative image patches with the closest similarity to the corresponding group. Y4. For each full slice image in the training set, calculate the proportion of image patches in each group to obtain G proportion values. Form an M×G matrix with all the proportion values ​​of the training set images and perform a T-test using M×1 labels to obtain G T statistics. Rank the T statistics of the G image patch groups to obtain new group values. Y5. By assigning RGB color values ​​to the group values ​​of image blocks, the mapped image blocks are placed back into the corresponding positions of the full slice image to obtain a spatial relationship quantization map. The spatial relationship quantization map is then encoded using a residual neural network ResNet18 to extract the latent features of the spatial relationship quantization map. Y6. The slice-level feature representation is concatenated with the feature representation of the spatial relationship quantization map and fed into the classifier for slice-level prediction to obtain the final prediction result.

7. The intelligent analysis method for whole-section pathological images of gliomas based on a dual-path analysis framework as described in claim 1, characterized in that, The similarity distance is calculated using the cosine similarity distance formula: ； In the formula, A and B represent the feature vectors of the image patch.

Images